Differential roles of delay-period neural activity in the monkey dorsolateral prefrontal cortex in visual–haptic crossmodal working memory

Significance

Neural activity was recorded from the dorsolateral prefrontal cortex (DLPFC) when a monkey performed visuo–haptic crossmodal and haptic–haptic unimodal delayed matching-to-sample (cue) tasks. Results indicate that neural networks in the DLPFC function sequentially in the crossmodal task from visual stimulus encoding and crossmodal information transferring between visual and tactile stimuli to the behavioral action. Our findings may clarify the neural mechanisms by which the cerebral cortex stores information in working memory, a cognitive function of prime importance in the coordination of behavior, speech, and reasoning.

Abstract

Previous studies have shown that neurons of monkey dorsolateral prefrontal cortex (DLPFC) integrate information across modalities and maintain it throughout the delay period of working-memory (WM) tasks. However, the mechanisms of this temporal integration in the DLPFC are still poorly understood. In the present study, to further elucidate the role of the DLPFC in crossmodal WM, we trained monkeys to perform visuo–haptic (VH) crossmodal and haptic–haptic (HH) unimodal WM tasks. The neuronal activity recorded in the DLPFC in the delay period of both tasks indicates that the early-delay differential activity probably is related to the encoding of sample information with different strengths depending on task modality, that the late-delay differential activity reflects the associated (modality-independent) action component of haptic choice in both tasks (that is, the anticipation of the behavioral choice and/or active recall and maintenance of sample information for subsequent action), and that the sustained whole-delay differential activity likely bridges and integrates the sensory and action components. In addition, the VH late-delay differential activity was significantly diminished when the haptic choice was not required. Taken together, the results show that, in addition to the whole-delay differential activity, DLPFC neurons also show early- and late-delay differential activities. These previously unidentified findings indicate that DLPFC is capable of (i) holding the coded sample information (e.g., visual or tactile information) in the early-delay activity, (ii) retrieving the abstract information (orientations) of the sample (whether the sample has been haptic or visual) and holding it in the late-delay activity, and (iii) preparing for behavioral choice acting on that abstract information.

Working memory (WM) is a central concept in cognitive sciences. The prefrontal cortex constitutes the highest stage in the cortical hierarchy of executive memories (1–5), and it seems to be essential for integrating sensory information of different modalities with subsequent action in goal-directed behavior (6–9).

Cells involved in WM (“memory cells”) were first recorded in the dorsolateral prefrontal cortex (DLPFC) of monkeys performing delayed-response tasks (10–12) and have also been reported by other subsequent primate studies (13–19). The persistent delay activity recorded in those studies reflects the maintenance of a working-memory representation and therefore underlies the representation of retrospective, current, and prospective information (20). From the results of those studies, it seems that, in addition to persistent delay activity that is sustained throughout the whole delay period in WM tasks, task/set cells, eye movement-related responses, and phasic sensory responses, etc. (14–18, 21), two other general types of prefrontal neurons have also been studied (22, 23). One is the so-called sensory-coupled cue cell, the discharge of which tends to diminish during the delay period of WM tasks. The other is the preparatory-set cell; its discharge tends to increase as the time for an expected behavioral response of a WM task approaches. These two types of cells may participate in two complementary processes: Sensory-coupled cells hold information of stimuli, and preparatory-set cells prepare for action in response to that information. These findings imply that the DLPFC plays a critical role in temporal organization of behavior by integrating action with recent sensory information across time (24).

Cells in the DLPFC have been shown to be attuned to stimuli of different modalities in memory tasks, such as colors (25–27), tactile vibrations (19), and tones (28). Functional imaging and event-related potential studies have also shown DLPFC activity in processing information from different modalities (29–33). In addition, monkeys with lesions in banks and depths of the arcuate sulcus (the posterior end of the DLPFC) were impaired in performance of a tactile–visual crossmodal matching task (34).

In line with these reports, DLPFC neurons have been revealed to be able to associate a visual stimulus with an auditory stimulus across time (35). In this pioneer study, cells in the DLPFC responded selectively to auditory stimuli, and most of them also responded to visual stimuli according to the task rule (crossmodal associations). A similar type of crossmodal delay activity was also found in the inferior temporal (IT) cortex in auditory–visual and visual–auditory tasks (36).

However, the mechanisms of temporal integration of sensory and action processing in crossmodal working memory remain unclear. Specifically, it is still unclear how the sensory component and the action component of crossmodal working memory networks, as well as the component that mediates crosstemporal contingencies throughout the whole delay, are timely and selectively activated in the task. Here, to better understand the role of the DLPFC in the neural processing of crossmodal working memory, we examined differential neural activity (different firing rates in response to different stimuli or task events) (10, 11, 13) during the performance of crossmodal and unimodal WM (delayed matching-to-sample) tasks. Monkeys were trained to perform a visuo–haptic (VH) crossmodal WM task that required memorization of a visual cue for a subsequent haptic choice, and a haptic–haptic (HH) unimodal task, in which the animals had to retain a haptic cue for a subsequent haptic choice. Moreover, we trained a monkey to perform a control task that was identical to the VH task in all respects but without the requirement to memorize the visual cue during the delay period for the subsequent choice. We intended to find answers to two questions: (i) How does the DLPFC represent information of two different associated modalities and (ii) how do cortical networks in the DLPFC integrate the temporally separated components, sensory and choice components of a WM task?

Results

Behavioral Performance in Monkeys.

Two monkeys (D and B) were trained in the VH crossmodal delayed matching-to-sample (DMS) task (Fig. 1, Upper), in which both monkeys’ average correct rate of task performance reached at least 75% (75.4% for D and 80.1% for B) and the spatial bias indices of both monkeys were lower than 0.1 (Fig. 2B), indicating that the monkeys did not use spatial clues for choices.

Fig. 1.

Schematic diagram of behavioral tasks. (Upper) The visual–haptic (VH) task. A trial starts with a presentation of a white rectangular light (duration of 0.5 s) at the center of a computer screen. Two seconds after the offset of the white light, a visual cue (an icon, duration of 2 s) is presented at the same position. A pair of black and white icons is used. The offset of the visual cue signals the beginning of the delay, which varies between 15 and 17 s (randomized duration). At the end of the delay, a click signals the accessibility of a pair of objects for the choice. The animal then extends his operating hand toward the objects for exploring, palpating and pulling the one that matches the visual cue to get a reward. (Lower) The haptic–haptic (HH) task. A trial begins with a click signaling that the sample object is accessible to touch. The animal extends his operating hand toward the object and briefly palpates it, and, after the palpation, the hand returns to its resting place, signaling the beginning of the delay (15–17 s). A second click signals the start of the choice period (see details in Materials and Methods).

Fig. 2.

(A) Anatomical locations of recording sites. PS, principal sulcus; AS, arcuate sulcus. The recording sites cover areas 9/46d and 8b in the DLPFC (on the surface of the DLPFC slightly dorsal to the posterior principal sulcus). (B) Behavioral performance and spatial bias in the VH task of two monkeys. Behavioral performance is calculated as correct trials per total trials. Behavioral bias is calculated as the absolute value of (X − Y)/(X + Y), where X is the number of left choices, and Y is the number of right choices. (C) A histogram and rasters of a cell in the VH task show average neuronal activities in different task periods (bin size = 500 ms). The time-locking event for the histogram is the offset of the visual cue (Icon-off), the beginning of the delay period.

To compare the delay activity between the crossmodal and unimodal tasks, the trained monkeys were also trained in the HH unimodal DMS task (Fig. 1, Lower). Task performance for both monkeys also reached 75% (79.2% for D and 82.5% for B).

Behavioral observations showed that, when performing both tasks, most of the time the two well-trained monkeys used the following strategy (two response options) at the choice: Option 1, if the first object touched by the monkey was the match to sample, the monkey pulled the object after palpation of it; option 2, if the first object was the nonmatch, the monkey switched to the other object and pulled it after palpation. In a very rare case, the monkey moved its hand back and forth between the objects and pulled the correct one eventually. However, no matter which option it chose, the monkey always palpated the object before pulling it (37).

Prefrontal Activity Related to the VH Task.

In the VH crossmodal task, a total of 403 neurons (Table 1) (150 from monkey D and 253 from B) were recorded in areas 9/46d and 8b of the lateral frontal cortex (Fig. 2A). Most of the neurons (92.1%) showed task-related activity (Fig. 2C). Among them, 136 (33.7%) units were responsive to the visual cue; 106 (26.3%) units were activated during the early-delay period, 159 (39.5%) during the late-delay period, and 87(21.6%) during the whole-delay period. In addition, 157 (39.0%) neurons were activated during the choice period.

Table 1.

Units in VH task in the prefrontal cortex

VH task*	Visual sample		Delay†						Haptic choice
			Early		Late		Whole
RS	136		106		159		87		157
	H	V	H	V	H	V	H	V	H	V
Diff	21	17	9	15	21	18	10	11	14	18
Total diff	38		24		39		21		32

Diff, differential; H, horizontal; RS, responsive; V, vertical.

^*There were 403 in total.

^†All of the numbers in the Delay column are exclusive.

Prefrontal Delay-Differential Activity in the VH Task.

We investigated differential activity between horizontal-sample and vertical-sample trials in each task period (Table 1). In the 136 sample-response cells, 38 (27.9%) units showed sample-differential activity. In the 106 early-delay cells, 24 (22.7%) neurons (Fig. 3A) were found to prefer, by the firing, either horizontal (9 horizontal-cells) or vertical icons (15 vertical cells). Among the 159 late-delay cells, 39 (24.5%) (Fig. 3B) showed significant differential activity, favoring either horizontal (21 neurons) or vertical (18 neurons) icons. Twenty-one neurons (21/87, 24.1%) (Fig. 3C) showed the selective activity across the whole delay period. Ten of those 21 whole-delay cells were horizontal-preferred and the other 11 were vertical-preferred. During the choice period, 32 (20.4%) differential neurons were observed.

Fig. 3.

Delay-period differential activity and grand average firing rate in the VH task. (A, Upper) Rasters and histograms (bin size = 50 ms) of a cell showing early-delay (0–5 s) differential activity. The time-locking event for histograms is Icon-off (the beginning of the delay period). The cell shows a significantly higher firing rate (P < 0.01) in vertical trials. (Lower) The grand average firing frequency calculated from 24 early-delay differential cells (P < 0.001). The grand average firing frequency (±1 SEM) of cells for preferred objects (preference determined by significant higher firing frequency of either object) is indicated by the histogram in red, and for nonpreferred object is indicated by the histogram in green. (B) Rasters and histograms of a cell (P < 0.001) (Upper) and the grand average firing rate (Lower) showing late-delay differential activity of 39 cells (P < 0.001). The time-locking event for the histograms is Choice-click (the end of the delay period). (C) Rasters and histograms of a cell (P < 0.001) (Upper) and the grand average firing rate (Lower) showing whole-time delay differential activity of 21 cells (P < 0.001). The time 0 is the offset of the visual cue.

A large proportion of differential units showed differential activity in more than one period (Table S1 and SI Materials and Methods). Among them, most neurons were distributed in two adjacent periods: sample/early-delay, early-delay/late-delay, or late-delay/choice.

Delay Differential Activity Recorded in Both VH and HH Tasks.

To further explore neural activity in the early- and late-delay periods, we studied a subgroup of neurons (172 out of 403 neurons) that were recorded in both VH and HH tasks. We firstly decoded the task modality (VH and HH) from the vectors of firing rates of the 172 neurons. Results showed the high decoding accuracy in the early-delay period but a chance level of accuracy in the late-delay period (Fig. S1 and SI Materials and Methods), suggesting that the early- and late-delay activities likely play distinct roles depending on working memory tasks.

In addition, an ANOVA test was carried out on all those 172 neurons individually to analyze the early- and late-delay activities with two main factors: stimulus identity (horizontal vs. vertical) and task modality (visual vs. haptic). During the early delay, there were 41 (out of 172, 23.8%) cells displaying a significant (P < 0.05) main effect of stimulus identity. In these 41 cells, 27 (65.8%) displayed both identity and modality main effects (P < 0.05), and 31 (75.6%) showed the interaction effect (P < 0.05). Post hoc analysis with the 31 cells revealed that 7 cells showed the significant [post hoc, Tukey’s honest significant difference (HSD), P < 0.05] identity difference only in one task (4 in VH task and 3 in HH task) (Fig. S1). In addition, 118 cells (out of 172, 68.6%) displayed the main effect of task modality (P < 0.05). During the late-delay period, there were 16 (out of 172, 9.3%) cells displaying a significant (P < 0.05) identity main effect (Fig. 4). None of these 16 cells showed a modality main effect, and 2 of them revealed an interaction effect (P < 0.05). Meanwhile, only 4 (out of 172, 2.3%) cells displayed the main effect of modality (P < 0.05).

Fig. 4.

(A) Rasters and histograms (bin size = 50 ms) of a cell showing activity in the late-delay period recorded in both VH (Upper, P < 0.001) and HH (Lower, P < 0.001) tasks. The time-locking event for histograms in these two tasks is the choice click (the end of the delay period). (B) The grand average firing rate of late-delay differential activity of 16 cells recorded in both VH (P < 0.01) and HH (P < 0.01) tasks.

Apparently, the majority of neurons showed the modality effect during the early-delay period whereas such effect was essentially absent during the late delay. Thus, together with the decoding accuracies, our data indicated that the firing rate of the majority of neurons in the early delay either occurred concomitant with or was significantly related to differences in modality whereas, in the late-delay period, the firing rate of neurons occurred essentially without concomitant changes or independently of differences in modality.

Late-Delay Differential Activity in the Visual–Haptic Identical Task.

After completing recording in both VH and HH tasks, nine late-delay differential cells were further recorded in an extra task [visual–haptic identical (VHI)] (see details in Materials and Methods). The late-delay differential activity in the VH task (Fig. 5A) was significantly reduced in this VHI (Fig. 5B) task. At the population level, the differential firing rates of the nine neurons in both VH and HH tasks were significantly higher than those in the VHI task (Fig. 5D). Due to the limitation of our animal-training equipment, there was no haptic–haptic identical control task used to further test those cells.

Fig. 5.

Late-delay activity and choice response in VH (A), VHI (B), and HH (C) tasks. The neuron recorded in the three tasks shows significant late-delay differential activity in the VH (P < 0.001) and HH (P < 0.001) tasks, but not in the VHI task (P > 0.3). Note that the late-delay period is indicated by the red box in the histogram. (D) The difference in firing rate (delta-spike per s) of the late-delay period between preferred and nonpreferred trials from nine neurons in the three tasks. The difference in both VH and HH tasks is significantly larger than that in the VHI (P < 0.01) task. There is no significant difference (ns) in firing rate between the VH and HH (P > 0.1) tasks.

Delay Activity in the Control Animal.

In monkey F, 187 neurons were collected during performance of the VH task without any working-memory requirement. Eight cells showed differential responses to the visual cue (3 favoring horizontal bars and 5 favoring vertical bars). However, neither delay-differential cells nor choice-differential cells were observed.

Eye Movement During the Delay Period.

To exclude the possibility that eye movement caused the frontal differential activity in both VH and HH tasks, the monkeys’ eye-movement information was tracked during the whole experiment and analyzed online and offline. No correlation was found between eye movement [both x axis (horizontal) and y axis (vertical) movements] and delay-differential activity.

Haptic Pull (Choice)-Related Activity in VH Task.

The neural activity during the period between the onset of the last touch of the chosen rod and the pull of it was calculated to examine haptic choice-only cells. Only one of those cells was observed.

Discussion

In the present study, we have observed differential neural activity in the dorsolateral prefrontal cortex (DLPFC) during the delay period in both unimodal and crossmodal working-memory tasks. Our findings provide previously unidentified neural evidence that the DLPFC is part of a neural network for visual–haptic crossmodal association and memory. Specifically, this network contains distinct groups of neurons showing specific firing patterns that may represent different neural processes in crossmodal association and memory.

In a delay task, the delay activity by and large reflects either information about the sensory (sample) stimulus or information about the impending action (2, 3, 8). Quintana and Fuster (38) found two types of units, working-memory cells and preparatory-set cells, indicating two frontal substrates of active representation, one for the recent past (stimulus) and the other for the anticipated future (action). Those two types of cells are probably part of the same cortical network of long-term memory, which represent the associated sensory and action components of a given task (35, 39). Accordingly, in the present study, the differential delay activity in the VH task could be evoked by the visual cue (early differential delay activity), but also by the expectation and preparation of the later haptic stimulus choice (late differential delay activity). Thus, to better understand the neural representation of delay differential activity, we concentrated on examining neural activity during early and late segments of the delay period in both VH and HH tasks and compared differential activities between the two tasks in those two segments.

Early-Delay Differential Activity.

Our findings indicate that, for cells recorded in both VH and HH tasks, results of the task modality decoding display the high decoding accuracy in the early-delay period (Fig. S1), but a chance level in the late-delay period. Further, an ANOVA analysis indicates that, in the early delay, the majority of those cells (118 out of 172 cells, 68.6%) show effects of task modality (visual vs. haptic) on their activity. In addition, some cells (27 out of 118, 22.9%) also show effects of stimulus identity (horizontal vs. vertical) on their activity. Even further, a small subset of cells (7 cells) show only differential early-delay activity in one of the modalities (visual or haptic) (Fig. S1). These results suggest that the activity of working-memory networks in the DLPFC during the early delay is likely modality-contingent. That is, the early-delay activity most likely encodes and maintains sample information in both visual and haptic modalities, but with different coding strengths (40–42). A similar finding showing modality-specific delay activity has been reported in the monkey prefrontal cortex with a go/no-go task (43).

The analysis of activity in the early-delay period in the present study does not purport to show that these early-delay cells form discrete subgroups. Instead, the analysis indicates that there is probably a continuum of neuronal early-delay response selectivity to different modalities.

Late-Delay Differential Activity.

Compared with early-delay activity, the late-delay differential activity may, however, be a cortical representation of stimulus information independent of modality. That is, this information is not related to the sensory modalities, and its cortical representation is activated when the animal needs it for a behavioral choice in the task. In contrast to the early delay in which the majority of cells exhibit a modality effect, there are very few late-delay cells (4 cells, 2.3% of 172 cells) showing the modality effect, and thus modality specificity of late delay activity is essentially absent.

Sixteen neurons showed the identity main effect in the late delay, and each of them demonstrated the same firing preference in both VH and HH tasks to the orientation of ridges (favoring either horizontal rod or vertical rod). None of these 16 cells displayed any modality effect. Furthermore, the late-delay differential activity was significantly diminished in the VHI task when the haptic choice was not required. Also no late-delay differential activity was observed in the control monkey performing the VH task without the requirement of working memory. All of these results thus indicate that the late-delay differential activity likely represents the neural process underlying retrieval of information (in this case, orientations of parallel ridges) for later haptic exploration and pulling in the task choice. This process may involve temporary activation of long-term cortical networks that store modality-independent specific sample information that has been built through a long period of task training (44–46). Apparently, the activity is not related to motor preparation and general arousal because of its selectivity. In addition, the differential activity of most late-delay differential neurons continues into the choice period, in which monkeys have to explore the haptic rod and pull the correct one to get a reward. Therefore, in this goal-directed behavior, neurons in DLPFC seem to be essential both in preparation for subsequent choice and in actual haptic exploration and pulling.

Whole-Delay Differential Activity.

Notably, most delay differential neurons in this study also showed differential activity across adjacent periods: i.e., sample/early delay, early delay/late delay, and late delay/choice. Further, there was a group of neurons showing differential activity throughout the whole delay period. These results indicate that the DLPFC is involved in the mediation of crosstemporal contingencies during sequential behavioral actions, which requires interactions between posterior and frontal memory networks (41). Such behavioral sequences are probably performed in chain-like fashion, one act leading to the next (35). Therefore, we propose that, in the DLPFC, the delay differential activity occurring in multiperiods (especially the one throughout the whole delay) plays a critical role in bridging and integrating the sensory (sample stimulus) component with later components related to the behavioral choice (action). This proposition is consistent with other recent work (42), in which researchers indicated that the mixed selectivity commonly observed in responses of prefrontal neurons could be interpreted as a signature of high-dimensional neural representations. Therefore, in the present work, the different groups of DLPFC neurons showing cue- differential, delay-differential, and/or choice-differential activities may also reflect the different levels of dimensionality. Specifically, the whole-delay differential activity corresponds to a high dimensional representation of combined information from early- and late-delay activities. Those different dimensionalities may be at the basis of the mechanisms underlying the remarkable adaptability of the neural coding in the DLPFC in both VH and HH working-memory tasks (42, 47). Compared with neurons in the primary somatosensory cortex that show differential activity related to a single function, such as haptic choice-only differential activity in the choice period (34), the lack of single-function neurons in the DLPFC further supports the idea that the DLPFC plays at a higher level an important role in integration of information between perception and action (3).

Summary

In the present study, we hypothesized that three neural processes would occur during the delay period of the crossmodal working-memory task: (i) activation of neural circuits that represent properties of the first stimulus (sample); (ii) activation of neural circuits that subserve crossmodal associations (information transfer between two modalities); and (iii) activation of neural circuits that retain information in working memory. We found three sets of delay neurons that showed differential activity in the early-, late-, and entire-delay periods. These neurons seem to participate in those three processes. Specifically, the early-delay differential activity may represent the neural process of encoding and maintaining sample stimulus information. The late-delay differential activity is most likely related to the activation of internal representation of information required for the behavioral choice. Finally, the whole-delay differential activity seems to be the bridge to connect the sensory component (visual) to the choice component (haptic). It seems that, in addition to sustained delay differential activity, the other two types of differential activities in the DLPFC are also involved in crossmodal working memory and associations in the VH crossmodal task. However, the neural processes underlying this flow are still poorly understood. In future studies, simultaneous multielectrode recordings of neural activity in early-, late-, and whole-delay differential neurons should be carried out to further explore how different groups of differential neurons function together in the temporal organization of behavior. In conclusion, neural networks in the DLPFC may consist of different populations of neurons that function sequentially in the task from visual stimulus encoding and crossmodal information transferring between visual and tactile stimuli to the haptic action.

Materials and Methods

Animal.

Three adult rhesus monkeys (Macaca mulatta), two males and one female, weighing 7–11 kg, were used for this study. They obtained water only during experimental sessions as the reward for correct behavioral responses. Animal care and surgical procedures were approved by the Animal Care and Use Committee at The Johns Hopkins University.

Behavioral Task.

Experiments were conducted in a sound-attenuated chamber. The monkey was trained to perform VH crossmodal (48) and HH unimodal (37, 48) delayed matching-to-sample (DMS) tasks in a fully automated, computer-controlled apparatus (Fig. 1) (SI Materials and Methods).

VH task (Fig. 1).

The animals (monkeys D and B) were trained in the VH task. A trial started with the presentation of a white rectangular light (duration 0.5 s) on the center of a computer screen in front of the animal, at eye level. This light signaled the beginning of a trial. Two seconds after the offset of the white light, a visual cue (an icon, duration of 2 s) was presented at the same position. A pair of black and white icons was used. One icon showed patterns of horizontal parallel stripes (3.5 mm apart), and the other showed vertical stripes. The offset of the visual cue signaled the beginning of the delay, which varied between 15 and 17 s (duration randomized). During the delay period, the monkey had to rest his operating hand on the handrest (also during the baseline period). Any break of the contact between the hand and the handrest would lead to an automatic abort of the trial. In the choice period, two rods (one horizontal and the other vertical) were presented side by side. One rod was 5.5 cm to the right of the center of the opening, and the other 5.5 cm to the left. The monkey reached out to grasp and pull the objects. A pull of the rod ended the trial and led to an immediate reward with about 1.5–2 mL of water if the chosen rod matched the sample (the horizontal rod matching the horizontal icon, and the vertical rod, the vertical icon). The visual cue and the position of the tactile choice objects were changed randomly between trials. (This arrangement prevented the animal from using spatial clues for the choice.) Eye movements were monitored and recorded continuously by an eye-tracking system (ISCAN ETL-200; ISCAN).

VH task performed by the third monkey (F).

The monkey (F) was trained to follow only the procedure of the VH task. That is, the monkey was not required to either actively discriminate between the two icons (horizontal vs. vertical) and then memorize the visual information or make a haptic choice between the two rods (horizontal vs. vertical). A pull of either rod was rewarded. No overall significant position bias was observed across all recording days. On one recording day, the monkey mostly pulled the rod in one position (left or right) in the choice period, but, on another day, the monkey would mainly pull the rod in the other position.

HH task (Fig. 1).

The monkeys (D and B) were also trained in the unimodal (HH) task. In this task, the trial began with a click signaling that the sample (a vertical cylindrical rod, either with horizontal or vertical ridges on its surface) was accessible for the monkey to touch in a fixed central position in front of the animal. About 1.5 s later, the monkey lifted its operating hand from the handrest and reached out to touch the rod. After the sample touch, the animal returned its hand to the handrest, initiating the delay period, which varied between 15 and 17 s trial by trial. A second click signaled the beginning of the choice period. During that period, two rods (horizontal vs. vertical ridges) were presented side by side. The monkey reached out again to grasp the objects. A pull of the matching rod led to immediate reward. Throughout the experiment, the rods were entirely hidden from view. The sample rod and the position of the rods in the choice period were randomized from trial to trial.

VHI task.

In some cases, after neurons from monkey D and monkey B had been recorded in both VH and HH tasks, they were also recorded in the VHI task. The VHI task was identical to the VH task, except that the haptic objects at the choice were replaced by two identical smooth rods; the animal did not have to discriminate and memorize sample icons or make a haptic choice between two objects because they were identical.

Implantation of Recording Chamber and Single-Unit Recordings.

For details on implantation of recording chamber and single-unit recordings, see SI Materials and Methods.

Data Analysis.

For details on data analysis, see SI Materials and Methods. Task modality decoding analysis was as follows: To test the hypothesis that the early-delay but not the late-delay activity conveyed different strengths of modality information, a decoding-based method (40) was used in our study. We trained a pattern classifier on the firing rates from the 172 neurons across 50% trials (six trials on average from each neuron) to “learn” to differentiate two conditions in modality. We then assessed how much differential information was present in a population of neurons by quantifying the accuracy in a way that the classifier predicted which modality was present in the “new testing trials” (the other 50% trials). Before training and testing the classifier, a normalization step was applied to ensure that all neurons could be used in the decoding analysis, rather than only using the neurons with high firing rates. A smoothed bootstrap estimate of the classification accuracy was repeated 50 times. Fig. S1 shows the classification accuracy averaged over all of the bootstrap and cross-validation trials.

To analyze the delay-differential activity in the VH task, we used a two-way ANOVA with two main factors: stimulus identity (horizontal vs. vertical) and the delay period (early-, late-, and whole-period). To further examine response properties of the neurons in both tasks, we performed an additional ANOVA analysis with two main factors: stimulus identity (horizontal vs. vertical) and task modality (crossmodal vs. unimodal) during the early- and late-delay periods. Based on the results of the ANOVA analyses, Tukey’s HSD (P < 0.05) test was performed for the post hoc analysis.