Short-Term Memory and Perceptual Decision for Three-Dimensional Visual Features in the Caudal Intraparietal Sulcus (Area CIP)

Abstract

The purpose of the present study was to examine whether neurons in the caudolateral part of the intraparietal sulcus (area CIP), a part of the posterior parietal cortex, contribute to short-term memory and perceptual decision of three-dimensional (3D) surface orientation, in addition to its purely visual nature of responding selectively to 3D surface orientation. Activities of CIP neurons were recorded while monkeys performed a modified delayed matching-to-sample (DMTS) task using stereoscopic stimuli. Seventy-seven neurons were examined with a routine of the DMTS task, and 94% (72 of 77) of them showed selectivity to surface orientation. Furthermore, 82% (63 of 77) of the examined neurons showed sustained activity during delay, and 60% (38 of 63) of them showed selective delay activity depending on the sample stimulus, suggesting that they contribute to short-term memory of 3D visual features. On the other hand, 53% (41 of 77) of the examined neurons showed modulation of visual response depending on whether a stimulus appeared as a sample, match, or nonmatch stimulus (contextual modulation). The majority (73%, 30 of 41) of these neurons with contextual modulation showed activity change depending on whether the test stimuli did or did not match the sample stimuli (match–nonmatch modulation), suggesting their involvement in matching, or perceptual decision, concerning 3D visual features. These findings suggest that CIP neurons play important roles not only in the perception of 3D visual features but also in cognitive functions such as short-term memory and perceptual decision of 3D visual information.

Introduction

Recently we found that neurons in the caudolateral part of the intraparietal sulcus (area CIP), a part of the posterior parietal cortex, are highly responsive to visual stimulation, and that the majority of visually responsive CIP neurons show selectivity to a specific surface orientation in space (Taira et al., 2000; Tsutsui et al., 2001, 2002). In these previous studies, monkeys were required to perform a modified delayed-matching-to-sample (DMTS) task, using three-dimensional (3D) stimuli as discriminative cues. During the recording sessions, we frequently observed CIP neurons showing sustained delay activities and context-dependent modulations of visual response, in addition to their pure visual response.

To perform the DMTS task, subjects have to maintain a memory of the sample stimulus during the delay and evaluate whether the test stimulus matches it during the test stimulus presentation. Delay activity during the performance of the DMTS task has been reported in a number of previous studies in inferotemporal (Gross et al., 1979; Mikami and Kubota, 1980; Miller et al., 1991, 1993; Eskandar et al., 1992; Miller and Desimone, 1994), posterior parietal (Constantinidis and Steinmetz, 1996; Sereno and Maunsell, 1998), and prefrontal (Miller et al., 1996; Sawaguchi and Yamane, 1999) cortices and has been interpreted as a neural correlate of short-term memory of a sample stimulus. Modulation of the visual response depending on the context in the DMTS task has been reported mainly in the inferotemporal (Gross et al., 1979; Mikami and Kubota, 1980; Miller et al., 1991, 1993; Eskandar et al., 1992; Miller and Desimone, 1994) but also in the posterior parietal (Constantinidis and Steinmetz, 2001) and prefrontal (Miller et al., 1996) cortices. It has been argued that the contextual modulation of visual response in the prefrontal cortex reflects its active involvement in the matching process or a perceptual decision concerning whether a test stimulus was the same as or different from a sample stimulus, whereas those in other cortical areas have much to do with short-term memory of a sample stimulus.

In the present study, we attempted to analyze the delay activity and the context-dependent modulation of visual response in CIP and compare them with those in other cortical areas previously reported to reveal how CIP is involved in perceptual and cognitive functions concerning 3D visual information processing. Monkeys were trained to perform a modified DMTS task, using solid-figure stereograms (SFSs) as discriminative cues, in which they had to report matching or nonmatching of the surface orientation by a go–no-go response (go–no-go DMTS task). Single-unit activities were recorded in CIP during the task performance.

Materials and Methods

Subjects and apparatus. Two male Japanese monkeys (Macaca fuscata) were used as the experimental subjects. Throughout the experiments, the monkeys were treated in accordance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals. This project was approved by the Ethical Committee of Nihon University School of Medicine. All experiments were conducted in the Department of Physiology, Nihon University School of Medicine.

All stimuli used in the present study were generated by a graphics computer (Indigo 2; SGI) and presented on a display (1240 × 1024 pixels, 21 inches) with a liquid crystal polarized filter (NUvision). The display was placed 44 cm in front of the monkey at eye level. The filter was switched at 120 Hz, whereby stimuli of 60 frames/sec were presented to each eye. The monkey wore polarized glasses to view these stimuli stereoscopically.

Stimuli and behavioral task. An SFS of a square plate with perspective cues was presented stereoscopically on a computer display. The size of the plate was 6.3 × 6.3° in visual angle when it was in the frontoparallel orientation. The thickness of the plate was set to minimum so that it appeared in the computer display as an ideal square plate rather than a parallelepiped. The plate was solid red without shadings and textures, and the background was black. (Pure red was chosen as the stimulus color to prevent ghost stimuli from appearing inappropriately in the eye when the filter was switched.) Nine different orientations were used in the task: a plate in the frontoparallel plane and plates of eight different orientations, which were slanted 45° against the frontoparallel plane and rotated every 45° around the sagittal (z) axis; therefore, the two-dimensional shape of the stimulus varied depending on the orientation. We used the definition from Stevens (1983) of the direction of slant as “tilt,” so that the tilt of the slanted plate ranged from 0 to 315° at 45° intervals. In the early phase of this study, we also used a stimulus set consisting of five orientations, omitting four orientations with oblique (45, 135, 225, and 315°) tilts. The fixation spot and the stimulus were presented at a distance of 44 cm from the monkey at eye level, either overlapping or next to each other. (The distance between the fixation spot and the stimulus center on the screen varied from 0 to 5°) The simulated viewing distance for the disparity and perspective cues of the stimuli was the same as the optical distance.

We used a go–no-go DMTS task (Fig. 1), in which the monkey had to judge whether the surface orientations of successively presented sample and test stimuli were the same or different. The difference in tilt angles between a pair of sample and test stimuli in a trial ranged from 0 to 180° at 45 or 90° intervals because the sample and test stimuli were chosen from a stimulus set consisting of nine or five orientations. Sample and test stimuli were chosen from a predetermined list of stimulus sequences. The stimulus sequence was determined pseudorandomly, complying with the rules that each of nine or five different surface orientations should appear as a sample and a test stimulus once in every nine or five trials, and that half of the trials should be match (go) trials and the other half should be nonmatch (no-go) trials in every 18 or 10 trials.

Figure 1.

Outline of go–no-go DMTS or successive same–different discrimination. Monkeys had to make a go or no-go response depending on whether the 3D orientation of sample and test stimuli were the same or different. FS, Fixation spot; Stm, stimulus.

The time sequence of the task was as follows. When a small fixation spot (0.2° in diameter) appeared, the monkey pressed the key and fixated on the spot. The monkey had to maintain fixation for at least 1 sec before the sample onset. The duration of sample stimulus presentation was 0.75 or 1 sec. After the offset of the sample stimulus, there was a delay of 2.3 sec until the test stimulus presentation. The duration of the test stimulus presentation was the same as that of sample presentation (0.75 or 1 sec). One second after the offset of the test stimulus, the color of the fixation spot changed from red to green (imperative cue). If the surface orientation of the sample stimulus had been the same as that of the test stimulus, the monkey had to release the key between 0.1 and 0.5 sec after the fixation color change (go trial); however, if the surface orientations had been different, the monkey had to withhold key release for 1.5 sec until the fixation spot was turned off and release the key between 0.1 and 0.5 sec after the fixation offset (no-go trial). The monkey was rewarded for the appropriate key release in both go and no-go trials (symmetrical reinforcement). After intensive training, monkeys could consistently perform this task with a success rate of >90%.

After completion of the training using the regular stimuli (SFSs), we conducted a behavioral probe test to assess whether the monkeys were performing a discrimination of 3D features. The probe stimuli used were the same set of solid figures, but they were displayed binocularly without disparities [two-dimensional solid figures (2D SFs)] and therefore novel to the monkey. The success rates for the last block of 100 trials with SFSs and those for the initial block of 100 trials with 2D SFs were 94 and 79% in one monkey and 88 and 74% in the other. In both monkeys, the success rate for 2D SFs was significantly lower than that for SFSs but significantly higher than the chance level (p < 0.01, χ² test). The difference of performance between SFSs and 2D SFs would not have occurred if the monkeys had performed a discrimination of the 2D features, because the 2D features were identical in two sets. The result suggests that the monkeys were performing a discrimination of the 3D features of SFS, and that the lower performance for 2D SFs was caused by a reduction of depth cues attributable to the absence of disparity cues. Both monkeys came to consistently perform the discrimination of 2D SFs with a success rate of >90% after 3 d of training (300 trials/d).

Single-unit recording. Before the single-unit recording, an atlas of the stereotaxic magnetic resonance image (MRI) of the brain of each monkey was constructed. For head fixation, a halo-like metal ring was implanted in each monkey's skull, and a recording chamber was stereotaxically implanted in the opening of the skull over the parietal cortex under sodium pentobarbital anesthesia. After recovery from the surgery, extracellular single-unit recordings were performed in CIP (Fig. 2) using tungsten microelectrodes. Because the recording chamber was implanted stereotaxically, the penetration track of the electrode could be accurately superimposed on the stereotaxic MRI brain map.

Figure 2.

Schematic illustration indicating the location of CIP in the top view of the left hemisphere.The intraparietal sulcus (ips), lunate sulcus (lu), and parietooccipital sulcus (po) are unfolded. AIP, Anterior intraparietal; VIP, ventral intraparietal; MIP, medial intraparietal; LIP, lateral intraparietal; PP, posterior parietal; PO, parietooccipital. CIP is located caudally and laterally in the intraparietal cortex between areas LIP and V3A and probably overlaps area LOP of the architectonic definition by Lewis and Van Essen (2000).

In the single-unit recording sessions, we first observed the response of neurons with a flat wooden plate held by the experimenter and examined whether there was a sufficient proportion of neurons that were selective to surface orientation to confirm that the cortical area we were recording from was CIP. After this initial testing, we examined the activity of individual neurons during the monkey's performance of the DMTS task with nine or five different surface orientations. If a neuron appeared to be related to the task according to the experimenter's visual inspection of rasters displayed on-line, we tried to maintain the recording for at least 45 or 25 trials. If the neuron appeared to be unrelated to the task, the recording was discarded before reaching 45 or 25 trials. The average number of trials collected for a neuron judged to be task-related by the experimenter was three correct match and three correct nonmatch trials for each orientation. Monkeys were required to gaze at the fixation spot from 1 sec before sample stimulus onset until test stimulus offset. The movement of the right eye was monitored using an infrared eye movement recording system (RMS). The trial was canceled immediately when the eye position exceeded the limit of 1° from the fixation spot. The eye movement during each trial was also monitored off-line to confirm that small saccades or vergence eye movements did not occur. Refer to our recent article (Taira et al., 2000) for a detailed description of eye movement recording.

Off-line analysis of neuronal activity. In the present study, we analyzed only the neuronal responses in successful trials. To analyze the neuronal activity related to task events in detail, we divided one trial into several periods: sample stimulus presentation (0.75 or 1 sec), former half of delay (1 sec, from 0.3 sec after sample stimulus offset to 1.3 sec after sample stimulus offset), latter half of delay (1 sec, from 1 sec before test stimulus onset to test stimulus onset), and test stimulus presentation (0.75 or 1 sec). Neuronal activities for 0.3 sec after sample stimulus offset were discarded to avoid including a visual off response into delay activity. For each neuron, data during the sample stimulus presentation and the delay were sorted by the orientation of the sample stimulus, and those during the test stimulus presentation were sorted by the orientation of the test stimulus. To test the responsiveness of a neuron in each trial event (sample, former and latter delay, and test), we compared the activity during each trial event with that of within 500 msec before sample stimulus onset independently for each orientation using Student's t test (paired, p < 0.05), in which the fixation period and the period of interest were matched for each trial. We did not apply a correction for multiple comparisons, because the purpose of the test was not to specify the difference of activity between different orientations or trial events but to examine whether a neuron was significantly responsive to at least one orientation during at least one trial event. If the neuron was judged as “responsive” to at least one orientation during one trial event, we conducted a Rayleigh test (p < 0.05) to judge the selectivity during that trial event.

To estimate the preferred orientation in each trial event, we transferred the response frequency for eight different orientations into vectors ( for 0,45... 315° tilt) so that the vector angle corresponded to the surface tilt and the vector length corresponded to the response frequency. The preferred orientation (tilt) was obtained by calculating the direction of the sum vector ( ).

To quantify whether delay neurons showed a decreasing or increasing trend of activity during delay, we calculated the delay trend index (DTI) for each delay neuron by dividing the difference of activity between the former half of delay and that in the latter half of delay by their sum as:

where R̄_former is the average activity (spikes per second) in the former half of delay, and R̄_latter is that in the latter half.

To test the context-dependent modulation of visual response, we further categorized neuronal activity during the test stimulus presentation into matching and nonmatching, and compared neuronal activities during sample, match, and nonmatch stimulus presentation using two-way ANOVA (nine or five surface orientations × three contexts) and Ryan's multiple comparison test. For the response to the preferred orientation of the neurons, the differential latencies between sample and match stimuli, sample and nonmatch stimuli, and match and nonmatch stimuli were calculated. Differential latency was defined as the latency of appearance of a significant difference in activity between two different types of stimuli (out of sample, match, and nonmatch). This was computed using the running mean Student's t test (p < 0.01) with a 40 msec window of neuronal activity in 10 msec increments from stimulus onset.

For a further analysis of visual response, we calculated an orientation–context index (OCI) for each neuron to quantitatively compare the effect of surface orientation and context in visual response. In calculating the OCI, we first calculated the mean response frequency for every combination of orientation and context independently. We then calculated the difference of response frequency between the most preferred and least preferred orientations (R̄_Omax–R̄_Omin) by pooling the average responses to different contexts and then the difference of response frequency between the most preferred and least preferred contexts (R̄_Cmax–R̄_Cmin)by pooling the average responses to different orientations. We defined the OCI as the difference of (R̄_Omax–R̄_Omin) and (R̄_Cmax–R̄_Cmin) divided by their sum as:

so that an OCI >0 indicates a stronger influence of orientation on the visual response, whereas one <0 indicates a stronger influence of context on the visual response.

Histology. After the completion of the single-unit recording, the monkeys were killed, and the entire brain was removed from the skull and soaked in 20% formalin. After the formalin fixation, 50-μm-thick sections were cut along the frontal plane in both hemispheres of two monkey brains. One of every two sections was stained with thionine to trace the penetrations. The unit recording sites were determined indirectly from the positions of the penetrations relative to the anatomical landmarks, as well as from the stereotaxic lesions made after performing all the experimental sessions.

Results

Activities of 243 neurons were recorded in CIP (Fig. 2), and after the initial on-line testing, 77 of them were tested with a test routine of the DMTS task (Fig. 1; for details of the recording procedure, see Materials and Methods.) To examine the memory and matching-related functions of these neurons, we analyzed activities during the sample and test stimulus presentation and those during the delay.

Visual response and selectivity to surface orientation

Of 77 neurons recorded during the performance of the DMTS task, all neurons were active during the sample and test stimuli presentation (Student's t test, p < 0.05), and 94% (72 of 77) of them were selective to the orientation of the sample stimulus, test stimulus, or both (Rayleigh test, p < 0.05). The majority of them (92%, 66 of 72) responded selectively to both sample and test stimuli; three (4%) responded selectively only to sample stimuli; and the other three responded selectively only to test stimuli.

Figure 3 shows the activity of a typical neuron that showed a surface orientation-selective response. This neuron showed consistent selectivity to 315° tilt whether it was presented as a sample or test (match–nonmatch) stimulus. For neurons showing selective response to both the sample and test stimuli, we examined the correlation between the orientation preferences during the sample and test stimulus presentations (Fig. 4). They were strongly correlated (r = 0.935; p < 0.001), indicating that the orientation selectivity was consistent and not influenced by the context of the stimulus presentation in most of the surface orientation-selective neurons.

Figure 3.

Responses of a typical CIP neuron showing consistent selectivity to a 3D surface orientation regardless of the context of presentation (sample, match, or nonmatch). The bar beneath each histogram indicates when the stimulus was on. Two markers in each raster line indicate stimulus onset and offset. Responses to three different orientations (135 and 315° tilt plus frontoparallel orientation) are displayed, although unit activity was recorded with a set of nine different orientations. Insets indicate the stimuli presented; solid lines represent the orthographic projection of the simulated plate onto the frontoparallel plane; dashed lines schematically represent the orientation of the surface in depth caused by binocular disparity; and the arrow represents the surface normal.

Figure 4.

Plot of the preferred orientation for sample (x-axis) and match (y-axis) stimuli. Each dot represents one neuron.

Delay activity

Of 77 neurons recorded during the DMTS task performance, 63 neurons (82%) were active during the delay period (Student's t test, p < 0.05), and 60% (38 of 63) of them showed selective delay activity according to the orientation of the sample stimulus (Rayleigh test, p < 0.05). To analyze the delay activity in detail, we divided the delay into two subperiods, the former and latter halves. Of these selective delay neurons, 87% (33 of 38) of neurons were selective in the former delay, and 50% (19 of 38) of neurons were selective in the latter delay. Thirty-seven percent (14 of 38) of these neurons showed selective activity in both the former and latter halves of the delay. The proportion of neurons selective in the former half of the delay was significantly higher than that in the latter half (χ² test, p < 0.01).

Figure 5 shows the activities of two typical neurons that exhibited selective delay activity. A neuron whose activity is shown in the left column responded selectively to the surface of 45° tilt during the sample stimulus presentation. During the delay, it showed sustained but gradually decreasing activity only when the preferred orientation had appeared as the sample stimulus (Fig. 5A, top row). Thus this neuron showed the same selectivity during the sample stimulus presentation and former half of the delay, but the activity returned to the spontaneous discharge level and became nonselective in the latter half of the delay (Fig. 5B). A neuron whose activity is shown in the right column responded selectively to the surface of 90° tilt during the sample stimulus presentation. During the delay, it showed gradually increasing activity in the latter half of the delay only when that preferred orientation had appeared as the sample stimulus (Fig. 5C, top row). Thus this neuron showed similar selectivity during the sample stimulus presentation and latter half of the delay, but the activity once became nonselective in former half of the delay (Fig. 5D).

Figure 5.

Activities of two typical sample-selective delay neurons showing decreasing (left column) and increasing (right column) trends. Note that all neuronal data during sample stimulus presentation and delay periods are sorted by a sample stimulus tilt angle. The broken-line graphs at the bottom indicate average discharge rates for different tilt angles during sample stimulus presentation (filled circles), former (F) half of delay (open triangles), and latter (L) half of delay (inverted open triangles). Dotted lines in the broken-line graphs indicate the spontaneous activity, and error bars indicate SE. FP, Frontoparallel orientation. Rasters and histograms above each broken-line graph display neuronal activities for three representative sample stimulus tilt angles. Bars at the top of the rasters indicate when the sample and test stimuli were on. Markers in each raster line indicate stimulus onset and offset. Other conventions of rasters and histograms are the same as in Figure 3.

Almost all (95%, 36 of 38) of the selective delay neurons also responded selectively during the sample stimulus presentation. To examine the relationship between the selective activity during the sample presentation and that during the delay, we plotted the preferred orientations during the delay against those during the sample presentation for these neurons. Figure 6A shows that the preferred orientations during the former half of delay were strongly correlated with those during the sample presentation (r = 0.844; p < 0.001). Similarly, Figure 6B shows that the preferred orientations during the latter half of the delay were strongly correlated with those during the sample presentation (r = 0.650; p < 0.01). Thus, the selectivity during the delay is consistent with the orientation preference during the sample stimulus presentation.

Figure 6.

Plot of the preferred orientation during sample stimulus presentation (x-axis) and delay (y-axis) periods. Each dot represents one neuron. Note that all neuronal data during sample stimulus presentation and delay periods are sorted by sample stimulus tilt angle. A, Plot of the preferred orientation during the former half of the delay against that during the sample presentation period for 31 neurons that showed selective activity during both the sample presentation and the former delay. B, Plot of the preferred orientation during the latter half of delay against that during the sample presentation period for 19 neurons that showed selective activity during both the sample presentation and the latter delay.

As in the two typical neurons shown in Figure 5, delay-selective neurons showed either a gradually decreasing or increasing trend of activity. To quantify the changes in activity within the delay, we calculated the DTI for each delay neuron by dividing the difference in activity between the former and latter halves of the delay by their sum (see Materials and Methods). Figure 7A shows the distribution of the index of selective delay neurons. The proportion of neurons having a decreasing trend in activity (index <0) was larger than that having an increasing trend (χ² test, p < 0.05). Figure 7B shows the distribution of the index of nonselective delay neurons. In contrast to selective delay neurons, the proportion of neurons having an increasing trend in activity tended to be larger than that having a decreasing trend (χ² test, p < 0.10).

Figure 7.

Distribution of delay trend indices showing the delay activity trend. A, Distribution of indices of 38 selective delay neurons. B, Distribution of 25 nonselective delay neurons. The index was calculated for each neuron by dividing the difference in activity between the former and latter delays by their sum, so an index >0 indicates an increasing trend, whereas one <0 indicates a decreasing trend.

Contextual modulation of visual response

As shown in Figure 4, the orientation preferences between sample and test (match–nonmatch) presentation periods were strongly correlated in selective visual neurons, indicating that the orientation selectivity was consistent and not influenced by the context of the stimulus presentation in these neurons. However, some of these neurons appeared to change their response magnitude depending on whether a stimulus was presented as a sample, match, or nonmatch stimulus. To examine whether the visual response was modulated depending on the context of the stimulus presentation, we further categorized test stimuli into match and nonmatch stimuli and performed two-way ANOVA (nine or five orientations × three contexts, p < 0.05) on the neuronal activity during stimulus presentation for 77 visually responsive neurons.

The main effect of orientation was significant in 72 neurons, indicating the selectivity to surface orientation. (This population was completely the same as neurons judged to be selective to surface orientation by the Rayleigh test.) The main effect of context was significant in 41 neurons, indicating modulation of the visual response by the stimulus presentation context. In 39 neurons, both main effects were significant; only two neurons responded nonselectively to surface orientation and showed the contextual modulation. Interaction of orientation and context was significant in 16 neurons. For 41 neurons with the contextual modulation of visual response, we performed the multiple comparison test of Ryan's procedure (p < 0.05) to compare the response magnitude for sample, match, and nonmatch stimuli. These neurons were then classified according to two different criteria. One was whether the response magnitude was different for the match and nonmatch stimuli (match–nonmatch modulation). Of 41 neurons, 24 were significantly more active during match stimulus presentation, whereas 6 were significantly more active during nonmatch stimulus presentation. The number of neurons responding more to match stimuli than to nonmatch stimuli was significantly larger than that of those responding more to nonmatch stimuli (χ² test, p < 0.05). The other criterion was whether the response magnitude was different for the sample and test stimuli (sample–test modulation). Of 41 neurons, 6 were significantly more active during sample stimulus presentation, whereas 8 were significantly more active during test stimulus presentation.

Figure 8 shows the activity (average response magnitudes to nine orientations in three contexts) of typical surface orientation-selective neurons with and without contextual modulations. Figure 8A shows the activity of a typical neuron without any contextual modulation. Figure 8B shows that of a typical neuron with match–nonmatch modulation. In this neuron, the response to match stimuli was stronger than that to nonmatch stimuli. Figure 8C shows that of a typical neuron with sample–test modulation. In this neuron, the response to test stimuli was stronger than that to sample stimuli. To analyze the time course of the contextual modulation, we constructed population average histograms (Fig. 9, top row) and their cumulative histograms (Fig. 9, bottom row) for each of four types of modulation (match > nonmatch, match < nonmatch, sample > test, and sample < test) by averaging the responses to preferred orientation of individual neurons. These figures indicate that the modulation of response started within a short latency after stimulus onset, and the difference in activity became significantly large at ∼500 msec after stimulus onset. If we assume the response to sample stimuli is a baseline visual response, it appears that match–nonmatch modulation (Fig. 9A,B) was mainly attributable to enhancement or inhibition of the response to match stimuli, and the modulation of the response to nonmatch stimuli was small and occurred with longer latency. In contrast, in cases of sample–test modulation (Fig. 9C,D), the modulations of response to match and nonmatch stimuli were almost the same in magnitude and latency.

Figure 8.

Broken-line graphs showing average response to stimuli of nine different orientations in different contexts: sample (closed circles), match (open triangles), and nonmatch (open inverted triangles). A, Responses of a typical neuron showing no contextual modulation of visual response, whose responses to three representative orientations are shown in the rasters and histograms in Figure 3. B, Responses of a typical neuron showing larger responses to match stimuli than to nonmatch stimuli. C, Responses of a typical neuron showing larger responses to test (match and nonmatch) stimuli than to sample stimuli. Dotted lines indicate the spontaneous activity, and error bars indicate SE. FP, Frontoparallel orientation.

Figure 9.

Population average histograms (top row) and their cumulative histograms (bottom row) of responses to preferred orientation in three different contexts (sample, match, and non-match). A, Averaged response of 24 match–nonmatch modulation neurons showing larger responses to match stimuli. B, Averaged response of six match–nonmatch modulation neurons showing larger responses to nonmatch stimuli. C, Averaged response of six sample–test modulation neurons showing larger responses to sample stimuli. D, Averaged response of eight sample–test modulation neurons showing larger responses to test stimuli. Differential latencies between the sample and match stimulus, sample and nonmatch stimulus, and match and nonmatch stimulus are indicated by closed triangles, open triangles, and arrows, respectively, in the cumulative histograms. The difference of activity between match and nonmatch trials did not reach statistical significance in neurons with sample–test modulation. For the calculation of differential latencies, see Materials and Methods.

Although a 1 sec interval intervened between the offset of the test stimulus and the onset of the imperative cue for key release (fixation color change), the match enhancement could be a reflection of motor preparation for key release or expectation for the imperative cue in go trails. To test this possibility, we examined whether the higher response to match stimuli continued until the imperative cue for key release in match > nonmatch type neurons (n = 24) by comparing the activity within 500 msec before the fixation color change (trigger for key release in go trial) in go (match) and no-go (nonmatch) trials using two-way ANOVA (orientation of test stimulus × match–nonmatch, p < 0.05). Of these neurons, 46% (11 of 24) showed significantly higher activity in go trials; however, 46% (11 of 24) showed no difference in activity between go and no-go trials, and 8% (2 of 24) showed significantly higher activity in no-go trials.

Comparison of surface orientation selectivity and contextual modulation

To quantitatively compare to what extent the two factors, surface orientation and context, can change the activity of CIP neurons, the difference in response frequency between the most and least preferred orientations averaged over contexts (R̄_Omax–R̄_Omin) and that between the most and the least preferred contexts averaged over orientations (R̄_Cmax–R̄_Cmin) were calculated for each neuron and then averaged over all neurons. Average R̄_Omax– R̄_Omin and R̄_Cmax–R̄_Cmin were 22.3 ± 1.98 and 4.15 ± 0.41 spikes/sec (mean ± SE), respectively, and the former was significantly greater than the latter (Student's t test, paired, p < 0.001). We also calculated the OCI for each neuron (for details, see Materials and Methods). The OCI varies from –1 to 1 depending on the relative strength of influence of the orientation and context on the neuronal response; an OCI >0 indicates a stronger influence of orientation, and one <0 indicates stronger influence of context. As shown in Figure 10, OCI was >0 in most neurons, indicating that their activity was influenced more by orientation than by context. Only 13% (10 of 77) had OCI <0, suggesting a stronger influence of context over that of orientation. Therefore, the main factor that could change the activity of CIP neurons was surface orientation, and the stimulus presentation context had a relatively small effect on their activity.

Figure 10.

Distribution of the OCIs showing the relative power of the factors of orientation and context over neuronal responses. An index >0 indicates a larger effect of orientation, whereas one <0 indicates a larger effect of context. For details concerning how we calculated the OCI, see Materials and Methods.

To assess the interaction between the factors of orientation and context that was present in 16 neurons, we attempted a quantitative analysis for interaction. It is shown in Figure 4 that the difference in context did not alter the preferred orientation. Therefore, we assumed that the interaction was caused by a stronger effect of context in the preferred orientation than in the nonpreferred orientation. We compared the average difference in response between the most and least preferred context in the most preferred orientation (R̄_OmaxCmax–R̄_OmaxCmin) and in the least preferred orientation (R̄_OminCmax–R̄_OminCmin). The mean R̄_OmaxCmax–R̄_OmaxCmin and R̄_OminCmax–R̄_OminCmin values were 11.5 ± 2.45 and 5.61 ± 1.71 spikes/sec, respectively, and there was a statistical tendency of the former being larger than the latter (Student's t test, paired, p < 0.07). Therefore, it was indicated that the influence of the context in the response is larger in the preferred orientation than in the nonpreferred orientation.

Finally, we examined whether contextual modulation and selective delay activity tended to occur together in each neuron. Of the 38 neurons with selective delay activity, 53% (n = 20) exhibited contextual modulation. Of the 39 neurons without selective delay activity, 54% (n = 21) exhibited contextual modulation. The percentage of neurons exhibiting contextual modulation was not significantly different between groups of neurons that did and did not exhibit selective delay activity (χ² test, p > 0.10).

Discussion

To perform the DMTS task, the monkey must be able to maintain an image of a sample stimulus during the delay and evaluate whether a test stimulus matches it. The main new finding of the present study is that many neurons in CIP show activity changes related to these cognitive functions, short-term memory and matching, or perceptual decision, in addition to the perception of 3D surface orientation. We found that 49% (38 of 77) of the examined neurons showed sample-selective delay activity. We also found that 39% (30 of 77) of the examined neurons showed modulation depending on whether the test stimuli did or did not match the sample stimuli (match–nonmatch modulation).

Delay activity and short-term memory of 3D surface orientation in CIP

It is evident from the task design that the observed selective delay activity is related to the short-term memory of 3D surface orientation, because the monkey had to maintain an image of the sample stimulus but could not intend or prepare to do any specific task-relevant response (go or no-go) during the delay of 2.3 sec. The fact that the selectivity of delay activity was highly correlated to the selective visual response during the sample presentation period also indicates the strong relationship between delay activity and short-term memory of 3D surface orientation.

As reported in previous studies using delay tasks (Kubota et al., 1974; Niki and Watanabe, 1976; Fuster and Jervey, 1982; Funahashi et al., 1989; Quintana and Fuster, 1999), there were two different trends of delay activity: decreasing and increasing. In the present study, selective delay neurons tended to show a decreasing trend, whereas nonselective delay neurons tended to show an increasing trend. It has been argued that a decreasing trend may be related to retrospective coding of previous sensory information, or a sensory trace, whereas an increasing trend may be related to prospective coding such as expectation of a forthcoming stimulus. Therefore, our finding that sample-selective delay neurons tended to show a decreasing trend is further support for the idea that observed selective delay activity in CIP is a neural substrate of short-term memory of 3D surface orientation. Conversely, nonselective delay activity, which tended to show an increasing trend, may be related to anticipation of the test stimulus presentation or other factors such as increasing demand in attentional or motivational resources.

It is noteworthy that the proportion of sample-selective delay neurons was relatively high (38 delay-selective neurons among 63 delay-active neurons, and these among 77 examined neurons). In studies of other visual areas, such a high proportion of selective delay neurons was observed only in areas related to the higher stage of visual perception. In the ventral visual pathway, many studies report that neurons with selective delay activity were found in the inferotemporal cortex during the performance of tasks requiring visual short-term memory (Fuster and Jervey, 1981; Fuster and Jervey, 1982; Miyashita and Chang, 1988; Fuster, 1990), but they were rarely found in earlier visual areas such as V1 and V4 (Haenny and Schiller, 1988; Haenny et al., 1988; Fuster, 1990; Maunsell et al., 1991; Chelazzi et al., 2001). Therefore, the observed high proportion of sample-selective delay activity of area CIP is in accordance with our previous finding that this area is involved in higher-order 3D visual perception (Tsutsui et al., 2001, 2002).

Contextual modulation of visual response and perceptual decision in CIP

We found, by comparing the visual responses to different orientations presented in different contexts (sample, match, and nonmatch), that the main factor affecting the visual response of CIP neurons is a stimulus feature of 3D surface orientation. The visual response was also modulated depending on the context, although it had less effect on the visual response of CIP neurons compared with 3D surface orientation. Concerning these results, there might be a possibility that the selection of the recording site on the basis of sensitivity for surface orientation (see Materials and Methods) would have biased the sampling of neurons toward neurons with a stronger orientation tuning and weaker contextual modulation. However, considering our previous findings that many neurons in CIP show orientation selectivity even in a passive viewing condition, and that those surface orientation-selective neurons are distributed widely within CIP (Shikata et al., 1996), our recording strategy would not have caused a strong sampling bias.

There were two types of modulation: (1) match–nonmatch modulation, i.e., different magnitudes of response to match and nonmatch test stimuli; and (2) sample–test modulation, i.e., different magnitudes of response to sample and test stimuli. The most prominent of these contextual modulations was the enhancement of the response to match stimuli (match enhancement). It is unlikely that the match enhancement is a simple reflection of a preparation for key release or an expectation of trigger stimulus for key release, because this modulation starts shortly after the onset of the test stimulus and does not remain until key release in all neurons of this type. Rather, it may be related to the cognitive process of matching, or perceptual decision. Match–nonmatch modulation of visual response during the performance of the DMTS task has been previously reported in the inferotemporal (Gross et al., 1979; Mikami and Kubota, 1980; Miller et al., 1991, 1993; Eskandar et al., 1992; Miller and Desimone, 1994), parietal (Constantinidis and Steinmetz, 2001), and prefrontal (Miller et al., 1996) cortices. It was reported that most neurons in the inferotemporal and parietal cortices showed suppressed responses to match stimuli (match suppression) (Eskandar et al., 1992; Miller et al., 1993; Constantinidis and Steinmetz, 2001), whereas most neurons in the prefrontal cortex showed enhanced responses to match stimuli (match enhancement) (Miller et al., 1996). Although we may not be able to directly compare the result of the present study with these studies because of the difference of the design of the DMTS task, CIP neurons seem to resemble prefrontal neurons, which have been suggested to have an active involvement in the matching process, in the sense that they show match enhancement rather than match suppression. The difference may be that the prefrontal cortex, in which neurons show less selectivity to a visual stimulus and a greater effect of match enhancement (Miller et al., 1996), have a more general role in the matching process. The interpretation of sample–test modulation may be more uncertain. It may be that neurons showing stronger responses to sample stimuli and those to match stimuli are related to the encoding and retrieval process in short-term memory. However, there may be other explanations: the stronger response to sample stimuli may be related to the novelty of the stimuli, or the stronger response to test stimuli may be related to the increasing demand for cognitive resources concerning the perceptional decision of whether test stimuli matched sample stimuli.

Similar to sample-selective delay activities, match–nonmatch modulation seems to be specific to higher-order cortical areas. Under the performance of a similar DMTS task, responses of neurons in V4 (Haenny et al., 1988; Maunsell et al., 1991) and MT/MST (Ferrera et al., 1994) were also modulated by the memory of a sample stimulus, but there was no tendency for the responses to test stimuli to be either enhanced or suppressed depending on whether they matched a sample stimulus. Instead, neurons in these areas were likely to show selective activity during test stimulus presentation depending on which stimulus was presented as a sample stimulus (sample-selective activity during test stimulus presentation). Therefore, the dominance of match enhancement in CIP suggests that this area is involved in higher-order information processing for visual perception.

Possible cortical network linking 3D visual perception and various cognitive functions

Two major findings of the present study, the high proportion of delay selectivity and match enhancement neurons, seem to imply a functional link between CIP and the prefrontal cortex. The prefrontal cortex has been indicated to play a major role in short-term memory, because lesions or reversible deactivation of the lateral prefrontal cortex impair performance in various delay tasks (Mishkin, 1957; Gross and Weiskranz, 1962; Mishkin et al., 1969; Goldman and Rosvold, 1970; Goldman et al., 1971; Passingham, 1975; Mishkin and Manning, 1978), and many prefrontal neurons are activated selectively by stimuli during the delay of these tasks (Niki, 1974a, b, c; Niki and Watanabe, 1976; Watanabe, 1986; Quintana et al., 1988; Funahashi et al., 1989; Wilson et al., 1993; Miller et al., 1996; Quintana and Fuster, 1999; Sawaguchi and Yamane, 1999). It was proposed that the prefrontal cortex plays a major role in short-term memory by activating a closed loop circuit between itself and other cortical areas (Goldman-Rakic, 1987; Fuster, 1997). Because the posterior parietal cortex including CIP seems to be directly connected with the prefrontal cortex (Schwartz and Goldman-Rakic, 1984; Selemon and Goldman-Rakic, 1988; Cavada and Goldman-Rakic, 1989b), it is likely that short-term memory for 3D visual information is achieved by a closed loop circuit between the prefrontal cortex and CIP. Similar to short-term memory, the prefrontal cortex was suggested to play a major role in the matching process, because prefrontal neurons show a greater effect of match enhancement and less selectivity to visual stimuli compared with other posterior association cortices (Miller et al., 1996). The high proportion of match enhancement neurons in CIP implies a close relationship between the prefrontal cortex and CIP, so the proposed closed loop circuit appears to be also important for matching or perceptual decision.

Our previous studies indicated that CIP neurons are involved in the coding of 3D surface orientation based on binocular disparity, linear perspective, and texture gradient, and that they are directly related to perception (Taira et al., 2000; Tsutsui et al., 2001, 2002). A recent functional MRI study of the monkey reported activation of the corresponding region of CIP during the presentation of 3D shapes defined by texture gradient as well as motion parallax (Sereno et al., 2002). Anatomically, CIP probably overlaps area LOP (lateral occipital parietal) of Lewis and Van Essen (2000) and lies in the caudal part of the lateral bank of the intraparietal sulcus. It receives fiber projection from V3 and V3A (Adams, 1997), so visual information is likely to reach this area in a bottom-up manner mainly through the pathway starting from layer 4b of V1, a thick stripe of V2, and through V3 and V3A (Sakata et al., 1997). It appears that CIP also receives input from V4 (Cavada and Goldman-Rakic, 1989a; Baizer et al., 1991). It may be that information from dorsal and ventral visual pathways meets in area CIP and is integrated to represent 3D surface orientation, which is directly related to subjective perception. The results of the present study suggest that CIP is the cortical area where perceptual information processed in a bottom-up manner meets the top-down cognitive signal, which probably comes from the prefrontal cortex.