Parallel processing of serial movements in prefrontal cortex

Abstract

A key idea in Lashley's formulation of the problem of serial order in behavior is the postulated neural representation of all serial elements before the action begins. We studied this question by recording the activity of individual neurons simultaneously in small ensembles in prefrontal cortex while monkeys copied geometrical shapes shown on a screen. Monkeys drew the shapes as sequences of movement segments, and these segments were associated with distinct patterns of neuronal ensemble activity. Here we show that these patterns were present during the time preceding the actual drawing. The rank of the strength of representation of a segment in the neuronal population during this time, as assessed by discriminant analysis, predicted the serial position of the segment in the motor sequence. An analysis of errors in copying and their neural correlates supplied additional evidence for this code and provided a neural basis for Lashley's hypothesis that errors in motor sequences would be most likely to occur when executing elements that had prior representations of nearly equal strength.

The problem of serial order in behavior is pervasive. As pointed out by Lashley (1), serial order is fundamental to all forms of skilled action, from speech to typing to reaching and grasping. He further believed that “analysis of the nervous mechanisms underlying order in the more primitive acts may contribute ultimately to the solution of the physiology of logic” (1). Speech is a prime example of elements organized serially at different levels: phonemes are uttered serially to make a syllable, syllables are strung serially to complete a word, words are emitted serially to make a sentence, etc. In other motor actions, simple movements are emitted serially to execute an integrated sequence (e.g., to reach for, grasp, and bring a cup to the mouth), a series of such sequences is performed to complete a task (e.g., get dressed), etc. In addition, arbitrary sequences can be learned and executed at ease (2). One common element in all these examples is the serially ordered nesting of lower-order units of action (e.g., phonemes, simple movements) within higher-order units (e.g., syllables, integrated motor sequences such as drawing a shape).

Lashley's major theoretical stance was the rejection of associative chaining theories, and the suggestion of an alternative model based on parallel response activation (1). A wealth of psychophysical evidence for this latter model has been gathered in speech and typing (3, 4), and theoretical models based on the idea of parallel response activation have also been developed (5–7). Lashley postulated the cotemporal activation of serially ordered action units: “There are indications that, before the internal or overt enunciation of the sentence, an aggregate of word units is partially activated or readied” (1). He also postulated a scanning mechanism by which these cotemporal representations could be translated into serial actions: “indications … that elements of the [sequence] are … partially activated before the order is imposed on them in expression suggest that some scanning mechanism must be at play in regulating their temporal sequence. The real problem, however, is the nature of the selective mechanism by which the particular acts are picked out in this scanning process and to this problem I have no answer” (1).

In these experiments we sought to (i) test Lashley's hypothesis above on cotemporal activation of action representations, (ii) seek a neural code for the serial order of these cotemporal representations, and (iii) use this code to investigate several aspects of serial order behavior. For that purpose, we trained two monkeys to copy simple geometrical shapes, which they drew as a series of movement segments. We then investigated the neural representation of the serial order of these segments in the prefrontal cortex, an area with an established role in serial ordering of motor actions and working memory (8, 9). Frontal lobe damage in human subjects disrupts serial integration of motor acts (10–12), and lesions of the prefrontal cortex in monkeys severely impair memory for the serial order of visual stimuli (13). Moreover, single cell recordings have identified prefrontal/premotor (A. F. Carpenter, G. Pellizzer, and A.P.G., unpublished observations) and motor (14) cortical cells coding for serial position in a context–recall task. Finally, with respect to the execution of motor sequences, neurophysiological studies (14–24) have demonstrated various kinds of sequence-specific engagement of single cells in several frontal areas (e.g., prefrontal, supplementary motor, dorsolateral premotor, primary motor cortex). However, these studies have not dealt explicitly with the cotemporal neural representation of serial order, which was the focus of this study. Here we provide evidence that all serial elements of a movement sequence are represented in an orderly fashion in the prefrontal cortex before the action begins.

Materials and Methods

Subjects and Task.

Two male rhesus macaques (M157 and M555, 8–10 kg body weight) were trained to draw geometric figures shown on a liquid crystal display projection screen, using a freely moving X–Y joystick with the left hand (model 541 FP, Measurement Systems, Norwalk, CT). The screen was located at a distance of 47 cm from the monkey. The task proceeded as follows. At the end of the intertrial period, a solid white start circle was presented on the left half of the screen. The monkey moved a cursor controlled by the joystick into the start circle to initiate a trial. The trial began with a waiting time (1 s for M157, 2 s for M555), during which there was nothing on the screen except the start circle and the cursor. If the monkey moved the cursor outside of the start circle during the hold period the trial was aborted. At the end of the hold period a shape was presented on the right half of the screen (M157 was trained to draw upright triangles, squares, trapezoids and an inverted triangle; M555 was trained to draw upright triangles and squares), and the cursor began leaving a trace on the left half of the screen. The monkey then had to use the joystick-controlled cursor to draw the shape presented. A trial was terminated if the monkey's trajectory strayed beyond a tolerance window. A correct trial was registered if the monkey completed the trajectory and returned to the start circle without straying outside the tolerance window. After a correct trial the monkey was given a juice reward. Trials were presented in blocks of shapes. M157 drew shapes in five blocks; the order of the shapes within a block was pseudorandomized. Each shape had to be drawn correctly 6 times before the next shape in the block was presented. Thus, 30 drawings were produced for each of four shapes in a complete set. M555 drew shapes in one block of 30 trials of each shape. The order of the shapes was pseudorandomized across sets. The x–y position of the joystick was sampled at 200 Hz. Care and treatment of the animals during all stages of the experiments conformed to the principles outlined in Principles of Laboratory Animal Care (NIH publication no. 86-23, revised in 1995). All experimental protocols were approved by the appropriate Institutional Review Boards.

Neural and Behavioral Data Acquisition.

The electrical activity of single neurons was recorded extracellularly by using 16 independently driven microelectrodes (Uwe Thomas Recording, Marburg, Germany) (25, 26). We recorded all cells encountered and analyzed all active cells recorded without any preselection. For each monkey, the recording chamber was made of titanium, and the inside diameter was 7 mm. The recording chamber was placed on the skull overlying the peri-principalis area in the right hemisphere, using stereotaxic coordinates and information obtained from MRI using a GE 1.5T magnet before their implantation; coronal images were taken every 1.5 mm. All surgical procedures were done aseptically under general gas anesthesia. Eye position was monitored by using a scleral search coil technique (CNC Engineering, Seattle) (27, 28). The horizontal and vertical components of the eye position were recorded at a sampling rate of 200 Hz simultaneously with neural recordings.

Analyses.

Motor trajectories were filtered (29) and differentiated with respect to time. The trajectories were then segmented by finding positive going zero acceleration crossings. If multiple submovements near a vertex were present, the zero crossing closest to the vertex was used. For M555, the triangle and the square were divided into 3 segments, because the square drawn usually had only 2 zero crossings. For M157, the triangle was divided into 3 segments, the square into 5, the trapezoid into 3, and the inverted triangle into 4.

We used standard linear discriminant analysis (30) to analyze the segment data. The function DDSCRM of imsl (Visual Numerics, Houston) was used to carry out all discriminant analyses. First, each trial was divided into a sequence of epochs, with each epoch spanning the drawing of a single segment of a shape. Then, the average firing rate during the drawing of each segment was calculated for all of the cells in an ensemble; these patterns of ensemble activity can be considered neural representations of the segments. Next, the ensemble pattern for each segment was averaged across correct trials and used to derive discriminant classification functions for that ensemble, namely one classification function per segment. These functions were then used to classify particular ensemble activity patterns as belonging to a specific segment. For that purpose, we calculated the posterior probability that the pattern belonged to different segments, and the pattern was classified as belonging to the segment category with the highest posterior probability, i.e., the “most probable” segment. The constraint was imposed that ensemble activity patterns in a given trial were classified only to those segments belonging to the shape drawn on that trial.

Two separate analyses were conducted. In the first, ensemble activity patterns were calculated in successive 25 ms bins throughout the waiting time (WT), the reaction time (RT), and throughout the drawing period. In the second analysis, the ensemble activity was averaged across the time in which the monkey drew each segment, rather than in 25 ms bins. In both bin- and segment-based analyses, firing rates were first converted to fractional interspike intervals (31) and then log-transformed, because neuronal responses are commonly proportional to the baseline activity level (32–34); the log-transformation normalizes distributions of the measurements and stabilizes their variance. Transformed activity patterns were then classified to the segments as above. The percent correct classification performance for the segments was assessed by using leave-one-out cross validation. Thus, new activity patterns were classified by using discriminant functions that were derived from all correct trials except the trial in which the pattern was recorded. In the segment-based analysis, ensemble activity patterns on both correct and error trials were classified. In the case of the classification of error trials, classification functions were derived by using correct trials truncated to the mean length of the error trials (error trials were shorter).

It is important to assess how clear-cut is the outcome of a classification. Obviously, this reduces to assessing the preponderance of the segment with the highest posterior probability (S₁) over the segment with next lower probability (S₂): the greater the distance between these two probabilities, the more secure the outcome of the classification. We evaluated this question by calculating the ratio S₁/S₂ and assessing its statistical significance across the whole population; for the latter objective, the S₁/S₂ ratios were log-transformed to normalize their distributions. Then, a paired t test was performed on the difference of these logratios and the antilog of the mean logratio difference computed (i.e., the geometric mean) to express the results in the original ratio scale.

Finally, we developed a metric built on the bin-based analysis to quantify the strength with which each ensemble represented the various segments in a shape across time, as follows. Each trial was converted to a sequence of represented segments by classifying the ensemble activity pattern in successive bins to segments, as described above. Then, across trials, we counted the number of trials that the ensemble activity was classified to each segment in a given bin. In that bin, the representation strength of each segment was then defined simply as the percentage of the total trials that the activity in that bin was classified to each of the segments. The representation strength is therefore the probability of representation of each segment in a bin. Aggregating these bin-wise data across all ensembles produced the representation strength functions of Fig. 2, which characterize the strength with which each segment is represented within prefrontal cortex across time.

Figure 2.

Plots for all four shapes of strength of representation vs. time. Each plot shows the strength of the representation of each segment for each time bin of the task. Time 0 indicates the onset of the template. Time bins during hold period and RT are 25 ms. Length of segments were normalized to permit averaging across trials. Plots show parallel representation of segments before initiation of copying. Further, rank order of strength of representation before coping corresponds to the serial position of the segment in the series. The rank order evolves during the drawing to maintain the serial position code. Line color corresponds to segments as follows: yellow, segment 1; green, segment 2; red, segment 3; cyan, segment 4; magenta, segment 5. Not all lines are defined for all shapes.

Results and Discussion

Monkeys drew a total of four geometrical shapes shown as templates on a screen (Fig. 1). Individual copying trajectories consisted of consecutive submovements characterized by bell-shaped velocity profiles. These submovements were serially ordered segments of the whole trajectory, qualitatively analogous, e.g., to phonemes in syllables. In both cases, the serially ordered segments, succeeding each other in time without interruption, give rise to a continuous motor output, namely drawing a shape or uttering a syllable. We recorded the activity of 511 cells in the prefrontal cortex during the copying task. These cells were located within Walker's area 46 in the dorsal and ventral banks of the posterior principal sulcus, and immediately adjacent gyral cortex. Fig. 1b illustrates the impulse activity from 16 cells recorded simultaneously. Small ensembles of 3–22 cells (mean = 9 cells, n = 58 ensembles) were recorded simultaneously. In >97% of cases, the ensemble activity patterns differed significantly among segments of a shape (multivariate ANOVA, Wilks' Lambda, P < 0.05) and, therefore, could be used for classification purposes. We found that, across the whole population, the posterior probability of the winner segment was, on the average, 14.5 times higher than that of first runner up segment (geometric mean, see Materials and Methods), a significant value (P < 10⁻²⁰, paired t test). These findings document the validity of the classification outcomes from which the following results were obtained.

Figure 1.

Shapes drawn, behavioral task and ensemble raster for one trial. (a) The four shapes drawn by the monkeys. The circle at the center of the top indicates the starting point for each shape. The arrow indicates the direction in which the shapes were copied. (b) The screens indicate the display during each of the phases of a single trial. The monkey had to maintain the cursor in the start hold circle for the WT. At the end of the WT, a template appeared and the monkey drew the shape. Shapes were drawn in blocks of trials such that several trials of the same shape had to be completed before a new shape was presented. This gave the monkey knowledge of the shape during the WT, before the template appeared. The raster at the bottom shows a single trial for 16 simultaneously recorded cells.

Strength of Segment Representation.

Shapes were copied in blocks of trials (see Materials and Methods). This provided advance knowledge to the monkeys during the WT (Fig. 1) concerning the shape they would copy; this was true for all but the first trial of the block which was not included in the analysis. Because a shape was drawn as a sequence of distinct segments, this advanced knowledge could be manifested as a proactive representation of the segments to be drawn during the WT and RT. Fig. 2 shows the time courses of the strength of representation of each segment during the WT, RT, and the drawing of each segment for several shapes copied. It can be seen that during the WT and the RT, the rank order of the strength of representation corresponds to the serial order of the segments. For example, a triangle was drawn in three segments [T1, T2, T3]. The rank order of the strength of representation S during the WT and RT of [T1, T2, T3] was S_T1 > S_T2 > S_T3, which corresponded to their serial position in the drawing: T1 (first), T2 (second), T3 (third). These results identified the strength of the representation of a segment in the population before drawing the shape as the determinant of the serial position of that segment in the shape drawn later. This relationship held for all shapes drawn, as can be seen in Fig. 2. In addition, there was an orderly temporal evolution of the strength of segment representation during the execution of the trajectories, as follows. The representation of a given segment peaked at roughly the middle of its execution and then began to fall. The representation of the following segment began to rise at roughly the middle of the current segment. Finally, the representation of the previous and subsequent segments crossed at approximately the point where the execution of the subsequent segment began. These patterns of time courses indicate that the drawing of a multisegment trajectory is actually an instance of a serial order process and support the hypothesis that the strength of segment representation is the neural code for serial order. Inhibitory interactions between prefrontal neurons may be part of the mechanism by which distinct ensemble activity patterns are established, a process known to play a role in the control of sequential events in prefrontal cortex (35). These results provide strong support for Lashley's (1) hypothesis that before the execution of a sequence all elements are simultaneously represented in a form of parallel response activation. In addition, our findings partially elucidate the kind of coding scheme by which the serial order of the shape segments to be drawn is represented during the time period preceding the drawing of the shape. This code is reflected in the strength of the representation of a given segment in the population and, as such, alleviates the need for a separate, qualitatively distinct mechanism of the serial order itself, a mechanism postulated by Lashley (the “schema of order,” ref. 1). A related question is what aspect of the segment is being coded in this activity. In a separate multiple linear regression analysis taking into account multiple aspects of each of the segments in the drawn copy, it was found that the serial position of the segment in the sequence comprising the shape was a prominent effect in the population that was statistically independent from the effects of other motor variables such as direction (unpublished observations).

Serial Order Errors and Their Neural Correlates.

Serial order processes possess a number of interesting behavioral characteristics. For example, errors in serial order tasks are characterized by a U-shaped serial position curve (36), such that performance (expressed, e.g., as percent correct) is better for early (i.e., primacy effect) and late (i.e., recency effect) elements, but poorer for elements in the middle of the sequence. This error pattern is observed for various such tasks across species (37). Indeed, we found a similar pattern of errors in our behavioral data (Fig. 3). Thus for most of the shapes, there were more errors committed during the execution of the middle segments of the shapes, than during the execution of the early or late segments. Interestingly, the strengths of the neural representations during the drawing of individual segments (Fig. 2) mirrored the serial position curves. Thus, when the whole shape was correctly drawn, the strength of the representation of the first and last segments was much higher than the strength of representation of the middle segments. We suppose that the strength of segment representation corresponds to performance, such that the stronger the representation the more likely would be a correct drawing of that segment. This would explain the high percent correct values for the first and last segments, i.e., the segments with strong representations. By the same token, the weaker representations of the middle segments would render them more prone to errors, as observed in the serial position error curve mentioned above.

Figure 3.

Serial position curves and classification performance on correct and error trials. (a) Graphs show the percent correct performance for each segment for each shape. The primacy/recency effect can be see in the higher percent correct performance for the early and late segments of a shape, and the relative decrease in performance for the middle segments of the shape. (b) Classification performance (mean ± 2 SEM) on correct (green lines) and error (red lines) trials. The neural activity pattern of each segment was classified by using the discriminant analysis. The neural activity patterns during the correct trials classified as the current segment much more often than the neural activity patterns during the error trials.

In addition, we analyzed the neural activity during the error trials by examining the distribution of segments to which the error segment was classified and comparing this distribution to that observed during correct trials. The results are shown in Fig. 3. It can be seen that when the shape was copied correctly, the current segment was classified in the proper serial position most of the times (in ≈70% of trials, on the average), whereas, when an error occurred, the current (error) segment was classified in the correct serial position much less frequently (in ≈35% of trials, on the average). This effect was statistically significant across all shapes and segments of both monkeys (ANOVA; F test, P < 0.0005).

Given that the neural activity during error trials was being misclassified much more often than it is during correct trials, we wanted to find out whether there was any systematic aspect to the misclassification during the error trials. In other words, if the neural activity was not representing the current segment being drawn, which segment was it representing? To investigate this, we plotted the segment to which the current segment was being classified on error and correct trials. We know that even during the execution of correct segments, the population is representing the other segments of the trajectory; this is the basis of the parallel representation. This can be seen in Fig. 2 by the nonzero probabilities of representation of segments other than the current segment being drawn, and also in Fig. 4 (green curve); the representation of the previous and following segments are approximately symmetric around the peak of the representation of the current segment. Now, the error data plotted in Fig. 4 (red curve) reveal a different picture. On error trials, segments subsequent to the current segment being drawn have an increased probability of representation. Thus, when the monkey commits an error, it is, on average, getting ahead of itself, and representing a subsequent segment prematurely.

Figure 4.

Classification probability for relative segment number. By using the same analysis as in Fig. 2, the segment to which the neural activity during the drawing of a segment was classified is plotted as a probability. On correct trials (green line), the neural activity usually was classified as the segment being drawn (segment 0). Furthermore, misclassifications are symmetric. On error trials, the neural activity classifies to the current segment much less often. Interestingly, the misclassifications are predominately to subsequent segments of the shape. Thus, on error trials, the ensembles tend to represent subsequent segments of the shape. Data points are mean ± 2 SEM.

This analysis of the errors and their neural correlates is connected closely to Lashley's ideas concerning error patterns in sequential behaviors. Indeed, he based most of his arguments for rejecting associative chaining theories of serial order behavior on the error patterns he observed in his own typing, a pattern which was rigorously documented later (3). A common finding across typing (3), writing (38), and speech (4) is that transposition errors, defined as the switching of elements of the sequence, are one of the most common classes of errors. Indeed, transposition errors are considered one of the cardinal characteristics of serial order processes, and formal models of serial order behavior, based on parallel response activation, have generally been based on reproducing these error patterns (5–7). With respect to neural coding, specific models have made certain predictions regarding the neural activity associated with transposition errors. Interestingly, the neural response profiles predicted by Houghton's model (6) mirror our empirically derived strength of representation quite accurately. Furthermore, most of these models generate transposition errors by allowing for noise in the strength of activation of the separate elements of the sequence. As can be seen in our data, noisy representation patterns would lead to a switch in the segment that had the strongest representation across the population, usually leading to an adjacent element of the sequence being represented. As can be seen in Fig. 4, during errors, subsequent segments were usually being represented, though a complete transposition could not occur in our task because a trial was aborted when the monkey's trajectory strayed too far from the ideal trajectory.

Conclusions

In this study, we sought to decode neuronal ensemble activity before and during the execution of a sequence of segments to investigate the neural representation of serial order. Before the execution of a sequence of segments, there was a cotemporal activation of all segments of the upcoming trajectory, with the strength of the representation of a given segment specifying its position in the upcoming sequence. The monkeys' behavioral performance was characterized by serial position curves, a common feature of serial order tasks. The serial position curves can be accounted for by the relative strengths of the representations of each segment during its execution. Initial and final elements of the sequence had stronger relative strengths of representation compared with the middle segments of the trajectories. Neural activity on error trials was less likely to represent the current segment being drawn. Thus, the relative strength of the representation of the current segment was lower on error trials, likely leading to the specification of a different segment. When the segment being represented was assessed relative to the segment being executed, it was found that on error trials subsequent segments were more likely to be represented; a neural trace of transposition errors. Thus, we have established a neural code for serial position, while developing a heuristic on multisegment copying, and used the code to try to understand the neural basis of serial position curves and transposition errors. Although this code was discovered in the prefrontal cortex, it is reasonable to suppose that serial order information is distributed in various brain areas (see above). The exact relations and modes of interplay among those areas remain to be investigated.

Abbreviations

WT

wait time

RT

reaction time