Action selection and refinement in subcortical loops through basal ganglia and cerebellum

Abstract

Subcortical loops through the basal ganglia and the cerebellum form computationally powerful distributed processing modules (DPMs). This paper relates the computational features of a DPM's loop through the basal ganglia to experimental results for two kinds of natural action selection. First, functional imaging during a serial order recall task was used to study human brain activity during the selection of sequential actions from working memory. Second, microelectrode recordings from monkeys trained in a step-tracking task were used to study the natural selection of corrective submovements. Our DPM-based model assisted in the interpretation of puzzling data from both of these experiments. We come to posit that the many loops through the basal ganglia each regulate the embodiment of pattern formation in a given area of cerebral cortex. This operation serves to instantiate different kinds of action (or thought) mediated by different areas of cerebral cortex. We then use our findings to formulate a model of the aetiology of schizophrenia.

1. Introduction

The higher-order circuitry of the brain comprises a large-scale network of cerebral cortical areas that are individually regulated by loops through subcortical structures, particularly through the basal ganglia and the cerebellum (Houk & Wise 1995; Kelly and Strick 2003, 2004). These subcortical loops form distributed processing modules (DPMs) that have powerful computational architectures (figure 1; Houk 2005). The final outcome of all of the computations in a given DPM is a spatio-temporal pattern of activity in the module's output vector, representing the activity in its set of cortical output neurons. This allows a given DPM to participate in the computations taking place in other areas of cerebral cortex or in the brainstem or spinal cord.

Figure 1.

The abstract signal processing operations posited (Houk 2005) for each DPM. Net excitatory pathways are shown with closed arrows, net inhibitory pathways with open arrows and the grey diamonds signify neuromodulatory and training inputs.

The loop through the basal ganglia is thought to regulate the selection and/or initiation of pattern formation (Houk & Wise 1995; Redgrave et al. 1999; Gurney et al. 2001; Houk 2001, 2005). The term embodiment is used in figure 1 to capture both possibilities, i.e. selection and initiation, the former occurring when disinhibition allows other cortical inputs to initiate and the latter when the selection is strong and does its own initiation. Embodiment is critically dependent on the refined, neuromodulated pattern classification operations that take place in the input layer of the basal ganglia, the striatum (Gruber et al. 2003). According to most contemporary models, bursts of striatal spiny neurons, via the direct pathway through the basal ganglia, disinhibit their targets in thalamus, allowing thalamo-cortical loops to embody patterns of activity that represent a ballpark estimate of an action or a thought. There are also mechanisms, via less direct pathways through the basal ganglia, for inhibiting the embodiment of patterns that would represent poor choices in action selection (Houk & Wise 1995; Gurney et al. 2001).

Once a tentative pattern has been selected and initiated through the operation of the loops through the basal ganglia, the loops through the cerebellum amplify and sculpt that pattern into a refined output vector (Houk & Mugnaini 2003). The amplification step appears to be implemented by the loop through the cerebellar nuclei. Regenerative positive feedback in this loop amplifies the output's intensity, duration and spatial extent. The restraint of this amplification process and, more importantly, sculpting it into an accurate representation of an action (or thought) is implemented by the loop through the cerebellar cortex. The cerebellar cortex is considered to be an exceptional neuronal architecture for learning difficult computations (Raymond et al. 1996; Houk & Mugnaini 2003) and so is well suited to this refinement task.

We will relate the computational features of DPMs to experimental results for two kinds of natural action selection. First, functional imaging during a serial order recall task will be used to study human brain activity during the selection of sequential actions from working memory. Second, microelectrode recordings from monkeys trained in a step-tracking task will be used to study the natural selection of corrective submovements. Our DPM-based model assists in the interpretation of puzzling data from both of these experiments. We come to posit that the many loops through the basal ganglia regulate the embodiment of pattern formation in a given area of cerebral cortex. This operation serves to instantiate different kinds of action (or thought) mediated by different areas of the cerebral cortex.

2. Serial Order Processing

Tasks in which lists of items are presented, after which the subject is required to recall the items in the same order in which they were presented, require serial order processing and sequential action selection. Here, we introduce a task dubbed Replicate without intending to identify novel behavioural phenomena. Instead, we aspire to establish a task paradigm that elicits many standard patterns of serial recall behaviour, but which also can be conveniently applied across research modalities and, in particular, across species. Benchmark properties of serial order recall include (i) a graded decline in recall accuracy with sequence length, (ii) transposition gradients reflecting a tendency for items to be recalled at serial positions near to their original positions, and (iii) item similarity effects including (a) a tendency for items to be recalled near the item where they originally appeared and (b) a tendency for sequences of similar items to be recalled less accurately than sequences of less similar items (Botvinick & Plaut 2006).

(a) The Replicate task

In Replicate, K targets are presented on an N×N grid of squares in a randomized sequence, and the subjects are required to remember their positions and serial order over a brief delay. The subjects are then cued to use a joystick to move a cursor to the K positions in the same order in which they were originally presented. The phase of target presentation requires the setting up of a working memory representation, which must be sustained through the delay and then decoded in order to produce correct joystick movements; we thus refer to the three phases of the task as the encoding, maintenance and decoding phases.

Behavioural studies with Replicate confirm that the task generates several standard patterns of recall behaviour. Thirty-two Replicate trials were performed, eight at each of four sequence lengths (three to six for one half of the subjects and four to seven for the other half). Each trial was initiated by the subject using the joystick to move a cursor into the central tile in a 5×5 grid. A target sequence then appeared, with each target location illuminated for a total of 500 ms. Following a 10 s delay, the joystick cursor changed colour, cuing the subject to replicate the target sequence, returning to the central tile when finished. A maximum of 3 s was allotted for identification of each location. Our error analysis demonstrates that the Replicate task yields the typical visual memory span of four to five items, and that errors frequently involve (i) transpositions of items located near to one another in the sequence and/or (ii) substitution of a location target with a nearby location in the grid. These results demonstrate that Replicate has several benchmark properties of serial order recall that have been studied with lists of more cognitive items.

(b) Functional neuroimaging of Replicate

For our brain imaging study, we employed a control task referred to as Chase. In Chase, a sequence of location cues appears just as in Replicate, but subjects use the joystick to track these cues immediately as they appear. Chase involves similar stimulus and response sequences to Replicate but eliminates the working memory component.

Brain functional neuroimaging (fMRI) activity of subjects performing Replicate utilized two primary blood-oxygen-level-dependent (BOLD) contrasts. An Execute contrast was made between the period of sensory-guided joystick movements in the Chase task and a rest period. This contrast was designed to show the neural correlates of motor execution. A Decode contrast was made between the memory-guided movement period of the Replicate task and the sensory-guided movement period of the Chase task. This contrast was designed to reveal the neural correlates of the decoding process while simultaneously controlling for BOLD activity related to pure motor execution. Whole-brain echo planar imaging data (24 6 mm slices, TR=2000 ms) were collected from 10 subjects, and a partial-brain scanning protocol focusing on the basal ganglia (12 6 mm slices, TR=1000 ms) was used for 9 subjects.

In the participants who provided whole-brain data, reliable decoding activity was observed in the right prefrontal (PF) cortex, left anterior cingulate, left supplementary motor area and portions of cerebellum. Activity related to the execution of joystick movements was observed in the contralateral primary motor cortex, contralateral putamen and ipsilateral cerebellar cortex.

The partial-brain imaging protocol provided better sensitivity to changes within the striatum of the basal ganglia. The differential BOLD activities in the right and left caudate and putamen were strikingly different for the Execute and Decode contrasts (figure 2). A significant increase in activity was found in the putamen for Execute, whereas a significant decrease in activity was found in the caudate nucleus for Decode. The deactivation representing a statistically significant decrease in blood flow in caudate during the decoding operation was puzzling. Brain processing is believed to require increased synaptic activity, which recruits increased metabolism and blood flow, as detected by an increase in BOLD signal (Logothetis 2002).

Figure 2.

Differential BOLD activity in the right and left head of the caudate and putamen for the Decode (grey) and Execute (black) contrasts. Error bars indicate standard error. ^* indicates a significant difference (t(8)≥2.36, p<0.05), while ^** indicates a highly significant difference (t(8)≥4.16, p<0.01). A decrease in activity was found in the caudate nucleus for decoding, whereas a significant increase in activity was found in the putamen for execution. Deactivation representing a statistically significant decrease in blood flow in caudate for the Decode contrast was surprising.

Decreases in BOLD are considered mysterious (Gusnard & Raichle 2001) and are usually explained by greater synaptic processing in the control task as opposed to the main task. In our Decode contrast, this could happen if caudate were actively engaged in the sensory-guided control task Chase, due to the presence of visual targets for each movement of the joystick. However, figure 2 indicates that caudate is not particularly active in the Execute contrast. The statistically significant decrease in BOLD for the Decode contrast seems to need a better explanation.

(c) Action selection in the loop through the basal ganglia

Although many authors have suggested that the loop through the basal ganglia plays an important role in action selection, there are diverse views concerning the mechanism by which this might occur. Most authors agree that action selection occurs in the input nucleus of the basal ganglia loop, namely the striatum (but see Rubchinsky et al. 2003). There are different views about the mechanisms for preventing actions, which will not be discussed here.

The dorsal part of the striatum, the neostriatum, comprises two divisions, the caudate nucleus and the putamen. The principal neurons of both the caudate and the putamen, the medium spiny neurons, are inhibitory GABAergic projection neurons. They emit an elaborate array of collaterals to neighbouring spiny neurons before they project to output stages of the basal ganglia, namely to either globus pallidus or substantia nigra pars reticulata. Figure 3a shows two of these spiny neurons with collaterals that inhibit each other and gives rise to an inhibitory feedback network entirely within the neostriatum. This local feedback network mediates a competitive pattern classification operation. Collateral inhibition is deemed an effective mechanism for competition by some authors (Plenz 2003) and ineffective by others, the latter believing that feed-forward inhibition regulates the pattern classification operation (Tepper et al. 2004). Beiser & Houk (1998) modelled both mechanisms and found that they worked, but the inhibitory feedback network worked more effectively than the feed-forward network.

Figure 3.

Competitive pattern classification between spiny neurons in the neostriatum. (a) Dendrites (above) and projection axons (below) of two spiny neurons. Two synaptic inputs from the cerebral cortex are illustrated. In the middle are two inhibitory collaterals. Note that one collateral inhibits a dendrite to mediate postsynaptic inhibition, whereas the other inhibits a synaptic terminal to mediate presynaptic inhibition. (b) Schematic illustration of why competition mediated by presynaptic inhibition is more effective than competition mediated by postsynaptic inhibition. The two time plots show net excitatory synaptic input (g_s) from cortex and membrane potential (V_m) of a spiny neuron as the cortical input slowly increases (between the two vertical dashed lines). In the absence of synaptic input, V_m is near the potassium equilibrium potential E_K. As synaptic input g_s increases, V_m moves in the positive direction in a sigmoidal fashion (typical of a down- to up-state transition). The upward arrows indicate times of GABA release from inhibitory collaterals. The open arrows illustrate how postsynaptic inhibition actually depolarizes (excites) spiny neurons that are in the down-state and only mediates shunting inhibition when V_m is at the chloride equilibrium potential E_Cl. The downward closed arrows show that presynaptic inhibition always decreases membrane potential (inhibits) and therefore is qualitatively more effective than postsynaptic inhibition. (c) Membrane potential responses of two model spiny neurons in response to stimulus A followed by stimulus B. The AB neuron responds strongly when B is delivered after A. The effect of presynaptic inhibition is shown by the suppression (arrow) of the BA neuron membrane potential. It is produced by collateral inhibition from unit A in figure 4, which fires in response to stimulus A (figure 5).

What has not been considered to date is the possibility that the inhibitory feedback network relies on presynaptic, as opposed to postsynaptic, inhibition. This is surprising since presynaptic inhibition of cortical input to the neostriatum has been demonstrated electrophysiologically (Calabresi et al. 1991; Nisenbaum et al. 1993) and morphologically (Lacey et al. 2005). Indeed, the operation of a presynaptic mechanism for collateral inhibition could also explain the mysterious fMRI BOLD deactivation that we found in caudate for the Decode contrast (figure 2). Synaptic input is believed to be a strong contributor to BOLD signals (Arbib et al. 2000; Logothetis 2002). Since presynaptic inhibition would decrease synaptic input that could explain the deactivation for caudate. The activation seen for putamen presumably results from a greater dependence on postsynaptic inhibition. The cause for this difference might relate to phylogeny. The DPM that operates on working memories via a loop through the caudate nucleus (Kelly & Strick 2004) is phylogenetically newer than the loop through the putamen to and from the primary motor cortex (M1). The latter, namely the M1-DPM, generates the voluntary motor commands that control the individual movements.

(d) Model of competitive pattern classification

Presynaptic inhibition should give rise to a computationally powerful mechanism for pattern classification. Beiser & Houk (1998) found that, since the equilibrium potential for postsynaptic GABAergic inhibition (E_Cl in figure 3b) is between the down- and up-state of spiny neurons, this mechanism for mediating competition between neighbouring spiny neurons is quite sensitive to spontaneous membrane potential and to model parameters. It performed better than feed-forward inhibition, but it was not optimal. Presynaptic inhibition has no equilibrium potential—it just reduces the synaptic input regardless of the membrane potential of the spiny neuron (figure 3b). This presynaptic advantage reflects a qualitative principled effect that should be robust to parameter selection.

We modelled a minimal network of recurrent loops from cortex through basal ganglia and back to cortex that encodes the serial order of two visual cues, A and B (figure 4). The reader is also referred to the ‘implementation details’ posted in the electronic supplementary material of this paper. Recurrent loops in the direct pathway through the prefrontal (PF) cortex, caudate (CD) nucleus, globus pallidus pars internus (GPi) and thalamus (T) are used to encode two visual cues, A and B. Computational units AB and BA are labelled for the sequence they respond to the best, whereas Ax (Bx) is activated by A (B) independent of its serial order. Prefrontal cortex projections are excitatory, with synaptic weights represented by dot sizes. Caudate spiny units are interconnected by inhibitory collaterals to form a competitive network (not explicitly shown in figure 4). Via their projections, CD units are inhibitory to GPi units. The high spontaneous activity of GPi units provides a tonic inhibitory background to thalamus, and inhibition of this background activity provokes a disinhibition of thalamic units. Rebound activity of thalamic units starts positive feedback and sustained activity in the reciprocal excitatory pathway between thalamus and cortex.

Figure 4.

Serial order encoding network. Recurrent loops in the direct pathway through the prefrontal (PF) cortex, caudate (CD) nucleus, globus pallidus pars internus (GPi) and thalamus (T) are used to encode two visual cues, A and B. Computational units AB and BA are labelled for the sequence they respond to the best; Ax (Bx) is activated by A (B) independent of its serial order. Prefrontal cortex projections are excitatory, with synaptic weights represented by dot sizes. Caudate spiny units are interconnected by inhibitory collaterals to form a competitive network (not explicitly shown). Caudate units are inhibitory to GPi units, which in turn inhibit thalamic units. This disinhibition activates thalamic units and interconnected PF Cortex units. The loop is completed by reciprocal excitatory connections between thalamus and cortex.

Spiny neurons were simulated using a minimal biophysical model (Gruber et al. 2003) with excitatory and postsynaptic inhibitory conductance inputs. Presynaptic inhibition was modelled by dynamically decreasing the excitatory synaptic weights of the input from PF cortex. The GPi–T–PF loop was abstractly modelled based upon the Beiser & Houk model (1998) with a sigmoidal function to transform membrane potentials into firing rates. The network was instantiated using either no inhibition, presynaptic inhibition or postsynaptic inhibition in caudate, and the model was then subjected to noise. The response to a sequence of A followed by B with zero noise is illustrated in figure 5. The effect of presynaptic inhibition can be appreciated in figure 3c which shows model membrane potentials of the AB and BA caudate spiny neurons designated in figure 4. The AB neuron is partially excited by Ax, the memory of A, and partially by B, thus strongly by the sequence AB. The BA neuron begins to be excited by stimulus A but then is suppressed by lateral presynaptic inhibition from other spiny neurons. A misclassification error in the illustrated example (in the presence of noise) would be firing of the BA neuron in PF cortex.

Figure 5.

Response to the stimulus A followed by B using presynaptic inhibition in caudate. Firing rates in caudate and prefrontal cortex are depicted. With reference to figure 4, inhibitory input from caudate units causes tonically active GPi units to hyperpolarize and pause, producing rebound responses in thalamic units. The respective prefrontal cortical units are then activated and sustained by positive feedback between thalamic and prefrontal units.

Presynaptic inhibition yielded improved noise tolerance and decreased energy requirements compared with postsynaptic inhibition. When the network was subjected to noisy inputs, the misclassification rate without inhibition was 54.6% but fell to 24.1% for postsynaptic inhibition and 19.4% for presynaptic inhibition (4.8% decrease with presynaptic versus postsynaptic inhibition, p<0.001). Presynaptic inhibition also decreased the summed magnitude of synaptic activity in caudate from 118 to 98.0 (difference of −16.9%, p<0.001). The decreased excitatory synaptic activity in the presence of presynaptic inhibition can account for the reduced fMRI BOLD signal seen in caudate during the decoding contrast that was illustrated in figure 2.

3. Embodiment of Corrective Submovements

Tracking movements that require both speed and accuracy consist of a primary movement that is often off-target, in which case it is accompanied by one or more corrective submovements in man (Novak et al. 2002) and monkey (Fishbach et al. 2005). The corrective submovements often occur before the primary movement is completed, which suggests that the neural control system uses a forward model to predict the movement endpoint based on a copy of the neural command (efference copy) and delayed sensory feedback. The methods for detecting submovements are described in Novak et al. (2000) and Fishbach et al. (2005).

Examples of a delayed and an overlapping submovement (OSM) together with the simultaneously recorded firing patterns of two neurons in the primary motor cortex (M1) are illustrated in figure 6a,b. In this task, the monkey turns a rotating handle to move a cursor horizontally on a screen (blue trace, position; red trace, velocity) to acquire a target (blue boxes). The individual spikes are represented by the vertical green lines at the top of figure 6a,b. The baseline-rate normalized cumulative sum histogram (CUSUM) (cf. Gibson et al. 1985) for both M1 neurons, shown by the green traces, marks the bursts of discharge with upward deflections. The first burst is large and occurs prior to the primary movements. The second burst is smaller and precedes the delayed corrective submovement (DSM) as shown in figure 6a and precedes the OSM as shown in figure 6b. We conclude that these neurons transmit motor commands that control both primary movements and corrective submovements.

Figure 6.

Segmented movements of a monkey and associated bursts of discharge in primary motor cortex. (a) The firing pattern of a motor cortical neuron during a trial that contains a delayed submovement. (b) The firing pattern of another motor cortical cell during a trial that contains an overlapping submovement. The motor cortical neurons each show two bursts of discharge, which are depicted by upward deflections in the green CUSUM traces.

(a) Abstract model of how corrective submovements are generated

Whether the update of motor commands is continuous or intermittent is still under debate. Our findings from a statistical analysis of the properties of submovements under conditions in which perturbations of target location were introduced at movement onset strongly support the hypothesis that the neural controller predicts the need for a correction and selects an appropriate one intermittently (Fishbach et al. 2007).

Our model of this action selection process is illustrated schematically in figure 7. Vision provides the information about the desired endpoint, which can be updated as rapidly as 180 ms when a visual perturbation is introduced at movement onset. The brain computes the predicted endpoint based on efference copy and sensory input, and it computes the prediction variance based on past experience. The normalized predicted error (Z-score) must exceed a threshold value T in order to initiate a corrective submovement. The executed submovement follows an approximately bell-shaped velocity profile.

Figure 7.

An abstract model of how corrective submovements are generated. Vision provides the information about the desired endpoint, which can be updated as rapidly as 180 ms when a visual perturbation is introduced at movement onset. The brain uses a forward model to compute the predicted endpoint based on efference copy and sensory input, and it computes the prediction variance based on past experience. The normalized predicted error (Z-score) must exceed a threshold value T in order to initiate a corrective submovement. The executed submovement follows an approximately bell-shaped velocity profile.

(b) Single-cell recordings from the basal ganglia in monkeys

In order to study how the basal ganglia participate in the embodiment of M1 commands for movements and submovements, we recorded from basal ganglia output neurons in monkeys that had been trained in the same one-dimensional step-tracking task described above. Since the output cells in GPi of the basal ganglia project, via thalamus, to many different areas in the cerebral cortex, neurons need to be sampled from the region of GPi that projects to the primary motor cortex (Roy et al. 2003). The sampled neurons should also be the ones that are well related to the task. Figure 8 is an example from our ongoing work that meets both of these criteria.

Figure 8.

Activity during a single trial of a GPi neuron. In this task, the monkey turns a rotating handle to move a cursor horizontally on a screen (blue trace, position; red trace, velocity) to acquire a target (blue boxes). The baseline-rate normalized CUSUM for the neuron (green trace) shows three downward deflections marking three pauses in the high tonic discharge rate in this GPi neuron. The first pause (1) is small and occurs prior to the primary movement; the second and third are stronger pauses in association with tiny corrective submovements (2 and 3).

The extracellular recording of the GPi neuron illustrated in figure 8 shows the typical high spontaneous discharge rate mentioned in §2d. Its firing rate is modulated in association with both the primary movement and with an OSM and a DSM which occurred in this particular trial (see the red velocity trace in figure 8 and compare this neuron with the motor cortical neurons illustrated in figure 6). The CUSUM (green trace) clearly shows three pauses in the high tonic firing rate. Pause 1 is small and occurs prior to the primary movement; pauses 2 and 3 precede tiny corrective submovements.

Figure 9 demonstrates the reliability of the single trial properties illustrated in figure 8. The average firing rate of the cell is shown for all trials that contained a single corrective submovement (blue trace) and for trials containing multiple corrective submovements (red trace). Both traces are aligned to the onset of the first submovement. Note that the pauses corresponding to the corrective submovements are as strong as or stronger than the pause for the primary movement, even though the corrections which they appear to control are typically much smaller than is the primary movement. These discrepant amplitude relationships in the firing rate data are puzzling. Motor cortex units show smaller bursts for the small corrective movements than for the larger primary movements (figure 6).

Figure 9.

The reliability of the previously illustrated single trial properties, from a block of trials for the figure 8 neuron. The average firing rate of the cell is shown for all trials containing a single corrective submovement (blue trace) and for trials containing multiple corrective submovements (red trace). Both traces are aligned to the onset of the first correction. Note that the pauses corresponding to the submovements are as strong as or stronger than the pause for the primary movement, even though the corrections they appear to control are typically much smaller than is the primary movement. In the text, we discuss the probable explanation for these discrepant amplitude relationships.

(c) Interpretation of the puzzling firing rate data

The DPM model posits that practice in a task allows regularly rehearsed processing steps to be exported from the basal ganglia and/or cerebellum to the area of cerebral cortex to which the channel projects (Houk & Wise 1995; Houk 2005). The control of primary movements can be exported to the motor cortex since they are rehearsed in every trial. In contrast, the corrective submovements vary substantially from trial to trial, so nothing regular is rehearsed. This model of knowledge transfer from the basal ganglia to the cerebral cortex is supported by combined recordings of single cell activity from the neostriatum and frontal cortex (Brasted & Wise 2004; Pasupathy & Miller 2005). It is also supported by Frank's (2005) simulations of dopamine modulation in the basal ganglia and by imaging data (Toni et al. 2002).

4. Discussion

Comprehension of brain dynamics may require an understanding of the assembly language of the brain, its machine language so to speak. The model used here (Houk 2005) assumes that the brain's assembly language is the firing rate of its individual neurons. The output of each DPM is a vector of firing rates in its population of output neurons. Although population discharge contains some information in addition to its rate code, due to the tendency for population activity to become synchronized, the evidence that synchronization is actually used to control actions is meagre (Fetz 1997; Houk 2005; for another view on this see Riehle et al. 1997). The key issue for us concerns whether or not there exists a biophysical mechanism for decoding a synchronicity signal into a selected action. Since thus far none have been documented, we treated synchronicity as an epiphenomenon and decided to focus our modelling efforts on firing rate, ignoring the detailed timing of action potentials. The CUSUM analyses of single cell discharge illustrated in figures 6 and 8 eliminate variations due to modest synchronization effects and therefore facilitate our analysis of single trial data. The analysis of single trial data is quite valuable since specific behaviour typically varies appreciably from one trial to the next. One can take advantage of this variation to test for reliable relations between single unit activity and behaviour.

(a) Special computational features of the neostriatum

Our model of action selection is motivated by the existence of powerful computational features in the loops through the basal ganglia. The pattern classification operation shown in figure 1 takes place in the striatal layer of a DPM. Computationally powerful pattern classification derives from several unique features of striatal medium spiny neurons (Houk 2005). These features include (i) a high convergence ratio (Kincaid et al. 1998) that presents nearly 20 000 different cortical inputs to any given spiny neuron, (ii) a three-factor learning rule that uses reward-predicting training signals from dopamine neurons to consolidate long-term potentiation learning (Houk et al. 1995), (iii) an attentional neuromodulatory factor (Nicola et al. 2000) that induces bistability and nonlinear amplification in spiny neurons (Gruber et al. 2003), and (iv) competition among spiny neurons mediated by presynaptic and postsynaptic collateral inhibition (§2c,d; Plenz 2003).

The anatomically demonstrated projections that loop back to the same area of cortex from which they derive (Kelly & Strick 2004) allow cortical-basal ganglionic modules to perform serial order processing (Beiser & Houk 1998). This feature allows them in principle to implement immediate serial order recall from working memories of a sequence. Long-term memories of serial order could be stored in cortico-cortical synapses or in the synapses between cortical neurons and striatal spiny neurons. The latter storage mechanism is thought to have a larger memory capacity for salient information (Houk & Wise 1995). The recall of previously learned sequences should also be efficient because cortical-basal ganglionic modules implement parallel searches through a vast repertoire of past experiences stored in the synapses of spiny neurons.

The DPMs in our model are generic in the sense that they all execute an identical set of processing operations, which are summarized in figure 1. A complete model of the Replicate task described in §2a requires at least three DPMs. One is needed to encode the sequence of visual targets into a serial order working memory. In psychological terms, its output is like a ‘thought’ about an action sequence. Its output vector is a representation of that thought. A second DPM is needed to decode the thought into a sequence of actions. Its output vector could be called a ‘plan’ in psychological terms, and that DPM's output vector is a representation of the plan. A third DPM is required to generate the set of commands that execute any given action. Its output vector can be called a ‘command’ in psychological terms. Much is known about the representation of voluntary motor commands in M1, the primary motor cortex (cf. Georgopoulos 1995).

Functional imaging of Replicate reveals activation patterns in a large network of brain areas (§2b), larger than the network of DPMs required to model this task. Psychological name tags for the output vectors in all these areas are difficult to come up with. The point being made here is that our spoken language starts to fail when we attempt to give names to all of the signals in large brain networks. However, we can still talk about the output vector of any given cortical area—it is the set of firing rates in that area's output neurons. Perhaps, we should begin to explore the use of brain language as an alternative to psychological terms when we attempt to describe the operation of large brain networks.

(b) Integrative control by basal ganglia and cerebellum

The present paper deals mainly with cortical-basal ganglionic loops, whereas most DPMs also have loops through cerebellum. Regarding the latter, presently we know most about signal processing in the loop between cerebellum and M1, the primary motor cortex (Houk & Mugnaini 2003). There are actually two loops in each cortical-cerebellar module. The one through the cerebellar nucleus is predominately excitatory and is responsible for the high firing rates of voluntary movement commands (Holdefer et al. 2005). This is the amplification block in figure 1—positive feedback appears to be responsible for the amplification. The longer loop through cerebellar cortex uses the strong inhibitory output from the Purkinje cells to restrain the positive feedback and, most importantly, to set the fixed points of this attractor network (Houk & Mugnaini 2003).

How do cortical-basal ganglionic and cortical-cerebellar modules work together? Figures 8 and 9 show an example of a GPi neuron in the basal ganglia helping to select a primary movement and subsequent submovements in a tracking task. These pauses will result in disinhibition of the M1 neurons to which the GPi neuron, via thalamus, projects, thus facilitating one or more bursts of discharge. Each of these bursts would then need to be amplified and refined by the cerebellum. Amplification in intensity and time would serve to generate any given element of the M1 output vector in figure 1, and spatial amplification would recruit the large population of M1 neurons (additional elements of that vector) that are required to produce a movement (Georgopoulos 1995). The cerebellar cortex would then restrain and refine the entire M1 output vector, shaping it into a composite motor command regulating the direction, velocity and duration of a primary movement and of the subsequent corrective submovements that home in on the target. Figure 6 shows examples of two elements in that output vector.

The abstract engineering operations in figure 7 satisfactorily superimpose on the neurophysiological operations abstracted in figure 1. With the help of dopamine neuromodulation, pattern classification in the neostriatum should be capable of generating the normalized predicted error in figure 7, utilizing convergent cortical input reflecting both phasic sensory and efference copy events, and tonic contextual cues. The three-factor learning rule in the neostriatum could have, through prior experience, stored these complex patterns of cortical activity in corticostriatal synaptic weights via reinforcement learning. The resultant output vector from the basal ganglia should then be able to embody appropriate motor cortical neurons for starting a movement in approximately the right direction, thus also initiating positive feedback and amplification in the loop through cerebellar nucleus. The Purkinje cells in the cerebellar cortex could then shape population discharge into an output vector that commands a reasonable bell-shaped primary movement together with the subsequent corrective submovements that are needed to ensure an accurate overall movement.

During the course of sensorimotor learning, the cerebral cortex, basal ganglia and cerebellum work in parallel but unique ways (Houk & Wise 1995; Lu et al. 1998; Doya 1999). The loop through the basal ganglia learns to discover ballpark actions that are appropriate in a given context (Houk 2005), utilizing reinforcement learning (Sutton & Barto 1998). The loop through the cerebellum learns to refine the ballpark action through a simplified form of supervised learning (Berthier et al. 1993). The cerebral cortex, driven by input from basal ganglia and cerebellum, learns through practice to perform these operations faster and more accurately, utilizing unsupervised Hebbian learning (Bliss & Collinridge 1993; Hua & Houk 1997). The coordination of these diverse forms of learning has been simulated by Doya (1999).

(c) Relation to other behavioural and neuroscientific studies

Our attempt to study the neuroscientific mechanisms underlying serial order recall interfaces well with models of considerably more complex forms of serial order behaviour which are investigated by Botvinick & Plaut (2006). These authors promoted an activation-based serial order memory analogous to the one that appears in the model by Beiser & Houk (1998). There is a great opportunity for collaboration between psychology and neuroscience on this forefront.

The cortical-basal ganglionic networks modelled by O'Reilly & Frank (2006) use the same operational principles as outlined here. Both are based on an action selection mechanism located in the striatal input stage of the basal ganglia, and both use direct and indirect pathways that loop back to the same cortical area that provides input to the striatal layer of the model. We tend to stress an action selection competition that is more complex than the Go versus No Go competition which they typically investigate. They are more interested in modelling a diversity of action selection tasks, whereas we are more concerned with task designs that facilitate linkages with the neuroscientific mechanisms which underlie action selection. These two approaches are quite complementary and serve to explore the diversity of linkages to psychology and neuroscience, which we all wish to achieve.

It seems clear that the basal ganglia are not the only site in the brain for action selection. For example, Cisek & Kalaska (2005) are studying spatial reaching decisions for which the operation of action selection appears to take place in the premotor cortex, perhaps with only a little help from the basal ganglia. We wish to draw analogy with the diminished role of the basal ganglia in the well-rehearsed elements of the step-tracking task discussed in §3 of this paper. Our view concerning this shift is that it is related to practice. Sensorimotor shortcuts through cortex could very well develop through practice-related learning mediated by the two-factor Hebbian learning rule in the cerebral cortex.

Humphries et al. (2006) seek to understand mechanisms for action selection that are embodied in the reticular formation. They plan to investigate how the reticular formation and the basal ganglia may interact. One idea that probably warrants consideration is that actions originally selected by the basal ganglia are exported to the reticular formation as a result of the long rehearsal period that begins in the neonate. This would fit with Swanson's (2000) claim that all of the cerebral cortex, including phylogenetically older parts, have outputs via the basal ganglia which parallel direct outputs from the cerebral cortex. Exploring the direct and indirect pathways from the amygdala to the reticular formation might be worthwhile.

(d) Understanding action selection deficits in brain disorders

Neurological and psychiatric disorders often target our capacity for action selection. Computational models of these decision-making processes offer a useful approach for investigating the aetiology of a particular disease and for exploring potential treatments of the deficits. To facilitate this, it is helpful if the model is capable of bridging from molecular processes to cellular neurophysiology to systems neurophysiology to behaviour. This was one of the motivations for developing the DPM model of mind agents (Houk 2005).

In another paper of this issue, Frank et al. (2006) discuss Parkinson's disease and attention-deficit/hyperactivity disorder (ADHD) and use computational models to relate these disorders to decision-making deficits in a diverse range of tasks. One additional task that might be considered for this purpose is the serial order recall task Replicate that was described in §2 of the present paper. Serial order processing challenges the capacity for pattern classification in the striatal input stages of the loops through the basal ganglia. Simulation studies of Replicate might help to identify deficits and potential treatments for Parkinson's disease and ADHD.

(i) Implications for schizophrenia

A simplified version of the Replicate task has been studied in patients suffering from schizophrenia (Fraser et al. 2004). The patients exhibited two prominent deficits that were anticipated from the existing models: (i) in line with predictions based on Manoach's (2003) capacity model, serial order processing became saturated at three or four items in the list, as contrasted with the normal capacity of 7±2 (Miller 1956) and (ii) in line with predictions based on the Beiser & Houk (1998) network model, targets presented later in the sequence were remembered most poorly. Both highly significant deficits were attributed to defective pattern classification in the caudate nucleus of the basal ganglia. This interpretation could be tested by imaging the Replicate task. One prediction to be tested is that the decrease in caudate blood flow in the Decode contrast (figure 2) will be attenuated or even reversed in schizophrenia, assuming that there is a deficit in GABAb-mediated presynaptic inhibition.

In fact, there is a modified expression of the GABAb receptor in schizophrenia (Enna & Bowery 2004). This implicates the modified GABAbR1 gene on chromosome 6p21.3 (Martin et al. 2001) as a major contributor to schizophrenia. Since the inheritance of schizophrenia is multigenic (Freedman et al. 2001), the gene identified by Freedman, Leonard and collaborators is also strongly implicated, a gene that causes altered expression of the alpha-7 nicotinic receptor. This receptor is prevalent in many of the loops between the cerebral cortex and the cerebellar nuclei. Altered transmission in these loops is thought to contribute to the cognitive dysmetria of schizophrenia (Andreasen 1999).

A central paradox of schizophrenia is that a condition which is genetic in origin survives in the population in spite of a fecundity disadvantage. The magnitude of the latter is such that any genetic predisposition should be eliminated from the population within a few generations. Instead, since the incidence of schizophrenia remains steady at 1–2%, there must be an accompanying genetic advantage (Huxley et al. 1964). In analysing this issue, Kuttner et al. (1967) offered three potential advantageous functions that accompany the inheritance of schizophrenia: (i) a capacity for complex social relations, (ii) intelligence, and (iii) language. Crow and colleagues have made a strong case for an evolutionary link between the origin of language and the aetiology of schizophrenia (Berlim et al. 2003). Their hypothesis is consistent with the prominent deficit in competitive pattern classification in schizophrenia mentioned above—language contains abundant examples of serial order processing.

How can the model presented here help us to understand the survival of genes responsible for schizophrenia? Our model suggests that superior action selection in the Replicate task results from competitive pattern classification mediated by the presynaptic inhibition of excitatory input to the neostriatum from the cerebral cortex. It is reasonable to assume that presynaptic inhibition in the caudate nucleus depends on the GABAbR1 receptor subunit coded by the gene on chromosome 6p21.3 (Martin et al. 2001). This could explain the genetic advantage. Schizophrenia patients suffer from defective pattern classification in caudate (Fraser et al. 2004). This could be explained by occasional (1–2%) malfunctioning variants of the GABAbR1 receptor subunit associated with unfavourable epigenetic expression leading to poor or absent presynaptic inhibition. As mentioned above, it should be possible to test this model of the aetiology of schizophrenia by imaging patients and normal controls.

5. Summary

We posit that both serial order recall and online error correction are prime examples of natural action selection. They appear to use analogous mechanisms for signal processing in their respective DPMs. Models comprising networks of DPMs may provide a useful substrate for studying complex behaviours and for exploring the underlying dynamics of the mind. Such simulations may help us to understand the aetiology and treatment of Parkinson's disease, ADHD and schizophrenia.

Supplementary Material

Electronic supplementary material 1

Model of Serial Order Processing. We implement a minimal model of serial order processing and use it to contrast the computational efficacy and metabolic costs of two cellular mechanisms for mediating collateral inhibition in the striatum. Presynaptic inhibition was found to be computationally more effective and metabolically less costly than postsynaptic inhibition. The results are illustrated in Figures 3–5 of Houk et al. 2007 [J. C. Houk, C. Bastianen, D. Fansler, A. Fishbach, D. Fraser, P. J. Reber, S. A. Roy & L. S. Simo 2007 Action selection and refinement in subcortical loops through basal ganglia and cerebellum. Phil. Trans. R. Soc. B 362 (doi:10.1098/rstb.2007.2063)

Electronic supplementary material 2

MatLab codes