Translating working memory into action: Behavioral and neural evidence for using motor representations in encoding visuo‐spatial sequences

Abstract

The neurobiological organization of action‐oriented working memory is not well understood. To elucidate the neural correlates of translating visuo‐spatial stimulus sequences into delayed (memory‐guided) sequential actions, we measured brain activity using functional magnetic resonance imaging while participants encoded sequences of four to seven dots appearing on fingers of a left or right schematic hand. After variable delays, sequences were to be reproduced with the corresponding fingers. Recall became less accurate with longer sequences and was initiated faster after long delays. Across both hands, encoding and recall activated bilateral prefrontal, premotor, superior and inferior parietal regions as well as the basal ganglia, whereas hand‐specific activity was found (albeit to a lesser degree during encoding) in contralateral premotor, sensorimotor, and superior parietal cortex. Activation differences after long versus short delays were restricted to motor‐related regions, indicating that rehearsal during long delays might have facilitated the conversion of the memorandum into concrete motor programs at recall. Furthermore, basal ganglia activity during encoding selectively predicted correct recall. Taken together, the results suggest that to‐be‐reproduced visuo‐spatial sequences are encoded as prospective action representations (motor intentions), possibly in addition to retrospective sensory codes. Overall, our study supports and extends multi‐component models of working memory, highlighting the notion that sensory input can be coded in multiple ways depending on what the memorandum is to be used for. Hum Brain Mapp 35:3465–3484, 2014. © 2013 Wiley Periodicals, Inc.

INTRODUCTION

Working memory (WM) refers to the ability to mentally maintain or manipulate a limited amount of information across short intervals during which this information is not available to perception [Baddeley and Hitch, 1974; Baddeley, 2012; Cowan, 1988]. Obviously, this mnemonic function is essential for many everyday behaviors, including the performance of movement sequences that are determined by previously presented (and currently unavailable) sensory input. Examples include the vocal reproduction of a just read sentence or the dialing of a just heard telephone number. To plan and execute such memory‐guided serial movements, the perceived sensory information needs to be transiently retained in WM and then translated into a sequential motor program [Lashley, 1951; Miller et al., 1960; Ohbayashi et al., 2003]. In humans, the ability for serial recall of visuo‐spatial sequences is frequently assessed using some version of the Corsi block‐tapping task (CBT) [Corsi, 1972; see also Berch et al., 1998; Milner, 1971]. This task requires participants to observe and subsequently reproduce a series of manual tapping movements on a board with scattered cubic blocks. Thus, the task comprises the encoding and retention of a temporo‐spatial pattern of visually presented manual actions (i.e., the tapping movements) and the conversion of this pattern into a corresponding movement sequence during recall. Despite the wide‐spread clinical use of this and similar “spatial span” tasks, their neural basis has rarely been examined.

Using positron emission tomography (PET), Owen et al. [1996, 1999] revealed activation in early visual areas (V1, V2), bilateral premotor and posterior parietal cortex, as well as right ventrolateral prefrontal cortex during spatial serial‐recall tasks. Subsequent PET studies reported that ordered versus irregular spatial target arrangements produced additional activation in middle frontal gyrus (i.e., dorsolateral prefrontal cortex, DLPFC), which was interpreted to facilitate the use of a pattern‐based chunking strategy during encoding [Bor et al., 2001, 2006]. Findings in patients with frontal lobe lesions likewise corroborated the contribution of both ventro‐ and dorsolateral PFC to spatial‐span task performance [Bor et al., 2006].

More recently, studies employing event‐related functional magnetic resonance imaging (fMRI) enabled separating activity related to encoding, retention, and recall. They revealed that lateral prefrontal, posterior parietal, premotor and sensorimotor cortices were consistently activated during both the encoding [Bor et al., 2003; Toepper et al., 2010] and retention [Pochon et al., 2001; Volle et al., 2005] phases of spatial serial‐recall tasks. Using a serial‐recall paradigm with two successive delays, Volle et al. [2005] were able to dissociate brain activity related to short‐term maintenance and response organization during retention. They argued that inferior posterior parietal, sensorimotor and premotor cortices as well as left DLPFC and other action‐related brain regions subserve WM‐guided response preparation rather than mere information storage in WM. This reasoning is supported by a study in which monkeys were trained to reproduce a previously memorized sequence of positional cues by making saccades in either the original or reverse order [Ohbayashi et al., 2003]. Firing rate of neurons in the rostral aspect of dorsal premotor cortex (dPMC) selectively increased upon occurrence of a cue indicating reproduction order, suggesting the participation of dPMC in translating visuo‐spatial order information from WM into a sequential motor program.

In standard serial‐recall tasks like the CBT (i.e., in tasks without a separate delayed instruction cue for response order or effector side), action sequence and effector are unequivocally determined by the stimulus sequence presented. Thus, the translation of temporo‐spatial sensory information into action‐related codes could occur immediately at encoding. Previous studies, however, have given no definite answer to this question. For instance, Pochon et al. [2001] compared conditions where participants either had to decide whether a given spatial sequence matched a previously shown one (recognition task) or had to reproduce a previously shown sequence via sequential button presses (reproduction task). The reproduction task elicited increased delay‐related activity in right DLPFC, left motor and premotor cortices, left supplementary motor area (SMA), and bilateral anterior putamen. In contrast, activity during sequence presentation (i.e., encoding) was not significantly different between recognition and reproduction tasks. Existent differences, however, might have failed significance because of the moderate statistical power associated with the small sample size (n = 8), since when compared against task‐specific control conditions, the reproduction but not the recognition task showed elevated activity during encoding in action‐related areas such as dorsal and ventral PMC, SMA, right inferior frontal gyrus, and left anterior putamen. Similarly, Macaluso et al. [2007] showed that delay activity before making a WM‐guided lateralized response reflected the effector (eye or hand) of the forthcoming movement rather than the sensory modality of the preceding instruction cue, but they did not report whether also cue‐related activity already predicted the effector.

Taken together, despite converging evidence for a large‐scale brain network subserving different subprocesses in spatial‐span performance, the question whether action‐related representations contribute to encoding visuo‐spatial sequences for delayed serial recall is still open. Obviously, visuo‐spatial sequences could simply be maintained as retrospective sensory representations, reflecting the just received input, which would correspond to the idea of a visuo‐spatial sketchpad [Baddeley, 2012; Baddeley and Hitch, 1974]. Instead (or in addition), they could also be maintained as prospective action codes, reflecting the intended output [cf. Eschen et al., 2007; Freeman and Ellis, 2003; Krigolson et al., 2012]. We used fMRI to test four predictions derived from the latter view:

If visuo‐spatial sequences were transformed into action‐related representations already at encoding (possibly in addition to sensory memory codes), then…

1. …there should be major commonalities between encoding‐ and reproduction‐related brain activity, especially in brain regions associated with action control (e.g., frontomedial or premotor cortex);

2. …encoding‐related activity (in brain areas associated with sequence reproduction) might predict effector side used for subsequent reproduction;

3. …introducing a prolonged retention interval, which is most likely bridged using rehearsal [Baddeley, 2012], should affect behavior and brain activity during recall, because rehearsing such codes during a long delay can be assumed to facilitate the implementation of concrete motor programs at reproduction [cf. Jeannerod, 2001], whereas sensory codes would need to be transformed from scratch into an action code at recall, irrespective of delay length;

4. …there might be activity in nonsensory, execution‐related brain areas during encoding that predicted subsequent correct recall, which would also provide evidence for the functional relevance of action‐related codes.

In summary, our study aimed at characterizing the brain networks implementing WM‐guided movement sequences and the type(s) of representation (i.e., sensory and/or motor) on which spatial serial recall performance operates.

METHODS

Participants

We examined 36 healthy volunteers [aged from 18 to 72 (M = 38.1; SD = 13.2) years; 15 females] without any record of neurological or psychiatric disorders and normal or corrected‐to‐normal vision. Absence of psychiatric disorders was assessed by using the Structured Clinical Interview for DSM‐IV [Wittchen et al., 1997]. All participants gave informed written consent to the study protocol approved by the local ethics committee of the RWTH Aachen University Hospital.

Experimental Protocol

Participants performed a visuomotor delayed‐recall task with the following trial structure (see Fig. 1): Each trial started by visually presenting the German word “Achtung” (Attention) for 500 ms. Subsequently, a schematic drawing of a right or left hand was presented for 1,000 ms. Successively appearing red dots (presented for 250 ms each) then indicated a sequence of fingers on the given hand. The short dot presentation time was chosen to be lower than typical response times in speeded four‐choice reaction tasks to suppress overt imitation of the indicated finger movements during sequence presentation. The success of this strategy was visually verified during both practice and experimental sessions. Total sequence length pseudo‐randomly varied (equally distributed) across trials between four and seven items. Participants were to memorize the sequence presented on the screen. After presenting the sequence, a black screen was shown for a variable period of either 500 or 7,000 ms (pseudo‐randomized across trials). The subsequent appearance (500 ms) of a green circle served as the “go” cue for sequence reproduction. Participants were instructed to reproduce the memorized sequence by pressing buttons of an MR‐compatible keyboard (LumiTouch, Burnaby, Canada) as fast and correctly as possible with the fingers corresponding to those previously marked on the schematic hand. After the response, a black screen was presented for a variable inter‐trial interval of 7–11 s (uniformly jittered, mean: 9 s).

Figure 1.

Temporal trial structure. Each trial started with an attention cue (Achtung) followed by a schematic left or right hand presented for 1,000 ms. Visual cues (red dots) indicating a 4‐ to 7‐item unilateral sequence of fingers to be memorized were shown for 250 ms each. After a go cue (green circle), presented after either a short (500 ms) or long (7,000 ms) delay, the participants' task was to reproduce the sequence they had been shown as fast and correctly as possible by a series of spatially corresponding finger movements. Trials were separated by a variable (7–11 s) inter‐trial interval, during which a blank screen was shown.

Each sequence length (four, five, six, or seven items) was crossed with both types of delay (500 or 7,000 ms) and the two possible response sides (left or right hand), yielding 16 distinct conditions. Each condition was presented six times throughout the whole experiment. Thus, in total, 96 trials were presented in a randomized fashion. The experiment was run on a Windows PC using Presentation 12.0 (Neurobehavioral Systems, Austin, Texas). All visual stimuli were displayed on a custom‐built, shielded TFT screen at the rear end of the scanner visible via a mirror mounted on the head coil (14° × 8° viewing angle, 245 mm distance from the participant's eyes).

Before scanning, participants received task instructions and practiced the task for 5 min. The instructions emphasized that participants should abstain from attempts to overtly imitate any sequence parts during encoding and retention. After the experiment, participants were asked to describe their strategies, if any, for remembering sequences.

Behavioral Data Analysis

The behavioral parameters recorded during the fMRI experiment were analyzed using MATLAB (MathWorks, Natick, USA). The following measures of performance were computed: percentage of correct reproductions, initial reaction time (i.e., the time interval between go‐signal onset and first button press), and mean inter‐response time (i.e., the time interval between first and last button press divided by sequence length minus one). Speed measures were derived from correct trials only. The effect of the experimental factors (reproduction onset: immediate or delayed; memory load: four to seven items) on performance was compared by 2 × 4 repeated‐measures analyses of variance (ANOVAs).

fMRI Data Acquisition

Images were acquired on a Siemens Trio 3‐T whole‐body scanner (Erlangen, Germany) using blood‐oxygen‐level‐dependent (BOLD) contrast [Gradient‐echo echo‐planar imaging (EPI) pulse sequence, TE = 30 ms; TR = 2.2 s; flip angle = 90°; in‐plane resolution = 3.1 × 3.1 mm²; 36 axial slices (3.1‐mm thickness; distance factor = 15%) covering the entire brain]. Image acquisition was preceded by four dummy scans (excluded from further analysis) allowing for magnetic field saturation.

fMRI Data Analysis

Images were analyzed using SPM5 (http://www.fil.ion.ucl.ac.uk/spm). First, the EPI images were corrected for head movement by affine registration using a two‐pass procedure by which images were initially realigned to the first image and subsequently to the mean of the realigned images. After realignment, the mean EPI image for each subject was spatially normalized to the Montreal Neurological Institute (MNI) single‐subject template using the “unified segmentation” approach [Ashburner and Friston, 2005]. The resulting parameters of a discrete cosine transform, which define the deformation field necessary to move the participants' data into the space of the MNI tissue probability maps, were then combined with the deformation field transforming between the latter and the MNI single‐subject template. The ensuing deformation was subsequently applied to the individual EPI volumes, which were hereby transformed into the MNI single‐subject space and resampled at 2 × 2 × 2 mm³ voxel size [cf. Kellermann et al., 2012; Langner et al., 2012]. The normalized images were spatially smoothed using an 8‐mm FWHM Gaussian kernel to meet the statistical requirements of the general linear model and compensate for residual anatomical variation between participants.

Statistical analysis relied on the general linear model as implemented in SPM5. Six conditions (sequence encoding as well as immediate and delayed recall, separately for left‐ and right‐hand trials, respectively) were modeled by convolving boxcar reference vectors with a canonical hemodynamic response function (HRF). Vector length corresponded to the individual duration of each condition (i.e., varied with sequence length both at encoding and recall). Thus, the task regressors also captured a major part of memory‐load‐dependent activity, obviating the additional assessment of parametric load effects. We chose this approach to capture each given trial epoch (i.e., encoding or recall) in its entirety and explain as much task‐related BOLD signal variance as possible. Evidently, the length of those epochs was naturally correlated with load (i.e., sequences containing more items took longer to present and reproduce). Given that we were not explicitly interested in neural load effects, this inclusive modeling of load effects also provided for a more valid estimation of WM‐related hemodynamic activity, as load effects form a core aspect of WM performance and related brain activity.

For each of the six conditions, trials were separated according to whether or not sequences were reproduced correctly, yielding a total of twelve regressors. The pseudo‐random order of left‐ vs. right‐hand sequences and immediate vs. delayed recall provided for low correlations between regressors and good parameter estimability. Additionally, the first‐order temporal derivatives were included as regressors of no interest to account for minor temporal variability of the HRF. Low‐frequency signal drifts were filtered using a cut‐off period of 128 s. Parameter estimates were subsequently calculated using weighted least squares to provide maximum‐likelihood estimators based on the temporal autocorrelation of the data [Kiebel and Holmes, 2003].

Regressors that modeled incorrect reproductions as well as temporal derivatives were not included in the subsequent group‐level analysis, which employed a random‐effects ANOVA (factor: condition, blocking factor: subject). Thus, individual effects for eight experimental conditions of interest were fed into this second‐level analysis: encoding in trials with subsequent correct recall, encoding in trials with subsequent incorrect recall, immediate recall, and delayed recall (each separately for left‐ and right‐hand sequences). In the modeling of variance components, we allowed for violations of sphericity by modeling nonindependence across images from the same participant and allowing unequal variances between conditions and participants using the standard implementation in SPM5. Simple main effects of each experimental condition (vs. the implicit resting baseline) and comparisons between conditions were tested by applying appropriate linear contrasts to the ANOVA parameter estimates. Activity shared between encoding and recall (i.e., activations present in all three main conditions: encoding, immediate and delayed recall) was tested by means of a conjunction analysis using the strict minimum t‐statistic [Friston et al., 2005]. The same approach was used to test for “effector‐independent” activity (i.e., activations associated with both left‐ and right‐hand sequences). It should be noted, though, that our use of the term “effector‐independent” only refers to the laterality of the hand used and is not to imply independence of the general type of effector (e.g., hand vs. eye). Put differently, we here assessed activity that was independent of the specific effector (left vs. right hand) but may still be specific to the particular effector system (hands). Except where noted, all resulting SPM{T} maps were thresholded at P < 0.05 (family‐wise error–corrected for multiple comparisons at voxel level; minimum cluster extent: 10 voxels) and anatomically localized using the SPM Anatomy toolbox 1.7 [Eickhoff et al., 2005, 2007; http://www.fz-juelich.de/inm/inm-1/EN/Forschung/_docs/SPMAnatomyToolbox/SPMAnatomyToolbox_node.html.

RESULTS

Behavioral Data

Higher WM load (i.e., longer sequences) produced significantly more errors in reproduction [F(3, 280) = 98.93, P < 0.001; Fig. 2A]; there was no load effect on initial reaction time (RT) and mean inter‐response time (all F < 1.2, ns; Fig. 2B,C). The length of the delay between encoding and reproduction had no effect on accuracy and mean inter‐response time (all F < 1.5, ns), but initial RT was significantly shorter after long delays [F(1, 254) = 12.72, P < 0.001; Fig. 2B]. There were no significant interaction effects on any performance measure (all F < 0.3, ns).

Figure 2.

Group‐averaged performance measures [(A) percentage of correct sequence reproductions; (B) response time of the first button press within a given sequence; (C) average response time for subsequent button presses within a given sequence] for each delay length and memory load. Error bars represent the standard error of the mean.

Imaging Data

Hypothesis I: Common activity during encoding and recall

A conjunction analysis was conducted to test for effector‐independent activity overlap during encoding and recall. Significant overlap, especially in execution‐related brain regions, would argue for using prospective, output‐related codes at encoding (possibly in addition to retrospective sensory codes). Subsequently, we tested for significant activity differences between encoding and recall. As the latter two contrasts were not motivated by a specific hypothesis, their results are reported in the Supporting Information (see Supporting Information Text, Fig. S1, and Table S1).

Effector‐independent activity shared between encoding visuo‐spatial sequences and reproducing them by sequential finger movements was revealed by a conjunction across encoding and recall epochs for both hands and both delays, restricted to trials with correct recall (Fig. 3, Table 1). Increased activity was found bilaterally in middle frontal gyrus (i.e., mid‐DLPFC) and a large frontomedial cluster comprising parts of SMA and pre‐SMA [corresponding to human cytoarchitectonic area 6 (see table notes for references to cytoarchitectonic assignments) and anteriorly adjacent cortex] as well as anterior midcingulate cortex (MCC: caudal aspects of areas a24′ and 32′). Further bilateral activity increases were observed in rostral aspects of both dorsal and ventral premotor cortices (dPMC and vPMC). The dPMC cluster centered around the superior frontal junction covering posterior aspects of superior and middle frontal gyrus, superior frontal sulcus, and superior aspects of the precentral sulcus; this partially overlaps with the presumed location of the human frontal eye fields [FEFs; Paus, 1996; cf. zu Eulenburg et al., 2012]. The vPMC cluster (rostral aspect of ventrolateral area 6) extended anteroventrally via inferior frontal junction (IFJ) and IFG pars opercularis (area 44) to anterior insula. Parietal activity was seen bilaterally in superior parietal lobule (SPL: areas 7A, 7P, and 7PC), intraparietal sulcus (IPS: areas hIP1‐3), and temporo‐parietal junction (TPJ: areas PF, PFt), extending to caudal aspects of primary somatosensory cortex (SI: area 2). Finally, enhanced neural activity during both sequence encoding and reproduction was also observed in bilateral posterior superior temporal sulcus (STS), right temporo‐occipital junction, bilateral cerebellar lobule VI, and a large bilateral cluster in the basal ganglia (dorsal and ventral aspects of globus pallidus and anteromedial putamen as well as substantia nigra], extending to the ventral anterior and ventrolateral thalamus (Fig. 3).

Figure 3.

Effector‐independent brain activity across both encoding and correctly recalling visuo‐spatial sequences, irrespective of delay length. Section coordinates refer to MNI space. Abbreviations: aMCC = anterior midcingulate cortex; dlPFC = dorsolateral prefrontal cortex; dPMC = dorsal premotor cortex; pre‐SMA = pre‐supplementary motor area; SPL = superior parietal lobule; STS = superior temporal sulcus; TOJ = temporo‐occipital junction; TPJ = temporo‐parietal junction; vPMC = ventral premotor cortex.

Table 1.

Effector‐independent brain activity across encoding and recalling visuo‐spatial sequences

Location	Histological Assignment	Local Maximum	t‐value
Left hemisphere
SMA	Area 6	−1 −2 56	16.0
Postcentral sulcus	Area 2	−44 −42 50	14.9
Ventral premotor cortex	—	−54 5 33	12.9
Dorsal premotor cortex	Area 6	−33 −9 62	12.8
Temporo‐parietal junction	PFcm, PFt, PFop	−51 −36 21	11.9
Anterior insula	—	−35 23 0	11.8
Anterior midcingulate cortex	Areas a24′, 32′	−1 15 42	10.9
Inferior frontal gyrus	Area 44	−51 8 18	10.6
Superior parietal lobule	7A	−30 −56 65	9.5
Pallidum	—	−15 8 −3	9.5
Cerebellum	Crus I	−39 −56 −30	9.4
Putamen	—	−18 8 −1	8.7
Mid‐dorsolateral prefrontal cortex	—	−38 27 27	7.8
Cerebellum	Lobule VI	−17 −69 −24	7.7
Intraparietal sulcus	hIP1, hIP2, hIP3	−32 −60 42	5.6
Posterior superior temporal sulcus	—	−56 −54 2	5.4
Parietal operculum	OP4	−60 −15 23	5.3
Thalamus	—	−17 −12 9	5.3
Right hemisphere
Intraparietal sulcus	hIP2	47 −39 47	12.9
SMA	Area 6	3 −2 54	12.8
Postcentral sulcus	Area 2	40 −36 44	12.6
Cerebellum	Lobule VI	33 −60 −27	12.4
Temporo‐parietal junction	PFm, PF	44 −45 48	12.2
Inferior frontal gyrus	Area 44	51 8 21	12.1
Dorsal premotor cortex	Area 6	30 −6 62	10.0
Ventral premotor cortex	—	57 8 33	9.8
Anterior insula	—	33 23 −1	9.7
Superior parietal lobule	7P	15 −74 59	9.3
Pallidum	—	21 6 0	8.4
Precuneus	7A	9 −68 48	8.1
Posterior superior temporal sulcus	—	51 −33 −3	7.8
Anterior midcingulate cortex	Areas a24′, 32′	8 14 33	7.5
Putamen	—	20 9 6	7.4
Mid‐dorsolateral prefrontal cortex	—	41 30 24	6.8
Temporo‐occipital junction	—	60 −56 −8	6.3
Cerebellum	Vermis	5 −72 −20	6.0
Thalamus	—	18 −15 14	5.1

Peak coordinates refer to MNI space. SMA = supplementary motor area; References to histological assignments: area 2: Grefkes et al. [2001]; area 6: Geyer [2004]; areas a24′, 32′: Palomero‐Gallagher et al. [2008]; area 44: Amunts et al. [1999]; hIP1, hIP2: Choi et al. [2006]; PFop, PF, PFt, PFcm, PGp, PFm: Caspers et al. [2006]; OP4: Eickhoff et al. [2006];7A, 7P: Scheperjans et al. [2008]; Cerebellum: Diedrichsen et al. [2009].

Hypothesis II: Effector‐specific activity common to both encoding and recall

Effector‐specific activity common to encoding and recall would be found if encoding entailed activity in brain regions involved in controlling motor output for sequence generation during recall, arguing for motor‐related representations during encoding. Brain activity selectively related to encoding left‐hand (vs. right‐hand) sequences that was shared with left‐hand recall was revealed by a conjunction analysis across the difference contrast “encoding left‐hand sequences > encoding right‐hand sequences” and the three main effects of encoding left‐hand sequences and reproducing left‐hand sequences for both delays, restricted to trials with correct recall. Conversely, activity selectively linked to encoding right‐hand sequences was revealed by an analogous conjunction analysis across the reverse difference contrast and the three main effects of encoding and reproducing right‐hand sequences. In both analyses, we observed increased activity in a large cortical cluster contralateral to the response hand, respectively (Fig. 4; Table 2). This cluster focused on aspects of somatosensory (mainly area 2 but extending to areas 1, 3a, and 3b) and primary motor cortex (areas 4a and 4p) that typically harbor hand‐specific representations; it extended posteriorly to SPL (areas 7A and 7PC) and anteriorly to dPMC and SMA (both area 6). Additionally, both analyses revealed significant contralateral activation in the vicinity of the ventral posterolateral nucleus of the thalamus as well as ipsilateral activation in the cerebellum (lobule VI). Importantly, both conjunction analyses revealed effector‐specific networks that closely resembled those for active (unilateral) hand use [Grefkes et al., 2008] but showed this activity also during the encoding of the visual material. It should be stressed that during encoding, no button presses were recorded, nor were overt finger movements detected visually, demonstrating the success of our strategy to discourage overt imitation by high‐speed dot presentation as well as respective instructions during the prescan training.

Figure 4.

Effector‐specific brain activity across both encoding and correctly recalling visuo‐spatial sequences, irrespective of delay length. Warm colors denote higher activity during encoding left‐hand (vs. right‐hand) sequences; cool colors denote higher activity during encoding right‐hand (vs. left‐hand) sequences. Section coordinates refer to MNI space. Abbreviations: dPMC = dorsal premotor cortex; MCC = midcingulate cortex; S1 = primary somatosensory cortex; SMA = supplementary motor area; SPL = superior parietal lobule.

Table 2.

Effector‐specific brain activity across encoding and recalling visuo‐spatial sequences

Location	Histological assignment	Local maximum	t‐value
Left‐ vs. right‐hand sequences
Left hemisphere
Cerebellum	Lobule VI	−33 −48 −26	6.1
Right hemisphere
Postcentral sulcus	Area 2	45 −35 51	11.3
Dorsal premotor cortex	Area 6	30 −12 60	10.6
Central sulcus	Area 3a	38 −28 42	9.3
Postcentral gyrus	Area 3b	39 −28 45	8.7
Superior parietal lobule	7PC	30 −53 59	8.5
Primary motor cortex	Area 4p	34 −27 50	8.3
SMA	Area 6	8 −6 54	7.9
Primary motor cortex	Area 4a	36 −26 57	7.8
Postcentral gyrus	Area 1	45 −34 57	7.3
Superior parietal lobule	7A	20 −62 68	6.0
Thalamus	—	17 −20 2	5.4
Right‐ vs. left‐hand sequences
Left hemisphere
Postcentral sulcus	Area 2	−41 −35 51	11.4
Postcentral gyrus	Area 3b	−41 −32 50	10.9
Dorsal premotor cortex	Area 6	−32 −18 57	9.6
Central sulcus	Area 3a	−32 −33 48	8.6
Superior parietal lobule	7PC	−36 −47 66	8.0
Primary motor cortex	Area 4a	−36 −23 57	7.9
Primary motor cortex	Area 4p	−37 −28 53	7.7
SMA	Area 6	−8 −8 53	6.8
Superior parietal lobule	7A	−28 −53 67	6.6
Thalamus	—	−17 −21 5	6.5
Postcentral gyrus	Area 1	−51 −18 47	5.2
Right hemisphere
Cerebellum	Lobule VI	33 −50 −29	5.9

Peak coordinates refer to MNI space. SMA = supplementary motor area; References to histological assignments: areas 1, 3a, 3b: Geyer et al. [1999]; area 2: Grefkes et al. [2001]; areas 4a, 4p: Geyer et al. [1996]; area 6: Geyer [2004]; 7A, 7PC: Scheperjans et al. [2008]; Cerebellum: Diedrichsen et al. [2009].

Hypothesis III: Effects of the retention interval

Differences in recall‐related brain activity after short versus long delays would argue for the use of action‐related codes during rehearsal (which can be assumed to be specific to bridging long delays), while no differences would be predicted from the assumption that retrospective sensory codes are used for rehearsal. The analysis of brain activity selectively associated with delayed, relative to immediate, recall was assessed across both hands and restricted to the delayed‐recall‐specific network via additionally including the contrast “delayed recall > encoding” and the main effect of delayed recall (for each response side, respectively) in the conjunction analysis. Results of this complex conjunction across six contrasts were considered significant at cluster‐level P < 0.05 (FWE‐corrected; cluster‐forming threshold at voxel level: P < 0.001). The analysis revealed significantly increased activity in a set of regions (Fig. 5; Table 3) that overlapped with major parts of the network previously found to be selectively associated with recall (vs. encoding) across both delays (see above). Specifically, delayed versus immediate recall evoked significantly higher bilateral activation in a cluster extending from SMA (area 6) to anterior MCC (caudal aspects of areas a24′ and 32′); in a cluster reaching from vPMC (ventrolateral aspect of area 6) further ventrally via IFJ to IFG pars opercularis (caudal area 44); in the central insula; and in the mid‐putamen (on the right side extending medially to the pallidum). Increased activation was also found bilaterally in a large anterior parietal cluster comprising SI/SII (areas 2, 3b, OP1, and OP4) as well as adjacent left SPL and bilateral inferior parietal areas PFt, PFop, PF, and PFcm. In summary, a large part of the global recall network was more strongly activated during delayed versus immediate recall.

Figure 5.

Brain regions showing higher activity during delayed than immediate (correct) recall, irrespective of effector side. Section coordinates refer to MNI space. Abbreviations: IPC = inferior parietal cortex; MCC = midcingulate cortex; S1 = primary somatosensory cortex; SMA = supplementary motor area; SPL = superior parietal lobule; TPJ = temporo‐parietal junction; vPMC = ventral premotor cortex.

Table 3.

Effector‐independent brain activity related to delayed (vs. immediate) recall

Location	Histological Assignment	Local Maximum	t‐value
Left hemisphere
Postcentral sulcus	Area 2	−57 −15 29	6.3
Parietal operculum	OP1	−60 −15 15	6.3
Putamen	—	−29 6 −5	5.9
Postcentral gyrus	Area 3b	−44 −23 36	5.9
Ventral premotor cortex	Area 6	−57 3 30	5.6
Temporo‐parietal junction	PFt, PFop, PFcm, PF	−51 −23 36	5.5
Superior parietal lobule	7PC	−36 −47 60	4.9
Anterior midcingulate cortex	Areas a24′, 32′	−11 14 32	4.9
Superior parietal lobule	7A	−30 −54 65	4.2
SMA	Area 6	−9 −14 69	4.1
Primary motor cortex	Area 4a	−38 −17 51	4.0
Central insula	—	−35 3 5	3.8
Inferior frontal gyrus	Area 44	−56 2 20	3.7
Parietal operculum	OP4	−62 −3 8	3.3
Right hemisphere
SMA	Area 6	3 −5 59	7.2
Postcentral gyrus	Area 3b	51 −17 39	6.4
Putamen	—	24 5 −6	6.0
Anterior midcingulate cortex	Areas a24′, 32′	8 6 41	5.8
Temporo‐parietal junction	PF, PFop, PFt, PFcm	62 −32 17	5.5
Pallidum	—	23 5 2	5.3
Postcentral sulcus	Area 2	41 −32 42	5.2
Inferior frontal gyrus	Area 44	59 5 17	5.1
Ventral premotor cortex	Area 6	39 −12 51	4.7
Parietal operculum	OP4	62 −17 20	4.6
Central insula	—	41 3 3	4.5

Peak coordinates refer to MNI space. SMA = supplementary motor area; References to histological assignments: area 2: Grefkes et al. [2001]; area 3b: Geyer et al. [1996]; area 4a: Geyer et al. [1996]; area 6: Geyer [2004]; 7A, 7PC: Scheperjans et al. [2008]; areas a24′, 32′: Palomero‐Gallagher et al. [2008]; area 44: Amunts et al. [1999]; OP1, OP3, OP4: Eickhoff et al. [2006];PFop, PF, PFt, PFcm: Caspers et al. [2006].

The reverse comparison (i.e., immediate relative to delayed recall) was analogously restricted to immediate‐recall‐specific activity (by including the contrast “immediate recall > encoding” and the main effect of immediate recall for each response side in the conjunction) but did not yield any significantly increased activation (at P < 0.05, FWE‐corrected at cluster level).

Two supplementary analyses tested for effector‐specific effects of delayed versus immediate recall. To this end, we performed the above‐described conjunction analysis separately for each response side and additionally restricted the results to side‐specific effects by including the contrast “delayed recall with left hand > delayed recall with right hand” (and vice versa, respectively). Both analyses revealed increased activity in cortical and subcortical sensorimotor‐related regions contralateral to the response hand, respectively, including SI and SII, primary and supplementary motor, inferior and superior parietal cortex as well as ventrolateral thalamus and posterior putamen (see Supporting Information Fig. S2 and Table S2). In summary, additional side‐specific activity during delayed (vs. immediate) recall was restricted to contralateral sensorimotor and thalamic structures, but several parts of the global delayed‐recall network (cf. Fig. 5) showed side‐specifically pronounced activation.

Hypothesis IV: Encoding‐related activity selectively associated with correct recall

The presence of activity in execution‐related brain regions during encoding that predicts accurate reproduction would not only suggest the use of action‐related codes at encoding but would also provide evidence for the functional significance of these codes. Activity during the encoding of sequences that were subsequently reproduced correctly (as opposed to those that were not) was initially assessed separately for each response hand. Contrasting correctly with incorrectly recalled left‐hand sequences (in conjunction with the main effects “encoding left‐hand sequences in correctly recalled trials” and “recalling left‐hand sequences across both delays”) revealed significantly increased neural activity in bilateral putamen. The corresponding contrast for right‐hand sequences showed increased activity only in the left putamen. Consequently, the subsequent conjunction analysis across both hands also yielded significant activation only in the left middle putamen (peak activity at: −24/5/−5; t = 5.7; see Supporting Information Fig. S3).

DISCUSSION

This study investigated neural correlates of encoding visuo‐spatial stimulus sequences and reproducing them by means of WM‐guided sequences of finger movements after variable delays. The major focus of the study lay on testing whether such visual‐spatial sequences are transformed into execution‐related codes already at encoding. In the following, we will discuss the behavioral and neural evidence obtained bearing on this question.

Behavioral Performance

Not surprisingly, longer sequences were harder to memorize, as evidenced by the gradual decrease in reproduction accuracy with increasing sequence length (see Fig. 2A). The speed of initiating sequence reproduction, however, was unaffected by sequence length. This argues against a programming of the complete sequence before initiation of the first movement, as previous research has shown that such preprogramming of entire movement sequences produces initiation delays that increase with sequence length or complexity [Haaland et al., 2004; Sternberg et al., 1978]. Full preprogramming, however, appears to be restricted to simple and short motor sequences performed under time pressure [Franks et al., 1998; Rosenbaum et al., 1987; van Donkelaar and Franks, 1991].

Partial or full sequence preprogramming during the (long) retention interval is also rather unlikely given that the repetitive nature of rehearsal would most probably have interfered with the parallel programming of a fixed motor sequence [Ilan and Miller, 1998]. Finally, motor preprogramming during the delay would be at odds with finding no difference in mean inter‐response time between the two retention intervals, since using the long delay to preprogram the entire sequence should otherwise have significantly accelerated execution. Sequence reproduction will hence have most likely relied on programming the second and subsequent elements of the sequence while executing preceding elements, making initiation speed independent of sequence length. Thus, recall processes will have occurred throughout the reproduction epoch [cf. Palmer, 2005; Palmer and Pfordresher, 2003]. Furthermore, the absence of differences in mean inter‐response time between sequences of different length is compatible with the hypothesis that the same type of (kinematic) programming was used for all sequence lengths.

If, however, the execution of the movement sequences was not preprogrammed during the retention interval, how did the acceleration of the response initiation occur? We suggest that encoding the sequence entailed the automatic generation of an abstract motor representation [Jeannerod, 1994], previously labeled “action plan” [Hoshi and Tanji, 2007] or “motor intention” [Andersen and Buneo, 2002; Jeannerod, 1999]. Such higher‐level representations are distinct from kinematic planning [Haggard, 2005] and refer to the result of integrating information on actions (e.g., to‐be‐used body parts and targets of their movement) to establish the goals of an intended action [cf. Hoshi and Tanji, 2007]. Action‐related (as opposed to stimulus‐related) sequence coding is also suggested by the independence of performance accuracy from delay length, because action codes are thought to be maintained easily at a rather stable level, that is, without the rapid decay that is typical of sensory memories [Toni et al., 2002].

Alternatively, motor circuits and, presumably, motor representations might have been automatically activated by observing an action [Caspers et al., 2010; di Pellegrino et al., 1992] or by receiving information that specifies a particular action [Cisek and Kalaska, 2004; Johnson et al., 2002; Ramnani and Miall, 2004]. As the serial presentation of red dots marking the order of subsequent button presses constitutes a prime example of such information, it may hence have resulted in an automatic action‐related representation of the visuo‐spatial stimulus sequence [cf. Heyes, 2011]. Such automatically generated, imitation‐related motor representations would probably be more concrete and include, for example, kinematic information, as compared with the more abstract action plans alluded to above. The presence of such, more concrete, motor‐related codes already at encoding is supported by the effector‐specific activity in primary sensorimotor regions during sequence presentation. It is, however, at odds with the aforementioned evidence against fully and concretely preprogramming the subsequently required motor sequence. It should moreover be noted that such preprogramming would necessitate some inhibitory process to hold well‐prepared responses in check prior to the “go” signal [Burle et al., 2010; Duque and Ivry, 2009; Jennings and van der Molen, 2005], which disagrees with our observing an initial‐RT benefit, rather than an inhibition‐induced slowing, after long (vs. short) delays [cf. Jaffard et al., 2008]. Thus, we tentatively conclude that encoding to‐be‐reproduced visuo‐spatial sequences may initially entail imitation‐related, concrete motor representations that, in turn, are retained in a more abstract form, so as to reduce memory load by reducing the level of detail and to prevent (mutual) interference between sequence elements during rehearsal.

Mentally rehearsing such sequential action plans during the long delay may, in turn, have facilitated programming the sequence upon “go” signal occurrence [Fadiga et al., 1995; Jeannerod, 1999, 2001; Léonard and Tremblay, 2007]. According to the sequence scheduling model by Rosenbaum et al. [1984, 1987], this facilitation will reduce initiation time, producing the initial‐RT benefit. Arguably, part of the initial‐RT benefit might also be due to improved temporal (i.e., nonspecific) preparation at the second imperative moment, which is the typical finding in variable‐delay RT paradigms [Langner et al., 2010a; Steinborn and Langner, 2012]. However, as the error in subjective time estimation would counteract this preparation benefit, especially in the current case of rather long delays [Niemi and Näätänen, 1981], nonspecific temporal preparation is unlikely to contribute substantially to the initial‐RT benefit after long delays.

In summary, our behavioral data suggest that action‐relevant visuo‐spatial stimulus sequences are likely to be encoded and retained as higher‐level action‐related representations (motor intentions), possibly mediated by some initial, more concrete imitative response priming. The ability (and necessity) to rehearse the given action plan during the long retention interval appears to have accelerated (kinematic) motor sequence programming during recall, thereby reducing the time until sequence execution started. In the following, we will discuss the neural data bearing on our research question.

Hypothesis I: Common effector‐independent brain activity during encoding and recall

Effector‐independent neural activity shared by both encoding and recall was found in a widespread network comprising frontal (DLPFC, IFG and premotor cortex), insular, and parietal (SII, SPL, IPS) areas as well as the basal ganglia. Overall, our results thus demonstrate substantial overlap of brain activity in many action‐relevant areas during encoding and recall, which supports the view that encoding visuo‐spatial cues for delayed, memory‐guided action recruits brain regions that subsequently implement the recall. Analogous observations were reported for oculomotor delayed‐response tasks [Curtis et al., 2004] and verbal WM [Chein and Fiez, 2001; Enschen et al., 2007]. That is, areas relevant for implementing a given action also seem to implement its memorization [see also Cowan, 1988; Hikosaka et al., 2002; Willingham, 1999]. As we observed overlap predominantly in areas related to action control, we suggest that visuo‐spatial sequence information is retained and rehearsed (also) by way of action‐related codes [see also Pau et al., 2013; Seidler et al., 2005], which is not meant to imply the absence or irrelevance of retrospective perceptual codes.

Hypothesis II: Common effector‐specific brain activity during encoding and recall

Hand‐specific activity during both encoding and recall was found in contralateral premotor, sensorimotor, and superior parietal cortex, thalamus as well as the ipsilateral cerebellum. Note that during encoding, these activations were not associated with overt hand movements. This response‐predictive pattern of lateralized brain activity during encoding strongly argues for a prospective, action‐related representation of the visuo‐spatial memoranda.

Hypothesis III: Delayed versus immediate recall

Retaining items in WM across intervals of 7 s without a decline in accuracy (cf. Fig. 2) typically requires active rehearsal during the delay period [Baddeley, 2012; Cowan, 1988]. Rehearsal of abstract motor intentions might have facilitated their conversion into motor programs at recall, which would also explain the faster initiation of sequence reproduction in recall trials with long delays. By comparing delayed with immediate recall, we examined the effect of rehearsal on reproduction‐related brain activity and found that a substantial part of the global recall network, including putamen, vPMC, SMA and SPL, showed bilaterally increased activity. Thus, long delays led to stronger activity increases in execution‐related brain regions during recall. In keeping with Jeannerod [1999, 2001], we interpret this finding as a facilitated implementation of the mentally rehearsed intention (or “simulated action”) in the sensorimotor system.

Notably, activity in SI and SII as well as the ventral posterolateral thalamus, which is the major thalamic relay for somatosensory signals, was strongly modulated by the delay in an effector‐specific way. This suggests that rehearsal mainly relied on simulating the proprioceptive and/or tactile action consequences leading to an expectancy‐induced activity increase in related brain areas [cf. Langner et al., 2011]. Moreover, rehearsing appropriate motor intentions might bias the spatial coding of the visually presented movement targets from gaze‐centered toward increasingly hand‐centered over the course of the delay period [cf. Bernier and Grafton, 2010; McGuire and Sabes, 2009]. The resulting bias toward manual action should then facilitate the concrete programming of each finger flexion during recall. Together, the effects of prolonged retention on neural activity lend further support to the view that visuo‐spatial sequences are retained and rehearsed (also) as action‐related codes that facilitate motor programming during subsequent WM‐guided reproduction.

Hypothesis IV: Encoding‐related activity associated with correct recall

We observed activation in the left anterior putamen during encoding that was selectively associated with subsequent correct (vs. incorrect) sequence reproduction. This part of the basal ganglia belongs to the so‐called skeletomotor loop among several basal ganglia–thalamocortical circuits, which is involved in motor control [Alexander et al., 1986]. The specific association of this execution‐related structure with encoding sequences that are subsequently reproduced correctly (with recall involving the same structure, among others) provides another strong piece of evidence for the use of action‐related codes at encoding. Furthermore, the specific association with correct recall also suggests that these prospective codes are functionally significant for the short‐term retention of sequences.

Intermediate Summary

None of our four predictions derived from the “execution‐oriented encoding” view was falsified. Together with the behavioral data, these findings lend strong support to the notion that memory‐guided movements are mediated by a movement plan already generated at encoding, rather than just prior to movement onset based on purely sensory target representations in memory [cf. Freeman and Ellis, 2003; Krigolson et al., 2012]. Nevertheless, the broad recruitment of visual areas during encoding is consistent with the notion that both execution‐related and sensory codes are used for sequence retention (and rehearsal), suggesting a dual‐coding approach in the human brain. Future research is required to test this hypothesis and delineate the contributions of different, potentially parallel representations to retaining to‐be‐reproduced visuo‐spatial sequences. In the following, we will briefly review the potential functional roles of brain regions involved in the encoding and WM‐guided reproduction of visuo‐spatial sequences and discuss possible underlying mechanisms.

Functional Significance of Brain Regions Involved in Translating Visuo‐spatial WM into Action

DLPFC

The DLPFC was activated by both encoding and recall without any statistically significant difference. This region has been frequently implicated in core WM processes such as information maintenance and updating [see Rottschy et al., 2012, for a recent meta‐analysis]. Both maintenance and updating are required for successful sequence encoding and reproduction: During encoding, early elements need to be retained while encoding later ones, and any new element must be added to the already stored elements as soon as it occurs; during recall, later elements need to be retained while reproducing earlier ones, and any just reproduced element must be eliminated (i.e., mentally marked as “done”) from the remaining sequence. These processes, in turn, rely on attentional selection (among perceptual inputs during encoding) and selection from memory (among elements defining potential motor acts during recall), which were shown to selectively recruit the (mid‐)DLPFC [Rowe et al., 2000]. At the same time, the single, serially presented sequence elements must be integrated over time into an ordered sequence for subsequent reproduction. In light of the “selection and temporal integration” framework underlying DLPFC function [Fuster, 2001; Quintana and Fuster, 1999], it is little surprising that activity in mid‐DLPFC was found to be specifically associated with coding the serial order of visual stimuli in WM [Amiez and Petrides, 2007]. Its essential involvement in WM for item order (as opposed to WM for items per se) is also supported by both lesion studies [for a review, see Marshuetz, 2005] and other neuroimaging work on delayed serial recall [Kellermann et al., 2012; Toepper et al., 2010]. By means of repetitive transcranial magnetic stimulation (rTMS), Hamidi et al. [2009] demonstrated a selective role of the right DLPFC in response formation and execution of WM‐driven responses beyond mere short‐term storage. It should be noted, though, that Volle et al. [2005], using fMRI, found left DLPFC to be selectively associated with preparing WM‐driven response sequences.

Taken together with our data, we propose that during sequence encoding, the DLPFC specifically subserves sequencing and chunking the stream of incoming spatial information [i.e., keeping sequence elements distinct and ordered, possibly grouping them into manageable, ordered bins; cf. Amiez and Petrides, 2007; Bor et al., 2003]. During recall, in turn, it might subserve selecting the appropriate motor intentions from WM and organizing their temporal order to guide the current action according to the instructed task set [cf. Rowe et al., 2000; Volle et al., 2005]. This view agrees with more general conceptualizations of the DLPFC as being essential for setting multiple behavioral goals (during encoding) and selecting appropriate actions (during recall) [Tanji and Hoshi, 2008], which can certainly be considered part of the functions ascribed to the “central executive” in current multi‐component models of WM [see Baddeley, 2012].

Ventral PMC and Posterior IFG

A large bilateral cluster extending from vPMC to posterior IFG (pars opercularis, area 44) showed increased activity during both encoding and recall. This pattern mirrors results of a recent meta‐analysis [Caspers et al., 2010], which revealed that this region is consistently involved in both action observation and imitation [Binkofski and Buccino, 2006]. Further, vPMC is also implicated in motor imagery [Binkofski et al., 2000; Gerardin et al., 2000], while inferior area 44 and frontal opercular cortex are involved in the learning of motor sequences [Seitz and Roland, 1992] and the acquisition of novel visuo‐motor associations [Toni et al., 2001]. These and other observations [reviewed in Binkofski and Buccino, 2006] suggest that encoding‐related vPMC/IFG activity reflects a kind of automatic “in sensu” (i.e., covert) imitation. Thus, the close coupling between the visual input (i.e., a sequence of dots presented on the fingers of a schematic hand) and its subsequent motor reproduction may have recruited brain regions involved in automatic imitation [Heyes, 2011]. Besides real human movements, such automatic imitative tendencies also occur when observing robotic actions [Press et al., 2005] or—as in our case—compatible movements of symbols [Brass et al., 2001]. We therefore assume that encoding to‐be‐reproduced visuo‐spatial sequences is mediated by automatic response priming, which entails the processing of proprioceptive information on the positions of fingers and response buttons, as corroborated by the hand‐specific activations in SI. This effector‐specific somatosensory (pre‐)processing (in the absence of overt movements) may support the conversion of the gaze‐centered spatial reference frame (in which the visual input is initially coded) into a hand‐centered reference frame. The latter is necessary for programming the proprioceptively guided recall (as fingers and buttons were not visible during the task). In fact, such visuomotor priming effects [Craighero et al., 1998] appear to be pronounced in memory‐guided (as opposed to visually guided) actions [Cant et al., 2005], possibly to facilitate the formation of a corresponding motor intention for short‐term storage [cf. McBride et al., 2012].

IFG activity during recall corresponds to this region's role in higher‐level movement control [Caspers et al., 2010; Gallese et al., 1996; Gerardin et al., 2000], which has prompted the notion that area 44 may contain bilateral hand and/or finger representations necessary for gesturing and imitation [Binkofski et al., 2000; Iacoboni et al., 2005]. More specifically, IFG may contribute to controlling the sequential aspects of the recall: Koechlin and Jubault [2006] proposed that area 44 subserves sequential behavior by “selecting/inhibiting simple action chunks through top‐down interactions that initiate and terminate successive selections of simple chunk components occurring in the premotor regions (i.e., single motor acts or sensorimotor associations)” (p 964). Besides IFG (and DLPFC, cf. above), these sequencing processes would presumably involve the cerebellum, which is likewise implicated in sequencing incoming sensory patterns and outgoing responses [Braitenberg et al., 1997; Mauk et al., 2000; Molinari et al., 2008]. Consistent with this view, we found bilateral cerebellar involvement in both sequence encoding and reproduction, possibly reflecting the preparation of sequential movements already at encoding [cf. Cui et al., 2000].

Anterior Insula

Further activity overlap between encoding and recall was observed in the anterior insula bilaterally. A recent meta‐analysis [Kurth et al., 2010] demonstrated that this region is associated with a broad range of cognitive tasks, including WM. Dosenbach et al. [2006, 2007] suggested that the anterior insula subserves the implementation and maintenance of task sets. Moreover, a recent study demonstrated significant coupling between right anterior insula and anterior MCC across different attention‐demanding tasks, suggesting that the right anterior insula may be critical for modulating cognitive control systems under challenging conditions [Eckert et al., 2009]. From this perspective, consistent activity in the anterior insula as observed in the present study may be related to maintaining an appropriate level of alertness [Langner et al., 2012; Sterzer and Kleinschmidt, 2010], which is considered a general prerequisite for efficiently performing cognitive tasks [cf. Langner et al., 2010b].

Basal Ganglia

A large bilateral cluster of common activity across encoding and recall comprised parts of the basal ganglia (middle putamen and pallidum, substantia nigra) and extended to the anterolateral thalamus. This network corresponds to the subcortical parts of what has been termed the skeletomotor loop among separate basal ganglia–thalamocortical circuits [Alexander et al., 1986]. Activity in this network is assumed to modulate cortical movement‐related information processing by disinhibiting thalamic motor nuclei that, in turn, send excitatory efferents to cortical motor zones [for review, see Chakravarthy et al., 2010; Haber and Calzavara, 2009]. The basal ganglia thus act as a “gate” that, via thalamic nuclei, selectively facilitates planning, preparation and execution of movements. Obviously, this function should be highly relevant for the memory‐guided production of movement sequences during recall. Parts of this loop, however, were specifically implicated in encoding. For instance, there is neuroimaging [McNab and Klingberg, 2008] and lesion‐based [Baier et al., 2010] evidence that the basal ganglia play a pivotal role in filtering WM input. Still earlier, Beiser and Houk [1998] proposed a computational model in which the above‐mentioned basal circuit enabled encoding the serial order of events into spatial activity patterns in the PFC. Based on this model and other evidence, Hazy et al. [2006, 2007] argued that the updating of WM content, for instance during serial stimulus presentation, is mediated by the gating function of the basal ganglia. Our finding of significant activation in this basal‐ganglia circuit across both encoding and recall is consistent with this view. Even stronger support for this “selective gating” function is provided by the selectively increased putaminal activity during encoding in trials that were later recalled correctly. According to the above reasoning, this should reflect stronger modulatory signals to PFC, achieving optimal degrees of selectivity in encoding (for additional discussion of the differential involvement of the basal ganglia in encoding vs. recall, see the Supporting Information).

Dorsal PMC

We found differential recruitment of different portions of the dPMC during encoding (ventro‐rostral dPMC) and recall (dorso‐caudal dPMC) (cf. Supporting Information Fig. S1 and Table S1). The stronger ventro‐rostral dPMC activity during encoding is consistent with the idea that this area implements arbitrary stimulus–response mappings [Cieslik et al., 2010; Hoshi and Tanji, 2006; Petrides, 1985, 1997; Taubert et al., 2010]. Since in the present study the sequence was fully predetermined during encoding, such transformation into an action‐related code, which was demonstrated to be a key function of the dPMC in monkeys [Hoshi and Tanji, 2000; Nakayama et al., 2008], could start immediately upon stimulus presentation. The location of the encoding‐selective parts of the dPMC overlaps with the presumed location of the human FEFs [Paus, 1996; zu Eulenburg et al., 2012]. This appears plausible, as the FEFs are crucially involved in facilitating visual target detection in both humans [Grosbras and Paus, 2003] and monkeys [Thompson and Schall, 1999], presumably via mediating attentional top‐down signals to bias perception [Corbetta and Shulman, 2002; Langner et al., 2011]. It has been shown that attention can be spatially distributed in parallel among several noncontiguous saccade goals, providing a basic input for abstract action plans [Baldauf and Deubel, 2009; Baldauf et al., 2006; Godijn and Theeuwes, 2003; Grefkes et al., 2010]. It must be acknowledged, though, that we cannot rule out small (reflexive) saccades driven by the visual stimulation, as eye movements were not recorded in our study (for additional discussion of the specific involvement of the dPMC in recall, see the Supporting Information).

Frontomedial Cortex

The encoding–recall distinction within the dPMC is mirrored by a similar differentiation between rostral and caudal aspects in the medial frontal cortex comprising the (pre‐)SMA and MCC. These frontomedial areas are known to be involved in both movement preparation and initiation [Cunnington et al., 2003; Nachev et al., 2008], exerting context‐specific influences on the motor cortex [Grefkes et al., 2008]. Similar to the above‐mentioned distinction within the dPMC, more rostral aspects of this region have been associated with more abstract (cognitive) aspects of motor control. In particular, the pre‐SMA is assumed to be engaged in complex, cognitively controlled tasks, whereas the more posterior “SMA proper” has direct projections to the primary motor cortex and is more strongly engaged in action execution, including performance of more simple, automatic movements [Eickhoff et al., 2011; Jakobs et al., 2009; Picard and Strick, 1996]. A similar differentiation holds true for the MCC, where anterodorsal parts are associated with enacting intentions, while more ventral and posterior parts (cingulate motor zone) have direct influence on lower‐level aspects of movement control [Hoffstaedter et al., 2013; Paus, 2001; Picard and Strick, 1996].

In the context of movement sequences it has been shown that frontomedial areas (rather than the lateral PMC) are essential for temporally organizing memorized behavioral sequences [for review, see Tanji, 2001], resolving competition between motor plans [Cieslik et al., 2011; Pardo et al., 1990], and triggering the initiation of (self‐paced) movements [Deiber et al., 1999; Hoffstaedter et al., 2013]. We therefore argue that the more anterior cluster encodes the temporal structure of the (complex) movement intention, while activity in the recall‐specific, more caudal cluster is related to structuring and implementing the sequential motor program according to this temporal information. This view agrees with our observation of increased activity in the caudal aspect of the frontomedial cluster after long (vs. short) retention intervals, which may reflect the facilitated programming of the movement sequence during recall due to rehearsal (for additional discussion of the specific involvement of frontomedial areas in recall, see the Supporting Information).

Posterior Parietal Cortex

An important aspect of translating the visuo‐spatial input into finger movements toward spatially corresponding but unseen targets is the transformation of retinotopically coded spatial information into hand‐centered coordinates [Lemay and Stelmach, 2005]. Representation of space and switching between various (e.g., gaze‐, head‐ or body‐centered) coordinate systems is considered crucially dependent on the posterior parietal cortex [Andersen et al., 1997; Buneo et al., 2002; Grefkes et al., 2004] and should form the basis for this region's involvement in planning visually guided reach and pointing movements [Andersen and Cui, 2009; Baldauf et al., 2008; Filimon, 2010]. The SPL and, in particular, the IPS are key regions for processing and storing visuo‐spatial information for action implementation [Grefkes and Fink, 2005; Rushworth et al., 2003] and the goal‐directed spatial allocation of attention [Corbetta and Shulman, 2002]. The common denominator may be the construction of a spatial salience map to guide perception and action in space, variably (and in parallel) coded in the appropriate body‐part‐specific reference frame(s) [Bisley and Goldberg, 2010; Ptak, 2011]. This view is consistent with our observation of activity in this region during both encoding and recall. The encoding‐related parietal activity along the IPS might reflect an automatic spatial coordinate transformation into a nonretinotopic, body‐ and/or hand‐centered reference frame, in which high priority values are assigned to the goal coordinates of subsequent proprioceptively guided movements. This view is supported by the lateralized, hand‐specific activations in SPL during encoding (see Fig. 4). During recall, this priority map could then be employed by motor and premotor areas to program the spatially corresponding finger movements according to the motor intention, which has also been argued to “reside” in posterior parietal areas [Andersen and Buneo, 2002; Andersen and Cui, 2009; Desmurget et al., 2009]. This interpretation agrees with the observation of a posterior‐to‐anterior gradient in this region from encoding to recall [cf. McGuire and Sabes, 2011].

In keeping with previous research [Chang and Snyder, 2010; Colby and Duhamel, 1996; Graziano, 2001], we take this pattern to reflect a gradient of various partially overlapping representations of space in parietal cortex. Its posterior aspects might hold a more vision‐oriented spatial representation with an eye‐centered reference frame. In contrast, motor‐oriented recall selectively activated the anterior aspect of the SPL, bordering on somatosensory areas, which may suggest a more body‐oriented, possibly hand‐centered representation. Finally, the parietal region commonly activated by encoding and recall lay in‐between, along the IPS, implying a more abstract (i.e., supramodal or hybrid) spatial representation [Andersen et al., 1997; Mullette‐Gillman et al., 2009].

In summary, the observed parietal activity concurs with a key role of the parietal cortex (particularly IPS/SPL) in the visuo‐motor transfer of spatial information [Filimon, 2010; Grefkes et al., 2004; Macaluso et al., 2003], which, within current multi‐component models of WM, may be interpreted as a neural correlate of the “visuo‐spatial sketchpad” [Baddeley, 2012]. We conjecture that this “sketchpad” codes space in various coordinate systems, depending on current needs (e.g., coding space for perception, action planning, or concrete motor programming/execution).

Theoretical Implications and Future Directions

In the final section, we will discuss implications our findings have for theoretical perspectives on visuo‐spatial WM. The evidence obtained for effector‐specific execution‐related representations at encoding suggests that such to‐be‐reproduced visuo‐spatial sequences are immediately coded in the form of configurational motor intentions (i.e., patterns of intended movements). This conclusion agrees with earlier behavioral findings of Cavallini et al. [2003], who observed that rehearsal in a typical CBT task was disturbed by nonspatial sequential finger movements. This interference between simple motor sequences and short‐term memory for to‐be‐reproduced visuo‐spatial sequences indicates common, execution‐related processing in both tasks. On a similar note, Smyth and Pendleton [1990] observed that encoding a sequence of movements to spatial targets for subsequent recall (as is required in CBT‐like tasks) negatively affects the serial recall of configured movements. Taken together with our data, this provides evidence for a third, execution‐related WM subsystem, besides the visuo‐spatial sketchpad and the phonological loop proposed by Baddeley and Hitch [1974]. This view agrees with similar suggestions of a proactive, movement‐related WM subsystem by Smyth and Pendleton [1989, 1990] as well as Logie and collaborators [Logie, 1995; Logie and Vecchi, 2006].

As mentioned before, however, we cannot conclude that encoding in the present task relied exclusively on prospective action‐oriented representations, as opposed to retrospective visual memories. First, there was sufficient neural evidence (e.g., the large parieto‐occipital cluster selectively activated during encoding) for a parallel short‐term storing of visual representations. Second, there are limitations to the validity of reverse inference (i.e., conclusions about cognitive processes drawn from brain activity), which depends critically on the selectivity of the given brain–behavior relationship [Poldrack, 2006]. Since this selectivity, however, is relatively high for brain regions involved in motor cognition and motor control, we can rather safely use the direct, within‐subject test of activation overlap between sequence encoding and motor reproduction to infer the occurrence of motor‐related processing during encoding.

Furthermore, it should be noted that the putative motor‐related WM subsystem is not necessarily used for encoding visuo‐spatial sequence information in all CBT‐like tasks. In particular, such motor representations should only be involved if all information necessary for subsequent motor reproduction is given at encoding, as was the case in our paradigm. Nevertheless, the possibility for dual coding (i.e., the parallel use of sensory and execution‐related codes) has implications for interpreting performance in typical spatial‐span tasks like the CBT, as existent deficits in visual WM might be compensated by relying on motor‐related codes, at least if the above‐mentioned necessary preconditions apply. Therefore, if CBT‐like tasks are employed to detect or diagnose specific impairments in visuo‐spatial short‐term memory, it might be worth considering to indicate the hand to be used for reproduction only after sequence presentation. This way, the use of execution‐related representations for spatial sequence retention would be suppressed or, at least, reduced.

Future studies are desirable that experimentally manipulate factors relevant to the question of which kind of representations are used under which conditions. For example, the “affordance” of the stimulus sequence could be varied by using stimuli ranging from highly abstract symbols to short films of realistic finger movements. Furthermore, secondary tasks could be used to selectively suppress nonspatial motor versus nonmotor spatial processing during encoding, retention, or both [cf. Lemay and Stelmach, 2005]. Other issues for future work concern differences between our task version and the original CBT, in which both the hand used for reproduction and the movement targets are seen during recall and in which observed and to‐be‐reproduced targets are equivalent (without requiring spatial remapping from the presentation screen to the response button box).

CONCLUSIONS

We delineated common neural correlates of encoding and reproducing visuo‐spatial stimulus sequences under conditions of immediate and delayed serial recall. Encoding visuo‐spatial sequences for subsequent manual reproduction activated prefrontal, premotor and parietal as well as subcortical structures known to be involved in action control. We also found effector‐specific brain activity already at encoding (i.e., in the absence of overt motor responses), and putaminal activity during encoding selectively predicted correct recall. Together with our behavioral data, this pattern of neural results suggests that the memoranda in proprioceptively guided serial‐recall tasks consist of prospective motor intentions, possibly encoded in parallel to retrospective perceptual (visual) memories. In delayed (as opposed to immediate) recall, the transformation of motor intentions into concrete (kinematic) motor plans appears to be facilitated by the preceding rehearsal of the motor plan. This influence of higher‐level motor cognition during encoding and retention/rehearsal on subsequent sequence reproduction suggests that hierarchical control models of sequential motor behavior [Grafton and Hamilton, 2007; Lashley, 1951; Rosenbaum et al., 1984, 2007] can be generalized to WM‐guided movements. Overall, our results support notions of an execution‐related WM subsystem, consistent with multi‐component models of WM that assume a flexible, goal‐dependent coding of sensory information to be mentally retained across short periods of time.

Supporting information

Supporting Information