The Representation of Order Information in Auditory-Verbal Short-Term Memory

Abstract

Here we investigate how order information is represented in auditory-verbal short-term memory (STM). We used fMRI and a serial recall task to dissociate neural activity patterns representing the phonological properties of the items stored in STM from the patterns representing their order. For this purpose, we analyzed fMRI activity patterns elicited by different item sets and different orderings of those items. These fMRI activity patterns were compared with the predictions made by positional and chaining models of serial order. The positional models encode associations between items and their positions in a sequence, whereas the chaining models encode associations between successive items and retain no position information. We show that a set of brain areas in the postero-dorsal stream of auditory processing store associations between items and order as predicted by a positional model. The chaining model of order representation generates a different pattern similarity prediction, which was shown to be inconsistent with the fMRI data. Our results thus favor a neural model of order representation that stores item codes, position codes, and the mapping between them. This study provides the first fMRI evidence for a specific model of order representation in the human brain.

Introduction

How do we store and retrieve a sequence of items? Almost any situation in daily life, whether making a cup of tea or remembering the words in a sentence, requires us to remember not only the individual elements but also the order in which they occurred. All computational models of serial order acknowledge this distinction between the representations of items in memory and the representation of the order in which they occur (Henson and Burgess, 1997). Broadly speaking, models can be divided into two classes dependent on how the order information is represented:

Chaining models assume that a sequence is stored by the formation of associations between representations of successive items (Fig. 1A). The order of items is retrieved by stepping along the chain of associations, such that each item becomes a cue for retrieving its successor (Lashley, 1951). Simple chaining models only assume pairwise associations between adjacent items, whereas more sophisticated chaining models assume remote associations as well as adjacent ones (Slamecka, 1985; Lewandowsky and Murdock, 1989).

Figure 1.

Positional and chaining models of order representation. A, An example how the sequences CAT PIG DOG and DOG CAT PIG are differentially encoded by the positional and chaining models. Numbers refer to the positions of the items (CAT, PIG, DOG) in the sequence. Items are color-coded to highlight the permutations. B, Difference in order encoding yields different pairwise similarity matrices for all possible permutations of the items CAT, PIG, and DOG. We assume that the pairwise similarity between two sequences according to a positional model is given by the Hamming distance between the sequences (see Materials and Methods for details). For the chaining model, pairwise similarity between two sequences can be described by the interitem distance between the sequences. The permutations of the items are number-coded on the axes; e.g., 123 refers to CAT PIG DOG, etc. C, Color-coded similarity matrices.

Positional models assume that order is stored by associating each item with its position in a sequence, and that the order is retrieved by reinstating each positional code which then cues the associated item (Fig. 1A; Lee and Estes, 1981; Burgess and Hitch, 2006; Brown et al., 2007). Hence, positional models need to store item codes, position codes, and the mapping between them. Importantly, these two classes of models make contrasting predictions about the similarity of the representations of different orderings of the items in a sequence.

Here we look to dissociate between positional and chaining models based on fMRI data. For this purpose, we used an immediate serial recall task, where participants vocally recalled sequences of nonwords immediately after presentation. We use multivoxel pattern analysis (MVPA) to dissociate fMRI activity patterns coding the phonological representations of the nonwords (items) from the patterns representing their order (Fig. 2). Importantly, because they encode different associations, positional and chaining models make different predictions about the similarity of orderings: for example, two sequences ABC and BCA are relatively more dissimilarly represented in the positional model than in the chaining one (Fig. 1A). This enables us to test whether the similarity structure of the activation patterns elicited by different orderings of items lend support to either positional or chaining models of order.

Figure 2.

Experimental task. A, Memory for the items in a sequence versus memory for the order of the items in a sequence. Items are nonoverlapping sets bisyllabic nonwords. B, Single trial. C, Structure of trials.

Previous research has suggested that a network of brain regions participate in order processing including the frontal lobe (Barone and Joseph, 1989; Histed and Miller, 2006; Berdyyeva and Olson, 2010), inferior parietal lobe (Moser et al., 2009), hippocampus (Kesner et al., 2010; Devito and Eichenbaum, 2011), and basal ganglia (Miyachi et al., 1997; Yin, 2010). However, previous imaging studies have only revealed the involvement of these brain areas in tasks requiring retention of order. In this study, we seek to go beyond anatomical localization to investigate the nature of order representation in the brain.

Materials and Methods

Participants

In total, 12 right-handed volunteers (7 female, 20–33 years old) gave informed, written consent for participation in the study after its nature had been explained to them. Subjects reported no history of psychiatric or neurological disorders and no current use of any psychoactive medications. The study was approved by the Cambridge Local Research Ethics Committee (Cambridge, UK).

Task

We used an immediate serial recall (ISR) task, where participants had to vocally recall sequences of three bisyllabic nonwords immediately in the order they were presented (Fig. 2B). We used sequences of three items to ensure that the entire sequence would fall within the participants' short-term memory (STM) capacity and could be accurately retained in STM. If we had used longer sequences where participants might make errors (e.g., 8 items; for review, see Cowan, 2005) then the representation of any given sequence would necessarily vary from trial to trial, and no consistent pattern of neural activity could be detected.

On each trial, participants were presented with a visual fixation cross to indicate the start of the auditory presentation of the sequence. Three nonwords were then played through the headphones in a male voice. The duration of each nonword was 800 ms and the whole three-item sequence 2.6 s (3 times 800 ms + 2 pauses of 100 ms). Presentation of a sequence was followed by a cue “?” indicating that participants were to verbally recall the sequence exactly as they had just heard it; or a cue “- - -” indicating not to respond and to wait for 2–10 s for the next list (Fig. 2). Participants were only instructed to recall on 2/3 of the trials (128 trials) and the length of the recall period was varied between 6 and 8 s. These measures (jittering of the length of the rest phase, varying the length of the recall period, and omitting the recall phase for one third of the trials) were taken to ensure a sufficient degree of decorrelation between the estimates of the BOLD signal for the encoding and recall phases of the task. Each participant was presented with 192 trials, with 96 trials per scanning run, in addition to an initial practice session outside the scanner. Participants were not informed that there were different types of trials.

For each trial, recall performance was measured as the Hamming distance between the presented sequence and the subject's recall. For two sequences of equal length, the Hamming distance is the number of positions at which the corresponding items are different. Put another way, Hamming distance measures how many items were not retained at their original positions.

Stimuli

We used two distinct sets of six bisyllabic nonwords. In one set, all consonants were stops (e.g., PAKEB); in the other they were liquids, glides, and nasals (e.g., NAEYUM). Within each set, the nonwords were arbitrarily divided into two subsets of three nonoverlapping items. Those three items were then presented in six possible different orders (Table 1; Fig. 2).

Table 1.

Stimuli

	Set 1 (stop consonants)	Set 2 (liquids, glides and nasals)
Subset A	BOEGID, GITOG, PAKEB	NAEYUM, REMIL, YOMING
Subset B	DOUGHBID, KETUB, PIDIK	LANING, NOWLEM, YANIM

Each ordering was presented 8 times over the course of the experiment. The nonwords were spoken by a male native speaker of British English and recorded at 44.1 kHz sampling rate and 16 bits per sample.

fMRI data acquisition and preprocessing

Participants were scanned at the Medical Research Council Cognition and Brain Sciences Unit (Cambridge, UK) on a 3T Siemens TIM Trio MRI scanner using a 32-channel head coil. Functional images were collected using 32 slices covering the whole brain (slice thickness 3 mm, 25% slice gap, in-plane resolution 3 × 3 mm) with a “silent” EPI sequence (Schmitter et al., 2008), with TR = 2.63 s, TE = 44 ms, and flip angle of 78°. In addition, high-resolution MPRAGE structural images were acquired at 1 mm isotropic resolution (see http://imaging.mrc-cbu.cam.ac.uk/imaging/ImagingSequences for detailed information). Each participant performed two scanning runs, 620 scans were acquired per run, including 16 dummy scans. Stimulus presentation was controlled by Presentation software version 4.19. Visual cues for sequence presentation and recall were rear projected onto a translucent screen outside the bore of the magnet and viewed via a mirror system attached to the head coil. Auditory stimuli were delivered with magnet-safe headphones installed inside ear defenders (NordicNeuroLab, noise attenuation of 30 dB). All trials were included in the analysis because the number of trials on which participants made errors was very small (see Results).

All fMRI data were preprocessed using SPM5 software (Wellcome Trust Centre for Neuroimaging, London) and analyzed using custom in-house software. Before analysis, all images were corrected for slice timing, with the middle slice in each scan used as a reference. Images were realigned with respect to the first image using trilinear interpolation, creating a mean realigned image. The mean realigned image was then coregistered with the structural image and the structural image was normalized to the MNI average brain using the combined segmentation/normalization procedure in SPM5.

Multivoxel pattern analysis

As noted above, we sought to dissociate fMRI activity patterns representing the phonological information of the items from the patterns representing their order. This cannot be done directly, because permutations of order only exist within a set of items, so we used two different analysis methods for item and order effects. To analyze order representations we used representational similarity analysis (RSA; Kriegeskorte et al., 2008) to compare patterns of brain activation to the similarity predictions from positional and chaining models. To determine which areas were sensitive to item information we used a multivoxel pattern classification approach to identify brain regions containing patterns that could differentiate between the two different item sets. The reason for using both pattern classification and RSA is that the item and order analyses address different questions. In examining order information, we compare two models that make different predictions about representational similarity. Our concern is not simply whether we can differentiate between different orders but in the more interesting question of what form those representations take. In the case of item representations, we have only two sets of items that are not sufficient to produce a rich pattern of similarity. In the case of a single dimension of similarity, a classifier, such as a support vector machine (SVM), is the best way to determine whether there are regions that distinguish between the two sets of items. However, a common set of fMRI data analysis steps were undertaken before the analyses were split.

We moved a spherical searchlight with a 3 voxel radius throughout the gray-matter masked and unsmoothed volumes to select, at each location, a local contiguous set of 113 voxels (3 mm isotropic). In each sphere, we estimated the BOLD response for the encoding and recall events of every trial (192 encoding and 128 recall events) for every voxel in the sphere. The event regressors were convolved with the canonical hemodynamic response (as defined by SPM8 analysis package) and passed through a high-pass filter (128 s) to remove low-frequency noise. In addition to six motion parameters (corresponding to translations and rotations of the image due to movement in the scanner) additional scan-specific regressors were also added to account for large head movements. Additional parameters were modeled to account for extreme interscan movements, which exceeded a translation threshold of 0.5 mm, rotation threshold of 1.33°, and between-images difference threshold of 0.035 calculated by dividing the summed squared difference of consecutive images by the squared global mean. These separate movement spike regressors remove variance due to head movement caused by participants speaking in the scanner during the response phase of the trials.

As a result, we obtained 192 encoding parameter estimates (β-values) for every voxel representing the 192 sequence presentation events and 128 recall β-values representing the 128 sequence recall events over the course of the experiment. As noted earlier, a sufficient degree of decorrelation between all of the regressors was ensured by: (1) jittering the length of the rest phase (between 2 and 10 s), (2) varying the length of the recall period (6–8 s), and (3) omitting the recall phase for one third of the trials. As a result, the voxels in the searchlight comprised a vector of activations resulting in one vector per trial.

Representation of item information

To identify voxels which encode item information we used a multivariate classification procedure. We labeled the searchlight voxel vectors according to the item set from which the sequence was derived (see Stimuli for details). Next, we split the vectors into two datasets: a training set used to train a support vector machine (with a linear kernel and a regularization hyper-parameter C = 40) to assign correct labels to the activation patterns, and a test set used to independently test the classification performance. The SVM classifier was trained to discriminate between two item sets with the training data and subsequently tested on the independent test data. The classification was performed with the LIBSVM (Chang and Lin, 2011) implementation. We used a standard k-fold cross-validation testing regime wherein k equaled the number of experimental trials. As a result, we performed a 192-fold cross-validation for every searchlight (191 training and 1 testing value). For every participant, the searchlight analysis resulted in an item set classification accuracy brain map. We assigned a score of zero to any sphere in which <33 voxels were inside the individual gray matter volume. These individual images were subsequently normalized to the MNI anatomical template and entered into random-effects analyses (one-sample t tests). By using a large number of train-and-test iterations (leave-1-out with 192 trials) we obtained a precise estimate of true classification accuracies. This approach provided a test–retest reliability within ±5% based on multiple simulations of 192 iterations conducted in four subjects. Next, we statistically assessed decoding significance with a two-tailed t test versus 50% chance decoding. To correct for multiple comparisons, we used a false discovery rate (FDR; Genovese et al., 2002) threshold of p < 0.05.

As a supplement to the t tests we also assessed statistical significance with nonparametric randomization tests (Gallivan et al., 2011; Stelzer et al., 2013). We permuted the correspondence between the test labels and data 100 different times to compute 100 mean classification accuracies for the testing labels. To this permuted distribution of accuracies, we added the mean accuracy obtained with the correct labeling. We then derived the distribution of the group-level mean accuracies by randomly sampling 1000 mean accuracies (with replacement) from each subject's permuted distribution. Next, we found the true group-level mean accuracy's empirical probability based on its place in a rank ordering of this distribution. The peak percentiles of significance (p < 0.001) are limited by the number of samples producing the randomized probability distribution at the group level. This nonparametric randomization test produced more searchlight volumes with significant p values than those found with the parametric t tests (Gallivan et al., 2011; Stelzer et al., 2013). This indicates that the t test group analysis provides a conservative estimate of the statistical significance of the classification accuracies.

Representation of order information

For every searchlight, we performed a representational similarity analysis (Kriegeskorte et al., 2008) between voxel activity patterns and two model predictions. Both models were expressed by a specific pairwise similarity function, which generated a different pairwise similarity matrix for the stimuli. Our stimuli were all possible permutations (orderings) of three items resulting in a 6 × 6 pairwise similarity matrix, which is diagonally identical (Fig. 3).

Figure 3.

Multivariate analysis of order information. First, we generate pairwise similarity matrices of orderings based on the positional and chaining models using the Hamming and interitem distances respectively. Next, we extract each participant's neural response to each sequence to estimate pairwise response pattern similarities. Pattern similarities are then compared with a predicted similarity structure given by the Hamming and the interitem distances between sequences by computing a correlation coefficient between the two matrices. The probability of the correlation is obtained by using a permutation test where the columns of the matrices was shuffled 10,000 times.

Positional model: item-position encoding.

We assume that the Hamming distance captures the pairwise similarity between item-position mappings in a positional model. For example, two sequences ABC and CBA share one item-position mapping: B at the second position. Hence, the Hamming distance, which measures how many items were not retained at their original positions, describes the similarity between item-position mappings for two sequences of equal length. Note, that although positional models also store positional codes independently of items and item-position mappings, we are not able to use our fMRI data for such correlations because all three items were presented during a single scan.

Chaining.

Chaining models represent sequences as successive interitem associations: for example, sequences like ABC and CBA can be represented as (A→B, B→C) and (C→B, B→A). It follows from this example that although ABC and CBA partially overlap (with B being the second item), there are no common interitem associations. In fact, between-sequence similarity calculated on the basis of which interitem associations are retained is equal to the inverse of the Hamming distance when three-item sequences are used. This is because two different sequences of three items can only have common item pairs when there is no overlap between item positions in the sequence (e.g., ABC vs BCA). Pairwise distances between permutations of three items can only take three possible values: complete overlap, no overlap, and partial overlap. It thus follows that whenever there is partial overlap between two permutations in Hamming distance there cannot be any overlap in the chaining distance and vice versa. As a result, pairwise Hamming and chaining distance matrices of all possible permutations of three items are inversely correlated with each other (Pearson's ρ = −1).

In this analysis we predict the similarity structure of the stimuli using the Hamming and interitem distances between sequences. The resulting two pairwise representational similarity matrices are then correlated with the response pattern similarity matrices from every searchlight to yield a Pearson's correlation coefficient per searchlight. For every participant, the searchlight analysis resulted in two brain maps of correlations between model predictions and neural response data. We assigned a score of zero to any sphere in which <33 voxels were inside the individual gray matter volume. These individual images were subsequently normalized to the MNI anatomical template and entered into permutation-based random-effects analyses using the SnPM toolbox for SPM (Nichols and Holmes, 2002; 10,000 permutations, 6 mm FWHM variance smoothing). The use of nonparametric tests avoids distributional assumptions regarding the nature of the descriptive maps, given that correlation scores are not normally distributed. Note that a positive and significant correlation with one of the models on a group level is enough to test our hypothesis, given the inverse correlation between our two predictors.

Similarity of order representations across brain areas

To assess to what extent two brain regions in the same subject represent the same information, we can compare the two regions' response pattern similarity matrices. We chose 10 regions of interest (ROIs) based on the peak voxels of the brain areas that showed significant correlation with the model predictions (see Results). Next, we correlated subjects' individual fMRI response-pattern similarity matrices between all ROIs during both encoding and recall phases. The group-level significance of the correlations was again assessed with permutation-based random-effects analyses as described above.

Three-way classification of sequences

We also sought to test whether the order effects were driven by a specific position in a sequence: for example, if a brain region only encoded the first item in the sequence and nothing else, it would still carry some information about the sequence and hence correlate positively with a model prediction. For this purpose, we ran a three-way classification analysis, where the trials were labeled according to the position of the items in three following ways: (1) first-item encoding: e.g., ABC, ACB versus BAC, BCA, versus CAB, CBA; (2) middle-item encoding: e.g., BAC, CAB versus ABC, CBA versus ACB, BCA; and (3) last-item encoding: e.g., BCA, CBA versus ACB, CAB versus ABC, BAC.

This resulted in three separate three-way classification analyses, which otherwise followed the procedure described above in Representation of item information.

Results

Behavioral results

Recall performance in the scanner was measured with an ISR task, where participants had to vocally recall sequences of three bisyllabic nonwords immediately in the order they were presented (see Materials and Methods, Task). Because three-item sequences fell comfortably within participants' memory span, we observed perfect recall on 97% of the trials. ANOVA over participants' recall scores for different stimulus types showed no significant effects of item set (stop consonants vs liquids, glides, and nasals: F_(1,44) = 1.01, MSE = 0.0003, p = 0.32; Table 1) or subset (F_(1,44) = 0.04, MSE = 0.0003, p = 0.84). Note that perfect recall was required to ensure that differences in neural activity patterns would not reflect recall performance.

Item information

We began by looking for regions that represented item-based phonological (or acoustic/phonetic) information of the sequences during the encoding phase. Significantly above chance item set classification was found bilaterally in superior temporal cortex anterior to the primary auditory cortex (STG), in bilateral ventral motor and premotor cortices (vMC, pMC), and in the right middle frontal gyrus and sulcus (Fig. 4A; Table 2). In the recall phase of the trials, we found that clusters of voxels in the superior temporal lobe, in the bilateral motor and premotor cortices, in the inferior parietal lobe (IPL), and in the dorsolateral prefrontal cortex encoded the phonological information of the sequences.

Figure 4.

Item information. Regions where the set of presented items was decoded significantly above chance across participants. Whole-brain FDR threshold of p < 0.05. A, Encoding phase of the task: red–yellow; recall phase of the task: navy–magenta. B, Brain areas where the information during the encoding phase predicted the activity patterns during the recall phase.

Table 2.

Item information: peak searchlight coordinates where the set of presented items was decoded significantly above chance across participants

	Region	Mean accuracy	t statistic	X	Y	Z	p	H
Encoding	STG	0.92	19.9567	−52	−10	4	<0.001	L
		0.87	17.6094	61	−3	4	<0.001	R
	TP	0.90	18.4244	−44	13	−31	<0.001	L
	aSTS	0.62	13.4918	−54	0	−21	<0.001	L
	SMG	0.68	10.3381	−58	−30	27	<0.001	L
	pSTG	0.69	10.053	53	−45	19	<0.001	R
	vMC	0.67	7.7004	64	−4	22	<0.001	R
Recall
	SPT	0.73	11.6549	−51	−28	9	<0.001	L
		0.86	18.4244	54	−30	15	<0.001	R
	pMTS	0.79	14.6601	52	−39	9	<0.001	R
	STG	0.74	12.4052	−56	−7	2	<0.001	L
		0.64	7.2791	61	−4	−5	<0.001	R
	MC	0.83	16.7471	−55	−19	50	<0.001	L
	vMC	0.69	9.4952	61	−5	19	<0.001	R

To further elucidate the nature of phonological representations we performed a post hoc analysis in the brain areas which showed significant phonological effects. Specifically, we tested whether the voxel patterns representing phonological information during the encoding phase predicted the activity patterns during the recall phase. This continuity of representations would indicate that decoded patterns relate to short-term memory, not only to perception. For this purpose, we trained the classification algorithm with data from the encoding phase and tested it on the data from the recall phase. This analysis was performed on all searchlights where item-decoding accuracy was significantly above chance (a two-tailed t test vs 50% chance decoding) within each subject's brain. We found that in the left and right anterior superior temporal gyrus, encoding patterns predicted recall patterns significantly above chance (Fig. 4B).

Order information

First, we correlated the model output with the activation patterns from the encoding phase of the trials. The positional model based on Hamming distance predicted similarities between activation patterns significantly above chance in the following brain areas: the superior planum temporale (SPT), IPL, supramarginal gyrus (SMG), ventral premotor cortex (vMC), and inferior frontal gyrus (IFG) (Fig. 5A; Table 3). No searchlight volumes showed significant correlation values for the chaining model. During the recall phase significant correlations followed the patterns observed in the encoding phase with the exception of more significant clusters in the superior temporal lobe posterior to the primary auditory cortex (SPT, Fig. 5A) and the lack of IFG clusters. Again, no searchlight volumes showed significant correlation values for the chaining model. Complementary ROI analyses revealed that both in the encoding and recall phases, order information was correlated between different brain regions significantly above chance (p < 0.001), including vMC, SPT, SMG, and IFG (Fig. 5B). This indicates that the representation of order information was similar and shared between those brain areas during both encoding and recall phases of the task.

Figure 5.

Order information. Regions where the similarity of activity patterns were predicted by the Hamming distance significantly above chance across participants. A, Encoding phase of the task: red–yellow; recall phase of the task: navy–magenta. B, Correlation of order information between different brain areas during the encoding and recall phases: (1) L-vMC, (2) R-IFG, (3) L-SPT, (4) L-IFG, (5) L-IPL, (6) L-SMG, (7) L-IFG, (8) R-SMG, (9) R-IPL, and (10) R-vMC. Only correlations for which p < 0.001 are shown. C, The similarity structure of fMRI activity patterns elicited by different orderings in three ROIs that encoded positional item-order associations. The similarity structure was visualized by performing nonmetric multidimensional scaling (in two dimensions) on the dissimilarity matrix of the multivoxel patterns evoked by all orderings using MATLAB's mdscale function. D, Similarity structure for three ROIs which encoded only item information and where not significantly correlated with the positional model prediction.

Table 3.

Order information: peak searchlight coordinates where the similarity of activity patterns were predicted by the Hamming distance significantly above chance across participants

	Region	Mean rho	X	Y	Z	p	H
Encoding	vMC	0.20	−62	−3	8	<0.001	L
	IFG	0.28	59	18	20	<0.001	R
		0.34	−58	5	9	<0.001	L
		0.29	−50	6	19	0.002	L
	SPT	0.24	−54	−39	19	<0.001	L
	IPL	0.25	−52	−21	40	<0.001	L
		0.26	38	−33	45	0.002	R
	SMG	0.37	−59	−27	17	0.002	L
		0.25	59	−17	15	0.002	R
Recall	SPT	0.29	57	−28	12	<0.001	R
	IPL	0.32	−52	−21	40	<0.001	L
		0.32	38	−33	45	<0.001	R
	vMC	0.27	61	6	25	0.002	R

The internal representations of item-order associations can be visualized by performing a multidimensional scaling of the activation patterns evoked by each of the six orderings (averaged across searchlights within the ROI). The resulting plot displays the similarity structure of the orderings as found in the ROIs (Fig. 5C,D). In the brain areas which represent item-order associations (Fig. 5C), sequences close in the Hamming space also lie closer to one another in the MDS plot. For example, orderings which retain one item-position mapping are closer to each other than to orderings which retain none.

To further elucidate the nature of representations we performed a post hoc analysis in the brain areas which showed significant order effects. For this purpose we ran a three-way classification of fMRI activity patterns based on the first, middle, or last item in the sequence (e.g., first-item decoding: ABC, ACB vs BAC, BCA, vs CAB, CBA) using the whole-brain searchlight approach (see Materials and Methods, Representation of order information). We did not observe above chance classification results in any brain areas suggesting that our results cannot be explained solely by the first, middle, or last items of the sequences.

Discussion

This study provides the first fMRI evidence for a specific model of order representation in the human brain. The representational similarity analysis revealed that a set of brain areas in the temporal lobe, supramarginal gyrus, and premotor cortex encode associations between items and order as predicted by a positional model (Fig. 5; Table 3). The chaining model of order representation leading to alternative pattern similarity prediction was shown to be inconsistent with the fMRI data. Our results thus suggest a neural model of order representation, which stores item codes, position codes, and the mapping between them.

These results show that item-order associations are encoded in a network of brain areas following a posterodorsal stream of cortical auditory processing pathways. Previous research on animal and human physiology has shown that the posterodorsal stream emanates from the posterior auditory belt areas (planum temporale) to parietal cortex, premotor cortex, and dorsolateral prefrontal cortex (Romanski et al., 1999; Kaas and Hackett, 2000; Hackett, 2011); and is selectively involved in the processing of spatiotemporal properties of auditory stimuli (Rauschecker, 1998; Rauschecker and Scott, 2009). Specifically, the inferior parietal lobe and the supramarginal gyrus have been shown to be sensitive to order tasks (Moser et al., 2009). Here we have shown that positional item-order associations are represented in the posterodorsal stream during both the encoding and recall phases of the order task. Furthermore, the representation of order was not detected in the primary auditory areas and was instead found in the posterior parts of the STG (including the PT) in the recall phase. This suggests that auditory sensory areas might not maintain the item-order associations beyond the initial encoding phase. We also showed that in different brain areas positional item-order associations are similar (including the vMC, SPT, SMG, and IFG; Fig. 5B) during both encoding and recall phases of the task. This suggests that these representations are relatively stable throughout the whole dorsal processing stream. We also established that the item-order associations we observed were not driven by a specific position in a sequence. This suggests that these brain areas encode item-position associations throughout whole sequence.

Here we have shown that fMRI data can be used to decode both item and item-order association codes. However, the positional models generally depend on the availability of a positional code, which provides a gradually evolving temporal context for the items in the specific sequence. Previous research on rodent neurophysiology has shown that such positional codes could be encoded by the hippocampus (Kesner et al., 2002, 2010; Manns et al., 2007; Devito and Eichenbaum, 2011). Our study was unsuited to the detection of positional codes for two reasons: first, the temporal resolution of the scanning protocol was not sufficient to acquire and deconvolve fMRI data for single items in the sequence (a whole sequence of three items was presented in 2.6 s, equal to the TR of the scanning sequence; see Materials and Methods, fMRI data acquisition and preprocessing). Second, because of auditory stimulus presentation we used a silent sequence, which caused a significant BOLD signal “drop-off” in the medial-temporal lobe (ibid). Thus, our study does not contradict previous research, which has found that the hippocampal memory system participates in representing serial order.

Theoretically, our results are consistent with a large body of modeling work on order representation, which assumes an existence of a gradually changing positional context signal (Henson and Burgess, 1997; Henson et al., 2003; Burgess and Hitch, 2006; Hitch et al., 2009). We have shown that the fMRI data support positional models over the alternative class of chaining models. It should be noted, however, that our results relate to a STM task with three-item sequences, which are comfortably within the STM span of the vast majority of people and are hence recalled without difficulty. As the sequence length increases people start relying of various cognitive strategies like grouping (Burgess and Hitch, 2006; Parmentier and Maybery, 2008; Kalm et al., 2012), which must also change the underlying representations of temporal order. Similarly, previous research suggests that the neural sequence representations change when longer sequences are transferred from short to long term memory through repetitions (Kalm et al., 2013). Furthermore, when participants recall sequences from long term memory their error profiles suggest chunking of adjacent items, which could be similar to chaining-like formation of interitem associations (Cowan and Chen, 2006; Orbán et al., 2008; Perlman et al., 2010). In fact, learning multiple overlapping sequences simultaneously would be impossible if participants could only rely on item-position associations (Kalm et al., 2013). Further research is required to study the change in order representations when long-term memory is involved. In sum, our evidence for positional models here applies to short sequences that can be recalled using short-term memory alone. Similarly, although our results support the broad class of positional models for encoding order in STM, the exact nature of the positional code in STM, including whether that position is encoded absolutely (Lee and Estes, 1981), temporally (Brown et al., 2000, 2007; Hitch et al., 2009), or relatively (Henson, 1998; Page and Norris, 1998) is unclear and will require additional research.