Contrasting visual working memory for verbal and non-verbal material with multivariate analysis of fMRI

Abstract

We performed a delayed-item-recognition task to investigate the neural substrates of non-verbal visual working memory with event-related fMRI (‘Shape task’). 25 young subjects (mean age: 24.0 years; STD=3.8 years) were instructed to study a list of either 1,2 or 3 unnamable nonsense line drawings for 3 seconds (‘stimulus phase’ or STIM). Subsequently, the screen went blank for 7 seconds (‘retention phase’ or RET), and then displayed a probe stimulus for 3 seconds in which subject indicated with a differential button press whether the probe was contained in the studied shape-array or not (‘probe phase’ or PROBE). Ordinal Trend Canonical Variates Analysis (Habeck et al., 2005a) was performed to identify spatial covariance patterns that showed a monotonic increase in expression with memory load during all task phases. Reliable load-related patterns were identified in the stimulus and retention phase (p<0.01), while no significant pattern could be discerned during the probe phase. Spatial covariance patterns that were obtained from an earlier version of this task (Habeck et al., 2005b) using 1, 3, or 6 letters (‘Letter task’) were also prospectively applied to their corresponding task phases in the current non-verbal task version. Interestingly, subject expression of covariance patterns from both verbal and non-verbal retention phases correlated positively in the non-verbal task for all memory loads (p<0.0001). Both patterns also involved similar frontoparietal brain regions that were increasing in activity with memory load, and mediofrontal and temporal regions that were decreasing. Mean subject expression of both patterns across memory load during retention also correlated positively with recognition accuracy (d_L) in the Shape task (p<0.005). These findings point to similarities in the neural substrates of verbal and non-verbal rehearsal processes. Encoding processes, on the other hand, are critically dependent on the to-be-remembered material, and seem to necessitate material-specific neural substrates.

1. Introduction

Visual working memory has been one of the best studied domains in Cognitive Neuroscience in recent decades with 2 major competing models: the classic and subsequently refined Working-Memory model by Baddely (Baddeley, 1981; Baddeley, 1992; Baddeley, 2003a) postulating dedicated memory systems, on the one hand, and the model by Cowan (Cowan, 2001; Cowan et al., 2002), on the other hand, abolishing difference between long-term and working memory, and interpreting working memory as the reactivation of long-term memory within a capacity-limited focus of attention. In the last decade or so, cognitive-neuroimaging experiments have brought (mainly visual) working memory to the fMRI scanner to identify neural correlates of encoding, maintenance and retrieval, and clarify their dependence on the to-be-remembered information. The empirical evidence produced in the pursuit of clarification and refinement of memory models has often brought up new questions, putting in doubt the completeness of both standard models (Postle, 2006).

The most common paradigm for studying working memory inside the MRI scanner has been the Delayed-Item Recognition task (DIR) (e.g. (D’Esposito et al., 1999; Rypma and D’Esposito, 1999)), using a variety of stimuli of different kinds, which often included, but were not limited to, verbal, shapes or locations. Typically, participants study a small list of stimuli, then are instructed to hold the information in mind while the screen goes blank, before they are tested on a probe item and have to indicate whether the item was contained in the original study array or not. Given the occasional confusion about the terms “working memory” and “short term memory”, we follow the conventional guidelines that postulate working memory to exceed short-term memory by involving active processing and manipulation, rather than passive shortage. By this conventional definition, the DIR task is strictly a short-term memory task, i.e. it can, but does not have to, involve working memory processing.

We used two simple DIR tasks to answer questions about the material-specificity of the neural correlates of encoding, maintaining and retrieving verbal and non-verbal information. To keep the executive demands minimal, we confined ourselves to simple information storage only, i.e. no additional manipulation of the to-be-remembered information was required. A clear left/right lateral organization for the storage of verbal/non-verbal information has been observed in functional imaging and brain-stimulation studies. Stimulation studies that are able to interfere with the normal functioning of brain regions have demonstrated this lateral organization (right = non-verbal, left=verbal) further for a variety of different brain regions: (Coleshill et al., 2004) used current micro-stimulation on epilepsy patients to disrupt normal processing of the hippocampus while patients studied words or faces. The effects of this disruption on subsequent recognition performance yielded a convincing double dissociation: stimulation of the left hippocampus was found to adversely affect word recognition only, whereas stimulation on the right adversely affected only face recognition. (Floel et al., 2004) likewise used transcranial magnetic stimulation (TMS) to disrupt neural activity in the prefrontal cortex (BA 45,47) while subjects were encoding verbal vs. non-verbal information with a similar left-right laterality of the impact on recognition performance: stimulation on the left side impacted word recognition, whereas stimulation on the right impacted recognition of abstract shapes.

Functional neuroimaging studies, while unable to test the necessity of brain regions for task performance, however, gave consistent results: (Golby et al., 2001) systematically manipulated the verbalizability of the studied material and presented words, scenes, faces, and abstract patterns, obtaining left-lateralized medial-temporal and frontal activation for words and right-lateralized activation for abstract patterns, with scenes and faces at an intermediate level of lateral symmetry. (Rothmayr et al., 2007) kept the nature of the presented stimulus material (orientation of Gabor patches) constant and only manipulated rehearsal strategies by instructing participants to maintain the information either verbally or non-verbally. Again, in a direct contrast of instructions for verbal or non-verbal rehearsal, verbal rehearsal activated mainly left language-associated temporal and parietal areas, while producing right dorsolateral prefrontal and medial prefrontal activation for non-verbal rehearsal.

Most of the studies probing the lateralization of brain activation with stimulus material or rehearsal instructions have proceeded on a mass-univariate or voxel or ROI level. Whether the areas underlying the encoding of different stimulus materials themselves were mutually correlated is an open question. It is conceivable that a single network of brain areas can accommodate all observed differences in laterality of mass-univariate results: this network, for instance, could include frontal, mediotemporal and parietal brain areas whose loadings would be weighted positively on the left side, and negatively on the right. Greater verbalizability of the to-be-remembered information would thus increase the expression of such a network, resulting in a net increase of activation on the left side, while causing a de-activation on the right side. Reducing the verbalizability would have opposite effects, but causing neural changes along a single dimension, rather than breaking up the activity into apparently distinct regional parts.

With multivariate analysis we can pursue this further, and performed a simple non-verbal Delayed-Item-recognition task using a common variant of the Sternberg task. The stimuli were 1,2, or 3 unfamiliar nonsensical line-drawings that should be unnamable (‘shape task’). We performed Ordinal Trend Canonical Variates Analysis (OrT CVA) (Habeck et al., 2005a) to look for memory-load related activation patterns in all three task-phases. We also re-derived the results of a verbal version of the task that used 1,3, or 6 letters, but whose task structure was otherwise identical (Habeck et al., 2005b) (‘letter task’). The load-related activation patterns that could be identified in all phases of the letter task were prospectively applied to the corresponding task phase of the Shape task, resulting in a network expression score per subject and load level. These network scores can be related to the score of the networks derived from the shape task itself. If both activation patterns are independent, their corresponding scores should be uncorrelated, whereas a positive or negative correlation indicates some degree of dependence, even if the areas with significant loadings in the networks are quite different.

Thus, we are equipped to answer specific questions and check the consistency of the answers with predictions by current models of working memory: (1) what are the neural substrates of storing and rehearsing verbal and non-verbal information, and (2), what are the similarities and differences of these neural substrates depending on the stimulus material (verbal/non-verbal)? The operational definition of similarities and differences was broken down further: (1) is the usage of the Letter patterns related to the usage of the Shape patterns, with similar brain-behavioral relationships? (2) Are the patterns similar in terms of topographic composition?

Apart from these primary questions, it is a natural exercise to ask what the two major models of Working Memory would predict for the results of our analysis. For these predictions we strictly assume a one-to-one correspondence between cognitive constructs and their neural substrates. The “embedded-process model” of Working Memory pioneered by N. Cowan (Cowan, 2001; D’Esposito, 2007; Miyake and Shah, 1999) and the earlier Working-Memory model formulated by A. Baddeley (Baddeley, 1992; Baddeley, 2003a). The predictions from both models are not necessarily in opposition to each other. The embedded-process model posits that short-term and working memory are not fundamentally different from long-term memory and might involve the same brain areas processes, just under the direction of the focus of attention; further, it does not insist on the transfer of information from brain areas involved in encoding to areas explicitly and exclusively to dedicated maintenance. Regarding different stimulus modalities, the model appears to us as agnostic and no particular differences are postulated on the basis of the kind of information that is to be remembered; on the other hand, such differences cannot be excluded either. The strongest prediction from the embedded-process model would be the following:

Cowan 1 – The neural substrates of encoding, maintenance and recollection show strong similarity, i.e. the topographic composition of the neural substrates are similar as well as the amount to which subjects utilize these areas is similar across task-phases.

In the Methods section we define rigorous measures of topographic similarity and similarity of subject utilization that are then applied in the Results section. For the current discussion, a merely conceptual understanding is sufficient.

When it comes to the Baddeley model, there are clear differences between the different phases of a working-memory task and between different kinds of to-be-remembered information: since maintenance of information involves dedicated storage buffers different from those used in the encoding process, we anticipate clear differences between task phases:

Baddeley 1 – The neural substrates of encoding, maintenance and recollection are different, i.e. involve different brain areas and subject utilization of these areas is different too.

Key differences particularly regarding the maintenance of information can be anticipated on account of the Baddeley model. For verbal information the phonological loop is involved, possibly employing sub-vocal articulatory rehearsal resulting in an activation of Broca’s area (Brodmann area 44). For the maintenance of non-verbalizable shape stimuli on the other hand, this should not be the case and the visual sketchpad with non-articulatory rehearsal strategies should result in different brain areas to become active. Thus, concerning material-specificity, we derived a second prediction:

Baddeley 2- Neural substrates of different task phases also show specificity with respect to the to-be-remembered information, i.e. the topographic composition of the neural substrates should be different and subject utilization should be different depending on whether letters or non-verbal shapes are to be remembered.

Since the analytic framework in this paper is spatial covariance analysis and our similarity measures are algorithmically formulated as spatial and subject correlations, the prediction “Baddeley 2” cannot strictly be refuted. Topographic similarity and similarity in subject utilization across stimulus modalities would still leave room for some crucial differences – it would just show that the similarities between both modalities, presumably pertaining to a generic episodic buffer that is involved in maintenance of any information, are more important and are more influential for the similarity measures computed within our analytic framework.

From these considerations one can see that “Cowan 1” and “Baddeley 1” are in direct contradiction, whereas “Baddeley 2” speaks to one particular aspect of the Baddeley model only, which is left unspecified in the embedded-process model.

2. Results

Behavioral performance

Twenty five young adults (12 males) performed the Shape DIR task (mean age 24.0 yrs, STD=3.8 yrs). Subjects encoded, retained, and were tested on, either 1,2, or 3 nonsense-shapes (for details see Experimental Procedure). There was a significant effect of memory load on median reaction time: RT(1)=1058ms ± 194ms, RT(2)=1218ms ± 208ms, RT(3)=1316 ms ± 182ms; F_(2,72)=11.16, p<0.0001. Recognition accuracy as measured by d_L was also affected and decreased with increasing memory load: d_L(1)=2.52 ± 0.57, d_L(2)=1.69 ± 0.74, d_L(3)=0.82 ± 0.42; F_(2,72)=52.11, p<0.0001. Forty young adults performed the Letter-DIR task (Habeck et al., 2005b) (behavioral and demographic summary available in the original paper). Subjects encoded, retained and were tested on either 1,3, or 6 letters (for details see Experimental Procedure). In contrast to the Letter DIR task, where only RT was affected and subjects essentially performed at ceiling accuracy regardless of memory load, the shape version of this task is more demanding and affects both RT and recognition accuracy.

Ordinal Trend analysis in brain imaging data

We performed Ordinal Trend analysis (OrT CVA) on all task phases of both Letter and Shape Tasks for the identification of activation patterns that show a monotonic change with memory load on a subject-by-subject basis. The task is explained more detail in the Experimental Procedure section; it consisted of a 3-second encoding phase (STIM), a 7-second maintenance phase (RET), and a 3-second recognition phase (PROBE). We reproduced earlier results from the Letter task (Habeck et al., 2005b) with minor modifications. All brain regions that were deemed reliable by the bootstrap procedure for all task phases in both Letter and Shape data are listed in tables A.1-A.5 in the Appendix. The brain regions for the Letter results in the tables in the Appendix appear a lot sparser than in the original paper, partly due to a more stringent threshold criterion with |Z|>3.09, p<0.001 and minimum cluster size of 10 voxels. We discuss the re-derived Letter results briefly in the Discussion section, but otherwise refrain from focusing on it further.

Apart from the listing of brain areas, we display the results of the OrT CVA in the following summary table. Load-related patterns were found everywhere but the PROBE phase in the Shape task.

We also checked for additional brain-behavioral correlations and detected one: mean expression across memory loads of the load-related pattern obtained from the RET phase in the Shape task correlated significantly with recognition accuracy: R²=0.41, p=0.0005. Thus, increased expression of the load-related pattern during the RET phase resulted in better recognition accuracy in the PROBE phase.

Relationship between Letter and Shape task

Of key interest is the question whether the neural correlates of encoding, maintenance and retrieval are specific to the type of stimulus material being studied and retained, or not. To answer this question, we prospectively applied each load-related pattern from one kind of stimulus material to the corresponding task phase of the other kind of stimulus material and checked whether the load-relationship in the pattern utilization was preserved. This was achieved with a permutation test of 10,000 iterations that broke the load-level assignment, but left the subject assignment intact. The test statistic was the number-of-exception criterion, i.e. the number of subject who failed to demonstrate a monotonic relationship in their pattern utilization and memory load, as explained in our OrT CVA methods paper (Habeck et al., 2005a). The results are shown in table 2.

Table 2.

Prospective application of load-related activation patterns to data of the other stimulus type; check whether load-relationship, i.e. ordinal trend, still holds. P-levels were estimated with a permutation test of 10,000 iterations. ‘Shape→Letter’ indicates that the pattern derived from Shapes is applied prospectively to the Letter data. ‘Letter→Shape’ indicates the converse operation.

Task Phase	Shape → Letter	Letter → Shape
STIM	No ordinal trend, p=0.26	No ordinal trend, p=0.11
RET	Ordinal Trend, p=0.0003	Ordinal Trend, p=0.0021
PROBE	N/A, i.e. no Shape Ordinal Trend	No ordinal trend, p=0.09

For the RET phase, the load-relationship was preserved for applications from and to both stimulus types. This is a first piece of evidence that load-related rehearsal of different stimulus materials shares some similarities, and might employ similar brain regions.

Next, we went beyond the load relationship and computed two similarity measures which are explained in detail in the Methods section. Briefly, the first similarity measure investigates to what extent utilization of both letter- and shape-derived activation patterns is similar across subjects, while the second similarity measure directly correlates the topographic loadings of both covariance patterns. P-levels were derived from permutation tests as explained in full in Section 4.

Table 3 shows that load-related rehearsal of Shapes and Letters has much in common: concerning both topography and degree of utilization in both data sets, the load-related patterns are correlated. Further, even the brain-behavioral relationship with recognition accuracy in the Shape task is preserved: mean expression across memory load of the Letter pattern in the Shape data correlated positively with d_L: R²=0.30, p=0.005.

Table 3.

Relationships between covariance patterns derived from both stimulus materials, and their subject expression. (The PROBE phase only produced a significant pattern in the Letter task, and was thus left out here.) The first row assesses the similarity of the brain regions involved in the covariance patterns of both stimulus materials, using the topographic correlation between pattern loadings as a test statistic. The second and third row assesses the similarity of utilization of both types of covariance patterns, listing the correlations between the subject expressions of both covariance patterns in data of both stimulus types. To help the reader, we explain the content of the field in the 2^nd row and 1^st column: in the STIM phase of the Letter data, utilization of both Letter- and Shape-patterns was not significantly related to one another as can be seen from the p-value of 0.17.

	STIM	RET
Topographic concordance of covariance patterns; CORR(vl, vs )	R=0.14, p=0.30	R=0.37, p=0.008
Mean expression of covariance patterns in Letter data; CORR(L · vl, L · vs )	R=−0.22, p=0.17	R=0.36, p=0.02
Mean expression of covariance patterns in Shape data; CORR(S · vl, S · vs )	R=−0.03, p=0.88	R=0.80, p=5 e- 6

Figure 1 displays load relationships and mutual dependence of both RET Shape and Letter pattern expressions.

Figure 1.

Illustration of the similarity of the neural substrates of Letter and Shape rehearsal. All neural data presented in this figure pertain to task-phase RET. For panels a-c, any score obtained from an application of a pattern to the Shape data is color-coded as blue; any pattern score obtained from an application to the Letter data is color-coded as red. A: subject expression of the load-related Shape pattern in the Shape and Letter data as a function of memory load. Expression values for each participant are connected by lines to represent the ties in the data. A significant Ordinal Trend was detected in both Shape and Letter data with p=0.006 and p=0.0003, respectively. B: subject expression of load-related Letter pattern. Here, also an Ordinal Trend was verified for both Shape and Letter data at p=0.0021 and p=0.001, respectively. C: mean expression of the load-related Shape pattern across all load levels (x-axis) against the corresponding expression of the load-related Letter pattern (y-axis), both in the Shape and Letter data. Expression values of both patterns were positively correlated in both data sets: greater usage of the Shape pattern was associated with greater usage of the Letter pattern hinting at functional similarity. Correlation values and p-levels of both pattern scores for the Shape and Letter data were R²=0.62 (p=2 e-6) and R²=0.15 (p=0.02), respectively. D, filled circles: mean expression of both rehearsal-pattern scores in Shape data against the corresponding discriminability measure d_L: greater subject expression of the Shape pattern in the Shape data correlates with better behavioral performance (R²=0.42, p=0.0005). Open circles: similar brain-behavioral correlation, but for the expression of the Letter pattern. Greater expression of the Letter pattern in the Shape data also correlates with better discrimination performance (R²=0.30, p=0.0043), even though the Letter pattern was derived from a different task and subject sample.

Both patterns display similar relationships to memory load, Shape recognition accuracy d_L, and are highly correlated to each other in their subject expression.

Figure 2 shows areas of significant loadings in both Shape and Letter patterns. For the areas increasing in activation with increasing memory load (fig 2a), one can perceive an overall lateralization on the left side of the brain, with the Letter pattern showing much bigger spatial extent than the Shape pattern.

Figures 2 a,b.

Significant positive and negative loadings as ascertained by the bootstrap test for both Shape (red color) and Letter pattern (blue color) derived during RET, overlaid on a probabilistic gray-matter mask. A threshold of |Z|>3.09 was adopted. Positive loadings, i.e. indicating increasing activation with increasing memory load, are shown in figure 2a, negative loadings, i.e. indicating decreasing activation with increasing memory load, are shown in figure 2b.

The same is true for the areas of negative loadings (fig 2b), where both patterns show de-activation in medial prefrontal cortex (BA 8,10), posterior cingulate (BA 23), and bilateral angular gyrus (BA 39), but the Letter pattern shows a larger extent of de-activation, with an additional left temporal area (BA 22).

3. Discussion

The current study produced somewhat complex results. We will first summarize the findings in an easy write-up, without a detailed listing of p-levels etc. Then we will try to integrate the findings into the larger context.

Load-related activation patterns in the non-verbal DIR task

We successfully identified memory load-related patterns in the encoding and rehearsal phases of a non-verbal DIR task. Employment of the load-related pattern during rehearsal was beneficial for task performance as measured by d_L, while there was no brain-behavioral correlation for the encoding phase. No load-related pattern could be produced from the probe phase data.

Brief commentary on re-derived load-related activation patterns in the Letter-DIR task

We also re-derived load-related patterns from data of the Letter version of this task that used verbal stimuli (Habeck et al., 2005b), after reprocessing with SPM 5. The results are overall consistent with the SPM 99 findings. (See table in the Appendix – note: more conservative thresholds were used than in the original article.) The load-related effect during STIM did not contribute as much variance as in the SPM 99 data, and subsequently necessitated more principal components to be captured (7 PCs as opposed to 2 PCs in the original paper). A further consequence is that the bootstrap weights are lower and that fewer areas survive the criteria for inclusion in the tables A.3 and A.4 in the Appendix.

The load-related effect in the RET phase, while involving frontoparietal brain regions, appeared more right-lateralized than in the original paper. We also tested the brain-behavioral correlation with NARTIQ and reaction time that was reported in the original paper (Habeck et al., 2005b): the load-related increase from the 1 to 6 letters correlated negatively with NARTIQ and positively with the corresponding reaction difference, prompting us to speculate that lesser utilization of the load-related pattern was a sign of greater efficiency at the task and higher NARTIQ. In the reprocessed data, the correlation between load-related increase and NARTIQ was strengthened with the same sign: R²=0.28, p=0.0004. However, the correlation with reaction time disappeared, R²=0.06, p=0.1156.

The PROBE phase produced a pattern, but -as in the original SPM 99 data- it did not contribute much variance and necessitated 8 principal components in its derivation. Due to the more stringent criteria demanded of the bootstrap maps, no areas were significant above threshold.

Material specificity of encoding and rehearsal substrates

We prospectively applied load-related patterns obtained from the STIM and RET phases of both Letter and Shape task to the data of the other stimulus type. The purpose was to answer the questions: (1) is the usage of the Letter patterns related to the usage of the Shape patterns and vice versa, with similar brain-behavioral relationships? (2) Are the patterns similar in terms of topographic composition? We aimed for a comprehensive documentation of how the nature of the to-be-remembered information influences the neural correlates of the cognitive processes investigated in the DIR task.

For the encoding phase (=STIM), we found no relationship between Letter and Shape patterns, not only in terms of subject expression, but also in terms of topographic composition. Noticeable was the finding that bilateral Insula (BA 13) and Superior Temporal Gyrus activation (BA 29, 38, 42) were identified with significantly positive loadings in the Shape pattern, but bilateral occipital areas in the Lingual and Fusiform Gyrus (BA 18, 19) (as well as the Anterior Cingulate Gyrus (BA 32), and the left Precuneus (BA 7)) loaded negatively. The lack of involvement of the occipital areas in load-related increases in activation is at first glance surprising and deviates from the Letter pattern, where mainly primary visual areas (BA 17) were found.

However, a closer survey of the literature reveals that the involvement of the Superior Temporal Gyrus is consistent with previous reports: Karnath et al. (Karnath, 2001) located it as the primary focus of lesions in spatial-neglect patients. This was later partially corroborated by a TMS study (Ellison et al., 2004) and a study using intraoperative current stimulation in awake brain-surgery patients (Gharabaghi et al., 2006). Both stimulation studies could establish the role of right Superior Temporal Gyrus as a necessary brain location for successful exploratory visual-search behavior. We can speculate that these areas are more important than extra-striate visual areas (Fusiform Gyrus BA 19, 18), which have been shown to activate in object-recognition and –matching tasks, and exhibit negative loadings in our covariance pattern. Increasing memory load during visual encoding of complex unfamiliar shapes thus might shift neural resources from the latter areas to the former, inducing a negative correlation between the two.

For the maintenance (=RET) phase, the results were different: employment of both Letter and Shape pattern was positively correlated, with a similar brain-behavior relationship to d_L; further, topographic composition of the patterns was significantly correlated as quantified by our similarity measure in a non-parametric test. When inspecting the areas that survived the threshold criteria of the bootstrap test, left lateralization is apparent, for both Shape and Letter pattern (see tables A.2 and A.4). This is in line with the results from our earlier paper (Habeck et al., 2005b) that focused on the verbal DIR task, where an essentially bilateral frontoparietal network was identified, including Broca’s area, i.e. the left Inferior Frontal Gyrus (BA 44/45). In the current study, Broca’s area -thought to be involved in articulatory rehearsal of verbalizable information- was again picked up as part of a large inferior frontal cluster in the re-processed Letter-task data in the load-dependent covariance pattern.

Beyond the immediate description of our results for the maintenance phases, we can speculate that our findings give additional support to the postulation of an episodic buffer operates independently of the modality of the to-be-remembered information (Baddeley, 2000; Jones et al., 1995; Pearson et al., 1999). Our findings of cross-applicability between modalities suggest that the contribution of this buffer to the neural substrates of modality-specific rehearsal is large enough to assure functional invariance and de-emphasizes the contributions from the modality-specific phonological loop and visual-spatial sketch pad. This suggestion is additionally supported by the implausibility of the phonological-loop involvement in the maintenance of the non-verbal stimuli: (1) as shown in Figure 4, the shape stimuli in our non-verbal task were chosen to minimize verbal strategies of encoding and maintenance; (2) anecdotally, task participants also reported that they did not report to naming or sub-vocal rehearsal, and (3) Broca’s area 44, present in our verbal maintenance pattern, was notably absent in our non-verbal pattern.

Figure 4.

Task structure for non-verbal working-memory task with sample shapes for a memory load of 2. One can appreciate that sample shapes are difficult to verbalize, minimizing the contribution of the phonological look to the rehearsal process.

The probe phase failed to produce a statistically significant load-related activation pattern in the Shape data. Prospective application of the load-related pattern derived from the probe phase in the verbal task could not establish a significant relationship between memory-load level and subject expression in the non-verbal task either. This analytical failure signals that, again, cognitive processes not modulated by memory load might be contributing more variance than load-related processes, making the detection with PCA difficult. Methods that might better hone in on load-related variance in the probe phase might then be successful; however, the additional failure of the prospective application of the Letter pattern argues that, if such a load-related Shape pattern can ultimately be derived, it will probably be independent in terms of both regional composition and subject expression of the Letter pattern.

Relation to models of working memory

We now turn to the question of how our findings about material-dependence of load-related activation patterns can inform the ongoing debate about visual working memory.

Two Working-Memory Models have garnered the major share of attention, and numerous neuroimaging studies have matched their theoretical predictions to empirical findings. Of these two models, the Baddeley model (Baddeley, 2003a; Baddeley, 2003b) has been applied most often and, as two reviews (D’Esposito, 2007; Postle, 2006) have pointed out, has often just been mapped onto neural substrates in a confirmatory manner without considering theoretical alternatives. One rival candidate, Cowan’s model (Cowan, 2001; Cowan et al., 2002), has been receiving more attention in recent years too. Relating neuroimaging finding to these models is, in our opinion, not always straightforward, but we tried to deduce falsifiable predictions from each model for the neuroimaging findings and checked whether our results were consistent with these predictions.

One of the key differences between the two models is the way working memory is conceptualized: Cowan’s sees the neural substrates of working memory as a subset of brain areas that are active in long-term memory as well; further, rehearsal and storage processes are postulated to involve the brain areas that are active in the perception of the to-be-remembered material. Ongoing delay activity in these regions first activated during encoding abolishes the need to transfer the information to dedicated storage buffers; this is intuitively appealing and avoids the logical pit fall of a proliferating number of different ‘grandmother’ storage systems depending on the information to be stored (D’Esposito, 2007). The solution for coding a large variety of sensory material through integration of a finite number of processing modules can thus be extended to working memory as well.

While appealing on theoretical grounds, this simple notion is not corroborated by our data: in the current Shape version of the DIR task for instance, the load-related covariance patterns identified during STIM and RET, involve different brain areas, and have no relationship in their degree of expression across subjects either. The patterns yield non-significant results for all the similarity measures outlined before, not only for the Shape, but also the Letter task (listings of values and p-levels omitted for brevity). The degree to which a participant utilizes the load-related pattern during STIM thus gives no information about her utilization of the load-related pattern obtained from RET. Thus our hypothesis “Cowan 1” presented in the Introduction is not validated, and it seems more likely that a transfer of information occurs from the areas responsible for the initial sensory processing to a generic, i.e. stimulus-independent, buffer in which information is held for further processing. The only possibility, eluding our analysis, is that there are different patterns for different levels of memory load, but these patterns are present to an equal degree in all task phases, i.e. STIM, RET, and PROBE. While unappealing, this possibility could be addressed by an analysis with additional constraints, but is beyond the scope of the current article.

The lack of constancy across task phases that invalidated the predictions captured in “Cowan 1” directly verify the predictions “Baddeley 1”. When it comes to the other prediction of the Baddeley-Hitch model, our results showed that neural correlates of the phonological loop and visuospatial sketchpad which are postulated with differential involvement for verbal and non-verbal visual information, respectively, seem very similar, at least in terms of the relationship between degree of pattern expression and behavioral performance. In fact, the Letter pattern could be substituted for the Shape pattern in our data with equal association to participants’ recognition accuracy in the Shape task as measured by d_L. While Broca’s area (BA 44), included in a large left inferior frontal cluster visible in Figure 2a, is a prominent super-threshold area in the covariance pattern obtained from the Letter-RET phase, this region’s material-specificity does not obviate the successful prediction of behavior when it is substituted for the covariance pattern derived from the Shape-RET task. Because of the design of our analyses, this finding cannot argue against a particular role of Broca’s area for implementation of the phonological loop, but it seems such differential activation is dwarfed by the commonalities in the covariance pattern i.e. the neural substrates of material-independent processing, reflecting the central executive and the episodic buffer. These two pattern-constituents produced the major portion of the variance in the data captured by our analysis, as well as accounting for behavioral performance.

The invariance with respect to stimulus type that was observed in the maintenance phase was even more impressive given that verbal and non-verbal tasks were run in different subject samples: the effect was strong enough to prevail in the face of task-unrelated subject variance, and cross-replicate from one sample to the other. In our experience, such favorable replication performance is rare for fMRI. On the other hand, the fact that both tasks were run in different samples does not permit a strong inference that the neural substrates of encoding are necessarily different: we merely observed a failure to reject the null-hypothesis (of difference), and rejection of the null-hypothesis might be made more difficult by the presence of task-unrelated subject differences in the two samples.

We can also investigate how our working-memory tasks, which only involve storage of information, relate to tasks that have higher processing demands, notably the well known N-back. N-backs combine encoding, maintenance, manipulation and recognition in one task phase, and are thus considerably more demanding than our delayed-response tasks, and can thus be expected to activate more brain areas. A meta-analysis of neuroimaging studies (Owen et al., 2005) provides a convenient summary of verbal and non-verbal N-back tasks that probe both object identity as well as object location. Most of the surveyed studies employed univariate analysis and are thus not totally compatible with our study, but the visual superposition is nevertheless instructive. We super imposed the positive loadings (p<0.001) in both our verbal and non-verbal maintenance patterns with the activation foci reported in (Owen et al., 2005) for the identity probes. All of the areas displayed represent increasing activation with increasing memory load. We centered 5 × 5 × 5 voxel cubes on the coordinates reported in (Owen et al., 2005) to enhance visibility.

Figure 3 conveys good similarity between the N-back findings and our covariance patterns for the dorsal aspect of both parietal and frontal regions. Material-specificity seems to be overall reasonably preserved between N-back and our tasks and particularly the verbal N-back results (blue) go along with our verbal results (cyan) quite well. One striking concordance is between our verbal results (cyan) and the non-verbal N-back in the left inferior frontal gyrus (BA 10), hinting at possible modality-independent contributions by an episodic buffer. Beyond this concordance, the N-Back tasks show more inferior frontal activation. Given that these tasks mixed encoding, maintenance, manipulation and recollection in one task episode it is to be expected that more activation foci appear than during more passive maintenance of information.

Figure 3.

Superposition of significant loadings in our maintenance patterns with activation foci reported for identity-monitoring N-back tasks (Owen et al., 2005). Warm color scale: super-threshold loadings (Z>3.09) for the Shape rehearsal pattern; cool color scale: super-threshold loadings (Z>3.09) for the Letter rehearsal pattern. Red: activation foci for non-verbal N-back tasks; blue: activation foci for verbal N-Back tasks.

Considering encoding in particular, we note that the encoding phases of our DIR tasks did not show the convergence evident for the maintenance phases. Our derived neural substrates manifested neither topographic similarity, nor similarity regarding subjects’ utilization. Our verbal encoding pattern mainly showed a small focus of bilateral activation with increasing memory load in the cuneus (BA 17), while our non-verbal encoding pattern shows extensive bi-literal activation with increasing memory load in the temporal lobe, while areas in the occipital lobe decreased in activation with increasing memory load. While results of our multivariate analysis again are not strictly compatible with earlier univariate analyses, we can consider the large amount of prior neuroimaging research regarding verbal and non-verbal encoding.

Prior research of word versus object encoding has postulated a “visual word form area” (VWFA) preferentially involved in the encoding of words (Braet et al.; Dien, 2009; Hillis et al., 2005; Mei et al.; Vigneau et al., 2005), although recent evidence points to similar involvement in object and face recognition (Mei et al.). The debate has not been settled as to the exact topographic location, and often subject-specific localizer tasks are employed to identify this area. However, usually the VFWA is localized more laterally in inferior-occipital cortex compared to the cuneus activation foci identified in our study. We can speculate that only iconic encoding processes show a clear relationship with memory load, while VWFA involvement might be constant and independent of memory load. If this was the case, our analysis which specifically identifies load-related neural substrates would be blind to VWFA.

Object encoding, on the other hand, involves the inferior temporal cortex and the long line of evidence gathered in humans and primates was discussed in a recent review (Hoffman and Logothetis, 2009). Our results for the non-verbal task seem more somewhat consistent with this evidence, although the positive loadings in our non-verbal encoding pattern are located about 10mm more superior to the usually reported inferior temporal locations.

Our activation patterns are therefore hard to reconcile with the extant literature; however, the details of our derivation is considerably different too. (1) We utilized multivariate analysis to identify patterns that show a monotonic increase with memory load in a distributed network of brain regions. Tightly circumscribed focal activation and activation that is constant across memory load will not play a prominent role in our findings. (2) Our analysis derives monotonically changing activation patterns during encoding, regardless of later retrieval success; this might be another reason for omissions of commonly identified areas.

4. Experimental Procedure

Delayed-Item-Recognition Task

The Delayed-Item-Recognition (DIR) task is a nonverbal adaptation of the Sternberg task (Sternberg, 1966; Sternberg, 1969), using unnamable shapes as task stimuli. Task parameters and training procedures were identical to those reported in our previous studies (Habeck et al., 2005b), where stimuli were letters. In brief, each trial of the DIR task consisted of set presentation (STIM), retention delay (RET) and probe presentation (PROBE). Based on pilot studies stimulus set size ranged from 1 to 3 shapes and was varied pseudo-randomly across trials via a random-without replacement scheme. The non-verbal stimuli consisted of 450 different computer-generated closed-curve shapes. Each shape was presented only once in the testing conditions of each participant. This offered an advantage in that both the novelty and appearance of the shapes maximized visual demands and limited the extent of phonologic processing.

There were three experimental blocks each consisting of 10 trials with 5 true positive and 5 true negative probes per set size yielding a total of 30 trials per block and 90 trials for the entire task per participant. Blank trials (presentation of a blank screen for two seconds, requiring no behavioral output) were pseudo-randomly interspersed between delayed item recognition trials to both provide a baseline condition for positive control purposes and reduce the likelihood of neural recognition predictive of the beginning of trials. The pseudo-randomization of these blank trials was via a random-without-replacement scheme (thus, more than one blank trial could occur sequentially, leading to an effectively jittered inter-trial interval), with a total of 70 blank trials per block. The participants indicated whether the probe item was included in the initial set by a differential button press (left hand=no, right hand=yes). The participants were instructed to respond as quickly as possible.

In order to minimize confusion when addressing different task phases, stimulus materials and memory loads, we provide a key for the most important short hands. Task-phase shorthand forms were already introduced as STIM, RET, and PROBE. Verbal and non-verbal tasks will be denoted as ‘Letter’ and ‘Shape’, respectively. Memory-load levels were denoted as ‘low’, ‘medium’, and ‘high’. ‘Medium’ corresponds to 2 items in the Shape task, but 3 letters in the Letter task; ‘high’ corresponds to 3 and 6 items in Shape and Letter task, respectively. Data pertaining to the probe phase of the Letter task for set size 3, for instance, would thus be denoted as ‘Letter-PROBE-Medium’.

fMRI data acquisition and image pre-processing

During the performance of each block of the delayed item recognition task, 207 BOLD images were acquired with an Intera 1.5 T Phillips MR scanner equipped with a standard quadrature head coil, using a gradient echo echo-planar (GE-EPI) sequence [TE/TR = 50 ms/3000 ms; flip angle = 90 degrees; 64 × 64 matrix, in-plane voxel size = 3.124 mm × 3.124 mm; slice thickness = 8 mm (no gap); 17 trans-axial slices per volume]. Four additional GE-EPI excitations were performed before the task began, at the beginning of each run, to allow transverse magnetization immediately after radio-frequency excitation to approach its steady-state value; the image data for these excitations were discarded. A T2-weighted, fast spin echo image was also acquired from each subject for spatial normalization purposes [TE/TR = 100 ms/2000 ms; flip angle = 90 degrees, 256 × 256 matrix; in-plane voxel size = .781 mm × .781 mm; slice thickness = 8 mm (no gap); 17 trans-axial slices per volume]. Task stimuli were back-projected onto a screen located at the foot of the MRI bed using an LCD projector. Subjects viewed the screen via a mirror system located in the head coil. Responses were made on a LUMItouch system (Photon Control Company). Task onset was electronically synchronized with the MRI acquisition computer. Task administration and data collection (RT and accuracy) were controlled using PsyScope (Macwhinney et al., 1997).

Image pre-processing and analysis were implemented using the SPM5 program (Wellcome Department of Cognitive Neurology) and other code written in MATLAB 7.0 (Mathworks, Natick MA). The following steps were implemented for each subject’s GE-EPI dataset: Data were temporally shifted to correct for the order of slice acquisition, using the first slice acquired in the TR as the reference. All GE-EPI images were realigned to the first volume of the first session. The T2-weighted (structural) image was then co-registered to the first EPI volume using the mutual information co-registration algorithm implemented in SPM5. This co-registered high-resolution image was then used to determine parameters (7 × 8 × 7 non-linear basis functions) for transformation into a Talairach standard space (Talairach, 1988) defined by the Montreal Neurologic Institute (MNI) template brain supplied with SPM5. This transformation was then applied to the GE-EPI data, which were re-sliced using sinc interpolation to 2 mm × 2 mm × 2 mm.

At the first-level GLM, the GE-EPI time-series were modeled with regressors that represented the expected BOLD fMRI response (relative to the blank intervals) to the three DIR trial components of memory set presentation, retention delay, and probe presentation, separately for set size (1-3) and probe type (true positive/true negative). DIR Trials without motor responses from the subject during the probe period were modeled separately, and were not included at the second-level GLM analysis. For the model neural response, two rectangular functions (and hence, two regressors) were used for the stimulus phase and the probe phase: one modeling a relatively brief (400 ms) neural response at the beginning of that trial component, and another modeling a neural response lasting throughout that entire component (3000 ms); a single rectangular function of 7000 ms duration was used for the retention delay. These rectangular functions pertaining to the neural response were convolved with the canonical hemodynamic response function supplied by SPM5 to construct the regressors that were finally used in the first-level GLM analysis. Contrasts were estimated for each load level, trial phase and probe type and were carried forward to the second level group analyses. Finally, all images were masked according to the SPM-supplied probabilistic gray-matter mask: only voxels with a probability of gray-matter probability of p>0.5 were retained in the analysis.

This manner of data pre-processing has been used previously (Zarahn et al., 2007) and differs in some details from the analysis presented earlier for the verbal version of this task (Habeck et al., 2005b): for this earlier study, SPM99, instead of SPM5, had been used for GLM model estimation, employing different assumptions about autocorrelation in the time series; second, all 3 task phases were just modeled with one rectangular function lasting throughout the task epoch (stimulus: 3000 ms, retention: 7000 ms; probe: 3000 ms). This was subsequently changed to the framework described for the current study to ensure maximum specificity of any modeled task activity, i.e. activity in the retention period, for instance, should only be detected by the appropriate retention-period predictor. The new framework ensures better specificity without overly sacrificing sensitivity ¹. To ensure better cross-applicability of neural data from both Shape and Letter task, we decided to reprocess the Letter data afresh in SPM5 with the updated modeling framework. For the sake of space, we kept discussion of these results to a minimum to avoid repetition, and only mentioned obvious deviations from the results of the earlier paper (Habeck et al., 2005b). Details of subject sample and behavioral results will not be repeated here, but can be obtained from the earlier publication (Habeck et al., 2005b). Tables of brain areas emerging from the analysis of reprocessed Letter data were given in an Appendix, but otherwise not elaborated on.

Group-level covariance analysis

Ordinal Trend Canonical Variates Analysis (OrT CVA) was performed on the data. This analysis is similar to other regional covariance analyses techniques, notably Partial Least Squares, to the extent that it applies principal components analysis (PCA) to the data matrix that is transformed using a matrix representing the experimental design. In addition, activity that changes as a function of study participant, but is invariant with respect to the memory-load parameter was removed from the data prior to the application of the design matrix (see the original paper (Habeck et al., 2005a) for detailed discussion). OrT CVA was designed to identify a covariance pattern in the MR signal whose expression increases across memory load levels (low/medium/high) for as many subjects as possible. Such a covariance pattern can be produced from a linear combination involving a small set of principal components. The coefficients for this linear combination are obtained through a multiple linear regression, aiming for an activation of the resulting pattern that best approximates a linear mean trend across the memory load levels.

A multivariate approach proceeds in a brain wide manner without any prior assumptions about brain localization. On the other hand, our analysis tests for types of activation with stringent constraints: patterns of sustained functional connectivity whose individual subject expression follows the experimental parameter (in this case: memory load) monotonically. By incorporating this constraint of a monotonic relationship with memory load on a subject-by-subject basis, we can increase the confidence of finding highly specific neural correlates of load-dependent encoding, retention, and retrieval processes.

Whether a voxel weight is reliably different from zero is assessed by a bootstrap procedure. Denoting the voxel weight obtained for the covariance pattern derived from the original data sample, i.e. the point estimate, as w, and the standard deviation resulting from the bootstrap resampling procedure as s_w, we can compute a z-score according to z=w/s_w. Sufficiently small variability of a voxel weight around its point estimate value in the resampling processes results in a z-value of large magnitude and indicates a reliable contribution to the covariance pattern. As the threshold criterion, we chose |z|>3.09; under the assumptions of a standard-normal distribution, i.e., z~N(0,1), this corresponds to a one-tailed probability of 0.001.

Individual subject’s expression of the activation pattern is quantified with the subject-scaling factor (SSF). The SSF is obtained by the operation of a dot product (=covariance across brain regions) between the covariance pattern in question and a subject’s task scan. It quantifies to what extent a subject expresses the activation pattern in a task scan with a single number, which can be used for further analysis.

Relating results to the data of the other stimulus type

Concordance of pattern utilization (similarity measure 1)

We can try to clarify this with simple linear algebra: imagine our data array S, for instance, every subject scan during the encoding phase for 3 nonsense line-drawings. (Letter data, likewise, would be denoted by L.) The shape task had 25 participants, thus S has 25 rows and several hundred thousand columns, i.e. one for each voxel. We can imagine both load-related covariance patterns as v_l and v_s, where v_s was derived from the shape-task data themselves, whereas v_l originated from the letter data. Pattern expressions of both networks can be computed with a Euclidean dot-product as columns vectors in the following manner:

Shape network expression (in shape data) = S · v_s

Letter network expression (in shape data) = S · v_l

We test whether the usage of both patterns during the shape encoding phase is related by computing the correlation coefficient CORR(S·v_l, S·v_s). This correlation could be significant even though some of the areas with significant loadings in the both covariance patterns might be different. In this case, the manner in which people employed both patterns would not be independent of each other.

Further, the monotonic relationship between memory-load and expression an Ordinal-Trend covariance pattern derived from one kind of stimulus material can be extended to the data of the other kind, and subjected to a statistical test. For instance, one possible null-hypothesis for this test is the absence of any meaningful load-relationship in the Letter data with the expression of the pattern derived from the Shape data. The test statistic for an Ordinal Trend, discussed in our original paper (Habeck et al., 2005a), can be computed for the expression values of the Letter-pattern in the Shape data, S · v_l. A permutation test was performed that re-assigns memory-load labels within subjects and re-computes the OrT-statistic. The relative frequency of iterations giving a value lower or equal to the point estimate value was then taken as the p-level.

Concordance of covariance patterns’ topography (similarity measure 2)

After laying out the test for a significant relationship of pattern utilization, we want to go one step further and ask whether the patterns display a significant relationship at the topographic level and involve similar brain areas. To provide a statistically rigorous answer to this is not straightforward: the smoothing performed on the brain images in the course of data pre-processing makes it difficult to estimate the real underlying number of degrees of freedom needed in any parametric test. Further, a realistic null-hypothesis should account for the generic activation and de-activation profile to be expected during visual-stimulation tasks, and for rejection demand a correlation beyond such generic activation profiles.

This prompted us to attempt a conservative non-parametric approach to estimate a null-hypothesis whose rejection would not incur an inflated p-level. For our task we had (40+25) × 3 × 3 = 585 parametric maps available: 65 participants (40 in the Letter, 25 in the Shape task) in 3 task phases (STIM/RET/PROBE) for 3 memory loads (low/medium/high). We selected a subset of these images to perform permutation tests. For each task phase we picked all 3 × 40=120 images for the Letter task and 3 × 25=75 images for the Shape task. One can construct 120 × 75 = 9,000 pairs of Letter-Shape images that were used to generate a null-hypothesis histogram. The test statistic for this histogram was a standard Pearson correlation coefficient, R, between Shape- and Letter-pattern loadings, using all voxel locations within the probabilistic (p>0.5) gray matter mask. With the generated null-histogram a two-tailed test was performed: if less than 5% of the histogram entries had an absolute value that was higher than the point estimate value |R|, the patterns were deemed topographically similar to each other.

After defining these two types of similarity measures, we will focus on the two most clear-cut scenarios (because they happened to materialize for our data in this report):

1. Maximum independence: the topographic similarity measure is non-significant and subject expression in both covariance patterns in Letter and Shape data are uncorrelated. In other words: participants use different brain regions for load-related processing depending on the stimulus material; further, the amount of utilization of these different brain regions is also independent between stimulus materials.

2. Maximum dependence: the topographic similarity measure is significant and subject expression of both covariance patterns in data of both stimulus types is correlated. In other words: participants use similar brain regions for load-relating processing for both stimulus materials. The amount of utilization is also similar for both types of stimulus materials.

One can conceive of several scenarios in-between, featuring single dissociations, but to avoid unnecessary complication, we decided to leave out their discussion.

Scenario (2) is the clearest indication that load-related processing between different stimulus materials is essentially interchangeable. It can further be bolstered if possible brain-behavioral correlations are preserved when swapping the load-related covariance pattern in question with that of the other stimulus material. For example: in task phase RET, not only was the topographic similarity measure between both Letter- and Shape-patterns significant, their subject expression was correlated in both Letter and Shape data. Further, the correlation between the Shape pattern’s expression in the Shape data with Shape-decision accuracy was matched when computing the expression of the Letter-pattern and correlating it with decision accuracy in the Shape task. Since verbal description of these complex findings can be challenging, we also give a symbolic description that might be easier to understand, using the shorthand notation introduced before. We list all significant correlations that demonstrate the maximum dependence of the load-related patterns during RET:

1. CORR(S · v_l, S · v_s) - subject expression of both patterns in the Shape data are correlated: the networks identified in the same task phase from both the Letter and Shape data are interchangeable in relation to the Shape data.

2. CORR(L · v_l, L · v_s) - subject expressions of both patterns in the Letter data are correlated: the networks identified in the same task phase from both the Letter and Shape data are interchangeable in relation to the Letter data.

3. CORR(v_l, v_s) - loadings of both patterns are correlated

4. CORR(d_L, S · v_s) -subject expression of Shape pattern in Shape data correlates with decision accuracy in Shape data (d_L)

5. CORR(d_L, S · v_l) -subject expression of Letter pattern in Shape data correlates with decision accuracy in Shape data (d_L) as well

These findings are discussed in more detail in Sections 2 and 3 of the article.

Highlights.

1. FMRI data from visual verbal and non-verbal working memory tasks were analyzed with spatial covariance analysis

2. Both tasks gave rise to activation patterns that changed monotonically with memory load.

3. The activation patterns underlying encoding showed no similarity between verbal and non-verbal task.

4. The activation patterns underlying maintenance showed great similarity between verbal and non-verbal task.

Table 1.

Global summary of reprocessed Letter and Shape results: the p-levels were determined from a permutation test of 1,000 iterations. The percentage of the variance accounted for (%VAF) was computed with respect to the raw data, i.e. prior to any application of a design matrix and prior to the removal of any taskindependent effects. ‘PC set’ indicates which Principal Components were used to construct the load-related patterns. One can appreciate the general relationship between PC-set and variance accounted for: the more PCs were used, the lower the variance accounted for.

Task phase	Ordinal Trend?	p-level	%VAF	PC set
Letter task
STIM	Yes	<0.001	2.5	1-7
RET	Yes	0.001	6.2	1-3
PROBE	Yes	<0.001	2.2	1-8
Shape task
STIM	Yes	0.001	4.2	1-6
RET	Yes	0.006	5.5	1-4
PROBE	No	N/A	N/A	N/A

Appendix

The following table list brain regions with significant positive and negative loadings in the load-related covariance patterns in both Letter and Shape tasks. The threshold criterion was |Z|>3.09, p<0.001, with a minimum cluster size of 100 voxels. In the PROBE phase of the Shape task, no load-related pattern could be identified. In the PROBE phase of the Letter data, a pattern could be identified, but no super threshold clusters were found in the bootstrap test. Coordinates are MNI coordinates in units of mm.

Table A1.

Shape STIM

X	Y	Z	Laterality	Lobe	Structure	Brodmann area	Z- value
Positive weights = increasing activation with memory load
48	−21	8	Left Cerebrum	Temporal Lobe	Superior Temporal Gyrus	Brodmann area 13	5.3232
−50	−38	15	Left Cerebrum	Sub-lobar	Insula	Brodmann area 13	4.6848
−57	−28	25	Left Cerebrum	Parietal Lobe	Inferior Parietal Lobule	Brodmann area 40	3.7765
63	−27	12	Right Cerebrum	Temporal Lobe	Superior Temporal Gyrus	Brodmann area 42	3.6238
Negative weights = decreasing activation with memory load
−26	−74	−13	Left Cerebrum	Occipital Lobe	Fusiform Gyrus	Brodmann area 19	5.6553
28	−76	−11	Right Cerebrum	Occipital Lobe	Fusiform Gyrus	Brodmann area 19	5.094
−2	25	39	Left Cerebrum	Frontal Lobe	Medial Frontal Gyrus	Brodmann area 8	5.0767
−30	−85	13	Left Cerebrum	Occipital Lobe	Middle Occipital Gyrus	Brodmann area 19	4.7736
12	−72	44	Right Cerebrum	Parietal Lobe	Precuneus	Brodmann area 7	4.58
8	−44	10	Right Cerebrum	Limbic Lobe	Posterior Cingulate	Brodmann area 29	4.5662
6	4	−2	Right Cerebrum	Sub-lobar	Caudate	Caudate Head	4.1759
2	17	58	Right Cerebrum	Frontal Lobe	Superior Frontal Gyrus	Brodmann area 6	3.6946
−24	−82	30	Left Cerebrum	Occipital Lobe	Cuneus	Brodmann area 19	3.5625
34	−80	22	Right Cerebrum	Temporal Lobe	Middle Temporal Gyrus	Brodmann area 19	3.5046
−12	−74	44	Left Cerebrum	Parietal Lobe	Precuneus	Brodmann area 7	3.5019

Table A.2.

Shape RET

X	Y	Z	Laterality	Lobe	Structure	Brodmann area	Z-value
Positive weights = increasing activation with memory load
8	−67	53	Right Cerebrum	Parietal Lobe	Superior Parietal Lobule	Brodmann area 7	4.754
34	−60	42	Right Cerebrum	Parietal Lobe	Inferior Parietal Lobule	Brodmann area 39	4.2701
−32	−56	43	Left Cerebrum	Parietal Lobe	Inferior Parietal Lobule	Brodmann area 7	4.1991
10	−79	−16	Right Cerebrum	Occipital Lobe	Lingual Gyrus	Brodmann area 18	4.1203
−40	29	26	Left Cerebrum	Frontal Lobe	Middle Frontal Gyrus	Brodmann area 9	4.0832
−8	−68	48	Left Cerebrum	Parietal Lobe	Precuneus	Brodmann area 7	3.5117
Negative weights = decreasing activation with memory load
53	−56	12	Right Cerebrum	Temporal Lobe	Middle Temporal Gyrus	Brodmann area 39	−4.8767
−12	−55	18	Left Cerebrum	Limbic Lobe	Posterior Cingulate	Brodmann area 23	−3.8281
−50	−63	27	Left Cerebrum	Temporal Lobe	Middle Temporal Gyrus	Brodmann area 39	−3.5857
−2	61	8	Left Cerebrum	Frontal Lobe	Medial Frontal Gyrus	Brodmann area 10	−3.4071

Table A.3.

Re-derived Letter STIM

X	Y	Z	Laterality	Lobe	Structure	Brodmann Label	Z-value
Positive Weights = increasing activation with memory load
−8	−80	6	Left Cerebrum	Occipital Lobe	Cuneus	Brodmann area 17	3.8588
14	−78	12	Right Cerebrum	Occipital Lobe	Cuneus	Brodmann area 17	3.8272
−54	−8	40	Left Cerebrum	Frontal Lobe	Precentral Gyrus	Brodmann area 6	3.5493
Negative Weights = decreasing activation with memory load
no super threshold clusters

Table A.4.

Re-derived Letter RET

X	Y	Z	Laterality	Lobe	Structure	Brodmann Label	Zvalue
Positive Weights = increasing activation with memory load
−52	2	30	Left Cerebrum	Frontal Lobe	Precentral Gyrus	Brodmann area 6	5.7687
−38	−54	48	Left Cerebrum	Parietal Lobe	Inferior Parietal Lobule	Brodmann area 40	5.5119
38	34	28	Right Cerebrum	Frontal Lobe	Middle Frontal Gyrus	Brodmann area 46	4.9514
28	2	60	Right Cerebrum	Frontal Lobe	Middle Frontal Gyrus	Brodmann area 6	4.8014
−46	40	8	Left Cerebrum	Frontal Lobe	Inferior Frontal Gyrus	Brodmann area 46	4.3022
−48	18	−2	Left Cerebrum	Frontal Lobe	Inferior Frontal Gyrus	Brodmann area 47	4.2955
6	−82	−22	Right Cerebrum	Occipital Lobe	Lingual Gyrus	Brodmann area 18	4.236
30	−62	58	Right Cerebrum	Parietal Lobe	Superior Parietal Lobule	Brodmann area 7	3.909
−28	−2	58	Left Cerebrum	Frontal Lobe	Sub-Gyral	Brodmann area 6	3.8504
42	−46	48	Right Cerebrum	Parietal Lobe	Inferior Parietal Lobule	Brodmann area 40	3.8373
−36	54	14	Left Cerebrum	Frontal Lobe	Middle Frontal Gyrus	Brodmann area 10	3.4723
Negative Weights = decreasing activation with memory load
38	−84	4	Right Cerebrum	Occipital Lobe	Middle Occipital Gyrus	Brodmann area 19	5.4123
12	−58	14	Right Cerebrum	Limbic Lobe	Posterior Cingulate	Brodmann area 30	4.9131
4	60	16	Right Cerebrum	Frontal Lobe	Medial Frontal Gyrus	Brodmann area 10	4.6876
−44	−66	24	Left Cerebrum	Temporal Lobe	Middle Temporal Gyrus	Brodmann area 39	4.3864
46	−64	10	Right Cerebrum	Occipital Lobe	Middle Temporal Gyrus	Brodmann area 19	4.1379
56	−48	2	Right Cerebrum	Temporal Lobe	Middle Temporal Gyrus	Brodmann area 22	3.9692
−50	−74	10	Left Cerebrum	Temporal Lobe	Middle Temporal Gyrus	Brodmann area 39	3.9004
−20	32	48	Left Cerebrum	Frontal Lobe	Middle Frontal Gyrus	Brodmann area 8	3.8673
−22	−42	−6	Left Cerebrum	Limbic Lobe	Parahippocampal Gyrus	Brodmann area 19	3.822