CNTRICS Final Task Selection: Working Memory

Abstract

The third meeting of the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) was focused on selecting promising measures for each of the cognitive constructs selected in the first CNTRICS meeting. In the domain of working memory, the 2 constructs of interest were goal maintenance and interference control. CNTRICS received 3 task nominations for each of these constructs, and the breakout group for working memory evaluated the degree to which each of these tasks met prespecified criteria. For goal maintenance, the breakout group for working memory recommended the AX-Continuous Performance Task/Dot Pattern Expectancy task for translation for use in clinical trial contexts in schizophrenia research. For interference control, the breakout group recommended the recent probes and operation/symmetry span tasks for translation for use in clinical trials. This article describes the ways in which each of these tasks met the criteria used by the breakout group to recommend tasks for further development.

Working memory function has perhaps been one of the most well studied cognitive domains in schizophrenia, since the seminal work of Park and Holtzman.¹ Furthermore, working memory is a domain that has received enormous attention in cognitive behavioral studies, as well as in both human and animal cognitive neuroscience.^2,3 Thus, there is a large body of extant work that outlines the psychological and neural processes that support working memory. Not surprisingly, working memory is a multidimensional construct, and research has increasingly indicated that it involves a number of different subcomponents.^4–6

In the psychological literature, working memory refers to a cognitive system involved in the maintenance and manipulation information over a short period of time (up to ∼30 s), serving as a temporary workspace (eg, a “mental blackboard”) enabling complex cognitive operations, such as planning, problem solving, and reasoning. Moreover, the contents of working memory are thought to be “buffered,” such that they are protected from interference due to either distracting information or decay over time. The information actively maintained in working memory includes both specific stimuli that will need to be used at a later point in time, as well the task goals that govern the current behavioral set.⁷ In addition, working memory capacity is inextricably tied to the ability to resist intrusion from currently irrelevant information. This idea is supported by demonstrations that proactive interference plays a prominent role in forgetting in working memory.^8–10

Some models of working memory, such as Baddeley's,¹¹ suggest that there are at least 3 major components of working memory; (1) a short-term storage buffer for visual information that is often referred to as the visuospatial scratch pad; (2) a short-term storage buffer for verbal information referred to as the phonological loop (which includes both articulatory rehearsal and phonological processing; (3) a central executive component that guides the manipulation and transformation of information held within the storage buffers. In recent years, the model has also included an episodic buffer, in which complex, multimodal events are integrated and stored online.⁵ In a model such as Baddley's, maintenance of specific information is governed by the buffer systems, while the regulation and coordination of this information—ie, updating and maintenance of task goals, management of interference, and manipulation and transformations of stored content—are handled by the central executive. Other models of working memory, such as the one put forth by Cowan, suggests that there are not qualitative or structural differences in the representations used to support working memory as compared to episodic memory, in the sense of there being dedicated storage buffers that only maintain information “contained” in working memory. Instead, Cowan's model suggests that the information contained in working memory is simply the activated portion of long-term memory that is currently in the focus of attention.⁶

Numerous studies have examined the integrity of different aspects of working memory in schizophrenia. There have been repeated demonstrations that individuals with schizophrenia exhibit deficits on a wide range of working memory tasks and that these deficits are associated with impairments in the function of neural circuits that support working memory function.^12–15 Some research even suggests that the degree of impairment in certain aspects of working memory predicts later onset of schizophrenia.^16,17 Further, the severity of working memory impairment predicts the degree of social and occupational impairment in individuals with schizophrenia.^18,19 In addition, there is evidence to suggest that working memory deficits are endophenotypic markers of risk for schizophrenia because individuals who share unexpressed genetic components of vulnerability to schizophrenia also experience impairments in working memory function.^20,21

Given the research outlined above, both the MATRICS and the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) initiatives included working memory as one of the core cognitive domains to focus upon in work geared toward developing more effective treatments for enhancing cognitive function in schizophrenia. At the first CNTRICS meeting, it was decided that 2 components of working memory were ready for immediate translation for development and use in clinical trials in schizophrenia: (1) goal maintenance and (2) interference control.

Goal maintenance was defined as: “The processes involved in activating task related goals or rules based on endogenous or exogenous cues, actively representing them in a highly accessible form, and maintaining this information over an interval during which that information is needed to bias and constrain attention and response selection.”²²

Interference control was defined as “The processes involved in protecting the contents of working memory from interference from either other competing internal representations or external stimuli.”²²

Much of the extant literature points to the greatest working memory deficits in schizophrenia falling in the domains of interference control and goal maintenance. Deficits are present in both medicated and unmedicated individuals and at both acute and chronic stages of the illness.^23–28 In addition, goal maintenance impairments are found in the first-degree relatives of individuals with schizophrenia,^29,30 as well as in individuals with schizotypal personality disorder.^31,32 There is some data that suggest that working memory maintenance is impaired in schizophrenia, even in the absence of interference,^14,33 but the effects tend to be weaker and less robust than that observed in high interference situations. For example, the seminal work of Oltmanns and Neale demonstrated a differential deficit in working memory span in the face of distraction as compared with no distraction,³⁴ a finding that has been replicated by a number of researchers.^35,36 In more recent work, Brahmbhatt and colleagues have shown heightened sensitivity to proactive interference among individuals with schizophrenia in the context of an n-back working memory task; a number of other studies have suggested impaired distractibility in the context of working memory tasks in schizophrenia.^37,38

As described in the other articles in this special issue, the goal of the third CNTRICS meeting was to solicit and evaluate nominations for promising tasks that would measure each of these 2 constructs. Nominations were solicited from both basic and clinical scientists, from both academia and industry. The nominators were asked to provide a description of the task and to provide information on five domains relevant to selecting the most promising tasks: (1) cognitive construct validity, (2) neural construct validity, (3) reliability, (4) other psychometric characteristics, and (5) the availability of animal models. The overview article at the beginning of this special section outlines why these criteria were selected and describes how they were applied during the group discussions. Here, we briefly review the ways in which each of these tasks met the criteria used for selection.

Goal Maintenance

CNTRICS received 3 initial nominations for the goal maintenance component of working memory: (1) the AX-Continuous Performance Task (CPT)/Dot Pattern Expectancy (DPX) task, (2) probabilistic reversal learning, and (3) the operation span/symmetry span. The working memory discussion group decided that the AX-CPT/DPX task would be recommended for further development as a measure of goal maintenance in working memory but that the other 2 tasks would not. The reasons for this are described next.

Tasks Not Selected for Translation as Measures of Goal Maintenance

In a probabilistic reversal learning task, subjects are presented with a set of stimuli concurrently (typically 2 or 3). The subject must choose one of the stimuli, and the choice is followed either by positive or negative feedback. The subject must figure out what characteristic of the stimuli (either a specific stimulus or a feature or dimension of a stimulus) is associated with positive vs negative feedback. After a certain number of correct choices, the stimulus or feature that is positively rewarded is changed, and the subject must learn to update the rule upon which stimulus selection is based. The task is called probabilistic when the “correct” choice is positive rewarded some of the time but not all the time (eg, 80% of correct choices are positively rewarded). In a deterministic version of the task, all correct choices are positively rewarded. This task clearly assesses the ability of a participant to learn the appropriate rule, use it to guide stimulus selection, and update the rule as the contingencies change. However, the working memory breakout group felt that this task did not specifically assess the ability to maintain a task goal in working memory. For example, the face validity of the task for this domain is low because published studies rarely refer to working memory maintenance in explanations of findings but rather inhibitory or reward-learning processes. In addition, the group felt that poor performance on this task could reflect a number of different factors (overgeneralization in reinforcement learning, inability to learn from positive or negative feedback, failure to update goals, inability to switch sets, etc), making it difficulty to interpret poor performance in terms of specific difficulties in goal maintenance.

The Operation Span/Symmetry Span tasks are variations on a set of “span” tasks that require subjects to maintain information (eg, a word) while they perform other computations (eg, mental arithmetic) that could interfere with the maintenance of information in working memory (for a comprehensive User's Guide, please see Conway et al³⁹). The breakout group felt that these types of span tasks were very important measures of some components of working memory function but felt that goal maintenance per se was not the critical determinant of performance on this task. However, the group did feel that these tasks were excellent measures of interference control (as described below) and recommended that they be considered for this construct rather than for goal maintenance.

Task Recommended for Translation as a Measure of Goal Maintenance AX-CPT/DPX Task

Description

The version of the AX-CPT developed by Cohen and colleagues (which we will refer to as the expectancy AX paradigm) was designed to measure the processing of context information, which was defined as “information that has to be held actively in mind in such a form that it can be used to mediate an appropriate behavioral response.”^40,41 Thus, the construct closely overlaps the conceptualization of goal maintenance. This version was a modification of the original AX-CPT.⁴² The original adaptation of the AX-CPT used a combination of Latin letters as cues and probes following the established A-then-X rule. More specifically, the rule for the AX-CPT is that “X” is the target when directly preceded by the letter “A” (AX trials), whereas a letter in any other sequence is a nontarget (eg, AY, BX, and BY trials). In this case, a B is any non-A cue, whereas a Y is any non-X probe. Different ratios of trial sequences have been used, including 80% AX/10% AY/10% BX,^41,43 70% AX/10% AY/10% BX/10% BY,^31,27,44 and 79% AX/8% AY/8% BX/5% BY.²⁹ Delay lengths between the cue and probe have also been varied from no delay to 10 seconds or longer.⁴⁵ The AX and BX trials both require the representation and maintenance of the cue or the goal of the task to respond appropriately. BX trials tend to be the most sensitive to impairments in the ability to maintain the goal representation (for an exception see Umbricht et al⁴⁶). That is, maintenance of the B cue is required to overcome the prepotent target response to the X on the trials (relatively infrequent) when it does not follow an A. The most difficult trial type for people with strong goal maintenance is often the AY condition. This is because goal-focused maintenance of the A cue should produce a high expectancy of a subsequent X probe and preparation of a target response, resulting in increased errors or slowing on AY trials. It has also been suggested that changing the delay interval from block to block detracts some from the prepotency manipulation, leading most current studies we are aware of to switch from a long to a short delay (and vice versa) only once during the experiment. Although the task can take up to 45 minutes when short and long delays are used, block lengths are kept purposefully short (from 10–40 trials per block). Thus, the task is not a CPT in the sense of demanding uninterrupted, sustained attention.

The DPX task is formally equivalent to the adaptation of the letter-based AX-CPT⁴⁷ but employs underlearned dot patterns rather than familiar letters (and a slightly different proportion of trials). The stimuli used in this variant, illustrated in figure 1A, were designed to provide a goal maintenance task with desirable characteristics for both experimental testing batteries and clinical applications in the following manner: (1) underlearned dot patterns increase the challenge of cue maintenance even over short-delay intervals; (2) higher proportions of AY and BX trials (12.5%) provide greater estimation power for the critical trial types; (3) dot-based configurations increase the potential for manipulating stimulus similarity, leading to increased AY difficulty for healthy participants. These characteristics of the DPX allow for shorter interstimulus intervals, faster accumulation of errors on critical trials, and an increased likelihood of an interpretable differential deficit. However, experimenters must pay close attention to participant behavior during training. Participants who are confused about the stimuli will make more errors on all types of trials (including BY trials, which are generally quite easy). This is hard to mistake as a specific deficit, and such participants may be removed from analyses based on their overall or BY error rates. Also, some participants may be overly focused on the fact that the “X” configuration of dots usually marks a target and may adopt a strategy of ignoring valid and invalid cues. Such participants will make close to 100% BX errors but have adequate performance in other conditions. Experimenters need to ensure that participants answer correctly on AY, BX, and BY trial types during training to confirm they understand all nontarget conditions.

Fig. 1.

A. Stimuli for Dot Pattern Expectancy (DPX) task. B. Confirmation factor analysis of expectancy AX and DPX trial types⁴⁷ (adapted with permission from the American Psychological Association). C. Increased activity in left dorsolateral prefrontal cortex (BA 9, x = −41, y = 18, and z = 28) following a cue to prepare to use a controlled (red) or automatic (green) response when responding to a Stroop stimulus (i)¹⁶⁵ or following a “B” or “A” cue in the AX task in healthy volunteers (ii) or patients with schizophrenia (iii)⁵³ (adapted with permission from the American Psychological Association). D. Patterns of impairments in patients on (i) the AX⁶⁶ and (ii) DPX tasks.

Construct Validity

The construct validity of the expectancy AX and DPX have been explored using formal connectionist modeling and studies of the convergent and divergent validity of the tasks.⁴⁷ Computational models that express a prepotent response and allow for the uploading of context information (goal information) such as A and B cues are capable of producing the patterns of responses observed in healthy participants and patients with schizophrenia.^48,49 Correlations have been observed between expectancy AX-CPT errors and an inability to selectively focus attention or maintain the gist of the previous sentence on paradigms that also require active goal or context maintenance.⁵⁰ As illustrated in figure 1B, confirmatory factor analyses have shown that the AX and BX conditions of both the AX and DPX measure a common latent construct, here called “context processing.” The AY condition measured a distinct latent factor perhaps reflecting “response preparation,” which was somewhat correlated with AX performance, too.

Neural Systems

The expectancy AX-CPT, but not the DPX, has been used in a number of imaging studies.^51–57 Contrasts within these studies generally implicate dorsolateral prefrontal cortex (DLPFC) (left, right, or both), bilateral anterior cingulate, and inferior parietal cortex (either left, right or both) as more activated when the cue must be maintained over a delay and when the “B” cue must be represented to overcome the prepotent target response. Figure 1C provides an illustrative example. In this study, a region of left DLPFC had been previously associated with preparing to attend to the nondominant aspect of a stimulus (the color of a Stroop stimulus, Figure 1Ci). This same region was found to be activated in an independent sample of healthy volunteers following the “B” cue compared with the A cue (Figure 1Cii). Patients with schizophrenia show no distinction between the B and A cues in this region (Figure 1Ciii). Furthermore, within the healthy participants, increased activity in this region was generally associated with better task performance. In recent work using intracranial recordings in monkeys, a similar result was found showing enhanced DLPFC activity to “B” cues as compared with “A” cues.⁵⁸

Pharmacological or Behavioral Manipulation of Task Performance

The N-methyl D-alanine antagonist ketamine has been found to increase AX and BX errors in healthy volunteers in a pattern reminiscent of that seen in patients with schizophrenia.⁴⁶ The administration of low-dose amphetamine (an indirect D1 agonist) can improve the ability to maintain goal information (A or B cues) in the face of distraction among healthy controls,⁵⁹ a result that is consistent with the work of Goldman-Rakic et al⁶⁰ suggesting that modulation of dopamine can influence working memory performance. Val/val homozygotes at the COMT val158met genotype, who are believed to have reduced availability of prefrontal dopamine, have also been found to have selective goal maintenance deficits on the DPX task.⁶¹

Animal Models

Killcross and colleagues have developed a biconditional choice task for use in rats that they view as an analogue of the expectancy AX-CPT/DPX.^62–64 This task does measure contextually influenced choices. However, it is not clear whether it assesses the maintenance of goal representations in working memory because the rats may be able to learn the task as a set of contextually bound stimuli that are stored as long-term memory representations. Javitt and colleagues have developed a version of the AX-CPT for use in macaques, in which the cues are symbols such as circles and squares and the targets are colored letters.⁵⁸ Like humans, monkeys made more errors on AY and BX trials than on AX and BY trials. However, healthy monkeys (modulo the fact that they had surgery to implant electrodes in DLPFC and ACC) did make slightly more BX than AY errors. As noted above, intracranial recordings suggest that analogous neural systems are activated by the task in monkeys and humans.

To facilitate the development of parallel task versions in humans and animals, recent work has also explored spatial variants of the AX-CPT/DPX. For example, pilot data from the TRiCAM laboratory and Minneapolis Veterans Affairs have found that a version of the task in which cues and probes are specified in terms of spatial locations relative to a landmark produce the same pattern of increased AY relative to BX errors observed in healthy controls as that observed in the expectancy AX and DPX tasks. This presentation modality was chosen because the capacity to move the landmark allows the dissociation between perceptual processes and the representation and maintenance of the cue in humans, nonhuman primates, and rats. Also, the similarity of valid and invalid cues and probes can be parametrically manipulated.

Performance in Schizophrenia

Servan-Schreiber, Cohen, and colleagues reported that, compared with patient and healthy controls, patients with schizophrenia made more errors on AX and BX trials following a brief delay, whereas AY errors went down.^41,50 Furthermore, as noted above, schizophrenia patients’ errors on BX trials after a delay tended to be correlated with the inability to selectively focus attention or maintain the gist of the previous sentence.⁵⁰

These results have been replicated to a varying degree and extended in several studies^29,43,65,66 (for an overview, see MacDonald⁴⁰). This work has suggested that the deficit is specific to schizophrenia as compared with psychosis, as well. As illustrated in Figure 1Di, Barch et al⁶⁶ investigated performance on the AX task by schizophrenia patients and a psychotic control group at first admission and 4 weeks later. While both groups showed impairments in context processing at baseline, these impairments generally remitted in psychotic controls but not in patients eventually diagnosed with schizophrenia.⁶⁶

In the first study using the DPX, schizophrenia patients and their nonpsychotic relatives showed a pattern of errors similar to that observed in the AX task (MacDonald, Goghari, et al, 2005). While control participants found AY trials to be the most difficult on this task, patients and their relatives were much less impaired on this condition relative to the BX condition, which was quite easy for control participants. Jones et al have replicated this pattern of performance in a recent sample of schizophrenia patients (figure 1Di; J. H. Jones, A. MacDonald, S. Sponheim, 2008). Again, patients showed a specific impairment on BX trials, relative to AY trials, while healthy participants made more errors on AY trials, relative to other conditions. These patterns of deficits are similar to those found by Servan-Schreiber, Cohen, and colleagues initially and support the convergent models of the validity of the task.⁴⁷

Psychometric Data

The retest reliability of the expectancy AX has been examined in an sample of schizophrenia patients, psychotic nonschizophrenia patients, and healthy controls using the sampling and follow-up method and task variant described elsewhere.⁶⁶ Intraclass correlation (ICC) coefficients for absolute agreement with time point entered as a fixed effect and participant as a random effect were calculated for a single point in time. For d’ context over the interval of 6 weeks postadmission to 6 months (total sample n = 131), the ICCs were 0.78 for the short delay, 0.73 for the long delay, and 0.81 for the 2 delays combined. When considering schizophrenia patients alone (n = 45), the corresponding ICCs over this interval were 0.81, 0.70, and 0.82. The ICCs for particular conditions were lower, with AX and BX ICCs close to 0.70 and for AY and BY 0.60 and 0.56, respectively. The practice of removing subjects who make more than 60% up to 90% errors systematically reduced ICCs of the total sample. The internal consistency of the expectancy AX was examined in a separate sample of patients, first-degree relatives, and healthy controls (total sample n = 63).^29,67 In this case, alpha (or KR-20) coefficients were .80 for the AX and .71 for the AY conditions, with similar coefficients irrespective of delay. For the BX, the alpha coefficient was .63 for a no-delay condition and .90 for a delay condition. BY internal consistencies were not reported. The DPX appears to have a similar level of internal consistency to the AX. Among a collapsed sample of schizophrenia patients and controls (n = 53), the internal consistency of the AX and BX conditions was greater than 0.85, whereas the AY and BY conditions were 0.56 and 0.58, respectively (Jones et al, 2008). In a version of the DPX task designed to take fewer than 10 minutes and consisting of only 44 AX trials, 15 AY, 15 BX, and 6 BY trials these coefficients were 0.90, 0.57, .84 and 0.34, respectively. Overall, the AX and BX trials appear to be the most reliably tapped components, whereas the AY condition is somewhat less so. The BY condition, which is included as a manipulation check, may not be as reliable because of the low range of errors and should probably only be used to determine whether a participant understood the task.

Future Directions

The AX paradigm has proved a useful method for testing individual differences in the capacity to use goal-related context representations to overcome automatic response tendencies. Ongoing work is directed toward making a variant of this classic paradigm efficient for use in clinical testing batteries. To this end, decreased task duration and an increase in the reliability of the AY and BX conditions will be important. The AY error rate in healthy controls provides important leverage for interpreting a differential deficit in the BX condition, which measures that aspect of goal maintenance related to the processing of context. In addition, further work is needed to develop a variant of the task that can be used in animal studies.

Methods for analyzing the AX and DPX paradigms continue to develop. The studies described herein have all used error rates, reaction times, and classical statistical approaches. Nonparametric approaches, such as the use of mixed-model logistic regression on trial-level data, might prove to be more sensitive to individual differences. Additionally, item-response theory and Rasch models may be appropriate in this context and, in conjunction with adaptive testing, may provide further efficiencies for the purpose of clinical testing.

Interference Control

CNTRICS received 3 initial nominations for the interference control in working memory construct: (1) the inhibition of currently irrelevant memories task, (2) the recent probes task, and (3) the ignore-suppress task. In addition, as noted above, the working memory breakout group felt that the Operation/Symmetry Span tasks nominated for the goal maintenance construct were potentially excellent measures of interference control in working memory. After extensive discussion, the group felt that the recent probes task and the Operation/Symmetry Span tasks should be recommended for translation as measures of interference control in working memory.

Tasks Not Selected for Translation as Measures of Interference Control

The inhibition of currently irrelevant memories is one in which subjects are shown a series of pictures (52 in total). In each run, some pictures are repeated.^68–72 In the first run, subjects are told to identify pictures that are repeated. In subsequent runs, they are instructed to only identify pictures that repeat in the current run and not pictures that were repeated in prior runs. Thus, this task measures the ability to control interference from previously presented information. The breakout group felt that this was a very interesting task but that it measured a construct different than the one of focus. More specifically, the group felt that it was a promising measure of interference control in episodic memory because the runs of pictures were fairly long and the interference effects could arise from stimuli presented 10–15 minutes previously. Such interference control mechanisms in episodic memory may rely on different mechanisms than those that contribute to interference control in working memory because the nature of the memory trace that creates the interference may be different for recently presented information (eg, working memory effects) vs more distally presented information (eg, episodic memory effects). In addition, the Inhibition of Currently Irrelevant Memories task also seems to require an episodic source discrimination process, which again may tap different processes than required for interference control in working memory.

The Ignore-Suppress Test is a variant on a classic Sternberg Item Memory Test of working memory in which subjects are shown a memory set and then tested on a series of probes and asked whether the probe was in the current memory set.⁷³ On each trial, subjects are presented with a set of words, half in one color and half in another color. In the ignore condition, the subjects are presented with a cue at the beginning of the trial that tells them to ignore words in one color but not the other. In the suppress condition, subjects are presented with a cue after the memory set was presented that tells them to forget words presented in one color or another. Subjects are then presented with a series of probes. Some of the probes are targets in the current memory set, and some are nontargets that were not part of the current memory set and not one of the to be ignored or forgotten words. However, some of the probes are the “to be ignored” or “to be forgotten” words, and these allow the experimenter to assess the ability of the participant to control interference from to be ignored/forgotten items. The breakout group was very excited about this task and felt that it was a highly promising direction for research on interference control in working memory. The group particularly liked the fact that one could assess both proactive (eg, ignore condition) and “reactive” (eg, suppress condition) interference control in the same paradigm. However, there was only a single published study on the task, with (as of yet) little information on many of the criteria critical to task nomination. Thus, although the breakout group felt that this task had the potential to be a very important instrument in future research, they felt it needed more basic science research before it was ready for translation into use in clinical trials in schizophrenia.

Tasks Nominated for Translation as Measures of Interference Control Recent Probes Task

Description

The recent probes task is a variant of the item recognition task introduced by Sternberg.^74,75 In this task, participants are presented with a small set of items (eg, words or letters) and are instructed to commit the items to memory. After a brief retention interval, participants are shown a recognition probe and respond whether the probe was a member of the set or not. The manipulation of interest involves trials in which the probe does not match any member of the current target set (hence requiring a negative response) but does match a member of the memory set on the previous trial (hereafter referred to as a recent negative probe). Participants are slower to reject recent negative probes compared with novel probes that have not appeared recently (probes that also require a negative response hereafter referred to as nonrecent negative probes). This delay in responding is thought to be due to the control processes that resolve conflicting contextual information in working memory induced by the high familiarity of recent negatives.⁷⁶ These 2 negative probe types (recent and nonrecent) are the trials of interest in this paradigm and are portrayed in figures 2 and 3. The extra time taken to reject recent negative probes is typically 50–100 milliseconds more than nonrecent probes. This cost is highly reliable in both response time (RT) and accuracy (ACC) but is typically more robust in RT due to high accuracy overall in this paradigm. Reduced costs reflect better control over memory and increased costs reflect poorer control over memory and greater susceptibility to proactive interference.

Fig. 2.

A Recent Negative Trial From the Recent-Probes Task. This figure illustrates a seqeuence to 2 trials that lead the second trial to be a recent negative trial. As can be seen in the figure, the probe on the second trial “golf” was a member of the memory set on the preceeding trial but not a member of the memory set on the current trial.

Fig. 3.

A Nonrecent Negative Trial From the Recent-Probes Task. This figure illustrates a seqeuence to 2 trials that lead the second trial to be a nonrecent negative trial. As can be seen in the figure, the probe on the second trial “lion” was not a member of the memory set on the current trial nor on the preceding trial.

Construct Validity

The construct validity of the recent probes task has been examined by determining the relationship between performance on this task and other measures of interference control. For example, Persson et al⁷⁷ recently demonstrated both positive and negative transfer between performance on the recent probes task and performance on 2 other cognitive tasks involving interference resolution. In addition, construct validity has been explored by examining how different behavioral manipulations affect task performance. Recent research has shown that having participants perform articulatory suppression during the retention interval (ie, counting “1-2-3” repeatedly) increases the recent probes effect by 70 milliseconds in response time and 11% in accuracy (A.S. Atkins, M.G. Berman, J. Jonides, P.A. Reuter-Lorenz, 2008). There is also preliminary data that suggest that when the materials are valenced (eg, positive and negative words) people with high scores on a depression inventory show reliably larger recent probe effects compared with age-matched controls.

In addition, Berman et al (M.G. Berman, J. Jonides, R.L. Lewis, unpublished data) have shown that the recent probes effect can be reliably decreased by having recent negative probes be members of a different semantic category than the relevant stimuli (eg, a recent negative probe that is a fruit, while the relevant stimuli are countries). This decrease is on the order of 45 milliseconds. In addition, the recent probes effect can be reliably decreased by 55 milliseconds if recent negative probes are taken from the 2-back set (eg, a trail occurring 2 trials previously) rather than the 1-back set (eg, the immediately preceding trial). Hence, intervening trials seem to interfere with past item memory traces, thus making them less familiar and less interfering in their own right (M.G.Berman, J. Jonides, R.L. Lewis, 2008). Other researchers have shown that high fluid intelligence (gF) participants can lower their recent probes effect by nearly 100 milliseconds when the proportion of recent negative probes is increased, while low gF participants cannot reduce the recent-probes effect in these high interference situations (G.C. Burgess, T.S. Braver, 2008).

Manipulations that have no bearing on recent probes effect are also found. For example, instructing participants to attempt to ignore past target sets does not alter the effect (M.G.Berman, J. Jonides, R.L. Lewis, unpublished data). Additionally, changing the duration of the intertrial interval has no impact on the effect (ie, the interference effect does not decay passively [M.G.Berman, J. Jonides, R.L. Lewis, unpublished data]). Performance of the task with words vs letters does not impact effect sizes. Limited work with visual (rather than verbal) materials has produced mixed results, often demonstrating varied recent probes effects both behaviorally and neurally.^82–85 Lastly, most studies use memory loads of 4 or 6 items, but there has not yet been a systematic study of how varying memory load affects the recent probes contrast.

Neural Systems

The recent probes task has also been explored quite extensively with neuroimaging techniques to investigate the neural mechanisms that underlie task performance.^{73,82,84,86–88} These studies have converged on the left inferior frontal gyrus (LIFG), most prominently in pars triangularis (BA 45), as the area critical for interference control in the recent probes task. The results from functional neuroimaging studies are bolstered by demonstrations that damage to LIFG greatly enhances the recent probes effect, while leaving other working memory functions relatively spared.⁸⁹ In addition, older adults, who show hypoactivity in this region compared with younger adults, show larger interference effects as well.⁹⁰ Therefore, it appears that LIFG plays a crucial role in interference control in the recent probes task.

Several theories have been proposed regarding the function of LIFG in controlling proactive interference. The most recent of these theories posits that LIFG is involved in context selection. By this account, when faced with highly familiar recent negative probes, LIFG functions to select the context in which the probe had been encountered (eg, the item was on the last trial not the current trial) in order to respond negatively.⁷³ This idea is corroborated by evidence that LIFG activation correlates with medial temporal cortex activation when retrieval demands increase by demands such as proactive interference (D.E. Nee, J. Jonides, 2008).⁷³ Hence, the medial temporal cortex, which is involved in episodic memory⁹² and contextual binding,¹⁰ may provide contextual information in order to arrive at correct recognition decisions in the face of interference.

Other neural regions have been found active in this task such as left anterior prefrontal cortex,^73,84 whose function may be to monitor retrieved information.^73,84 However, regions other than LIFG have been reported inconsistently in the recent probes task, and so these results need to be taken with some caution.

While there are no computational models that have simulated performance in the recent probes task, the biased-competition model has provided a conceptual model of recent probes performance.^76,93–95 According to this model, when a recent negative probe is presented it activates attributes/features that are associated with positive probes. These features include familiarity (which is high), its context (seen on the previous trial), its semantic representation, etc. Because high familiarity is generally associated with positive responses, the familiarity of the recent negative probe will bias response decision processes toward a positive response. At the same time, the context of the recent negative probe does not match that of the current item's context, and this contextual mismatch will produce a negative response tendency, which is the correct response. This contextual information must be favored/selected to arrive at the correct response, and this is the function of LIFG. The resolution of these competing tendencies slow participants compared with nonrecent negative probes that have very low item familiarity and thus little competition.⁷⁶

Pharmacological or Behavioral Manipulation of Task Performance

The recent probes task has not yet been studied with either pharmacological or behavioral manipulations (other than practice⁷⁷) designed to modify performance.

Animal Models

There are no animal models specifically of the recent probe task.

Performance in Schizophrenia

The recent probes task in its original form has not yet been studied in individuals with schizophrenia. However, an analogous version of an N-Back working memory task that included repeated lures (conceptually similar to recent negatives) did reveal enhanced false alarms to familiar nontarget items.³⁷ Such a result is consistent with the idea that individuals with schizophrenia demonstrated deficits in interference control in working memory and that they would show deficits on the recent probes task.

Psychometric Data

The reliability of the recent probes effect, ie, the difference in reaction time between recent negative and nonrecent negative probes, has been examined in recent unpublished work by Nee and Jonides. In this work, reliability was calculated in 2 ways for 10 studies that utilized the recent probes task from the Jonides laboratory. First split-half reliabilities were calculated by correlating the average recent probes effect for odd and even trials. The average split half reliability was 0.44 with a range of 0.16–0.68. Second quasi test-retest reliabilities were calculated by correlating first half recent probes effects with second half effects. The average test-retest reliability was 0.42 with a range of 0.1–0.69. These reliabilities may be somewhat conservative given that the recent probes effects were highly significant for each of these studies individually and given that the variance in effect size may be quite small, yielding a restricted range. However, even though these reliabilities are typical for a cognitive experimental task, they are still well below the value expected for tasks used in clinical trials. In addition, current versions of the recent probes task can show both practice effects (across sessions) and fatigue effects (within sessions)⁷⁷.

Future Directions

There are several clear future directions for the further development of this task. First and foremost, this task needs to be studied in patients with schizophrenia to confirm that it elicits the expected pattern of deficits. Second, the degree to which such deficits can be attributed to a specific deficit in interference resolution vs a generalized deficit will need to be examined, given that performance in the recent negative condition is typically more variable than performance in the nonrecent negatives condition. This opens the possibility that the recent negatives condition has greater discriminating power than the nonrecent negatives condition. Third, psychometric development work is needed to enhance the reliability of the task and to determine more precisely the magnitude of practice effects. For example, future research may need to explore the number of trials needed to provide adequate reliability and may need to explore the impact of variations in the frequency of recent negative on reliability and the magnitude of the interference effect. Fourth, future research will need to examine whether interference control in the recent probes task can be modified either pharmacologically or behaviorally to determine whether it will be a feasible target for cognitive enhancement in schizophrenia.

Operation/Symmetry Span Tasks

Description

The operation span and the symmetry span have the same “deep structure” as all complex span tasks. Subjects are presented with trials in which they must “process” some type of information while they “maintain” some type of information in working memory. The amount of information that must be maintained in working memory can be manipulated across trials.⁹⁶ The first complex span task was the reading span, in which subjects had to read sentences and remember the sentence final word. The operation span is similar but replaces the sentences with a set of mathematical operations. Thus, on each trial, a mathematical operation is presented, followed by a word to be remembered (eg, (2/1) + 1 = 3 ? BEAR). When each trial is presented, participants must read it aloud, judge the correctness of the mathematical operation, and then memorize the final word for later recall. In the original procedure, trials progress from smaller (2–3 operation + word strings) to larger sets (4–6 strings), with a participants’ span being scored as the highest set that can be performed correctly. Some variants use a procedure in which the number of trials per set is pseudorandomly varied rather than ascending, and others use trial-based rather than set-based scoring to determine span.⁹⁷

In the symmetry span, participants must recall sequences of red squares presented in a matrix while performing a symmetry judgment task. The symmetry judgment task requires subjects to decide whether a 8 × 8 matrix with some squares filled in black is symmetrical about its vertical access.

Numerous studies have established the construct validity of traditional versions of span tasks such as the operation span and the symmetry span. However, participant-paced versions of complex span tasks result in lower or nonsignificant correlations with criterion measures as compared with experimenter-paced versions that constrain the maximum amount of time the participant is allowed to work on the secondary task.^98,99 To combat this problem, Engle and colleagues developed automated versions of operation¹⁰⁰ and symmetry span. In the automated version of the operation span, participants remember letters rather than words (see figure 4). Individual differences in working memory capacity (WMC), as measured by performance on operation and symmetry span (see figure 4), reflect primarily the ability to maintain and retrieve information in interference-rich situations,¹⁰¹ as well as the ability to override prepotent but contextually inappropriate responses in favor of the behavior necessary for accomplishing a specific goal.¹⁰² Thus, the operation span and the symmetry span tasks were designed with several goals in mind. First, each participant completes math operations/symmetry judgments in isolation (ie, a single task condition). The maximum time allowed to solve the operations/judgments in the dual task condition is set to equal 2.5 SDs above each individual's mean from this single-task condition. This controls for individual differences in performance of the secondary task because partialing outprocessing time does not mediate the relationship between operation span and intelligence.¹⁰⁰ Scores are automatically calculated immediately upon finishing the task. As described above, there are multiple scoring methods in use; these are thoroughly discussed elsewhere.^103,104

Fig. 4.

Screenshots of the Automated Operation Span (Above) and Symmetry Span (Below) Tasks. The numbers in the right-hand corner (not visible to the participants) indicate the order in which the screens are presented on a given trial. The first 3 panels illustrate the sequence of events on a single trial, and the last panel indicates the memory test that participants are given at the end of a set of trials. For the operation span, participants must remember the letter presented at the end of each individual trial across the remaining trials in that set until they are given the memory test. Similarly, for the symmetry span, participants must remember the location of the red block in the grid presented at the end of each individual trial across the remaining trials in the set until they are given the memory test.

Construct Validity

Performance on operation and symmetry span are predictive of performance in a number of situations requiring selective attention and interference control.¹⁰⁵ Often, researchers administer operation and symmetry span to a large sample and then compare how the individuals who score below the 25th percentile (low spans) and above the 75th percentile (high spans) perform on other tasks. For example, compared with high spans, low spans: (a) make more incongruent trial errors and have a larger interference effect on the Stroop task^106,107; (b) make more errors on antisaccade trials^108,109; and (c) show more interference on incompatible trials in various versions of the flanker task.^110,111 In addition, Miyake and colleagues^112,113 demonstrated that complex span tasks were positively related to separate latent variables representing updating and resistance to proactive interference. On the basis of these studies and others, all conducted with healthy young adults, performance on operation and symmetry span reflect an ability to select a goal-related representation to successfully perform a task but only in the conditions where interference is present. As evidence that the tasks show discriminant and predictive validity, high and low spans do not differ on congruent Stroop trials, prosaccade trials, or compatible flanker trials. However, it must be noted that all these conditions are likely to have lower discriminating power than the incongruent Stroop, antisaccade, and incompatible flanker trials.

In addition, numerous large-sample, latent-variable studies have examined the validity of complex span tasks.¹¹⁴ These studies showed that complex span measures: (a) exhibit construct validity by strongly correlating with each other^115–117; (b) display criterion-related validity by consistently accounting for variance in fluid intelligence and reading comprehension^118–120; (c) possess psychometrically desirable internal consistency as measured by Cronbach's alpha^{115–117,121}; (d) have satisfactory reliability as assessed by test-retest correlations¹²¹; and (e) show a developmental trajectory, as complex span scores increase throughout childhood¹²² and are decreased in older relative to young adults.¹²³

Importantly, a recent study¹²⁴ found that an incentive manipulation did not change an individual's relative complex span score. High and low spans’ performance determined the bonus money they received as compensation. Although scores improved in the incentive session, the relative ordering of the participants’ scores was unchanged, indicating that motivation cannot account for the individual differences observed on complex span tasks.

Laboratories at Georgia Tech, University of North Carolina-Greensboro, and University of Georgia have administered automated operation and symmetry span to nearly 4000 young adults, finding that they are strongly correlated, r₃₈₅₈ = 0.54. Independently, Salthouse and Pink¹²⁵ reported a correlation of 0.52 in a sample of 754 individuals from 18 to 98 years of age. In addition, the automated versions predicted fluid intelligence in separate, large-sample studies.^100,125

Neural Systems

Two separate neuroimaging studies^126,127 found increased activity in left and right DLPFC in complex span tasks relative to the single tasks. A series of neuroimaging studies^128–131 examined individual differences in the performance of verbal and spatial complex span tasks. The main conclusion was that high spans show greater activity in dlPFC and anterior cingulate cortex (ACC) than low spans, and the amount of functional connectivity between these areas is also significantly greater in the high spans. This result corroborates research indicating that connectivity between the dlPFC and ACC is important for performance on the Stroop task.¹³² Raz et al¹³³ used structural magnetic resonance imaging to account for anatomical variation in complex span tasks. They found that behavioral performance was significantly positively correlated with PFC, limbic cortex, and inferior parietal lobule volumes.

D'Esposito et al⁸⁶have focused on neurotransmitter systems that may account for performance on complex span tasks. A recent study¹³⁴ observed that high spans produce more dopamine (DA) in the striatum than do low spans. Previous research¹³⁵ manipulated the amount of DA available by administering either a placebo or a DA agonist (bromocriptine) and comparing the performance of individuals across sessions on several cognitive tasks. The results showed that high spans performed worse in the bromocriptine session, whereas low spans performed better after taking the DA agonist. These results indicate that individual differences in complex span performance are tied to baseline DA levels, and because DA follows an inverted U dose-response¹³⁶ high and low spans show differential effects of altering the amount of DA available.

Pharmacological or Behavioral Manipulation of Task Performance

To our knowledge, complex span tasks such as the operation or symmetry span have not been used as dependent measures in pharmacological or behavioral intervention studies. However, as noted above, performance on complex span tasks have been used to predict individual differences in the response to manipulations of DA levels.¹³⁵

Animal Models

Kolata et al^137,138 have developed a task paradigm that is designed to tap the same type of interference control in working memory assessed by complex span tasks such as operation span and symmetry span. Kolata et al^137,138 trained mice to performance two 8-arm radial mazes. After stable performance levels had been reached on each maze, Kolata et al^137,138 have the mice perform trials of one maze interleaved with performance on trials of the other maze, putatively requiring them to inhibit interference from one maze while performing the other maze. Nevertheless, because there is no specific retention of items across the interleaved sets, it is not clear whether the task places the same demands on storage as the span tasks.

Performance in Schizophrenia

All studies using complex span tasks such as operation and symmetry span have shown that individuals with schizophrenia have significantly lower complex span performance than healthy controls,^139–147 whereas the results with simple span tasks such as digit span are much more inconsistent. In addition, individuals with schizophrenia show a strikingly similar pattern of deficits in the Stroop, antisaccade, and AX-CPT tasks as shown by low spans relative to high spans. Children of a parent with schizophrenia show decreased performance on complex span tasks,¹⁴⁸ and this deficit is consistent over several years.¹⁴⁹ Finally, complex span measures are significantly related to formal thought disorder symptoms in patients with schizophrenia.^150,151 Interestingly, Berenbaum et al¹⁵⁰ found that digit span was not related to language disorganization, indicating the benefits of using operation and symmetry span tasks to predict symptom profiles in schizophrenia.

Psychometric Data

Because these complex span tasks have been used almost as widely as many standardized tests, several studies have examined the effect of various administration manipulations upon the tasks’ construct validity.¹⁰⁴ First, most researchers have concluded that the nature of the processing or storage component can be verbal, numerical, or spatial without impacting the ability to predict other cognitive abilities.^97,117,152 Second, manipulating the difficulty of the math task does not affect operation span's predictive utility.¹⁵³ Although the difficult version resulted in lower scores, the rank order of individuals and the predictive validity did not change.

Internal consistency as measured by Cronbach α is .78 and .80 for operation¹⁰⁰ and symmetry span (Unsworth et al, submitted), respectively. The test-retest reliability for operation span is r₇₇ = 0.83¹⁰⁰ and for symmetry span is r₁₃₉ = 0.77 (Unsworth et al, submitted). Finally, Salthouse et al¹⁵⁴ report that automated operation and symmetry span are both significantly negatively correlated with chronological age, as would be expected. Overall, operation and symmetry span possess excellent validity and reliability properties based on several independent, large samples using the same administration and scoring procedures.

One challenge to the use of complex span tasks in clinical trials setting is that these task show large practice effects.¹⁵⁵ Currently, each of these tasks have only one version available, and their large practice effects may preclude their use as dependent measures in clinical trials that involve repeated testing of the same individual.

Future Directions

As noted above, one major limitation to the use of complex span tasks in clinical trials settings is the potential for large practice effects, given that only one version of each is currently available. Thus, it would be important to determine whether psychometrically matched alternative versions of the operation span and symmetry span can be developed so as to facilitate their use as dependent measures in clinical trials. However, it may still be useful to use the operation span and symmetry span as predictors of individual differences in response to various pharmacological manipulations on tasks that can be used repeatedly. If so, we strongly advocate studies using multiple complex span tasks as indicators of interference control in working memory in future research with patients. In addition, as noted above, Kolata, Matzell, and colleagues have created a rat model of working memory,^156–158 but understanding of complex span performance would be greatly aided by more translational research, including developing a nonhuman primate model. Further, we are not aware of any genetics studies that have used either operation or symmetry span, despite strong evidence that working memory capacity is heritable.¹⁵⁹

General Conclusion

The above review was meant to provide the reader with an overview of the data used to select among the different tasks nominated to measure the constructs of goal maintenance and interference control in working memory. As should be apparent in these descriptions, no task is perfect and much research and work remains to be done with each of these tasks. The largest challenge facing the next phase of the CNTRICS process is to translate these tasks into forms that are usable in a clinical trials context. Paradigms derived from cognitive neuroscience have numerous potential advantages that stem from their close ties to modern theories of human and animal neuroscience that might facilitate a more theoretically driven approach to novel treatment development. However, compared with standard neuropsychological measures, these paradigms are often long, sometimes complex, and as of yet unstandardized. The second CNTRICS meeting was entirely devoted to addressing the challenges of this translation process from a practicality and feasibility perspective that takes into account the need to maintain the construct validity of any translated measures. The articles published as part of the special issue that reported on the second CNTRICS meeting outline in detail the nature of these challenges and some potential pathways for solving some of the critical translational hurdles.^160–164 We think that these challenges are surmountable in regard to the tasks described in the current article, several of which are already in the process of being refined for potential use in clinical trials. Thus, the next concrete step is to form teams of clinical and basic scientists that can work together to facilitate the translation process in a way the preserves the strengths of these cognitive neuroscience paradigms (eg, tight grounding in empirical cognitive neuroscience) while making them truly usable tools in the search for more effective means of improving cognition in schizophrenia.

Funding

National Institutes of Health (to D.M.B., S.R.S.); McDonnell Center for Higher Brain Function (to D.M.B.); and National Alliance for Research on Schizophrenia and Depression to (D.M.B.); Veterans Affairs Medical Research Service (to S.R.S.).