Functional neuroimaging and behavioral correlates of capacity decline in visual short-term memory after sleep deprivation

Abstract

Sleep deprivation (SD) impairs short-term memory, but it is unclear whether this is because of reduced storage capacity or processes contributing to appropriate information encoding. We evaluated 30 individuals twice, once after a night of normal sleep and again after 24 h of SD. In each session, we evaluated visual memory capacity by presenting arrays of one to eight colored squares. Additionally, we measured cortical responses to varying visual array sizes without engaging memory. The magnitude of intraparietal sulcus activation and memory capacity after normal sleep were highly correlated. SD elicited a pattern of activation in both tasks, indicating that deficits in visual processing and visual attention accompany and could account for loss of short-term memory capacity. Additionally, a comparison between better and poorer performers showed that preservation of precuneus and temporoparietal junction deactivation with increasing memory load corresponds to less performance decline when one is sleep-deprived.

Air or ship traffic control and long-distance driving, as well as patient status monitoring in intensive care units, are examples where failure to detect, register, and process visual information as a result of sleep deprivation (SD) may have disastrous outcomes. One of the bottlenecks constraining visual information processing is the capacity to capture and briefly retain items in visual short-term memory (VSTM) to detect behaviorally salient events (1).

VSTM capacity is limited to approximately four easily discriminated objects (2, 3). The neural substrate for temporary storage of visual information has been localized to the parietal lobes (4). When VSTM is deliberately engaged, the intraparietal sulcus (IPS) in both hemispheres shows a monotonic increase in activation corresponding to set size until memory capacity is reached, whereupon activation asymptotes. When the same arrays are viewed without having to remember the constituent elements, parietal activation is insensitive to perceptual load. In contrast, activation in the extrastriate visual region is sensitive to perceptual load, increasing monotonically with increasing set size (at least up to a set size of eight). These observations provide dissociable regions to examine the neural substrates of storage capacity and perceptual load.

The response of posterior cortical regions to SD is of interest, because prior experiments involving verbal short-term memory in the context of SD have consistently shown reduced task-related activation in the parietal cortex (5–8). This decline in activation has been shown to reliably correlate with the extent of performance decline in verbal short-term memory after SD. Additionally, we found that of various metrics, response-time variability best correlated with SD-related reduction in posterior parietal activation (9). This suggests that degraded attention rather than loss of storage capacity per se may account for the observed changes in brain activation and performance in “working memory” tasks. This latter view is supported by behavioral studies suggesting that VSTM and attention are largely intertwined cognitive processes (10–12). However, as important as attention is, it is not the sole determinant of VSTM capacity, because interference effects using different tasks suggest the existence of attention and storage-specific subsystems (13), which neuroimaging may be suited to tease apart.

Evaluating the activation in the parietal region in a working memory task alone is of little help in discriminating between loss of storage capacity and degraded attention, because this region is implicated in both functions. We reasoned that, by including a task that evaluates the neural correlates of increased perceptual load without demands on storage, we could examine the contributions of both faculties to performance decline after SD and, in so doing, gain better insight into the neural changes that take place in this state.

We studied the effect of 24 h of SD on VSTM and visual attention in 30 individuals. All participants completed a task of VSTM and a task of visual attention, and memory/perceptual load was parametrically varied within both tasks (Fig. 1). In the VSTM task, participants viewed briefly presented arrays of one to eight colors, retained this array for 1 s, and then determined whether the color of a single probe square had been presented earlier. In the visual attention task, participants were presented with similar arrays but had only to indicate the presence or absence of a square in the center of that array.

Fig. 1.

Schematic of the VSTM and VAC tasks. In the VSTM task, participants were asked to remember the different colors in the memory array that varied in size from one to eight colors. The memory probe appeared 1 s later, and participants indicated whether the color shown had been presented in the earlier array. There were four runs of the VSTM task, and each run consisted of 12 15-s blocks interleaved with 18 s of fixation. There were eight blocks of each set size, and each block consisted of four trials. In the VAC task, participants attended to the center of the array and responded to whether a colored square was present in the center of the array. There were three runs of the VAC task. Each run similarly consisted of 12 15-s blocks interleaved with 18 s of fixation. There were six trials in each block with eight blocks of each set size (1, 2, 3, 4, 6, and 8).

If SD affects VSTM capacity primarily by compromising the storage of visual information, we would expect to observe a reduction in parietal activation at the capacity limit of VSTM but not at subcapacity set sizes. This is because, in a capacity-limited system, performance will be at ceiling (or intact) at subcapacity loads, and failures (incorrect responses) will emerge only at the limit of the system's capacity. Further, in this account, visual processing should not be affected, and we would expect ventral extrastriate activation to keep apace with increasing set size.

Alternatively, if the degradation of attention is the predominant mechanism through which VSTM capacity is reduced after SD, we could expect attenuation of extrastriate cortex and parietal activation at all set sizes, as might be expected if a factor fundamental to task performance was compromised.

Results

Behavioral Findings.

VSTM.

Similar to other studies, we found that short-term memory capacity averaged approximately three colors after a normal night's sleep. Performance on the memory task, as indexed by K (12), was maximal at set size 4 and reached a plateau thereafter (Fig. 2). A 2 [state: rested wakefulness (RW), SD] by 6 (set size) repeated-measures ANOVA on K scores indicated significant effects of state, F(1, 29) = 35.29, P < 0.001, set size, F(5, 145) = 34.61, P < 0.001, as well as a significant interaction, F(5, 145) = 6.92, P < 0.001.

Fig. 2.

Behavioral performance (K) in the VSTM task and IPS activation in the VSTM and VAC tasks after a normal night's sleep. At RW, IPS activation paralleled K in the VSTM task; activation/performance increased to set size 4 before reaching an asymptote. In contrast, for the VAC task, IPS activation was similar for set sizes 1–4 and increased slightly for set sizes 6 and 8. After SD, there was a decrease in behavioral performance as well as activation in both tasks.

K varied significantly with set size at RW, F(5, 145) = 31.95, P < 0.001. Paired-sample t tests (α = 0.01 to correct for multiple comparisons) showed significant stepwise increments in K up to set size 4 (largest P = 0.001). K did not increase significantly between set sizes 4, 6, and 8 (smallest P = 0.45). Despite the significant decline in K after SD, an effect of set size remained, F(5, 145) = 10.41, P < 0.001. When participants were sleep-deprived, their short-term memory capacity averaged around two colors, with maximal performance seen at set size 3. Although K was significantly different between set sizes 2 and 3, t(29) = 4.04, P < 0.001, there was no significant increase in K beyond set size 3.

Although K is considered a marker of short-term memory capacity, state-related differences in task performance is better assessed using task accuracy as the use of set size, because a product can inflate the difference in performance between the two states for the larger set sizes. Consistent with this, we found that accuracy on the VSTM task declined consistently with increasing set size, F(1, 145) = 148.45, P < 0.001, and after SD, F(1, 29) = 38.02, P < 0.001 [supporting information (SI) Table 1]. However, the interaction of set size and state for accuracy was not significant, F(5, 145) = 1.57, P = 0.17.

Visual array-size control (VAC).

There was a significant decline in accuracy on the control task after SD, F(1, 29) = 23.35, P < 0.001 (SI Table 1). However, unlike performance on the memory task, accuracy did not vary as a function of set size, F(5, 145) = 0.67, P = 0.65. There was similarly no interaction of set size and state, F(5, 145) = 0.13, P = 0.99.

Nonlapsers vs. lapsers.

Nonlapsers (n = 14) comprised individuals who responded to at least 95% of the trials in both tasks when sleep-deprived. The remaining subjects (lapsers) did not respond to >5% of trials (mean 14% of VSTM trials; 8% of VAC trials). Nonlapsers had better short-term memory capacity when sleep deprived (SI Table 2). A 2 (state) by 6 (set size) by 2 (group) repeated-measures ANOVA indicated significant interactions between state and group for both K, F(1, 28) = 12.66, P < 0.01, and accuracy, F(1, 28) = 19.75, P < 0.001.

Imaging Findings.

Activation associated with the two tasks was analyzed by using separate whole-brain voxel-by-voxel 6 (set size) by 2 (state: RW, SD) repeated-measures ANOVA (SI Table 3). Consistent with Todd and Marois (4), an effect of set size was present in the IPS for the VSTM task, whereas an effect of set size was expressed predominantly in extrastriate cortex for the VAC task. Although a number of brain regions showed task-related activation that is modulated by set size and state (SI Table 3), we confine our discussion of results to the posterior cerebral cortex, specifically the IPS, the ventral occipital (VO) extrastriate cortex, the precuneus (PC)/posterior cingulate cortex (PCC), and the temporoparietal junction bilaterally. The first two regions figure prominently in the imaging literature pertaining to VSTM (4, 14). The latter two regions deactivate when one engages in task-relevant processing (15).

IPS Responses in the Visual Memory and Control Conditions.

In the scans performed after normal sleep, we found that the IPS showed a differential parametric response to set size in the two tasks, suggesting that it is involved in the short-term storage of visual information (4, 14).

In the memory task, an effect of set size was evident in bilateral IPS (η² for right IPS = 0.48). Additionally, activation here paralleled the increase in K. Although activation did not differ between set sizes 1 and 2, there were significant stepwise increases between set sizes 2–4 (largest P = 0.008) and reached asymptote thereafter (P = 0.07 for the contrast between set sizes 4 and 6 and P = 0.85 for the contrast between set sizes 6 and 8). In comparison, in the control condition, the effect of set size on activation in the IPS was much weaker (η² = 0.23) and was not tightly coupled to K, being similar at set sizes 1–4 (smallest P = 0.30) before increasing for set sizes 6 and 8 (Fig. 3).

Fig. 3.

IPS activation for the VSTM and VAC tasks as a function of set size and state for 30 participants. There were strong effects of set size and state for the VSTM task. An effect of state was also evident in the control task.

SD severely depressed task-related activation in both the memory and control conditions. In the memory condition, there was a clear effect of state (η² = 0.55) and a marginal interaction of state and set size (η² = 0.12) (SI Table 3; Fig. 3). There was a reduced coupling of IPS activation and K after SD (Fig. 2). Unlike the pattern reported for K, stepwise increases in activation were less marked; comparisons between activation at adjacent set sizes were not significant (smallest P = 0.08). Significant differences in activation were present only in comparisons between set size 1 and set sizes 3, 4, 6, and 8 (largest P = 0.01).

To eliminate the possibility that the decrement in IPS response was because of participants falling asleep, we analyzed activation data related to the short-term memory condition only for nonlapsers (n = 14). A state by set size ANOVA revealed similar effects of set size, F(5, 65) = 23.08, P < 0.0001 (η² = 0.59) and state, F(1, 13) = 19.12, P < 0.001 (η² = 0.51) within the IPS for this subgroup (SI Fig. 6A).

Storage deficiency, as reflected by IPS activation, was unlikely to be solely responsible for the sleep-deprivation-related drop in memory capacity, because the IPS response to the control condition, which tracked visual processing without mnemonic demands, was also dramatically reduced after SD (η² = 0.61; SI Table 3). These observations also applied to the subgroup of 14 participants who did not lapse dramatically after SD.

VO Responses to Task and State.

After a good night's sleep, VO extrastriate activation increased in tandem with increasing set size (η² = 0.19) in the memory condition (SI Table 3). SD engendered a precipitous decline in VO activation across all set sizes (η² = 0.48; SI Table 3; Fig. 4). Similar effects of set size and state were found for the visual control condition. VO activation increased monotonically with set size (η² = 0.67) after normal sleep; a series of pairwise t tests for VO activity at RW indicated nonsignificant differences between set size 1 and 2, t(29) = 1.84, P = 0.08 and between set sizes 2 and 3, t(29) = 1.12, P = 0.27. However, all other comparisons were significant (highest P = 0.005). VO activity was severely attenuated after SD (η² = 0.43; Fig. 4), and pairwise comparisons at each set size indicated significant effects of state at all set sizes (largest P = 0.01).

Fig. 4.

VO activation for the VSTM and VAC tasks as a function of set size and state for all 30 participants. A significant effect of state was present across all set sizes for both tasks.

To rule out the possibility that decreased VO activation was a result of sleep-related eye closure, we performed a subset analysis on nonlapsers. State-related declines in VO activation in both the memory (η² = 0.43) and visual control conditions (η² = 0.25; SI Fig. 6B) remained in this subset analysis.

PC/PCC and Temporoparietal Deactivation.

The finding of decreased posterior cortex responses to visual stimuli after SD could raise suspicion of a generalized state-related reduction in cerebral perfusion or change in blood–oxygen-level-dependent response. This was addressed by finding preserved stimulus-related responses in the PC that shares the same arterial perfusion as the parietal and occipital regions of interest (16). We found task-related deactivation within the PC and temporoparietal junction to be relatively preserved after SD (Fig. 5 and SI Fig. 7).

Fig. 5.

Activation in the PC/PCC for the VSTM task as a function of set size and state for 30 participants. There was a strong effect of set size. The effect of state was not significant when all participants were analyzed as a group. However, when a comparison was made between participants who responded to at least 95% of the trials on the VSTM and VAC tasks after SD (nonlapsers; n = 14) and lapsers (n = 16), there was a significant interaction of state, set size, and group. This interaction reflects the finding of lesser deactivation after SD for those who lapsed more when sleep deprived and indicates that this effect was present mainly at higher visual loads.

After normal sleep, and in all 30 volunteers, the magnitude of task-driven deactivation in the PC/PCC (η² = 0.40) paralleled set size. After SD, nonlapsers showed preserved parametric change in PC/PCC deactivation with increasing array size. In contrast, the 16 subjects who were less able to maintain performance after SD showed significantly reduced deactivation in the PC/PCC when sleep-deprived, as evinced by the significant state by group interaction, F(1, 28) = 6.67, P = 0.01 (Fig. 5). There was also a significant state by set size by group interaction, F(5, 140) = 2.39, P = 0.04. This suggests that, unlike the SD-induced functional deficit within the visual cortices that were present even with low item loads, poorer performers showed reduced deactivation only with higher visual item load. A similar pattern of activation was also present in the temporoparietal junction, which shows a similar state by set size by group interaction, F(5, 140) = 2.72, P = 0.02.

Discussion

SD Reduces VSTM Capacity Through Several Mechanisms.

We found that SD reduces VSTM capacity, as evidenced by a reduced K when healthy adults have to temporarily retain color information of a simple array.

After both normal sleep and SD, IPS activation increased monotonically in accordance to VSTM capacity in the memory condition but showed a much reduced response to set size in the perceptual load condition. After normal sleep, extrastriate cortex activation increased with set size in both the memory and perceptual load control conditions. Taken together, these patterns of activation corroborate prior suggestions that the IPS participates as a memory store (4, 14), and that the extrastriate activation serves to track visual processing load.

Critically, SD reduced the coupling between IPS activation and K in the memory condition that was observed after normal sleep. SD disproportionately depressed IPS activation relative to K at each set size. This cannot be explained solely by a decline in visual item storage.

SD Impairs the Neural Response to Perceptual Load, Signifying a Basic Fault in Perceptual Processing and/or Attention.

We took care to show that SD-related reductions in posterior cortex activation were neither due to generalized state-related reduction in blood–oxygen-level-dependent response nor a result of eye closure linked to an attenuation of volunteer effort. This left impaired visual processing or degraded attention as possible explanations for the observed reduction in activation, and these two may be interrelated.

Ventral visual pathway activation, from the occipital lobe to the fusiform and parahippocampal region, can be modulated by attention, increasing when attention is focused and reduced when attention is divided or directed elsewhere (17, 18). In present experiments, attention was not divided or directed elsewhere, and the reduction in activation in the posterior cortex can be reasonably attributed to neurophysiological changes accompanying SD. Such a state-related change might be expected to affect brain activation in a manner relatively consistent across set size and task. This is exactly what we observed in the present experiment, where neural responses in the occipital and parietal areas were attenuated even at lower set sizes.

Prolonged wakefulness is accompanied by elevation of adenosine levels, which in turn inhibits basal forebrain cholinergic output (19). Activity of the latter system is important in the maintenance of wakefulness. In addition, acetylcholine figures prominently in the maintenance of sustained attention (20), and reduction in acetylcholine-mediated drive for wake maintenance is a candidate mechanism for attention decline after SD. Acetylcholine has been shown to enhance visual working memory through increasing selectivity of perceptual processing (21, 22). The neural accompaniment of acetylcholine-mediated enhancement of visual memory has been observed in occipital extrastriate (23, 24) and superior parietal (25) areas spatially similar to those showing SD-attenuated activation in the present study.

We posit that degraded perceptual processing, possibly related to an impaired capacity to sustain attention, results in a percept that is harder to maintain in short-term memory, leading to a drop in VSTM capacity. Such a mechanism may link deficits in attention with those affecting memory and learning (22).

Connecting Attention, Working Memory, and SD.

The present work lends support to our suggestion that decline in working memory after SD may be strongly influenced by degraded attention (9). This might not have been appreciated in prior work involving SD, because the parietal region that reliably shows state-dependent reduction in task-driven activation (6, 9) participates in multiple cognitive operations, including the temporary storage of visual information (4), sustained attention (26), and visual selection (27, 28). Further, although the overlap of working memory and visuospatial attention networks has been demonstrated by behavioral (11–13) and functional imaging studies (29), modality-specific (memory) storage and attention mechanisms may make distinct contributions to VSTM capacity (13).

Decreased Deactivation, a Secondary Marker for VSTM Capacity Reduction in SD.

In addition to cortical regions showing task-related activation, those showing deactivation may make independent contributions to performance when sustained attention is required (26). Here, we observed attenuated deactivation in the PC/PCC gyrus and temporoparietal junctions in volunteers whose VSTM capacity declined to a greater extent after SD.

Prior work has shown that task-related changes in activation within the PC/PCC may correlate with a decline in psychomotor vigilance after SD (30), as well as with memory loss in elderly adults (31). The temporoparietal junction has been reported to show greater deactivation with increasing VSTM load but not perceptual difficulty. It is thought to interrupt ongoing “background,” task-irrelevant processes for the analysis of potentially behaviorally relevant visual events (32).

In the present work, reduced deactivation in the poorer-performing group engendered by SD was evident only at larger set sizes, indicating it may be more important to inhibit task-irrelevant processes as visual item load increases.

In sum, the present experiments highlight how attention and memory are interrelated in studies of cognition in the setting of SD, and that there are several determinants of the capacity limit of short-term memory under this condition.

Methods

Participants.

Participants were selected from respondents to a Web-based questionnaire. They had to: (i) be right-handed, (ii) be between 18 and 35 years of age, (iii) have habitual good sleeping habits (sleeping no less than 6.5 h each night for the past 1 month), (iv) score no greater than 22 on the Morningness–Eveningness scale (33), (v) not be on any long-term medications, (vi) have no symptoms associated with sleep disorders, and (vii) have no history of psychiatric or neurologic disorders. The sleeping habits of all participants were monitored throughout the 2-week duration of the study, and only those whose actigraphy data indicated habitual good sleep (i.e., they usually slept no later than 1:00 a.m. and woke no later than 9:00 a.m.) were recruited for the study after informed consent.

Thirty persons successfully completed this study. They were right-handed, healthy, university undergraduates and graduates (16 females; mean age = 21.33 years, SD = 1.63 years, range = 19–25 years). All participants indicated they did not smoke, consume any medications, stimulants, caffeine, or alcohol for at least 24 h before scanning. All participants had normal or corrected-to-normal vision, and normal color vision was confirmed with Ishihara's Tests for Color Deficiency.

Experimental Tasks.

VSTM.

Participants were presented with and instructed to remember arrays of different colors, up to a maximum of eight differentiable colors. These colors were presented as black-outlined squares on a white background and occupied 16 possible locations (in an invisible 4 × 4 grid), which subtended a maximum visual angle of 4° × 4°. There was a pool of 18 possible colors (Red–Green–Blue color coordinates: 255,0,0; 0,0,255; 0,65,0; 0,255,0; 204,153,255; 255,102,0; 255,255,0; 128,99,65; 173,216,230; 255,0,255; 153,51,0; 0,0,0; 0,250,154; 128,0,128; 255,139,107; 255,255,255; 116,113,112; 202,255,112).

On each trial, the memory array was presented for 500 ms, followed by a single memory probe 1,000 ms later (Fig. 1), and participants indicated whether the color was present/absent from the memory array by pressing a “yes” or “no” button. All participants completed four runs of the task. Each run consisted of 12 blocks of four trials, each block lasting 15 s, interleaved with 13 blocks of 18 s of fixation. For every run, there were two blocks of each set size (1, 2, 3, 4, 6, and 8), and these blocks were presented in a random order for every participant.

VAC.

In this task, participants were similarly presented with visual arrays of up to eight colors but had to respond only to the presence or absence of a square in the center of the array (Fig. 1). Array sizes of 1, 2, 3, 4, 6, and 8 were used. There were three runs of this task. Each run consisted of 12 15-s stimulus blocks interleaved with 13 blocks of 18-s fixation, and the order of the blocks in each run was randomized across subjects.

Study Procedure.

Participants visited the laboratory thrice, approximately once per week. The first visit was a briefing session during which they were informed of the study's protocol and requirements and were given practice on two runs of the VSTM task. Scanning took place in the two remaining sessions (RW and SD). The order of scans was counterbalanced across the participants.

The scans at RW took place at 8:00 a.m. For the SD session, participants were monitored in the laboratory from 6:00 p.m. onward, and scanning took place at 6:00 a.m. the next day. During the night, participants were allowed to engage only in nonstrenuous activities, such as reading, working on a computer, and conversing. A short battery of psychometric tests was performed hourly according to a previously described protocol (6, 9). To minimize the possible cognitive and hemodynamic confounding effects of nicotine (24) and caffeine (34), participants were not allowed drinks containing any stimulants or caffeine (e.g., coffee, chocolate, and colas), and smoking was not permitted.

Practice runs of both tasks were administered immediately before participants entered the scanner to ensure that subjects understood all task requirements. The order of administration of the two tasks was counterbalanced across subjects. Performance was continuously monitored, and participants were prompted to respond through the intercom system when they failed to respond on two consecutive trials.

Imaging Procedure and Analysis.

Stimuli were projected onto a screen by using a liquid crystal display projector and viewed by participants through a rear-view mirror. Participants responded by using a button box held in the right hand. A bite bar and foam padding were used to reduce head motion. Images were acquired on a 3T Allegra system (Siemens, Erlangen, Germany). A gradient echo-planar imaging sequence was used with a repetition time of 3,000 ms, field of view of 192 × 192 mm, and a 64 × 64-mm pixel matrix. Thirty-six oblique axial slices (3 mm thick with a 0.3-mm interslice gap) approximately parallel to the intercommissural plane line were acquired. High-resolution coplanar T1 anatomical images were also obtained. For the purpose of image display in Talairach space, an additional high-resolution anatomical reference image was acquired by using a 3D–Magnetization–Prepared–Rapid–Gradient Echo sequence.

The functional images were processed by using Brain Voyager QX version 1.7.9 (Brain Innovation, Maastricht, The Netherlands). Intrasession image alignment to correct for motion across runs was performed by using, as the reference image, the first image of the functional run that was acquired immediately before the coplanar T1-weighted image. Interslice timing differences attributable to slice acquisition order were adjusted by using linear interpolation. Gaussian filtering was applied in the spatial domain by using a smoothing kernel of 8-mm FWHM for group level activation maps. After linear trend removal, a high-pass filter of period 408 s was applied. The T1 images were used to register the functional data set to the volunteers' own 3D image, and the resulting aligned data set was transformed into Talairach space.

The functional image data were analyzed for both sessions by using a general linear model with six predictors, one for each set size. All predictors were convolved by using a canonical hemodynamic-response function and analyzed by using a mixed-effects model.

Data Analysis.

Performance on the VSTM task was indexed by K, an index of memory capacity defined as ((hit rate + correct rejection rate) − 1) × array size (12). The effects of set size and state on both behavioral and neural findings were investigated by using 6 (set size) by 2 (state) repeated-measures ANOVA. Similar effects of state and set size were found for the IPS and VO bilaterally. As such, effects are reported only for regions in the right hemisphere. Effect sizes were specified in terms of η² (i.e., SS_effect/SS_effect + SS_error; SS denotes sum of squares).

Abbreviations

VSTM

visual short-term memory

IPS

intraparietal sulcus

VAC

visual array-size control

VO

ventral occipital

SD

sleep deprivation

PC

precuneus

PCC

posterior cingulate cortex

RW

rested wakefulness.