The roles of item exposure and visualization success in the consolidation of memories across wake and sleep

Abstract

Memory consolidation during sleep does not benefit all memories equally. Initial encoding strength appears to play a role in governing where sleep effects are seen, but it is unclear whether sleep preferentially consolidates weaker or stronger memories. We manipulated encoding strength along two dimensions—the number of item presentations, and success at visualizing each item, in a sample of 82 participants. Sleep benefited memory of successfully visualized items only. Within these, the sleep–wake difference was largest for more weakly encoded information. These results suggest that the benefit of sleep on memory is seen most clearly for items that are encoded to a lower initial strength.

Of the vast amount of information encoded each day, most of it will be forgotten. Only a few select memories are retained for long-term storage. Sleep plays an important role in these memory consolidation processes (Diekelmann and Born 2010; Rasch and Born 2013). The initial strength of a memory impacts its long-term trajectory, although the specific effect of sleep remains unclear. Some studies report that weaker memories receive a greater benefit of sleep (Drosopoulos et al. 2007; Djonlagic et al. 2009; Schapiro et al. 2017), while others find strong memories are prioritized (Tucker and Fishbein 2008; Schoch et al. 2017; Wislowska et al. 2017).

How might these findings be reconciled? It is notable that across studies a wide variety of different tasks were used, with large variation in what constituted a weak versus strong memory. To date, most studies have only included two levels of encoding strength and have implemented between-subject designs that precludes the investigation of weak versus strong encoding within an individual (c.f. Schapiro et al. 2017). Finally, some studies allocated participants to an encoding strength group via a post-hoc median split of initial learning performance. This conflates weak versus strong encoding with poor versus good learners, which may not tap into the same process (Creery et al. 2015).

We sought to directly investigate the role of two different processes that influence memory strength. Our first dimension of encoding strength was the number of item presentations, or encoding opportunities. Two previous studies using such an approach have shown sleep–wake differences for weaker memories only (Drosopoulos et al. 2007; Schapiro et al. 2017). Increasing neural pattern similarity across successive item presentations is associated with better memory (Xue et al. 2010; Lu et al. 2015), and a large number of repetitions leads to stronger cortical memory representations (Brodt et al. 2018). As such, it is possible that after multiple presentations, memory traces have already been rendered strong enough that sleep does not or cannot strengthen them further, resulting in the sleep benefit being strongest for initially weak memories.

Our second dimension of encoding strength was the ability to successfully visualize items. Other work has shown that successful visualization of word pair objects interacting together in a scene leads to stronger memory than visualizing the objects separately or failing to visualize (Kirchhoff 2009; Murray and Kensinger 2012). Compared with inducing strength via repetition of stimuli, successful visualization may lead to qualitatively deeper memory representations compared with items that are not visualized (Galli 2014). Deeper memories formed by successful visualization may lead to a stronger associative link between the two objects (Murray and Kensinger 2012). Given that sleep appears to convey a benefit for associative memories (Diekelmann et al. 2009), successful visualization may be necessary for sleep-based consolidation to occur.

A total of 82 participants (M_age = 21.3 yr (SD = 2.9, range = 18–32), 56 female) took part in the study. Participants were English-speaking, with no history of any sleep, neurological, or psychiatric disorders. They reported a typical bedtime of no later than 2 a.m. and slept on average at least 6 h per night. Participants were recruited through local college job boards and were financially compensated for their time. In the 3 d prior, they were asked to keep a regular sleep schedule. On all experimental days, they were asked to abstain from caffeine and alcohol. They were instructed not to nap between visits. Two participants were excluded due to self-reporting taking a nap between sessions, and one participant was excluded due to a recording error. The study received IRB approval from Beth Israel Deaconess Medical Center.

Subjective sleep was assessed via a sleep log that asked participants about their sleep in the three nights prior to the experiment, the Pittsburgh Sleep Quality Index (PSQI), which measured general sleep quality over the preceding month (Buysse et al. 1989), and the Stanford Sleepiness Scale (SSS), which was used to assess subjective sleepiness and alertness before the start of each experimental session (Hoddes et al. 1972). See Supplemental Material for subjective sleep data.

The study and experimental design are shown in Figure 1. There were four experimental groups, with all participants making two visits to the laboratory (Fig. 1A). A list of 90-word pairs were created. All words described objects, and the two words of each pair were unrelated to each other (e.g., bucket-car). Words were selected from an online list of 5000 words (https://www.wordfrequency.info). Word frequency ranged from 4991 to 27,524 and contained five to eight letters. During session 1, participants completed an encoding phase, where they studied pairs of words (Fig. 1B). Participants were instructed to try and remember the pairs by visualizing a scene containing the two objects (i.e., integrative visualization). Each word pair was randomly assigned to one of three encoding conditions: weak (one presentation, hereafter referred to as 1X), intermediate (two presentations [2X]), or strong (four presentations [4X]). On each trial, a fixation cross appeared on the screen for 1000 msec, following by the word pair for 1500 msec. After the pair disappeared, participants pressed a key indicating whether they had successfully visualized a scene including both items, responding either “yes,” “no,” or “don't know.” After encoding, participants were told that their memory would be tested after a 5-min rest period. On each recall trial (Fig. 1B), the first word appeared on the screen and participants had to type in the second word. After completing this for each pair, participants were told that when they returned for their second visit, their memory for the same word pairs would be tested again. Upon returning for session 2, participants completed a second (delayed) recall test (Fig. 1B). The timing of the first and second session varied between the experimental groups as follows: In the 12-h sleep (n = 19) and 24-h sleep first (n = 19) groups, participants performed session 1 in the evening (start time 8–10 p.m.) and the second session either 12 or 24 h later. The 12-h wake (n = 20) and 24-h wake first (n = 20) groups performed session 1 in the morning (start time 8–10 a.m.), and session 2 either 12 or 24 h later. Overall word pair recall is shown in Table 1.

Figure 1.

Experimental design. (A) Timing of the four experimental groups. (S1) Session 1 (encoding + immediate recall), (S2) session 2 (delayed recall). (B) Example of a trial at each phase. During encoding, participants studied 90 pairs of words. They were instructed to visualize a scene containing the two objects. Following a fixation cross of 1000 msec, the word pair appeared for 1500 msec. Following presentation, participants indicated whether they had successfully visualized the word pairs (yes, no, don't know). Thirty of the word pairs were presented one time, 30 presented two times, and 30 presented four times. There was a total of 210 trials, with a 1-min break every 35 trials. Order of trials was pseudorandomized for each participant, with at least two trials separating multiple presentations of any item. During recall, the first word of the pair appeared on the screen, and participants were instructed to type in the second word as quickly and as accurately as possible. They were told there was no penalty for guessing, and if they did not know the answer, they could press the “Enter” key to move on to the next pair. If there was no response after 7 sec, a warning message appeared telling the participant to respond. If there was no response after a further 3 sec (10 sec from initial word onset), the program automatically advanced. Each pair was tested once (n = 90 trials total). Both immediate and delayed recall tests followed the same procedure. Order of presentation of word pairs was randomized for each participant and each test.

Table 1.

Percent of word pairs recalled at immediate and delayed test

We first examined how many items were successfully visualized during encoding (Fig. 2A). Successful visualization was defined as responding “yes” to the visualization prompt on at least one presentation. Averaged across all trials and groups, participants reported successfully visualizing 54% of the trials (SD = 14%). There was a significant effect of item presentation number on visualization success (F_(2,150) = 29.2, P < 0.001, ηp² = 0.28), with a significant increase in the percentage of items successfully visualized as presentation number increased (all P < 0.001, all d > 0.40). There was no effect of group (F_(3,75) = 0.68, P = 0.56, ηp² = 0.03), nor was there a significant interaction between presentation number and group (F_(6,150) = 0.29, P = 0.94, ηp² = 0.01).

Figure 2.

Results. (A) Percentage of items visualized during encoding. (B) Immediate recall accuracy. (C) Relative change in recall for items visualized during encoding. (D) Relative change in recall for items not visualized during encoding. (1X) One presentation during encoding, (2X) two presentations, (4X) four presentations, (VIS) successfully visualized during encoding, (NVIS) not successfully visualized during encoding. Error bars represent the within-participant standard error

Next, we looked at immediate recall accuracy (Fig. 2B). Successful visualization had a significant effect on immediate recall (F_(1,75) = 19.40, P < 0.001, ηp² = 0.21), with visualized (VIS) items showing higher recall accuracy than not visualized (NVIS) items. There was also a significant effect of presentation number (F_(2,150) = 377.88, P < 0.001, ηp² = 0.83). Immediate recall accuracy increased significantly with each increase in presentation number (all P < 0.001, d > 1.69). There was also a significant item visualization * presentation number interaction (F_(2,150) = 3.88, P = 0.023, ηp² = 0.05). VIS items were better recalled than NVIS items for all item presentation conditions (all P < 0.027, all d > 0.25). However, the difference between VIS and NVIS items was smaller for 1X items (M = 7%, SD = 28%) than either 2X (M = 13%, SD = 32%; t₍₇₇₎ = 3.5, P = 0.001, d = 0.14) or 4X (M = 16%, SD = 28%; t₍₇₇₎ = 2.35, P = 0.02, d = 0.27). The difference between 2X and 4X was not significant (t₍₇₇₎ = 1.3, P = 0.21, d = 0.14). The main effect of group, and all interactions involving group, were nonsignificant (all P values >0.19, all ηp² < 0.06).

We next sought to understand how item presentation and visualization influenced change in memory across either a 12-or 24-h delay (Fig. 2C,D). Change in recall was measured as the relative change in recall (i.e., the change from immediate to delayed test as a percent of pairs recalled at immediate test).

To disentangle the effects of sleep wake pattern (i.e., sleep closely following learning, or wake closely following learning) and retention interval (12 h, 24 h) we ran a 3 (item presentation condition: 1X, 2X, 4X) × 2 (sleep–wake pattern) × 2 (retention interval) ANOVA. Separate ANOVAs were run for VIS and NVIS items. For VIS items, one participant's 1X change score was excluded on the basis of being an outlier (standardized change in memory score of Z > 3.29). There was a significant effect of sleep–wake pattern (F_(1,70) = 11.77, P = 0.001, ηp² = 0.14). Across encoding conditions, memory was better when learning was followed by sleep (M = −7.59%, SD = 7.95%) compared with when it was followed by wake (M = −16.31%, SD = 18.90%). There was no significant main effect of retention interval (F_(1,70) = 0.17, P = 0.68, ηp² = 0.002); however, there was a significant interaction between sleep–wake pattern and retention interval (F_(1,70) = 5.36, P = 0.023, ηp² = 0.07). Follow-up tests showed a significant main effect of sleep following learning when the retention interval was 12 h (t₍₃₂₎ = 3.54, P = 0.001, d = 1.20), but not when the retention interval was 24 h (t₍₃₄₎ = 0.02, P = 0.98, d < 0.01). There was also a significant main effect of item presentation (F_(2,139) = 5.12, P = 0.007, ηp² = 0.07). There was significantly less forgetting of 4X items compared with 1X (t₍₆₉₎ = 2.49, P = 0.015, d = 0.30) and 2X items (t₍₇₆₎ = 3.29, P = 0.002, d = 0.38).

There was a significant interaction between sleep–wake pattern and item presentation (F_(2,139) = 3.63, P = 0.029, ηp² = 0.05, Fig. 3A). Follow-up one-way ANCOVAs, with immediate recall performance entered as a covariate, showed memory was better when sleep closely followed learning for 1X (F_(1,67) = 10.64, P = 0.002, ηp² = 0.14) and 2X (F_(1,74) = 7.59, P = 0.007, ηp² = 0.09) items, but not 4X (F_(1,76) = 3.47, P = 0.07, ηp² = 0.04). In both cases, the effect was driven by higher item maintenance between sessions (i.e., successfully recalled at both immediate and delayed test) (all P < 0.005, d > 0.70), and not by item gains (unsuccessfully recalled at immediate test and successfully recalled at delayed test) (all P > 0.20, d < 0.29) (see Supplemental Material).

Figure 3.

Interaction between item presentation and sleep/wake retention for visualized items. (A) Forgetting across item presentation conditions when either sleep or wake occurred first following learning. (B) Pairwise contrasts between each group that had sleep compared with the group who had no sleep. (1X) One presentation during encoding, (2X) two presentations, (4X) four presentations. Error bars represent the within-participant standard error. (***) P < 0.001, (**) P < 0.01, (*) P < 0.05.

Within the wake first group, there was significantly less forgetting of 4X items compared with 2X (t₍₃₈₎ = 2.64, P = 0.012, d = 0.42) and 1X items (t₍₃₂₎ = 2.76, P = 0.009, d = 0.48). In the sleep group, the 4X versus 1X difference was not significant (t₍₃₆₎ = 0.33, P = 0.75, d = 0.05), although there was still less forgetting of 4X compared with 2X (t₍₃₇₎ = 2.29, P = 0.03, d = 0.37). When the sleep first group was separated into 12-h and 24-h conditions, and contrasted to the 12-h wake only group (Fig. 3B), we found significantly less forgetting of 1X items after a 24-h delay with sleep first compared with a 12-h delay containing no sleep (t₍₃₄₎ = 2.29, P = 0.03, d = 0.75). Comparatively, the 12-h sleep condition showed less forgetting regardless of the number of initial item presentations, when compared directly with 12-h wake (all P < 0.02, all d > 0.76). There was no encoding condition × retention interval interaction (F_(1,139) = 0.29, P = 0.75, ηp² = 0.004), nor was there a significant three-way interaction (F_(2,139) = 1.50, P = 0.23, ηp² = 0.02).

For NVIS items, we observed a significant main effect of item presentation (F_(2,145) = 10.33, P < 0.001, ηp² = 0.13). Across groups, there was significantly more forgetting of 1X items compared with both 2X (t₍₇₆₎ = 3.04, P = 0.003, d = 0.35) and 4X (t₍₇₄₎ = 4, P < 0.001, d = 0.47) items. There was no difference between 2X and 4X (t₍₇₅₎ = 1.08, P = 0.28, d = 0.12). All other main effects and interactions were nonsignificant (all P > 0.057).

Together, these results suggest that effects of sleep on memory were only seen for visualized items. A direct contrast revealed significantly better memory for VIS items in the sleep group compared with the wake group (t₍₆₈₎ = 2.56, P = 0.01, d = 0.61), with no difference for NVIS items (t₍₇₃₎ = 0.81, P = 0.41, d = 0.19).

A logistic mixed effects model largely confirmed these results. This approach complements the ANOVA results in two ways. First, individual participant and trial variation can be modeled as random effects. Second, by using items maintained at both sessions (i.e., successfully recalled at both tests) as the dependent variable, it eliminates any potential problems with baseline memory influencing the change in recall metric. For 1X VIS items, both the 12 h sleep and 24 h sleep first groups showed significantly higher odds of remembering an item at both sessions than the 12 h wake group. For 2X VIS items, the only significant contrast was significantly higher odds in the 12 h sleep group compared with 12-h wake (see Supplemental Material).

To understand how memories of different initial strength are consolidated after already decaying across a prior 12-h period, we used the decay that occurred across the first 12 h (indexed by the 12 h sleep/12 h wake group) to infer the amount of decay that occurs across the second 12 h in the 24 h groups (see Supplemental Material for details; Talamini et al. 2008). We focused this analysis on just the VIS items.

We first looked at the inferred decay across the sleep-filled portion of the 24 h wake-first group, after memory had deteriorated over preceding wakefulness (Fig. 4A). For the 1X items, there was a nonsignificant gain across the sleep interval (t₍₁₅₎ = 1.76, P = 0.09, d = 0.44). While the magnitude of this effect was numerically higher than for 2X and 4X items, the differences were not significant (all P > 0.13, all d < 0.40). The inferred decay across the sleep period in the 24 h wake first group was equivalent to the decay observed in the 12 h sleep group across all item presentation conditions (all P > 0.10, all d < 0.57).

Figure 4.

Forgetting across sleep and wake intervals. (A) Inferred decay during sleep after a day of wakefulness. Gray dashed line shows inferred decay in the second half of the wake-sleep interval, calculated for each subject by subtracting mean decay in the 12 h wake group from the 24 h wake first decay score. (B) Inferred decay across wake after a night of sleep. Like A, the gray dashed line shows inferred decay across the second half of the sleep–wake interval. Here, inferred forgetting was calculated for each subject by subtracting mean decay in the 12 h sleep group from the 24 h sleep first decay score. (1X) Items presented once during encoding, (2X) items presented twice during encoding, (4X) items presented four times during encoding.

We next focused on the 24 h sleep first group and inferred the amount of decay that occurred over the wake period following sleep (Fig. 4B). For 1X items, forgetting over the sleep-filled first 12 h was not different from zero (t₍₁₇₎ = 0.06, P = 0.95, d = 0.01). Forgetting occurred over the following wake period, with the rate of decay across wake being significantly greater than the decay over the preceding sleep interval (t₍₃₅₎ = 3.11, P = 0.004, d = 1.02). Despite this, the rate of wake-associated decay following a night of sleep was 18% lower than an equivalent wake delay that was not preceded by a night of sleep (Fig. 4B). For 2X and 4X items, the amount of decay that occurred over the sleep period and the wake period was equivalent (all P > 0.68, d < 0.13).

Here, we examined two distinct dimensions of encoding strength, item repetition and visualization success, and investigated their impact on memory retention across a delay period filled with sleep or wake. When sleep occurred close to learning, we found significantly less forgetting of successfully visualized items, compared with when a wake-filled delay followed learning. No difference between groups was observed when considering items that were not successfully visualized during encoding. Visualizing two items together leads to a stronger associative memory (Murray and Kensinger 2012), suggesting a deeper level of encoding for visualized items. Successful visualization also recruits the hippocampus (Kirchhoff 2009), which is critically involved in both the encoding and consolidation of memories (Rauchs et al. 2011; Sawangjit et al. 2018; Schapiro et al. 2019). A deep encoding strategy may be required for any consolidation during sleep to occur. Other research has suggested that selective benefits of sleep may occur for more deeply encoded memories (Alger and Payne 2016). Direct investigations of how shallow versus deep encoding influence subsequent consolidation are warranted.

Within the visualized items, items encoded to different strengths via presentation number showed differential forgetting rates across periods of either wake or sleep. Over a wake delay, there was significantly more forgetting of weakly encoded items compared with strong ones. After just a single presentation, the neural memory representation is weaker and less stable compared with a memory for an item presented multiple times (Xue et al. 2010; Lu et al. 2015). Over a wake-filled delay, it appears those weak memory traces fade from memory faster than stronger, more durable traces.

This pattern was not seen when sleep closely followed learning. In this group, forgetting was equivalent for memories encoded to different strengths. Relative to wake, the largest between-group difference in memory was for the weakly encoded items. This suggests that during sleep weaker memories receive a consolidation benefit when compared with wake where those same memories deteriorate the most. While the neurophysiological mechanism underlying this effect is unclear, recent research has shown that the consolidation of weak memories is uniquely associated with sleep spindles (Baena et al. 2020; Denis et al. 2020), 12- to 15-Hz waxing and waning oscillations that are characteristic of nonrapid eye movement sleep. This builds on other evidence showing that sleep spindles facilitate memory consolidation (Schabus et al. 2004; Cox et al. 2012), Furthermore, memory replay in the hippocampus during quiet rest is preferential for weakly encoded memories (Schapiro et al. 2018). Whether this would extend to hippocampal replay during sleep would be an interesting target for future investigation.

When the sleep group was broken down into the 12- or 24-h delay groups, preferential consolidation of weak items was only seen in the 24-h group. Compared with 12-h wake, there was less forgetting only for 1X items despite the identical amounts of wakefulness and a longer overall retention time in the 24-h group. Within the 24-h sleep first group, there was significant forgetting across the wake, but not the sleep, period. Despite this, wake-based forgetting was attenuated when following post-learning sleep, compared with a wake only period. This may suggest that sleep had to some degree stabilized weakly encoded memories, making them more resilient to subsequent deterioration. Given that such patterns were not observed for the 2X and 4X items, this provides evidence for a differential function of sleep and wake on the rate of forgetting for weakly encoded information. While a stabilizing effect of sleep on weak memories over a 24-h period was seen, future work should investigate whether these effects are persistent over an even longer interval (e.g., months or years).

Using the 24-h wake first group, we looked at how consolidation across sleep occurred after 12 h awake, meaning memories of all strengths had already undergone some deterioration (albeit significantly more in the case of the weak items). We found a nonsignificant gain in recall for 1X items across the sleep period. Although not significantly different, the magnitude of this gain was numerically higher for weak items relative to stronger items. This result should be interpreted with caution. As well as being nonsignificant, other studies have reported no benefit of sleep when it occurs after a long wake-filled delay (Gais et al. 2006; Talamini et al. 2008; Payne et al. 2012a,b). Furthermore, sleep-based consolidation of episodic memories typically manifest as reduced forgetting (Fenn and Hambrick 2013), with little evidence that absolute gains in memory can be seen (Muehlroth et al. 2020; but also see Dumay 2016).

In this study memory strength was manipulated during encoding, and both immediate and delayed retrieval occurred without feedback. However, the way in which memories are retrieved can also influence memory strength. While some work has shown that the presence of an immediate test is necessary for sleep-associated consolidation to be detected (Schoch et al. 2017), other research has found no benefit of sleep following a retrieval without feedback condition (Bäuml et al. 2014; Abel et al. 2019). This finding was interpreted by the authors in relation to a bifurcation model, whereby the successfully retrieved items are strengthened to such a high degree that sleep does not need to consolidate them any further, whereas failed retrievals are too weak to receive a benefit (Bäuml et al. 2014). This is conceptually similar to the proposal that memories are consolidated during sleep based on an inverted u-shape distribution, with memories closer to the middle of the strength scale receiving the largest benefit (Stickgold 2009).

It is possible that in the present study, weak items that were successfully retrieved during immediate recall were still within a range that sleep could act upon. Partial support for this hypothesis comes from our results showing that the benefit of sleep was expressed as a higher level of item maintenance between sessions (i.e., successfully retrieved at both tests), with no sleep effects being seen for items not successfully retrieved at immediate test (Muehlroth et al. 2020). Future research is needed to better specify the interaction between encoding and retrieval-based modulations of encoding strength, which might reflect an additional dimension of memory strength. It will be useful to design more continuous measures of encoding strength to pinpoint exactly where on the strength continuum sleep-based consolidation effects are seen, and how wide that window is. Together, this work provides more evidence that sleep-based memory consolidation is not uniform across all levels of initial strength, and that differences in memory between sleep and wake are largest for memories that initially are relatively weakly encoded.