Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Sep 1.
Published in final edited form as: J Cogn Enhanc. 2020 Nov 27;5(3):386–395. doi: 10.1007/s41465-020-00196-y

Multisensory Facilitation of Working Memory Training

Anja Pahor a,b, Cindy Collins a, Rachel N Smith b, Austin Moon a, Trevor Stavropoulos a, Ilse Silva a, Elaine Peng a, Susanne M Jaeggi c, Aaron R Seitz a
PMCID: PMC8415034  NIHMSID: NIHMS1650289  PMID: 34485810

Abstract

Research suggests that memorization of multisensory stimuli benefits performance compared to memorization of unisensory stimuli; however, little is known about multisensory facilitation in the context of working memory (WM) training and transfer. To investigate this, 240 adults were randomly assigned to an N-back training task that consisted of visual-only stimuli, alternating visual and auditory blocks, or audio-visual (multisensory) stimuli, or to a passive control group. Participants in the active groups completed 13 sessions of N-back training (6.7 hours in total) and all groups completed a battery of WM tasks: untrained N-back tasks, Corsi Blocks, Sequencing, and Symmetry Span. The Multisensory group showed similar training N-level gain compared to the Visual Only group, and both of these groups outperformed the Alternating group on the training task. As expected, all three active groups significantly improved on untrained visual N-back tasks compared to the Control group. In contrast, the Multisensory group showed significantly greater gains on the Symmetry Span task and to a certain extent on the Sequencing task compared to other groups. These results tentatively suggest that incorporating multisensory objects in a WM training protocol can benefit performance on the training task and potentially facilitate transfer to complex WM span tasks.

Keywords: working memory, multisensory, training, transfer

1. Introduction

The human brain has evolved to learn in information-rich environments in which integration of information from multiple sensory inputs is ubiquitous. While perception and cognition are often studied within one sensory modality at a time, there is growing evidence that much of the neocortex is inherently multisensory (Ghazanfar & Schroeder, 2006; Murray, Lewkowicz, Amedi, & Wallace, 2016) and that multisensory processing is behaviorally advantageous, particularly when the sources are temporally and semantically related (Diaconescu, Alain, & McIntosh, 2011). Multisensory facilitation can benefit subsequent unisensory perceptual processing (R. S. Kim, Seitz, & Shams, 2008; Shams, Wozny, Kim, & Seitz, 2011; von Kriegstein & Giraud, 2006) and memory retrieval of unisensory information (Heikkilä, Alho, & Tiippana, 2017; Lehmann & Murray, 2005; Murray et al., 2004; Thelen, Talsma, & Murray, 2015). Models suggest that multisensory facilitation can arise through cross-sensory connections between unisensory representations and/or feedback to unisensory representations supporting stronger encoding in those regions as well as by modification or formation of multisensory representations, so that later presentation of unisensory stimuli activates an expanded, multisensory network of brain regions (Shams & Seitz, 2008).

Working memory (WM) is an example of a system that is thought to be multisensory in nature (Quak, London, & Talsma, 2015). Indeed, parts of the WM circuitry such as the intraparietal and dorsolateral prefrontal cortices have been linked to multi-modal maintenance of information (Cowan et al., 2011). WM training has shown to lead to improved performance on tasks that rely on short-term and WM components, particularly when the untrained tasks share similar processes as the trained task (Holmes, Woolgar, Hampshire, & Gathercole, 2019). It has been suggested that transfer effects only occur if the training and transfer tasks engage specific overlapping brain regions and processes (Dahlin, Neely, Larsson, Bäckman, & Nyberg, 2008) and that paradigm-specific effects may reflect changes in strategies developed throughout training, rather than an improvement in the efficiency of WM (Forsberg, Fellman, Laine, Johnson, & Logie, 2020). It is worth noting that mechanisms leading to transfer, and the extent of transfer beyond the domain of WM, are topics of substantial controversy with even metanalyses reaching inconsistent conclusions (Au et al., 2015; Melby-Lervåg, Redick, & Hulme, 2016; Schwaighofer, Fischer, & Bühner, 2015; Soveri, Antfolk, Karlsson, Salo, & Laine, 2017; Weicker, Villringer, & Thöne-Otto, 2016); however, this is goes beyond the scope of the current paper.

In the present paper, we address issues of how to promote learning and (near) transfer within the same cognitive domain, namely, WM. A WM training protocol that incorporates multisensory objects would not only provide redundancy of information but might further enable stimuli to be encoded into richer, multisensory representations (R. Kim, Seitz, & Shams, 2008; Shams & Seitz, 2008), which may facilitate learning and transfer (Deveau, Jaeggi, Zordan, Phung, & Seitz, 2014). However, WM training protocols tend to be restricted to the visual domain and/or are not specifically designed to induce multisensory facilitation. For example, many extant procedures, such as the dual n-back, require each modality stream to be processed separately, rendering multisensory integration disadvantageous to task performance. While this type of dual task performance is certainly demanding, it was later shown that dual n-back training does not lead to greater learning and transfer compared to the single n-back training (Jaeggi et al., 2010; Jaeggi, Buschkuehl, Shah, & Jonides, 2014). To date, there are no procedures that specifically implement multisensory facilitation and thus, the assumption that multisensory training can benefit WM training outcomes is yet to be tested.

Here we test the hypothesis that multisensory WM training will facilitate transfer to untrained tasks within the WM domain compared to visual WM training alone or training that contains visual and auditory stimuli but are presented as separate WM tasks. To address this hypothesis, we randomly assigned participants to one of 3 n-back training conditions and compared their learning and transfer to untrained WM tasks. The conditions consisted of a vision-only training, training that alternated between auditory and visual stimuli, or multisensory training where auditory and visual stimuli were presented concurrently. In the latter condition, pairs of visual-auditory stimuli were constant and congruent, for example a picture of a bell was always accompanied with the sound of a bell. The performance of the three interventions was further compared with that of a passive control group. These conditions were chosen because they represent commonly used WM training protocols, with visual-only being the most common one, and very few studies include auditory-only WM training (Pergher et al., 2019).

2. Methods

2.1. Participants

Undergraduates from UC Riverside (UCR) and UC Irvine (UCI) were recruited to participate in the study via flyers, emails, and word of mouth between Fall 2018 and Fall 2019. A total of 306 students completed a consent form to participate in the study, 240 of which were subsequently enrolled based on their availability (5 times a week for 1 hour). Participants were randomly assigned to one of 4 groups: 3 active training groups and a passive control group (see Table 1 for demographics). 42 participants were dropped due to attrition or technical errors (see Figure 1): 6 from the Multisensory group, 10 from Visual Only, 15 from Alternating, and 11 from Passive control group. All procedures were approved by UCR and UCI Institutional Review Boards. Participants provided informed consent and received monetary compensation.

Table 1:

Demographics of the four groups.

Multisensory Visual Only Alternating Control
Age (years (SD)) 20.09 (2.57) 20.35 (5.23) 20.98 (3.98) 20.46 (4.86)
Female (%) 58 62 54 62
Male (%) 38 34 44 32
Other or gender unknown (%) 4 4 2 6
N 48 50 50 50
Open in a new tab

Figure 1:

Figure 1:

Open in a new tab

Enrollment

2.2. Procedure

Training and assessments were administered on tablet computers via software developed in-house (“Recollect the Study”; available on Google Play and Apple App Store). In order to prevent fatigue associated with long assessment sessions, participants completed 3 sessions of pre-test and 3 sessions of post-test separated by at least 8 working days. The active groups conducted training tutorials and practice (day 1), short 20-minute training sessions (days 2, 3, 12 and 13) to ease into training, as well as full 40-minute training sessions on days 4–11, for a total of 400 minutes of training (excluding day 1 practice), whereas the passive control group only took part in the assessments (see Figure 2). An added benefit of administering testing sessions over 6 days is management of expectations of the passive control group1 (Green et al., 2019). To further probe expectations, at the end of the study (after the final post-test session), participants responded to the following question using a 5-point Likert scale: “Do you think that the sessions you completed during the study helped you perform better on any of the tasks you completed in the last 3 days?”, wherein 1= Not at all, 2= Not really, 3= Can’t say, 4= Quite a bit, 5= Very much. If the answer selected was 3 or higher, they were also asked to select which task(s), if any, they thought they had performed better on because of earlier sessions.

Figure 2:

Figure 2:

Open in a new tab

Timeline of training and testing for passive and active groups. A 5–7 minute break was given between testing and training blocks and between two 20-minute training blocks.

2.3. Training

The adaptive N-back training game was designed based on principles thought to promote learning (Deveau et al., 2014; Mohammed et al., 2017). In the ‘Visual Only’ condition, all visual stimuli were paired with an identical auditory cue, representing a single N-back training protocol commonly used in the literature (Heinzel et al., 2014; Jaeggi et al., 2010; Küper & Karbach, 2016; Miró-Padilla et al., 2019). An ‘Alternating’ visual/auditory condition consisted of 2 unisensory N-back blocks per day: visual n-back with a placeholder sound and auditory n-back with a visual cue, the order of which was counterbalanced across days (Figure 2). In the ‘Multisensory’ training condition, each N-back visual stimulus type was paired with a different matching sound.

2.4. Transfer Tasks

To assess near transfer, untrained N-back tasks were administered at pre- and post-test, as well as three other WM tasks. Table 2 shows a correlation matrix of baseline performance on transfer tasks. While this dataset is part of a larger study in which a number of other cognitive tasks were administered2, here we test the specific hypothesis that multisensory facilitation improves WM training outcomes. For all assessments, the app provided instructions, examples, as well as performance-gating practice trials with feedback.

Table 2:

Spearman’s rank correlation coefficients of performance on working memory measures at pre-test (N=172).

Sequencing Symmetry Span Corsi Blocks Untrained N-back
Sequencing 1.00
Symmetry Span .25** 1.00
Corsi Blocks 0.10 .27** 1.00
Untrained N-back 0.13 0.03 0.06 1.00
Open in a new tab
**

p < 0.01

Untrained N-back

A visual N-back task with two versions, featuring pictures of animals or pictures of vehicles (counterbalanced across participants), was administered to test generalization to untrained stimuli in a non-game setting. On a given trial, a picture appeared on a black screen for 2500 ms with a 500 ms ISI, and participants were asked to tap the picture if it matched the N-back rule. Each block consisted of 30+N trials with 9 targets (30% target and 30% lure rates). All participants completed N-levels 1–3 but could progress to 4-back (and beyond) if no more than 2 errors were made on the previous level. The main dependent measure was the highest N-level reached at pre-test and post-test.

WM measures

Three WM tasks were used to assess transfer of n-back training. A simple span task, Corsi Blocks, was used to assess WM storage and two complex span tasks, Sequencing and Symmetry Span, were used to measure WM storage-and-processing (Cowan, 2008). In Corsi Blocks, participants viewed a sequence of 12 gray squares turning blue and were asked to reproduce that sequence by tapping on the displayed squares in the same order in which they appeared. Sequencing is a tablet version of the WMS-III Letter-Number Sequencing test (Wechsler 1997), in which participants see a mixed sequence of letters and numbers appearing one by one (e.g. ‘5K8G2’), and are then prompted to enter the numbers in numerical order (‘258’) followed by letters in alphabetical order (‘GK’). A given sequence did not include any of the characters in the previous trial or consecutive numbers and letters. Symmetry Span is a tablet-based version of Automated Symmetry Span (Unsworth, Redick, Heitz, Broadway, & Engle, 2009), in which participants are required to recall sequences of red squares in a 4 × 4 matrix while performing a symmetry judgement task.

In all three span tasks, the ISI was 500 ms while stimulus duration was 1000 ms for Corsi Blocks and Sequencing, and 650 ms in Symmetry Span. All three tasks had 2 trials per set size and started at the lowest set size of 2 items. The next set size (e.g., 3 items) was displayed if at least 1 trial was correct, and the task ended when both trials in a set size were incorrect. The highest possible set size was 10 for Corsi Blocks and 15 for the Symmetry span and Sequencing tasks, ensuring that high performing individuals and/or training-related improvements were not masked by ceiling effects. An individual’s span was defined as the highest set size at which at least 1 of the 2 trials was correct.

2.5. Data Analysis

IBM SPSS Version 24 (IBM corp.) and JASP 0.9.2 (JASP Team, 2019) were used for statistical analyses. For Untrained N-back, data was available for all 198 participants and no outliers were removed based on |z| > 3. For the three span tasks, task data were removed for participants whose individual span was at a minimum (span = 2) either on pre-test or post-test, indicating poor understanding of task demands or inattention. For Corsi blocks, 1 participant had missing data, but no data points were removed. For Sequencing and Symmetry span, data were available for 196 and 191 participants, respectively, 15 of which were removed from each group (see Table 4 for sample sizes per group). The highest span achieved at pre-test was 10 for Corsi Blocks, 9 for Sequencing and 8 for Symmetry Span.

Table 4:

Descriptive statistics and within-group Pearson’s r, Cohen’s d4 and Bayes Factor5 for performance on pre-test and post-test on transfer tests. Significant evidence in favor of the alternative hypothesis is marked with an asterisk (BF10 > 3); strong to extreme evidence is marked bold (BF10 > 10).

Pre-test Post-test Paired comparisons (pre vs. post)
N Mean SD N Mean SD r d BF10
Best N-level
Visual Only 50 3.06 0.24 50 4.02 1.02 −.09 0.90 >1000 *
Alternating V/A 50 3.04 0.20 50 3.86 1.07 .03 0.76 >1000 *
Multisensory 48 3.04 0.20 48 3.75 0.96 .39 0.79 >1000 *
Passive 50 3.06 0.24 50 3.12 0.44 .32 0.14 0.25
Corsi Blocks
Visual Only 50 6.74 1.08 50 6.72 1.13 .34 0.02 0.16
Alternating V/A 50 6.58 1.13 50 6.86 1.41 .59 0.24 0.57
Multisensory 48 6.88 1.02 48 7.06 1.14 .43 0.16 0.28
Passive 50 6.66 0.96 50 6.74 1.61 .36 0.05 0.16
Sequencing Span
Visual Only 46 5.61 1.24 46 5.89 1.32 .49 0.22 0.44
Alternating V/A 46 5.61 1.47 46 6.17 1.60 .55 0.39 3.43*
Multisensory 42 5.26 1.23 42 5.98 1.47 .47 0.51 16.30 *
Passive 47 5.36 1.05 47 5.70 1.25 .50 0.29 0.98
Symmetry Span
Visual Only 44 4.80 1.25 44 4.84 1.12 .19 0.03 0.17
Alternating V/A 44 4.82 1.21 44 4.71 1.59 .38 0.07 0.18
Multisensory 40 4.45 1.01 40 5.13 1.34 .56 0.59 49.05 *
Passive 48 4.44 1.22 48 4.69 1.08 .33 0.19 0.35
Open in a new tab
*

= p < 0.05, BF10 < 1 = Evidence for H0

Gain scores were obtained by subtracting pretest from posttest scores. Since the gain scores were not normally distributed, nonparametric tests were conducted on all transfer tasks. As a first step, we investigated whether the gain itself (pre- vs. post- test change) was significant in any of the groups. If at least one group showed significant gain, Kruskal-Wallis H tests were used to investigate how the four groups, on average, differed in gains, followed by Mann-Whitney U-tests.

3. Results

3.1. Training

Participants in the Visual Only and Multisensory groups showed similar training progress, outperforming the Alternating group towards the end of training. In Figure 3, visual and auditory N-back blocks of the Alternating group are shown separately. The highest N-level gain, calculated as average N-level in the last session minus the average N-level in the first session, was observed for the Multisensory group (M = 1.95, SD = 0.21), followed by the Visual Only (M = 1.75, SD = 0.18) and Alternating group (M = 0.99, SD = 0.12). An independent-samples Kruskal-Wallis test showed that the 3 groups significantly differed in terms of N-level gain (H(2) = 15.82, p < 0.001). Pairwise comparisons with adjusted p-values showed that there was no difference in training gain between Multisensory and Visual Only groups (p = 1.0, r3 = 0.03), in contrast, the Multisensory group achieved significantly higher N-level gain than the Alternating group (p = 0.001, r = 0.36) and the same pattern was observed for the Visual Only group (p = 0.003, r = 0.33).

Figure 3:

Figure 3:

Open in a new tab

Mean N-level achieved on a given day, split by visual and auditory blocks for the Alternating group. A dip on day 8 reflects transition from less abstract stimuli to more abstract stimuli (equivalent across groups).

3.2. Transfer

There is no evidence that the 4 groups differed significantly at baseline on any of the measures as indicated by Kruskal-Wallis one-way tests, which was expected given the random assignment to groups (see Table 3).

Table 3:

Kruskal-Wallis H tests to assess potential group differences at pretest.

Pretest measure Kruskal-Wallis H test
Best N-level H(3) = 0.38, p = .94
Corsi Blocks H(3) = 3.68, p = .30
Sequencing Span H(3) = 2.69, p = .44
Symmetry Span H(3) = 5.05, p = .17
Open in a new tab

Untrained n-back.

Wilcoxon Signed Ranks Tests demonstrated that as expected, all groups except the control group reached significantly higher N-levels on the untrained tasks at post-test relative to pretest (Visual Only: Z = −4.89, p = < 0.001; Alternating: Z = −4.36, p = < 0.001; Multisensory: Z = −4.12, p = < 0.001; Control: Z = −1.00, p = 0.32). The groups differed significantly in N-level gain as shown by a Kruskal-Wallis test (H(3) = 31.07, p < 0.001; see Figure 4A): Visual Only vs. Control (U = 568.50, z = −5.42, p = < 0.001, r = −0.54), Alternating vs. Control (U = 711.00, z = −4.51, p = < 0.001, r = −0.45), and Multisensory vs. Control (U = 730.00, z = −4.26, p = < 0.001, r = −0.43). In the active sample (N = 148), N-level gain throughout training showed a significant positive correlation with pre/post N-level gain on untrained N-back tasks (rho = .45, p < 0.001), which was also observed separately for each group ( Visual Only: rho = .41, p = 0.003; Alternating: rho = .30, p = 0.03; Multisensory: rho = .59, p < 0.001).

Figure 4:

Figure 4:

Open in a new tab

Difference scores for each group: V = Visual Only, A = Alternating Audio-visual, M = Multisensory, and C = Control (Passive); color coded as red, blue, green and gray, respectively. (A) Difference between highest N-level achieved at post-test relative to pre-test on an untrained visual N-back task. (B-D) Change in span on Forward Corsi Blocks, Sequencing, and Symmetry Span, respectively. Error bars represent S.E.M. * p <0.05, ** p < 0.01. Stars without lines indicate significant difference at post-test relative to pre-test in each group (Wilcoxon signed-rank test), whereas stars with lines indicate significant differences in change scores in one group relative to another (Kruskal-Wallis test).

Corsi Blocks.

We next examined transfer to Corsi Blocks Forward, a measure of WM capacity. The change from pre- to post-test on span was not significant in any of the groups as indicated by Wilcoxon Signed Rank Tests (Visual Only: Z = −0.30, p = .77; Alternating: Z = −1.76 p = .08; Multisensory: Z = −1.17, p = .24; Control: Z = −1.15, p = .25) hence no further analyses were conducted (Figure 4B).

Sequencing.

Wilcoxon Signed Rank Tests demonstrated significant gain in sequencing span for Alternating (Z = −2.21, p = .03) and Multisensory (Z = −2.85, p = .004) groups whereas Visual Only (Z = −1.37, p = .17) and Control groups (Z = −1.84, p = .07) did not show significant gain (Figure 4C). Overall, the groups did not differ significantly in span gain (H(3) = 4.27, p = 0.23); however, there was a trend for the Multisensory group to show larger span gain compared to the Visual Only group (U = 759.50, z = −1.78, p = 0.075, r = −0.19) and to the Control group (U = 778.50, z = −1.77, p = 0.077, r = −0.19). Training gain did not predict span gain (rho = .01, p = .96) in the active sample (N = 134).

Symmetry Span.

The only group that showed significant improvement in symmetry span was the Multisensory group (Z = −3.31, p = .001), whereas the other groups did not improve (Visual Only: Z = −0.16, p = .88; Alternating: Z = −0.58, p = .56; Control: Z = −1.34, p = .18). While a Kruskal-Wallis test did not show significant differences in span gain among the groups (H(3) = 6.33, p = 0.10), a comparison of gain in the three active groups was marginally significant (H(2) = 5.96, p = 0.05), prompting comparisons between pairs of active groups. As can be seen in Figure 4D, the multisensory group showed higher span gain than the Visual Only group (U = 667.50, z = −1.96, p = 0.05, r = −0.25) and the Alternating group (U = 634.00, z = −2.26, p = 0.03, r = −0.21), albeit the effect sizes are small. Moreover, training N-level gain showed a significant positive correlation with symmetry span gain in the active sample (rho = .23, p = 0.01, N = 128); however, this was not observed for the individual groups (Visual Only: rho = .11, p = 0.47; Alternating: rho = .20, p = 0.20; Multisensory: rho = .25, p = 0.13).

3.3. Self-reported transfer

Out of 182 participants that answered the question “Do you think that the sessions you completed during the study helped you perform better on any of the tasks you completed in the last 3 days?”, the most common answer across all four groups was 4 (“Quite a bit”). The highest average score was reported by the Multisensory group (M = 3.74, SD = 0.93, N = 46), followed closely by Visual Only (M = 3.71, SD = 0.76, N = 45), Alternating (M = 3.52, SD = 0.85, N = 48), and the Passive control group (M = 3.47, SD = 0.80, N = 43). A Kruskal-Wallis H test indicated that the four groups did not show a statistically significant difference in their answer to this question (H(3) = 5.33, p = .15). These results provide tentative evidence that the four groups perceived a similar rate of change in post-test performance as a result of participating in the study (even the passive control group). Figure 5 shows the percentage of participants per group that selected a given task as one in which they thought they had improved on as a result of participating in the study. Note that this question was only presented to participants who answered 3 (“Can’t say”) or higher on the previous question, i.e. to 163 participants (89.6%) and that they were not required to select any tasks.

Figure 5:

Figure 5:

Open in a new tab

Self-reported improvement on tasks at post-test as a result of earlier sessions, shown as percentage per group; V = Visual Only, A = Alternating Audio-visual, M = Multisensory, and C = Control (Passive).

4. Discussion

Based on the rich literature on benefits of multisensory exposure to learning and memory (Matusz, Wallace, & Murray, 2017; Quak et al., 2015; Shams & Seitz, 2008), it was hypothesized that WM training could be facilitated by introducing multisensory objects in an n-back training task. In order to test this, performance on multisensory n-back training was contrasted against visual-only and alternating audio-visual n-back training, as well as a passive control group. Results showed that participants in the Multisensory group had equal N-level gain compared to Visual Only training yet showed significant transfer to some untrained WM tasks, whereas the Visual Only condition did not. This is in line with studies suggesting that multisensory experience can facilitate memory (Thelen, Matusz, & Murray, 2014) and learning (Shams & Seitz, 2008). In fact, it has been demonstrated that even a single multisensory exposure can influence memory for visual and auditory objects (Matusz et al., 2017). Multisensory cues can also improve delayed retention of incidental learning (Broadbent, Osborne, Mareschal, & Kirkham, 2019) and aid perceptual learning (Seitz, Kim, & Shams, 2006). The main question of interest here, however, is whether multisensory training can boost near transfer.

As expected, all three active training groups improved on an untrained N-back task compared to the control group. Training N-level gain predicted transfer to untrained N-back tasks in the active sample irrespective of condition. There was no evidence of training-related enhancement of WM capacity as measured by a simple span task for any of the groups, possibly due to the higher capacity used in this task compared to that experienced during training (e.g. typical N-back span was 4 whereas typical Corsi span was 7). However, the Multisensory group showed significant performance improvement on complex span tasks, particularly symmetry span, and to some extent, sequencing. Training on a task that requires rapid updating of temporary bindings in WM may have strengthened the processes involved in manipulation of information that are crucial for performance on complex span tasks. Indeed, research indicates that updating tasks, complex span tasks, and binding tasks share a large proportion of variance (Wilhelm, Hildebrandt, & Oberauer, 2013).

We note that a key component of the Multisensory training employed here is that the auditory and visual stimuli at each time-point supported the same task-response. This is a key element of multisensory facilitation (Shams & Seitz, 2008) and is in contrast to dual WM tasks in which participants have to perform a secondary task while maintaining information in WM (Fougnie & Marois, 2011), or dual N-back tasks that require updating of two separate streams of stimuli, one in the visual and the other in the auditory domain. We suggest that while these dual tasks may give rise to some benefits of multi-tasking (Anguera et al., 2013); they put the senses at competition with each-other and don’t give rise to facilitation (Deveau et al., 2014).

While the current study examined transfer to a variety of visual working memory tasks, we note that the question of transfer to auditory or multisensory working memory tasks has not been addressed. Further work will be required to test the hypothesis that the multisensory condition would also be advantageous to transfer to working memory tasks that involve auditory, and maybe even other stimulus modalities. We also acknowledge a limitation in that we did not directly assess placebo effects and/or enjoyment; however, post-test self-report results indicated that the four groups perceived a similar rate of transfer overall, meaning that even the passive control group felt some improvement – likely due to retest effects. When asked whether they observed improvement on specific tasks, the pattern of results mirrored the measured transfer effects, indicating good insight into individual outcomes of the study (Tsai et al., 2018). At present there is extremely limited and mixed data on whether expectation-based effects can drive transfer effects (Green et al., 2019), hence, this should be addressed by future research. A further limitation of the study is the use of a passive control group. While passive control groups only control for retest effects, active control groups can help manage participants’ expectancies, equalize participant-experimenter contact time, and reduce demand characteristics (Boot, Simons, Stothart, & Stutts, 2013). On the other hand, designing an appropriate control training protocol that does not rely on WM is no trivial task, and certainly not as easy as administering a placebo drug. Furthermore, a recent meta-analysis did not find any evidence that control group type meaningfully impacts effect sizes from cognitive measures (Au, Gibson, Bunarjo, Buschkuehl, & Jaeggi, 2020). Even though the use of active control groups is still strongly recommended, research involving passive control groups may still be informative and useful, especially in early, proof-of-concept studies such as this one (cf. Green et al., 2020).

In summary, the present results suggest that incorporating multisensory objects in a WM training protocol can benefit performance on the training task compared to training WM in each sense separately and that multisensory training can potentially facilitate transfer to complex WM span tasks. To gain a better understanding of multisensory facilitation of WM training, future research should include an auditory-only training group, auditory WM transfer tests, and explore whether multisensory facilitation affects far transfer tasks as well. Evidence of multisensory facilitation of WM training could inform development of training protocols, particularly those targeting older adults, where supporting learning via multiple senses may be advantageous to those who experience transient deficits in either sense.

Acknowledgments

Funding: This work has been supported by the National Institute of Mental Health (Grant No. 1R01MH111742) and by the National Science Foundation (Grant No. BCS-1057625) to ARS and SMJ, and in addition, SMJ is supported by the National Institute on Aging (Grant No. 1K02AG054665).

Footnotes

Conflict of Interest: On behalf of all authors, the corresponding author states that there is no conflict of interest.

2

Three additional tasks were administered (math test, countermanding task and matrix reasoning) that do not assess WM, therefore, they were not included in this paper. Furthermore, some of the tasks were not consistent across conditions. Those tasks were included for pilot purposes in preparation for a larger scale study in which we are targeting 30,000 participants and a wide range of training conditions and outcome measurers (https://www.scientificamerican.com/article/does-brain-training-actually-work/).

3

r is an effect size estimate calculated as z /√N

4

Accounts for the correlation between pre- and post-test measures: (MeanPost−MeanPre)/(SDPre2+SDPost2−2rPrePost∗SDPre∗SDPost)

5

BF10 grades the intensity of the evidence that the data provide for H1 versus H0. BF10 between 1 and 3 is considered to be only anecdotal evidence for H1; 3–10: moderate evidence; 10–30: strong evidence; 30–100: very strong evidence; >100 extreme evidence (Wagenmakers et al., 2018).

Publisher's Disclaimer: This Author Accepted Manuscript is a PDF file of a an unedited peer-reviewed manuscript that has been accepted for publication but has not been copyedited or corrected. The official version of record that is published in the journal is kept up to date and so may therefore differ from this version.

Bibliography

  1. Anguera JA, Boccanfuso J, Rintoul JL, Al-Hashimi O, Faraji F, Janowich J, … Gazzaley A (2013). Video game training enhances cognitive control in older adults. Nature, 501(7465), 97–101. doi: 10.1038/nature12486 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Au J, Gibson BC, Bunarjo K, Buschkuehl M, & Jaeggi SM (2020). Quantifying the Difference Between Active and Passive Control Groups in Cognitive Interventions Using Two Meta-analytical Approaches. Journal of Cognitive Enhancement, 4(2), 192–210. doi: 10.1007/s41465-020-00164-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Au J, Sheehan E, Tsai N, Duncan GJ, Buschkuehl M, & Jaeggi SM (2015). Improving fluid intelligence with training on working memory: a meta-analysis. Psychonomic Bulletin & Review, 22(2), 366–377. doi: 10.3758/s13423-014-0699-x [DOI] [PubMed] [Google Scholar]
  4. Boot WR, Simons DJ, Stothart C, & Stutts C (2013). The Pervasive Problem With Placebos in Psychology: Why Active Control Groups Are Not Sufficient to Rule Out Placebo Effects. Perspectives on Psychological Science : A Journal of the Association for Psychological Science, 8(4), 445–454. doi: 10.1177/1745691613491271 [DOI] [PubMed] [Google Scholar]
  5. Broadbent HJ, Osborne T, Mareschal D, & Kirkham NZ (2019). Withstanding the test of time: Multisensory cues improve the delayed retention of incidental learning. Developmental Science, 22(1), e12726. doi: 10.1111/desc.12726 [DOI] [PubMed] [Google Scholar]
  6. Cowan N (2008). What are the differences between long-term, short-term, and working memory? Progress in Brain Research, 169, 323–338. doi: 10.1016/S0079-6123(07)00020-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Cowan N, Li D, Moffitt A, Becker TM, Martin EA, Saults JS, & Christ SE (2011). A neural region of abstract working memory. Journal of Cognitive Neuroscience, 23(10), 2852–2863. doi: 10.1162/jocn.2011.21625 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Dahlin E, Neely AS, Larsson A, Bäckman L, & Nyberg L (2008). Transfer of learning after updating training mediated by the striatum. Science, 320(5882), 1510–1512. doi: 10.1126/science.1155466 [DOI] [PubMed] [Google Scholar]
  9. Deveau J, Jaeggi SM, Zordan V, Phung C, & Seitz AR (2014). How to build better memory training games. Frontiers in Systems Neuroscience, 8, 243. doi: 10.3389/fnsys.2014.00243 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Diaconescu AO, Alain C, & McIntosh AR (2011). The co-occurrence of multisensory facilitation and cross-modal conflict in the human brain. Journal of Neurophysiology, 106(6), 2896–2909. doi: 10.1152/jn.00303.2011 [DOI] [PubMed] [Google Scholar]
  11. Forsberg A, Fellman D, Laine M, Johnson W, & Logie RH (2020). Strategy mediation in working memory training in younger and older adults. Quarterly Journal of Experimental Psychology, 73(8), 1206–1226. doi: 10.1177/1747021820915107 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Fougnie D, & Marois R (2011). What limits working memory capacity? Evidence for modality-specific sources to the simultaneous storage of visual and auditory arrays. Journal of Experimental Psychology. Learning, Memory, and Cognition, 37(6), 1329–1341. doi: 10.1037/a0024834 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Ghazanfar AA, & Schroeder CE (2006). Is neocortex essentially multisensory? Trends in Cognitive Sciences, 10(6), 278–285. doi: 10.1016/j.tics.2006.04.008 [DOI] [PubMed] [Google Scholar]
  14. Green S, Bavelier D, Kramer AF, Vinogradov S, Ansorge U, Ball KK, … Witt CM (2019). Improving methodological standards in behavioral interventions for cognitive enhancement. Journal of Cognitive Enhancement, 3(1), 1–28. doi: 10.1007/s41465-018-0115-y [DOI] [Google Scholar]
  15. Heikkilä J, Alho K, & Tiippana K (2017). Semantically congruent visual stimuli can improve auditory memory. Multisensory Research, 30(7–8), 639–651. doi: 10.1163/22134808-00002584 [DOI] [PubMed] [Google Scholar]
  16. Heinzel S, Schulte S, Onken J, Duong Q-L, Riemer TG, Heinz A, … Rapp MA (2014). Working memory training improvements and gains in non-trained cognitive tasks in young and older adults. Neuropsychology, Development, and Cognition. Section B, Aging, Neuropsychology and Cognition, 21(2), 146–173. doi: 10.1080/13825585.2013.790338 [DOI] [PubMed] [Google Scholar]
  17. Holmes J, Woolgar F, Hampshire A, & Gathercole SE (2019). Are Working Memory Training Effects Paradigm-Specific? Frontiers in Psychology, 10, 1103. doi: 10.3389/fpsyg.2019.01103 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Jaeggi SM, Buschkuehl M, Shah P, & Jonides J (2014). The role of individual differences in cognitive training and transfer. Memory & Cognition, 42(3), 464–480. doi: 10.3758/s13421-013-0364-z [DOI] [PubMed] [Google Scholar]
  19. Jaeggi SM, Studer-Luethi B, Buschkuehl M, Su Y-F, Jonides J, & Perrig WJ (2010). The relationship between n-back performance and matrix reasoning — implications for training and transfer. Intelligence, 38(6), 625–635. doi: 10.1016/j.intell.2010.09.001 [DOI] [Google Scholar]
  20. JASP Team. (2019). JASP 0.9.2 (0.10.2). Computer software, Computer software. [Google Scholar]
  21. Kim RS, Seitz AR, & Shams L (2008). Benefits of stimulus congruency for multisensory facilitation of visual learning. Plos One, 3(1), e1532. doi: 10.1371/journal.pone.0001532 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kim R, Seitz AR, & Shams L (2008). Neural mechanisms of multisensory perceptual learning. Journal of Vision, 8(6), 976a. [Google Scholar]
  23. Küper K, & Karbach J (2016). Increased training complexity reduces the effectiveness of brief working memory training: evidence from short-term single and dualn -back training interventions. Journal of Cognitive Psychology, 28(2), 199–208. doi: 10.1080/20445911.2015.1118106 [DOI] [Google Scholar]
  24. Lehmann S, & Murray MM (2005). The role of multisensory memories in unisensory object discrimination. Brain Res Cogn Brain Res, 24(2), 326–34. [DOI] [PubMed] [Google Scholar]
  25. Matusz PJ, Wallace MT, & Murray MM (2017). A multisensory perspective on object memory. Neuropsychologia, 105, 243–252. doi: 10.1016/j.neuropsychologia.2017.04.008 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Melby-Lervåg M, Redick TS, & Hulme C (2016). Working Memory Training Does Not Improve Performance on Measures of Intelligence or Other Measures of “Far Transfer”: Evidence From a Meta-Analytic Review. Perspectives on Psychological Science : A Journal of the Association for Psychological Science, 11(4), 512–534. doi: 10.1177/1745691616635612 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Miró-Padilla A, Bueichekú E, Ventura-Campos N, Flores-Compañ M-J, Parcet MA, & Ávila C (2019). Long-term brain effects of N-back training: an fMRI study. Brain Imaging and Behavior, 13(4), 1115–1127. doi: 10.1007/s11682-018-9925-x [DOI] [PubMed] [Google Scholar]
  28. Mohammed S, Flores L, Deveau J, Cohen Hoffing R, Phung C, M. Parlett C, … R. Seitz A (2017). The benefits and challenges of implementing motivational features to boost cognitive training outcome. Journal of Cognitive Enhancement, 1(4), 491–507. doi: 10.1007/s41465-017-0047-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Murray MM, Lewkowicz DJ, Amedi A, & Wallace MT (2016). Multisensory Processes: A Balancing Act across the Lifespan. Trends in Neurosciences, 39(8), 567–579. doi: 10.1016/j.tins.2016.05.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Gonzalez Andino S, & Schnider A (2004). Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage, 21(1), 125–135. doi: 10.1016/j.neuroimage.2003.09.035 [DOI] [PubMed] [Google Scholar]
  31. Pergher V, Shalchy MA, Pahor A, Van Hulle MM, Jaeggi SM, & Seitz AR (2019). Divergent research methods limit understanding of working memory training. Journal of Cognitive Enhancement, 1–21. doi: 10.1007/s41465-019-00134-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Quak M, London RE, & Talsma D (2015). A multisensory perspective of working memory. Frontiers in Human Neuroscience, 9, 197. doi: 10.3389/fnhum.2015.00197 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Schwaighofer M, Fischer F, & Bühner M (2015). Does Working Memory Training Transfer? A Meta-Analysis Including Training Conditions as Moderators. Educational Psychologist, 50(2), 138–166. doi: 10.1080/00461520.2015.1036274 [DOI] [Google Scholar]
  34. Seitz AR, Kim R, & Shams L (2006). Sound facilitates visual learning. Current Biology, 16(14), 1422–1427. doi: 10.1016/j.cub.2006.05.048 [DOI] [PubMed] [Google Scholar]
  35. Shams L, & Seitz AR (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12(11), 411–417. doi: 10.1016/j.tics.2008.07.006 [DOI] [PubMed] [Google Scholar]
  36. Shams L, Wozny DR, Kim R, & Seitz A (2011). Influences of multisensory experience on subsequent unisensory processing. Frontiers in Psychology, 2, 264. doi: 10.3389/fpsyg.2011.00264 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Soveri A, Antfolk J, Karlsson L, Salo B, & Laine M (2017). Working memory training revisited: A multi-level meta-analysis of n-back training studies. Psychonomic Bulletin & Review, 24(4), 1077–1096. doi: 10.3758/s13423-016-1217-0 [DOI] [PubMed] [Google Scholar]
  38. Thelen A, Matusz PJ, & Murray MM (2014). Multisensory context portends object memory. Current Biology, 24(16), R734–R735. doi: 10.1016/j.cub.2014.06.040 [DOI] [PubMed] [Google Scholar]
  39. Thelen A, Talsma D, & Murray MM (2015). Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition, 138, 148–160. doi: 10.1016/j.cognition.2015.02.003 [DOI] [PubMed] [Google Scholar]
  40. Tsai N, Buschkuehl M, Kamarsu S, Shah P, Jonides J, & Jaeggi SM (2018). (Un)great expectations: the role of placebo effects in cognitive training. Journal of Applied Research in Memory and Cognition, 7(4), 564–573. doi: 10.1016/j.jarmac.2018.06.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Unsworth N, Redick TS, Heitz RP, Broadway JM, & Engle RW (2009). Complex working memory span tasks and higher-order cognition: a latent-variable analysis of the relationship between processing and storage. Memory, 17(6), 635–654. doi: 10.1080/09658210902998047 [DOI] [PubMed] [Google Scholar]
  42. von Kriegstein K, & Giraud A-L (2006). Implicit multisensory associations influence voice recognition. PLoS Biology, 4(10), e326. doi: 10.1371/journal.pbio.0040326 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Wagenmakers E-J, Love J, Marsman M, Jamil T, Ly A, Verhagen J, … Morey RD (2018). Bayesian inference for psychology. Part II: Example applications with JASP. Psychonomic Bulletin & Review, 25(1), 58–76. doi: 10.3758/s13423-017-1323-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Weicker J, Villringer A, & Thöne-Otto A (2016). Can impaired working memory functioning be improved by training? A meta-analysis with a special focus on brain injured patients. Neuropsychology, 30(2), 190–212. doi: 10.1037/neu0000227 [DOI] [PubMed] [Google Scholar]
  45. Wilhelm O, Hildebrandt A, & Oberauer K (2013). What is working memory capacity, and how can we measure it? Frontiers in Psychology, 4, 433. doi: 10.3389/fpsyg.2013.00433 [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES