Consistent Shifts of Stimulus Modality Induce Chunking in Sequence Learning

Abstract

The ability to learn event sequences is crucial to human behavior because it enables us to interact adaptively with our environment. The sensory environment is essential in guiding the acquisition of these sequences, so the role of the stimulus modality in sequence learning is of high relevance. The present study examined structured stimulus modality shifts in sequence learning using the serial reaction time task (SRT). Participants had to respond to numbers that were presented either in the visual or in the auditory modality. Importantly, the numbers, as well as the stimulus modality, followed a fixed pattern. We found better performance in sequenced than in random blocks, indicating sequence learning. Moreover, the performance was better when the stimulus modality remained the same than when the modality changed between successive trials (the modality shift effect, MSE). However, sequence learning facilitated performance primarily in modality repetitions, so that the MSE became progressively larger in the sequenced blocks, while it was small in the random blocks, and this pattern was particularly pronounced for the participants who showed a high recall level for the response sequence. To account for this effect, we assume that consistent modality shifts induce parsing of the sequence into chunks. Because chunk retrieval at chunk boundaries incurs RT costs, the acquired sequence knowledge is more efficiently expressed in modality repetitions (i.e., within chunks). Together, the data suggest that the formation of explicit knowledge enhances chunking in sequence learning. 

Introduction

The ability to learn complex sequences of actions is crucial to human behavior because it enables us to interact with our environment by learning motor skills (Abrahamse, Jiménez, Verwey, & Clegg, 2010; Clegg, DiGirolamo, & Keele, 1998; Lashley, 1951). Conversely, our sensory environment essentially guides the acquisition of action sequences. For instance, people often learn skills by watching and imitating others or by listening to instructions (i.e., observational learning, see, e.g., Willingham, 1998). Since our sensory environment is rather complex, we are constantly perceiving, processing, and integrating information from different modalities, and we are forced to switch between them (e.g., Abrahamse, van der Lubbe, & Verwey, 2008); Abrahamse, van der Lubbe, & Verwey, 2009; Lewkowicz & Ghazanfar, 2009; see also Wahn & König, 2017, for a more recent generic review of multisensory processing). The aim of the present study was to explore the interaction of structured modality shifting and sequence learning by using the serial reaction time (SRT) task (Nissen & Bullemer, 1987).

The SRT Task and Sequence Knowledge: Differentiating Implicit and Explicit Learning

In the SRT task, participants respond to successive stimuli by performing corresponding responses, while the sequence of stimuli or responses (or both) follows a certain pattern. In order to separate sequence learning and concurrently occurring general practice effects when performing motor sequences, the sequence-specific learning effect is often measured as the performance difference between blocks with a fixed sequence versus blocks with a random sequence of stimuli (see, e.g., Abrahamse et al. 2010; Dienes & Berry, 1997; Keele, Ivry, Mayr, Hazeltine, & Heuer, 2003; Schwarb & Schumacher, 2012, for reviews). Typically, this indirect measure of sequence learning yields robust sequence learning effects even if participants report no explicit awareness of the sequence (see, e.g., Dienes & Berry, 1997; Keele et al., 2003; Shanks & St John, 1994, for reviews).

It has been demonstrated that the awareness of the sequence (implicit compared to explicit learning) influences sequence learning (see Shanks, 2005, for a review). Implicit learning may be operationally defined as the acquisition of knowledge while this knowledge is difficult to express, whereas explicit learning “can be communicated or demonstrated on demand” (Berry & Dienes, 1993). There is an ongoing debate on how to appropriately assess whether learning took place implicitly or explicitly. Interviews in which participants were asked to reproduce the sequence have been shown to not exhaustively measure the available knowledge (Shanks & St John, 1994; Shanks, 2005). To overcome these drawbacks, more sensitive tasks, like the generation and recognition tasks were introduced. Nevertheless, even with these more sophisticated instruments, the assessment of whether learning took place implicitly or explicitly remains challenging (Destrebecqz & Cleeremans, 2001). Thus, the classification of learning as having occurred implicitly or explicitly remains a tricky methodological issue.

However, earlier studies examining sequence learning and sequence reproduction demonstrated facilitated learning for the participants classified as explicit learners compared to implicit learners, not only in terms of generally improved learning but also in terms of specific interactions, suggesting that the distinction between implicit and explicit learning can involve even some qualitative changes in processing. For example, Zirngibl and Koch (2002) compared sequence learning with vocal and manual responses and found that explicit learning facilitated learning with vocal responses much more than with manual responses. In another study, Koch, Philipp, and Gade (2006) investigated the influence of task sequences on task inhibition (in n−2 task repetitions) for two groups of participants. While one group was explicitly informed about the task sequence, the other group was not. Interestingly, they found reduced task inhibition in the sequence relative to the random blocks primarily for those participants who were explicitly informed about the task sequence. This finding was interpreted in terms of chunking based on these task repetitions, so that n−2 task repetitions were embedded in a chunk. Since performance within a chunk is facilitated, the chunking process should facilitate reactions in n−2 repetitions and thus decrease task inhibition, as found by the authors.

Sequence Learning and Chunking

A large body of studies suggest chunking as an important mechanism in sequence learning (e.g., Chan, Immink, & Lushington, 2017; Jiménez, Méndez, Pasquali, Abrahamse, & Verwey, 2011; Kirsch, Sebald, & Hoffmann, 2010; Koch & Hoffmann, 2000; Sakai, Kitaguchi, & Hikosaka, 2003; Stadler, 1993, Stadler, 1995). As a general definition (e.g., Gobet et al., 2001), a chunk is a group of elements, and these elements are strongly associated with each other, while they are only weakly associated with elements of other groups (Chase & Simon, 1973; Cowan, 2001; Simon, 1974). According to the self-organizing consciousness framework (e.g. Perruchet & Gallego, 1997; Perruchet & Vinter, 2002), at the beginning of the learning process, one attends to single elements, or primitive units, and in the course of learning, successive primitive units are concatenated to larger units or chunks. In many respects, these larger units act as primitive units and can themselves become part of even larger units, so that progressively larger chunks are created (Perruchet & Gallego, 1997). Typically, patterns of chunking are identified by looking at response latencies during the performance of the sequence, because reactions within chunks are supposed to be facilitated and thus faster than reactions at chunk boundaries (e.g. Koch et al., 2006; Nissen & Bullemer, 1987; Song & Cohen, 2014).

Stadler (1993) showed that the insertion of regular pauses into the stimulus sequence improved sequence learning. He explained this finding by enhanced parsing of the sequence due to pauses. In other words, the insertion of pauses enhanced the chunking of the sequence into subsequences. Moreover, it was demonstrated that not only pauses, but also certain relational structures in the sequence, like ascending and descending runs (e.g., 123–321), enhanced learning, suggesting a facilitated formation of chunks due to induced sequence structure (Kirsch et al., 2010; Koch & Hoffmann, 2000). Furthermore, Jiménez et al. (2011) demonstrated facilitated learning when using color cues that consistently demarcated subsequences. Together, these results suggest that the integration of subsequences into larger units was facilitated when sequence structure was induced by additional features, such as temporal, relational, and color transition patterns.

Structured Modality Shifting and Chunking

Similarly to the studies presented in the preceding paragraph, in the present study, we induced additional sequence structure by using a cross-modal stimulus sequence with consistent, predictable modality shifts, which, in turn, should enhance the formation of chunks. This is where the modality shift effect (MSE), a well-established cross-modal attention phenomenon, becomes relevant.

The MSE denotes the phenomenon in which participants respond more slowly when the modality just changed compared to when it remained the same (e.g., Driver & Spence, 1998; Spence & Driver, 1997). The mechanism underlying the MSE is not fully understood yet. However, Töllner, Gramann, Müller, and Eimer (2009) proposed a modality-weighting account (MWA), which assumes a weighting mechanism for perceptual modalities, where the total weight is limited and weighting of one modality results in a facilitated processing of stimuli in this modality compared to stimuli in other modalities. More specifically, this facilitation involves an enhanced coding of relevant stimuli and/or the transmission of modality-specific stimulus information to a cross-modal stage of processing. When a modality shift occurs, initially, the wrong modality is weighted, resulting in a time-consuming reorientation of attentional weight from the old to the new stimulus modality in order to enhance the stimulus signal and its salience at the cross-modal processing stage.

In our study, we expected to observe a MSE and its interaction with sequence learning. That is, the performance benefit of modality repetitions compared to shifts should be enhanced in the sequenced blocks, because the structured modality sequence should induce a parsing of the response sequence, in which modality shifts act as a boundary for chunk formation and modality repetitions are chunked together (see Koch et al., 2006; Rosenbaum, Kenny, & Derr, 1983; Schneider & Logan, 2015). To be more specific, whereas the time-consuming shifting of attentional weight (as proposed by Töllner et al., 2009) and the activation of a new chunk slow down performance during modality shifts, there should be a benefit for modality repetitions due to enhanced chunking and facilitated responses within chunks.¹ Moreover, to the degree that this chunking process is facilitated with high sequence recall, as it has been demonstrated in earlier studies (e.g. Koch et al., 2006; Koch, 2007; Schneider & Logan, 2015), we should also find that this interaction of modality shifting and sequence learning is larger with high compared to low sequence recall. In other words, individuals who are able to recall large parts of the sequence should exhibit a particularly enlarged MSE for the sequence trials relative to random trials compared to those individuals who show low sequence recall.

Method

Participants 

Thirty-two undergraduates (28 females and four males; M_age = 21.4 ± 3 years) from RWTH Aachen University participated in the experiment and received partial course credit in exchange. Before participation, they provided written informed consent according to the Declaration of Helsinki. 

Stimuli and Tasks   

The experiment was programmed in E-Prime 2.0 (Schneider, Eschman, & Zuccolotto, 2002). As stimuli, we employed the numbers 1, 2, 3, and 4 and presented them either (a) visually as digits in the center of the computer screen and with a height of 2.5 cm or (b) auditorily via headphones, spoken by a female voice and presented with a sound pressure level of 65 dB. Participants were instructed to press v with the index finger of their right hand if there was a visual or auditory 1, b with the middle finger for a 2, n with the ring finger for a 3, and m with the little finger for a 4 on a German QWERTZ computer keyboard.  

Procedure  

Participants were informed that the numbers would either be presented visually or auditorily and about how to respond to them. They were instructed to respond as fast and accurately as possible. The visual stimulus was presented until the participant pressed a key and the auditory stimulus (with a mean duration of 613 ms) was repeated until a response was made. If the response was correct, a white screen was presented for 500 ms, whereas if the response was incorrect, there was a feedback signal presented for 200 ms (visual ups, German for “oops,” in visual trials, auditory “oops” in auditory trials) followed by a white screen for another 300 ms. Accordingly, we used a constant responsestimulus interval (RSI) of 500 ms, because changes in RSI have been shown to affect learning (e.g., Stadler, 1995). The experiment began with a practice block of 12 trials with random number and modality sequences, followed by ten experimental blocks consisting of 72 trials each. Two types of sequences, a number sequence and a modality sequence, were present during the experiment, while no mention was made of any sequence to be learned. There was a repeating number sequence in Blocks 3 to 8 and Block 10, whereas the numbers were pseudorandom in Blocks 1, 2, and 9. Similarly, there was a repeating modality sequence in Blocks 3 to 8 and in Block 10, whereas the modality sequence was random in Blocks 1 and 2. In the random number Block 9, the fixed modality sequence was maintained for half of the participants, whereas it was randomized for the other half². Two 12-number response sequences (241232143134 and 234124213143, adopted from (Zirngibl & Koch, 2002) were counterbalanced across participants and presented in Blocks 3 to 8 and Block 10. None of them contained repetitions or runs of four elements, and they were identical with regard to the number of reversals and runs. The pseudorandom number sequences in Blocks 1, 2, and 9 followed the same constraints and contained no immediate repetitions and no fourelement runs, while each number was as likely to occur as the other. In Blocks 3 to 8 and Block 10, there were four different modality sequences that were counterbalanced across participants. They differed with regard to whether the sequence started with a visual (V) or an auditory (A) stimulus and whether it started with a modality repetition or a shift. The sequences were counterbalanced with regard to the number of visual and auditory trials and regarding the number of repetitions and shifts. There were six visual and six auditory trials as well as six modality repetitions and six modality shifts in each modality sequence. The four sequences were: V-A-V-V-V-A-A-A-V-A-A-V, A-V-A-A-A-V-V-V-A-V-VA, V-V-V-A-A-A-V-V-A-A-V-A, and A-AA-V-V-V-A-A-V-V-A-V. The pseudorandom modality sequence in the other blocks was also counterbalanced regarding repetitions and shifts as well as auditory and visual trials. After completing the SRT task, a structured interview was conducted to identify participants who were able to explicitly recall the sequences. To this end, participants were informed about the existence of the 12-element number and modality sequences and were asked to recall the sequences as correctly as possible (see Hoffmann & Koch, 1997, for a detailed description of the procedure). The experiment took about 20 minutes.

Design  

The independent within-subject variables were modality (visual vs. auditory), modality transition (shifts vs. repetitions), and sequential predictability (random vs. predictable). Note that we were particularly interested in the variables of sequential predictability and modality transition (mean RTs of shifts vs. mean RTs of repetitions in the sequence; this variable represents the MSE), and their interaction to address our theoretical considerations. Regarding the predictability variable, we investigated three different contrasts. First, to assess the “pure” MSE, we examined modality and modality transition in the first two blocks, in which both number and modality sequence was random. Second, we investigated the influence of modality and modality transition in the predictable sequence of Blocks 3 to 8 (note that this analysis cannot distinguish sequence-specific effects from general practice effects). Finally, to isolate sequence-specific performance effects, we computed the difference between the transfer Block 9 and the averaged sequence Blocks 8 and 10. Based on the post-experimental interview, we additionally introduced sequence recall (high vs. low) as a post hoc between-subjects variable. Reaction time and error rates were the independent variables. Significance was tested at α = .05.

Results

For the analysis of error rates, the first trial of each experimental block was excluded from the analysis. For the analysis of RTs, the first trial of each block, errors (4.1%), trials after errors (4.1%), and RTs below 50 ms (0.1%) were excluded from the analysis. Based on the remaining trials, z-values were calculated for each participant in each block and RTs above z = 2.5 and below z = −2.5 were discarded as outliers (2.3%).

Before presenting the RT data collapsed for the 32 participants, we shortly present preliminary analyses with regard to the variation in the transfer Block 9. In Block 9, while the response sequence was random for every participant, the modality sequence was maintained for half of the participants (Group 1), whereas it was random for the other half (Group 2): The two subgroups did not differ in statistical analyses with regard to Block 9, that is, there was no significant main effect of group, F(1, 30) = .008, p > .92, η_p² = .000, and there were no significant interactions of group, F(1, 30) < 2, p > .15.

Moreover, we ran separate analyses for the two groups in order to identify potential differences with regard to main effects and interactions in transfer Block 9 and the surrounding sequence Blocks 8 and 10. The RT data were submitted to repeated-measures analyses of variance (ANOVAs) with the within-subject variables of block (transfer Block 9 vs. averaged sequence Blocks 8 and 10) and modality transition (modality repetition vs. modality shift). The analyses yielded highly similar results for both groups:  They showed a significant main effect of block (Group 1: F[1, 15] = 12.395, p < .01, η_p² = .452; Group 2: F[1, 15] = 8.821, p < .05, η_p² = .370), indicating shorter RTs in the sequence blocks compared to the transfer block (Group 1: M = 500 ms vs. M = 574 ms; Group 2: M = 488 ms vs. M = 562 ms). The main effect of modality transition was significant (Group 1: F[1, 15] = 20.528, p < .001, η_p² = .578; Group 2:  F[1, 15] = 22.370, p < .001, η_p² = .599), revealing a MSE for both groups (Group 1: M = 519 ms for repetitions vs. M = 555 ms for shifts; Group 2: M = 506 ms for repetitions vs. M = 545 ms for shifts). Finally, the interaction of block and modality transition was significant (Group 1: F[1, 15] = 17.426, p < .01, η_p² = .537; Group 2: F[1, 15] = 23.392, p < .001, η_p² = .609), indicating a larger MSE in the sequence blocks compared to the transfer block for both groups (Group 1: MSE in sequence blocks: M = 60 ms vs. MSE in the transfer block: M = 12 ms; Group 2: MSE in sequence blocks: M = 62 ms vs. MSE in the transfer block: M = 16 ms). Thus, in the following, we present the data of the two groups collapsed across the variation in Block 9. The larger sample size of 32 participants allows for the introduction of the post hoc variable of the level of sequence recall based on the free recall data.

Free Recall Data

For each participant, the maximum number of elements of the sequence that could be reproduced correctly was determined following standardized questioning according to Hoffmann and Koch (1997). A common finding in the sequence learning literature is that a better recall performance goes along with a higher learning score in the SRT task (see Shanks, 2005, for a review). In the present study, the correlation between the number of correctly recalled numbers and the RT learning score was high, r(30) = .80, p < .001 (see Figure 1), supporting the validity of our measure for differentiating levels of sequence recall. Moreover, there was a correlation between sequence recall and the MSE in sequence Blocks 8 and 10,³ r(32) = .35, p < .05, indicating that a higher level of sequence recall goes along with a larger MSE. Fourteen of the 32 participants were able to recall five or more numbers of the response sequence in the correct order and were categorized as the high sequence recall group, whereas the other 18 participants who could recall less than five numbers were considered the low sequence recall group (according to Zirngibl & Koch, 2002, who used the same number sequences and five as the cut-off for sequence recall).⁴ In the following analyses, performance of the high sequence recall group is contrasted with that of the low sequence recall group to investigate how the level of sequence recall modulates the influence of consistent modality shifting on sequence learning.

Reaction Time Data

Blocks 1 and 2. To investigate initial RT levels in Blocks 1 and 2 (random sequence), a repeated-measures ANOVA with the between-subjects variable sequence recall (representing a dummy variable in these blocks) and the within-subject factors of block, modality, and modality transition was calculated (see Table 1⁵). The effect of block was not significant, F(1, 30) = 1.046, p > .3, η_p² = .034, but the effect of modality was significant, F(1, 30) = 120.916, p < .001, η_p² = .801, showing that responses to visual stimuli (M = 526 ms) were faster than responses to auditory stimuli (M = 619 ms). The ANOVA also yielded an effect of modality transition, F(1, 30) = 13.572, p < .01, η_p² = .311. That is, participants responded faster in modality repetitions (M = 564 ms) than in shifts (M = 581 ms), thus showing an MSE of 16 ms.

Figure 1.

Mean learning score as a function of length of correctly recalled part of the 12-number response sequence for learners with low versus high sequence recall. Bars represent SEs.

Moreover, there was an interaction of block and modality, F(1, 30) = 4.463, p < .05, η_p² = .130. While RTs for the visual trials changed only minimally from Block 1 (M = 525 ms) to Block 2 (M = 526 ms), RTs for the auditory trials became shorter from Block 1 (M = 627 ms) to Block 2 (M = 612 ms). However, this effect is likely due to the practice of the task, since there was no sequence to learn yet.

As expected, the effect of sequence recall was not significant in the first two blocks (i.e., prior to the introduction of the structured sequences), F(1, 30) = 2.475, p > .12, η_p² = .076, showing no initial difference in RTs between high and low levels of sequence recall. No further interaction was significant, F < 1.8 in each instance.

Blocks 3 to 8. The number and modality sequence was introduced in Block 3. Another four-way repeated-measures ANOVA was calculated on RTs of these blocks. The effect of block was significant, F(2.61, 78.3) = 20.565, p < .001, η_p² = .407,⁶ indicating increasingly shorter RTs in the course of the blocks (Block 3: M = 566 ms, Block 8: M = 484 ms). The effect of modality was again significant, F(1, 30) = 89.143, p < .001, η_p² = .748, showing faster responses to visual stimuli (M = 484 ms) than to auditory stimuli (M = 565 ms). Moreover, there was an effect of modality transition, F(1, 30) = 109.696, p < .001, η_p² = .785, indicating an MSE of 61 ms (repetitions: M = 494 ms, shifts: M = 555 ms).

Furthermore, there was a significant interaction of modality and modality transition, F(1, 30) = 6.323, p < .05, η_p² = .174. That is, the MSE was significantly larger for the auditory (M = 74 ms) than for the visual modality (M = 46 ms).

In these blocks with structured sequences, there was a main effect of sequence recall F(1, 30) = 7.708, p < .01, η_p² = .204, demonstrating shorter RTs for the high sequence recall group (M = 480 ms) compared to the low sequence recall group (M = 568 ms). Moreover, there was an interaction between sequence recall and block, F(5, 150) = 5.975, p < .001, η_p² = .166, showing higher practice-related general RT gains for the high sequence recall group (RT gain: M = 131 ms) than for the low sequence recall group (RT gain: M = 35 ms) in the course of Blocks 3 to 8. Most notably, there was a significant three-way interaction between block, modality transition, and sequence recall, F(5, 150) = 3.006, p < .05, η_p² = .091 (see Figures 2 and 3). That is, the MSE became larger for the high sequence recall group (Block 3: M = 37 ms, Block 8: M = 79 ms), while it remained largely unchanged for the low sequence recall group (Block 3: M = 63 ms, Block 8: M = 58 ms). No further interactions were significant, F < 1.7 in each instance.

Table 1.

Mean RTs in Milliseconds (and SDs in Parentheses) as a Function of Modality (Auditory vs. Visual), Modality Transition (Shift vs. Repeat), and Block of Trials

Figure 2.

Mean RTs (in milliseconds) as a function of modality transition (shift vs. repetition) and block of trials. Reaction times are split up for the participants who showed high versus low sequence recall. R indicates a random response sequence. Bars represent SEs.

Figure 3.

Mean modality shift effect (in milliseconds) as a function of block, split up for the participants who showed low and high sequence recall. R indicates a random response sequence. Bars represent SEs.

Blocks 8 and 10 versus Block 9. For the critical test of sequence-specific learning, the comparison between RTs in the transfer Block 9 (random response sequence) and the averaged sequence Blocks 8 and 10 was studied in another fourway ANOVA. The effect of block was significant, F(1, 30) = 41.002, p < .001, η_p² = .578, revealing increased RTs in the transfer block (M = 567 ms) compared to the training blocks (M = 485 ms) and thus confirming sequence-specific learning. The effect of modality was, again, also significant, F(1, 30) = 105.548, p < .001, η_p² = .779, showing shorter RTs to visual stimuli (M = 487 ms) compared to auditory stimuli (M = 565 ms). There was an interaction of block and modality, F(1, 30) = 4.884, p = .036, η_p² = .139. That is, the differences in the learning score were larger for auditory (M = 85 ms) than for visual (M = 70 ms) trials. Furthermore, the effect of modality transition was significant, F(1, 30) = 44.789, p < .001, η_p² = .599, indicating that participants’ RTs were shorter in modality repetitions (M = 506 ms) than in shifts (M = 546 ms), showing an overall MSE of 40 ms. Importantly, this MSE was larger in the training blocks (M = 62 ms) relative to the transfer block (M = 16 ms), as indicated by a significant interaction of block and modality transition, F(1, 30) = 41.129, p < .001, η_p² = .578 (see Figures 2 and 3). Moreover, the interaction of modality and modality transition was significant, F(1, 30) = 11.113, p < .01, η_p² = .270, showing that the MSE was larger in the auditory modality (M = 57 ms) than in the visual modality (M = 21 ms).

Also in this analysis, there was a main effect of sequence recall, F(1, 30) = 8.657, p < .01, η_p² = .224, showing overall shorter RTs for the high sequence recall group (M = 485 ms) compared to the low sequence recall group (M = 568 ms), but, as expected, this main effect was qualified by a significant interaction of sequence recall and block, F(1, 30) = 19.915, p < .01, η_p² = .399, confirming that the sequencespecific learning effect (the increase in RT in the transfer block) was especially strong for the high sequence recall group (M = 138 ms) compared to the low sequence recall group (M = 25 ms; see Figure 2).

Again, most notably, the sequence recall variable also modulated the interaction of block and modality transition, F(1, 30) = 5.000, p < .05, η_p² = .143, indicating that the modulation of the MSE through sequence learning was especially large in the high sequence recall group (MSE in Blocks 8 and 10: M = 71 vs. MSE in Block 9: M = 10 ms) compared to the low sequence recall group (MSE in Blocks 8 and 10: M = 54 ms vs. MSE in Block 9: M = 24 ms).

A separate ANOVA was calculated for those 18 participants who showed low recall of the sequence, revealing a significant main effect of block, F(1, 17) = 9.457, p < .01, η_p² = .357 and thus confirming sequence-specific learning for this subgroup. Moreover, there was a significant interaction between block and modality transition even in that subgroup, F(1, 17) = 12.227, p < .01, η_p² > .418. Thus, the MSE was increased in the sequence blocks in both groups; although, as shown by the three-way interaction, this effect was significantly larger for the participants who could recall the sequence. No further interaction was significant, F < 1.8 in each instance.

Error Data

With regard to sequence learning, the error data showed a similar pattern as the RT data (see Table 2⁷ and 3). Therefore, we briefly summarizethese data here, focusing on the critical comparison of Block 9 (random response sequence) with Blocks 8 and 10 (fixed response sequence). The number of errors was lower in the averaged sequence Blocks 8 and 10 (M = 4.0%) and went up in the transfer Block 9 (M = 5.3%), thus confirming a significant learning effect, F(1, 30) = 5.817, p < .05, η_p² =.162. However, neither the main effect of modality transition, F(1, 30) < 1, p > .3, η_p² < .04, nor the interaction of block and modality transition, F(1, 30) = 2.952, p = .096, η_p² = .090, were significant. Moreover, the between-subjects variable of sequence recall (high vs. low sequence recall of the response sequence) did not yield a significant main effect or significant interactions, ​F < 1.5.

Assessment of Chunk Structure

In order to assess chunk structures, RTs for the serial positions of the response sequence were examined and the data-driven approach of k-means clustering was selected to identify the chunking pattern in sequence Blocks 8 and 10 (Song & Cohen, 2014). Typically, reactions at the beginning of chunks are slower, whereas reactions within chunks are faster (e.g., Koch et al., 2006; Nissen & Bullemer, 1987; Song & Cohen, 2014). K-means clustering denotes an algorithm which partitions objects according to patterns (see Steinley, 2006 for a review). Applied to our context, it self-sorts RTs into two clusters, namely fast and short RTs. The analyses were conducted separately for (a) Modality Sequences 1 and 2, and (b) for Modality Sequences 3 and 4.⁸ Figure 4 depicts mean RTs in the course of the 12-elements response sequence on the left side, whereas on the right side, the results of k-means clustering sorted as either fast or slow are presented for the two types of modality sequences. The results roughly followed the expected pattern: The modality shifts were expected to represent the start of a chunk and thus, RTs in modality shifts were expected to be comparatively long, whereas the modality repetitions were presumed to be within the chunk and thus, RTs in modality repetitions were expected to be shorter. In Modality Sequences 1 and 2, there were two deviations from this general RT pattern, namely, on Positions 7 and 11. In Modality Sequences 3 and 4, there was only one deviation from the expected RT pattern (Position 2). Thus, k-means clustering illustrated the presumed chunking structure.

Dicussion

The aim of the present study was to investigate sequence learning in a more complex sensory environment. More specifically, we examined whether consistent modality shifting affects the process of sequence learning in a cross-modal SRT task. To our knowledge, we are the first to investigate this interaction. Participants responded to a repeating sequence of numbers that were presented either visually or auditorily. Not only the response sequence but also the shifting between modalities followed a fixed pattern.

The MSE was present throughout the experiment, showing the typical performance benefit for modality repetitions over shifts (e.g., Driver & Spence, 1998). However, the major finding was that the MSE interacted with sequence learning: The MSE was comparatively smallin the first two random blocks and the transfer block (Block 9), whereas it was especially large in the sequenced blocks. In particular, the modality-repetition trials benefited from sequence learning, whereas the learning-specific effect on modality-shift trials was comparatively much smaller. Furthermore, it was shown that this modulation of the MSE was especially large for those participants who could recall large parts of the response sequence, and smaller for those who could not. However, this modulation of the MSE by sequence learning was present (and significant) in both groups (i.e., high vs. low sequence recall), only to a different degree, so that a higher level of sequence recall simply seems to increase the general interaction of sequence learning and cross-modal attention shifting, which we attribute to an influence of modality shifting on chunk formation.

Figure 4.

The chunking pattern was analyzed separately for (a) Modality Sequences 1 and 2 (n = 16) and (b) Modality Sequences 3 and 4 (n = 16). Reaction times in sequenced Blocks 8 and 10 were selected for analyses. Left: Mean RTs (collapsed across modalities) for serial positions in the 12-element response sequence which was accompanied by a 12-element modality sequence. The x-axis indicates the position in the sequence and whether a modality repetition (MR) or shift (MS) occurred at this point. Right: Using k-means clustering, mean RTs for the 12 serial positions in the response sequence were self-sorted as either fast (F) or slow (S), illustrating the chunking pattern

Table 2.

Mean Error Rates in % (and SDs in Parentheses) as a Function of Modality (Auditory vs. Visual), Modality Transition (Shift vs. Repeat), and Block of Trials

Table 3.

Mean Error Rates in % (and SDs in Parentheses) as a Function of Recall Performance (Low vs. High), Modality Transition (Shift vs. Repeat), and Block of Trials

The interaction of sequence learning with consistent cross-modal shifting is a novel finding. In principle, there could be two possible mechanisms underlying this effect: Either a facilitated learning of the response sequence during modality repetitions or a facilitated behavioral expression of the acquired response sequence during modality repetitions. Notably though, the modulation of the MSE was especially large for participants who demonstrated a high sequence recall and thus knew large parts of the sequence, irrespective of modality repetitions or shifts. Thus, it is unlikely that the effect was due to enhanced learning of the sequence only during repetition trials. Instead, the influence of sequence recall suggests that the modulation of the MSE is due to a facilitated reproduction of the response sequence during modality repetitions compared to shifts, based on facilitated chunk formation induced by modality shifts which served as chunk boundaries.

Indeed, chunking has frequently been suggested as an important mechanism in sequence learning (e.g. Jiménez et al., 2011; Koch & Hoffmann, 2000; Sakai et al., 2003; Stadler, 1993, Stadler, 1995). For example, it has been demonstrated that regular pauses (Stadler, 1993), relational structures in the sequence (Kirsch et al., 2010; Koch & Hoffmann, 2000), or color cues (Jiménez et al., 2011) enhanced sequence learning. Thus, additional sequential structure facilitated the parsing or chunking of the sequence into subsequences (Kirsch et al., 2010; Koch & Hoffmann, 2000; Stadler, 1993). Similarly, we induced additional sequential structure in the present study by inducing regular modality shifts. We presume that the numbers that were presented in the same modality were grouped together and that modality shifts served as chunking points. Thus, visually presented numbers were grouped into one chunk and auditorily presented numbers into the other, and the modality shift served as the chunking point at which one chunk ended and the next one had to be retrieved, which incurs some retrieval costs in terms of increased RT (e.g., Koch & Hoffmann, 2000; Rosenbaum et al., 1983). Indeed, the chunking pattern which was illustrated by examining RTs for the sequential positions in the sequence and through k-means clustering supported this assumption: It showed that during the repeating, predictable 12-element sequence, RTs in modality shifts were longer, whereas RTs to modality repetitions were shorter, indicating that modality shifts represent the beginning of a chunk, whereas modality repetitions were within chunks and thus, facilitated. Hence, the data suggest that consistent modality shifts induce chunks, which, in turn, explains why the reproduction of the sequence is especially facilitated during modality repetitions and not during shifts.

Notably, we found that this specific effect of modality shifts on sequence learning was more pronounced in those participants who demonstrated high sequence recall. Generally, we found that free recall of the sequence, as a direct measure of sequence learning, corresponded quite well with the indirect RT score of learning (i.e., r = .8), which is consistent with previous results using this type of experimental design (e.g., Hoffmann & Koch, 1997, who found a correlation of r = .81; or Zirngibl & Koch, 2002, who found a correlation of r = .71). Additionally, there was a correlation of r = .35 between the level of sequence recall and the MSE in the sequence blocks, indicating that a high sequence recall goes along with a larger MSE and thus with enhanced chunking. Moreover, it is notable that even those participants who could not recall large portions of the sequence showed a significant modulation of the MSE, suggesting that chunking was induced even in those participants who showed low sequence recall, albeit to a lesser degree.

At this stage, we would like to point out that our original aim was not to draw conclusions about implicit or explicit learning, because our assessment of sequence recall might not have been an optimally sensitive measure for differentiating implicit or explicit learning (see, e.g., the methodological discussion in Shanks & St John, 1994; Shanks, 2005). Therefore, we believe that our data primarily illustrate how high sequence recall can benefit sequence performance on the basis of more efficient chunking processes and thereby add to earlier studies which demonstrated facilitated chunking with high sequence recall (e.g. Koch et al., 2006; Koch, 2007; Schneider & Logan, 2015).

Furthermore, note that, based on the present data, we can only make statements about shifting between the auditory and visual modality, whereas we cannot make inferences regarding the influence of structured modality shifts involving other modalities like tactile or olfactory stimulation. This should be examined in future studies. The same applies to different lengths of modality or response sequences and different lengths of the RSI (which has been shown to influence sequence learning, e.g., Koch & Hoffmann, 2000; Stadler, 1995).

To conclude, the present study showed that sequence learning is influenced by modality shifting, and to our knowledge, is the first to examine and demonstrate this interaction. More specifically, we showed that modality repetitions facilitated the reproduction of a learned equence compared to shifts, suggesting that the consistent modality shifts induced enhanced chunking of the sequence. Moreover, this pattern was particularly pronounced for the participants who showed high recall performance, indicating that the chunking process was enhanced with high sequence recall.