Reliability of the serial reaction time task: If at first you don’t succeed, try, try, try again

Cátia M Oliveira; Marianna E Hayiou-Thomas; Lisa M Henderson

doi:10.1177/17470218241232347

. 2024 Mar 7;77(11):2256–2282. doi: 10.1177/17470218241232347

Reliability of the serial reaction time task: If at first you don’t succeed, try, try, try again

Cátia M Oliveira ^1,^✉, Marianna E Hayiou-Thomas ¹, Lisa M Henderson ¹

PMCID: PMC11529135 PMID: 38311604

Abstract

Procedural memory is involved in the acquisition and control of skills and habits that underlie rule and procedural learning, including the acquisition of grammar and phonology. The serial reaction time task (SRTT), commonly used to assess procedural learning, has been shown to have poor stability (test–retest reliability). We investigated factors that may affect the stability of the SRTT in adults. Experiment 1 examined whether the similarity of sequences learned in two sessions would impact stability: test–retest correlations were low regardless of sequence similarity (r < .31). Experiment 2 added a third session to examine whether individual differences in learning would stabilise with further training. There was a small (but nonsignificant) improvement in stability for later sessions (Sessions 1 and 2: r = .42; Sessions 2 and 3: r = .60). Stability of procedural learning on the SRTT remained suboptimal in all conditions, posing a serious obstacle to the use of this task as a sensitive predictor of individual differences and ultimately theoretical advance.

Keywords: Language, serial reaction time task, procedural memory, sequence learning, reliability, individual differences

Introduction

Procedural memory underlies the encoding; storage; and retrieval of motor, perceptual, and cognitive skills that involve the integration of sequenced, statistical, and probabilistic knowledge across the lifespan (Eichenbaum, 2002; Eichenbaum & Cohen, 2001; Koch et al., 2020; Ullman, 2004). Learning in this system relies on the basal ganglia (specifically, the striatum), the cerebellum, and portions of the parietal and frontal cortices (Packard & Knowlton, 2002; Parent & Hazrati, 1995; Poldrack & Packard, 2003) and tends to be gradual, yet once the skills have been learned they are used rapidly and automatically. The procedural memory system is proposed to be involved in language acquisition. Specifically, Ullman and colleagues (Ullman, 2004; Ullman et al., 2020) propose that the procedural memory system supports the acquisition of rule-based linguistic knowledge, such as phonology and grammar; while the declarative system is mostly associated with acquisition of more arbitrary and explicit knowledge, such as vocabulary. Supporting this, language and procedural memory share brain systems, including basal ganglia and frontal cortex, especially Broca’s area (Ullman, 2001; Ullman & Pierpont, 2005), and clinical populations with impairments of the basal ganglia tend to show both motor and linguistic impairments (Ullman & Pierpont, 2005). Aligning with the declarative/procedural model, some previous studies have shown small to moderate correlations between procedural learning and language and literacy abilities (Clark & Lum, 2017; Desmottes et al., 2017; Lum et al., 2012). However, other studies have failed to replicate these associations (Desmottes et al., 2017; Gabriel et al., 2015; Henderson & Warmington, 2017; Siegelman & Frost, 2015; Vakil et al., 2015; West et al., 2019). This inconsistency, coupled with recent concerns about the psychometric properties of tasks used to measure procedural learning—serial reaction time task (SRTT; Kalra et al., 2019; Siegelman & Frost, 2015; Stark-Inbar et al., 2017; West et al., 2018); contextual cueing and Hebb tasks (West et al., 2018); and statistical learning tasks (Arnon, 2020)—calls for further research to systematically examine the reliability of markers of procedural learning.

The SRTT (Nissen & Bullemer, 1987) is the most widely used measure of procedural (or sequence) learning that requires participants to connect a series of events and form high-order associations to predict future positions (Keele et al., 2003). It has been shown to rely on the same neural networks as other measures of procedural learning (Clark et al., 2014; Hardwick et al., 2013). For example, patients with basal ganglia disorders (e.g., Huntington’s disease) show impaired procedural learning on the SRTT (Willingham & Koroshetz, 1993), and functional magnetic resonance imaging (fMRI) studies demonstrate that procedural learning captured by the SRTT elicits activation in the basal ganglia (putamen: Willingham et al., 2002; ventral striatum: Doyon et al., 1996; and the cerebellum: Hardwick et al., 2013). In the SRTT, a stimulus is presented in an array (e.g., four squares presented horizontally across a screen) and participants are required to press a corresponding button on a keypad or button box to the position of the stimulus on screen as quickly as possible. Unbeknown to the participant, some of the stimulus transitions follow a sequence, with procedural learning being measured as the response time difference between the sequenced and random trials. Faster responses to sequenced than random trials are taken as a “procedural learning effect,” indicating that the participant has learned the sequence and is therefore able to anticipate the next position.

SRTTs can be deterministic or probabilistic. Deterministic sequences usually comprise random and sequenced blocks. The first blocks typically contain the repeating sequence, with a sudden switch to a random block, followed by a final sequenced block; however the opposite pattern (random–structured–random) is also frequently adopted. Reaction times (RTs) tend to decrease progressively during practice in sequenced blocks but then increase in random blocks; this difference in RT is taken as evidence of procedural learning. In contrast, probabilistic SRTTs usually comprise two second-order conditional sequences, one that occurs with a higher probability than the other (e.g., sequence A [85%]: 121432413423; sequence B [15%]: 323412431421; Siegelman & Frost, 2015). Each block starts with a random bigram (e.g., 43) and the next location selected will be either the location that followed that bigram in sequence A (i.e., 2, termed a “probable” trial) or the location that following that bigram in sequence B (i.e., 1, termed an “improbable” trial). Procedural learning in probabilistic SRTTs is measured as the difference in response times between probable and improbable trials. Importantly, despite participants showing evidence of procedural learning, they often have little to no awareness of the presence of a probabilistic sequence (Destrebecqz & Cleeremans, 2001). Deterministic sequences, on the contrary, have been found to yield more explicit awareness of the sequence (Jiménez & Vázquez, 2005; Stark-Inbar et al., 2017; Stefaniak et al., 2008). Thus, the probabilistic sequences may represent purer measures of implicit procedural learning (Stefaniak et al., 2008).

The SRTT is well known for producing robust effects at the group level, thus recently there has been increased interest in using the SRTT as a marker of individual differences (Siegelman & Frost, 2015). However only a few studies have explored the psychometric properties of the task. Reliability refers to the ability of a task to rank individuals’ performance consistently across time, with higher reliability indicating stable scores obtained at test and retest (Hedge et al., 2018). Split-half reliability, a measure of internal consistency within a single session that reflects the correlation between scores within a test (Nunnally & Bernstein, 1994), has been shown to be moderate to adequate on the SRTT in children and adults, respectively (children: rs = .49−.75; adults rs = .84−.92, West et al., 2018, 2021). However, test-retest reliability (i.e., the stability of the test scores over different sessions) is notably poorer and below acceptable psychometric standards: that is, r < .70 (Burlingame et al., 1995; Nunnally & Bernstein, 1994), in both children (probabilistic SRTT: r = .21, 500 trials, West et al., 2018; r = .26, 1000 trials, West et al., 2021) and adults (deterministic SRTT: r = .38, Kalra et al., 2019; r = .07, Stark-Inbar et al., 2017; probabilistic SRTT: r = .47, Siegelman & Frost, 2015; r = .70, West et al., 2021; and alternating SRTT: r = .46, Stark-Inbar et al., 2017). In one exception, West et al. (2021) obtained a test–retest reliability of .70 using a probabilistic SRTT with 46 adults aged between 18 and 61 years. The unusually high stability reported here could be due to one or more of a number of methodological differences: for example, a large number of trials (i.e., 1,500), the same sequence was administered twice, the gap between tests was 2–3 days, and use of a 250-ms interstimulus interval (ISI).

According to classical test theory (Fleiss, 1986), observed scores reflect true scores and measurement error, and higher degrees of measurement error lead to greater fluctuations in scores across time. This translates into poor test–retest reliability as participants’ relative ranking will change between test and retest (Berchtold, 2016; Nunnally & Bernstein, 1994). Poor reliability may contribute to noisier predictions; increased uncertainty in parameter estimation (Loken & Gelman, 2017); and attenuation of the association between measures (Rouder et al., 2019; Rouder & Haaf, 2019, 2021). In small samples, as demonstrated by Loken and Gelman (2017), measurement error can lead, by chance, to overestimation of the effect size. Thus, the poor reliability of the SRTT may contribute to the inconsistently reported correlations between language/literacy measures and procedural learning (LeBel & Paunonen, 2011). It is, however, important to note that in the one study to date which reports adequate test–retest reliability for the SRTT (r = .70; West et al., 2021), only negligible correlations were observed between procedural learning and word and nonword reading measures (rs from −.06 to −.11; West et al., 2021). Thus, even in the face of adequate stability, this lack of association remains contrary to the predictions of the declarative/procedural model. Nevertheless, it is a single study, and identifying optimal conditions for achieving better reliability remains imperative. Indeed, only a robust and reliable task can test the boundaries of the procedural/declarative model of language acquisition, including the procedural deficit hypothesis, and permit a better understanding of the role of procedural learning and language development and disorder (Matheson, 2019). Systematically examining the stability of the SRTT also has clear methodological value, in revealing design modifications to enhance its psychometric properties, and clinical value, in working towards developing a tool that can identify procedural learning weaknesses (Berchtold, 2016). Generally, it has been claimed that a larger number of trials in any task tends to increase reliability, due to a reduction in measurement error (D. H. Baker et al., 2021; Rouder & Haaf, 2019, 2020). However, studies by West and colleagues (2018, 2021) showed only modest (and nonsignificant) numerical improvements in test–retest reliability when they increased the number of trials in their SRTT.

In addition to examining reliability, agreement, also called repeatability, was examined using the Bland–Altman method (Bland & Altman, 1986, 1999, 2010). As argued by Berchtold (2016), the concept of test–retest refers to both the reliability and agreement of a measurement tool, with agreement referring to the ability of a test to produce the same scores when participants are tested under the same conditions. Thus, while reliability reflects the test’s ability to rank participants consistently within or across sessions, agreement instead focuses on the consistency of the scores, independently of the range and distribution of the variables. Thus proving particularly important for clinical applications whereby participants’ scores, instead of ranking order, may be used to track response to intervention.

Therefore, here, we examine further factors that may influence stability. Of particular focus here are the similarity of the sequences to be learned (Experiment 1) and the number of sessions across which learning is assessed (Experiment 2). To allow for a comprehensive understanding of reliability, a multi-measurement analytic approach will be taken: we will assess the psychometric properties of the SRTT across different measures of procedural learning (difference scores or random slopes) and different psychometric measures (split-half reliability, test–retest reliability, and agreement).

Experiment 1

There are several reasons why the similarity of sequences to be learned over two or more sessions may influence both the size of the procedural learning effect and potentially also its stability, and each predicts that greater similarity between sequences should result in better learning at later sessions. First, learning the same or similar sequences reduces the likelihood of proactive interference, in which the memory of the first-learned sequence disrupts the learning of the second-learned sequence (Borragán et al., 2015; Darby & Sloutsky, 2015). Second, greater similarity increases the likelihood that consolidation of the first sequence will benefit learning of the second, such that individuals benefit from prior knowledge when exposed to the new material (Nemeth et al., 2010; Robertson et al., 2004; Siegelman & Frost, 2015). Third, the well-established phenomenon of practice effects is likely to lead to an improvement in performance for later sessions (Hausknecht et al., 2007; Scharfen et al., 2018), which is why the use of alternate forms is generally recommended (Beglinger et al., 2005); although see Scharfen et al. (2018) for evidence that alternate forms do not reduce practice effects in working memory capacity tasks. Finally, greater similarity may also lead to increased explicit awareness of the sequence at subsequent sessions and improve performance (Rüsseler et al., 2003) as explicit knowledge has been shown to increase with extended training in the SRTT and is more likely to lead to offline consolidation (Robertson et al., 2004).

While greater similarity in sequences used in different sessions may result in larger procedural learning effects in later sessions, they may also reduce the stability of procedural learning (Stark-Inbar et al., 2017). Individual differences in any one of the above factors would introduce variability in procedural learning at retest, thus leading to changes in the rank order of scores (Hedge et al., 2018; Stark-Inbar et al., 2017). Practice effects have been shown to vary according to participants’ characteristics (e.g., age: Brown et al., 2009; Hodel et al., 2014) and cognitive skills (Schaefer & Duff, 2017), thus introducing additional variability at retest. To our knowledge there has been no direct examination of the effect of sequence similarity on either the magnitude of the procedural learning effect, or the test–retest reliability of the SRTT. However, two recent studies in the literature are consistent with our prediction: Siegelman & Frost (2015) used the same sequences at both testing sessions and reported lower test–retest reliability than West et al. (2021), who used different sequences. While West et al. (2021) showed no significant differences in the learning effect between sessions, Siegelman and Frost (2015), on the contrary, reported that after 3 months the majority of participants (64 out of 75) showed a better performance at retest.

Experiment 1 examined the effect of similarity of the two sequences to be learned, to ascertain (1) the impact on the magnitude of the procedural learning effect, and (2) the effect on test–retest reliability (referred to here as stability). Similarity was operationalised in terms of the Levenshtein distance (LD), which has been widely used to determine the distance between strings across fields such as biology, computer science, and linguistics (Berger et al., 2021; Eriksen & Tougaard, 2006; Faes et al., 2016; Konstantinidis, 2005). Three types of operations are considered—substitutions, deletions, and insertions—with a small distance between sequences indicating higher similarity and a large distance revealing that the sequences are dissimilar (Levenshtein, 1966). We used sequences of varying similarity in a probabilistic SRTT to test four main hypotheses:

H1: Participants will demonstrate procedural learning in both sessions, as indexed by faster responses to probable versus improbable elements of the sequence;
H2: Similarity between sequences will impact the magnitude of the procedural learning. Higher levels of similarity between Sessions 1 and 2 will result in a larger procedural learning effect in Session 2, whereas lower levels of similarity between Sessions 1 and 2 will result in a relatively smaller of procedural learning effect;
H3: Within session reliability (indexed by the split-half correlation coefficient) will be higher than stability across sessions, indexed by test–retest reliability;
H4: Sequence similarity will be negatively associated with stability: more similar sequences at Sessions 1 and 2 will be associated with lower test–retest reliability.

Methods

Participants

A total of 103 undergraduate students from the University of York (91 females), aged between 18 and 25 years (M = 19.18, SD = 1.09), participated in exchange for course credit. The sample included monolingual, bilingual, and multilingual individuals from various nationalities; all identified as fluent English speakers. The sample size was determined based on West et al. (2021), doubling the number of participants to allow for a median split of participants based on similarity of the sequence. Sensitivity analyses, in line with those conducted by Farkas et al. (2023) and presented in Supplementary Materials 1, suggest that sample sizes above 100 participants offer limited gains in precision. Furthermore, with a sample size of 103, we have 80% power to detect correlations equal to, or above, .30. The experiment was approved by the Ethics Committee of the Psychology Department at the University of York and each participant gave written informed consent.

Measures

SRTT: A nonverbal probabilistic SRTT was used, following West et al. (2018, 2021) given the task used in this previous study has produced the highest reported stability in the existing literature. On each trial, four black outlined rectangles were presented horizontally and a stimulus (i.e., a smiley face) appeared in one of the four rectangles, with participants asked to respond as quickly and accurately as possible by pressing one of four corresponding keys (Z, X, N, M) on the keyboard. The stimulus remained visible until the key press. Participants rested their index and middle fingers of each hand on the four keys so they were ready to respond.

Two versions of this task were generated, each containing two different underlying second-order conditional 12 item sequences. The first two sequences were taken from Shanks et al. (2003): probable sequence A—314324213412; improbable sequence A—431241321423, while the second sequences were taken from Schvaneveldt and Gomez (1998): probable sequence B—121342314324; improbable sequence B—123413214243. In second-order conditional sequences, each trial can be predicted based on the previous two trials (Schwarb & Schumacher, 2012). For each SRTT, each block started with the consecutive generation of two random digits (e.g., 21), with that bigram then followed by the digit in probable sequence A (e.g., 3) with 90% of probability or followed by the digit in improbable sequence A (e.g., 4) with 10% probability (after West et al., 2018, 2021). After each response a new bigram was created which continuously followed the same principles. See Additional Analyses 1¹ for a series of simulations manipulating (1) the overall number of trials and (2) the ratio between trials per condition.

The task comprised 1,000 trials per session, as in West et al. (2021), divided into 20 blocks of 50 trials each. Within each block, trials immediately followed the participants’ response, with no ISI. Breaks between blocks comprised a fixation cross presented centrally on screen for a random duration between 8 and 12 s. The stimuli were programmed in Psychopy 2 (Peirce et al., 2019); response accuracy and RT (from stimulus onset) were recorded.

Sequence similarity

Varying the degree of similarity between inputs was achieved by generating a new stimulus set for each participant (i.e., given the probabilistic nature of the SRTT, each participant was exposed to a different set of 1,000 trials). To achieve variability in the stimulus sets, half of the participants were exposed to stimuli that conformed to the same sequence structure at Sessions 1 and 2 (A/A), while others were exposed to stimuli that were generated by different sequence structures at both time points (A/B) (see Figure 1). Crucially, due to the probabilistic nature of the task, none of the participants was exposed to the exact same stimulus set at both sessions, as new stimuli were generated per session. Furthermore, variability in the input was increased by randomly matching the digits of the sequence (1, 2, 3, 4) to a different position on screen (left, centre–left, centre–right, right). A measure of similarity of the resulting sequences actually presented to each participant was computed using the LD. LD computes the minimum operations required (insertion, deletion, and substitution) for both strings to be identical, thus providing an indication of similarity between stimulus sets (Levenshtein, 1966). The LD was calculated for each participant by comparing the stimulus sets, that is, two sets of 1,000 trials. Across participants, the LD between pairs of stimuli varied between 248 and 437. More details on the distribution of LD between sessions can be found in Additional Analyses 2. The similarity ratio index of the total number of triplets in common between sequences was also computed (Pasquali et al., 2019; Wierzchon et al., 2012). Given the use of second-order conditional sequences, whose minimum unit of sequential information is three sequential locations or triplets, this additional computation ensured that these triplets were captured by the LD scores. Pearson’s correlations between the LD scores and the similarity ratio index revealed a high correlation between measures (r = .86).

Figure 1. — Visual representation of the process of stimulus set generation.

Procedure

All participants were tested individually or in a quiet testing room in groups of up to six. All participants performed the SRTT at both sessions (SRT1 refers to SRTT at Session 1; SRT2 for Session 2). Each session lasted approximately 30 min, with Session 2 occurring 1 week after Session 1 for all but two participants, who were tested 9 and 10 days apart. Once the SRT2 task was completed, task enjoyment and explicit knowledge were assessed via a question and a generation task, to ensure that the levels of explicit awareness were equivalent to previous studies using probabilistic tasks (see Supplementary Materials 2).

Statistical analyses

R software—version 4.1.1 (Rstudio Team, 2020) and lme4 package (Bates et al., 2015) were used to perform two separate linear mixed effects analyses of the performance of the participants on the SRTT and all figures produced using the package ggplot2 (Wickham, 2016). p-values were obtained for the linear mixed effects model using the lmerTest package (Kuznetsova et al., 2017) and corrected for multiple comparisons using the Holm–Bonferroni method (Holm, 1979). All reported p values are non-adjusted; however, all analyses which have not survived correction for multiple comparisons after correction for familywise error rates have been stated.

For the following data analyses, RTs were grouped into epochs of five blocks, comprising 200 trials. The first two trials of each block were removed as these were not predictable since the sequence follows a higher order structure with the third trial being predicted based on the previous bigram (two trials). All incorrect trials were removed from the analyses. Due to the unequal number of probable and improbable trials, a moving criterion based on sample size was used to identify outlier RTs (Cousineau & Chartier, 2010; Van Selst & Jolicoeur, 1994). Participants with overall RTs > 2.5 SD from overall mean were excluded from the analyses (based on z scores averaged across probable and improbable conditions for each group/session separately). Two participants were removed from the analyses for both sessions while the remaining two participants were removed for one of the sessions.

As RTs were right-skewed based on visual inspection and tests of normality, a log transformation was used to normalise the distribution of RTs (Brysbaert & Stevens, 2018). Visual inspection of the residual plots after log transformation did not reveal any obvious deviations from homoscedasticity or normality.

The fixed-effects structure represented the maximal-fixed-effects structure. The random intercept structure included solely participants, as item order was not consistent across participants due to randomisation procedures. The random structure followed the forwards best path approach (Barr et al., 2013) starting from the minimal intercepts-only structure and building the random structure according to likelihood-ratio tests (p < .2) (Barr et al., 2013) and the Akaike information criterion (AIC; Akaike, 1974) to avoid overfitting (Brewer et al., 2016).

H1 and H2: the procedural learning effect and similarity

The first model—RT model, designed to explore the procedural learning skills of the sample, included the within-group variables—probability (probable or improbable), epoch (contrasts between successive Epochs 2-1 [i.e., Epoch 2 vs Epoch 1], 3-2 [i.e., Epoch 3 vs Epoch 2], 4-3 [i.e., Epoch 4 vs Epoch 3], 5-4 [i.e., Epoch 5 vs Epoch 4]), and Session (1 or 2) into a linear mixed effects model, with participants as a random effect, to account for participant variability in performing the SRTT, and Session, Epoch, and Probability as random slopes. The second model—similarity model—was formulated to explore the relationship between similarity and procedural learning in more detail. Due to the continuous nature of the similarity variable, it was centred and standardised before running the analysis. In both models, the outcome variable is log-transformed RTs to address issues of non-normality, although raw means are reported for ease of interpretation. The model with similarity included only RTs from the last three epochs to avoid the inclusion of epochs where procedural learning is not yet robust as suggested by Conway et al. (2019). Probability (probable or improbable), Session (1 or 2), and Similarity were entered as fixed effects and Participants as a random effect. Thus, unlike the first model, Epoch was not included as the goal was to explore the role of similarity when procedural learning was more robust, independently of its progression across epochs. After building the random structure following the method previously described, Session and Probability were included as a random slope.

After model selection, the influence.ME package was used to detect influential data as these values may lead to changes in regression estimates (Nieuwenhuis et al., 2012). Dfbetas were standardised and participants whose z-scores were greater than ±3.29 were identified as influential cases as opposed to the 2.5 SD threshold to avoid loss of a high number of participants (Walker et al., 2020). Three participants were identified as influential cases for the response times model and four for the similarity model.

H3 and H4: reliability and agreement

Test–retest and split-half reliability of the RTs were analysed using Pearson’s correlations, with a reliability of .70 or greater being considered adequate (Nunnally & Bernstein, 1994). Although we have compared our findings against this arbitrary threshold, reliability should be viewed in a continuum. As poor reliability results in the attenuation of the effect sizes of interest, researchers should take these measurement issues into account when designing a study, especially when making design choices, such as the number of trials per individual per task, has a critical impact on the effect sizes within a task and correlations across tasks (Green et al., 2016; Rouder & Haaf, 2019). Two² different indices of procedural learning, commonly used in previous studies, were computed to better capture stability. Simple difference scores, the most commonly used measure for the SRTT, were computed for each participant as the simple difference between improbable and probable RTs, with a positive value indicating procedural learning. Random slopes for each participant/session were obtained by running a linear mixed effects model with log transformed RTs as a dependent variable and Probability (probable or improbable) as a predictor, for the random structure participants were introduced as a random intercept and probability as a random slope (Lammertink et al., 2020; Llompart & Dąbrowska, 2020; Milin et al., 2017). Random slopes were computed as this measure better captures the learning trajectory for each participant and are less likely to be influenced by extreme scores.

To measure split-half reliability for both sessions, trials were separated into probable and improbable trials. Consecutive trials were labelled as odd or even. Split-half reliability was calculated by correlating the overall mean difference in RTs for even and odd trials. The split-half and test–retest reliability were computed both for the entire task and the last 600 trials, following the suggestion that the later stages of procedural learning may be more stable (Conway et al., 2019). Agreement was examined using the Bland–Altman method (Bland & Altman, 1986). The Bland–Altman method involves plotting the mean of the measures for each participant (e.g., (Diff2 + Diff 1)/2 against the difference in the paired measurements in Sessions 2 and 1 (e.g., Diff2—Diff 1), with 95% of the data points being expected to lie within ±1.96 SDs of the mean difference, referred to as the 95% limits of agreements. According to Bland and Altman (1999), while a consistent tendency in the scores where performance is superior in one of the sessions than the other can be adjusted for by subtracting the difference between sessions from the one with higher scores (bias), wide limits of agreement pose a more serious problem. Determining whether the limits are adequate will depend on how precise the instrument must be for its use in clinical or research settings.

Results

Data were available for 100/103 participants for Session 1 and for 98/103 participants for Session 2. Data from five participants were lost due to computer malfunction and one due to a participant being unable to attend the second session. Four of these participants contributed data for one of the sessions, but two participants’ data were lost for both sessions. Three participants were identified as outliers for each session. Data from 97 participants for Session 1 and from 95 participants for Session 2 were therefore included in the analysis. Participants showed high accuracy rates across sessions (Session 1: Macc = 95%, SD = .09; Session 2: Macc = 95%, SD = .08).

H1: procedural learning in the SRTT

Results from the mixed effects model are presented in Table 1. As evidenced in Figure 2, RTs decreased with practice as observed by faster RTs with successive epochs. There was evidence of procedural learning, with RTs faster for probable than improbable trials. This “procedural learning effect” increased over epochs, as shown by the significant interaction between Epoch × Probability for Epoch 2-1 (i.e., Epoch 2 vs Epoch 1), Epoch 3-2 (i.e., Epoch 3 vs Epoch 2), and Epoch 4-3 (i.e., Epoch 4 vs Epoch 3; no longer significant after correction for multiple comparisons), but not for the last contrast, possibly indicating a plateau in learning after Epoch 4. The significant interaction between Probability × Session, indicates that participants showed a larger procedural learning effect in Session 2 than Session 1, but this was not significant after correction for multiple comparisons. The absence of a three-way interaction between Epochs × Probability × Session indicates that the within-session progression of procedural learning was similar for both sessions.

Table 1.

Predictors of the magnitude of procedural learning.

Fixed effects	b	SE	t	p	CI
(Intercept)	6.074	0.013	474.610	<.001	6.049	6.100
Epoch 2-1	−0.019	0.004	−4.259	<.001	−0.028	−0.010
Epoch 3-2	0.008	0.004	2.130	.035	0.001	0.015
Epoch 4-3	−0.010	0.003	−3.001	.003	−0.017	−0.004
Epoch 5-4	−0.019	0.004	−4.921	<.001	−0.026	−0.011
Probability	0.024	0.001	15.806	<.001	0.021	0.027
Session	0.061	0.004	16.080	<.001	0.054	0.069
Epoch 2-1 × Probability	0.011	0.002	5.020	<.001	0.007	0.016
Epoch 3-2 × Probability	0.012	0.002	5.018	<.001	0.007	0.016
Epoch 4-3 × Probability	0.005	0.002	2.309	.021	0.001	0.010
Epoch 5-4 × Probability	0.004	0.002	1.653	.098	−0.001	0.008
Epoch 2-1:Session1	−0.017	0.004	−3.816	<.001	−0.025	−0.008
Epoch 3-2:Session1	−0.004	0.003	−1.392	.166	−0.010	0.002
Epoch 4-3:Session1	−0.012	0.003	−3.398	<.001	−0.019	−0.005
Epoch 5-4:Session1	−0.008	0.003	−2.328	.021	−0.015	−0.001
Probability × Session	−0.002	0.001	−2.297	.022	−0.003	0.000
Epoch 2-1 × Probability × Session	0.001	0.002	0.618	.537	−0.003	0.006
Epoch 3-2 × Probability × Session	−0.001	0.002	−0.396	.692	−0.005	0.004
Epoch 4-3 × Probability × Session	−0.004	0.002	−1.638	.102	−0.008	0.001
Epoch 5-4 × Probability × Session	0.000	0.002	0.079	.937	−0.004	0.005
Random effects		Variance				SD
Participant: (Intercept)		0.0156				0.1250
Participant: Session (Slope)		0.0013				0.0360
Participant: Epoch 2-1 (Slope)		0.0014				0.0372
Participant: Epoch 3-2 (Slope)		0.0008				0.0291
Participant: Epoch 4-3 (Slope)		0.0006				0.0248
Participant: Epoch 5-4 (Slope)		0.0008				0.0286
Participant: Probability (Slope)		0.0002				0.0127
Participant: Session × Epoch 2-1 (Slope)		0.0013				0.0361
Participant: Session × Epoch 3-2 (Slope)		0.0004				0.0203
Participant: Session × Epoch 4-3 (Slope)		0.0006				0.0254
Participant: Session × Epoch 5-4 (Slope)		0.0006				0.0252

Open in a new tab

Indicated in bold are the contrasts that survived correction for multiple comparisons using the Holm–Bonferroni method.

Figure 2. — Mean response times for probable and improbable trials per epoch and session (Session 1 on the left and Session 2 on the right). Bars indicate 95% CI.

H2: the effect of similarity on procedural learning

In the model incorporating sequence similarity (results are presented in Table 2), a similar pattern of results was obtained in terms of significant effects of probability and session. Turning to the effect of similarity, in line with our predictions, LD and LD × Probability were not significant predictors of RT, but there were Probability × Session × LD interactions. This indicates that greater similarity was associated with larger procedural learning effects in Session 2. This was further examined by Pearson’s correlations between the LD for each participant and their procedural learning effect (for each session separately). As expected, LD and procedural learning were not significantly correlated in Session 1 (given sequence similarity between the two sessions should have no effect on Session 1), overall: r(91) = .09, p = .40, 95% CI = [−.12, .29]; last 600 trials: r(92) = .11, p = .271, 95% CI = [−.09, .31], but were moderately negatively correlated in Session 2, overall: r(91) = −.34, p < .001, 95% CI = [−.51., −.14]; last 600 trials: r(91) = −.34, p < .001, 95% CI = [−.51, −.15]. This further confirms that participants who were exposed to more similar sequences (i.e., lower LD) in Sessions 1 and 2 demonstrated larger procedural learning effects in Session 2 (Figure 3).

Table 2.

Predictors of the similarity effect on the magnitude of procedural learning.

Fixed effects	b	SE	T	p	CI
(Intercept)	6.068	0.013	474.508	<.001	6.042	6.068
Probability	0.033	0.002	19.253	<.001	0.030	0.033
Session	0.051	0.003	15.181	<.001	0.044	0.051
Levenshtein distance	−0.016	0.015	−1.060	.292	−0.046	−0.016
Probability × Session	−0.002	0.002	−1.524	.131	−0.006	−0.002
Probability × Levenshtein distance	−0.003	0.002	−1.518	.133	−0.007	−0.003
Session × Levenshtein distance	−0.005	0.004	−1.204	.232	−0.013	−0.005
Probability × Session × Levenshtein distance	0.006	0.002	3.178	.002	0.002	0.006
Random effects			Variance			SD
Participant (Intercept)	0.0145		0.120
Participant: Session (Slope)	0.0009		0.030
Participant: Probability (Slope)	0.0002		0.013
Participant: Session × Probability (Slope)	0.0001		0.012

Open in a new tab

Indicated in bold are the contrasts that survived correction for multiple comparisons using the Holm–Bonferroni method.

Figure 3. — Relationship between Levenshtein distance and difference scores for both sessions for all trials.

H3: reliability

Split-half reliability (see Table 3) was very similar in both sessions for the overall task and the last 600 trials; using random slopes rather than raw difference scores as the metric of learning yielded numerically higher estimates of reliability. The split-half coefficients ranged from .55 to .71 (>.70 is considered adequate (Furr & Bacharach, 2008).

Table 3.

Split-half reliability of the procedural learning measures for overall and last 600 trials of the SRTT for Session 1 (SRT1) and Session 2 (SRT2).

Task	Trials	Split-half reliability
Task	Trials	N	Difference scores	N	Random slope
SRT1	1,000	95	r = .55 (.39, .67)	95	r = .68 (.56, .78)
SRT1	Last 600	94	r = .50 (.34, .64)	94	r = .71 (.59, .80)
SRT2	1,000	91	r = .62 (.47, .73)	94	r = .70 (.58, .79)
SRT2	Last 600	93	r = .55 (.39, .68)	93	r = .63 (.49, .74)

Open in a new tab

Split-half reliability correlations are significant (p < .05). SRTT: serial reaction time task.

Test–retest reliability of the RTs themselves (e.g., the RT for probable trials in Session 1 with the RT for probable trials in Session 2) was high with a value equal or superior to .80. However, test–retest reliability of procedural learning effect was poor (r = .08–.17), irrespective of which measure was used and whether all RTs were included or just the final 600 trials (Table 4).

Table 4.

Test–retest reliability of the procedural learning measures for overall and last 600 trials of the SRTT.

Task	Trials	Test–retest reliability
Task	Trials	N	Difference scores	N	Random slopes
SRT1–SRT2	1,000	91	r = .14 (−.06, .34)	91	r = .17 (−.04, .36)
SRT1–SRT2	Last 600	91	r = .08 (−.12, .28)	91	r = .17 (−.04, .36)

Open in a new tab

Test–retest reliability correlations are nonsignificant (p > .05). SRTT: serial reaction time task.

The levels of agreement between difference scores were explored via creating Bland–Altman plots (Figure 4). The Bland–Altman plots for the difference scores reveal that very few data points lie outside the limits of agreement (−57.53, 55.47), with a mean difference of −1.03; 95% CI = [−7.03; 4.98]. However, although most data points lie within the limits of agreement, there are still considerable discrepancies between time points as evidenced by the poor precision of these limits, indicating a high degree of variance between sessions compared with between-subject variance, thus suggesting that the degree of agreement is not acceptable (Bland & Altman, 1999).

H4: similarity and test–retest reliability

Following the significant interaction between similarity and procedural learning, test–retest reliability was compared for participants with low and high sequence similarity scores (achieved by performing a median split). Test–retest reliability was poor for both the high- and low-similarity groups, with no significant differences between groups (overall task: z = .83, p = .41; last 600 trials: z = .15, p = .88) (Table 5).

Table 5.

Test–retest reliability of the procedural learning measures for high and low-similarity groups measured for overall and the last 600 trials of the SRTT.

Similarity	Random slopes		Test–retest reliability
Similarity	Trials	N	Random slopes
Low	1,000	46	r = .30 (.01, .55)
Low	Last 600	46	r = .22 (−.08, .48)
High	1,000	47	r = .13 (−.17, .40)
High	Last 600	48	r = .20 (−.10, .46)

Open in a new tab

All correlations are nonsignificant (p > .05). SRTT: serial reaction time task.

Also, we tested for the possibility that similarity might have an impact on test–retest reliability by including similarity in a linear regression model which predicted the regression slopes in Session 2 from the regression slopes in Session 1 (mean-centred), LD (mean-centred) and their interaction. We found no evidence that LD moderated the relationship between procedural learning Sessions 1 and 2. Although LD was predictive of the size of the effect in Session 2 (overall: b = −.37, SE = .11, t = −3.55, p < .001, 95% CI = [−.58, −.16]; last 600 trials: b = −.37, SE = .11, t = −3.47, p < .001, 95% CI = [−.58, −.16]), there was no significant interaction between procedural learning in Session 1 and LD, thus suggesting that similarity did not influence the test–retest reliability of the SRTT (overall: b = .06, SE = .09, t = .69, p = .493, 95% CI = [−.12, .24]; last 600 trials: b = −.006, SE = .11, t = −.06, p = .952, 95% CI = [−.22, .20]).

Discussion

Experiment 1 examined the reliability of the procedural learning effect, as captured by a probabilistic SRTT, and examined the impact of the similarity of the sequences on the magnitude and stability of procedural learning. As expected, robust procedural learning effects (i.e., faster responses to probable than improbable trials) were observed. However, the level of procedural learning in a subsequent session was substantially influenced by how similar the new sequence was to a previously learned sequence. That is, greater similarity between sequences was associated with larger procedural learning effects for the new sequences. Furthermore, despite observing adequate levels of split-half reliability within each session (random slopes: .68−.72), test–retest reliability was very poor, regardless of the level of similarity between sequences (r < .18).

The positive correlation between the procedural learning effect and sequence similarity aligns with previous results (e.g., Siegelman & Frost, 2015). West et al. (2021) tested participants on a probabilistic SRTT, with a 3- to 4-day interval between sessions, and found no significant differences in performance between sessions. However, West et al. used distinct sequences at test and retest with the aim of reducing practice effects. Together with the present results, these studies suggest that the SRTT is prone to practice effects when subsequent sessions use similar sequences. The present study cannot speak of the mechanism/s that underlie the benefit of similarity on procedural learning. However, in light of the lack of evidence for a relationship between explicit awareness and the level of similarity between sequences (see Supplementary Materials 2), one possibility is that consolidated knowledge of the first-learned sequence aids the acquisition of the second-learned sequence (Brown et al., 2009; Press et al., 2005; Robertson et al., 2004) or that knowledge of the first-learned sequence proactively interferes with the acquisition of the second-learned sequence (Desmottes et al., 2017).

The suboptimal test–retest reliability of the SRTT observed here is also generally consistent with previous findings. However, our test–retest coefficients were considerably lower than Siegelman and Frost (2015; r = .47) and West et al. (2021; r = .70), irrespective of similarity between sequences at both time points. Our coefficients are more akin to those obtained by West et al. (2018, 2021) in children (r = .21; r = .26, respectively). The low test–retest reliability of the SRTT is striking, particularly in the context of robust group-level procedural learning effects and despite high stability of overall RTs. One possibility is that difference scores, in general, are intrinsically less reliable than their component parts. This has been suggested by Hedge et al. (2018) as difference scores contain measurement error from both measures which leads to an increase in the proportion of measurement error relative to between-subject variance. Yet, the limitations of using difference scores does not seem to pose as much of an issue when analysing the split-half reliability, nor does it explain the better test–retest reliability observed by Siegelman and Frost (2015) and West et al. (2018) despite also analysing difference scores. Furthermore, if difference scores were solely responsible for poor reliability, one would expect better outcomes for the random slopes. Unfortunately, that was not the case. Thus, other factors must contribute to the pattern of lower stability than split-half reliability.

It is possible that specific differences in design between our experiment and West et al. (2021) can account for the divergent findings. First, West et al. (2021) recruited older participants (18–61 years, M = 25.33 years, SD = 10.33 years) than in Experiment 2 (17–34 years, M = 20.09 years, SD = 2.09 years). This could have contributed to increasing the stability of the SRTT as test–retest reliability has been found to increase with age in intelligence measures (Schuerger & Witt, 1989). While presentation rates and age of participants have been shown to affect the procedural learning effect on the SRTT (presentation rates: e.g., Arciuli & Simpson, 2011; Emberson et al., 2011; Frensch & Miner, 1994; Soetens et al., 2004; Willingham et al., 1997; age: e.g., Brown et al., 2009; Juhasz et al., 2019) there is no evidence, to our knowledge, of its impact on the test–retest reliability of the task. Second, West et al. (2021) included a 250-ms ISI between trials, which was absent in our experiment with the aim of reducing explicit awareness (Destrebecqz & Cleeremans, 2001). The inclusion of an ISI, however, could have contributed to the higher test–retest reliability by inducing stronger representations of the sequence (Cleeremans & Sarrazin, 2007; Gaillard et al., 2009), with explicit awareness possibly emerging as a consequence of the increased signal strength (Cleeremans, 2011; Timmermans et al., 2012). However, our data did not show indication that the magnitude of procedural learning was associated with explicit awareness (for more details see Supplementary Materials 2). Furthermore, a follow-up experiment (fully described in Supplementary Materials 4) replicated more closely the design adopted by West et al. (2021) by including a 250-ms ISI and participants with ages between 18 and 60 years. Yet, this experiment still revealed suboptimal test–retest reliability (r < 21). Explicit awareness levels were also similar between groups with and without an ISI. Taken together, this suggests that the superior reliability observed by West and colleagues (2021) may be explained by other design or sampling factors.

In sum, Experiment 1 obtained clear evidence of procedural learning, which was larger in the second session, particularly when the second-learned sequences were more similar to the first-learned sequences. However, test–retest reliability of procedural learning was very poor regardless of the level of similarity between sequences. Another possibility, examined in Experiment 2, is whether this variability in the procedural learning effect across sessions will diminish with further training—that is, individuals will eventually reach a “plateau” which more accurately reflects their intrinsic procedural learning capacity. Given the lack of evidence for any impact of sequence similarity on reliability of the SRTT, and the larger procedural learning effect for those learning sequences with higher similarity, sequences with high similarity were adopted in Experiment 2 to maximise the chances of participants reaching a “plateau” at an earlier stage of learning.

Experiment 2

Experiment 2 examined whether the inclusion of three sessions would increase the test–retest reliability of the SRTT, since, as suggested by Conway et al. (2019) the poor reliability of probabilistic procedural learning may be related to the measurement of earlier stages when learning might not be as robust. Palmer et al. (2018) have demonstrated patterns of increased stability on a variety of measures of cognitive ability commonly used to assess striatal dysfunction by increasing the number of training sessions. They reported that practice effects diminished in patients with striatal impairments by the third session, thus increasing the stability of the measures. Although Palmer et al. (2018) did not consider the SRTT, it is possible that it would follow a similar stabilisation trajectory, since the striatum has also been strongly implicated in performance on this task (Robertson et al., 2001; Torriero et al., 2004).

Experiment 2 also carried out a preliminary examination of the relationship between procedural learning and language and literacy. According to the Procedural/Declarative model (Ullman et al., 2020; Ullman & Pierpont, 2005), performance on language measures (particularly grammar and phonology) and literacy measures (e.g., spelling, which requires procedural learning) should be associated with procedural learning. However, such correlations have not been consistently found in previous studies. If these correlations are masked by the low stability of the SRTT and if incorporating multiple sessions increases stability, then stronger correlations would be expected with procedural learning effects measured at later sessions. This hypothesis is supported by West et al. (2021), who found, in their children’s sample, small to moderate correlations between linguistic/literacy measures and procedural learning captured in a second session, but not a first session.

Finally, Experiment 2 considered the role of attention in relation to procedural learning stability. An extensive literature has considered the role of attention in procedural learning in the context of dual task paradigms. Such studies demonstrate a detrimental effect on procedural learning when participants simultaneously perform the SRTT alongside a secondary task (deterministic sequences: Coomans et al., 2014; Schumacher & Schwarb, 2009; Shanks et al., 2003; probabilistic sequences: Shanks et al., 2005). In line with this, a positive correlation between sustained attention and procedural learning in children has been found by Sengottuvel and Rao (2013) and West and colleagues (2021). In the latter, it was also observed that the attentional demands of the SRTT may vary depending on the session: although attention was found to positively correlate with procedural learning at both sessions, stronger correlations were observed for Session 2. Furthermore, when attention was entered as a predictor of children’s attainment (on measures of reading, grammar, and arithmetic), in a latent variable path model which also included the SRTT, measures of declarative learning and attention, attention and declarative memory contributed unique variance, but the SRTT did not. This suggests that while the SRT may be a weak correlate of language and related skills, this may be the result of overlapping variance with other variables, such as attention. This is further supported by the strong correlation between attention and procedural memory (r = .56) observed in West et al. (2021).

However, in West et al. (2021), a 9-point observational rating scale was used to estimate the levels of attention throughout the SRTT, while Sengottuvel and Rao (Sengottuvel & Rao, 2013) assessed the offline attention skills through a two-choice RT task. For both attentional tasks information regarding their psychometric properties is lacking, with the operationalisation of attention used by West et al. (2021) potentially tapping into other constructs such as motivation/boredom required for children to remain focused on the task (e.g., R. S. J. d. Baker et al., 2010; Godwin et al., 2016). Here, a direct measure of attention (i.e., a psychomotor vigilance task) was adopted to further explore the relationship between procedural learning and attention.

Experiment 2 used the same SRTT as in Experiment 1 but on three separate sessions, to address three research questions and test the following accompanying preregistered hypotheses (https://osf.io/yb3sv):

H1: Participants are expected to demonstrate evidence of procedural learning in all three sessions.
H2: Moderate to low test–retest reliability levels are expected between Sessions 1 and 2;
H3: If stability of performance increases with the number of sessions, test–retest reliability will be higher between Sessions 2 and 3 than between Sessions 1 and 2;
H4: Split-half reliability will be higher for later sessions when compared with Session 1;
H5: Procedural learning is expected to correlate with language and literacy performance/scores in all sessions;
H6: Higher associations between language and procedural learning will be expected in later sessions if the procedural learning effects are more reliable at later sessions;
H7: Participants with better attention skills will be expected to show more procedural learning;
H8: Higher correlations between procedural learning and attention are expected for later sessions.

No hypotheses were preregistered regarding how attention influences stability between sessions as, to our knowledge, this has not been previously tested using the SRTT. Exploratory analyses were therefore performed to examine relationships between attention and stability.