Test–retest reliability, practice effects and estimates of change: A study on the Mindmore digital cognitive assessment tool

Abstract

The present study aimed to establish test–retest reliability and investigate practice effects of the Mindmore cognitive assessment tool, a digital adaptation of traditional pencil and paper tests designed for self‐administration. Additionally, normative change scores for the most frequently used tests were derived. A total of 149 healthy Swedish adults (aged 20–79) completed the test battery twice, 1 month apart. The battery assessed attention and processing speed, memory, language, visuospatial functions, and executive functions. Test–retest reliability, measured by ICC and Spearman coefficients, and practice effects were estimated for 22 main‐scores and 33 sub‐scores. Regression models were used to assess change in performance while controlling for demographics, computer equipment, testing location (online or in‐laboratory) and baseline performance for 12 main‐scores and nine sub‐scores. Test–retest reliability was good for 11 main‐scores (≥0.70), satisfactory for five (0.60–0.69), and minimal for six (<0.60) albeit three having satisfactory sub‐scores. Practice effects were observed for tests with a major speed component, but not for reaction time, sustained attention, verbal memory and naming (alternate forms), nor visuospatial functions. Trackpad negatively influenced change for one test. Demographics and testing location did not significantly affect the change scores. Our study provides support for test–retest reliability and practice effects of the Mindmore cognitive assessment tool which were comparable to those of traditional tests. These findings, together with the normative change scores, can aid researchers and clinicians in interpreting test results and distinguishing between normal variations in performance and changes indicative of clinical impairment.

INTRODUCTION

Cognitive tests are integral in healthcare and neuropsychological research. Clinicians and researchers use these tools to evaluate cognitive function, providing insights into cognitive health and dysfunction. Recent years have seen advancements in digital cognitive testing tools which offer standardized administration, precise measurement of variables, and efficient record‐keeping (Zygouris & Tsolaki, 2015). Additionally, they may be found to be cost‐effective and to provide the flexibility to be tailored to individual patient needs (Mackin, Insel, Truran et al., 2018; Ruano, Sousa, Severo et al., 2016). The adoption of these digital tests, especially of those with remote administration, has grown in response to the challenges brought on by the COVID‐19 pandemic (Bilder, Postal, Barisa et al., 2020). However, many clinicians and researchers are hesitant of fully adopting these new tools (Bauer, Pozehl, Hertzog, Johnson, Zimmerman & Filipi, 2012; Miller & Barr, 2017), as they often lack extensive psychometric data on norms, validity, reliability, sensitivity, and specificity (Charalambous, Pye, Yeung et al., 2020; Giaquinto, Battista & Angelelli, 2022).

Digital cognitive tests for routine monitoring of cognitive changes could facilitate more regular assessments, which may be helpful in identifying subtle changes in cognitive function. However, the reliability of these assessments must first be thoroughly verified. For researchers and clinicians alike, psychometric evidence on test–retest reliability, practice effects and normative change, is essential. Such information is necessary to determine whether shifts in cognitive scores represent actual changes or are artifacts of the testing process.

Test–retest reliability assesses the consistency of test scores over time and is essential for establishing the stability and reproducibility of test results. Practice effects refer to the change in test scores that occurs as a result of prior exposure. They are essential for understanding the potential reasons of an improvement, or lack of improvement, observed in the test results. Normative change‐scores take reliability, practice effects, and their interaction with patient demographics and other variables into account. Clinicians can thus use these to compare the observed change in test scores in their patients to a calculated expected change. Importantly, these statistical aids should be used in combination with clinical observations, symptom severity, and patient report when drawing conclusions on the clinical implications of variations in cognitive test performance from one test occasion to another (e.g., see Lansdall, McDougall, Butler et al., 2022).

This study investigated the test–retest characteristics of the collection of traditional cognitive tests adapted for self‐administration on the digital platform Mindmore. These tests have been shown comparable in terms of correlations and score equivalency to traditional pencil and paper variants (Bjorngrim, van den Hurk, Betancort, Machado & Lindau, 2019), and there is a large platform‐specific normative database for the healthy Swedish population (Van den Hurk, Bergman, Machado, Bjermo & Gustavsson, 2022). Several studies are ongoing in clinical populations (e.g., Amirpour, Eckerblad, Bergman & Nilsson, 2024; Franke Foyen, Sennerstam, Kontio, Lekander, Hedman‐Lagerlof & Lindsater, 2023; Van den Hurk, Johansson, Classon, Gustavsson, Nägga & Segernäs, 2024).

We expected that the reliability and practice effects of the digital tests would be comparable to their traditional counterparts, as supported by comparative validity studies (Bjorngrim et al., 2019; Latendorf, Runde, Salminen & Steinert, 2021; Ye, Ko, Phi et al., 2020). According to the Criteria and guiding principles for evidence‐based grading of measurement instruments by Sweden's Local Governments and Regions (SKR, 2023), we anticipated reliability coefficients to be satisfactory (0.60–0.69) to good (≥0.70) for most cognitive tests, though often minimal (<0.60) for test scores subject to floor or ceiling effects, variability in practice effects, or time variability in cognitive abilities (Calamia, Markon & Tranel, 2013). Practice effects are anticipated for most test results given the healthy population and relatively short test interval, particularly for test results heavily influenced by processing speed (Fernandez‐Marcos, de la Fuente & Santacreu, 2017; Lezak, Howieson, Bigler & Tranel, 2012). Minimal practice effects were expected for tests in the visual memory and visuospatial domains (Calamia et al., 2012), though significant variability in effect sizes might occur within each domain.

The aim of the study was to add psychometric data for the digital cognitive assessment tool Mindmore by investigating test–retest reliability and practice effects over a four‐week interval for 22 main test scores and their sub scores. Results were compared to those documented in the literature for the traditional pencil and paper variants of the tests. In addition, normative change scores were computed for the most frequently used tests, while controlling for demographic variables, input method, computer type, screen size, testing location, and baseline scores.

METHODS

Participants

Healthy Swedish‐speaking adults were recruited on two separate occasions, first in 2020 and subsequently in 2021/2022. No reimbursement for participation was provided. The study was initiated shortly before the onset of the COVID‐19 pandemic. Initially, 52 students from Skövde University were approached for in‐laboratory test–retest on touchscreen tablets. As the pandemic unfolded, the study transitioned to a remote format to enable wider recruitment, outside of the student population and with a wider age range. In 2021/2022, the focus was on collecting normative data for remote online testing and 200 participants who had done baseline testing were approached for a retest. Participants in the remote online group were recruited via social networks (friends, family, sharing at social media, Linkedin, etc.), Mindmore's website, a Facebook advertisement, and Accindi.se (a webpage where research studies advertise to find participants).

In both groups, the subjects received questionnaires about mental and physical health, Hospital anxiety and depression scale (HADS; Zigmond & Snaith, 1983) and, due to an increasing prevalence of stress related disorders in Sweden (Norlund, Reuterwall, Hoog, Lindahl, Janlert & Birgander, 2010) and its relatedness to cognitive impairment (Malmberg Gavelin, Domellöf, Åström, et al., 2021), the Karolinska exhaustion disorder scale (KEDS; Beser, Sorjonen, Wahlberg, Peterson, Nygren & Asberg, 2014) to verify eligibility. Follow‐up interviews were given to clarify given answers. In addition, all subjects above age 65 who no longer participated in working life were given a short cognitive screening test, short portable mental state questionnaire (SPMSQ; Pfeiffer, 1975; Svedjebrant & Lindmark, 2011) or a Mini‐mental state examination (MMSE; Folstein, Folstein & McHugh, 1975). It was required that all participants had basic computer skills (e.g., being able to use touchscreen, mouse or trackpad) and for those performing the tests online having access to a laptop or stationary computer with internet connection.

The exclusion criteria, applied after the participants were entered into the study were: (1) unsatisfactory testing conditions (e.g., environmental disturbances reported not to be limited only to single tests, or lack of basic computer skill); (2) less than fluent in Swedish language (and for the normative change scores, exclusion from verbal tests if not having Swedish as mother tongue); (3) previous experience of Mindmore tests or experience as test administrator of corresponding traditional tests; (4) SPMSQ ≥ 3, MMSE < 25 or under examination/has applied for examination at a memory clinic; (5) HADS‐D ≥ 11 or currently affected by depression; (6) KEDS ≥ 19 or currently affected by stress related exhaustion disorder; (7) ADHD/ADD with or without medication; (8) active disease in the central nervous system or psychiatric condition; or (9) history of episodes or conditions which can affect cognition (e.g., stroke, significant head trauma, earlier or present substance abuse). In addition, dyslexia meant exclusion of TMT B (i.e., a test requiring the alphabet being learned by heart) and colorblindness exclusion of tests with colored stimuli. Treated chronic diseases (e.g., diabetes mellitus, hypertension, heart disease) were not an exclusion criterion.

Of the initial sample of 252 subjects, four had unsatisfactory testing conditions, four had high HADS‐D scores and 29 high KEDS scores, three had ADHD/ADD and two a history of episodes or conditions which can affect cognition. Of the remaining 210 subjects, one student cancelled the baseline test and 60 baseline tested (six students and 54 online) did not return for a retest assessment. Independent samples t‐tests showed that these non‐returners were similar in demographics and, in general, also in cognitive characteristics to those who returned for follow‐up testing (see Table S1 in Appendix S1 Part A). The demographics of the final sample, the remaining 149 volunteers, are shown in Table 1.

Table 1.

Demographics. screen size and used computer equipment per location and the total sample

Location	Age			Education			Screen size a		n
	Range	M	SD	Range	M	SD	M	SD	Men	Women	Mouse	T‐pad	L‐top b	D‐top b	Tot
In‐laboratory	20‐55	25.8	6.59	9–18	14.0	2.05	12.3	0.00	15	22					37
Online	21–79	52.9	18.1	7–26	15.3	2.98	16.3	4.65	36	76	77	35	92	19	112
Total sample	20–79	46.2	19.8	7–26	15.0	2.83	15.3	4.38	51	98	77	35	92	19	149

Notes: In‐laboratory overlapped with touchscreen, T‐pad = trackpad, L‐top = laptop, D‐top = desktop.

^a Missing data on screen size for two participants were replaced with the series mean.

^b Data was missing on computer type for one participant.

As the table shows, there was a predominance of female participants in both the in‐laboratory and online subgroups. An independent‐samples t‐test showed that the participants in the in‐laboratory subgroup were significantly younger [t(147) = 8.91, p < 0.001]. One hundred and eighteen participants had 13 or more years of formal education, corresponding to university or college, while 27 had 10–12 years corresponding to high school or equivalent, and only four had 6–9 years of formal education corresponding to primary school or equivalent. All participants but seven had Swedish language as mother tongue. The majority of the participants tested online used laptop with mouse as input method. Screen size online varied between 9.3 and 32 inches, and was 12.3 inches (same for all) for those tested in laboratory on touchscreen tablet.

Procedure

Participants tested online were instructed to find a quiet environment without distractions and to complete the test in one sitting. The test battery was followed by an online debriefing on test conditions during the assessment (e.g., disruptions and technical issues). About 4 weeks later (mean = 27.9 days; SD = 6.1), the test battery was repeated at the same location (in‐laboratory or at‐home) using the same test order. We selected a 4‐week interval, which, while shorter than the intervals commonly used in clinical settings, is typical for psychometric studies (Batista, Giatti, Barreto, Galery & Passos, 2013; Cangoz, Karakoc & Selekler, 2009; Delaney, Prevey, Cramer & Mattson, 1992). This interval was chosen to strike a balance between minimizing variability in cognitive function between the test sessions and limiting the impact of practice effects. Additionally, it proved to be a practical duration for the participants in our study. The study was approved by the Regional Ethics Review Board in Stockholm (issue number 2021‐05215).

Cognitive tests

The tests assessed attention and processing speed, memory, language, visuospatial functions, and executive functions. All included tests and their main and sub scores are presented in Table 2 and some illustrations of the tests are shown in Fig. 1. For detailed test descriptions, see Appendix S1 Part B.

Table 2.

Included neuropsychological tests

Test	Description	Main scores	Sub scores
Attention and processing speed
Trail making test part A (TMT A; Army individual test battery, 1944)	Mental speed (drawing version for touchscreen)	TMT A index = correct connections/Time (s)	Correct connections Time (s)
Trail making test part A (CTMT A; Army individual test battery, 1944)	Mental speed (click‐version for mouse/trackpad)	CTMT A time (s)
Trail Making Test part D (CTMT D; Army individual test battery, 1944)	Motor speed (click‐version for mouse/trackpad)	CTMT D time (s)
Symbol digit processing test (SDPT; see Fig. 1b)	Visual scanning (similar to SDMT; Smith, 1982).	SDPT correct = correct answers	Errors
Simple reaction time (SRT; in‐house developed test)	Simple reaction time (version for touchscreen).	SRT index = correct reactions/RT mean (ms)	Correct reactions RT mean (ms)
Simple reaction time (click‐SRT; in‐house developed test)	Simple reaction time (click‐version for mouse/trackpad)	Click‐SRT mean (ms)
Continuous performance test (CPT; in‐house developed test)	Sustained attention (click‐version for mouse/trackpad)	CPT index = commissions + omissions	Commissions Omissions RT mean (ms) RT SD (ms)
Memory
Rey auditory verbal learning test (RAVLT; Rey, 1964; Schmidt, 1996)	Verbal episodic memory (five learning trials of 15 words, distraction list, delayed recall after about 20 min, recognition)	Learning Immediate recall Delayed recall Recognition = hits – false alarms	Double recalls Intrusion errors
Corsi block‐tapping test (Corsi, 1972; Milner, 1971)	Visual working memory (forward followed by backward condition)	Corsi index = sum of forward and backward partial credit scores (PCSs)	Span forward PCS forward Span backward PCS backward
Language
Boston naming test (BNT‐15; Jørgensen, Johannsen & Vogel, 2017; Kaplan, Goodglass & Weintraub, 1983)	Naming (15‐items short‐version; see Fig. 1d)	BNT‐15 = number of correct answers
FAS word fluency test (Benton, Hamsher & Sivan, 1989)	Word fluency (1 min per letter)	FAS index = sum of declared words on letters F, A and S.	Letter F Letter A Letter S
Visuospatial functions
Cube copying test (Necker, 1832)	3D copying	Cube copying (scoring = 0, 1, 2 or 3; see the text).
Clock drawing test (Critchley, 1953; Shulman, Shedletsky & Silver, 1986)	Spatial orientation	Clock drawing (scoring = 0, 0.5 or 1; see the text).
Executive functions
Trail making test part B (TMT B; Army individual test battery, 1944)	Mental flexibility (drawing version for touchscreen)	TMT B index = correct connections/Time (s)	Correct connections Time (s)
Trail making test part B (CTMT B; Army individual test battery, 1944)	Mental flexibility (click‐version for mouse/trackpad; see Fig. 1a)	CTMT B time (s)
Complex reaction time (CRT; in‐house developed test)	Complex reaction time (version for touchscreen.	CRT index = (correct reactions – commissions)/RT mean (ms)	Correct reactions Commissions RT mean (ms)
Complex reaction time (click‐CRT; in‐house developed test)	Complex reaction time (click‐version for mouse/trackpad)	Click‐CRT mean (ms)	Commissions
Paced auditory serial addition test (PASAT; Gronwall, 1977; Gronwall & Sampson, 1974)	Information processing	PASAT index = correct responses – errors	Correct responses Errors Omissions Longest sequence
Victoria Stroop test (Strauss, Sherman & Spreen, 2006; Stroop, 1935)	Concentration effectiveness (silent version; see Fig. 1c)	Stroop index = CW correct responses/CW time (s): see the text	Interference (ms) W time (ms) W correct CW time (ms) CW correct

Fig. 1.

Examples on Mindmore tests. (a) The CTMT B practice task preceding the testing task. Click in incorrect circle is indicated with red color. (b) The SDPT test. Mirrored geometric symbol pairs are presented in positions 1 and 8, 3 and 5, and 2 and 9. The random geometric symbols were chosen to have low complexity yet distinct and equally simple to perceive. (c) The silent Victoria Stroop test, incongruent condition. (d) The BNT 15 test (fictive stimulus item).

In addition to the learning, recall and recognition trials, the RAVLT scores double recalls and intrusion errors were included as scores reported to be sensitive to ADHD (Pollak, Kahana‐Vax & Hoofien, 2008). In addition to span, the Corsi test included partial credit scoring, in which credit is given to partly correct items (Conway, Kane, Bunting, Hambrick, Wilhelm & Engle, 2005).

Scoring of the cube was based on the method in the Rowland universal dementia assessment scale (RUDAS; Storey, Rowland, Basic, Conforti & Dickson, 2004). That is, the cube was given one point if it was based on a square rather than a rectangle or other shape. It could be skewed, warped and asymmetric. A second point was given if all six internal lines appeared correct and without additional lines, and a third point was given if all six external lines forming the outer contour appeared correct. The corners of the cube were required to be distinct, not indistinctly rounded, and all lines were required to be connected to the corners with due tolerance for limited precision of the input method. As a general rule for scoring the cube it was regarded better to err on the side of caution. The clock was scored 0 = failed, 0.5 = only hands or numbers correct, or 1 = both hands and numbers correct. Equivalent forms of the two tests were administered irrespective of if the test taker was using touchscreen, mouse or trackpad.

Performance indices were calculated taking both accuracy and speed into account, so called “throughput measures” (Thorne, 2006) for the following tests: TMT A and B (correct connections/time), SRT (correct responses/reaction time), CRT [(correct responses – commission errors)/reaction time], and Stroop (correct responses CW/reaction time CW), where CW stands for conflicting (and W for non‐conflicting) color of the ink and the word. In addition, indices were calculated for CPT (commissions + omissions), RAVLT recognition (hits – false alarms), PASAT (correct responses – errors), Stroop interference (reaction time CW − reaction time W), and the Corsi test (forwards partial credit score + backwards partial credit score). The Corsi index was founded on the premise that the Corsi test relies on processing within the working‐memory systems and is conceptually similar to verbal digit span tasks, but because of the visual presentation, the distinction between forward (attention span) and backward (working memory span) conditions are not as clear‐cut (Kessels, Van den Berg, Ruis & Brands, 2008).

The RAVLT, FAS and BNT‐15 tests used speech to text software to deliver verbatim transcripts. The transcripts were afterwards corrected manually by trained human raters listening to the sound files. The same raters were not used at each assessment. Inter‐rater reliability (Spearman) for the human raters versus the speech to text rating software, assessed for 400+ online test‐takers, is 0.97 for RAVLT and .94 for FAS and BNT‐15 (Bergman, 2023). Alternate forms A and B for RAVLT and BNT 15 were given to all participants. In addition, randomized stimuli were given for the SDPT, however, due to the time of the release, only to participants tested online.

The tests were administrated in the same order across the first and second test occasion, however, the order differed between in‐laboratory and online settings. In‐laboratory the order was: TMT A and B, PASAT, RAVLT learning and immediate recall, SRT, CRT, Stroop, Corsi, RAVLT delayed recall and recognition, SDPT, FAS. Online the order was: CTMT A, B and D, RAVLT learning and immediate recall, SDPT, Corsi, Stroop, RAVLT delayed recall and recognition, cube copying, PASAT, clock drawing, FAS, BNT‐15, click‐SRT, click‐CRT (or CPT). A new order was designed for the online setting due to the replacement of TMT A and B, SRT and CRT by tests designed for mouse/trackpad use: CTMT, click‐SRT, click‐CRT, and CPT. Additionally, the online battery included the additional tests cube copying, clock drawing, and BNT‐15. For about half of the participants the online test battery included the click‐CRT, for the other half it included the CPT, because these tests are based on the same design and cannot be administrated in one test battery. The T1 (T2) test batteries took in median 47 (45) min to complete in the laboratory, 55 (54) min online with click‐CRT, and 70 (67) min with CPT.

Statistical analysis

Test–retest reliability was estimated by computing intra‐class (ICC; Absolute agreement, Single scorings) and Spearman correlation coefficients. ICC estimates were replaced with Spearman estimates when any of the distributions of baseline and follow‐up scores were highly skewed (i.e., skewness larger than ±1) and when measurements were taken from an ordinal scale (i.e., the “throughput” indices produce ordinal scales by dividing one interval scale with another).

Practice effects were tested for significance by paired t‐tests and expressed in effect size units [Cohen's d = M (follow‐up score – baseline score)/SD (follow‐up score – baseline score); Cohen, 1992]. For the SDPT, practice effects for the group tested online given randomized stimuli and the group tested in laboratory given non‐randomized stimuli were compared.

We applied the fourth‐order polynomial formula by Piovesana & Senior (2018) to compute minimum sample sizes to obtain stable estimates of the change score's population mean and standard deviation within the 90% confidence intervals (CIs). The formula takes the varying skewness of the test sample distributions into account to estimate minimum sample sizes. Test scores showing sample sizes sufficient to obtain stable estimates, and satisfactory or better test–retest reliability (≥0.60), were included to constitute the change norms.

In assessment of the significance of change in test scores over time we corrected for demographics (age, education, sex), and administrative variables (input method, computer type, screen size and location) according to the regression‐based change approach by Van der Elst, Van Boxtel, Van Breukelen & Jolles (2008). Since this method does not control for practice effects and regression to the mean, we added a second step in which we included the baseline score as an additional predictor to the significant predictors from the first step. A discussion on method selection and details of the two steps are included in Appendix S1 Part D.1.

The assumptions of regression analyses, homoscedasticity, normal distribution of the residuals, and absence of influential cases, were tested for each model. Uncertainty in the regression line was accounted for by using the standard error of the estimate (SEest), which is the standard deviation of the residuals, as error term (McSweeny, Naugle, Chelune & Lüders, 1993). Extreme scores were accounted for by subgrouping individuals according to predicted change scores for regression residuals that do not fulfill the homogeneity of variance assumption (Van der Elst, Van Boxtel, Van Breukelen & Jolles, 2008). That is, homoscedasticity was evaluated by subgrouping the participants into tertiles of predicted scores and applying Levene's test of homogeneity of variance to the residuals. Further, following Van der Elst, Van Boxtel, Van Breukelen & Jolles (2008), we present the 90% prediction intervals (PIs) of the observed 5th and 95th percentiles of the distribution of the standardized residuals. Alpha was set to 0.01. All analyses were performed using SPSS version 7.5 and MedCalc version 22.023.

Literature search

Findings on test–retest reliability and practice effects were compared with those reported for traditional tests (i.e., excluding RT and CPT which are not available in pencil and paper). For that purpose, we searched in the PubMed database and Google Scholar for reports in English on traditional pencil and paper tests' test–retest reliability and practice effects. The search was limited to reports on samples of healthy adults with retest intervals up to one month and, where not found, and for meta‐studies, up to one year. In the identified reports, all included traditional tests corresponding to digitized Mindmore tests were included in the comparison.

Removal of single test results

Eight test results, distributed among three participants, were not registered due to technical issues. Two TMT B results were removed due to dyslexia, and the FAS and BNT‐15 results were removed for one participant who was not test‐naïve due to profession. However, the main cause of removal of single test results was reported or unreported measurement error. Measurement error was operationally defined as a significant discrepancy between an individual's hypothetical true score and the actual score obtained. The criterion for the removal of unreported measurement error was an extreme change score outside the outer lower or upper interquartile fences (i.e., below Q1 – (3 × IQR) or above Q3 + (3 × IQR).

Out of 4,168 collected main test results (calculated as up to 22 test results for each of the 149 participants across both test occasions), 15 single test results (0.4%) were missing or removed from the data due to reported measurement error (13 occurred at baseline, 2 at follow‐up). These were distributed among 13 participants and consisted of: failure on practice round leading to skipped test (six participants); accidently skipped test (five participants); affected test performance due to environmental disturbance (one participant); and uncertainty on instructions (one participant). All 15 affected T1‐T2 pairs were excluded from the data analysis.

Additionally, 37 single test results (0.9%), distributed among 24 participants (with three overlapping those with reported measurement errors), were removed due to unreported measurement error (20 occurred at baseline and 17 at follow up). All 37 affected T1‐T2 pairs have been excluded from the data analysis.

RESULTS

Test–retest reliability and practice effects

The final sample results are shown in Table 3. Test–retest reliability was good (≥0.70) for 11 main test scores: CTMT A, CTMT D, SDPT, RAVLT learning, immediate recall, and delayed recall, Corsi, FAS, CTMT B, PASAT, Stroop; It was satisfactory (0.60–0.69) for five main test scores: TMT A, SRT, RAVLT Recognition, clock drawing, click‐CRT; while it was minimal (<0.60) for six main test scores: click‐SRT, CPT, BNT, cube copying, TMT B, CRT. Of all 33 additional subscores, seven showed good (≥0.70), nine satisfactory (0.60–0.69), 15 minimal (<0.60), and two were in absolute agreement with all or nearly all participants (SRT correct, TMT A correct) reaching the maximum score. All other accuracy scores, except CPT commissions and PASAT errors and omissions, showed minimal test–retest reliability due to floor or ceiling effects.

Table 3.

Test–retest results: n, test–retest correlations (ICC/Spearman), M and SD of the baseline, follow‐up and change, t‐values of the paired differences, and their effect sizes

Cognitive test score	n	ICC/r s	Baseline		Follow‐up		Change		t	P	Cohens d
			M	SD	M	SD	M	SD
TMT A index	37	0.61b	0.99	0.25	1.09	0.28	0.10	0.21	2.75	**	0.48
Time (s)	37	0.61a	25.8	6.57	23.7	6.69	−2.11	6.32	−2.03	*	0.33
Correct	37	‐	24.0	0.00	24.0	0.16	0.03	0.16	1.00	ns	0.18
CTMT A time (s)	93	0.79	36.3	10.6	33.4	10.0	−2.94	6.15	−4.60	***	0.48
CTMT D time (s)	92	0.90	22.9	6.21	22.1	5.80	−0.83	2.56	−3.09	**	0.32
SDPT correct	95	0.92	42.7	11.0	43.9	11.3	1.17	4.37	2.61	**	0.27
Errors	95	0.13a	0.95	1.45	1.01	1.63	0.06	1.86	0.33	ns	0.03
SRT index	35	0.67b	19.0	2.86	18.7	2.45	−0.25	2.22	−0.68	ns	0.11
Mean (ms)	35	0.63	485	79.9	488	63.8	2.96	62.4	0.28	ns	0.05
Correct	35	–	9.00	0.00	9.00	0.00	0.00	0.00	0.00	ns	–
Click‐SRT mean (ms)	65	0.57	310	62.0	324	70.6	13.6	61.2	1.79	ns	0.22
CPT index	33	0.53a	1.70	1.74	1.51	1,62	−0.18	1.51	−0.69	ns	0.12
Commissions	33	0.63a	1.24	1.58	1.24	1.44	0.00	1.22	0.00	ns	0.00
Omissions	33	0.30a	0.45	0.71	0.28	0.63	−0.18	0.73	−1.44	ns	0.25
RT mean (ms)	33	0.76a	474	109	458	101	−15.5	64.1	−1.39	ns	0.24
SD (ms)	33	0.19a	125	63.0	119	64.1	−6.00	83.8	−0.41	ns	0.07
RAVLT learning	144	0.78	51.7	10.6	51.5	10.1	−0.20	6.90	−0.35	ns	0.03
RAVLT immediate recall	142	0.77	11.1	3.01	11.1	2.89	−0.01	2.02	0.08	ns	0.00
RAVLT delayed recall	142	0.80	11.0	3.18	11.0	3.08	0.06	1.96	0.34	ns	0.03
Double recalls	135	0.48a	4.04	4.79	3.46	5.56	−0.58	5.19	1.31	ns	0.11
Intrusion errors	137	0.54a	7.46	2.68	7,87	2.96	0.41	2.67	1.79	ns	0.15
RAVLT recognition	141	0.61a	13.5	1.84	13.1	1.96	−0.42	1.48	−3.36	***	0.28
Corsi index	129	0.74	86.3	27.3	93.1	30.2	6.88	19.8	3.94	***	0.35
Fw span	131	0.58	6.13	1.18	6.40	1.08	0.27	1.02	2.99	**	0.26
Fw partial credit score	129	0.68	45.8	15.4	49.5	15.0	3.65	11.9	3.50	***	0.31
Bw span	132	0.51	5.55	1.26	5.83	1.38	0.28	1.30	2.48	*	0.22
Bw partial credit score	130	0.57	40.2	16.0	43.6	18.0	3.33	15.6	2.44	*	0.21
FAS index	128	0.86	48.1	13.9	50.5	13.5	2.41	6.89	3.95	***	0.35
Letter F	128	0.68	16,3	4.95	17.6	4.94	1.30	3.79	3.87	***	0.34
Letter A	128	0.74	13.9	4.81	14.0	4.72	0.14	3.47	0.46	ns	0.04
Letter S	128	0.75	18.0	5.64	19.0	5.91	0.97	4.03	2.72	**	0.24
BNT‐15	110	0.56a	12.3	2.36	12.6	2.02	0.31	1.75	1.85	ns	0.18
Cube copying	93	0.58a	2.62	0.72	2.61	0.78	−0.01	0.63	−0.16	ns	0.02
Clock drawing	90	0.63a	0.87	0.27	0.89	0.24	0.03	0.19	1.39	ns	0.16
TMT B index	35	0.54b	0.42	0.14	0.50	0.13	0.07	0.12	3.55	***	0.58
Time (s)	35	0.61a	62.3	24.9	50.6	14.1	−11.7	16.9	−4.12	***	0.69
Correct	35	−0.20a	23.3	1.48	23.5	1.22	0.23	2.07	0.65	ns	0.11
CTMT B time (s)	87	0.78	50.6	18.5	46.6	16.4	−3.95	11.1	−3.32	***	0.36
CRT index	35	0.57b	17.4	2.64	17.1	2.11	−0.27	2.19	−0.74	ns	0.12
Mean (ms)	35	0.65	574	82.2	578	67.9	3.79	63.4	0.35	ns	0.06
Correct	35	0.36a	9.91	0.28	9.94	0.24	0.03	0.30	0.57	ns	0.10
Commissions	35	0.12a	0.11	0.40	0.17	0.38	0.06	0.54	0.63	ns	0.11
Click‐CRT mean (ms)	52	0.68	466	94.7	455	97.7	−11.0	77.3	−1.02	ns	0.14
Commissions	52	0.01a	0.56	0.87	0.58	0.85	0.02	1.21	0.11	ns	0.02
PASAT index	124	0.75a	44.8	13.7	48.9	12.2	4.03	8.42	5.34	***	0.48
Correct	124	0.77a	48.7	10.8	51.9	9.16	3.19	6.04	5.87	***	0.53
Errors	124	0.63a	3.86	4.31	3.02	3.88	−0.85	3.43	−2.75	**	0.25
Omissions	124	0.77a	7.42	8.99	5.10	7.02	−2.32	5.06	−5.12	***	0.46
Longest sequence	124	0.54	23.6	17.1	29.4	17.7	5.83	16.3	3.99	***	0.36
Stroop index	130	0.91b	13.8	4.06	14.8	4.20	1.02	1.82	6.41	***	0.56
Interference (ms)	130	0.67a	362	328	280	264	−81.8	194	−4.80	***	0.42
W time (ms)	130	0.86	1,183	263	1,162	276	−21.0	140	1.70	ns	0.15
W correct	130	0.21a	19.8	1.19	19.9	0.32	0.12	1.23	1.14	ns	0.10
CW time (ms)	130	0.92a	1,545	467	1,442	420	−103	172	−6.81	***	0.60
CW correct	130	0.18a	19.5	1.39	19.7	0.79	0.18	1.24	1.69	ns	0.15

Notes: ^a,b Spearman coefficient is shown instead of ICC because the ICC assumptions were not fulfilled due to ^a highly skewed T1 and/or T2 scores (ie., skewness larger than ±1) or ^b measurements taken from ordinal scale. BNT = Boston naming test, click‐CRT = click version of CRT, click‐SRT = click version of SRT, CPT = continuous performance test, CRT = complex reaction time, CTMT = click version of TMT, PASAT = paced auditory serial addition test, RAVLT = Rey auditory verbal learning test, SDPT = symbol digit processing test, SRT = simple reaction time, TMT = trail making test.

^* p ≤ 0.05,

^** p ≤ 0.01,

^*** p ≤ 0.001, ns = P > 0.05.

Interestingly, three of the six main test scores showing minimal coefficients still showed satisfactory coefficients on subtest scores (i.e., the time scores of TMT B, CRT and CPT, and CPT commissions). See Fig. 2 for a graph on the primary test–retest reliability coefficients, the sub score coefficients exceeding minimal main score coefficients, and their 95% confidence intervals. See also Fig. 3 and Appendix S1 Part C for test–retest scatterplots.

Fig. 2.

Plot on test–retest reliability coefficients in descending order and their 95% confidence intervals for all main scores and the sub scores exceeding minimal main score coefficients (see the text).

Fig. 3.

Test–retest scatterplots on tests with good (a, b), satisfactory (c, d) and minimal (e, f) test–retest reliability.

Significant practice effects were observed on 10 of the 22 main test scores and 14 of the 33 additional subscores. Largest effect sizes, corresponding to, or close to, medium (d ≥ 0.48), were seen for TMT A index, CTMT A time, TMT B index and time, PASAT index and correct, Stroop index and CW time.

A comparison of the SDPT practice effects between those tested online given randomized stimuli as shown in Table 3 (n = 95, mean age = 55.9 years, ICC = 0.92) and those tested in laboratory given non‐randomized stimuli (n = 37, mean age = 25.8 years, r _s  = 0.64) showed that randomized stimuli lowered the practice effect with 45%, although still being significant (1.2 points, d = 0.27, p = 0.011), as compared to non‐randomized stimuli (3.2 points, d = 0.49, p = 0.005). A partial correlation of age with the practice effects when controlling for randomized/non‐randomized stimuli could not explain the subgroup difference (r = −0.009, p = 0.918).

The results of the literature search for reports on the corresponding traditional pencil and paper tests' test–retest reliability and practice effects for healthy adults at short retest intervals are shown in Table 4. At least one report was identified for all but the cube copying test.

Table 4.

Present findings compared to traditional tests' test–retest reliability and practice effects for healthy adults at short retest intervals (i.e., except for the meta study, up to 1 month or, if no report found, up to 1 year)

Cognitive test score	Present study		Corresponding pencil and paper measure
	TRR	PE	References	n	Interval	TRR	PE
TMT A time	0.61	−2.1	Cangoz et al. (2009)	20	1 month	0.78	na
CTMT A time	0.79	−2.9	Feinkohl, Borchers, Burkhardt et al. (2020)	45	1 week	0.67	−3.3
			Siciliano, Chiorri, Battini et al. (2019)	40	3 weeks	0.80	−3.1
			Calamia et al. (2013), meta study		327 days	0.66	na
CTMT D time	0.90	−0.8	Homack, Lee & Riccio (2005)	101	na	0.77	na
SDPT correct a	0.92	1.2	Benedict et al. (2012)	25	1 week	0.84	4.2
			Benedict, Duquin, Jurgensen et al. (2008)	25	1 month	0.74	~3.0
			Magnin, Sagawa, Moulin & Decavel (2020)	19	1 week	0.92	4.8
RAVLT learning b	0.78	−0.2	Geffen, Butterworth & Geffen (1994)	51	6–14 days	0.77	0.6
			Lemay, Bedard, Rouleau & Tremblay (2004)	37	2 weeks	0.72	−1.2
			Calamia et al. (2013), meta study		275 days	0.74	na
RAVLT immediate b	0.77	0.0	Geffen et al. (1994)	51	6–14 days	0.70	0,2
			Lemay et al. (2004)	37	2 weeks	0.76	−0.3
			Calamia et al. (2013), meta study		105 days	0.63	na
RAVLT delayed b	0.81	0.1	Geffen et al. (1994)	51	6–14 days	0.66	−0.4
			Lemay et al. (2004)	37	2 weeks	0.81	−0.7
			Calamia et al. (2013), meta study		244 days	0.88	na
RAVLT recognition b	0.61	−0.4	Geffen et al. (1994)	51	6–14 days	0.38	−0.3
			Lemay et al. (2004)	37	2 weeks	0.32	−0.3
			Calamia et al. (2013), meta study		299 days	0.28	na
Corsi Fw span	0.58	0.3	Sanchez‐Benavides, Pena‐Casanova, Casals‐Coll et al. (2016)	113	1 year	0.50	0.1
Corsi Bw span	0.51	0.3	Sanchez‐Benavides, Pena‐Casanova, Casals‐Coll et al. (2016)	113	1 year	0.50	0.1
FAS index	0.86	2.4	Harrison, Buxton, Husain & Wise (2000)	90	1–8 weeks	0.82	na
			Ross, Calhoun, Cox, Wenner, Kono & Pleasant (2007)	53	44.6 days	0.84	4.9
			Calamia et al. (2013), meta study		224 days	0.79	na
BNT‐15 b	0.56	0.3	Morris, Heyman, Mohs et al. (1989)	278	1 month	0.56	0.1
Cube copying	0.58	0.0	No study found
Clock drawing	0.63	0.0	Shulman (2000), review c			0.63–0.94	na
TMT B time	0.61	−11.7	Cangoz et al. (2009)	20	1 month	0.73	na
CTMT B time	0.78	−4.0	Feinkohl et al. (2020)	45	1 week	0.69	−11.0
			Siciliano et al. (2019)	40	3 weeks	0.81	−18.3
			Calamia et al. (2013), meta study		328 days	0.77	na
PASAT correct	0.77	3.2	Correia & Figueroa (2016), review		2 weeks	0.73–0.90	6.0
Stroop interference	0.68	−0.1	Malek, Hekmati, Amiri, Pirzadeh & Gholizadeh (2013)	30	2‐weeks	0.64	na
Stroop CW time	0.92	−0.1	Lemay et al. (2004)	37	2 weeks	0.82	−0.2
			Saurman (2017), meta study			0.80	na

Notes: PE = practice effect expressed in raw scores, TRR = test–retest reliability, na = not available.

^a Randomized stimuli.

^b Alternate lists.

^c The studies reviewed used more complex scoring systems.

Of the 18 test results we were able to compare, 14 showed good (≥0.70) or satisfactory (0.60–0.69) test–retest reliability across both digital and traditional formats. Three test results, Corsi forward and backward span and BNT‐15, showed minimal reliability in both modalities. Despite a ceiling effect (see Fig. 3d), the RAVLT recognition showed satisfactory reliability in this study, although minimal reliability in the literature.

The test–retest reliability coefficients of TMT A and B for touchscreen were satisfactory but lower than those of CTMT A and B and those of traditional pencil and paper format. This discrepancy is likely due to the narrow age span (mean age of 25.8 years) and relatively narrow performance ranges for the in‐laboratory participants who make up the sample for TMT A and B for touchscreen.

Practice effects observed in this study for CTMT A time, TMT B time, and Stroop CW time were generally consistent with those found in their pencil and paper formats. The practice effect for the digital SDPT was, however, smaller than that for the pencil and paper SDMT, likely because only the digital SDPT employed randomized stimuli. Somewhat lower practice effects were also observed for the digital PASAT compared to its pencil and paper format, possibly due to a slightly longer test–retest interval.

The practice effect was lower for CTMT B, the click version, as compared to TMT B, the drawing version, irrespective of whether the test was digitized or in a pencil and paper format. The CTMT B and TMT B both require mental flexibility, but the TMT B also requires drawing the lines between circles in the correct sequence. This added requirement when combined with the requirement on mental flexibility could explain the slower initial performance on the TMT B compared to CTMT B.

Models on normative change

Seven of the 22 main test scores did not fulfill the required minimum sample size to obtain stable estimates, the cube copying and clock drawing tests were excluded from the change scores due to being essentially qualitative tests, and the BNT‐15 test was excluded due to not fulfilling the criterion on satisfactory retest reliability. In addition, the results for verbal tests were excluded for the non‐native Swedish speakers (n = 7 for RAVLT and n = 6 for FAS). After exclusion of one influential outlier on Stroop interference (Cook's distance = 2.3) the final models on the remaining 12 main test scores and nine sub scores are as shown in Table 5. Incidentally, these are the most commonly used Mindmore tests administered to patients at healthcare facilities in Sweden. Age ranges for these models were 21–79 years for CTMT, and else 20–79 years.

Table 5.

Final multiple regression models, standard error of the estimate and 90% PIs of the standardized change scores

Change score	n	Prediction equation a	P	r 2	Predicted change score	SEest b	90% PI (z)
							Lower	Upper
CTMT A time (s)	93	5.193 + (−0.224 × T1)	***	0.149		5.704	−1.627	1.520
CTMT D time (s)	92	2.596 + (−0.149 × T1)	***	0.131	≤−1.828 −1.827 to −0.586 ≥−0.585	4.275	−1.829	1.576
						1.869
						1.681
SDPT correct c	95	1.168		0.000		4.365	−1.748	1.067
RAVLT learning d	139	13.559 + (−0.267 × T1)	***	0.164		6.396	−1.595	1.604
RAVLT immediate recall d	135	2.855 + (−0.256 × T1)	***	0.150		1.866	−2.090	1.216
RAVLT delayed recall d	135	2.509 + (−0.226 × T1)	***	0.132		1.863	−1.663	1.406
RAVLT recognition d	134	2.228 + (−0.200 × T1)	**	0.063	≤−0.242 −0.241 to 0.292 ≥0.293	1.167	−1.890	1.603
						1.881
						2.445
Corsi index	129	26.609 + (−0.191 × T1) + (−15.091 × TP)	***	0.139		18.402	−1.453	1.657
Fw partial credit score	129	19.548 + (−0.340 × T1)	***	0.181		11.126	−1.626	1.718
FAS index	122	10.812 + (−0.170 × T1)	***	0.109		6.645	−1.413	1.869
Letter F	122	6.416 + (−0.310 × T1)	***	0.156		3.559	−1.535	1.764
Letter A	122	4.610 + (−0.312 × T1)	***	0.174		3.180	−1.664	1.785
Letter S	122	5.224 + (−0.230 × T1)	***	0.098		3.924	−1.643	1.819
CTMT B time (s)	87	10.516 + (−0.293 × T1)	***	0.228		9.807	−2.001	1.434
PASAT index	124	17.189 + (−0.293 × T1)	***	0.229	≤5.452 5.453 to 11.320 ≥11.321	5.054	−1.771	1.524
						9.464
						15.451
Correct	124	17.579 + (−0.296 × T1)	***	0.278	≤4.378 4.379 to 8.910 ≥8.911	3.709	−1.976	1.651
						8.867
						7.814
Omissions	124	0.298 + (−0.353 × T1)	***	0.395	≤−9.591 −9.590 to −4.647 ≥−4.646	9.205	−2.138	1.228
						7.385
						2.784
Stroop index	130	1.022	–	0.000		1.820	−1.621	1.748
Interference (ms)	129	72.825 + (−0.440 × T1)	***	0.391		151.225	−1.708	1.424
W time (ms)	130	−20.959	–	0.000		140.438	−1.622	1.732
CW time (ms)	130	151.547 + (−0.165 × T1)	***	0.200		154.413	−1.561	1.495

Note: PI (z) = standardized prediction interval, SEest = standard error of the estimate, T1 = baseline score, TP = trackpad.

^a Predicted change score y _change = intercept + B _T1  × T1 + B _TP  × TP. Predicted retest score y _retest = intercept + (1 + B _T1 ) × T1 + B _TP  × TP where TP is 1 if trackpad was used and 0 if mouse or touchscreen was used.

^b The SEest is specified per tertile of predicted change scores for test scores where the Levene test of homogeneity of variance was not fulfilled.

^c Randomized stimuli.

^d Alternate word lists.

^* p ≤ 0.05,

^** p ≤ 0.01,

^*** p ≤ 0.001, ns = p > 0.05.

The change‐score regression analyses resulted in negative significant associations with the baseline scores for all but three of the 21 test scores. Otherwise, there only appeared a single influence of trackpad on the Corsi index, showing that the trackpad users performed on average about 15 points worse than mouse and touchscreen users at follow‐up. In our data, we could not observe any significant influence of demographics, computer type, screen size, or testing location.

The prediction equations in Table 5 can be used to calculate the standardized value (z) of the change scores:

$$z=\left(\mathrm{sign}\right)\left(\text{observed change score}-\text{predicted change score}\right)/\mathit{SE}\mathrm{est}$$

where (sign) is positive when a higher raw test score indicates better performance, and negative for worse performance with higher score. Table 5 also shows the 90% PIs of the standardized change scores. The lower and upper ends of the 90% PIs in the table are arranged so that the lower end always indicates worse performance and the upper end better performance.

DISCUSSION

This study provides evidence of test–retest reliability and practice effects of 22 main tests and 33 subscores of the Mindmore digital cognitive assessment tool over a 1‐month interval in a healthy population. Test–retest reliability coefficients were in line with our expectations and were largely similar to those previously reported for traditional pencil and paper tests. Satisfactory or better test–retest reliability for clinical use was demonstrated for 16 main test scores. Among the six main test scores with minimal test–retest reliability, three had clinically important subscores that demonstrated satisfactory or better test–retest reliability. Practice effects were found to be comparable to those of traditional pencil and paper tests and were observed in about half of all tests. As expected, these practice effects were primarily evident in test results heavily influenced by processing speed. Change scores were established for 12 main scores and nine subscores with sufficient sample sizes allowing for stable estimates and with satisfactory or better retest reliability.

Clinical implications

The implications of these results extend to both clinicians and researchers. Our findings clarify the psychometric properties of various tests, enabling informed decision‐making regarding test selection and interpretation. For instance, seeing improvements between two time points in tests with large practice effects (e.g. TMT A index, or Stroop index), can now be interpreted with the knowledge that healthy test‐takers are expected to improve following repeated testing. Data on practice effects is particularly important when assessing if cognition has worsened due to a disease, or improved following an intervention. Further, detecting genuine cognitive shifts becomes more plausible when armed with the knowledge of normative change scores. For example, consider a patient using trackpad tested at two occasions whose RAVLT recognition scores declined from 12 to 9, and Corsi index scores from 78 to 46. These scores represent observed score differences of −3 for RAVLT and −32 for Corsi. To determine the significance of these changes, we calculate predicted changes using regression formulas from Table 5, resulting in predicted changes of −0.172 for RAVLT (calculated as 2.228 + [−0.200 × 12]), and −3.380 for Corsi (calculated as 26.609 + [−0.191 × 78] + [−15.091 × 1]). Subsequently, we determine the standardized z‐scores: −1.503 for RAVLT ([−3 − −0.172]/1.881), and −1.555 for Corsi ([−32 − −3.380]/18.402). Clinically, the RAVLT score change does not indicate significant memory impairment, as the z‐score of −1.503 is above the clinical significance threshold (z‐score > −1.890). In contrast, the Corsi index's z‐score of −1.555 falls below the clinical significance threshold (z‐score < −1.453), indicating a significant decline in visuospatial working memory. This suggests potential clinical implications for the patient's ability to process and recall visual and spatial information, which could affect daily activities such as orientation and location memory.

While many tests demonstrated commendable test–retest reliability, certain tests fell short. The minimal reliability observed in click‐SRT may stem from the variability introduced by consumer‐grade computers. To mitigate such testing errors, dedicated hardware tailored for reaction time tasks is recommended (Holden, Francisco, Tommerdahl, et al., 2020). Nevertheless, the click‐CRT test, which demands additional inhibitory control – potentially making it more sensitive to cognitive impairment – showed satisfactory reliability and is of greater clinical relevance. The cube copying test's limited reliability does not significantly impact its clinical use, given its primary value lies in qualitative assessment. The restrained reliability of the BNT‐15 test is likely due to its constrained scoring range and an evident ceiling effect. Floor and ceiling effects are also apparent in most of the accuracy scores included and the limited scoring ranges observed for the Corsi span forwards and backwards. The clear floor and ceiling effects observed are likely due to our healthy sample, and test–retest reliability would presumably look differently in a clinical population. Still, for any of these scores, an attained clinical score worse than normally expected should be interpreted with caution.

Methodological considerations

The design of our study included a retest interval of roughly 1 month, a duration that may be seen as brief in clinical contexts where intervals of 6–12 months are more typical. At such extended intervals, practice effects tend to decrease (Calamia et al., 2012), implying that an impairment in the same order between baseline and follow‐up at a longer retest interval will in many cases reflect a smaller total change due to the smaller practice effect. It is also important to note that practice effects decrease with repeated assessments (Benedict, Smerbeck, Parikh, Rodgers, Cadavid & Erlanger, 2012; Sacks, Clark, Pols & Geffen, 1991). Therefore, we urge caution when extrapolating our change scores to contexts involving longer intervals or multiple retests to prevent false‐positive findings.

It fell outside the scope and design of this study to examine the effects of screen dimensions, computer type, input method, or the physical location of test administration on test–retest reliability and practice effects. The sample distribution was too limited to stratify based on these variables. To our knowledge, no prior report exists on the potential effects of these variables on test–retest properties. Additionally, it was not possible to examine every variable for all test results' change scores; the CTMT and BNT‐15 tests were exclusively administered online, and in‐laboratory administration overlapped with touchscreen input for the Corsi, PASAT, and Stroop tests. Consequently, our findings that no discernible effects on change scores of these variables were observed, except for the Corsi index, where follow‐up scores were lower when using trackpad, are preliminary. More extensive studies with larger and more evenly distributed samples are needed to thoroughly investigate the impact of screen dimensions, computer type, input method, and test administration location on test outcomes.

Unlike other studies reporting change scores (e.g., Eliassen, Kirsebom, Fladby, et al., 2023; Karlsen, Karr, Saksvik, et al., 2020), we did not find any significant influence of age, education or sex. In part, this might be attributed to the limited education range in this study's sample. However, in contrast to the other studies we also employed a slightly different method which reduces the risk of demographic regression artifacts (for details, see Appendix S1 Part D.2). Nevertheless, the present change score norms should be used with caution for persons with low educational attainment.

Lastly, the present norms were derived from a population who with few exceptions were ethnically and culturally Swedish adults who live in economically stable urban and rural regions in Sweden. This limits the generalizability of the findings and calls for caution when using the provided change scores for persons with other ethnic and cultural background than Swedish.

CONCLUSION

Our study demonstrates the test–retest reliability and practice effects of the Mindmore cognitive assessment tool, showing they are comparable to those of traditional pencil and paper tests. Additionally, we derived Swedish normative change scores for the most frequently used tests. This psychometric data can assist researchers and clinicians in distinguishing between normal variations in performance and changes indicative of clinical impairment, providing valuable guidance when interpreting retest results and making informed decisions regarding patient care.

We express our sincere gratitude to the participants who generously contributed their time and effort to the present study. We also extend our thanks to Professor Sakari Kallio for facilitating data collection at Skövde University, and to research assistant Gustav Martling for his valuable assistance in the project. Ingvar Bergman, Ludwig Franke Föyen and Wobbie Van den Hurk, are or have been employees of Mindmore AB. Anders Gustavsson indirectly owns shares in Mindmore AB. All study participants gave their informed consent prior to study inclusion. The data that support the findings of this study are available from the corresponding author upon reasonable request.

Supporting information

Table S1. Independent samples t‐test comparisons of demographics and cognitive characteristics at baseline between the final sample and the 60 subjects who did not return for retest.

Table S2. Final change score models with all covariates entered into the regression analyses in one step. Models that differ from those in Table 5 are not intended to be used as norms as they likely reflect regression artifacts.

Figure S1. Measures with good reliability.

Figure S2. Measures with satisfactory reliability.

Figure S3. Measures with minimal reliability.

Figure S4. Scatter plot of Corsi forwards partial credit baseline score and test score change at low and high age. Red‐white dots and dashed red lines signify “low” age (M = 28.7 years, n = 65) and solid blue dots and solid blue lines signify “high” age (M = 66.2 years, n = 64) according to a median split. The horizontal lines demark the change score means for the two age groups.