Regression-Based Norms for the Symbol Digit Modalities Test in the Dutch Population: Improving Detection of Cognitive Impairment in Multiple Sclerosis?

Abstract

Background/Aims

Appropriate and timely screening instruments that sensitively capture the cognitive functioning of multiple sclerosis (MS) patients are the need of the hour. We evaluated newly derived regression-based norms for the Symbol Digit Modalities Test (SDMT) in a Dutch-speaking sample, as an indicator of the cognitive state of MS patients.

Methods

Regression-based norms for the SDMT were created from a healthy control sample (n = 96) and used to convert MS patients' (n = 157) raw scores to demographically adjusted Z-scores, correcting for the effects of age, age², gender, and education. Conventional and regression-based norms were compared on their impairment-classification rates and related to other neuropsychological measures.

Results

The regression analyses revealed that age was the only significantly influencing demographic in our healthy sample. Regression-based norms for the SDMT more readily detected impairment in MS patients than conventional normalization methods (32 patients instead of 15). Patients changing from an SDMT-preserved to -impaired status (n = 17) were also impaired on other cognitive domains (p < 0.05), except for visuospatial memory (p = 0.34).

Conclusions

Regression-based norms for the SDMT more readily detect abnormal performance in MS patients than conventional norms, identifying those patients at highest risk for cognitive impairment, which was supported by a worse performance on other neuropsychological measures.

Introduction

Cognitive impairment is common in multiple sclerosis (MS) with prevalence rates ranging from 45 up to 70% [1]. Early detection of patients at risk is vital, as it has important implications in clinical planning and management. Currently, it is difficult to assess the MS-associated cognitive disorders in routine neurological visits. Brief cognitive-screening measures have been proposed to substitute for the more comprehensive neuropsychological test batteries, which are time consuming and do not allow application in daily clinical practice [2]. However, the lack of appropriate normative data, from different language groups, currently limits their applicability [3].

The Symbol Digit Modalities Test (SDMT) is a valuable neuropsychological test, especially in the area of MS research, and has been recommended as part of the Brief International Cognitive Assessment for MS, a brief assessment for cognitive monitoring purposes in MS clinics that is internationally being validated at present [4]. It is a widely accepted measure of attention and information processing speed that embodies high sensitivity to detect cognitive deficits in MS patients, and has been extensively used as a research tool in studies assessing the relationship between cognitive performance and disease burden on MRI [4, 5, 6, 7, 8, 9]. Despite its excellent psychometric properties and timely application, one of the weaknesses in implementing the SDMT in clinical settings is the reliance on published normative data, varying in size and quality and with poor generalizability [10].

Recently, an alternative approach to normalization has been proposed using linear regression modelling, which allows to control for demographic effects on test performance [10, 11]. The regression-based normalization methodology takes advantage of general developmental and demographic trends to generate continuous norms while requiring relatively few individuals. Previous studies on this topic, comparing regression-based norms with the traditional manual-based norms, have reported higher rates of impairment on the SDMT when correcting for demographics [10, 11]. Although some observations were in favour of the regression-based approach, the exact value and clinical interpretation of this finding remain unclear.

Therefore, the aim of this study was to evaluate newly derived, regression-based norms for the oral SDMT in a Dutch-speaking sample, as an indicator of the cognitive state of MS patients. With this in mind, we compared impairment-identification rates between SDMT regression-based and conventional normalization methods, using the means and SDs of the same control group as a reference. Then we evaluated the overall cognitive performance of the resulting SDMT-preserved and -impaired patient groups, as measured by an extensive set of neuropsychological tests.

Materials and Methods

Subjects

Study protocols were approved by the institutional ethics review board of the VU University Medical Centre and all subjects gave written informed consent prior to study participation. Normalization was achieved using data from 96 healthy controls (HCs; 57 females; mean age 46.0 ± 10.6 years; range 23-63), free of neurological or psychiatric disorders, who were recruited from the community and volunteered to participate. Educational level was graded using a Dutch classification system ranging from 1 (less than primary or elementary education) to 7 (academic education). To evaluate whether our HC sample can be considered representative for a normal Dutch population, we compared previously published normative data from the Netherlands with our sample [12]. In addition we assessed a total of 157 MS patients (104 females; mean age 41.08 ± 8.24 years; range 23-65), who were all diagnosed with clinically definite MS based on the criteria of McDonald [13], and were part of a cohort that has been followed for 5–7 years since the time of diagnosis (mean disease duration 7.5 ± 2.2 years) [14]. Subjects were largely overlapping with previous publications on the same cohort [14, 15, 16]. MS subtypes included relapsing-remitting MS (133 patients), primary progressive MS (15 patients) and secondary progressive MS (9 patients) [17]. Physical disability was measured with the Expanded Disability Status Scale (EDSS) and was relatively mild (median EDSS 2.5; range 0-8.0) [18]. Patients were relapse-free and without steroid treatment for at least 2 months before examination and followed standard treatment options since diagnosis as described previously [16].

Neuropsychological Evaluation

All subjects underwent an extensive set of neuropsychological tests combined in the Brief Repeatable Battery of Neuropsychological tests (BRB-N) [19], in addition to the concept shifting test (CST) [20], the Stroop color-word test [21], and the memory comparison test (MCT) [22], performed by an experienced neuropsychologist. This resulted in an evaluation of the following 6 cognitive domains:

• Information-processing speed – measured by the oral SDMT [23]. The Rao [19] adapted SDMT from BRB-N, version A, was used in all subjects within this study.

• Verbal memory – assessed with the Selective Reminding Test.

• Visuospatial memory – assessed with the 10/36 Spatial Recall Test.

• Working memory – measured by the MCT.

• Attention – assessed with the Stroop color-word test.

• Executive functioning – measured by the Word List Generation test and the CST.

Conventional Normalization Method

For the conventional analysis of neuropsychological data, individual raw test scores of patients were converted to Z-scores, using the means and SDs of respective test scores of the HCs as a reference. Cognitive domain summary measures, based on these Z-scores, were calculated for all patients. The exact construction and use of these domain measures have been described elsewhere [14, 15]. Patients who scored at least 2 SDs (i.e., Z ≤-2) below the average of controls were defined as impaired on that domain.

Regression-Based Norms for the SDMT

Regression-based norms for the SDMT were created following the same procedures as published by Parmenter et al. [10], albeit using the adjustment as described by Berrigan et al. [11], with centring of the age variable (age minus mean age) to avoid multicollinearity and for ease of interpretation. First, the HCs' raw scores were converted to a scaled score metric (mean 10, SD 3) based on the cumulative frequency distribution presented in Table 1. Next, the resulting scaled scores were regressed on 4 demographic variables: age centered (agec = age − 46), agec², gender (female vs. male) and education, entered en block. The factor agec² was added to account for possible non-linear effects of age on test performance. Participants' educational level was converted to years of education. None of the assumptions of regression analyses were violated (no influential cases, normality of the residuals, or homoscedasticity).

Table 1.

Raw score to scaled score conversions using the cumulative frequency distribution

Scaled scores	Raw scores
3	≤43
4	44–46
5	47
6	48–49
7	50–52
8	53–55
9	56–58
10	59–62
11	63–65
12	66–68
13	69–72
14	73–75
15	76–77
16	78–81
17	≥82

To obtain demographically adjusted SDMT Z-scores for the MS patients, we first converted each participants' raw score to a scaled score based on the raw-to-scaled score conversions derived from the HCs (Table 1). Next, predicted scores were calculated using the multiple regression equation, based on β weight values for all 4 demographic variables and their predictive constant:

$$\text{Scaled}\hspace{0.17em}{\text{score}}_{\text{predicted}}=\text{constant}+{\beta }_{\text{agec}}(\text{agec})+{\beta }_{{\text{agec}}^2}({\text{agec}}^2)+{\beta }_{\text{gender}}(\text{gender})+{\beta }_{\text{education}}(\text{education})$$

Then, the predicted scaled scores were subtracted from each participants' actual obtained scaled score and the difference was divided by the root mean squared error (RMSE) of the HCs:

$$Z-\text{score}=(\text{scaled}\hspace{0.17em}{\text{score}}_{\text{predicted}}-\text{scaled}\hspace{0.17em}{\text{score}}_{\text{actual}})/\text{RMSE}$$

Only the final Z-scores, which compare the actual scaled scores to the demographically predicted scaled scores, were interpreted. SDMT performance was classified as impaired (Z ≤-2 SD) or preserved (Z >-2 SD) based on the Z-scores derived from each normalization method (conventional and regression-based). To assess the consistency between methods, Z-scores of the HCs based on conventional norms were correlated with the regression-based equivalents running a Pearson's correlation. The McNemar test was used to determine whether the proportion of patients that was classified as SDMT-impaired varied, when using different normalization methods.

The SDMT-preserved and -impaired patient groups were compared in terms of demographic, clinical and cognitive characteristics. It was hypothesized that the regression-based approach would result in an increase in the detection of impairment in MS patients, corresponding to a worse cognitive performance on other neuropsychological measures.

Statistical Analyses

The statistical analyses were performed in SPSS for Windows version 20.0 (Chicago, IL, USA). All variables were checked for normality using Kolmogorov-Smirnov tests and histogram inspection. A one-sample t tests and chi-square goodness-of-fit test were used to determine whether means and distribution of cases in our HC sample were different to previously reported Dutch normative data [12]. Group differences were evaluated using analysis of variance (ANOVA) for normally distributed variables. Non-parametric analysis was carried out when variables were not normally distributed using Mann-Whitney U tests. Pearson chi-square tests were performed when analysing group differences in terms of categorical variables. A p value of <0.05 was considered statistically significant.

Results

Subject Descriptives

In Table 2, the demographic data of the subjects are summarized per group: HCs and MS patients. MS patients were younger than controls (p < 0.001) but demographically matched on gender (p = 0.27) and education (p = 0.14).

Table 2.

Demographic and clinical measures of healthy controls and MS patients

	Healthy controls (n = 96)	Patients (n = 157)	p value
Age, yearsa	46.0 (10.6)	41.1 (8.2)	<0.001
Gender, female/male	57/39	104/53	0.27
Education, yearsb	15.0 (6.0–18.0)	15.0 (6.0–18.0)	0.14

^aData are mean (SD) for normally distributed variables

^bbecause of non-normal distribution, median and range are provided.

Comparison of Previously Published and Current Dutch Normative Samples

Our HCs were group matched to a Dutch normative sample previously used by Boringa et al. [12] (n = 140) in terms of age (46.0 vs. 45.8; p = 0.89) and female to male ratio (57/39 vs. 78/62; p = 0.29). The effect of education (p < 0.001) was seen in 57 subjects (59%) who obtained more than 10 years of education in our sample compared to 53 HCs (38%) observed in the previous publication. In addition, we found a significant difference (p < 0.001) in SDMT performance, with higher test scores in our HCs (61.0 ± 9.7; range 43-99) than in Boringa's controls (56.1 ± 12.4; range 27-106).

SDMT Impairment Rates Using Conventional and Regression-Based Norms

Table 3 shows the regression model for the SDMT used to derive demographically adjusted Z-scores for the MS patients, including agec (age − 46), agec², gender and education as predictor variables. The regression analyses revealed that agec was the only significant predictor (p = 0.011) of normal SDMT performance in our HCs. The conventionally obtained Z-scores of the HCs strongly correlated with the regression-derived equivalents (r = 0.90, p < 0.001).

Table 3.

Final regression model for the SDMT

Measure	Predictor	B	SE B	Standardized B	t	p value	Total R2	RMSE
SDMT	Constant	7.653	1.591
	Agec	–0.0806	0.031	–0.289	–2.60	0.011
	Agec2	–0.000449	0.003	–0.018	–0.18	0.86
	Gender, female	0.470	0.605	0.079	0.78	0.44
	Education, years	0.1515	0.099	0.169	1.53	0.13
							0.150	2.777

SDMT, Symbol Digit Modalities Test; Agec, age centered; B, unstandardized regression coefficient; SE B, standard error of B; t, t test statistic; standardized B, standardized regression coefficient; RMSE, root mean squared error.

When impairment classification rates were examined using either the SDMT conventional or regression-based norms, statistically more MS patients were considered impaired using the regression equation (exact p < 0.001). Based on conventional norms, 15 (10%) MS patients were identified as impaired and 142 (90%) patients as preserved. After correcting for demographics, the number of impaired patients had increased to 32 (20%) with a concomitant reduction in preserved patients to 125 (80%). This change was a consequence of 17 patients becoming impaired when using the regression-based norms.

Clinical and Cognitive Characteristics of SDMT-Preserved and -Impaired Patient Groups

Table 4 shows the differences between the SDMT-preserved and -impaired patient groups, as defined by the combination of conventional and regression-based norms. First, we compared the preserved patients (n = 125) with the group of patients that was defined as impaired based on regression-based norms (n = 32). We found no significant differences in terms of age (p = 0.18), gender (p = 0.62) and education (p = 0.05). The impaired patients scored significantly worse than the preserved patients on all clinical and cognitive measures, with more progressive MS phenotypes (34 vs. 10%; p = 0.01), higher EDSS scores (p = 0.001) and worse Z-scores on all other cognitive domains (p < 0.001; except for visuospatial memory [p = 0.013]).

Table 4.

Characteristics of SDMT-preserved and -impaired patients based on the combination of conventional and regression-based norms

	SDMT preserved (regression-based; n = 125)	SDMT impaired (regression-based; n = 32)	SDMT impaired (regression-based only; n = 17)
SDMT rawb	58.9 (9.7)	38.5 (7.1)***	41.1 (7.6)***
Age, years	40.6 (8.1)	42.9 (8.5)	43.8 (1.9)
Gender, female/male	84/41	20/12	13/4
Education, yearsb	15 (12 to 15)	12 (12 to 15)	12 (12 to 15)
MS characteristics
Disease type (RR/progressive)	112/13	21/11**	14/3
EDSSb	2.0 (2.0 to 3.5)	3.0 (2.5 to 5.0)***	2.5 (2.0 to 3.5)
Cognitive Z-scoresb
Executive functioning	−0.2 (−0.6 to 0.0)	−1.0 (−1.6 to −0.6)***	−0.7 (−1.1 to −0.3)**
Visuospatial memory	0.3 (−1.0 to 0.8)	−0.7 (−1.6 to 0.3)*	−0.2 (−1.3 to 0.6)
Working memory	−0.3 (−0.9 to 0.2)	−1.3 (−2.6 to −0.6)***	−0.6 (−1.6 to −0.3)*
Verbal memory	−0.1 (−0.7 to 0.6)	−1.2 (−1.8 to −0.5)***	−0.7 (−1.5 to −0.1)*
Attention	−0.1 (−0.6 to 0.3)	−0.7 (−1.6 to −0.2)***	−0.7 (−1.0 to 0.0)*

SDMT, Symbol Digit Modalities Test; RR, relapsing-remitting multiple sclerosis; progressive, primary and secondary MS types; EDSS, Expanded Disability Status Scale.

^aData are mean (SD) for normally distributed variables

^bbecause of non-normal distribution, median and interquartile range are provided

^*p < 0.05

^**p < 0.01

^***p < 0.001.

Then, we compared the SDMT-preserved patients (n = 125) with the group of patients being impaired only when using the regression-based norms (n = 17). Groups were found equal in terms of demographics (age: p = 0.81; gender: p = 0.44; education: p = 0.06) and disease characteristics (disease type: p = 0.38; EDSS: p = 0.14). As seen in Table 4, the SDMT regression-based only impaired patients scored significantly worse on other cognitive domain measures (executive functioning: p < 0.01; all other domains: p < 0.05), except for visuospatial memory (p = 0.34).

Discussion

The SDMT is increasingly used as a screening and follow-up tool to detect cognitive deficits in MS patients; however, appropriate normalization in different language groups is needed [3, 4]. In previous studies, abnormal test performance was defined by comparing an examinee's result to the average performance of the most suitable normative (sub)group, taking no account of any potential demographic influence [9, 12, 15, 24]. Parmenter et al. [10] were the first authors to provide regression-based norms for the SDMT with the potential to resolve this limitation and found higher rates of impairment than previously published discrete norms, which was confirmed later by Berrigan et al. [11]. In this study, we compared newly derived regression-based norms for the SDMT in a Dutch-speaking sample with conventional normalization methods, using the means and SDs of the same control group as a reference. Demonstrating the concurrent validity of our approach, we found strong correlations between HCs' Z-scores obtained by either of the normalization methods. As predicted, applying the regression-based norms to the MS patients resulted in a larger group of impairment, yielding lower Z-scores on the SDMT than the conventional norms, which was reflected by worse cognitive functioning on other neuropsychological measures. These observations lend further support to the utility of regression-based norms for the SDMT as a screening method for cognition in MS.

Demographic variables, such as age, gender and education, have been shown to affect normal SDMT performance in previous studies [12]. In the HC sample presented here, age turned out to be the only significantly influencing characteristic. Arguably, it could be decided to include only age as the predictor variable in the final regression model. Reporting the results of a separate McNemar's test, we found no significant difference (exact p = 0.63) in impairment classification rates in the MS patients, using either all 4 demographic variables (20%) in the regression equation or age (centered) alone (22%). Following this approach, demographically adjusted Z-scores can easily be obtained by multiplying agec by its relevant β weight value and subtracting this from its predictive constant (10.027 − 0.0992*agec). To support the appropriate and timely application in clinic, we recommend maintaining a spreadsheet on a computer that can aid in generating a patients' demographically adjusted Z-score by simply entering their demographics and SDMT raw score into the spreadsheet. Such a sheet that contains current normative data can be provided upon request. The regression-based norms presented in this study were established in Dutch HCs; hence, they can only be applied in Dutch patients.

Various other normative data for the SDMT have been published, exploring the effects of relevant demographic predictors of test performance [10, 11, 12, 23, 25]. Boringa et al. [12] provided normative values for the SDMT in a Dutch healthy population, which have been used ever since for many neuropsychological and natural history studies [9, 12, 15, 16, 24]. Although they found an influence of age, gender and education, they did not account for any of the reported effects, which potentially lead to an underestimation of cognitive dysfunction in earlier publications. The lack of correlation between education and SDMT scores in our sample may be a combined issue of range restriction, given the relatively high educational level of the HCs (59% of the HCs obtained more than 10 years of education), and a low N. The effect of gender on SDMT performance remains unclear. While some researchers found gender differences, reporting that women outperformed men, gender had limited impact on test performance in the current normative sample [12, 23, 25]. In addition, we found no quadratic influence of age, which is notable, as one would expect the relationship between age and test performance to be the strongest for subjects in the extreme ends of age (under 25 and over 60), given developmental influences on the information-processing speed [26]. Unfortunately, the small number of controls in these certain age ranges originally included in this study, has led to the generation of a regression equation that inadequately captures the association of age with test performance.

It is important to note that our control group was not age matched to the larger clinical sample, which should be considered when one is planning to use the regression-based norms. However, when comparing the current sample to previous Dutch normative data [12], we found no significant difference in terms of age. Most importantly, we included a sizable sample of HCs (n = 96) and MS patients (n = 157) representing a mix of demographic trends to generate continuous norms that by itself require relatively few individuals of a given demographic trait. Nonetheless, the general small sample size of about 100 controls will tend to decrease the stability of our findings. These concerns notwithstanding, we wish to note that the use of SDMT regression-based norms in the MS sample yielded a frequency of impairment on par with previous neuropsychological studies in early MS patients [27].

To our knowledge, this was the first study to examine the relationship between the overall cognitive functioning aspects in MS patients as measured with a comprehensive set of neuropsychological tests and the SDMT, which was measured with regression-based and conventional normalization methods. The most important limitations lie in the fact that our sample size was relatively small and educational level was high. Nevertheless, our results indicate that demographically adjusted norms for the SDMT more readily detect abnormal performance in MS patients, identifying those patients at highest risk for cognitive impairment, which was supported by a worse performance on other neuropsychological tests. For easier application in clinical settings, we propose the implementation of a spreadsheet in a computer-aided design that automatically generates a patient's demographically adjusted Z-score. Future studies, including more representative normative datasets, should be carried out when considering the actual implementation of regression-based norms for the SDMT in routine daily practice.

To conclude, this study confirms the utility of the SDMT regression-based approach as a potential screening method for cognition in MS, in both research and clinical practice.

Disclosure Statement

Dr. Jessica Burggraaff and Dr. Dirk L. Knol have no conflicts of interest to declare. Prof. Bernard M.J. Uitdehaag has received personal compensation for consulting from Biogen Idec, Genzyme, Merck Serono, Novartis, Roche en TEVA. The VUmc MS Center Amsterdam has received financial support for research activities from Bayer Schering Pharma, Biogen-Idec, GlaxoSmithKline, Merck Serono, Novartis, and Teva.