Validation of the Brief International Cognitive Assessment for Multiple Sclerosis in Japan

Short abstract

Background

The Brief International Cognitive Assessment for MS (BICAMS) is a practical battery for measuring cognitive function in multiple sclerosis (MS).

Objectives

We aimed to validate a Japanese version of the BICAMS in patients with MS and healthy controls.

Methods

The Symbol Digit Modalities Test (SDMT), the California Verbal Learning Test-Second Edition (CVLT2) and the Brief Visuospatial Memory Test Revised (BVMTR) were administered to 156 patients with MS and 126 healthy controls (HCs). The BICAMS was re-administered in a subset of 27 MS patients and 30 HCs.

Results

The mean (±SD) raw scores in the MS and HC groups were as follows: SDMT: MS 47.9 ± 14.0, HC 61.0 ± 9.5; CVLT2: MS 48.6 ± 12.6, HC 55.7 ± 10.5; BVMTR: MS 23.5 ± 8.4, HC 28.3 ± 5.4, respectively, and significant differences were found between the two groups on all tests (p < 0.0001). Cohen’s d values were 1.07, 0.60, and 0.67 in SDMT, CVLT2, and BVMTR, respectively. The test-retest reliability coefficients for each test were as follows: SDMT: r = 0.93; CVLT2: r = 0.82; and BVMTR: r = 0.77 (p < 0.0001).

Conclusions

This study provides results that support the reliability and validity of the BICAMS in Japan.

Introduction

Multiple sclerosis (MS) is an immune-mediated inflammatory disease of the central nervous system (CNS), and a progressive, neurodegenerative course is common, as represented by decline in clinical metrics and progressive brain atrophy.¹ While physical disability as evaluated by the Expanded Disability Status Scale (EDSS)² remains the most commonly applied clinical outcome in MS, attention to cognitive function in MS is increasing. There is modest correlation between EDSS score and cognitive abilities assessed via neuropsychological testing.^3–5 However, cognitive impairment tends to progress over time, sometimes independently of the accumulation of physical disability. Despite cognitive tests having been validated in MS, a neuropsychological evaluation is not routinely implemented alongside EDSS scoring. In fact, 25% of patients with an EDSS score of less than 4.0 have cognitive impairment that is not captured by the EDSS alone.⁶

In general, neuropsychological studies suggest that 40–65% of patients with MS are cognitively impaired. Most often cognitive processing speed (CPS) and memory are involved.⁷ The Minimal Assessment of Cognitive Function in MS (MACFIMS) was the first effort to develop consensus on an optimal neuropsychological test battery for patients with MS.⁸ The MACFIMS requires at least 90 min of face-to-face examination, preferably with board-certified neuropsychologists or well-trained technicians. In 2012, a panel of experts was convened to improve the accessibility of neuropsychological testing in MS centers by narrowing the focus of the battery to only tests of CPS and new learning. The resulting Brief International Cognitive Assessment for MS (BICAMS)⁹ was also designed for potential international use, as the components are easily translated.¹⁰ The BICAMS includes the Symbol Digit Modalities Test (SDMT),¹¹ the second edition of the California Verbal Learning Test (CVLT2)¹² and the revised Brief Visuospatial Memory Test (BVMTR).¹³ The BICAMS has been validated in several English speaking and non-English speaking populations, and usefulness of the battery has been confirmed in several countries.^14–21

The objective of this study was to validate and assess the reliability of the Japanese version of the BICAMS in the Japanese population.

Methods

Patients with MS and healthy individuals

This study was conducted between March 2016 and May 2017 with 156 Japanese patients with MS (female/male = 107/49) diagnosed using the 2010 revised McDonald criteria²² and 126 age-, sex-, and education duration-matched Japanese healthy controls (female/male = 91/35) at nine sites (Sapporo Neurology Hospital, Kyushu University, Tokyo Women’s Medical University Yachiyo Medical Center, Saitama Medical Center, Osaka University Graduate School of Medicine, Chiba University, Tokyo Medical and Dental University, Tohoku Medical and Pharmaceutical University, and Hokkaido Medical Center), which cover major regions in Japan (Table 1). All patients underwent neurological examination, and those deemed to have visual or motor disabilities too severe to perform BICAMS were excluded. Patients with neuromyelitis optica spectrum disorders²³ and patients with MS who had relapses and received steroids within one month before the examination were excluded from this study. Patients with MS who had history of major psychiatric disorders including depression preceding MS and received anti-psychotics were also excluded from this study. Patients were categorized according to MS subtype of disease course:²⁴ one had primary progressive, 137 had relapsing–remitting and 18 had secondary progressive disease. Demographic and clinical data are shown in Table 1. Differences in sex ratio, duration of education, and age at examination between the patients and controls were not significant (p > 0.05). People with diseases of the CNS or major medical illnesses were excluded from the healthy control group. All participants had adequate vision, speech, and functions of upper extremities to complete testing based on clinician judgment. The study protocol was approved by the ethics committee of each participating site, and all patients and healthy controls gave their written informed consent to participate in the study.

Table 1.

Demographic and clinical profiles for MS patients and controls

	MS patients	Controls
N (female/male)	156 (107/49)	126 (91/35)
Age at examination (years)* (range)	41.4 ± 9.3 (23–70)	39.3 ± 11.9 (21–75)
Education (years)* (range)	14.1 ± 1.9 (12–22)	14.3 ± 1.6 (12–18)
Age at onset (years)* (range)	31.0 ± 9.2 (10–64)
Disease duration (years)* (range)	10.3 ± 7.2 (0.7–39)
EDSS* (range)	2.4 ± 2.0 (0–8)

BICAMS

The BICAMS battery includes three individual tests: the SDMT (faux example in Figure 1) for CPS, the first five recall trials of the CVLT2 (faux example in Figure 2) for auditory/verbal learning and memory, and the first three recall trials of the BVMTR (faux example in Figure 3) for visual/spatial memory.⁹ The BICAMS, which was originally written in English, was translated into Japanese and used for assessment of neuropsychological functions. Especially for the CVLT2, words were culturally adapted for the Japanese population, and the Japanese version of the CVLT2 has been recently validated.²⁵ At baseline, original versions of the SDMT and the CVLT2 and Form 1 of the BVMTR were used.

Figure 1.

Faux stimuli presented to subjects for the Symbol Digit Modalities Test (SDMT). Adapted from Benedict et al.¹⁰

Subjects are asked to see a series of nine symbols, each paired with a single digit in a key at the top of a standard sheet of paper, and then, to answer the digit associated with each symbol as rapidly as possible for 90 s. Subjects are explained these procedures in Japanese like「上の段に符号, 下の段に数字があり, それぞれの符号とその下の数字は対応しています．それぞれの数字にはそれぞれ異なった符号が割り当てられているので, 90秒間に上段の符号に対応する数字を, 一番上の符号と数字の対応表を見て, 出来るだけ多く回答してください．」.

Figure 2.

Faux stimuli presented to subjects for the California Verbal Learning Test-Second Edition (CVLT2).

Subjects are asked to listen to 16 words, and report as many of the items as possible. There is no instruction as to the order in which items are recalled. After recall is recorded, the entire list is read again followed by a second attempt at recall. Altogether, there are five learning trials. Subjects are explained these procedures in Japanese like「これから単語のリストを読み上げますのでよく聞いてください．読み終わったら, どんな順序でも構いませんので, リストにあった単語を出来るだけたくさん言ってください．これを5回繰り返します．」.

Figure 3.

Faux stimuli presented to subjects for the Brief Visuospatial Memory Test Revised (BVMTR). Adapted from Benedict et al.¹⁰

Subjects are asked to view six figures for 10 s, and the display is removed from view. Then, subjects render the stimuli via pencil on paper manual responses. There are three learning trials. Subjects are explained these procedures in Japanese like「6つの図計が描かれたシートをお見せします．10秒間, シート全体をお見せするので, できる限り多く覚えて下さい．シートを伏せた後, 回答用紙にそれぞれの図形を正しい形で正しい位置に描いて下さい．これを3回繰り返します．」.

The test battery was administered in the following fixed order: SDMT, first five trials of CVLT2, and first three recall trials of BVMTR.¹⁰

Retest was performed at an interval ranging from one to three weeks as the paper for international validation recommends¹⁰ in 27 patients with MS and 30 controls among 156 patients with MS and 126 controls. For retest, alternate forms were employed (SDMT Alternate Form 1;²⁶ CVLT2 Alternate Form; BVMTR Form 2) to reduce practice effects from prior exposure to the stimuli.

Statistical analysis

All data are expressed as mean ± standard deviation (SD). Statistical analysis was performed using the SAS 9.4 software package (SAS Institute Inc., Cary, NC, USA) and the GraphPad Prism (GraphPad Software, Inc., La Jolla, CA, USA). Comparisons between groups were performed using paired-sample t-tests and the Mann–Whitney U test as appropriate. Measurements of the linear correlation between data sets were assessed using Pearson's r coefficient. The international expert consensus committee recommended that a more valid measure of test–retest reliability is the Pearson correlation coefficient, and these r values for test–retest correlation were considered adequate if >0.70 and good if >0.80.¹⁰ p values less than 0.05 were considered statistically significant.

Results

BICAMS data in patients with MS and healthy controls

Scores were uniformly lower in patients with MS than healthy controls (Table 2). Cohen’s d effect sizes were calculated. The SDMT had the highest Cohen’s d value (1.07), followed by BVMTR (0.67) (Table 2). Cronbach’s alpha coefficients for all three BICAMS test scores were 0.82 in patients with MS, 0.52 in the healthy control group and overall 0.79, suggesting an adequate level of within-construct reliability.

Table 2.

BICAMS raw score in MS patients and controls

	MS patients	Controls	p-value
SDMT
mean ± SD (range)	47.9 ± 14.0 (2–84)	61.0 ± 9.5 (29–89)	<0.0001
Cohen’s d	1.07
Effect size r	0.47
CVLT2
mean ± SD (range)	48.6 ± 12.6 (11–73)	55.7 ± 10.5 (20–78)	<0.0001
Cohen’s d	0.60
Effect size r	0.29
BVMTR
mean ± SD (range)	23.5 ± 8.4 (0–36)	28.3 ± 5.4 (12–36)	<0.0001
Cohen’s d	0.67
Effect size r	0.32

BICAMS: Brief International Cognitive Assessment for Multiple Sclerosis; MS: Multiple Sclerosis; SDMT: Symbol Digit Modalities Test; CVLT2: California Verbal Learning Test Second Edition; BVMTR: Brief Visuospatial Memory Test Revised. SD: standard deviation.

As expected, a significant negative correlation between age at examination and each BICAMS component was found in both patients with MS and controls (Table 3). Negative correlations between education level and the SDMT or the CVLT2 was found, although there were no correlations between education and the BVMTR in the control group (Table 3). On the other hand, in the MS patients group, no correlation was found between duration of education and each of the BICAMS tests (Table 3).

Table 3.

Correlation between age at examination or duration of education and the BICAMS

	Age at examination				Education duration
	MS patients		Controls		MS patients		Controls
	r	p-value	r	p-value	r	p-value	r	p-value
SDMT	–0.37	<0.0001	–0.44	<0.0001	0.07	n.s.	0.24	<0.01
CVLT2	–0.25	<0.01	–0.23	<0.01	0.13	n.s.	0.25	<0.05
BVMTR	–0.30	<0.001	–0.25	<0.01	0.001	n.s.	0.05	n.s.

BICAMS: Brief International Cognitive Assessment for Multiple Sclerosis; MS: Multiple Sclerosis; SDMT: Symbol Digit Modalities Test; CVLT2: California Verbal Learning Test Second Edition; BVMTR: Brief Visuospatial Memory Test Revised. SD: standard deviation; n.s.: not significant (p > 0.05)

Correlation between disease duration or EDSS and the BICAMS

Negative correlations were found between disease duration and score in the SDMT and the BVMTR, although there was no association between disease duration and the CVLT2 (Table 4). As for disease severity and cognitive impairment, relatively strong negative correlations were found between the EDSS and scores of all three tests of the BICAMS in MS patients (Table 4).

Table 4.

Correlation between disease duration or EDSS and the BICAMS

	Disease duration		EDSS
	r	p-value	r	p-value
SDMT	–0.30	<0.001	–0.56	<0.0001
CVLT2	–0.12	n.s.	–0.29	<0.001
BVMTR	–0.27	<0.001	–0.46	<0.0001

EDSS: Expanded Disability Status Scale; BICAMS: Brief International Cognitive Assessment for Multiple Sclerosis; SDMT: Symbol Digit Modalities Test; CVLT2: California Verbal Learning Test Second Edition; BVMTR: Brief Visuospatial Memory Test Revised. SD: standard deviation; n.s.: not significant (p > 0.05)

Tests–retest reliability

Table 5 shows mean raw data of tests and retests in patients with MS and controls. As for correlation coefficients between the tests and the retests, the BVMTR in the control group was not so highly correlated (r = 0.58) although the p-value was less than 0.001 (Table 5). The BVMTR overall had adequate correlation, and other data had good correlation over 0.80 (Table 5).

Table 5.

Test–retest reliability of the BICAMS tests in MS patients and controls

	Overall			MS patients			Controls
	test	retest	r/p-value	Test	retest	r/p-value	test	retest	r/p-value
SDMT	53.1 ± 13.8(16–80)	55.0 ± 15.3(12–79)	0.93/<0.0001	45.1 ± 13.2(16–66)	47.3 ± 15.9(12–76)	0.94/<0.0001	60.2 ± 10.0(29–80)	61.9 ± 11.1(35–79)	0.87/<0.0001
CVLT2	50.3 ± 12.9(11–73)	54.7 ± 11.3(23–77)	0.82/<0.0001	48.6 ± 13.1(11–73)	52.4 ± 11.6(23–69)	0.80/<0.0001	51.9 ± 12.7(20–73)	56.8 ±10.8(35–77)	0.83/<0.0001
BVMTR	27.5 ± 6.3(2–36)	29.8 ± 4.5(8–36)	0.77/<0.0001	25.5 ± 7.6(2–34)	28.6 ± 5.4(8–36)	0.82/<0.0001	29.2 ± 4.2(17–36)	30.8 ± 3.3(22–36)	0.58/<0.001

BICAMS: Brief International Cognitive Assessment for Multiple Sclerosis; MS: Multiple Sclerosis; SDMT: Symbol Digit Modalities Test; CVLT2: California Verbal Learning Test Second Edition; BVMTR: Brief Visuospatial Memory Test Revised. SD: standard deviation. For each test, raw data are expressed as mean ± standard deviation scores (ranges).

Discussion

Some degree of cognitive impairment is found in roughly half of the MS population, and diminished CPS and reduced learning on memory tasks are particularly common.²⁷ Previously, we reported on cognitive dysfunction in MS evaluated with the Brief Repeatable Battery of Neuropsychological tests (BRB-N), a historically well-established battery.²⁸ This battery takes around 45 min to administer,²⁹ and requires considerable expertise, thus it is not well suited to routine clinic activity. On the other hand, it takes 15 min to administer the BICAMS battery, which is reliable and valid in many languages.^9,10 BICAMS has been shown to correlate well with the BRB-N (Kappa 0.46) and identify very similar numbers of MS patients as cognitively impaired.³⁰ Cultural background influences neuropsychological performance,³¹ and this battery has been validated in several countries including the Czech Republic,¹⁴ Italy,¹⁵ Hungary,¹⁶ Lithuania,¹⁷ Brazil,¹⁸ Ireland,¹⁹ Argentina²⁰ and Greece.²¹ The present study meets an important clinical need in Japan, and we find that despite the marked differences between English and Japanese, we again show significant differences between patients and controls (p < 0.0001 in all tests).

There are some limitations in this study, including application to patients with visual impairments and the cross-sectional nature of the study. Subjects with major psychiatric disorders were excluded from this study. However, in this study, detailed psychiatric investigations with some batteries such as Beck Depression Inventory Second Edition (BDI-II) were not done, and minor psychiatric conditions might have affected some cognitive functions. Future studies will reveal the association between depressive state and data of BICAMS in the Japanese population. In fact, in this study, patients who had visual or hand disabilities too severe to perform BICAMS were excluded. Our current study provides data for the Japanese BICAMS in 126 healthy controls, and whilst this does not constitute a full normative sample for the population as a whole, it is matched to the demographics of the main MS population and is sufficient to address the key questions of test validity central to this paper. As presented in the consensus panel BICAMS papers,^9,10 the purpose of the increasing number of small-scale country/language validations^14–21 is to provide a normative sample exclusively for MS clinic patients. The lack of 17–20-year-old healthy subjects in this study could hamper efforts to developed regression-based norms that depend on adequate sampling of all ages. We again find that the BICAMS measures are influenced by variables such as age, sex, and level of education.^28,32,33 Importantly, the rank order of these correlations was as expected, with CPS being most strongly related to age and verbal memory most strongly related to education. We (Benedict, Langdon, Niino) are planning to extend the normalization process and integrate these data with norms from other countries to further our endeavor to provide international regression-based norms for clinical use.

As for disease duration and scores on the BICAMS tests, negative associations were found in the SDMT and the BVMTR, but not in the CVLT2. In our previous study with the BRB-N, a negative association was demonstrated between disease duration and verbal memory evaluated by a selective reminding test.²⁸ The battery for verbal memory in this study was different from the battery in our previous study, and that may be one of the reasons for the different result. As for other reasons, the included patients were different between the two studies. In fact, there is a study that demonstrated negative associations between disease duration and the BICAMS scores of all three tests.¹⁷ Previous studies, including our study, demonstrated that cognitive impairment evaluated by batteries such as the BRB-N had correlations with physical disability as evaluated by EDSS.^4,28 Our current data also demonstrated similar findings.

The international expert consensus committee reported that test–retest reliability has high priority and is very relevant for the validation of BICAMS.¹⁰ Regarding this issue, we assessed about 60 participants including patients and controls to undergo retest within 1–3 weeks post baseline. In the retest, an alternative version of each test was used. As a result, strong correlations have been established with assessing the test–retest reliability in all tests other than the BVMTR in controls. Our study confirmed good test–retest reliability. The controls’ performance shows slightly weaker test–retest correlation in the BVMTR in controls in our study. On the other hand, Sandi et al. reported a slightly weaker correlation in the CVLT2.¹⁶ The difference might be due to the included participants or cultural background.

Consistent with prior research, our data reveal the largest effects and best reliability for the SDMT in support of our previous results.²⁸ The SDMT could provide a better index of CPS, which seems to be more frequently impaired in patients with MS, and the SDMT is the best psychometric measure available for assessing CPS in patients with MS.^34,35 Further, the SDMT is a good test to predict cognitive impairment in patients with MS, even in the early stages of the disease.³⁶

In summary, the BICAMS is a useful battery to screen cognitive impairment in patients with MS, and international validation for BICAMS has been done. The study provides validation of Japanese version of the BICAMS, the first project in Asia, and confirms that this battery can be useful for clinical practice in Japanese patients with MS. Nationality significantly influences BICAMS performance, hence the importance of the inclusion of a nationality variable when international norms for the BICAMS are constructed.³⁷ It is important to investigate the potential impact of nationality on international validation in the future.

Conflicts of interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by the Health and Labour Sciences Research Grant on Intractable Diseases (Neuroimmunological Diseases) from the Ministry of Health, Labour and Welfare of Japan.