Abstract
Abstract
Multiple sclerosis (MS) is a neurological disease of the central nervous system which can lead to a range of severe physical disabilities. A large proportion of those affected will experience cognitive impairment, which is associated with a worse prognosis. Effective assessment of cognition in MS has been problematic due to a lack of suitable scales. The Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS) was developed in 2010 as part of an international endeavour to facilitate cognitive assessment.
Aim
The aim of this systematic review and meta-analysis was to synthesise the available literature published as part of the BICAMS international validation protocol.
Methods
A literature search conducted using PubMed, PsycINFO and Google Scholar identified 16 studies for inclusion in the systematic review, 14 of which could be included in a meta-analysis.
Results
BICAMS has been widely validated across 11 languages and 14 individual cultures and locations. The meta-analysis demonstrated that BICAMS identified significantly reduced cognitive functioning in adults with MS compared to healthy controls. This was true for all three cognitive domains assessed by BICAMS: information processing speed (g = 0.943, 95% CI 0.839, 1.046, g < 0.001), immediate verbal recall memory (g = 0.688, 95% CI 0.554, 0.822, p < 0.001) and immediate visual recall memory (g = 0.635, 95% CI 0.534, 0.736, p < 0.001).
Conclusion
BICAMS has been widely applied across cultures and languages to assess cognition in MS. BICAMS offers a feasible, cost-effective means of assessing cognition in MS worldwide. Further validation studies are underway to support this project.
Keywords: Brief International Cognitive Assessment for Multiple Sclerosis, BICAMS, Cognition, Meta-analysis, Multiple sclerosis, Systematic review
Introduction
Multiple sclerosis (MS) is a chronic autoimmune, inflammatory neurological disease of the central nervous system [1]. MS can lead to a range of physical, psychiatric and cognitive impairments [2]. Cognitive difficulties have a negative impact over and above physical impairments. Cognition affects employment, disease management, personality, and many aspects of psychosocial function [3–5]. Patient self-report of cognitive performance is confounded by psychosocial variables, most markedly depression [6], and health professionals are poor at detecting cognitive impairment in routine clinical consultations [7]. Objective evaluation of cognitive status in MS is therefore required.
Traditional Cognitive Batteries for MS
Over the last 30 years, a handful of batteries have been developed to assess MS cognition. The seminal work of Rao and colleagues resulted in the Brief Repeatable Battery (BRB), which comprises the Selective Reminding Test (SRT, verbal memory), the Symbol Digit Modalities Test (SDMT, information processing speed), the Paced Auditory Serial Addition Test (PASAT, information processing speed), the 10/36 Spatial Recall Test, and the Word List Generation Test (WLG, executive skills) [8]. The battery takes about 45 min to complete. Until recently, the Rao battery was the most widely used assessment of MS cognition, with validation in several countries and frequent use in pharmacological trials. In an effort to develop a more comprehensive battery to assess MS cognition, an expert group was convened and recommended the Minimal Assessment of Cognitive Function in MS (MACFIMS) [9]. The MACFIMS replaces the 10/36 with the BVMT-R for visual memory, replaces the SRT with the CVLT-II for verbal memory, and adds the Judgment of Line Orientation (spatial skills) and the Delis–Kaplan Executive Function System (D-KEFS) Sorting Task (executive skills–flexibility). This battery takes about 90 min to complete. The MACFIMS has been validated in a few countries and used quite extensively in research. However, both of these batteries must be administered by a trained neuropsychologist, which limits their feasibility for widespread use in routine clinical assessment. In addition, the tests have been validated in few countries outside of the United States, which limits their validity for multinational trial use.
The Development of the Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS)
Historical Background
The Brief International Cognitive Assessment for MS (BICAMS) was designed by an expert consensus group to facilitate the assessment of cognition in MS [9]. In June 2010, an expert committee of seven neurologists and five neuropsychologists convened to develop recommendations for a clinical tool for neurologists and healthcare professionals working with people with MS. The purpose of the monitoring instrument would be for baseline ratings and regular follow-up assessments, which could be incorporated into routine clinical practice. It was decided that the recommended battery should be completed in 15 min and should not require any special equipment (beyond paper, pen and stopwatch). It should also be appropriate for international use (i.e. translation).
Development and Content
The committee critically evaluated the available cognition scales, their psychometric properties and their feasibility in the clinical setting. The group agreed that the monitoring tool should assess the domains of information processing speed and immediate verbal and visual recall, and would be sufficiently specific and sensitive to measure significant cognitive impairment in large clinical samples. The following three tests were selected.
Symbol Digit Modalities Test (SDMT)
The Symbol Digit Modalities Test (SDMT) was selected to measure information processing speed. The test consists of a number of single digits, each paired with a particular abstract symbol. The patient is presented with rows of the nine symbols that are arranged pseudo-randomly, and must say the numbers that go with each symbol in turn. The SDMT can be completed within 5 min, including the delivery of instructions and time allocated for practice and testing. The SDMT has shown superior psychometric properties. In particular, it has been reported to have high sensitivity of 82% to cognitive impairment in MS [10–12] and cognitive change [13–15], and moderate specificity of 60% [16], and has been validated in several countries [10–12]. It is most closely linked to magnetic resonance imaging (MRI) parameters [17]. The SDMT also has good external clinical validity and is associated with current [18] and future employment status [15].
California Verbal Learning Test-II (CVLT-II)
The California Verbal Learning Test-II (CVLT-II), which comprises five learning trials, is an examination of immediate verbal recall. In the CVLT-II, patients are read a list of words at a slightly slower rate than one item per second, and asked to recall as many items as possible in any order, across five trials. The test is a 16-item word list, with four items belonging to each of four categories, arranged randomly. The first five recall trials of the CVLT-II have a high degree of interdependence compared to other sections [19, 20], and the committee decided that they had sufficient psychometric rigour to be sensitive to MS impairment when used alone [19, 20]. Total time to administer the CVLT-II five recall trials is 5–10 min including instructions, testing and responses. The full CVLT-II also has external clinical validity and has been shown to be able to differentiate employed MS patients from patients unemployed due to MS [20].
Brief Visuospatial Memory Test–Revised (BVMT-R)
The Brief Visuospatial Memory Test–Revised (BVMT-R) is an assessment of immediate visual recall. The first three recall trials of the BVMT-R were selected by the committee. The psychometric properties of the BVMT-R recall trials are good [21]. The BVMT-R requires the patient to observe a 2 × 3 stimulus array of abstract geometric figures. There are three learning trials, each 10 s in length. The array is hidden from view, and the patient is required to draw the geometric figures in the correct position from memory after each 10 second exposure.
International Validity
The BICAMS committee subsequently published an international validation protocol [22]. A number of countries have published national validations. The aim of the present systematic review and meta-analysis is to synthesise the relevant national validation literature regarding BICAMS.
Method
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement was followed as a guide [23] for standardised conduct and reporting of the current systematic review and meta-analysis. According to the Cochrane Database (searched January 2018), there are no previously published reviews in the proposed area; therefore, the current review is the first to synthesise studies related to the validation of the BICAMS.
Systematic Literature Search
Search terms were developed and used to identify studies which were conducted as part of the BICAMS international validation protocol (Table 1). These keywords were searched within the title or abstract of the PubMed, PsycINFO, Medline and PsycARTICLES databases in January 2018.
Table 1.
Search terms for systematic review
| Search terms |
|---|
|
“multiple sclerosis” OR “ms” AND |
| “Brief International Cognitive Assessment for Multiple Sclerosis” OR “BICAMS” |
Eligibility Criteria
To refine the studies which were included as part of the international validation of the BICAMS protocol, only the following eligibility criteria were applied. The inclusion criteria for the studies in the present review were as follows: (a) peer-reviewed studies with no date restriction, written in the English language; (b) samples including adults with any clinical subtype of MS (or subtype combination); (c) studies which were undertaken as part of the international validation of the BICAMS protocol. The exclusion criteria were studies which included BICAMS but were not part of the international validation of the BICAMS protocol.
The additional criteria for inclusion in the meta-analysis were as follows: (d) studies included a healthy control (HC) comparison group; (e) they reported standard quantitative information based on the SDMT, CVLT-II and BVMT-R subscales (mean, standard deviation and sample size) of the MS cases and HC comparison group; and (f) there were a minimum of four studies, as specified by Rosenthal [24], which met the above criteria to be included in a meta-analysis.
To examine the eligibility criteria, all titles and abstracts which were returned were screened. The full texts of all studies considered to meet the eligibility criteria were accessed (see Fig. 1).
Fig. 1.
PRISMA flowchart for selection process of studies in systematic review and meta-analysis
Data Extraction
A headed table was developed to guide the extraction of relevant information from full texts, and to assess their eligibility for the final review. Extraction was initially carried out by the authors (FC and DL), and studies were organised according to whether they met criteria for the systematic review, meta-analysis or both. Following data extraction, 21 studies were excluded from the final review according to the exclusion criteria. Extrinsic moderators, including participant characteristics, were extracted from the 16 short-listed studies, comprising recruitment selection, diagnostic criteria, age, and subtype of MS, disease duration and time since diagnosis, education, and score on the Expanded Disability Status Scale (EDDS) [25]. Where appropriate, the characteristics of the control group were detailed, including sample size and age. The profession of the examiner administering the BICAMS was also recorded. Fourteen studies met the criteria for the meta-analysis from those included in the systematic review. For the meta-analysis, the standard quantitative information based on the subscales of SDMT, CVLT-II and BVMT-R (mean, standard deviation and sample size) of the MS cases and HC comparison group were extracted for baseline assessments of the BICAMS. All of the relevant data for the present review and meta-analysis was obtained from numerical information in texts, tables and graphs, and statistical analysis.
Quality Assessment
The quality ratings for the studies included in the systematic review were derived from the Effective Public Health Practice Project (EPHPP) [26] quality assessment tool. This instrument assigns ratings in quantitative studies according to selection bias, study design, confounders, blinding, data collection methods, and withdrawals or drop-outs. A total quality rating can be derived from the individual ratings of the measures. The EPHPP is particularly useful for examining the quality of studies in health care settings and has previously demonstrated strong inter-rater reliability [27]. The two authors (FC and DL) independently examined the articles, and any disagreements were discussed and resolved.
Statistical Analysis
The meta-analysis was conducted using the Comprehensive Meta-Analysis version 3 software program [28]. Three individual analyses were performed based on the averages of the SDMT, CVLT-II and BVMT-R subscales, which measure information processing speed and immediate verbal and visual recall, respectively. Effect sizes were calculated as standardised mean differences with Hedges’ g using the following interpretation: 0.2 = small; 0.5 = medium; 0.8 = large. Hedges’ g was selected because it offers the same interpretation as Cohen's d, while correcting for any potential biases that occur from small sample sizes, whereas Cohen's d is disposed to overestimating the absolute value of the standardised mean difference in small samples.
The effect size was modelled using a random-effects model, which estimates a mean of a distribution of effects, to examine the degree of difference between the MS group and the HC based on their performance across the former BICAMS subscales. The random-effects model assumes that the allocation of study weights is based on the inverse of the total variance, which includes both within- and between-study variance. Compared to the fixed-effect model, this model yields a wider confidence interval (CI) when there is significant heterogeneity among effect sizes. The predicted direction of results was that HC would have superior BICAMS subscale totals compared with adults with MS.
Heterogeneity was assessed using Cochran’s Q test, and the magnitude of heterogeneity using the I2 statistic. A significant Q statistic indicates dissimilar effect sizes across the included studies, and methodological or sample differences might be introducing variance in the results across studies. The I2 statistic assesses the proportion of heterogeneity across studies not due to random error and is interpreted as a small (25%), moderate (50%) or high (75%) level of heterogeneity [29].
Forest plots were created to visually assess for the presence of outliers. Sensitivity analyses were conducted to assess for publication bias by visually inspecting funnel plots of standardised mean differences against standard error, and then assessed using Egger’s test of funnel plot asymmetry [30] and Rosenthal’s fail-safe N [24]. If publication bias was indicated (Egger regression test: p < 0.1), then the trim-and-fill method [31] for random-effects models would be applied in order to impute “missing studies” to redress funnel plot asymmetry with adjusted pooled effect sizes and 95% CIs reported after the addition of potential missing studies.
Compliance with Ethics Statement
This article is based on previously conducted studies and does not contain any studies with human participants or animals performed by any of the authors.
Results
Systematic Review of BICAMS Validation Studies
Systematic Review Overview
Sixteen studies met the criteria for inclusion in the systematic review (Table 2). The BICAMS has been widely validated across 14 countries: the United States, Argentina (two papers), Belgium, Brazil, Canada, Czechoslovakia, Greece, Hungary, Iran, Ireland, Italy (two papers), Japan, Lithuania and Turkey. BICAMS was validated twice in both Italy and Argentina. From its original form in English, the instrument has been extensively translated into 11 languages, including Dutch, Czech, Greek, Hungarian, Italian, Japanese, Lithuanian, Persian, Portuguese, Spanish and Turkish. It is important to note the distinction between language and culture. BICAMS was validated across three English-speaking countries (USA, Canada and Ireland), since the cultural norms of these specific populations are markedly different, and this is likely to interfere with the reliability of cognitive testing (e.g. [32]). The proportion of individuals tested in the current study relative to the estimated total number of those individuals in the country [33] is presented in Table 3.
Table 2.
Study design and participant demographics
| Study (first author) | No. of subjects | Age, years Mean (SD) |
Gender % Female |
Education, years Mean (SD) |
% Type MS (RR/SP/PP/PR/CIS) | Disease duration Mean (SD) |
EDDS Mean (SD) |
SDMT | CVLT-II | BVMT-R |
|---|---|---|---|---|---|---|---|---|---|---|
| Costers [40] | ||||||||||
| MS | 97 | 45.42 (9.24) | 70% | 14.28 (1.86) | 84/12/4/0/0 | 12.97 (7.16) | 3.50 (2.50) | 52.1 (13.1) | 60.1 (12.9) | 25.4 (29) |
| HC | 97 | 43.52 (12.69) | 77% | 14.69 (1.61) | – | – | – | 61 (10.2) | 61.3 (60) | 28.2 (5.1) |
| Dusankova [36] | ||||||||||
| MS | 369 | 34 (10) | 68% | 14 [3] | 68/26/3/3/0 | 8 (7) | 3 (1.5) | 50 (13) | 60 (8) | 23 (7) |
| HC | 134 | 34 (9) | 71% | 14 (2.5) | – | – | – | 65 (9) | 52 (11) | 29 (4) |
| Eshaghi [41] | ||||||||||
| MS | 156 | 34.05 (9.09) | 71% | 14.13 (2.97) | 80/20/0/0/0 | 6.07 (5.08) | 2.5a (7.5)a | 43.93 (16.55) | 49.23 (10.39) | 20.88 (7.65) |
| HC | 90 | 33.65 (9.48) | 63% | 14.27 (3.56) | – | – | – | 55.58 (16.48) | 55.31 (8.46) | 25.01 (7.38) |
| Giedraitienė [42] | ||||||||||
| MS | 50 | 38.8 (10.2) | 32% | 15.9 (2.8) | 88/6/2/0/4 | 11.7 (9.2) | 3.3 (1.3) | 42.7 (13.9) | 55.9 (10) | 23.1 (7) |
| HC | 20 | 36.7 (16.4) | 10% | 17.5 (3.5) | – | – | – | 57.7 (11.5) | 65.7 (5.9) | 29.6 (4.1) |
| Goretti [43] | ||||||||||
| MS | – | – | – | – | – | – | – | – | – | – |
| HC | 23 and 243 | 38.9 (13) and 38.2 (13.1) |
66% 65% |
14.9 (3) | – | – | – | 56.3 (11.3) |
56.3 (9) |
27.9 (6.1) |
| Niccolai [44] | ||||||||||
| MS | 192 | 41.4 (10.8) | 74% | 12.3 (3.5) | 100/0/0/0/0 |
12.7 (1.7) |
2.7 (1.7) |
46.4 (12.8) | 49.9 (12.1) |
23.7 (8) |
| HC | 200 | – | – | – | – | – | – | 56.3 (11.3) |
56.3 (9) |
27.9 (6.1) |
| Niino [45] | ||||||||||
| MS | 156 | 41.4 (9.3) | 69% | 14.1 (1.9) | 88/11/1/0/0 |
2.4 (2.0) |
47.9 (14) | 48.6 (12.6) | 23.5 (8.4) | |
| HC | 126 | 39.3 (11.9) | 72% | 14.3 (1.6) | – | – | – |
61 (9.5) |
55.7 (10.5) | 28.3 (5.4) |
| O’Connell [46] | ||||||||||
| MS | 67 | 43.9 (12.1) | 73% |
13.6 (2.7) |
70/28/2/0/0 |
10.2 (8.4) |
1.8 (0.9) |
46 (12.9) | 45.3 (10.2) | 17.9 (7.1) |
| HC | 66 | 42.7 (12.8) | 68% |
14.1 (3.1) |
– | – | – | 56.1 (10.6) | 53.6 (9.1) | 20.9 (6.5) |
| Ozakbas [47] | ||||||||||
| MS | 173 | 37.5 (10.7) | 71% |
13.9 (7.3) |
151/18/4/0/0 |
9.2 6.1) |
2.4 (1.7) |
43.2 (12.5) | 45.7 (11.3) | 16.9 (8.5) |
| HC | 153 | 36.9 (8.9) | 71% |
15.4 (8.8) |
– | – | – | 53.5 (9.5) | 53.9 (7.7) | 22.5 (9.2) |
| Polychroniadou [48] | ||||||||||
| MS | 44 | 40.2 (9.9) | 61% |
14.1 (4.8) |
77/12/9/0/2 | RR 9.6 (3.7); SP 7.5 (3.7); PP 6.0 (2.6); CIS 1.6 (0.5) | – | 45 (17.2) | 55.5 (12.3) | 18.5 (8.3) |
| HC | 79 | 36.2 (10.6) | 61% |
15.4 (5.5) |
– | – | – | 61.4 (13.1) | 60.5 (10.7) | 22.1 (6.5) |
| Sandi [49] | ||||||||||
| MS | 65 | 41.9 (8.9) | 75% |
n = 31 < 12 years; n = 34 ≥ 13 years |
100/0/0/0/0 |
11.1 (7.6) |
2.5 (1.8) |
55.62 (15.48) | 55.37 (10.71) | 22.54 (8.54) |
| HC | 65 | 40.9 (11.8) | 75% |
n = 31 < 12 years; n = 34 ≥ 13 years |
– | – | – | 66.82 (12.44) | 59.03 (8.29) | 26.68 (5.56) |
| Spedo [50] | ||||||||||
| MS | 58 | 41.2 (12.2) | 69% | 12.7 (5.2) | 100/0/0/0/0 |
8.3 (6.6) |
2.2 (2) |
35.9 (16.1) | 42.1 (12.4) | 19.9 (8.6) |
| HC | 58 | 40.3 (11.9) | 55% | 12.5 (3.6) | – | – | – | 47 (13) | 53.4 (10.8) | 23.8 (7.7) |
| Strober [10] | ||||||||||
| MS | 65 |
45 (9.9) |
80% | 14 (2.4) | 72/22/3/1.5/1.5 |
10.2 (7.6) |
3.3 (1.7) (n = 54) |
48.4 (10.1) | 48.1 (9.8) | 26.2 (5.8) |
| HC | 46 | 45.2 (9.9) | 85% |
15 (1.8) |
– | – | – | 62.1 (10.7) | 56.7 (8.1) | 21.1 (6.4) |
| Vanotti [51] | ||||||||||
| MS | 50 | 43.42 (10.17) | 74% | 14.86 (2.78) | 78/18/4/0/0 | 13.06 (9.08) | 3.29 (2.55) | 45.14 (16.07) | 50.92 (12.37) | 20.7 (7.74) |
| HC | – | – | – | – | – | – | – | – | – | – |
| Vanotti [52] | ||||||||||
| MS | 50 | 43.42 (10.17) | 74% |
14.86 (2.78) |
78/18/4/0/0 | 13.06 (9.08) | 3.29 (2.55) | 45.14 (16.07) | 50.92 (12.37) | 20.7 (7.74) |
| HC | 100 | 42.37 (10.07) | 75% |
14.94 (2.49) |
– | – | – | 56.71 (10.85) | 60.88 (10.46) | 23.44 (5.84) |
| Walker [53] | ||||||||||
| MS | 57 | 45.44 (9.93) | 72% |
15.44 (2.68) |
77/16/7/0/0 | 10.11 (7.72) | 2.70 (1.85) | 49.65 (10.78) | 51.63 (10.05) | 24.6 (6.54) |
| HC | 51 | 41.92 (10.78) | 86% |
16.31 (2.11) |
– | – | – | 59.11 (8.46) | 57.72 (7.97) | 29.8 (3.63) |
aMedian and (range)
Table 3.
Comparison of number of people with MS in validation studies and total number in each country
| Study | Country | Number of people with MS in study | Number of people with MS in country | Percentage in study (%) |
|---|---|---|---|---|
| Costers [40] | Belgium | 97 | 12,000 | 0.81 |
| Dusankova [36] | Czech Republic | 369 | 16,000 | 2.31 |
| Eshaghi [41] | Iran | 156 | 50,000 | 0.31 |
| Giedraitienė [42] | Lithuania | 50 | 2621 | 1.91 |
|
Goretti [43] Niccolai [44] |
Italy | 23 | 68,000 | 0.03 |
| 192 | ||||
| Niino [45] | Japan | 156 | 12,000 | 1.30 |
| O’Connell [46] | Ireland | 67 | 7000 | 0.96 |
| Ozakbas [47] | Turkey | 173 | 40,000 | 0.43 |
| Polychroniadou [48] | Greece | 44 | 7000 | 0.63 |
| Sandi [49] | Hungary | 65 | 20,000 | 0.33 |
| Spedo [50] | Brazil | 58 | 30,000 | 0.19 |
| Strober [10] | United States | 65 | 400,000 | 0.02 |
| Vanotti [51, 52] | Argentina | 50 | 8000 | 0.06 |
| Walker [53] | Canada | 57 | 97,366 | 0.06 |
Quality Rating
The overall quality of the studies ranged from “moderate” to “weak” on the EPHPP template [26]. It is interesting to note that high-quality validation studies are rated as “weak” on several dimensions (e.g. “blinding” and “data collection method”), because the requirements for stringent international validation do not coincide with the parameters typically applied to general studies. This was evidenced by the polarity in scoring, with other dimensions rated strongly (“selection bias” and “study design”). Sensitivity for assessing the quality of validation studies using this tool is likely to be unsatisfactory for this reason.
Recruitment Method
Adults with MS were recruited from a variety of locations, including university hospitals, MS centres, specialist clinics and tertiary referral centres. In comparison, HC were either recruited from the community or an established normative sample, or were known to the adults with MS.
Sample Size
Thirteen studies included a group of adults with MS and an HC group, while two studies did not. The studies which deviated from this sampling style included either an HC group or an MS group only. The sample size differed greatly between studies. In adults with MS, this ranged from 369 to 44, while for HC the samples ranged from 200 to 20. Three studies contained equal-sized groups.
Gender
The gender ratio in the majority of the samples of adults with MS and HC disproportionally favoured women. This ranged from 61 to 80% in those with MS and from 55 to 86% in HC. It is likely that this trend is related to the gender differentiation in presentation, as the sex ratio in MS disproportionally favours women to men [34]. However, there was one anomaly: the gender ratio favoured mem in just one study, which is likely to have been influenced by the relatively small sample size (32% women with MS; 10% healthy women). Only two studies reported groups that were exactly matched on gender (61 and 75%).
Mean Age
There was a relatively consistent range of mean ages among adults with MS reported across the studies, from 34.0 (10.0) to 45.4 (9.9) years. The age of HC followed a similar spread, 33.7 (9.5) to 45.2 (9.9). The average age of adults with MS was 39 years, and among HC was 40 years.
Diagnosis and Selection Criteria
There was a great deal of disparity between the inclusion and exclusion criteria of the studies in the review. The most common form of diagnosis of MS was through the revised McDonald criteria [35] or later versions.
Type of MS
The most common subtype of MS represented across studies was relapsing-remitting (RRMS); other subtypes included secondary-progressive (SPMS), progressive-relapsing (PRMS), primary-progressive (PPMS) and clinically isolated syndrome (CIS). Three studies included only RRMS, while the remaining studies included a mix, although this was not uniformly represented in each study.
Disease Duration and Time Since Diagnosis
The average disease duration varied among studies from 6.07 (5.08) to 12.97 (7.16) years.
BICAMS Type and Mode of Delivery
All of the studies used the BICAMS paper version with the MS or HC groups. As noted above, various translations of the BICAMS have been created as part of the international validation protocol. Three studies reported the profession of the examiner: a neuropsychologist, MS nurse specialist and PhD student. Most studies did not report who administered BICAMS.
Meta-Analysis of BICAMS Validation Studies
The results from all three subscales of BICAMS—SDMT, CVLT-II and BVMT-R—showed that adults with MS performed significantly worse than controls. To examine the overall effect size for this, a meta-analysis was conducted.
SDMT
Figure 2 depicts a forest plot showing the effect size for each study which included SDMT. Overall, information processing speed was significantly lower in adults with MS than in HC, with a large effect size (g = 0.943, 95% CI 0.839, 1.046, p < 0.001). There was no evidence of outliers, heterogeneity (Q = 20.65, p > 0.050) or publication bias (Egger test: p > 0.050, two-tailed). The funnel plot (see Fig. 3) shows that the effect sizes were symmetrical. Trim-and-fill analysis [31] estimated that there were no studies missing from the analysis.
Fig. 2.
SDMT forest plot
Fig. 3.
SDMT funnel plot
CVLT-II
The effect size for each study which included the CVLT-II is shown in the forest plot in Fig. 4. Overall, verbal memory for immediate recall was significantly lower in adults with MS than in HC, with a medium effect size (g = 0.688, 95% CI 0.554, 0.822, p < 0.001). There was no evidence of outliers, although heterogeneity was indicated (Q = 36.07, p < 0.001) to a moderate extent (I2 = 63.95). Using the trim-and-fill method, one study would need to fall on the left of the mean effect size to make the plot symmetrical (see Fig. 5). Assuming a random-effects model, the new imputed mean effect size remained medium (p = 0.674, 95% CI 0.541, 0.808). The Egger test remained non-significant (p = 0.735, two-tailed), indicating that there was no publication bias.
Fig. 4.
CVLT-II forest plot
Fig. 5.
CVLT-II funnel plot
BVMT-R
Figure 6 displays the forest plot of effect sizes for the studies which included the BVMT-R. Overall, immediate recall of visual memory was significantly lower in adults with MS than in HC, with a medium effect size (g = 0.635, 95% CI 0.534, 0.736, p < 0.001). There was no evidence of outliers or heterogeneity (Q = 20.694, p > 0.050). The funnel plot presented in Fig. 7 shows that the effect sizes were symmetrical, and the trim-and-fill analysis estimated that there were no studies missing from the analysis. There was no evidence of publication bias according to the Egger test (p = 0.801, two-tailed).
Fig. 6.
BVMT-R forest plot
Fig. 7.
BVMT-R funnel plot
Sensitivity
Only one paper compared BICAMS to a longer battery [36]. In this large study, applying the operational criterion of “one or more abnormal tests”, BICAMS identified 58% of MS patients as cognitively impaired, while the MACFIMS [37], applying the MACFIMS operational criterion of “two or more abnormal tests”, identified 55% of MS patients as cognitively impaired. Interestingly, for the MS patients with disease duration over 21 years (n = 25), BICAMS identified a greater proportion as cognitively impaired (96 vs 76% on the MACFIMS).
Discussion
Summary of Findings
The aim of the current paper was to synthesise the national BICAMS validations to date. A total of 16 studies were included in the systematic review, of which 14 articles were then assimilated into the meta-analysis. The systematic review showed that BICAMS had been widely applied in many different languages, cultures and locations, with a diverse range of clinical samples including different disease subtypes, durations and severity. Most studies had included an HC sample with a similar educational background. BICAMS was administered in person by a neuropsychologist in most cases, although this was underreported. BICAMS was able to identify cognitive difficulties in adults with MS compared to HC, with difficulties identified in all three areas of information processing speed and immediate verbal and visual recall. Cognitive impairment was most marked in the domain of information processing speed. These findings fit with established knowledge and opinion and show that the selected scales can detect cognitive dysfunction in a 15-min battery. BICAMS also relies on previous findings regarding the component scales for its psychometric provenance.
Strengths
Several strengths of this review must be considered. Countries reported that BICAMS could be feasibly administered in around 15 min, with minimal materials, and was recommended for routine clinical cognitive assessment in MS. Within the validation studies there was a wide representation of cultures, languages and locations involved in the initiative. With the success of the international validation protocol and the number of those estimated within the population [33], it is possible to generalise the findings beyond the review. The proportion of samples represented varied between 0.2 and 1.91%, with the largest representation of adults with MS recruited from Lithuania. This is a significant first step in translating an understanding of cognitive impairment in MS globally. The direction of the results was as expected on all three scales; however, of note was that the CVLT-II was the most heterogeneous. This is likely a result of the extra linguistic and cultural demands of the stimuli in the subscale.
Limitations
There are a number of limitations which need to be recognised. In terms of sampling, there was a great deal of heterogeneity in sample size and MS type, ranging from disproportionally mixed subtypes to the inclusion of only one subtype. It is possible this may have biased the size of effect found, as cognitive impairment is more common in progressive forms of the illness. The HC were recruited from the community or were related or known to the adult with MS. In cases where HC were related to the adult with MS, this would pose a threat to the statistical assumption of independence between the samples examined. There was also a disparity in inclusion criteria, with variations of the McDonald criteria used [35], the control of comorbidity and medication, the details of which would have likely contributed to the observed heterogeneity. The aforementioned is likely to be in addition to the unequal gender ratio, in favour of women. It is important to consider the average age of those tested in the current review, and that the degree of cognitive impairment identified by BICAMS is likely to be negatively skewed, as the range of illness duration exceeded 6 years. The studies were cross-sectional, and baseline rather than re-test scores were included in the review; therefore, further longitudinal data is needed to understand the course of cognitive decline. Although BICAMS is reportedly able to be administered by qualified clinical health professionals, due to underreporting in the current studies, it appeared that most BICAMS were delivered by a neuropsychologist. Clinical thresholds of BICAMS scores have been proposed, but these were not recorded in the current review [38]. Few studies included accompanying neurocognitive assessments. Finally, this review provides only an interim analysis of the wider international validation of the BICAMS endeavour, since many countries have not yet published their data.
Conclusions
BICAMS has been shown to be relatively robust across a variety of cultures, languages and locations as a valid measure of cognition in MS. Working together with multiple sources of funding, the international MS community has participated in this endeavour to great success in improving optimisation of cognitive assessment. As a result of this international cooperation, a psychometrically sound assessment has become available, leading to increased awareness of MS cognition, which will improve cognitive symptom management.
Future Directions
A further dozen international validations are in progress. The range of countries with access to cognitive assessment for MS is increasing. Large combined datasets are facilitating the development of illuminating regression models [39]. It is anticipated that the new iPad version of the BICAMS, which is currently undergoing validation, will further extend the validity and reach of BICAMS within the MS community.
Acknowledgments
This study was made possible through the efforts of the international MS community in the validation of the BICAMS. The study was completed as part of a Doctorate in Clinical Psychology training by the first author at Royal Holloway, University of London.
Funding
No funding or sponsorship was received for this study or publication of this article.
Authorship
All named authors meet the International Committee of Medical Journal Editors (ICMJE) criteria for authorship for this article, take responsibility for the integrity of the work as a whole, and have given their approval for this version to be published.
Disclosures
Freya Corfield has nothing to disclose. Dawn Langdon has participated in speaker bureaus for Bayer, Merck, Almirall, EXCEMED, Teva, Roche, Novartis, Biogen and Sanofi; has served as consultant for Novartis, Bayer, Merck, Biogen, Teva and Sanofi; and has received research grants from Bayer, Merck, Novartis and Biogen. All are paid to DL’s institution.
Compliance with Ethics Guidelines
This article is based on previously conducted studies and does not contain any studies with human participants or animals performed by any of the authors.
Data Availability
All data generated or analyzed during this study are included in this published article/as supplementary information files.
Open Access
This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Footnotes
Enhanced digital content
To view enhanced digital content for this article go to 10.6084/m9.figshare.6490256.
References
- 1.Thompson AJ, Baranzini SE, Geurts J, Hemmer B, Ciccarelli O. Multiple sclerosis. Lancet. 2018;391(10130):1622–1636. doi: 10.1016/S0140-6736(18)30481-1. [DOI] [PubMed] [Google Scholar]
- 2.Benedict RHB, DeLuca J, Enzinger C, Geurts JJG, Krupp LB, Rao SM. Neuropsychology of Multiple Sclerosis: looking Back and Moving Forward. J Int Neuropsychol Soc. 2017;23(9–10):832–842. doi: 10.1017/S1355617717000959. [DOI] [PubMed] [Google Scholar]
- 3.Bruce JM, Hancock LM, Arnett P, Lynch S. Treatment adherence in multiple sclerosis: association with emotional status, personality, and cognition. J Behav Med. 2010;33(3):219–227. doi: 10.1007/s10865-010-9247-y. [DOI] [PubMed] [Google Scholar]
- 4.Kavaliunas A, Danylaite Karrenbauer V, Gyllensten H, Manouchehrinia A, Glaser A, Olsson T, et al. Cognitive function is a major determinant of income among multiple sclerosis patients in Sweden acting independently from physical disability. Mult Scler. 2017 doi: 10.1177/1352458517740212. [DOI] [PubMed] [Google Scholar]
- 5.Roy S, Drake A, Fuchs T, Dwyer MG, Zivadinov R, Chapman BP, et al. Longitudinal personality change associated with cognitive decline in multiple sclerosis. Mult Scler J. 2018 doi: 10.1177/1352458517753720. [DOI] [PubMed] [Google Scholar]
- 6.Kinsinger SW, Lattie E, Mohr DC. Relationship between depression, fatigue, subjective cognitive impairment, and objective neuropsychological functioning in patients with multiple sclerosis. Neuropsychology. 2010;24(5):573–580. doi: 10.1037/a0019222. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Romero K, Shammi P, Feinstein A. Neurologists’ accuracy in predicting cognitive impairment in multiple sclerosis. Mult Scler Relat Disord. 2015;4(4):291–295. doi: 10.1016/j.msard.2015.05.009. [DOI] [PubMed] [Google Scholar]
- 8.Rao SM, Cognitive Function Study Group of the National Multiple Sclerosis Society . A manual for the brief repeatable battery of neuropsychological tests in multiple sclerosis. Milwaukee, WI: Medical College of Wisconsin; 1990. [Google Scholar]
- 9.Langdon D, Amato M, Boringa J, Brochet B, Foley F, Fredrikson S, et al. Recommendations for a Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS) Mult Scler J. 2012;18(6):891–898. doi: 10.1177/1352458511431076. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Strober L, Englert J, Munschauer F, Weinstock-Guttman B, Rao S, Benedict RHB. Sensitivity of conventional memory tests in multiple sclerosis: comparing the Rao Brief Repeatable Neuropsychological Battery and the Minimal Assessment of Cognitive Function in MS. Mult Scler J. 2009;15(9):1077–1084. doi: 10.1177/1352458509106615. [DOI] [PubMed] [Google Scholar]
- 11.Deloire MSA, Bonnet MC, Salort E, Arimone Y, Boudineau M, Petry KG, et al. How to detect cognitive dysfunction at early stages of multiple sclerosis? Mult Scler. 2006;12:445–452. doi: 10.1191/1352458506ms1289oa. [DOI] [PubMed] [Google Scholar]
- 12.Camp SJ, Stevenson VL, Thompson AJ, Miller DH, Borras C, Auriacombe S, et al. Cognitive function in primary progressive and transitional progressive multiple sclerosis. A controlled study with MRI correlates. Brain. 1999;122(7):1341–1348. doi: 10.1093/brain/122.7.1341. [DOI] [PubMed] [Google Scholar]
- 13.Amato MP, Portaccio E, Goretti B, Zipoli V, Iudice A, Della Pina D, et al. Relevance of cognitive deterioration in early relapsing-remitting MS: a 3-year follow-up study. Mult Scler. 2010;16(12):1474–1482. doi: 10.1177/1352458510380089. [DOI] [PubMed] [Google Scholar]
- 14.Benedict RH, DeLuca J, Phillips G, LaRocca N, Hudson LD, Rudick R. Multiple sclerosis outcome assessments consortium validity of the symbol digit modalities test as a cognition performance outcome measure for multiple sclerosis. Mult Scler. 2017;23(5):721–733. doi: 10.1177/1352458517690821. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Morrow SA, Drake A, Zivadinov R, Munschauer F, Weinstock-Guttman B, Benedict RHB. Predicting loss of employment over three years in multiple sclerosis: clinically meaningful cognitive decline. Clin Neuropsychol. 2010;24(7):1131–1145. doi: 10.1080/13854046.2010.511272. [DOI] [PubMed] [Google Scholar]
- 16.Parmenter B, Weinstock-Guttman B, Garg N, Munschauer F, Benedict RHB. Screening for cognitive impairment in multiple sclerosis using the Symbol Digit Modalities Test. Mult Scler J. 2007;13(1):52–57. doi: 10.1177/1352458506070750. [DOI] [PubMed] [Google Scholar]
- 17.Vollmer T, Huynh L, Kelley C, Galebach P, Signorovitch J, DiBernardo A, et al. Relationship between brain volume loss and cognitive outcomes among patients with multiple sclerosis: a systematic literature review. Neurol Sci. 2016;37:165–179. doi: 10.1007/s10072-015-2400-1. [DOI] [PubMed] [Google Scholar]
- 18.Einarsson U, Gottberg K, von Koch L, Fredrikson S, Ytterberg C, Jin Y-P, et al. Cognitive and motor function in people with multiple sclerosis in Stockholm County. Mult Scler. 2006;12(3):340–353. doi: 10.1191/135248506ms1259oa. [DOI] [PubMed] [Google Scholar]
- 19.Fink F, Eling P, Rischkau E, Beyer N, Tomandl B, Klein J, et al. The association between California Verbal Learning Test performance and fibre impairment in multiple sclerosis: evidence from diffusion tensor imaging. Mult Scler. 2010;16(3):332–341. doi: 10.1177/1352458509356367. [DOI] [PubMed] [Google Scholar]
- 20.Stegen S, Stepanov I, Cookfair D, Schwartz E, Hojnacki D, Weinstock-Guttman B, et al. Validity of the California verbal learning test-II in multiple sclerosis. Clin Neuropsychol. 2010;24(2):189–202. doi: 10.1080/13854040903266910. [DOI] [PubMed] [Google Scholar]
- 21.Gaines JJ, Gavett RA, Lynch JJ, Bakshi R, Benedict RH. New error type and recall consistency indices for the brief visuospatial memory test—revised: performance in healthy adults and multiple sclerosis patients. Clin Neuropsychol. 2008;22(5):851–863. doi: 10.1080/13854040701565216. [DOI] [PubMed] [Google Scholar]
- 22.Benedict RHB, Amato MMP, Boringa JJ, Brochet B, Foley F, Fredrikson S, et al. Brief International Cognitive Assessment for MS (BICAMS): international standards for validation. BMC Neurol. 2012;12(1):55. doi: 10.1186/1471-2377-12-55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Moher D, Liberati A, Tetzlaff J, Altman DG, Altman D, Antes G, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement (Chinese edition) J Chinese Integr Med. 2009;7(9):889–896. [Google Scholar]
- 24.Rosenthal R. Meta-Analytic procedures for social science research. Beverley Hills: Educational Researcher; 1991. [Google Scholar]
- 25.Kurtzke JF. Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS) Neurology. 1983;33(11):1444–1444. doi: 10.1212/wnl.33.11.1444. [DOI] [PubMed] [Google Scholar]
- 26.Thomas BH, Ciliska D, Dobbins M, Micucci S. A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions. Worldviews Evid Based Nurs. 2004;1:176–184. doi: 10.1111/j.1524-475X.2004.04006.x. [DOI] [PubMed] [Google Scholar]
- 27.Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG. Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: Methodological research. J Eval Clin Pract. 2012;18(1):12–18. doi: 10.1111/j.1365-2753.2010.01516.x. [DOI] [PubMed] [Google Scholar]
- 28.Borenstein M, Hedges L, Higgins J and Rothstein H. Comprehensive Meta-Analysis version 3. 2005. p. Englewood: Biostat.
- 29.Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. Br Med J. 2003;327(7414):557–560. doi: 10.1136/bmj.327.7414.557. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test measures of funnel plot asymmetry. Br Med J. 1997;315(7109):629–634. doi: 10.1136/bmj.315.7109.629. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Duval S, Tweedie R. Trim and Fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56(2):455–463. doi: 10.1111/j.0006-341x.2000.00455.x. [DOI] [PubMed] [Google Scholar]
- 32.Chevalier TM, Stewart G, Nelson M, McInerney RJ, Brodie N. Impaired or not impaired, that is the question: navigating the challenges associated with using Canadian normative data in a comprehensive test battery that contains American tests. Arch Clin Neuropsychol. 2016;31(5):446–455. doi: 10.1093/arclin/acw031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Browne P, Chandraratna D, Angood C, Tremlett H, Baker C, Taylor BV, et al. Atlas of Multiple Sclerosis 2013: a growing global problem with widespread inequity. Neurology. 2014;83(11):1022–1024. doi: 10.1212/WNL.0000000000000768. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Boström I, Landtblom AM. Does the changing sex ratio of multiple sclerosis give opportunities for intervention? Acta Neurol Scand. 2015;132(S199):42–45. doi: 10.1111/ane.12430. [DOI] [PubMed] [Google Scholar]
- 35.Polman CH, Reingold SC, Banwell B, Clanet M, Cohen JA, Filippi M, et al. Diagnostic criteria for multiple sclerosis: 2010 Revisions to the McDonald criteria. Ann Neurol. 2011;69(2):292–302. doi: 10.1002/ana.22366. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Dusankova JB, Kalincik T, Havrdova E, Benedict RHBB. Cross cultural validation of the minimal assessment of cognitive function in multiple sclerosis (MACFIMS) and the brief international cognitive assessment for multiple sclerosis (BICAMS) Clin Neuropsychol. 2012;26(7):1186–1200. doi: 10.1080/13854046.2012.725101. [DOI] [PubMed] [Google Scholar]
- 37.Benedict RHB, Fischer JS, Archibald CJ, Arnett PA, Beatty WW, Bobholz J, et al. Minimal neuropsychological assessment of MS patients: a consensus approach. Clin Neuropsychol. 2002;16(3):381–397. doi: 10.1076/clin.16.3.381.13859. [DOI] [PubMed] [Google Scholar]
- 38.Beier M, Gromisch E, Hughes A, Alschuler K, Madathil R, Chiaravalloti N, et al. Proposed cut scores for tests of the Brief International Cognitive Assessment of Multiple Sclerosis (BICAMS) J Neurol Sci. 2017;381:110–116. doi: 10.1016/j.jns.2017.08.019. [DOI] [PubMed] [Google Scholar]
- 39.Smerbeck A, Benedict RHB, Eshaghi A, Vanotti S, Spedo C, Blahova Dusankova J, Sahraian MA, Marques VD, Langdon D. Influence of nationality on the Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS) Clin Neuropsychol. 2018;32(1):54–62. doi: 10.1080/13854046.2017.1354071. [DOI] [PubMed] [Google Scholar]
- 40.Costers L, Gielen J, Eelen PL, Van Schependom J, Laton J, Van Remoortel A, et al. Does including the full CVLT-II and BVMT-R improve BICAMS? Evidence from a Belgian (Dutch) validation study. Mult Scler Relat Disord. 2017;18:33–40. doi: 10.1016/j.msard.2017.08.018. [DOI] [PubMed] [Google Scholar]
- 41.Eshaghi A, Riyahi-Alam S, Roostaei T, Haeri G, Aghsaei A, Aidi MR, et al. Validity and reliability of a Persian translation of the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS) Clin Neuropsychol. 2012;26(6):975–984. doi: 10.1080/13854046.2012.694912. [DOI] [PubMed] [Google Scholar]
- 42.Giedraitienė N, Kizlaitienė RKG. The BICAMS Battery for Assessment of Lithuanian-Speaking Multiple Sclerosis Patients: relationship with age, education, disease disability, and duration. Med Sci Monit. 2015;10:3853–3859. doi: 10.12659/MSM.896571. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Goretti B, Niccolai C, Hakiki B, Sturchio A, Falautano M, Minacapelli E, et al. The brief international cognitive assessment for multiple sclerosis (BICAMS): normative values with gender, age and education corrections in the Italian population. BMC Neurol. 2014;14(1):171. doi: 10.1186/s12883-014-0171-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Niccolai C, Portaccio E, Goretti B, Hakiki B, Giannini M, Pastò L, et al. A comparison of the brief international cognitive assessment for multiple sclerosis and the brief repeatable battery in multiple sclerosis patients. BMC Neurol. 2015;15(1):204. doi: 10.1186/s12883-015-0460-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Niino M, Fukazawa T, Kira J, Okuno T, Mori M, Sanjo N, Ohashi T, Matsui M. Validation of the brief international cognitive assessment for multiple sclerosis in Japan. Mult Scler J Exp Transl Clin. 2017;263(4):205. doi: 10.1177/2055217317748972. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.O’Connell K, Langdon D, Tubridy N, Hutchinson M, McGuigan C. A preliminary validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) tool in an Irish population with multiple sclerosis (MS) Mult Scler Relat Disord. 2015;4(6):521–525. doi: 10.1016/j.msard.2015.07.012. [DOI] [PubMed] [Google Scholar]
- 47.Ozakbas S, Yigit P, Cinar BP, Limoncu H, Kahraman T, Kösehasanoğulları G. The Turkish validation of the Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS) battery. BMC Neurol. 2017;17(1):208. doi: 10.1186/s12883-017-0993-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Polychroniadou E, Bakirtzis C, Langdon D, Lagoudaki R, Kesidou E, Theotokis P, et al. Validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) in Greek population with multiple sclerosis. Mult Scler Relat Disord. 2016;9:68–72. doi: 10.1016/j.msard.2016.06.011. [DOI] [PubMed] [Google Scholar]
- 49.Sandi D, Rudisch T, Füvesi J, Fricska-Nagy Z, Huszka H, Biernacki T, et al. The Hungarian validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) battery and the correlation of cognitive impairment with fatigue and quality of life. Mult Scler Relat Disord. 2015;4(6):499–504. doi: 10.1016/j.msard.2015.07.006. [DOI] [PubMed] [Google Scholar]
- 50.Spedo CT, Frndak SE, Marques VD, Foss MP, Pereira DA, de Carvalho LF, et al. Cross-cultural adaptation, reliability, and validity of the BICAMS in Brazil. Clin Neuropsychol. 2015;4046:1–11. doi: 10.1080/13854046.2015.1093173. [DOI] [PubMed] [Google Scholar]
- 51.Vanotti S, Smerbeck A, Eizaguirre MB, Saladino ML, Benedict RRHCF, Vanotti S, Smerbeck A, Eizaguirre MB, Saladino ML, Benedict RRH, et al. BICAMS in the Argentine population: relationship with clinical and sociodemographic variables. Appl Neuropsychol Adult. 2017;23:1–10. doi: 10.1080/23279095.2017.1323751. [DOI] [PubMed] [Google Scholar]
- 52.Vanotti S, Smerbeck A, Benedict RHB, Caceres F. A new assessment tool for patients with multiple sclerosis from Spanish-speaking countries: validation of the Brief International Cognitive Assessment for MS (BICAMS) in Argentina. Clin Neuropsychol. 2016;4046:1–9. doi: 10.1080/13854046.2016.1184317. [DOI] [PubMed] [Google Scholar]
- 53.Walker LAS, Osman L, Berard JA, Rees LM, Freedman MS, MacLean H, et al. Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS): Canadian contribution to the international validation project. J Neurol Sci. 2016;362:147–152. doi: 10.1016/j.jns.2016.01.040. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
All data generated or analyzed during this study are included in this published article/as supplementary information files.