Psychometric Properties of Cognitive Assessment in Amyotrophic Lateral Sclerosis: A Systematic Review

Abstract

Purpose

We aimed to list all tests used to assess cognitive change in patients with amyotrophic lateral sclerosis (ALS) and to provide a descriptive synthesis of the psychometric properties of tests that were evaluated in a population of ALS patients.

Materials and Methods

The protocol is registered in PROSPERO (ID: CRD42017055603). We systematically search for literature in 11 databases. Full-text articles, in any language, with original research were included. All included articles were scrutinised by two independent authors. Disagreement was resolved by consensus. The framework of Lezak informed conceptualises of the tests identified. To evaluate methodological quality, we used the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN). Data were synthesised using criteria proposed by the Cochrane Back Review Group.

Results

Of 319 included articles, 46 articles reported information on the psychometric properties of cognitive tests used in patients with ALS. We found that the highest level of evidence was supported for the Reading the Mind in the Eye Test (RME), Addenbrooke’s Cognitive Evaluation (ACE) and Frontal Assessment Battery (FAB). Moderate level of evidence was found for the screening tests; Edinburgh Cognitive and Behavioural ALS Screen (ECAS) and the Montreal Cognitive Assessment (MoCA).

Conclusion

The screening test, ECAS and the social cognition test, RME, may have some advantages over other tests that have been used for assessing cognitive change in ALS patients. Recommendations of ALS-specific tests with sound psychometric properties are urgently needed.

Introduction

Insight about the cognitive changes that might occur in amyotrophic lateral sclerosis (ALS) has increased during the last decades. Such changes are now recognised as one of the integral features that characterizes this multisystem disease.1 Cognitive changes may influence ALS patient survival,2 caregiver burden,3 and affect decision-making processes regarding treatment.4 These findings stress the importance of accurately identifying those affected and detecting the extent of cognitive change.1,5 Early and repeated assessment may help healthcare professionals to better prepare families, enable individually tailored support, and provide timely opportunities for making essential decisions before impairment emerges.1,4

Recent studies show that the prevalence of cognitive changes in ALS varies between 30% and 62%.6,7 This wide range could be due to real differences in ALS patient characteristics (ie disease stage) or different tests used to assess cognitive changes.8–10 During the past years, more focus has been directed on cognitive assessment and Strong et al have recently published revised diagnostic criteria for ALS-specific cognitive impairment.1 To better define the extent and nature of changes, the authors differentiate between brief assessments and screening in clinical settings, assessment for prognostic stratifications in clinical trials and assessment focusing on research. A diagnosis of ALS-specific cognitive change depends on evidence of changes in executive function, including social cognition and/or language.1 Although controversy, evidence of memory dysfunction in patients with ALS are increasing.1,7 Changes in these cognitive functions may occur separately or in combinations with each other and should be assessed formally.1 As a minimum, Strong et al suggest assessment of Phonemic/letter fluency early in the course of ALS, and that all patients undergo screening with the ALS Cognitive Behavioural Screen (ALS-CBS) or the Edinburgh Cognitive and Behavioural ALS Screen (ECAS).1,5,11 The necessity of further assessment may depend on whether changes in cognition are registered and/or the setting in which an assessment takes place.1 Moreover, the patient’s physical disability, risk of exhaustion and possible practice effect should be taken into account. Further, Strong et al provide a list of tests, sensitive to ALS and recommended for evaluation of executive function, language and memory. According to the authors, extended assessment should include neuropsychologists or speech and language pathologists to support administration and interpretation of cognitive profiles and may thus not be feasible.1 Despite these efforts, not all instruments/tests are available everywhere yet and more effort and knowledge are needed among both clinicians and researches. Better application of the Strong criteria,1 using tests covering a wide spectrum of cognitive symptoms may help to better understand ALS-specific cognitive changes.9

To aid healthcare professionals to make informed choices as they select tests capable of detecting ALS-specific cognitive changes, this review has the following objectives: (i) to list cognitive tests applied in ALS-specific healthcare (Phase I); (ii) to provide a descriptive synthesis of the psychometric properties of tests that were evaluated in a population of ALS patients (Phase II).

Materials and Methods

To ensure transparent reporting, we report our systematic review according to the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA).12

Protocol and Registration

The protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) before any study procedures were undertaken. The protocol was last updated on February 12, 2020, at the PROSPERO site, available from: http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42017055603.13

Eligibility, Exclusion, and Inclusion Criteria

Eligible for inclusion in this systematic review were research articles, published in full text and reporting on cognitive function in patients with ALS. Patients had to be diagnosed using recognised criteria common to ALS-specific healthcare.14–16 We excluded articles about studies that employed experimental cognitive paradigms and paradigms used in combination with PET/CT/FMRI when focused on neuroanatomical questions of ALS. We also excluded articles reporting on children or animal. Included in Phase I were all relevant articles, regardless of the publication date, language they were written in and published. Of these results, we selected in Phase II, only those articles reporting on psychometric properties of cognitive tests that were evaluated in patients with ALS.

Information Sources

Systematic online searches were carried out in the Cochrane Library, McMaster⁺, Epistemonikos, Medline, Embase, SveMed+, CINAHL, AMED, OT seeker, PEDro and PsycINFO. We also checked the reference lists of systematic reviews we identified for additional relevant articles.

Search

A trained librarian carried out database searches. Our first search was carried out in February 2017 in order to identify the cognitive tests used in ALS-specific health care (Phase I). Search terms covered subject headings and free text words for the following three concepts: ALS/MND, cognition and specific cognitive domains known in ALS, and assessment tools. A second search was performed in December 2019 in order to find validating studies of cognitive tests used in patients with ALS (Phase II). In this second search we added a forth concept; psychometrics, to the searches in the main medical databases (Medline, Embase, PsycINFO, CINAHL, Cochrane Library), according to the search strategy filters developed by the Hedges Team for detecting clinical prediction studies.17,18 See Supplementary material 1 for the search strings and the exact terms that we used for Medline and Embase in the first and the second search.

Study Selection and Data Collection Process

In Phase I, all references from the first search were screened. In Phase II in order to find validating studies, included references from Phase I and new references from the second search were screened. Two of the authors independently screened all references and extracted the data. Consensus for inclusion was reached by discussion among the authors. For articles written in languages others than English, we consulted professional colleagues who are knowledgeable about health terminology in those languages. For eight articles, we contacted the authors for additional clarifying information.

Data Items

In Phase I, a neuropsychologist listed all the tests with sufficient supporting references used in the selected studies and classified them according to the cognitive domains they address; attention, concept formation and reasoning, construction and motor performance, executive functioning, learning and memory, perception, social cognition, verbal functions and language skills. This procedure was mainly based on well-established work in the field of neuropsychology19,20 as well as on test manuals or aim (s) of the studies of the included articles. In this evaluation, we were aware that the classification could to some extent vary, depending on the perspective of the investigators. Batteries, rating scales, inventories and screening tests were categorised separately when all subtests were applied. Since several articles, however, reported on just a few subtests from the complete test battery, and names used seemed not always identical with the original ones, names of the batteries were assigned to the above domain categories. When a complete battery was used, it was assigned separately.20

In Phase II, we extracted various characteristics of the included articles and published information on each identified test regarding psychometric properties. We were particularly interested in studies that evaluated the feasibility, internal consistency, reliability (test–retest reliability, inter-rater reliability), content validity, construct validity (structural validity and hypotheses testing), criterion validity and responsiveness of tests, as most of these evaluation parameters are specifically addressed in the Consensus-based Standards for the Selection of Health Status Measurement Instruments (COSMIN).21 In Table 1, definitions and explanations of the above-mentioned psychometric properties (except feasibility) are presented together with recommendations of preferred statistical methods and minimum sample size to include in analyses.

Table 1.

List of the Reported Psychometric Properties in Phase II – Definitions, Explanations and Recommendations of Preferred Statistical Method and Minimum Sample Size Included in Analyses

Psychometric Property – Definition and Explanations21	Preferred Statistical Method21	Sample Size21,22
Internal consistency:
Internal consistency measures the degree of	-Cronbach’s alpha	50
interrelatedness among items. Internal consistency is only relevant when items form a reflective model, meaning all items are a manifestation of the same construct.	-Goodness of fit statistics	100-500
Assessment of the unidimensionality of the test is a prerequisite for internal consistency statistics to be evaluated objectively.	-Factor analysis or item response theory analysis	100-500
	-Internal consistency coefficient for each (unidimensional) (sub)scale separately.	50
Reliability:
Reliability is the degree to which a test is free from measurement error.	-ICC -Cohen’s kappa or weighted kappa depending on the scale used.	50
-Test-retest reliability refers to the robustness of the measurement error related to different testing times.
-Interrater reliability refers to the robustness of the measure-error related to different scorers. For evaluation of inter-rater reliability, the raters need to be independent.
Content validity (including face validity)
Content validity represent the degree to which the content of a test adequately reflect the construct to be measured. Content validity of a test is determined by asking patients for patient-reported tests or experts within the field of interest to judge the relevance and comprehensiveness of a test’s items. Evaluation of face validity is subjective; therefore, no standard exists.
Construct validity
Construct validity represent the degree to which the scores of the test are consistent with hypotheses based on the assumption that the test validly measures the construct to be measured.
-The structural validity aspect of construct validity refers to the degree to which the score of a test is an adequate reflection of the dimensionality of the construct to be measured. It is only relevant for tests that are based on a reflective model.	-Factor analysis	100
-The hypothesis testing aspect of construct validity concerns the relationships to scores of other instruments. It is recommended to formulate a priori hypotheses, to specify the expected direction (positive or negative) of correlations one expects, or to quantify the range of expected differences between groups or scores. It is important that the comparator test is clearly described and has adequate measurement properties in order to interpret correlations properly.	-Correlations	50
Criterion validity
Criterion validity refers to the degree to which the scores of a test are an adequate reflection of a ‘gold standard’ (an external criterion of the phenomenon being measured). For health-related patient reported outcome measures, the only possibility is to compare a shortened version of a test to the original long version.	-Correlations or the area under the ROC. -Sensitivity and specificity for dichotomous scores	50
Responsiveness
Responsiveness is the ability of a test to detect change over time in the construct being measured. When evaluating responsiveness, at a minimum, at least some of the patients should display changed performance (eg, improvement or decrement). Therefore, it is important to describe precisely what occurs during the follow-up. Responsiveness may be evaluated by hypotheses testing or comparison to a ‘gold standard’.	-Hypotheses testing: Correlations -Comparison to a gold standard: Correlations or the area under the ROC. -Sensitivity and specificity for dichotomous scores	50

Abbreviations: eg, for example; ICC, intraclass correlation coefficient; ROC, receiver operating characteristic.

Assessment of Study Quality (Risk of Bias)

In Phase II, we evaluated the risk of bias and methodological quality of the studies in the included articles. We used the four steps of the COSMIN checklist.21 Only the standards corresponding to the psychometric property of interest were used and accompanied by specific explanations and instructions given in the COSMIN checklist.21 As feasibility is not part of the COSMIN checklist, we did not evaluate the methodological quality of studies that assessed the feasibility of the tests they used.

Level and Synthesis of Evidence

For tests identified in Phase II, we first calculated a quality score (excellent, good, fair, or poor) per measurement property, according to the scoring system developed by Terwee and colleagues.22 For each test of a study, psychometric properties were assessed and were rated as adequate (+), not adequate (-) or unclear (?). A rating of 0 was given when no information was available in the article. We then synthesised individual ratings using the criteria proposed by the Cochrane Back Review Group, leading to categories of strong evidence, moderate evidence, limited evidence, conflicting or unknown evidence.23

Ethical

Ethical approval and consent to participate are not applicable for this systematic review and analysis since no research subjects were recruited. This study adheres to the principles of the Declaration of Helsinki.24

Results

In Phase I for identifying cognitive tests, we included 319 articles. The flow of information through Phase I of the systematic review is presented in Figure 1. Of these 319 articles identified in our first search in 2017, 35 articles reported information on the psychometric properties of cognitive tests used in patients with ALS. In our second search conducted in 2019, we identified 11 new references reporting information on the psychometric properties of cognitive tests used in patients with ALS. A total of 46 articles were therefore included in Phase II of our systematic review. See Figure 2 for flow of information through Phase II of the systematic review.

Figure 1.

Flow of information through Phase I of the systematic review: Identifying cognitive tests.

Notes: Adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:10.  Creative Commons license and disclaimer available from: http://creativecommons.org/licenses/by/4.0/legalcode.12

Abbreviation: ALS, amyotrophic lateral sclerosis.

Figure 2.

Flow of information through Phase II of the systematic review: Evaluating psychometric properties.

Notes: Adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:10.  Creative Commons license and disclaimer available from: http://creativecommons.org/licenses/by/4.0/legalcode.12

Abbreviation: ALS, amyotrophic lateral sclerosis.

Phase I: Cognitive Tests Identified for Use in ALS-Specific Healthcare

Table 2 lists the most frequently reported; cognitive tests classified into eight cognitive domains, test batteries, and rating scales evaluated in the included articles. The total number of tests referenced within each cognitive domain and the total number of times a test is referenced are shown.

Table 2.

List of the Most Frequently Reported Cognitive Tests, Test Batteries and Scales Used in Patients with Amyotrophic Lateral Sclerosis (ALS) in 319 Articles Identified in Phase I

Cognitive Domains and Tests (Total Number of Tests Identified per Category)a		Number of Articlesb
Cognitive domain, followed by 2 most-frequently used testsc
Attention, orientation and working memory (n=34)
	Digit Span backward	89
	Stroop Test/Color- Word Interference Test	78
Concept formation and reasoning (n=15)
	Card sorting tests	77
	Raven’s Coloured Progressive Matrices	52
Construction and motor performance (n=12)
	Block Design	17
	Complex Figure Test (copy)	9
Executive functioning (n=12)
	Phonemic fluency tests	148
	Category fluency tests	99
Learning and memory (n=37)
	Complex Figure Test (recall)	43
	Rey Auditory- Verbal Learning Test	41
Perception (n=14)
	Benton Judgement of Line Orientation	23
	Visual Object and Space Perception Battery	16
Social cognition (n=12)
	Reading the Mind in the Eyes Test	10
	Judgement of Preference Test	4
Verbal function and language skills (n=42)
	Boston Naming Test	52
	North American Adult Reading Test	34
Other kind of batteries or inventories, followed by 2 most-frequently used testsc
Test batteries (n=13)
	Wechsler Adult Intelligence Scale	12
	Mental Deterioration Battery	2
Rating scales, inventories and screening tests (n=32)
	Mini-Mental State Examination	134
	Frontal Assessment Battery	44

Notes: ^aCognitive tests are categorised according to the cognitive domain they assess, whereas the other kinds are categorised according to type of test (eg, scale, battery, etc.), as indicated. ^bA given article could report on more than one test. ^cSee Supplemental material 2 for the complete list of all reported tests.

A complete list of all cognitive tests identified in Phase I are presented in Supplementary material 2. We observed that for some tests, several different versions were used, which indicates that the tests were in the process of being developed or had been revised (eg the Wechsler Adult Intelligence Scale), or that the tests were modified or adapted specifically for use in patients with ALS.

Phase II: Cognitive Tests That Were Evaluated in Patients with ALS

Table 3 summarises the characteristics of articles and studies reported therein that were included in Phase II of this review. All of the included studies reporting on feasibility of tests, focused on the suitability of specific tests for use in patients with motor and speech impairment. The authors of those studies indicated that the Edinburgh Cognitive and Behavioural ALS screen (ECAS) might be more suitable for patients with ALS than the Frontal Assessment Battery (FAB) and the Montreal Cognitive Assessment (MoCA) screening test. Table 4 lists cognitive tests evaluated in patients with ALS identified in the first (2017) and second search (2019) of the review, and examined in Phase II according to their cognitive domain classification. Available in Table 4 is also information in which country each test has been validated.

Table 3.

Characteristics of Articles Included in Phase II

Study Characteristics	Values
Articles, number	46
Publication year, range	1985-2019
ALS-sample size per study, range	5-274
Mean age (year) of ALS patients, range	50.7 – 68.5
Recruitment setting, number of articles
Hospital or clinical	40
Register data	2
Others	4
Continent, number of articles
America	14
Asia	6
Europe	22
Oceania	4
Study design, number of articles
Cross sectional	38
Longitudinal	8
Cohort study	3
Case-control	19
Diagnostic	22
Combination of designs	2
Psychometrics, number of articles
Internal consistency	9
Reliability	5
Content validity	1
Construct validity – structural validity	4
Construct validity - hypothesis testing	28
Criterion validity	29
Responsiveness	3
Feasibility	5

Abbreviation: ALS, amyotrophic lateral sclerosis.

Table 4.

List of Cognitive Tests Evaluated in an ALS Population, Identified in the First and Second Search (2017 and 2019) of the Review and Examined in Phase II, Classified into Cognitive Domains

Test	Country of Validation (Alphabetical Order)	Reference Number
Executive functioning
Category fluency tests	Italy	25
Phonemic fluency tests	Italy, Scotland	25,26
Learning and memory
Prospective memory test	China	27
Social cognition
Awareness of Social Inference Test	Australia	28
Ekman 60 Faces Test	England	29
Reading the Mind in the Eyes Test	Ireland	30
Screening tests
Addenbrooke’s Cognitive Examination	Australia, Canada, China, Greece, Japan	31–36
ALS Brief Cognitive Assessment	United States of America	37
ALS Cognitive Behavioural Screen	Brazil, Italy, Spain, United States of America	38–42
Edinburgh Cognitive and Behavioural ALS Screen	China, Czech Republic, Germany, Greece, Ireland, Scotland, Spain, Switzerland	5,35,43-53
Frontal Assessment Battery	Australia, Canada, Italy, Japan, Korea, United States of America	25,36,54-58
Mini-Mental State Examination	China, Italy, United States of America	25,34,37
Montreal Cognitive Assessment	Canada, Japan	55,58
Penn State Brief Exam of Frontal and Temporal Dysfunction Syndromes	United States of America	59
University of California San Francisco Screening Battery	United States of America	60,61
Test batteriesa
ALS-Computerized Frontal Battery	Canada	62
Battery I: Frontal Behavioral Inventory, Phonemic fluency test, Beck Depression Inventory	United States of America	63
Battery II: Addenbrooke’s Cognitive Examination-Revised, Digit Span-backword, Hayling Test, Iowa Gambling Test, Ekman 60 Faces-modified, Cambridge Behavioural Inventory-Revised	Australia	64
Battery III: ALS Cognitive Behavioural Screen, Phonemic fluency test, Semantic fluency test, Boston Naming Test-short version, Comprehension of Instructions, Peabody Picture Vocabulary Test-Revised	Greece	65
Battery IV: Token Test-modified, Graded Naming Test, British Picture Vocabulary Scale-II, Test for the Reception of Grammar, Pyramids and Palm Trees Test, Kissing and Dancing Test, Cookie Theft Picture	England	66
Battery V: Frontal Behavioural Inventory-Modified, Mini-Mental State Examination, Rey Auditory- Verbal Learning Test, Controlled Oral Word Fluency Test	United States of America	67
Communicative Participation Item Bank	United States of America	68
Mental Deterioration Battery	Italy	69

Notes: ^aBatteries I–V were developed specifically for use in patients with ALS.

Abbreviation: ALS, amyotrophic lateral sclerosis.

Phase II: Assessment of Study Quality

Assessment of the methodological quality of included studies using the COSMIN checklist21 was for each psychometric property of interest (see Table 1 for definitions and explanations):

Internal Consistency

None of the nine studies evaluating internal consistency explicitly described whether the tests used were based on a reflective or formative model. Nonetheless, all of the nine studies calculated Cronbach’s alpha or Item response theory (IRT), methods that are both adequate parameters of internal consistency. Four studies assessed the unidimensionality of the test they used. Of these four studies, two conducted the unidimensional subtest separately, and all four had a sample size of greater than 50.

Reliability

None of the five studies that assessed reliability evaluated test–retest reliability. For evaluation of inter-rater reliability, the raters were independent in four studies. Three studies used intraclass correlation coefficient, a preferred statistical measure of reliability. Two studies used Pearson’s correlation, which does not take systematic error into account. Two studies had a sample size of less than 50 participants.

Content Validity

One study assessed the content validity of the test they used; however, they did not provide a detailed description of how this was done.

Construct Validity – Structural Validity

None of the four studies evaluating structural validity explicitly described whether the tests they used were based on a reflective or formative model. Three studies used IRT methods and followed the recommendation of checking the dimensionality of the items. One study that relied on classical test theory did not use factor analysis for this check, which is the preferred statistical method to determine the structure of the instrument.

Construct Validity – Hypothesis Testing

Of 28 studies evaluating hypothesis testing, 26 studies formulated a priori hypotheses. Specification of expected direction (positive or negative) of correlations one expected, or quantification of the range of expected differences between groups or scores were lacking completely or described only vaguely in 23 studies. Of the 14 studies that used hypothesis testing to investigate the convergent validity of tests, all 14 studies provided references for the comparator test; however, eight of these studies lacked information about measurement properties. In 27 studies, correlations were used. In seven studies, the sample size was inadequate.

Criterion Validity

In 24 of 29 studies evaluating criterion validity, we found that the authors compared the scores of the new test to a widely used one, not the original longer version. Statistic methods like correlation, receiver operating characteristics and sensitivity/specificity were used in all 24 studies. The sample size were inadequate in eight studies.

Responsiveness

All the three studies evaluating responsiveness tested the subjects at least two times during the assessment period and reported the time interval between measurements. Precise descriptions of what occurs during the follow-up were not adequately described in any of the included studies. To evaluate responsiveness, all three included studies tested a specific hypothesis. All three studies formulated hypotheses a priori, but the hypotheses were formulated only vaguely. Two studies reported p-values for the hypothesis testing they did, even if this should be avoided since it is not relevant to examine whether correlations statistically differ from zero.70 The sample size in the analyses were too small in all the included studies.

Phase II – Synthesis of Evidence

The results from the synthesis of the evidence on the quality of the test psychometrics in Phase II of this review are presented in Table 5.

Table 5.

Levels of Evidence: Quality of Psychometric Properties and Feasibility of Tests in Articles Identified During Phase II

Test	Internal Consistency	Reliability	Content Validity	Structural Validity	Hypothesis Testing	Criterion Validity	Responsiveness	Feasibility
Executive
Category fluency tests						?
Phonemic fluency tests					?	?
Learning memory
Prospective memory test					?
Social cognition
Awareness of Social Inference Test					++
Ekman 60 Faces Test					?
Reading the Mind in the Eyes Test	+++			+++
Screening test
Addenbrooke’s Cognitive Examination	?			?	+++	?	+	x
ALS Brief Cognitive Assessment						?
ALS Cognitive Behavioural Screen						?		x
Edinburgh Cognitive and Behavioural ALS Screen	?	?			++	++	?	x
Frontal Assessment Battery					+++	?	+	x
Mini-Mental State Examination						+/-
Montreal Cognitive Assessment					++	?		x
Penn State Brief Exam of Frontal and Temporal Dysfunction Syndromes
University of California San Francisco Screening Battery		?			+	?
Test batteries a
ALS-Computerized Frontal Battery					?
Battery I					?			x
Battery II	?			?
Battery III						?
Battery IV		?			+
Battery V					?
Communicative Participation Item Bank	+		?			?
Mental Deterioration Battery				+

Notes: +++ or — = Strong evidence with consistent positive or negative results from multiple studies with good methodological quality or one study with excellent methodological quality. ++ or – = Moderate evidence with consistent positive or negative results from multiple studies with fair methodological quality or one study with good quality. + or - = Limited evidence with positive or negative results from a study with fair methodological quality. +/− = Conflicting evidence with conflicting results from individual studies. ? = Unknown level of evidence due to results from studies with poor methodological quality. x =Feasibility was evaluated. Gray-highlighted box indicate high to moderate level of evidence. ^aBatteries I–V were developed specifically for use in patients with ALS.

Abbreviation: ALS, amyotrophic lateral sclerosis.

We found that the highest level of evidence was supported for the screening tests; Addenbrooke’s Cognitive Evaluation (ACE) and Frontal Assessment Battery (FAB), and the Reading the Mind in the Eye Test (RME), a test of social cognition. We found support for a moderate level of evidence for the ECAS and the MoCA screening tests and the Awareness of Social Inference Test, another test of social cognition. It should be noted that when evaluating the quality of screening tools, only the ECAS has alternative versions limiting practice effects. Moreover, it is available in several languages. Of all the included articles in Phase II, we observed that the greatest number of psychometric properties of interest was evaluated for the ACE and the ECAS. The Mini-Mental State Examination (MMSE) showed inconsistent levels of evidence.

Discussion

In this review, we had two aims: (i) to compile a comprehensive list of all cognitive tests used in ALS-specific healthcare and (ii) to provide a descriptive synthesis of the psychometric properties of tests validated in patients with ALS.

Our literature search and analyses indicated that a large number of tests are being used in ALS-specific healthcare. Of these tests, however, only a fraction have been evaluated for use within a population of ALS patients. This may lead to different conclusions about a patient`s cognitive functioning.22 Although commonly used, some tests, especially the MMSE, have produced conflicting evidence when it comes to assessing ALS-specific cognitive changes.25,34,56,67

To date, of the tests used to evaluate cognitive changes in ALS, none fulfils all of the essential measurement properties stipulated by the COSMIN checklist.21 We did, however, observe acceptable measurement properties for four screening tests; the ACE, FAB, ECAS and MoCA. Feasibility studies, however, showed that the ECAS might be more suitable, and hence preferable, for screening cognitive function in ALS patients compared to the FAB and MoCA, a finding that is in line with that of another systematic review.71 While several screening tests have been developed to detect cognitive changes, the ECAS has been designed specifically to assess cognitive changes in patients with ALS.5 Strong et al recommend that either the ECAS or the ALS-CBS be administered to all patients.1 However, in our review, we found the psychometric properties of the ALS-CBS poorly evaluated. Our analysis revealed that, of all the screening tests evaluated in the included studies, the psychometric properties of the ECAS and ACE were the most scrutinised. It should be noticed, however, that effort has been made to provide alternate versions of the ECAS.72 The challenge in recurrent testing is that the disease may sometimes deteriorate quickly. Therefore, the tests should be easy to perform, but not be easily remembered and learned at the same time. This enables recurrent testing which is desirable to follow the development of cognitive changes over time. This way, significant practice effects can be reduced if not eliminated. Moreover, translations in several languages would allow international multicentre studies, which could increase knowledge and patient care.

Of two tests of social cognition evaluated in patients with ALS, we found that the highest level of evidence was supported for the RME. Although not specifically developed for use in patients with ALS, the RME accounts for potential physical disability as it can be conducted in either written or spoken.73 The RME is recommended by Strong et al for evaluation of the ability to read the emotions of others, a social cognitive process most likely in patients with ALS who meet the criteria for frontotemporal dementia.1

Our data suggest that the ECAS may be preferred over other screening tests for validly and reliably screening cognitive change in patients with ALS and that the RME may be suitable for the extended assessment of social cognition. The advantage of ECAS is that it is an ALS-specific test and therefore may show changes that are characteristic for ALS. However, our data did not allow us to give clear recommendations on which specific tests to use for extended assessment of other cognitive changes that may occur in patients with ALS. Similar methodological flaws recurring across the included studies preclude us from making such recommendations, and also prompt us to highlight the need for researchers and clinicians to pay particular attention to the methods applied in psychometric studies. Hence, there is a great potential for improving the methodological quality of studies that aim to evaluate the measurement properties of a cognitive test used in ALS-specific healthcare.

Study Strengths and Limitations

The strengths of the current study lay in the rigorous quality assessment achieved by the use of the COSMIN checklist, a widely used and validated tool to assess the risk of bias in evaluating measurement properties of a test.21 Also, in combination with the COSMIN checklist, we used recommended quality criteria21 to synthesise the level of evidence.22 Other strengths are that a trained librarian conducted the literature search and that the searches were relatively comprehensive and exhaustive. However, we did not consult all databases, nor grey literature, a strategy that future reviews might consider.

This review did not take studies in healthy controls into account. A more accurate reflection of the psychometric properties of each test would include studies with healthy controls. Therefore, the reliability and validity of some tests may be underestimated. The analyses demonstrate that in particular, screening instruments have been evaluated so far, and more research is needed with regard to evaluate further cognitive/neuropsychological tests. Another limitation can be the inclusion of articles since 1985, since this may risk confusing the results as different versions of cognitive tests may have varying psychometric properties. We argue that our aim was to provide a comprehensive overview possibly also including results from countries where not all tests/versions are available yet. Furthermore, the analyses in the present study are limited to the assessment of cognition as one aspect of behavior. We acknowledge the importance to investigate tests assessing emotional aspects of behavior as a future.

The quality of included studies was rated as fair to poor in many cases, which limits the conclusions we can draw. Available data did not allow us to conduct meta-analyse and we could not perform assessments of heterogeneity. Thus, at this point, we can only give guarded advice for tests that should be used in ALS-specific healthcare. We are aware that the presence of publication bias may have affected our results. Finally, some evaluator bias may have also affected our result, because three of the authors of this review are involved in the validation of the Norwegian version of the ECAS.74

Conclusions

To the best of our knowledge, this is the first study to systematically identify and evaluate tests that are used to measure the cognitive change in ALS and those tests that have been validated in patients with ALS. The screening test, ECAS and the social cognition test, RME; may have some advantages over other tests that have been used for assessing cognitive change in ALS patients. Psychometric properties of tests that were evaluated in a population of ALS patients have mainly been examined in screening instruments. Thus, more research regarding the validation of comprehensive neuropsychological assessment seems appropriate to implement the recent ALS criteria.1

Disclosure

Tina Taule reports three of the authors of this review are involved in the validation of the Norwegian version of the ECAS. The authors report no other potential conflicts of interest for this work. This work was funded by grants from the Western Regional Norwegian Health Authority [grant number: 912,158, 2017].