Abstract
Aim
The Mini‐Mental State Examination is a widely used cognitive assessment tool. However, it has several limitations, including the learning effect and interrater reliability. Therefore, we developed a Computer‐Based Cognitive Assessment Tool (CompBased‐CAT), which runs on a tablet or personal computer. In this study, we examined the validity and discrimination ability of the CompBased‐CAT.
Methods
Participants were recruited from the Otasha‐Kenshin study carried out in 2016. We included 773 community‐dwelling older individuals in Japan (332 men, 441 women, aged 65–97 years). CompBased‐CAT scores were converted to z‐scores, and the correlation with Mini‐Mental State Examination scores was examined using Pearson's correlation coefficient. Furthermore, the ability to discern cognitive impairment was examined using the receiver operating characteristic curve.
Results
The Pearson's correlation coefficient for the Mini‐Mental State Examination scores and each task component of the CompBased‐CAT ranged from 0.24 to 0.41 (P < 0.001), and the correlation coefficient of the total z‐scores was 0.51 (P < 0.001). The sensitivity, specificity and area under the receiver operating characteristic curve of the discriminating ability of the CompBased‐CATool for cognitive impairment were 0.81, 0.77 and 0.85, respectively.
Conclusions
The CompBased‐CAT certainly possesses validity, discriminating ability and utility as a new cognitive assessment tool in community‐dwelling older individuals. Geriatr Gerontol Int 2020; ••: ••–••.
Keywords: assessment, cognitive function, community‐dwelling elderly, dementia, screening
Introduction
The incidence of dementia has increased due to the rapidly aging population.1 It is estimated that the global number of patients with dementia will reach 115.4 million by 2050.2 Dementia is related to a decline in quality of life, and increased economic and patient care burdens.2
Unfortunately, current drugs are unable to prevent or cure dementia.3, 4 Therefore, early detection of cognitive decline is essential to prevent dementia. Early detection can effectively identify the disease in its treatable stage and provide lifestyle guidance in an attempt to slow cognitive impairment. This will allow patients and their families to adequately prepare themselves if the patients are at risk for developing dementia.5 In recent years, tests such as neuroimaging and genetic biomarkers have been developed. However, they are expensive and not readily available to community‐dwelling older individuals.6
The Mini‐Mental State Examination (MMSE) is a widely used cognitive assessment test, which screens for functional cognitive impairment and tracks changes in cognitive function over time.7, 8 However, the MMSE, which consists of a series of questionnaires, has several limitations, including a learning effect from repetitive measurements, interrater reliability and the need for several trained personnel for its administration.9 Furthermore, the MMSE primarily assesses memory and language function, and shows poor sensitivity toward detecting frontal lobe dysfunction.9
Therefore, we developed a Computer‐Based Cognitive Assessment Tool (CompBased‐CAT) consisting of six tasks, that runs on a tablet computer (PC), which automatically executes the test.
In the present study, we first examined the relationship between the CompBased‐CAT and MMSE scores to verify the validity of the CompBased‐CAT.
Methods
Participants
Participants were recruited from comprehensive health checks that were part of the Otasha‐Kenshin study carried out in 2016 at the Tokyo Metropolitan Geriatric Hospital and Institute of Gerontology (Tokyo, Japan).10, 11 The inclusion criteria were older community‐dwelling individuals, aged ≥65 years. We excluded 54 participants, who did not consent to undergo cognitive assessment and those who found the CompBased‐CAT difficult. Thus, data from 773 individuals were analyzed. All participants provided informed consent. This study was approved by the ethics committee of the Tokyo Metropolitan Institute of Gerontology (approval number H14,2016).
Cognitive measurements
Measurement of the CompBased‐CAT
The CompBased‐CAT was created to assess specific cognitive functions. It consisted of the following six tasks (Fig. 1): (i) digit span forward (attention and concentration)12 (ii) digit span backward (attention and concentration)12 (iii) memory of item names (immediate memory); (iv) memory recall of item names (remote memory); (v) Stroop (executive function and selective attention)13 and (vi) recognition of figures (space perception). This test was carried out using a tablet PC (operating system: Windows 10; ASUS TransBook T100HA, ASUS, Taipei, Taiwan, China,). Participants wore noise‐canceling headphones and responded to all tasks according to the instructions of the computer. Each test included one practice trial, and individuals were instructed to respond each task after practicing.
Figure 1.
The Computer‐Based Cognitive Assessment Tool tasks. (a) Task 1 and 2: digit span forward and backward tasks. (b) Tasks 3 and 4: memory and recall of item names tasks. (c) Task 5: Stroop task. (d) Task 6: recognition of figures.
Tasks 1 and 2: Digit span forward and backward tasks
In task 1, participants were required to immediately recall a set of numbers that were presented in random order on the PC screen. In task 2, participants recalled the same set of numbers in reverse order. The number of digits gradually increased (3, 5, 7, 8). The maximum number of digits that a participant could recall in the correct order (forward or reverse) was recorded.
Tasks 3 and 4: Memory and recall of item names tasks
In task 3, participants were instructed to memorize 10 target words that were presented on the PC screen. Subsequently, 20 words, including 10 target and 10 distracter words, were displayed, and participants were required to select the 10 target words. This was repeated for two trials and the number of correct answers on the second attempt was recorded. Furthermore, participants were instructed to recall the 10 target words, after all other tests, and the total number of target words recalled was recorded.
Task 5: Stroop task
In task 5, the names of colors were displayed in an incongruent fashion (for example, “aka” [red in English] was written in blue font) on the PC screen. Participants were required to say the color of the font and not read the word, among four options (the correct answer for the above would be blue). The total number of correct answers for each of the 20 items in the task was recorded.
Task 6: Figure recognition task
In task 6, blocks were stacked sterically and displayed on the PC screen. Participants were asked to state the number of the blocks. We recorded the total number of correct answers for each of the eight items in the task.
Measurement of the MMSE
We used the Japanese version of the MMSE, which measures cognitive function in both clinical and research settings, to assess the validity of the CompBased‐CAT.14 This test was administered by trained and registered clinical psychologists. The MMSE and Comp‐Based CAT were assessed on the same day.
Other measurements
We included age at enrollment, sex, health status and educational level as demographic variables; body mass index and maximum gait speed as physical factors; the simplified Japanese version of the World Health Organization‐Five Well‐Being Index as a psychosocial factor;15 and the Tokyo Metropolitan Institute of Gerontology Index of Competence as an instrumental activity of daily living factor.16 We also assessed the independence in carrying out five basic activities of daily living (bathing, dressing, walking, eating and continence).
Statistical analysis
CompBased‐CAT scores were converted into z‐scores. We calculated the total score for each individual as the CompBased‐CAT score. We used Pearson's correlation coefficient to compare the CompBased‐CAT and MMSE scores. Student's t‐test and Pearson's χ2‐test were used to compare the characteristics between individuals with and without cognitive impairment in accordance with earlier criteria (cognitive impairment absent: MMSE score ≥24, cognitive impairment present: MMSE score <24).17 Furthermore, we used receiver operating characteristic (ROC) curve analysis to determine the cut‐off CompBased‐CAT score that was equivalent to an MMSE score <24. We calculated the cut‐off point using the Youden Index. The area under the ROC curve for the CompBased‐CAT score was used to measure its accuracy of discrimination between individuals with and without cognitive impairment. All analyses were carried out using IBM spss version 21.0 (IBM Japan, Tokyo, Japan). Statistical significance was set at P < 0.05.
Results
The characteristics of the participants are presented in Table 1. The mean age (SD) was 72.6 years (6.4 years), the range was 65–89 years and 441 participants were women (57.1%). The mean duration of schooling (SD) was 12.8 years (2.7 years), and the mean MMSE score (SD) was 28.3 (1.9). A total of 27 (3.4%) participants were assessed to have cognitive impairment. An overwhelming majority of participants (98.6%) could independently carry out activities of daily living. The original score and z‐score of the CompBased‐CAT are presented in Table 2. The z‐score of each item and total z‐score of participants with cognitive impairment (total z‐score: mean −6.04, SD 6.34) were significantly lower than those for participants without cognitive impairment (total z‐score: mean 0.19, SD 3.03). Table 3 presents the Pearson's correlation coefficient between the MMSE scores and each task component of the CompBased‐CAT, which ranged from 0.24 to 0.41 (P < 0.001). The correlation coefficient between the MMSE and CompBased‐CAT total scores was 0.51 (P < 0.001). Figure 2 shows the ROC curve of the total z‐score of the CompBased‐CAT, which discerns the presence of cognitive impairment (MMSE <24). The area under the ROC curve sensitivity and specificity were 0.85, 0.81 and 0.77, respectively.
Table 1.
Participants’ characteristics
| Variable | Mean (SD) or n (%) |
|---|---|
| Age (years) | 72.6 (6.4) |
| Women, n (%) | 441 (57.1) |
| Health status, n (%) | |
| Cerebral stroke | 48 (6.2) |
| Heart disease | 110 (14.2) |
| COPD | 11 (1.4) |
| Diabetes mellitus | 98 (12.7) |
| Depression | 32 (4.1) |
| Education (years) | 12.8 (2.7) |
| BMI (kg/m2) | 23.0 (3.4) |
| Max gait speed (m/s) | 2.1 (0.4) |
| WHO‐5 | 16.3 (5.0) |
| TMIG‐IC | 12.3 (1.2) |
| MMSE | 28.3 (1.9) |
BMI, body mass index; COPD, chronic obstructive pulmonary disease; MMSE, Mini‐Mental State Examination; TMIG‐IC, Tokyo Metropolitan Institute of Gerontology Index of Competence; WHO‐5, World Health Organization‐Five Well‐Being Index.
Table 2.
Comparison between the original scores and z‐scores of the Computer‐Based Cognitive Assessment Tool of participants with and without cognitive impairment
| Scale range | All | MMSE ≥24 | MMSE <24 | P‐value | |
|---|---|---|---|---|---|
| n = 773 | n = 746 | n = 27 | |||
| Digit span forward | 0–8 | 5.3 (1.4) | 5.4 (1.4) | 3.9 (1.6) | <0.01 |
| 0.04 (0.97) | −1.04 (1.16) | <0.01 | |||
| Digit span backward | 0–8 | 4.5 (1.5) | 4.5 (1.5) | 3.1 (1.4) | <0.01 |
| 0.03 (0.99) | −0.91 (0.92) | <0.01 | |||
| Memory of the item name | 0–10 | 8.9 (1.2) | 9.0 (1.1 ) | 7.0 (2.4) | <0.01 |
| 0.06 (0.90) | −1.55 (2.00) | <0.01 | |||
| Memory recall of the item name | 0–10 | 7.1 (3.4) | 7.2 (3.3) | 3.4 (3.9) | <0.01 |
| −0.04 (0.26) | −0.33 (0.30) | <0.01 | |||
| Stroop task | 0–20 | 2.3 (1.5) | 15.9 (6.0) | 10.0 (6.4) | <0.01 |
| 0.32 (9.83) | −9.22 (10.5) | <0.01 | |||
| Recognition of figures | 0–6 | 4.7 (1.6) | 4.7 (1.6) | 3.8 (1.8) | 0.02 |
| 0.03 (0.98) | −0.52 (1.10 ) | <0.01 | |||
| Total z‐score | NA | NA | NA | NA | |
| 0.19 (3.03) | −6.04 (6.34) | <0.01 |
The top row shows the original scores and the bottom row shows the z‐scores. NA, not available.
Table 3.
Pearson's correlation coefficients of the Mini‐Mental State Examination and z‐scores of each task component of the Computer‐Based Cognitive Assessment Tool
| Correlation coefficient | ||
|---|---|---|
| r | P | |
| Digit span forward | 0.35 | <0.001 |
| Digit span backward | 0.33 | <0.001 |
| Memory of the item name | 0.41 | <0.001 |
| Memory recall of the item name | 0.30 | <0.001 |
| Stroop task | 0.28 | <0.001 |
| Recognition of the figure | 0.24 | <0.001 |
| Total z‐score | 0.51 | <0.001 |
Figure 2.
Receiver operating characteristics curve of the total score of the discrimination ability of the Computer‐Based Cognitive Assessment Tool for assessing cognitive impairment (Mini‐Mental State Examination score <24). AUC, area under the curve; CI, confidence interval.
Discussion
The present study developed the CompBased‐CAT, and investigated its validity and accuracy for assessing cognitive impairment. We found a moderate correlation between the CompBased‐CAT total and MMSE scores. Furthermore, the CompBased‐CAT showed high sensitivity and specificity for detecting cognitive impairment.
The characteristics of the participants of the present study, which included MMSE scores, the prevalence of cognitive impairment, educational history, maximum walking speed and Tokyo Metropolitan Institute of Gerontology Index of Competence, were similar to other large‐scale cohort studies carried out in Japan.18, 19, 20 Therefore, the samples in this study were regarded as representative of community‐dwelling older individuals in Japan.
The original CompBased‐CAT score, z‐score of each item and total z‐score, which was calculated from the z‐score of each item, were significantly lower in those individuals with cognitive impairment than those without, indicating that CompBased‐CAT reflects cognitive function. As each domain of the CompBased‐CAT has a different range of scores, we used the z‐score for each item to calculate the total z‐score and to make comparisons between the domains. The scores of the item‐memory and Stroop tasks showed a remarkable difference between the cognitive impairment and non‐cognitive impairment groups. It was suggested that the decline in the scores of the item‐memory and Stroop tasks was pronounced in patients with cognitive impairment. This finding is consistent with those of earlier studies.21
The correlation coefficient between the z‐score of each domain and the MMSE score was approximately 0.24–0.41, whereas the correlation coefficient between the total z‐score and the MMSE score was the highest at 0.51, suggesting that the total z‐score is more effective in discriminating cognitive impairment than the z‐score for each domain alone.
Previous studies have reported the discrimination ability of computer‐based cognitive tests, with sensitivity ranging from 0.68 to 1.00, specificity ranging from 0.73 to 0.98 and the area under the ROC curve ranging from 0.80 to 0.99.22 The discrimination ability of the CompBased‐CAT was similar to these previous studies, which indicates that the CompBased‐CAT possesses validity and discrimination ability as a cognitive test.
Several computer‐based cognitive assessment tools were developed in earlier studies and showed a correlation with established neurocognitive measurements.23, 24, 25 However, the sample size of those studies was very small, and only included individuals who visited clinics or had no cognitive problems. Therefore, they might not be representative of community‐dwelling older individuals in Japan. In the present study, we recruited a large number of individuals with a wide range of MMSE scores that were highly representative of the general older adult population in the community, and showed that CompBased‐CAT has high validity in community‐dwelling older people in Japan. Furthermore, the scores reported in this study can be used for reference among community‐dwelling older individuals.
The CompBased‐CAT has several advantages over the MMSE. First, the CompBased‐CAT tasks are randomized by the PC; therefore, the CompBased‐CAT can eliminate any learning effect. Interestingly, an earlier study shows that there are few learning effects associated with randomizing problems.9 Second, the CompBased‐CAT is measured automatically by the PC; therefore, there is no instruction bias or discrepancy due to personal judgment26 and multiple personnel with specialized knowledge are not required to carry out the test. Third, the measurements are recorded, and the scores calculated automatically, which eliminates human error and reduces examination time, thus providing immediate results. In addition, the CompBased‐CAT enables us to administer the test and calculate the scores in 10–15 min. The MMSE reportedly takes approximately 10 min (not including the time required for scoring) if the examination is carried out by a skilled physician.27 More time is required when unskilled examiners carry out the examination or, especially, calculate the scores. Thus, the CompBased‐CAT requires approximately 10 min less than the conventional MMSE test to score and record results. Therefore, CompBased‐CAT is beneficial in terms of its time efficiency, and useful for evaluating a large number of participants during community health checkups while providing immediate feedback to the participants.
Finally, the MMSE primarily assesses memory and language function, whereas the CompBased‐CAT includes a task that assesses extensive cognitive function, which includes memory and language. Therefore, the CompBased‐CAT can measure cognitive function in a more comprehensive manner, which gives it greater utility compared with the MMSE.
The present study had the following limitations. First, our working definition of cognitive impairment was biased, as we defined it using the MMSE, and a definitive diagnosis of dementia was not established. The main purpose of this study was to determine the efficacy of the CompBased‐CAT for screening functional cognitive impairment; therefore, we considered the MMSE, which assesses cognitive function, to be an appropriate test for comparison. Second, we were unable to determine the reliability of the CompBased‐CAT. Earlier studies have shown that similar computer‐based cognitive tests have a high test–retest reliability.22 However, it is necessary to verify the reliability of CompBased‐CAT in the future. Finally, this was a cross‐sectional study, and a longitudinal study is necessary to examine the ability of the CompBased‐CAT to predict the onset of dementia.
In summary, we showed the validity and the discrimination ability of the CompBased‐CAT for assessing cognitive dysfunction. We also showed its utility for assessing community‐dwelling older individuals in Japan. We believe that using the CompBased‐CAT in a large‐scale cohort study or local health checkup could help prevent the onset of dementia by screening individuals with a high risk of dementia. This tool is available only in Japanese; thus, the development of an English version in addition to the verification of its validity and reliability are issues that need to be approached in the future. However, it is assumed that the need for evaluating cognitive function through simple tests will continue to increase in the countries that have aging populations. Therefore, the present study is relevant, as it shows the validity of computer‐based cognitive function tests in community‐dwelling older individuals.
Disclosure statement
The authors declare no conflict of interest.
Acknowledgements
We are grateful to the individuals who participated in this study. This work was supported by JSPS KAKENHI (grant numbers: 15K01469 and 16K01853) and Research Funding for Longevity Sciences (grant numbers: 28‐30 and 29‐42), from the National Center for Geriatrics and Gerontology, Japan.
Takahashi J, Kawai H, Suzuki H, et al. Development and validity of the Computer‐Based Cognitive Assessment Tool for intervention in community‐dwelling older individuals. Geriatr. Gerontol. Int. 2020;20:171–175. 10.1111/ggi.13836
References
- 1. Livingston G, Sommerlad A, Orgeta V et al Dementia prevention, intervention, and care. Lancet 2017; 390: 2673–2734. [DOI] [PubMed] [Google Scholar]
- 2. Prince MR, Albanese E, Wimo A, Ribeiro W, Ferri CP. The global prevalence of dementia: a systematic review and metaanalysis. Alzheimers Dement 2013; 9: 63–75.e2. [DOI] [PubMed] [Google Scholar]
- 3. Morley JE, Berg‐Weger M, Lundy J. Nonpharmacological treatment of cognitive impairment. J Nutr Health Aging 2018; 22: 632–633. [DOI] [PubMed] [Google Scholar]
- 4. The Gerontological Society of America . The Gerontological Society of America Workgroup on Cognitive Impairment Detection and Earlier Diagnosis. [Cited 29 June 2019.] Available from URL: https://www.geron.org/images/gsa/documents/gsaciworkgroup2015report.pdf), 2015.
- 5. Morley JE, Morris JC, Berg‐Weger M et al Brain health: the importance of recognizing cognitive impairment: an IAGG consensus conference. J Am Med Dir Assoc 2015; 16: 731–739. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Frisoni GB, Boccardi M, Barkhof F et al Strategic roadmap for an early diagnosis of Alzheimer's disease based on biomarkers. Lancet Neurol 2017; 16: 661–676. [DOI] [PubMed] [Google Scholar]
- 7. Strauss E, Sherman EMS, Spreen O. A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary, 3rd edn. Oxford, England: Oxford University Press, 2006. [Google Scholar]
- 8. O'Bryant SE, Humphreys JD, Smith GE et al Detecting dementia with the mini‐mental state examination in highly educated individuals. Arch Neurol 2008; 65: 963–967. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Nieuwenhuis‐Mark RE. The death knoll for the MMSE: has it outlived its purpose? J Geriatr Psychiatry Neurol 2010; 23: 151–157. [DOI] [PubMed] [Google Scholar]
- 10. Kera T, Kawai H, Hirano H et al Comparison of body composition and physical and cognitive function between older Japanese adults with no diabetes, prediabetes and diabetes: a cross‐sectional study in community‐dwelling Japanese older people. Geriatr Gerontol Int 2018; 18: 1031–1037. [DOI] [PubMed] [Google Scholar]
- 11. Kawai H, Ihara K, Kera T et al Association between statin use and physical function among community‐dwelling older Japanese adults. Geriatr Gerontol Int 2018; 18: 623–630. [DOI] [PubMed] [Google Scholar]
- 12. Gerton BK, Brown TT, Meyer‐Lindenberg A et al Shared and distinct neurophysiological components of the digits forward and backward tasks as revealed by functional neuroimaging. Neuropsychologia 2004; 42: 1781–1787. [DOI] [PubMed] [Google Scholar]
- 13. Yucel M, Pantelis C, Stuart GW et al Anterior cingulate activation during Stroop task performance: a PET to MRI coregistration study of individual patients with schizophrenia. Am J Psychiatry 2002; 159: 251–254. [DOI] [PubMed] [Google Scholar]
- 14. Tombaugh TN, McIntyre NJ. The mini‐mental state examination: a comprehensive review. J Am Geriatr Soc 1992; 40: 922–935. [DOI] [PubMed] [Google Scholar]
- 15. Awata S, Bech P, Koizumi Y et al Validity and utility of the Japanese version of the WHO‐Five Well‐Being Index in the context of detecting suicidal ideation in elderly community residents. Int Psychogeriatr 2007; 19: 77–88. [DOI] [PubMed] [Google Scholar]
- 16. Koyano W, Shibata H, Nakazato K, Haga H, Suyama Y. Measurement of competence: reliability and validity of the TMIG Index of Competence. Arch Gerontol Geriatr 1991; 13: 103–116. [DOI] [PubMed] [Google Scholar]
- 17. Tsoi KK, Chan JY, Hirai HW et al Cognitive tests to detect dementia: a systematic review and meta‐analysis. JAMA Intern Med 2015; 175: 1450–1458. [DOI] [PubMed] [Google Scholar]
- 18. Taniguchi Y, Kitamura A, Murayama H et al Mini‐Mental State Examination score trajectories and incident disabling dementia among community‐dwelling older Japanese adults. Geriatr Gerontol Int 2017; 17: 1928–1935. [DOI] [PubMed] [Google Scholar]
- 19. Watanabe Y, Arai H, Hirano H et al Oral function as an indexing parameter for mild cognitive impairment in older adults. Geriatr Gerontol Int 2018; 18: 790–798. [DOI] [PubMed] [Google Scholar]
- 20. Kugimiya Y, Ueda T, Watanabe Y et al Relationship between mild cognitive decline and oral motor functions in metropolitan community‐dwelling older Japanese: the Takashimadaira study. Arch Gerontol Geriatr 2019; 81: 53–58. [DOI] [PubMed] [Google Scholar]
- 21. Mitchell AJ, Beaumont H, Ferguson D, Yadegarfar M, Stubbs B. Risk of dementia and mild cognitive impairment in older people with subjective memory complaints: meta‐analysis、risk of dementia and mild cognitive impairment in older people with subjective memory complaints: meta‐analysis. Acta Psychiatr Scand 2014; 130: 439–451. [DOI] [PubMed] [Google Scholar]
- 22. De Roeck EE, De Deyn PP, Dierckx E et al Brief cognitive screening instruments for early detection of Alzheimer's disease: a systematic review. Alzheimers Res Ther 2019; 11: 21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Makizako H, Shimada H, Park H et al Evaluation of multidimensional neurocognitive function using a tablet personal computer: test‐retest reliability and validity in community‐dwelling older adults. Geriatr Gerontol Int 2013; 13: 860–866. [DOI] [PubMed] [Google Scholar]
- 24. Maruff P, Lim Y, Darby D et al Clinical utility of the cogstate brief battery in identifying cognitive impairment in mild cognitive impairment and Alzheimer's disease. BMC Psychol 2013; 1: 30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Inoue M, Jinbo D, Nakamura Y, Taniguchi M, Urakami K. Development and evaluation of a computerized test battery for Alzheimer's disease screening in community‐based settings. Am J Alzheimers Dis Other Demen 2009; 24: 129–135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Ridha B, Rossor M. The mini‐mental state examination. Practic Neurol 2005; 5: 298–303. [Google Scholar]
- 27. Deveugele M, Derese A, van den Brink‐Muinen A et al Consultation length in general practice: cross sectional study in six European countries. BMJ 2002; 325: 472. [DOI] [PMC free article] [PubMed] [Google Scholar]