A preliminary study of the reliability and validity of the Uyghur version of the NUCOG cognitive screening application

Abstract

Introduction

Technological advances and artificial intelligence now make it feasible to administer cognitive assessments on touch-screen devices. The aim of this study is to develop a Uyghur version of the NUCOG cognitive screening application and evaluate its reliability, validity, and optimal cutoff scores among Uyghur people with cognitive impairment.

Methods

The English version of the NUCOG app was translated and adapted into the Uyghur version (NUCOG-U). A total of 250 Uyghur people aged 55–80, including 90 normal controls, 91 patients with mild cognitive impairment (MCI), and 69 dementia patients, were randomly selected and administered with NUCOG, MoCA-U, Mini-Mental State Examination (MMSE), and other neuropsychological batteries. ROC curves were generated to determine the optimal cutoff values.

Results

NUCOG-U version showed high internal consistency (Cronbach’s α = 0. 826), inter-rater reliability (ICC = 0.999), and test − retest reliability (r = 0.998, p < 0.001). NUCOG scores were significantly correlated with those of MoCA-U (r = 0.896, p < 0.001) and MMSE(r = 0.899, p < 0.001). NUCOG scores were significantly different among the three groups (p < 0.001). The optimal cutoff value for MCI was 80.5, with a sensitivity of 100% and specificity of 73%, and 70 for dementia, with a sensitivity of 94.1% and specificity of 100%.

Conclusion

The NUCOG-U shows high reliability and validity and is suitable for screening cognitive function in the elderly Uyghur population. The optimal cutoff scores to detect mild cognitive impairment and dementia in the Uyghur people are 80.5 and 70, respectively.

Introduction

The aging population is facing increasing challenges in mental health, with dementia syndromes such as Alzheimer’s disease imposing a heavy burden on families and society. Early detection and intervention of cognitive impairment are crucial in preventing or delaying the progression of dementia [1, 2]. Therefore, the development of simple, quick, and efficient cognitive screening tools is of utmost importance. With the advancement of Information and Communication Technologies (ICT) and Artificial Intelligence (AI), clinicians are showing a demand for portable intelligent electronic devices in the field of cognitive screening. In recent years, studies have shown that electronic cognitive screening tools offer several advantages over traditional paper-based versions, such as standardization of operations, automatic data collection and calculation, reduction of human errors, accurate measurement of potential factors in responses, automatic comparison with age-related mean values in databases, as well as saving manpower and costs [3].

The Neuropsychiatry Unit Cognitive Assessment Tool (NUCOG), which was developed by Walterfang, is a simple, quick, and easy-to-administer cognitive screening tool [4]. Compared to the Mini-Mental State Examination (MMSE), which is widely used for screening cognitive function, NUCOG has a higher sensitivity and broader coverage for cognitive testing, especially for early Mild Cognitive Impairment (MCI) patients who usually perform in the normal range of MMSE [4]. It has been translated into several languages, including Malay, Persian, and Chinese [5–7]. Several studies have demonstrated the tool’s reliability and validity in screening for cognitive impairment caused by a variety of conditions, with good sensitivity and specificity [4, 7]. NUCOG was translated into Chinese in 2014 [7]. In 2016, Walterfang developed an intelligent application software version of NUCOG for use on iPads. However, since memory and language tests are closely correlated with language habits, characters, and even living style, the efficiency of the NUCOG application will be affected by the culture, language, society, economy, and education level. China is a multi-national country where each nationality possesses a unique cultural background and language. The socioeconomic conditions and education levels are largely different among regions. Therefore, the Chinese version of NUCOG does not necessarily apply to minorities in China. Regarding the different cultural spans, the NUCOG should be appropriately adjusted to formulate versions that are suited to specific regions or nationalities. Uyghur is spoken in northwestern China by at least 10 million native speakers [8]. In this study, we translated the NUCOG tool into the Uyghur version of the NUCOG software (NUCOG-Uyghur version, NUCOG-U) with some modifications. The reliability and validity of NUCOG-U were tested in Uyghur people in Xinjiang Urumqi of China and the optimal cutoff value (COV) was estimated.

Materials and methods

Study subjects and ethical considerations

Patients were selected from the Neurology Department of Xinjiang People’s Hospital (the largest general hospital in Xinjiang), and residents were selected from local communities (both rural and urban) between October 2022 and October 2023. The average age was 67.3 ± 6.6 years (age range 55–80). All the participants were native Uyghur people and volunteered to take part in the study. Patients were excluded if they had any of the following: (1) advanced cancer, stroke, active epilepsy, depression, or a history of mental illness; (2) disease that significantly affects cognitive assessment (e.g. hypothyroidism); (3) Impairments in hearing, vision, physical activity, or any other communication problems that may impact test completion. A total of 250 participants were included, divided into three groups: normal control (NC, n = 90), MCI (n = 91), and dementia (n = 69). Participants with dementia in this study included patients with Alzheimer’s disease and vascular dementia.

The enrollment criteria for NC were: No or mild memory impairment; no objective cognitive impairment; normal range in MMSE score; Clinical dementia rating (CDR) = 0; Global deterioration scale (GDS) = 1; early Alzheimer’s disease test (AD8) = 0; Activity of daily living is normal, Activity of daily living scale (ADL) < 18.

Enrollment for MCI patients was based on Petersen’s criteria [9]: (1) Complaint of memory impairment (confirmed by a family member or a witness); (2) Objective memory impairment not explained by age or education level; (3) Generally normal cognition; (4) Activities of daily living retained; (5) Not meeting the criteria of dementia; (6) Excluding memory impairment induced by other causes. Candidates met the following characteristics: MMSE was > threshold of dementia test; CDR = 0.5; GDS = 2–3; ADL < 18; AD8 = 1–2.

Enrollment for dementia was based on the Diagnosis and Statistics Manual for Mental Disorders (DSM-V) and also included case history and imaging results [10]. In this study, MMSE was ≤ threshold of dementia test; CDR = 1–2; GDS = 4–6; ADL may be normal or abnormal; AD8 > 2.

This study was approved by the Xinjiang People’s Hospital Ethics Committee in China (equivalent to an Institutional Review Board). It conformed to the Declaration of Helsinki, and written informed consent was obtained from all participants or their next of kin. Participants were informed and consented to their answers being used for this study’s research aims.

Translation of NUCOG

With the authorization of the original authors’ team for the application, the original English version of NUCOG was translated into Uyghur with some modifications by a neurological clinician who is fluent in both Uyghur and English. The NUCOG-U was then revised and corrected by a Uyghur expert and then translated back to English by another expert familiar with both Uyghur and English but blind to NUCOG. The meanings of the back-translated English version were similar to the original English version. Twenty normal Uyghur people aged 50 or older, coming from diverse educational and cultural backgrounds, were chosen to pre-test NUCOG-U. Based on their feedback, minimal adjustments were made, and the final version of NUCOG-U was established. It included five cognitive domains: attention, visuospatial ability, memory, executive function, and language. The total score was 100, with each domain assigned 20 points and scored separately. The software version automatically calculated scores and time to completion, comparing them with the participant’s previous scores and the age-related means in the database. Additionally, results were automatically sent via email to the participant and the researcher.

Modifications to the final version of NUCOG-U included: (1) Memory: to account for the cultural background and word frequencies of the Uyghur language, the word “guitar” in the English version was changed to “dutar” in Uyghur (a traditional musical instrument). (2) Executive function: considering that direct translation of foreign terms may affect accuracy, the word “pineapple “ in the original version was changed to “pear” in Uyghur. (3) Language: to ensure that word frequency and compound words remained consistent with the original version, the words “banana”, “artillery”, and “constitutional” in the English version were changed to “strawberry”, “sunflower,” and “wooden bed” in Uyghur, respectively; and regarding the grammatical features of the Uyghur language and the customs, the writing sentence “The boy’s aunt made a large pie out of steak and dough” was modified in syntax while the structure and basic meaning remained intact.

Survey tools

MoCA [11]: this tool includes visuospatial abilities, memory, concentration, language fluency, delayed recall, and orientation domains. In this study, the Uyghur version of MoCA was used after obtaining permission from the tool’s author, and the optimal cutoff values for the Uyghur population were applied.

MMSE [12]: includes time, location, orientation, language, short-term memory, calculation and reading comprehension, and picture description. The maximal score is 30. The cutoff value (COV) for dementia in this study was illiteracy ≤ 17, elementary school education ≤ 20, middle school and higher education ≤ 24. The MMSE does not require special permissions since the licensing period has expired.

CDR [13]: includes memory, orientation, judgment and problem-solving, community affairs, home and hobbies, and personal care. The score employs a uses a five-point scale. COVs were: none = 0, questionable = 0.5, mild = 1, moderate = 2, and severe = 3.

ADL [14]: fourteen ADL items were used in the present study, covering instrumental and physiological activities of daily living. Each item consists of 4 grades. An individual was considered as having “ADL impairment” if scores of at least two items were 3 or 4.

GDS [15]: used to evaluate dementia and includes seven grades. Values between 1 and 3 were categorized as pre-dementia and 4–7 as dementia.

AD8 [16]: includes eight questions for cognitive evaluation. Each “yes” answer receives one point. Scores ≥ 2 suggest cognitive impairment.

Survey methods

A double-blind method was used to evaluate participants with NUCOG-U, MoCA-U, MMSE, CDR, GDS, ADL, and AD8. Each individual was tested independently in a quiet room by two trained clinicians who had received consistency training. Experienced clinicians grouped the participants based on diagnosis standards, case history, and other assessments except NUCOG-U. After two weeks, 20 participants were selected randomly and re-tested using NUCOG-U.

Statistical analysis

SPSS 24.0 (SPSS Inc., US Chicago) was used to analyze data. The average age, education, and NUCOG-U total score of the three groups were analyzed by analysis of variance (ANOVA). The sex factor was analyzed using the χ²test. Correlation between NUCOG-U and other assessments was tested using Pearson and receiver operator characteristic curve (ROC), and corresponding sensitivity, specificity, and optimal cutoff value were generated. p < 0.05 was considered significant.

Results

The basic sociodemographic characteristics and cognitive screening results of participants are presented in Table 1.

Table 1.

General information of participants

Item		NC	MCI	Dementia	F/χ2	p
Number		90	91	69	-	-
Age	(Year ± SD)	66 ± 5.9	68 ± 5.3	68 ± 7.9	1.3	0.501
Sex	M(%)	15(50)	16(51.6)	18(46.2)	0.223	0.894
	F(%)	15(50)	15(48.4)	21 (53.8)
Education	(Year ± SD)	12 ± 2.8	11 ± 2.1	8 ± 3.5	19.1	< 0.001
MoCA-U	(± SD)	24 ± 1.2	19 ± 2.3	9 ± 5.4	158.1	< 0.001
MMSE	(± SD)	27 ± 1.2	24 ± 1.2	15 ± 5.1	113.7	< 0.001
NUCOG-U	(± SD)	88.4 ± 2.7	80.9 ± 7.2	42.2 ± 20.9	114.3	< 0.001
Visuospatial	(± SD)	19.7 ± 0.6	17.5 ± 2.5	9.3 ± 3.8	20.8	< 0.001
Executive	(± SD)	16.3 ± 1.3	13.8 ± 3.2	6.4 ± 4.5	12.3	< 0.001
Attention	(± SD)	17.3 ± 1.2	14.1 ± 1.2	9.1 ± 4.2	10.1	< 0.001
Memory	(± SD)	15 ± 1.8	14.1 ± 4.6	6.3 ± 3.7	11.1	< 0.001
Language	(± SD)	19.7 ± 0.6	18.7 ± 1.3	12.9 ± 5.9	5.0	0.019

There was no significant difference (p > 0.05) in age or sex among the three groups.

Internal consistency was tested using Cronbach’s α. The results showed that Cronbach’s α of NUCOG-U is 0.826, suggesting high internal consistency. Twenty people were randomly selected after 11.7 ± 2.6 days of the first test and re-evaluated with NUCOG-U. The variation of the total NUCOG-U score was 0.8 ± 0.3, and the correlation coefficient of the two tests was 0.998 (p < 0.001), suggesting high test-retest reliability.

Intraclass correlation coefficient (ICC) was used to test the reliability of the two raters and showed high inter-rater reliability ICC = 0.999 (95%CI: 0.999—1.000). Total NUCOG-U scores from the NC, MCI, and dementia groups were 88.4 ± 2.7, 80.9 ± 7.2, and 42.2 ± 20.9, respectively. and the difference was significant (p < 0.001). Then, the three groups underwent an LSD-t test for between-group comparison, and the results were all significantly different (p < 0.001). suggesting that NUCOG-U can be used to differentiate the three groups. The dementia group received the lowest score, MCI was in the middle, and NC patients had the highest scores. The differences among the three groups included visuospatial abilities, executive function, concentration, and memory, suggesting a good discriminating validity.

MMSE and MoCA-U scales were selected as criteria, and their correlations with NUCOG-U were analyzed separately. Correlation analysis showed that the correlation coefficient was 0.899 (p < 0.001) with MMSE and 0.896 (p < 0.001) with MoCA-U, suggesting high criterion-related validity.

Based on the MCI and DSM-V diagnostic criteria as the gold standards, we plotted the ROC curves between the NC and MCI groups and between the dementia and MCI groups. The area under the curve (AUC) for MCI and dementia were 0.934 (95% CI:0.889 ~ 0.978) and 0.981 (95%CI: 0.962-1.000), respectively, suggesting that NUCOG-U has capacity for screening the above patients. The optimal cutoff value for MCI was 80.5, with a sensitivity of 100% and specificity of 73%, and 70 for dementia, with a sensitivity of 94.1% and specificity of 100%.

Discussion

In this study, we translated the NUCOG app into the Uyghur language and determined the optimal cutoff score to detect mild cognitive impairment and dementia in the Uyghur people. Early detection and intervention for dementia can significantly improve the quality of life for patients and reduce associated socioeconomic costs [17]. Early detection relies on the age at which initial cognitive screening is conducted, the frequency of screenings, and the specificity of the screening tools [18]. MoCA has a higher sensitivity; however, it does not provide clinicians with detailed performance across various cognitive domains. In contrast, NUCOG assesses overall cognitive function and provides specific scores for each cognitive domain, thus enhancing the breadth of information available for clinical diagnosis [4].

With the advancement in medicine and technology, computer-aided technologies under artificial intelligence are increasingly being utilized in clinical practice [19]. In recent years, researchers found that AI and computer-aided systems have shown certain effectiveness in the rehabilitation and diagnosis of cognitive impairment [20–24]. The development of AI technology can effectively address the uneven distribution of healthcare resources between urban and rural areas, especially in developing countries and remote areas [25].

The Uyghurs are a Turkic ethnic group living in Eastern and Central Asia. Today, Uyghurs live primarily in Xinjiang Uyghur Autonomous Region in China, with a large population of 11,624,300 (2020). The Uyghur language, which belongs to the Turkic language family, is an agglutinative language and has a subject-object-verb word order. The written language is the reformed Uyghur Arabic alphabet. However, there are no reports about the evaluation of cognitive function for Uyghur people living in Xinjiang, particularly on evaluation using the application of AI and computer-aided technologies. Therefore, it is necessary to develop the Uyghur version of the NUCOG app. When NUCOG is applied to a specific population or region, and while maintaining basic consistency with the original version, the optimal cutoff value should be identified based on the cultural, language, and text characteristics. In the clinic, ignorance of cultural differences or direct translation of literal meaning will reduce the accuracy of test results. Considering the differences in culture and language in Uyghur people, we translated NUCOG into NUCOG-U with flexibility rather than literally to keep the same content, structure, and word frequency as the English version. Then NUCOG-U was translated back to English, pre-tested, and developed into the final version of NUCOG-U. The translation quality was controlled by a group of Uyghur language experts. Therefore, NUCOG-U is an equivalent of NUCOG and is more applicable to the Uyghur population. Most participants in this study finished the test in 15 min and were able to cooperate and respond actively, suggesting that NUCOG-U is applicable to the Uyghur population.

The Cronbach’s α of NUCOG-U is 0.826, which is lower than 0.915 of the original English version [4], 0.922 of the Chinese version [7], and 0.895 of the Malay version [5], but nonetheless higher than 0.7, suggesting a high internal consistency. The test-retest reliability is 0. 998. Consistency between raters is 0.999, higher than 0.91 of the original English version, suggesting that adequate staff training is necessary prior to the study.

Criterion-related validity was tested not by MMSE alone but by a combination of MoCA and MMSE. The results showed that NUCOG-U has higher correlation and criterion-related validity with MMSE and MoCA. There was a significant difference among the three groups of NUCOG-U score, indicating good criterion validity and discriminative validity.

Although there were differences in education years among the three groups, there was no significant difference between the NC and the MCI groups (F = 4.82, p = 0.07). Therefore, in this study, NUCOG-U can be used as a screening tool to distinguish between cognitively normal individuals and those with MCI, effectively excluding the influence of educational level. This finding differs from the results of the Chinese version validation study, where all groups were affected by educational background. The sub-items with significant differences in average scores among the three groups included visuospatial ability, executive function, attention, and memory, similar to the original English version. The English and Chinese versions suggest that NUCOG has a certain ability to differentiate between different types of cognitive impairment compared to MMSE. However, further evaluation of cognitive impairment caused by different diseases still needs to be combined with clinical assessments and other neuropsychological evaluations. This study did not group participants based on different types of cognitive impairment.

Different COVs in NUCOG-U correspond to varying sensitivity and specificity figures. In this study, we chose the optimal COV at the maximum of the Youden index. The results showed that with a cutoff score of 80.5, NUCOG-U had a sensitivity of 100% and a specificity of 73%, higher than 80 of the original version and 78.5 of the Malay version. This study included patients with MCI, while the original version and the Malay version primarily focused on screening dementia patients.

AUC reflects the accuracy of the test. Generally, the diagnosis accuracy is considered higher with AUC > 0.9. The AUCs for diagnosis of MCI and dementia using NUCOG-U were 0.934 and 0.981, respectively, suggesting that NUCOG-U has higher accuracy in the MCI and dementia screening test.

However, this study has some limitations: It was conducted only among the Uyghur population aged 55–80 in the Urumqi region and participants were recruited from hospital visitors. Therefore, selection bias cannot be ruled out. A larger sample size and more in-depth research are needed to determine whether the results and current cutoff values are suitable for other regions and different age groups, and the illiterate population within the Uyghur population. The cover of optimal COV in this study is not broad. Further research is required to determine the optimal COV across both sexes, various age groups, and different educational backgrounds, especially when comparing individuals with dementia to those with MCI and normal cognition. Longitudinal follow-up is also required to assess NUCOG-U’s predictive validity over time.

Conclusion

The NUCOG-U version translated and validated in this study can be used to differentiate between normal cognition, MCI, and dementia in the Uyghur-speaking population. The optimal cutoff values for MCI and dementia in this population were determined to be 80.5 and 70, respectively, using ROC curve analysis.

Author contributions

NY: conceptualization, study design, software translation and production, and manuscript writing. AR: data collation, data analysis, and manuscript writing. LH: research management, supervision, writing the final manuscript.

Funding

This study was funded by the People’s Hospital of Xinjiang, Fund number 20200307.

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Declarations

Ethics approval and consent to participate

This study was approved by the Xinjiang People’s Hospital Ethics Committee in China (equivalent to an Institutional Review Board). It conformed to the Declaration of Helsinki, and written informed consent was obtained from all participants or their next of kin.

Competing interests

The authors declare no competing interests.