Montreal Cognitive Assessment (MoCA) XpressO: Validation of a Digital Self-administered Cognitive Pre-screening Tool

Abstract

Background:

The need for cognitive testing is increasing with the aging population and the advent of new Alzheimer disease therapies. To respond to the increased demand, the XpressO was developed as a self-administered digital cognitive pre-screening tool that will help distinguish between populations of subjective and objective cognitive impairment according to the Montreal Cognitive Assessment (MoCA).

Methods:

This is a prospective validation study. XpressO is composed of tasks that assess memory and executive functions. It is validated compared to the digital MoCA as a gold standard. Out of 118 participants screened from the MoCA Clinic and a family practice clinic, 88 met inclusion criteria, 2 participants had missing data due to incomplete tasks, 86 participants were included in the analysis; the mean age was 70.34 years. A logistic regression model was built, and its accuracy was evaluated by the sensitivity, specificity, and Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC).

Results:

Analysis showed strong correlation between (1) XpressO memory tasks scores and the MoCA Memory Index Score (p-values<0.001), and between (2) XpressO sub-test scores and MoCA total score (p-values<.005). The AUC for predicting MoCA performance is 0.845. To classify individuals with normal and abnormal MoCA scores, two threshold values were introduced for the total XpressO scores: sensitivity of 91% at a cut-off of 72, specificity of 90% at a cut-off of 42, and an undetermined range in between.

Conclusion:

XpressO demonstrated high AUC, high sensitivity and specificity to predict cognitive performance compared to the digital MoCA. It may provide efficient cognitive pre-screening by identifying individuals who would benefit from further clinical assessments, potentially reducing waiting times and high burden on healthcare clinics.

Registration:

ClinicalTrials.gov Identifier: NCT05879562

Introduction

The increasing prevalence of dementia, and awareness for the importance of early diagnosis, adds to overwhelming demand for cognitive screening. Potential treatments for cognitive decline are beneficial when provided in early stages¹. Individuals with subjective cognitive impairment (SCI) are at higher risk for future cognitive decline² though often remain cognitively intact³. A brief cognitive pre-screening tool will allow rapid identification of the population with SCI, distinguishing them from those with objective cognitive impairment, who may require intervention. Self-administered cognitive screening tools are not new to the medical field: a recent systematic review reported 21 digital cognitive screening tools that are self-administered⁴. Among these 21 digital tools, only 7 were reported to be free and available on the web. However, these tests are not commonly used. A pre-screening test that correlates with the Montreal Cognitive Assessment (MoCA)⁵ might be used more commonly to distinguish cognitive impairment from subjective cognitive concerns of patients. XpressO was developed (by ZN and YG) to serve as brief digital cognitive pre-screening, self-administered through a tablet device, phone, or desktop. XpressO is unique by offering clinicians predictive measures of cognitive performance compared to the MoCA as gold standard. A Delphi Global Consensus in 2023 ranked MoCA highest among 53 tools as a measure of cognitive abilities in clinical practice⁶. XpressO was designed to differentiate between individuals with intact cognition (MoCA score≥25/30) and individuals with cognitive impairment (MoCA score<25/30). Our hypothesis is that XpressO is predictive of MoCA score.

ClinicalTrials.gov identifier: NCT05879562.

Methods

Study Design.

This is a validation study with a randomized design where participants complete both the XpressO and MoCA tests in a randomized order.

Participants.

118 participants were recruited. Ten participants did not complete the study due to impaired comprehension. Additional 20 participants were excluded from analysis due to missing data: skipped/uncompleted tasks (n=11), technology interference e.g., wifi or software interruptions (n=5), or receiving assistance (n=4). Eighty-eight participants were included in the primary analysis. Table S1 describes demographic characteristics. Eligibility criteria included: age ≥50 years; formal education ≥6 years; fluency in English/French; recent completion of MoCA<3 months; no history of MoCA score ≤11/30. Experience with tablets was not a requirement for the study, however data was collected on their technology experience (self-reported) to explore possible confounding effects for lack of experience with tablets. Ethics Committee approval was obtained; all participants provided informed consent for the study.

Setting.

Participants were enrolled at the MoCA clinic and a family practice clinic between October 2022 – January 2023. This study conformed to the Standards of Reporting Diagnostic Accuracy Studies (STARD) 2015 Statement⁷.

Digital MoCA.

The digital MoCA⁸ is identical to the paper MoCA⁵ rather administered on a tablet device. It was administered and rated by trained research staff in either English/French according to the participant’s preference.

XpressO.

In this study, XpressO was completed on an 11-inch Android tablet without use of a stylus, independently by the participant, in a quiet room at the MoCA clinic or the family practice clinic. XpressO Includes 3 tasks that repeat in 3 variations, to a total of 9 tasks. It evaluates memory recall, executive functions, and processing times. Full details on the development of XpressO tasks and demonstrations of XpressO are available in the supplementary material and in the following link: https://mocacognition.com/digitaltools/. Participants were informed that each task is timed. Participants who required assistance were excluded from analysis for maintaining XpressO as self-administered, assuring a similar time interval for the delayed recall task and avoiding impact on the total processing time.

Intervention.

The order of administration for MoCA versus XpressO first/second was randomized, to account for potential bias. They were administered consecutively. The total MoCA score was dichotomized into a binary variable, defined as “1” for MoCA score ≤ 24/30 and “0” otherwise. This cut-off was used based on a recently suggested optimal range between 24–26^8,9.

Statistical Analysis.

Demographic characteristics were described using means, standard deviations, frequencies, and percentages (Table S1). We described and evaluated the correlation between: pairwise XpressO sub-test results and binary MoCA scores, XpressO memory task scores and MoCA Memory Index Score (MIS), and XpressO logical task scores and MoCA Executive Index Score (EIS), using t-tests for continuous variables. Standardized Mean Differences (SMDs) for all pairwise comparisons were calculated. The predictive model was built based on logistic regression. Composite proportion predictors based on XpressO sub-test results were calculated. For the memory recall tasks, predictors included the proportion of correct objects among the placed objects, and the proportion of both correct objects and positions among placed objects. For logical tasks, predictors included the proportion of 100% correct objects (for type, size, sequence, and color) among placed objects. Two additional participants with no placed objects were excluded from the model-building process because of missing data in XpressO memory tasks, reducing the sample size to 86. To assess the performance of the selected predictive model on unseen data, the original data were randomly split into a training set (70%) and a testing set (30%). With the training set, 10-fold cross-validation was used to select predictors and build the predictive model. The model’s performance was evaluated using the testing set to provide an independent assessment of how well the model generalizes to unseen data. The Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) was used to evaluate the predictive performance of all candidate logistic models. The final predictive model was then built on the entire dataset.

Definition of XpressO score.

To scale the predicted probabilities for XpressO score from 0 to 100, we set the minimum total duration to 100 seconds, maximum number of entries per memory task to 6, and maximum number of entries per logical task to 10. With these values, the outcomes of the logistic regression, i.e., the predicted probability of a participant having a MoCA score ≤24/30, were scaled on a range from 0–100. Two cut-off points and an intermediate indeterminate zone for XpressO were used⁹. Our approach is based on two cut-off points for the predicted probabilities of the final logistic regression to achieve either high sensitivity or high specificity.

Results

Descriptive Analysis Results.

Participants (n=88) had a mean age of 70.34 years with a standard deviation (SD) of 7.23 years. Most of the participants were female (57%). Participants were recruited from two clinics: a Family clinic (n=55) and the MoCA clinic (n=33) (Supplementary Table S1). The highest probability of intact cognition (MoCA score ≥25/30) is for individuals with XpressO scores ≥72, and the highest probability of cognitive impairment (MoCA score ≤24/30) is for individuals with XpressO scores ≤42, creating a grey zone for XpressO scores between 43–71, which predict MoCA scores between 24–26/30. Table 1 describes participants’ XpressO sub-test results among high and low MoCA scores. P-values<0.05 are considered statistically significant, and SMD>0.5 is considered medium-high effect size. The mean MoCA score of all participants was 23.5/30 (SD=3.8). The association between the binary MoCA score and most of XpressO sub-test results, were statistically significant with SMDs in the medium-high range. XpressO memory task scores were highly associated with MoCA MIS¹⁰ (p-values<0.001); XpressO logical task scores were highly associated with MoCA EIS¹⁰ (p-values<0.03). Associations between XpressO processing times and MoCA scores are summarized in Table 1.

Table 1.

XpressO sub-test results of participants by binary MoCA score

	MoCA ≤ 24	MoCA ≥ 25	P values	SMD
Sample size	46	42
Memory Tasks: Average number of objects placed (mean (SD))	4.69 (0.75)	4.83 (0.50)	0.317	0.217
Memory Tasks: Average number of same objects (mean (SD))	4.23 (1.01)	4.71 (0.52)	0.008*	0.591
Memory Tasks: Average number of same objects and position (mean (SD))	3.56 (1.35)	4.33 (0.72)	0.001*	0.709
Logical Tasks: Average number of objects placed (mean (SD))	6.91 (1.36)	6.81 (1.97)	0.789	0.057
Logical Tasks: Average number of 100% correct answers (mean (SD))	3.79 (2.14)	5.56 (2.17)	<0.001*	0.820
Logical Tasks: Average number of correct colors (mean (SD))	5.21 (1.73)	6.19 (1.95)	0.014*	0.532
Total processing time in seconds (mean (SD))	463.30 (197.04)	377.93 (103.60)	0.014*	0.542
Average processing time of memory tasks in seconds (mean (SD))	57.63 (24.29)	49.32 (15.17)	0.060	0.411
Average processing time of logical tasks in seconds (mean (SD))	39.17 (21.52)	27.34 (8.66)	0.001*	0.721
Total drags (mean (SD))	5.52 (0.48)	5.73 (0.50)	0.050	0.424
Average drags of memory tasks (mean (SD))	6.13 (0.65)	6.30 (0.83)	0.302	0.220
Average drags of logical tasks (mean (SD))	4.91 (0.79)	5.17 (0.71)	0.118	0.337

Abbreviations: SD, standard deviation; SMD, standardized mean difference.

Note:

^*p-value < 0.05 indicates a statistically significant difference. SMD values > 0.5 are considered medium to large effect sizes.

Predictors of the logistic regression model.

The predictors included in the final predictive logistic regression include: total processing times (seconds), proportion of correct objects among placed objects in memory tasks, proportion of 100% correct objects (for type, size, sequence, and color) among placed objects in logical tasks, average entries in memory tasks, and average entries in logical tasks. The AUC was 0.828 on the training set and 0.827 on the testing set, demonstrating good predictive performance of both training and testing datasets. For the final predictive logistic regression model built on the entire dataset, the AUC was 0.845 (Figure 1).

Figure 1.

The receiver operating characteristic curve (ROC) Curve, area under the ROC curve (AUC), sensitivities, and specificities with two cut-off points.

Two cut-off points and an indeterminate zone.

At XpressO cutoff point of 72, sensitivity was 91.3%, indicating that 8.7% of participants with true cognitive impairment are predicted to present MoCA scores ≥25/30. At XpressO cutoff point of 42, specificity was 90%, implying that 10% of participants with true intact cognition are predicted to present MoCA scores ≤24/30. XpressO scores between 43–71, inclusive (43≤score≤71) constitute a gray zone that is indeterminate. Figure 1 shows the ROC curve and AUC values for the predictive logistic model based on analysis of 86 participants. Figure 2 displays the sensitivity and specificity across the total XpressO scores. Figure S1 in the Supplemental Material displays a box plot of MoCA scores among XpressO score groups. Additionally, XpressO was shown to predict the MoCA range regardless of the participants’ years of formal education. Analysis for the effect of education, the effect of randomization order, and of processing times are included in the Supplemental Material.

Figure 2.

Sensitivities and specificities across total XpressO scores. The two dashed vertical lines denote the two separate cut-off points. Scores between the two points are indecisive.

Discussion

This validation study found XpressO, a brief cognitive screening test, to have high accuracy in predicting specific scoring ranges of the MoCA, as indicated by the AUC (0.845), high sensitivity (91%) and specificity (90%) at XpressO cut-off points of 72 and 42, respectively. Key additional findings were significant associations of XpressO memory and logical subtests and those of MoCA MIS and EIS respectively, which allows identifying amnestic cognitive impairment with high accuracy.

Advantages of XpressO include self-administration, availability on various digital devices, and the non-verbal nature of all XpressO tasks, providing simpler use regardless of the patient’s first language or level of education. XpressO was shown to predict MoCA range score regardless of the participants’ formal years of education, potentially due to the design of all XpressO tasks being non-verbal. Moreover, the use of XpressO is intended to be free of charge for non-commercial purposes. However, the test has few limitations: although XpressO identifies impaired cognition, it does not specify the level of cognitive impairment (e.g., mild, or severe). Additionally, it requires technology access, which may be limited for socioeconomically disadvantaged individuals or those who require technological assistance. Also, being self-administered, XpressO does not permit caregiver assistance.

Strengths and Limitations of the Study

Strengths.

The study sample included similar proportions of different MoCA scores (Figure S1), aiming to reflect a broad range of cognitive functions. Additionally, the study population included patients with low levels of formal education, and explores the specific association of XpressO memory tasks to MoCA MIS, and XpressO logical task to MoCA EIS.

Limitations.

The study identifies the range of cognitive level, and not specific scores; the range of XpressO scores between 43–71 is indeterminate: high sensitivity and specificity are achieved only for XpressO results ≤42 or ≥72 points. In addition, the sample size of the study was relatively small, and confirmation of results will require investigation with larger and more diverse populations, including a broader range of racial and ethnic backgrounds. Although we used random data splitting and 10-fold cross validation to select a predictive model and evaluate overfitting, given the relatively small sample size, the results may still depend on the specific random split used in the initial train-test split. Participants randomized to complete the MoCA before the XpressO had significantly higher scores on MoCA (MoCA average score of 24.6/30, SD 3.93 versus MoCA average score of 22.5/30, SD 3.48 respectively; p-value = 0.008). The lower scores on the MoCA as the second test may suggest cognitive fatigue. However, the order of randomization did not impact the XpressO scores, and this exploratory analysis was found to improve the prediction model accuracy.

Conclusions

XpressO is a brief self-administered cognitive screening tool with high accuracy in predicting scoring ranges on the MoCA. Thus, cognitive pre-screening with XpressO may distinguish mild cognitive impairment from normal cognition and facilitate early diagnosis and intervention. Another potential benefit may be reducing the demand and resources needed for further cognitive screening. Future studies of XpressO should be conducted in larger samples and more diverse populations.

Key points:

• In a sample of 88 participants, we validated a self-administered tablet-based cognitive pre-screening tool known as XpressO, by comparing it to the MoCA test as the gold standard.

• XpressO is a brief pre-screening tool that evaluates three cognitive domains: memory, executive function, and processing time.

• The operating characteristics of the XpressO support its validity as a self-administered cognitive screening tool in older adults compared to the full digital MoCA. This research suggests that XpressO holds significant promise for clinical care by introducing a convenient brief cognitive pre-screening tool.

Why does this paper matter?

The potential impact of this research on clinical care includes introduction to brief self-administered cognitive pre-screening XpressO that predicts the MoCA Score. This may help to distinguish mild cognitive impairment from normal cognition and facilitate early diagnosis and intervention. In addition, it may reduce the demand on healthcare specialists and resources needed for further cognitive screening.