A virtual reality cognitive screening tool based on the six cognitive domains

Abstract

INTRODUCTION

Early detection of cognitive impairment enables interventions to slow cognitive decline. Existing neuropsychological paper‐and‐pencil tests may not adequately assess cognition in real‐life environments. A fully‐immersive and automated virtual reality (VR) system—Cognitive Assessment using VIrtual REality (CAVIRE)—was developed to assess all six cognitive domains. This case–control study aims to evaluate the ability of CAVIRE to differentiate cognitively‐healthy individuals from those with cognitive impairment.

METHODS

One hundred nine Asian individuals 65–84 years of age were recruited at a primary care setting in Singapore. Based on the Montreal Cognitive Assessment (MoCA), participants were grouped as either Cognitively Healthy (MoCA ≥26, n = 60) or Cognitively Impaired (MoCA <26, n = 49). Subsequently, all participants completed the CAVIRE assessment.

RESULTS

Cognitively‐healthy participants achieved higher VR scores and required shorter completion time across all six cognitive domains (all p’s < 0.005). Receiver‐operating characteristic curve analysis showed area under the curve of 0.7267.

DISCUSSION

The results demonstrated the potential of CAVIRE as a cognitive screening tool in primary care.

Highlights

• CAVIRE is a virtual reality (VR) system that assesses the six cognitive domains.

• CAVIRE can distinguish healthy individuals from individuals with cognitive impairment.

• It has potential as a cognitive screening tool for older people in primary care.

1. INTRODUCTION

Cognition refers to brain processes by which an individual becomes aware of the situation and required actions, and subsequently executes strategies for optimal living. ¹ Based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM‐5), cognition is categorized into six cognitive domains: complex attention; executive function; language; learning and memory; perceptual‐motor function; and social cognition. ²

Aging is associated with cognitive decline. ³ Cognitive decline may lead to cognitive impairment, where there is deficit in one or more cognitive domains, or subsequently dementia, where the deficits in cognition interfere with the individual's independence to perform daily activities. ²

Due to the aging population and increasing prevalence of dementia, ⁴ , ⁵ it is important to identify deficits in cognition early. Recent literature suggests that signs of cognitive impairment could appear as early as 9 years before the diagnosis of dementia. ⁶ There is currently no known medical cure for dementia; therefore early detection of cognitive decline could allow for various interventions to slow cognitive decline. ⁷ , ⁸ , ⁹

Early identification of cognitive impairment in primary care involves using neuropsychological paper‐and‐pencil screening tests, for example, the Mini‐Mental State Examination (MMSE) and Montreal Cognitive Assessment (MoCA). ¹⁰ However, these tests are unable to detect deficits across all six cognitive domains. For example, MMSE can assess memory, language, and attention, but is less effective in assessing executive function. ¹¹ More importantly, these tests lack in ecological validity; they are unable to adequately predict the individual's cognition in real‐world environments, and the testing conditions do not resemble the individual's daily surroundings. ¹² , ¹³

Virtual reality (VR) allows an individual to interact with a three‐dimensional (3D) computer‐generated environment via engagement of different senses. ¹⁴ VR has shown potential as a cognitive screening tool. As compared to paper‐and‐pencil tests, VR has better ecological validity. ¹⁵ VR can create multimodal sensory stimuli in an interactive manner, and combines the features of laboratory control with a simulation that mimics real‐world environments. ¹⁶

Nevertheless, existing VR systems for detecting cognitive changes have limitations. For example, VR systems that utilize a two‐dimensional computer screen ¹⁷ create a less‐immersive testing environment for the user, where the user may be distracted by external stimuli. Moreover, many existing VR cognition tests assess a limited number of cognitive domains, but not all six cognitive domains. ¹⁸ , ¹⁹

We developed a fully‐immersive and automated VR system (CAVIRE) to assess all six cognitive domains. ²⁰ Results from a previous study indicated that the time taken to complete the CAVIRE assessment was significantly shorter compared to that of the MoCA. ²¹ Younger participants achieved significantly higher VR scores and required shorter time to complete the VR tasks, as compared to older participants. ²² Overall, the previous study demonstrated the feasibility of using CAVIRE in primary care for cognitively‐healthy individuals.

In this study, we aim to evaluate the ability of CAVIRE to differentiate older individuals (65‐ to 84‐years‐old) who are cognitively healthy from individuals with cognitive impairment. We hypothesize that cognitively‐healthy individuals will achieve higher VR scores and require shorter time to complete the VR tasks, as compared to the cognitively‐impaired individuals. This study also aims to evaluate the acceptability of using CAVIRE for participants. We hypothesize that most participants will find CAVIRE acceptable.

RESEARCH IN CONTEXT

Systematic review: Sources were reviewed from Google Scholar and PubMed. Existing screening tests for cognitive impairment have some limitations. CAVIRE is a virtual reality (VR) assessment based on the six cognitive domains. CAVIRE involves 13 different virtual environments consisting of tasks that simulate common daily activities. Previous studies demonstrated the feasibility of using CAVIRE for cognitively‐healthy individuals.

Interpretation: The current study involved individuals 65–84 years of age. Our findings show that CAVIRE can distinguish cognitively‐healthy individuals from individuals with cognitive impairment, based on the Montreal Cognitive Assessment (MoCA). No participant dropped out due to VR‐induced symptoms (nausea, headache, and/or giddiness). CAVIRE has the potential to be used as a cognitive screening tool among older individuals in the primary care setting.

Future directions: Further studies are required to establish the full psychometric properties of CAVIRE, including test–retest reliability, and validation of CAVIRE against clinical diagnosis by neurologists.

2. METHODS

This article presents the results from a sub‐analysis of a study protocol on using CAVIRE to assess the six cognitive domains. ²⁰

2.1. Study site

This case–control study was situated at a public primary care clinic in Singapore, from October 2020 to March 2022.

2.2. Study population

All participants fulfilled the following criteria: 65–84 years of age; understood English; and were agreeable to completing the study procedures. Individuals with any of the following were excluded from participation: pre‐existing diagnosis of cognitive impairment or dementia; neurological deficits that affect vision, hearing, speech, or arm movements; previous history of motion sickness or epilepsy; or inability to provide informed consent.

2.3. Study procedures

MoCA was used to determine the participants’ cognitive status. A cutoff score of 26 was chosen, as this value is widely used in the screening for mild cognitive impairment (MCI) among Asian individuals among Asian individuals in Singapore. ²³ Participants with a MoCA score of ≥26 were enrolled in the cognitively‐healthy group, whereas participants with a MoCA score of <26 were enrolled in the cognitively‐impaired group. Subsequently, demographic information was collected (gender, ethnicity, education level, type of housing). Participants also completed other cognitive assessments: Abbreviated Mental Test (AMT) and MMSE; and functional status assessments: Barthel Index–Basic Activities of Daily Living (BADLs) ²⁴ and Lawton Instrumental Activities of Daily Living (IADLs). ²⁵

Next, participants attempted the CAVIRE VR assessment. The software of CAVIRE is built upon the Unity game engine, with integrated application programming interface (API) for voice recognition. The hardware of CAVIRE includes: (1) HTC Vive Pro Head Mounted Display (HMD), which enables the display of the 3D virtual environment; (2) Lighthouse sensors, which provide tracking of the HMD; (3) Leap Motion device mounted onto the HMD, which allows for the tracking of natural hand and finger movements; and (4) Rode VideoMic Pro microphone, which enables the capture of speech.

CAVIRE consists of a tutorial session, a cognitive assessment with 13 different segments, and an automated scoring algorithm. Each segment consists of virtual tasks mimicking common ADLs. Participants performed these tasks using their hand and head movements, as well as using speech. During the cognitive assessment, automated visual and voice instructions in English were present in the VR system to guide the participants. For each segment, participants were allowed multiple attempts to complete the tasks within the time limit. Based on the performance matrix, the first correct attempt would yield the highest possible score for that segment. Subsequent correct attempts would yield a lower score. Once the time limit was exceeded, the participants automatically proceeded to the next segment.

The tasks in these 13 different segments are designed to assess the six cognitive domains based on the DSM‐5 criteria, ² as shown in Figure 1. Examples of the different segments in CAVIRE, as well as an illustration of a participant performing the virtual tasks in CAVIRE using hand movements, are displayed in Figure 2.

FIGURE 1.

Cognitive domain assessed in each segment of CAVIRE.

FIGURE 2.

(A) Segment on preparing peanut butter bread for breakfast. (B) Segment on selecting tsshe corrects fruit from memory. (C) Participant performing the virtual tasks in CAVIRE using hand movements.

At the end of the CAVIRE assessment, participants provided feedback on their acceptability of using CAVIRE by filling up a questionnaire, adapted from the Spatial Presence Experience Scale (SPES). ²⁶ The questionnaire answers were rated on a Likert scale (score from “1” to “5,” corresponding with “strongly disagree” to “strongly agree”).

2.4. Outcome measures

Based on the automated scoring algorithm, the CAVIRE system calculates the VR scores and time taken to complete the VR tasks based on the six cognitive domains for participants in the cognitively‐healthy group and those in the cognitively‐impaired group, respectively. The total VR score is also automatically calculated and compared with the MoCA score. Participants who experience nausea, headache and/or giddiness during the CAVIRE assessment would be advised to discontinue the assessment, and are considered as dropouts. With regard to the acceptability of using CAVIRE, an average score of ≥80% on the questionnaire would be considered a positive outcome.

2.5. Data management and monitoring

Data from the questionnaires were added into a secure database and audited by a data management officer. Data from the CAVIRE assessment was combined with the audited questionnaire data from the database, and then given to the statistician for analysis.

2.6. Statistical analyses

Demographic characteristics, functional status assessments, cognitive assessments, VR score, time taken to complete the VR tasks, and participant feedback were summarized based on the cognitively‐healthy and cognitively‐impaired status of participants. Categorical and continuous variables were summarized as frequency (percentages), mean (SD) with minimum–maximum range, or median (interquartile range [IQR]) with minimum–maximum range as appropriate. The differences between cognitively‐healthy and cognitively‐impaired groups were tested using a chi‐square test and two‐sample t‐test (even though normality assumptions failed) for categorical and continuous variables, respectively. Linear regression was used for continuous outcomes to adjust for education, and the results were expressed in terms of β estimate with 95% confidence interval (95% CI). Receiver‐operating characteristic (ROC) curve and Youden's index analysis were used to find the optimal cutoff to distinguish between cognitively‐healthy and cognitively‐impaired individuals. Area under the curve (AUC) from ROC curve was determined to report the predictive ability of CAVIRE, MMSE, and AMT. Because there were a few outcomes to test for the difference between cognitively‐healthy and cognitively‐impaired groups, Bonferroni corrected p‐values for multiple comparisons were used. For the participant feedback scores, Mann–Whitney U test was performed for ordinal variables, as the distribution was non‐normal. All tests were two‐sided and a p‐value < .05 was considered statistically significant unless otherwise stated. All analyses were carried out using SAS version 9.4.

3. RESULTS

Two hundred twenty‐seven individuals 65–84 years of age were screened for eligibility, of whom 85 individuals were excluded. Among the 142 individuals who fulfilled the criteria, 110 agreed to participate: 60 participants (cognitively‐healthy); 50 participants (cognitively‐impaired).

One participant from the cognitively‐impaired group did not complete the study due to discomfort during the MMSE assessment, and hence did not proceed with the CAVIRE assessment. The remaining 109 participants (99%) completed all study procedures. No participant dropped out due to VR‐induced symptoms (nausea, headache, and/or giddiness). The participant recruitment flowchart is depicted in Appendix 1.

3.1. Participant demographics

Demographic characteristics are shown in Table 1. Other than education level, there was no difference between cognitively‐healthy and cognitively‐impaired groups.

TABLE 1.

Demographic characteristics of all included participants.

Characteristic	Cognitively‐healthy (n = 60)	Cognitively‐impaired (n = 49)	Total (n = 109)	p‐value
Gender, n (%)
Male	27 (45.0)	22 (44.9)	49 (45.0)	1.0000
Female	33 (55.0)	27 (55.1)	60 (55.0)
Ethnicity, n (%)
Chinese	53 (88.3)	43 (87.8)	96 (88.1)	0.1051
Non‐Chinese	7 (11.7)	6 (12.2)	13 (11.9)
Education, n (%)
Up‐to‐secondary	30 (50.0)	42 (85.7)	72 (66.1)	<0.0001
Post‐secondary/tertiary	30 (50.0)	7 (14.3)	37 (33.9)
Socioeconomic status (housing), n (%)
Public housing	41 (68.3)	39 (79.6)	80 (73.4)	0.2001
Private housing	19 (31.7)	10 (20.4)	29 (26.6)

Note: Values are summarized as frequency (percentages). p‐values are based on chi‐square test.

3.2. Cognitive assessments and functional status assessments

The cognitively‐healthy group achieved significantly higher scores on the MoCA compared to the cognitively‐impaired group. After adjusting for education, the p‐value remained significant (p < 0.0001). In terms of time taken to complete the MoCA, the cognitively‐healthy group required a significantly shorter time.

The cognitively‐healthy group also achieved significantly higher scores on the other assessments: Instrumental Activities of Daily Living (IADLs); Abbreviated Mental Test (AMT); and Mini‐Mental State Examination (MMSE). However, there was no significant difference for the Basic Activities of Daily Living (BADLs). The results are shown in Table 2.

TABLE 2.

Comparison of functional status, cognitive, and virtual reality assessments between cognitively‐healthy and cognitively‐impaired participants.

Type of assessment	Cognitively‐healthy (n = 60)	Cognitively‐impaired (n = 49)	Mean difference (95% CI)	p‐value	Mean difference (95% CI) 1	p–value 1
Functional status assessments
BADL score	99.5 (1.77) 90–100	97.7 (4.34) 85–100	1.85 (0.52, 3.17)	0.0070	2.16 (0.66, 3.67)	0.0014
IADL score	22.8 (0.86) 18–23	20.7 (3.18) 9–23	2.02 (1.08, 2.95)	<0.0001	1.86 (0.81, 2.91)	0.0001
Cognitive assessments
AMT score	9.7 (0.50) 8–10	9.1 (1.19) 5–10	0.60 (0.24, 0.96)	0.0015	0.57 (0.15, 0.99)	0.0024
MMSE score	28.4 (1.57) 24–30	25.5 (3.40) 14–30	2.89 (1.83, 3.94)	<0.0001	2.77 (1.56, 3.98)	<0.0001
MoCA score	28.0 (1.23) 26–30	21.5 (3.57) 12–25	6.52 (5.46, 7.58)	<0.0001	6.25 (5.06, 7.44)	<0.0001
Time taken to complete MoCA (sec)	641.1 (204.30) 355–1500	1083.0 (428.78) 463–2100	−441.90 (−590.00, −293.80)	<0.0001	−413.77 (−571.82, −255.71)	<0.0001
Virtual Reality (VR) Assessment
Total VR score	1703.3 (298.53) 1100–2175	1299.5 (319.36) 500–1900	403.80 (285.40, 522.30)	<0.0001	365.94 (221.48, 510.4)	<0.0001
Time taken to complete VR (s)	519.0 (70.01) 354–643	586.1 (72.86) 445–714	−67.10 (−94.44, −39.75)	<0.0001	−64.07 (−97.86, −30.28)	<0.0001

Note: Values are summarized as mean (SD) and min—max. Mean difference is based on difference between cognitively‐healthy and cognitively‐impaired groups. p‐values are based on two‐sample t‐test.

¹ Adjusted mean difference with 95% confidence interval (CI) and associated p‐value after adjusting for education. Statistical significance was set at p < 0.00625 after adjusting for Bonferroni's correction of multiple comparisons.

3.3. Virtual reality assessment

Participants from the cognitively‐healthy group achieved significantly higher total VR scores on the CAVIRE assessment compared to participants from the cognitively‐impaired group. Upon adjusting for education, the p‐value was still significant (p < 0.0001). In terms of time taken to complete the CAVIRE assessment, the cognitively‐healthy group required a significantly shorter time. The p‐value remained significant at p < 0.0001, even after adjusting for education. The results are presented in Table 2.

Cognitively‐healthy participants also achieved significantly higher VR cognitive function scores for each of the six cognitive domains: complex attention; executive function; language; learning and memory; perceptual‐motor function; and social cognition. Upon adjusting for education level, the p‐values were still significant for all six cognitive domains. The results are available in Table 3.

TABLE 3.

Comparison of virtual reality score between cognitively‐healthy and cognitively‐impaired based on the six cognitive domains.

Cognitive domain	Cognitively‐healthy (n = 60)	Cognitively‐impaired (n = 49)	Mean difference (95% CI)	p‐value	Mean difference (95% CI) 1	p‐value 1
Complex attention	218.75 (73.01) 50–325	137.24 (52.32) 50–275	81.51 (57.65, 105.40)	<0.0001	70.85 (44.7, 96.99)	<0.0001
Executive function	291.67 (62.89) 100–400	230.61 (50.59) 100–375	61.05 (39.51, 82.60)	<0.0001	56.2 (32.45, 79.94)	<0.0001
Language	313.33 (69.42) 125–400	232.65 (64.39) 50–325	80.68 (55.22, 106.10)	<0.0001	73.51 (45.95, 101.08)	<0.0001
Learning and Memory	280.00 (72.75) 125–400	214.29 (59.51) 100–375	65.71 (40.60, 90.83)	<0.0001	61.46 (33.77, 89.15)	<0.0001
Perceptual‐motor	330.42 (58.96) 100–400	284.18 (79.66) 75—400	46.23 (19.02, 73.45)	0.0011	40.74 (12.32, 69.16)	0.0054
Social cognition	269.17 (58.47) 150–375	200.51 (81.57) 50–350	68.66 (41.05, 96.26)	<0.0001	63.19 (34.45, 91.93)	<0.0001

Note: Values are summarized as mean (SD) and min—max. Mean difference is based on difference between cognitively‐healthy and cognitively‐impaired groups. p‐values are based on two‐sample t‐test.

¹ Adjusted mean difference with 95% confidence interval (CI) and associated p‐value after adjusting for education. Statistical significance was set at p < 0.0083 after adjusting for Bonferroni's correction of multiple comparisons.

In terms of time taken to complete the VR tasks based on the six cognitive domains, cognitively‐healthy participants also required a significantly shorter time. After adjusting for education level, the p‐values remained significant for all six cognitive domains. The results can be found in Table 4.

TABLE 4.

Comparison of time taken to complete the virtual reality tasks based on the six cognitive domains between cognitively‐healthy and cognitively‐impaired.

Cognitive domain	Cognitively‐healthy (n = 60)	Cognitively‐impaired (n = 49)	Mean difference (95%CI)	p‐value	Mean difference (95% CI) 1	p‐value 1
Complex attention	126.07 (27.75) 75–190	144.53 (22.86) 97–190	−18.46 (−28.07, −8.85)	0.0002	−16.00 (−26.55, −5.46)	0.0033
Executive function	199.93 (42.18) 109–283	233.67 (39.38) 150–300	−33.74 (−49.26, −18.22)	<0.0001	−32.27 (−49.19, −15.34)	0.0003
Language	182.65 (31.25) 127–254	208.02 (26.12) 164–256	−25.37 (−36.26, −14.48)	<0.0001	−23.08 (−35.05, −11.11)	0.0002
Learning and Memory	81.37 (16.96) 45–121	94.96 (17.79) 61–123	−13.59 (−20.25, −6.94)	0.0001	−13.82 (−20.99, −6.64)	0.0002
Perceptual‐motor	169.50 (35.85) 91–259	198.67 (40.07) 126–259	−29.17 (−43.78, −14.56)	0.0001	−29.45 (−45.09, −13.81)	0.0003
Social cognition	92.33 (26.03) 54–160	109.10 (24.35) 61–160	−16.77 (−26.36, −7.18)	0.0008	−14.17 (−24.55, −3.79)	0.0079

Note: Values are summarized as mean (SD) and min—max. Mean difference is based on difference between cognitively‐healthy and cognitively‐impaired groups. p‐values are based on two‐sample t‐test.

¹ Adjusted mean difference with 95% confidence interval (CI) and associated p‐value after adjusting for education. Statistical significance was set at p < 0.0083 after adjusting for Bonferroni's correction of multiple comparisons.

3.4. Potential of CAVIRE to distinguish between cognitively‐healthy and cognitively‐impaired individuals

ROC curve analysis was undertaken over continuous total VR scores of all participants to evaluate the ability of CAVIRE to distinguish between cognitively‐healthy (MoCA ≥26) and cognitively‐impaired (MoCA <26) individuals. An optimal statistical cutoff was achieved at 1600 points (sensitivity = 83.67%, 95% CI = 70.34–92.68; specificity = 61.67%, 95% CI = 48.21–73.93), as shown in Appendix 2 with an AUC of 0.7267 (95% CI = 0.65–0.81). The maximum possible VR score is 2400 points.

3.5. Comparison between CAVIRE and the paper‐and‐pencil cognitive assessments

In assessing the correlation between the CAVIRE score and the paper‐and‐pencil cognitive assessment scores, the Pearson correlation coefficient was 0.63 for CAVIRE and MoCA scores; 0.55 for CAVIRE and MMSE scores; and 0.48 for CAVIRE and AMT scores, as shown in Appendix 3.

In comparing the predictive ability of CAVIRE versus MMSE and AMT, the AUC based on CAVIRE scores (continuous scale) was 0.90 (95% CI = 0.86–0.95); the AUC based on MMSE (continuous scale) was 0.70 (95% CI = 0.62–0.78); and the AUC based on AMT (continuous scale) was 0.81 (95% CI = 0.74–0.88). The results are shown in Appendix 4.

3.6. Correlation of cognitive domains measured by CAVIRE and MoCA

Both CAVIRE and MoCA assess five common cognitive domains: complex attention; executive function; language; learning and memory; and perceptual‐motor function. The Pearson correlation coefficient for CAVIRE and MoCA was 0.371 for complex attention; 0.535 for executive function; 0.375 for language; 0.516 for learning and memory; and 0.351 for perceptual‐motor function (Appendix 5).

3.7. Acceptability of using CAVIRE virtual reality

Acceptability of using CAVIRE was assessed via a questionnaire with 10 responses rated on a Likert scale. As shown in Table 5, the median feedback score for each of the 10 questions was 4 of 5, whereas the median total feedback score overall was 40 of 50. The largest proportion of responses to all the 10 questions was either “agree” or “strongly agree.”

TABLE 5.

Questionnaire to assess acceptability on using CAVIRE.

	Feedback Score
Feedback questionnaire item	Cognitively‐healthy (n = 60)	Cognitively‐impaired (n = 49)	Total (n = 109)	p‐value
The virtual reality (VR) system was easy to use.	4 (4 to 5) 2–5	4 (4 to 5) 2–5	4 (4 to 5) 2–5	0.1565
The amount of time I spent on the VR test is acceptable to me.	4 (4 to 5) 3–5	4 (4 to 4) 2–5	4 (4 to 5) 2–5	0.0048
During the VR test, I did not experience any symptoms such as: nausea, headache, or giddiness.	4 (4 to 5) 3–5	4 (4 to 5) 2–5	4 (4 to 5) 2–5	0.4247
It seemed that I was actually there in a new environment.	4 (4 to 5) 2–5	4 (4 to 4) 1–5	4 (4 to 5) 1–5	0.2225
The environment seemed similar to the real world.	4 (4 to 5) 2–5	4 (4 to 4) 2–5	4 (4 to 5) 2–5	0.0043
I was able to interact with the objects around me to perform the tasks.	4 (4 to 5) 2–5	4 (3 to 4) 2–5	4 (4 to 5) 2–5	0.0007
The use of VR helps to make the experience in the clinic more interactive.	4 (4 to 5) 3–5	4 (3 to 4) 1–5	4 (4 to 5) 1–5	0.0003
The use of VR to help diagnose a medical condition appeals to me.	4 (4 to 5) 2–5	4 (3 to 4) 2–5	4 (4 to 5) 2–5	0.0005
In the future, I would like to see more VR applications being used in the clinic.	4 (4 to 5) 2–5	4 (3 to 4) 2–5	4 (4 to 5) 2–5	0.0002
Overall, I enjoyed the VR experience in the clinic.	5 (4 to 5) 3–5	4 (4 to 4) 1–5	4 (4 to 5) 1–5	0.0004
Total Feedback Score	42.5 (40 to 48.5) 34–50	40 (36 to 41) 24–50	40 (37 to 47) 24–50	<0.0001

Note: Values are summarized as median (interquartile range [IQR]) and min—max. p‐values are based on the Mann–Whitney U test.

In comparing the median total feedback score between cognitively‐healthy and cognitively‐impaired participants, cognitively‐healthy participants responded with significantly higher feedback scores overall (p < 0.0001). Nevertheless, for three question items there was no significant difference between cognitively‐healthy and cognitively‐impaired participants: “The virtual reality (VR) system was easy to use”; “During the VR test, I did not experience any symptoms such as: nausea, headache or giddiness”; and “It seemed that I was actually there in a new environment”.

4. DISCUSSION

The results indicate the potential of CAVIRE as a screening tool to assess cognitive function across six cognitive domains in older Asian adults. Based on the six cognitive domains, participants from the cognitively‐healthy group achieved significantly higher VR scores and required a significantly shorter time to complete the VR tasks, as compared to participants from the cognitively‐impaired group. The Pearson correlation coefficient of 0.63 indicates moderate positive correlation between the VR and MoCA scores. ²⁷

The AUC from the ROC analysis was 0.7267, which suggests that CAVIRE can distinguish between cognitively‐healthy and cognitively‐impaired participants. An AUC of >0.700 is considered acceptable. ²⁸ AUC is an indicator of a biomarker's predictive ability to discriminate between healthy and diseased individuals. ²⁹ AUC values of MoCA have been reported to vary between 0.71 and 0.99 across different populations. ³⁰ In Singapore, a local study reported lower AUC values for MoCA: 0.63 and 0.65. ³¹ Hence, the predictive ability of CAVIRE to detect cognitive impairment seems promising. Further enhancements can be made to improve CAVIRE's predictive ability.

At the optimal cutoff score of 1600 points, the sensitivity was 83.67%, whereas the specificity was 61.67%. Having high sensitivity but low specificity may lead to an increased number of false positives, resulting in patients who are disease‐free being informed that they have the possibility of having the disease. ³² Nevertheless, the intent is for CAVIRE to be used as a screening tool, not a diagnostic tool. Thus an increased rate of false positives will allow more suspected individuals to be identified and receive appropriate interventions, rather than missing early identification of cognitive impairment, leading to undesirable consequences related to progressive cognitive impairment or even dementia. However, an increased rate of false positives may potentially cause more anxiety and higher impact on health care costs. Therefore, further improvements are planned to increase the sensitivity and specificity of CAVIRE. For example, the current CAVIRE system utilizes a Leap Motion device (mounted onto the HTC Vive Pro VR headset) to detect the hand and finger movements of the participants. However, because the Leap Motion and HTC Vive are two separate systems, the integration of the Leap Motion device with the HTC Vive may not be optimal to detect the hand and finger movements in the most accurate manner. Future versions of CAVIRE will explore the use of newer VR headsets that have fully integrated hand‐ and finger‐tracking functions to enhance the performance of CAVIRE.

To our knowledge, CAVIRE is currently the only VR system that assesses the six cognitive domains through multiple virtual environments. The 13 different segments in CAVIRE depict ADLs in common local environments. Other studies utilizing VR to assess cognitive function focus on limited cognitive domains in a limited number of virtual environments. ¹⁸ , ¹⁹ Furthermore, unlike other VR systems, CAVIRE does not require external devices (e.g., mouse, keyboard, touchscreen, or VR controller) for the VR tasks. In CAVIRE, there is direct interaction with the virtual environment using natural hand movements and speech. This allows for enhanced ecological validity, reflecting the user's real‐world functional ability. ¹³

In the current study, the use of CAVIRE seems to be well received by senior participants. The median total feedback score was 40 of 50; the largest proportion of responses to the 10 questions assessing acceptability was “agree” or “strongly agree.” This fulfills the pre‐defined acceptability level in which a total feedback score of 80% and above will be considered a favorable outcome. ²⁰ Overall, cognitively‐healthy participants were more accepting of VR compared to cognitively‐impaired participants. Cognitively‐healthy participants were able to perform the VR tasks better, and hence had a more positive VR experience.

However, there was no significant difference between the response of cognitively‐healthy and cognitively‐impaired participants for three question items: “The virtual reality (VR) system was easy to use;” “It seemed that I was actually there in a new environment;” and “During the VR test, I did not experience any symptoms such as: nausea, headache, or giddiness.” These results show that despite the difference in cognitive status, both “cognitively‐healthy” and “cognitively‐impaired” participants agree that CAVIRE was easy to use, and CAVIRE was able to simulate the presence of being inside a new environment. Moreover, both cognitively‐healthy and cognitively‐impaired participants generally agreed that they did not experience VR‐induced adverse symptoms. No participant dropped out of the study due to adverse symptoms during the VR assessment.

AMT and MMSE were used to assess the cognitive function of participants, in addition to MoCA. BADLs and IADLs were used to assess the functional ability of participants in performing ADLs. The scores for AMT and MMSE reflect that of the MoCA, where there is significant difference between cognitively‐healthy and cognitively‐impaired groups. BADL score did not show a significant difference between both groups, whereas IADL score showed a significant difference. This was expected, as BADLs include skills needed to manage a person's basic physical needs, whereas IADLs include skills that require more complex processing. Hence, IADLs are more susceptible to deterioration due to decline in cognitive function compared to BADLs. ³³

In comparing CAVIRE with the paper‐and‐pencil cognitive assessments, CAVIRE showed higher correlation with MoCA, as compared with MMSE and AMT, respectively. MoCA has been shown to be superior in assessing cognitive impairment, and less susceptible to ceiling effects, as compared to MMSE and AMT. ³⁰ , ³⁴ , ³⁵ Therefore, it was appropriate to use MoCA cutoff scores to differentiate the cognitive status in this proof‐of‐concept study for CAVIRE. In addition, CAVIRE also exhibited higher predictive ability to discriminate between cognitively‐healthy and cognitively‐impaired individuals, compared to the MMSE and AMT.

Sub‐analysis was conducted to explore the correlations of the individual cognitive domains between CAVIRE and MoCA. The calculation of the cognitive domain scores of MoCA was adapted from a previous study. ³⁶ The results show lower correlations of each individual cognitive domain scores compared to the correlation of the total CAVIRE score (0.63). The disparity could be due to the difference in the spread of the individual cognitive domain scores of CAVIRE versus that of MoCA. For example, for the cognitive domain of Language, the CAVIRE scores range from 0 to 400, across intervals of 25. In contrast, the MoCA scores range from 0 to 6, with intervals of 1 in‐between.

This study has several limitations. Demographic characteristics showed that a higher proportion of cognitively‐healthy participants received post‐secondary or tertiary education, compared to cognitively‐impaired participants. Education level could be a confounding factor. However, the limited sample size in this study did not permit stratification of participants based on education. Stratification by education level could be incorporated into future studies to minimize confounding effects. Nevertheless, analyses were performed by controlling for education level, as shown in Tables 2, 3, and 4. After adjusting for education, the p‐values did not change much, and the results remain statistically significant. This suggests that education level was not a confounding factor.

The recruitment resulted in an unequal number of participants for each group (60 cognitively‐healthy; 49 cognitively‐impaired), as the study was funded by different grants. A similar local study was done previously using another VR system to assess cognitive function. ³⁷ In the previous study, 37 participants were recruited in the cognitively‐healthy group, whereas 23 participants were recruited in the cognitively‐impaired group. A post hoc power calculation showed that group sample sizes of 60 for the cognitively‐healthy group and 49 for the cognitively‐impaired group achieved adequate power (> 80%) to reject the null hypothesis of equal means, when the population mean difference is 40.0 (conservative approach) with SD of 57.0 and significance level (alpha) of 0.006, using a two‐sided two‐sample unequal‐variance t‐test. Therefore, this study would be adequately powered.

The current study involved only a single visit for each participant. The test–retest reliability of the CAVIRE system will be assessed in a future study in which participants will perform the same virtual tasks in a second visit after a stipulated short time interval. In addition, further studies are planned to validate CAVIRE against clinical diagnosis. In the next phase of the development of CAVIRE, we will enroll adults who are diagnosed with MCI by neurologists with imaging evidence and compare their CAVIRE performance with that of cognitively‐healthy individuals.

The study population was limited to English‐speaking participants, as the current version of CAVIRE uses English to narrate and provide instructions to carry out the virtual tasks. Other language versions of CAVIRE, such as Mandarin and Malay language, will be developed to cater to the multi‐ethnic Asian population in Singapore. Language versions are essential, as the results show that the cognitively impaired persons tend to be lower in their education level and hence may be less literate in English. The other language versions are required to undertake similar validation studies.

In conclusion, the study demonstrates the capability of the CAVIRE virtual reality system to distinguish between cognitively‐healthy and cognitively‐impaired individuals. The significant differential performance across the six cognitive domains, coupled with the general acceptability of using CAVIRE among the participants, allude to the potential of CAVIRE as a screening tool for cognition that can be deployed in primary care.

CONFLICT OF INTEREST STATEMENT

The authors declare that there are no conflicts of interest related to this study.

CONSENT STATEMENT

Written informed consent was obtained from all human participants.

Supporting information

Supplementary Information

Supplementary Information

Supplementary Information

Supplementary Information

Supplementary Information

Supplementary Information

Tan NC, Lim JE, Sultana R, Quah JHM, Wong WT. A virtual reality cognitive screening tool based on the six cognitive domains. Alzheimer's Dement. 2024;16:e70030. 10.1002/dad2.70030