Web-based cognitive assessment in older adults: Where do we stand?

Abstract

Purpose of review

The use of digital tools for remote cognitive measurement of older adults is generating increasing interest due to the numerous advantages offered for accessibility and scalability. However, these tools also pose distinctive challenges, necessitating a thorough analysis of their psychometric properties, feasibility and acceptability.

Recent findings

In this narrative review, we present the recent literature on the use of web-based cognitive assessment to characterize cognition in older adults and to contribute to the diagnosis of age-related neurodegenerative diseases. We present and discuss three types of web-based cognitive assessments: conventional cognitive tests administered through videoconferencing; unsupervised web-based assessments conducted on a computer; and unsupervised web-based assessments performed on smartphones.

Summary

There have been considerable progress documenting the properties, strengths and limitations of web-based cognitive assessments. For the three types of assessments reported here, the findings support their promising potential for older adults. However, certain aspects, such as the construct validity of these tools and the development of robust norms, remain less well documented. Nonetheless, the beneficial potential of these tools, and their current validation and feasibility data, justify their application [see Supplementary Digital Content (SDC)].

INTRODUCTION

Thirty years ago, a group of researchers and clinicians at the Institut universitaire de gériatrie de Montréal (IUGM) created an award-winning computerized memory battery. This battery was poised to revolutionize cognitive assessment by providing precisely controlled measurements in terms of temporal parameters and delivery, while streamlining the workflow for clinicians [1]. Unfortunately, the anticipated promises did not materialize as expected. Decades later, the field has yet to fully embrace web-based tools in both clinical and laboratory settings. Changing the way a field practices is notoriously difficult and it is hard to implement innovation in practice. There are many reasons why this might have been the case for web-based cognitive assessment. Most clinicians and researchers were trained in, and frequently use, traditional in-person psychometric testing. By contrast, the online environment is newer and hence less familiar. Clinicians may also have hesitation due to less control/standardization of online formats and they may wonder whether they are sufficiently valid and normed. Given the consequences of a diagnosis of dementia from a cognitive assessment, the need for properly designed and validated tests is particularly critical. Another possible reason is that switching to online testing requires access to computerized tests, capacity hosting web-based data collection and storage, as well as capacity to process and analyze large-scale web-based data. In addition, clinicians may feel overwhelmed by the number of web-based tests, especially as there are no recommendations or major consensus from the field on which tools to use. Finally, clinicians may think that older adults will not be comfortable with computerized testing due to the digital divide. For all these reasons, experts may be open to the idea of web-based testing in theory, but have difficulty implementing it in practice.

However, this situation has the potential to change dramatically over the next few years. The SARS-CoV-2 COVID-19 pandemic has reshaped the landscape, ushering a new era where clinicians and researchers have become aware of the enormous but under-exploited potential of technology for cognitive assessment. Extended periods of confinement and safety concerns, which limited hospital visits and direct contact with vulnerable older adults, compelled many clinicians and researchers to adopt remote assessment methods (e.g.: [2,3]).

It is important to acknowledge that telehealth and web-based assessments have existed long before the COVID-19 pandemic. One example is the Cambridge Neuropsychological Test Automated Battery (CANTAB) system, which was initially developed in the 1980 s to assess older adults suspected to have dementia [4,5]. As early as 2012, the Academy of Clinical Neuropsychology and the National Academy of Neuropsychology published a position paper with recommendations on this topic [6]. However, the COVID-19 pandemic brought a dramatic shift that transformed web-based cognitive assessments from sophisticated tools for specific research interests into tools likely to make a substantial impact in both clinical practice and research.

In this narrative review, we examine recent literature (2021–2023) on the use of web-based cognitive assessment in older adults, focusing on its potential for characterizing cognition and contributing to the diagnosis of age-related neurodegenerative diseases. Furthermore, we examine the limitations of these tools and explore potential avenues for further advancements in this field. Digital neuropsychology refers to the assessment of cognition using either computerized or web-based resources [7,8]. Web-based cognitive assessment refers to any cognitive test or battery that requires access to the internet. Here, we present and discuss three types of web-based cognitive assessments: conventional cognitive tests administered through videoconferencing; unsupervised web-based assessments conducted on a computer; and unsupervised web-based assessments performed on smartphones or other portable devices. For each type of assessment, we present available recent data on psychometric validation, examining criterion validity (via concurrent validity and within-individual comparability), construct validity, and test-retest reliability. Concurrent validity measures whether web-based tests are related to in-person versions, while within-individual comparability measures whether the web-based and in-person versions yield similar scores. Construct validity measures whether the test accurately reflects what it is intended to measure by examining their sensitivity to characteristics expected to be associated with these tests (for instance, diagnostic group or age) or show similar inter-correlations. Test-retest reliability measures consistency over repeated testing. Where available, we present findings on aspects of the user experience, including acceptability and usability of the tools. We also discuss the unique challenges and potential contributions associated with each assessment type. To conclude, we provide an overview of the overall potential, identify key issues, and share perspectives in this regard. 

Box 1.

no caption available

VIDEOCONFERENCING

Videoconferencing has been used to administer conventional in-person cognitive tests remotely and under supervision, following the same protocols as a conventional in-person test. Thus, the examiner is present virtually and can provide instructions, record responses and the time required to perform the test, as well as provide feedback and verify that the test is carried out correctly. In some cases, the tests need to be adapted for remote administration by removing or modifying items that are not suitable for remote testing. This includes elements of the test that involve motor components or require copying tasks. In most cases, however, test elements do not need to be modified and can be used largely the same way as when administered in-person.

A few recent studies or literature reviews [3,9,10^▪▪,11,12^▪] have investigated the concurrent validity and comparability between videoconference-based assessments of commonly used cognitive tests and their in-person counterparts. Hunter et al.[10^▪▪] conducted a literature review identifying 12 studies (740 participants) that compared in-person with videoconference-based assessment in older adults with a range of cognitive impairments (from mild cognitive impairment (MCI) to dementia). Results indicate that many of the tests currently used in clinical practice [e.g.: Mini-Mental State examination (MMSE), Severe MMSE (SMMSE), Letter Fluency, and Hopkins Verbal Learning Test-revised (HVLT-R)] show good concurrent validity compared to in-person administration. It should be noted, however, that the participants included in the studies had, on average, a relatively high level of education (mean of 14.1). Additionally, concurrent validity was often measured in controlled environments (usually a research facility) and therefore may not be as high for people who are disadvantaged or less proficient with technology, or when the test is completed in the person's home. Beishon et al.[12^▪] examined the literature on common global cognition tests [e.g., Montreal Cognitive Assessment (MoCA), MMSE, Rowland Universal Dementia Assessment Scale (RUDAS)] and also reported high concurrent validity, but scores were often not comparable, suggesting insufficient comparability (see also [9]). It is interesting to note that the remote version of the Kimberley Indigenous Cognitive Assessment (KICA-screen) showed good concurrent validity and comparability compared to the in-person version when tested in Aboriginal and Torres Strait Islander people [13^▪]. This supports the hypothesis that remote assessment could reduce barriers related to accessing dementia care in rural and isolated areas.

Videoconferencing assessment proved to have good diagnostic validity, particularly in distinguishing people with dementia from healthy older adults. However, its precision in identifying individuals with MCI was found to be relatively lower [10^▪▪,11], although there are still very few studies that have assessed diagnostic validity.

Finally, a limited number of studies have examined the user-experience, which report good acceptability of remote testing through videoconferencing, even among a diverse population and patients with dementia [13^▪,14,15]. However, challenges have been reported, specifically for certain linguistic tasks (e.g. monosyllabic repetition) or visual items (e.g. Visual Object and Space Perception Battery (VOSP) object decision task) [14].

In sum, most studies comparing videoconference-based to in-person traditional cognitive tests report good concurrent validity but poor or fair comparability. Lack of comparability means that individual participants do not necessarily obtain the same score on the videoconference and in-person versions of the assessment. Thus, despite the apparent similarities and seemingly minor changes when the test is administered remotely, there are differences in the underlying construct and/or level of difficulty. Differences in the underlying construct imply that the presence of an impairment may indicate different cognitive deficits when using different versions of the assessment. Differences in the level of difficulty indicate that cut-off values and normative data cannot be transposed directly from in-person to remote versions of the same test. It is therefore important to provide more research data to not only determine the criterion validity of cognitive tests administered through videoconferencing compared to their in-person version, but also to understand their construct validity. Furthermore, there is a need to develop norms and cut-off values that are appropriate for these particular versions of assessments.

UNSUPERVISED COMPUTER WEB-BASED COGNITIVE ASSESSMENTS

Unsupervised web-based computer cognitive assessments consist of self-administered online tasks completed remotely. These assessments are typically short web-versions of lab or clinical cognitive tasks, although a few recent developments include more ecologically valid tasks or a virtual reality environment [16]. Unsupervised tests allow the collection of data from tens of thousands of individuals [17–19]. The scale and age range in these large datasets offer an unprecedented opportunity for researchers to study population-level variability in cognitive performance [17,18], examine how cognition relates to demographic or lifestyle risk factors for dementia (e.g., age, sex, education, smoking [20,21]), and detect subtle effects, such as sex differences [22].

A recent systematic review identified over 3000 unsupervised web-based cognitive assessment tools [23]. However, it is important to note that these tools exhibit substantial differences in development and validation. The main foreseeable challenge for unsupervised computer assessments is the increase in confounding factors due to loss of experimenter control and standardization. However, counterbalanced within-person testing has demonstrated that the testing environment does not significantly impact the scores of older adults from unsupervised at-home testing compared to supervised lab testing [24]. Within-person studies also show good test-retest reliability [25,26,27^▪,28], good concurrent validity compared to a gold-standard clinician administered neuropsychological assessment [29,30,31^▪] and poor to good convergent validity compared to analogous neuropsychological tasks [24,25,28,32,33^▪,34^▪] in older adults and in age-related neurodegenerative diseases or conditions (MCI, Alzheimer's disease, dementia with Lewy bodies, and Parkinson's disease). Unsupervised online assessments have successfully identified effects that are consistently found with supervised in-person testing, such as age differences and between-task correlations [35,36^▪]. This suggests excellent construct validity, although with slightly lower scores on fine motor or fast reaction time tests due to differences in device latencies and interfaces [19,32]. Unsupervised computer assessments also show good diagnostic validity for neurodegenerative diseases [30,34^▪,37,38] and associations with neurodegenerative biomarkers for dementia [26].

Due to the unsupervised nature of such tests, it is especially critical to assess and report measures of user experience. User experience studies find that most older adults successfully understand and complete online tasks [24,39–41], even though they subjectively report greater anxiety and difficulty than younger adults [28]. Studies have indicated relatively positive user experiences and comparable scores across variations in familiarity or attitudes to technology use [24,27^▪,41]. Interestingly, some studies have reported that older and middle-aged adults can even present more reliable scores than younger adults, possibly due to fewer distractions [35,42], and have lower attrition levels than younger adults for multiple tests [27^▪]. Older adults also report an eagerness to use technology, and actively incorporate it into their daily lives, particularly touchscreen devices, as the direct motor-screen interactions are easy to navigate [43]. Unsupervised computer assessments have also demonstrated feasibility in people with MCI and mild AD [44].

While unsupervised computer assessments herald a new era of efficiency and scalability in cognitive testing, they come with unique challenges. Rigorous data examination and cleaning are imperative when handling large online datasets [36^▪]. Furthermore, test developers should pay attention to task adaptations for older adults, such as clear task instructions, more practice trials, video tutorials, or a lower/adaptive starting difficulty [39,45].

UNSUPERVISED MOBILE WEB-BASED COGNITIVE ASSESSMENTS

Unsupervised mobile web-based cognitive assessments consist of digitized versions of existing conventional cognitive tools or newly developed laboratory tools. They are completed online by individuals without supervision using their own mobile devices, usually smartphones.

A major asset of mobile web-based assessments is the possibility of performing repeated high-frequency assessments over multiple days and on various occasions [27^▪]. This approach provides an estimate of the day-to-day cognitive functioning of individuals, accounting for within-individual variability, and improves reliability compared to conventional cognitive assessments that capture performance only at a single time point [46^▪▪]. Repeated assessments may be particularly relevant when assessing individuals with neurodegenerative disorders as their cognitive performance can vary significantly depending on the time of day [47,48^▪▪]. The variability observed in mobile cognitive performance assessments may have the potential to identify subtle changes in cognitive function that may help detect MCI [49].

Although there have been few studies conducted on mobile assessments as this field is fairly new, recent studies show that mobile tools can in some cases achieve high between-person reliability (i.e., whether inter-individual differences are comparable over repeated testing) [46^▪▪,50–54] and good test-retest reliability over intervals ranging from several days to several months[46^▪▪,50,55,56^▪]. This suggests that high-frequency measurement can be used with confidence to assess within-person fluctuations in cognitive performance. However, test-retest reliability is highly variable, ranging from poor to exceptional, and is not systematically reported [57].

Some mobile-based assessment tools (e.g., Trail Making Test, MoCA, or MMSE) also demonstrate very good concurrent validity, as evidenced by high correlation against their conventional in-person versions [49,50,55]. Recent studies also show good construct validity of mobile-based assessments through sensitivity to age and education effects, as well as fluid and neuro-imaging biomarkers that index AD progression. These biomarkers also include measures such as cortical thickness, hippocampal volume, and levels of amyloid and tau [46^▪▪,50,55,58–60]. Moreover, mobile-based assessments have good diagnostic accuracy in distinguishing people with MCI or AD from cognitively healthy older adults [55], and sometimes have greater accuracy than conventional in-person tools, such as the MoCA [61].

In terms of user experience, studies show good-to-high completion rates, with relatively low drop-out rates, even for cognitive assessments completed several times per day [46^▪▪,50,51,54,55,56^▪]. Older participants have shown a tendency to stay engaged in remote studies for longer durations compared to younger participants [62]. Additionally, they have expressed positive perceptions of mobile-based assessments, finding them acceptable and enjoyable (e.g., [50]).

Mobile-based assessments can be used to design tests that are less expensive and time consuming than conventional in-person assessments. However, frequent assessments can be challenging to complete, hence concise or shortened measures are recommended to provide participants the autonomy they need to plan tasks according to their daily schedule [63]. Furthermore, while many tools are now available, many do not provide psychometric data that are critical for clinical use, such as normative data and sensitivity or specificity values [23]. Finally, there are also several practical limitations to consider, such as potential variability in finger and hand dexterity, as well as variations in screen size [64].

CONCLUSION

Interest in the vast potential offered by remote assessment possibilities is growing. Web-based assessments (especially supervised) demonstrate good concurrent validity, and expected associations (e.g., with biomarkers, age, clinical status), but can show insufficient comparability with conventional in-person versions which suggests they reflect slightly different constructs or difficulty. Psychometric properties are largely good, but vary considerably from one tool to another. Web-based assessments are generally well accepted by older adults, although further user experience research is needed.

Web-based testing offers advantages of better accessibility and inclusivity (e.g., for remote or vulnerable populations) (e.g., [13^▪]) and thus overcomes barriers to recruitment, testing, and retention [65^▪]. Unsupervised assessments can be easily and inexpensively distributed world-wide, and mobile device options offer particularly exceptional portability. Such assessments enable measuring cognition across multiple sessions, shifting from traditional single-session in-person neuropsychological testing to repeated assessments [35,66]. This scalability offers larger sample sizes, greater sample diversity, and lower experimenter effects [46^▪▪].

Although web-based assessments offer clear advantages, they also present certain challenges [67^▪]. Issues remain that might hinder validity and usability, such as hardware differences or technical issues [8,62]. While web-based assessments save time and resources in the long term, it is important to initially invest in tool development, validation, and technical adaptation. Older adults still face challenges due to the digital divide (e.g., technology anxiety, limited digital literacy [68], although confinement from the COVID-19 pandemic increased technology usage in this group [69,70]. Finally, privacy and ethical issues must be considered, particularly when communicating results [63].

In conclusion, although cognitive assessments administered in person by a clinician are currently considered the gold standard for clinical diagnosis and characterization, the landscape is evolving. The role of web-based assessments in cognitive evaluation is gaining recognition, although its widespread adoption depends on objectives and contexts [67^▪]. For example, unsupervised assessments may be useful for initial screening efficiency and reach, and to indicate when further testing is needed. Supervised videoconferencing may represent an interesting option for vulnerable people or those living in rural or remote areas. In the long term, advances in digital health are expected to change how clinical and scientific work is conducted. It is however important to ensure scientifically and ethically sound practices when developing and adopting digital health solutions.

Acknowledgements

We would like to thank Ève Samson for her assistance in preparing the manuscript and Annie Webb for English revision.

Financial support and sponsorship

This work received support from grants awarded by the Canadian Institutes of Health Research (CIHR), Canada Research Chairs program, and National Sciences and Engineering Research Council of Canada (NSERC) to SB; a fellowship from the Quebec Consortium for the early identification of Alzheimer's disease (CIMA-Q), Lemaire Family Foundation, and a salary supported by the Canadian Consortium for Neurodegeneration in Aging (CCNA) funds to AAL; and a fellowship from the Research Center of the Institut universitaire de gériatrie de Montréal (CRIUGM), from SB's NSERC and CIHR funds to RP. SB also received support from the Fonds de recherche du Québec – Santé, the Social Sciences and Humanities Research Council of Canada and the Canadian Consortium on Neurodegeneration and the Alzheimer’ Society of Canada.

Funding support: Canadian Institutes of Health Research, Canada Research Chairs and National Sciences and Engineering Research Council of Canada.

Conflicts of interest

There are no conflicts of interest.

REFERENCES AND RECOMMENDED READING

Papers of particular interest, published within the annual period of review, have been highlighted as:

• ▪ of special interest

• ▪▪ of outstanding interest