Unsupervised online neuropsychological test performance for individuals with mild cognitive impairment and dementia: Results from the Brain Health Registry

Abstract

Introduction

The purpose of this study is to compare online neuropsychological test performance of older adults across self-reported diagnoses of being cognitively normal, mild cognitive impairment, and dementia due to Alzheimer's disease and to determine the association of memory concerns and family history of dementia on cognitive performance.

Methods

Participants completed the Cogstate Brief Battery unsupervised at home.

Results

Data from 6463 participants over the age of 55 years were analyzed. Adults with the diagnosis of mild cognitive impairment and Alzheimer's disease were associated with poorer performance on all cognitive tests than cognitively normal adults (P < .05 for all), and online cognitive test performance significantly improved diagnostic classification (P < .001). Poorer performance on all cognitive measures was associated with memory concern (P < .001 for all) but not family history of dementia.

Discussion

Our results provide preliminary support for the use of cognitive tests taken online without supervision as a means to improve the efficiency of participant screening and recruitment for clinical trials.

1. Introduction

The Brain Health Registry (BHR; www.brainhealthregistry.org) was launched in March of 2014. The goal of the BHR was to establish a national online research registry for all types of research trials using comprehensive assessments of medical, family, and psychiatric history as well as assessments of cognitive functioning. The inclusion of self-administered online cognitive assessments within a research registry represents an important potential avenue to improve upon the effectiveness of traditional registry methodology, particularly for studies focused on individuals with known or suspected cognitive dysfunction due to neurodegenerative diseases such as Alzheimer's disease (AD). Specifically, self-administered online cognitive test results may be particularly helpful to improve recruitment efforts focused on identifying individuals with mild cognitive deficits in the earliest stages of disease process. However, few studies have been conducted to evaluate the degree to which self-administered online cognitive test performance is sensitive to cognitive dysfunction.

The availability of online tests of cognition has increased dramatically over the past few years, and there are currently over 40 different online cognitive assessment batteries available [1], [2]. However, despite the recent growth in online tests of cognition, unlike computerized cognitive assessments that are not available online, the vast majority have not been validated clinically [2], [3], [4] or have been validated only in small samples or disease-specific populations [5], [6], [7], [8]. As a result, the effectiveness of using cognitive tests within online research registries to identify participants based on cognitive performance is not yet known. The BHR offers a significant opportunity to evaluate the potential for cognitive tests to be used in online research registries to improve participant screening and recruitment.

One of the cognitive test batteries available on the BHR website is the Cogstate Brief Battery (CBB). The CBB is a suite of computerized cognitive assessments [9], [10] that has been used extensively in supervised settings to evaluate cognitive functioning in clinical trials. In older adults, measures of information-processing speed, attention, and memory from the CBB have been shown to be sensitive to cognitive dysfunction and longitudinal cognitive decline in individuals with mild cognitive impairment (MCI) and dementia due to AD [9], [11], [12], [13]. Furthermore, poor performance on these measures from the CBB has been shown to be associated with patient report of subjective memory complaint in older adults [12], [14]. In contrast, despite reports that family history of dementia may be associated with cognitive dysfunction in older adults [15], [16], performance on the CBB has not yet been evaluated with respect to family history of memory problems or dementia. Although early reports have suggested that unsupervised CBB performance is comparable to supervised in-clinic assessments [17], [18], to date, there have not been studies conducted to evaluate patterns of performance on unsupervised CBB assessments in older adults with self-reported diagnoses of MCI and dementia.

The aim of this study was to evaluate performance on an online version of the CBB in an unsupervised context for assessment of cognition in older adults who are enrolled in the BHR. To achieve this, we first examined the extent to which performance on the tests forming the CBB varied with age, education, and gender. We then sought to determine the extent to which individuals with a self-reported diagnosis of early or established dementia performed differently from those who classified themselves as cognitively normal. Finally, we evaluated the strength of relationships between performance on the CBB tests and measures of self-reported concern about memory problems and family history of dementia. The first hypothesis was that performance on each BHR-CBB test would be associated with age in the older adults, with increasing age being associated with decreased performance. The second hypothesis was that performance on the BHR-CBB would be worse in individuals who report that they have received a diagnosis of MCI or dementia compared with that in individuals with no diagnosis of cognitive disorder. The third hypothesis was that performance on the BHR-CBB would be associated with levels of self-reported concern of memory problems and a family history of dementia with both factors contributing to worse cognitive performance.

2. Methods

The BHR (BrainHealthRegistry.org) functions within the University of California, San Francisco, and is approved by institutional review board. BHR registrants receive no compensation for completing study procedures. Currently, more than 52,000 participants have registered with the BHR. After registration and giving consent, each participant completes a series of questionnaires, including measures of demographics, overall health, medication use, memory complaints, family history of AD, mood, sleep, diet, and exercise. Participants also complete online neuropsychological tests including the CBB. All cognitive tests and questionnaires are administered online with no supervision, and scores are not reported to participants. All cognitive tests and questionnaires are completed on a voluntary basis, and not all registrants complete all measures.

2.1. Measures of cognition

The CBB is a computerized cognitive assessment battery that has been extensively validated under supervised administrations in a variety of patient populations [9], [19], [20]. All CBB scores used in primary analyses for the study described in the following were obtained consistent with those of previously published methods for supervised administrations [9], [19], [20]. The CBB consists of 4 cognitive tests:

• The detection task (DET): The DET is a measure of psychomotor function and information-processing speed that uses a simple reaction time paradigm with playing-card stimuli. The subject is asked to press the “yes” key as soon as the card in the center of the screen turns face up. The software measures the speed and accuracy of each response. The primary outcome variable for this test is reaction time in milliseconds for correct responses normalized using a logarithmic base 10 (Log 10 transformation).

• The identification test (IDN): The identification task is a measure of visual attention and uses a choice reaction time paradigm with playing-card stimuli. The subject is asked whether the card displayed in the center of the screen is red. The subject responds by pressing the “yes” key when the joker card is red and “no” when it is black. The primary outcome for this test is reaction time in milliseconds for correct responses normalized using a logarithmic base 10 (Log 10 transformation).

• The one card learning test (OCL): The one card learning task is a measure of visual learning and memory that uses a pattern-separation paradigm with playing-card stimuli. In this task, the playing cards are identical to those found in a standard deck of 52 playing cards. The subject is asked whether the card displayed in the center of the screen was seen previously in this task. The subject responds by pressing the “yes” or “no” key. The primary outcome variable is the proportion of correct responses (accuracy) normalized using an arcsine transformation.

• The one-back test (ONB): The one-back task is a measure of working memory and uses a well-validated n-back paradigm with playing-card stimuli. In this task, the playing cards are identical to those found in a standard deck of 52 playing cards. The subject is asked whether the card displayed in the center of the screen is the same as the card presented immediately before. The subject responds by pressing the “yes” or “no” key. The primary outcome variable for this test was accuracy of correct response.

2.2. Family history of memory problems and dementia/memory concern

Family history of AD is obtained for participants based on their response to the following yes/no question: “Please indicate if a parent had or has memory problems including AD and other forms of dementia.” Memory concern was evaluated using the following yes/no question: “Are you concerned that you have a memory problem?”

2.3. Diagnosis of MCI/dementia

Diagnosis of MCI and dementia was obtained from the BHR medical history questionnaire question, “Please indicate whether you currently have or have had any of the following conditions in the past.”

2.4. Statistical analyses

BHR participants over the age of 55 years were included in this analysis. Participants who endorsed diagnoses of major depression and post traumatic stress disorder were excluded (n = 1657). To evaluate the association of diagnostic group, memory concern, and family history of memory problems/dementia with cognitive performance, ordinary least squares regression procedures were employed. Each cognitive test was modeled separately and included age, gender, and education as covariates. Education was parameterized as a three-level factor with groups for those with less than, equal to, or more than 4 years of college. Model fits were inspected by an analysis of the residuals. Receiver operating characteristics analysis was conducted to evaluate the screening performance of demographic characteristics (age, education, and gender) alone and in addition to unsupervised online cognitive test results for classifying diagnosis of MCI and dementia. We show receiver operating characteristics curves and report area under curve as an estimate of classification accuracy for each diagnostic group. All analyses were done in R version 3.1.1 (www.r-project.org).

3. Results

3.1. Sample characteristics

Of the 52,000 registrants who have enrolled in the BHR, 6463 were aged 55 years or older, had completed at least one test in the CBB, completed family history and memory concern questionnaires, and also provided self-reported diagnostic information (cognitively normal, MCI, and AD). These participants were included in subsequent analyses. The mean age of the sample was 66.1 years (standard deviation = 7.1), 70% of the sample was female, and 73% had completed a 4-year college degree (Table 1). In the sample, 89% were identified as white/Caucasian, 2% as black/African-American, and 2% as Asian/Pacific Islander. For the sample, 52% of registrants (n = 3376) reported a family history of memory problem/dementia, and 49% of respondents endorsed memory concerns (Table 1).

Table 1.

Participant characteristics for demographics, memory concerns, family history of memory problems, and dementia (n = 6463)

Variables	Control (CN; n = 5748)	MCI (n = 638)	AD (n = 77)	Total (n = 6463)	P (CN vs. MCI; CN vs. AD)
Mean age (SD)	66.01 (7.07)	66.90 (7.55)	67.25 (7.30)	66.12 (7.13)	.011; .116
Gender					.002; .022
Male	1663 (28.9%)	223 (35.0%)	32 (41.6%)	1918 (29.7%)
Female	4085 (71.1%)	415 (65.0%)	45 (58.4%)	4545 (70.3%)
Education
<16 years/college	1467 (25.5%)	219 (34.3%)	31 (40.3%)	1717 (26.6%)	<.001; <.001
16 years/college	1812 (31.5%)	183 (28.7%)	27 (35.1%)	2022 (31.3%)
>16 years/college	2469 (43.0%)	236 (37.0%)	19 (24.7%)	2724 (42.1%)
Race					.027; .067
White/Caucasian	5106 (88.8%)	585 (91.7%)	63 (81.8 %)	5754 (89.0%)
Black/African-American	124 (2.2%)	9 (1.4%)	2 (2.6%)	135 (2.1%)
Asian and Pacific Islander	143 (2.5%)	10 (1.6%)	2 (2.6%)	155 (2.4%)
Hispanic/Latino	84 (1.5%)	5 (0.8%)	3 (3.9 %)	92 (1.4%)
Native American	59 (1.0%)	6 (0.9%)	2 (2.6%)	67 (1.1%)
More than one race	110 (1.9%)	12 (1.9%)	4 (5.2%)	126 (1.9%)
Memory concern					<.001; <.001
Yes	2500 (43.6%)	574 (90.1%)∗	72 (93.5%)∗	3146 (48.8%)
No	3238 (56.4%)	63 (9.9%)	5 (6.5%)	3306 (51.2%)
Family history of memory problem/dementia					.667; .64
Mother	2261 (39.3%)	251 (40.2%)	28 (36.4%)	2540 (39.4%)
Father	1167 (20.3%)	123 (19.7%)	22 (28.6%)	1312 (20.3%)
Any parent	3011 (52.4%)	325 (52.1%)	40 (51.9%)	3376 (52.3%)

Abbreviations: CN, cognitively normal; MCI, mild cognitive impairment; AD, Alzheimer's disease.

^∗Participants who self-reported a diagnosis of MCI and AD were proportionately more concerned about their memory.

3.2. Association of patient demographic characteristics and self-reported diagnostic group on CBB performance

Across self-reported diagnosis groups, regression results (Table 2) indicate that performance on the test of working memory (ONB) was associated with age, gender, and education. Performance was also associated with MCI diagnosis (β = −0.074, standard error (SE) = 0.013, P < .001) and AD diagnosis (β = −0.173, SE = 0.013, P < .001). Performance on the test of visual learning (OCL) was associated with age, gender, and education; performance was also associated with MCI diagnosis (β = −0.051, SE = 0.007, P < .001) and AD diagnosis (β = −0.090, SE = 0.019, P < .001). Performance on the test of visual attention (IDN) was associated with age and education but not gender; performance was also associated with MCI diagnosis (β = 0.034, SE = 0.003, P < .001) and AD diagnosis (β = 0.050, SE = 0.008, P < .001). Performance on the test of psychomotor function (DET) was associated with age, gender, and education; performance was also associated with MCI diagnosis (β = 0.049, SE = 0.004, P < .001) and AD diagnosis (β = 0.065, SE = 0.012, P < .001). The association of each cognitive test with age is also shown in Fig. 1. Mean performance on CBB for each self-reported diagnostic groups is shown in Fig. 2. Fig. 3 shows the receiver operating characteristics curve for demographic variables (age, education, and gender) alone and in combination with all CBB scores for classification of MCI and dementia participants. The accuracy of predicting MCI diagnosis using demographics alone was 57.8% (95% CI [55.4%–60.1%]), and with online cognitive test performance in addition to demographics, the accuracy increased to 68.3% (95% CI [66.0%–70.6%]; z = −7.98; P < .001). Similarly, when predicting dementia diagnosis, the accuracy using demographics alone was 65.5% (95% confidence interval (CI) [59.4%–71.6%]), and with online cognitive test performance in addition to demographics, the accuracy increased to 76.3% (95% CI [70.4%–82.1%]; z = −4.27; P < .001).

Table 2.

Regression for cognitive test results across diagnostic groups (n = 6463)

	R2	β	SE	P
Memory: ONB	0.029
Age		−0.004	0.001	<.001
Gender		0.044	0.008	<.001
Education (16 years)		0.025	0.010	.011
Education (>16 years)		0.039	0.009	<.001
MCI		−0.074	0.013	<.001
AD		−0.173	0.035	<.001
Learning: OCL	0.047
Age		−0.004	0.000	<.001
Gender		0.011	0.005	.013
Education (16 years)		0.020	0.005	<.001
Education (>16 years)		0.046	0.005	<.001
MCI		−0.050	0.007	<.001
AD		−0.090	0.019	<.001
Attention: IDN	0.060
Age		0.002	0.000	<.001
Gender		0.003	0.002	.016
Education (16 years)		−0.010	0.002	<.001
Education (>16 years)		−0.011	0.002	<.001
MCI		0.034	0.003	<.001
AD		0.050	0.008	<.001
Psychomotor: DET	0.073
Age		0.003	0.000	<.001
Gender		0.025	0.003	<.001
Education (16 years)		−0.013	0.003	<.001
Education (>16 years)		−0.015	0.003	<.001
MCI		0.049	0.004	<.001
AD		0.065	0.011	<.001

Abbreviations: MCI, mild cognitive impairment; AD, Alzheimer's disease; DET, detection task.

Fig. 1.

Associations of cognitive test performance with age (n = 6463). Estimated mean cognitive scores over age shown in red.

Fig. 2.

Cognitive test result differences between self-reported diagnostic groups (n = 6463). Abbreviations: MCI, mild cognitive impairment; AD, Alzheimer's disease. Mean cognitive score and 95% confidence interval shown in red.

Fig. 3.

ROC curve for ability of online cognitive test performance to classify MCI and AD (n = 6463). Abbreviations: MCI, mild cognitive impairment; AD, Alzheimer's disease; ROC, receiver operating characteristics.

3.3. Association of self-reported memory concern with CBB performance

Across self-reported memory concern, results of regression (Table 3) indicated that performance on the test of working memory (ONB) was associated with age, gender, and education. Performance was also associated with having a memory concern (β = −0.050, SE = 0.008, P < .001). Performance on the test of visual learning (OCL) was associated with age, gender, and education; performance was also associated with having a memory concern (β = −0.025, SE = 0.004, P < .001). Performance on the test of visual attention (IDN) was associated with age and education but not gender; performance was also associated with having a memory concern (β = 0.014, SE = 0.002, P < .001). Performance on the test of psychomotor function (DET) was associated with age, gender, and education; performance was also associated with having a memory concern (β = 0.020, SE = 0.003, P < .001).

Table 3.

Regression for cognitive test results and subjective memory concern (n = 6463)

	R2	β	SE	P
Memory: ONB	0.026
Age		−0.005	0.001	<.001
Gender		0.047	0.008	<.001
Education (16 years)		0.025	0.010	.014
Education (>16 years)		0.039	0.009	<.001
Memory concern		−0.050	0.008	<.001
Learning: OCL	0.041
Age		−0.004	0.000	<.001
Gender		0.013	0.005	<.001
Education (16 years)		0.021	0.005	<.001
Education (>16 years)		0.046	0.005	<.001
Memory concern		−0.025	0.004	<.001
Attention: IDN	0.044
Age		0.002	0.000	<.001
Gender		0.001	0.002	.491
Education (16 years)		−0.011	0.002	<.001
Education (>16 years)		−0.011	0.002	<.001
Memory concern		0.014	0.002	<.001
Psychomotor: DET	0.059
Age		0.003	0.000	<.001
Gender		0.023	0.003	<.001
Education (16 years)		−0.015	0.003	<.001
Education (>16 years)		−0.016	0.003	<.001
Memory concern		0.020	0.003	<.001

Abbreviation: DET, detection task.

3.4. Association of family history of memory problems or dementia with CBB performance

Across self-reported family history of memory problems or dementia, there were no significant associations between cognitive test performance and family history of memory problems (P > .05 for all).

4. Discussion

The BHR was created to accelerate clinical neuroscience trials by enrolling a large cohort of adults who provide information on medical, family, and psychiatric history, as well as information on a number of lifestyle factors. This registry is unique because it is not a trial specific registry or a disease-specific effort and can be used by a large number of researchers conducting a wide range of clinical trials. Perhaps most importantly, the BHR is the first registry to use unsupervised online measures of cognitive functioning taken at home which could offer significant advantages for screening and identifying participants for clinical trials for AD and other neurological disorders. Specifically, online cognitive tests taken at home offer the ability to quickly screen a large number of potential research participants for research trials involving cognitive criteria at very low cost. Our most significant findings from this study are results that show that self-reported MCI and AD diagnoses were associated with poorer performance on unsupervised online cognitive tests, and online test performance improved diagnostic classification of participants. In addition, our data suggest that age and registrants' report of cognitive concern may also be associated with cognitive performance across several domains consistent with the extant literature. Finally, our data suggest that family history of dementia was not associated with online cognitive performance. Each of these findings will be discussed in the following.

The identification and screening of eligible participants represents among the most challenging and costly aspect of clinical trials [21], [22], [23], particularly for studies targeting individuals at the earliest stage of neurodegenerative disease. Historically, research registries served as a vital mechanism to accelerate clinical trial research [24], [25], [26], [27], [28]. More recently, online research registries have become critically important for the identification of large numbers of eligible research participants at low cost [29], [30]. However, to date, online measures of cognitive functioning have not been used in the majority of online research registries. Inclusion of online neuropsychological tests in these research registries holds particular promise to facilitate enrollment into clinical trials for which cognitive decline or cognitive symptoms are inclusion criteria given the potential to identify a large number of individuals at low cost. Our primary goal for this study was to evaluate the degree to which a diagnosis of MCI or AD was associated with poorer performance on online neuropsychological tests.

Previous studies have demonstrated that the cognitive tests used in this study have shown group differences related to MCI and AD diagnosis when taken under supervision [9], [11], [12], [13]. However, to date, there have not been studies evaluating the degree to which these tests taken at home in unsupervised settings are associated with self-reported diagnosis of MCI or AD. Our results show that self-reported MCI and AD diagnoses and their associations with poorer performance on each of these tests are consistent with a strong literature base for the validity of these tests across the age span and in many clinical samples [9], [10], [19], [20], [31]. Our results are also consistent with those of a recent study demonstrating that online test performance is comparable with tests given under supervision and can be used as valid estimates of cognitive functioning [17]. Finally, we demonstrated that using online cognitive test results significantly improved diagnostic classification of MCI and AD in this sample. Although these improvements were modest, given the difficulties in identifying and enrolling participants in clinical trials and the low cost of online screening procedures, these results are encouraging. We recognize that further investigation with clinically confirmed diagnoses is necessary, in combination, these results support the potential for online neuropsychological tests to be used to improve screening and recruitment for research studies with specific cognitive criteria in older adults. We also recognize the importance of establishing the clinical validity of these online tests of cognition in future studies. As such, we are currently in the process of conducting a validation study which includes comparisons of online cognitive tests with standard neuropsychological tests, and these results will be made available soon.

As an additional mechanism to explore the potential predictive validity of the online cognitive test data collected, we evaluated how cognitive test performance was associated with participant report of a memory concern. Our data show a significant association of registrants' memory concerns with test performance on all four online cognitive tests. These findings would suggest that registrants with concerns about their memory function may be experiencing some difficulty with several different cognitive domains relative to their peers without memory concern. Such a relationship would be consistent with previous studies demonstrating that perception of memory problems or subjective cognitive complaints are associated with poorer performance on measures of learning and memory, speed of information processing, and working memory [32], [33], [34], [35]. Furthermore, our results would suggest that participant memory concerns, which are easily obtained with online research registries, may be useful for identifying individuals at the earliest stage of cognitive decline for participation in research studies. Finally, our results suggest that participant reports of memory concerns may not be domain specific and instead may reflect perception of more general cognitive difficulties across several domains. In combination, our results suggest that memory complaint in addition to poor online test performance may be particularly useful in identifying individuals experiencing the earliest symptoms of cognitive decline.

In contrast, our findings did not support an association of family history of memory problem or dementia with online cognitive test performance. These results are inconsistent with those of previous studies showing a relationship of family history of dementia with poorer performance on measures of cognitive functioning in older adults [16], [36]. One potential explanation for this discrepancy is that our assessment included an evaluation of family history of memory problems and did not require a specific endorsement of dementia. As such, our results, in conjunction with the extant literature, could be interpreted as suggesting that the association of cognition to family history of dementia may be stronger than a family history of memory problems. Furthermore, these existing studies used traditional neuropsychological assessments that may be more sensitive to subtle cognitive dysfunction among individuals with a family history of AD. Alternatively, our sample of several thousand participants may have offered greater statistical power to evaluate for group difference in cognition associated with family history of memory problems and dementia. Finally, our data showing association with declining cognitive performance on each of the four cognitive tests with increasing age are consistent with a large literature base [37], [38], [39], [40]. Although it is important to clarify that these age effects are subtle and do not reflect cognitive impairment, these results do offer another line of evidence partially supporting the validity for the use of online cognitive tests.

This study has a number of strengths including a large sample size. However, we do recognize that our sample is currently underrepresented for men, individuals of low education level, and with respect to ethnic diversity. In addition, we recognize that our measures of cognitive functioning have not yet been adequately validated for online use, and as such, our results should be interpreted cautiously. In addition, by design, we did not investigate the impact of specific medical illnesses, psychiatric diagnoses, or lifestyle factors on cognitive performance which will be addressed in future investigations. Furthermore, we only included individuals who completed online cognitive tests, and as such, our results may not be representative of the population. Finally, we recognize that our methodology employed self-report of MCI and AD diagnosis and should be replicated using clinically confirmed diagnosis.

To achieve our goal of improving clinical trials research, further validation of these online measures of cognitive functioning is necessary, as is further study of the efficacy of using online test data as a screening tool for clinical trials. However, in aggregate, our data show preliminary support for the use of online research registries that use cognitive assessments as a significant avenue to improve clinical research trials by maximizing the efficiency of participant recruitment. In turn, improvements in participant selection would decrease the time required to complete clinical trials and decrease the cost of such trials—two major obstacles in developing more effective treatments for identifying effective treatments for AD and other neurological disorders.

Research in Context.

• 1.Systematic review: The authors reviewed the literature using traditional (e.g., PubMed) sources and meeting abstracts and presentations. The inclusion of self-administered online cognitive tests within research registries represents an important avenue to improve upon the effectiveness of traditional registry methodology; however, few studies have evaluated if these tests can differentiate clinical populations. Relevant citations are appropriately cited.
• 2.Interpretation: Our findings suggest that age, self-reported diagnostic group (normal, mild cognitive impairment, and dementia), and memory concern were associated with worse performance on unsupervised online neuropsychological tests completed on an online research registry. These results suggest that online neuropsychological tests may be an important tool to improve upon research registry methodology.
• 3.Future directions: Additional studies conducted to determine clinical validity of online neuropsychological tests with clinically defined patient groups will be important to determine the potential effectiveness of using these tests as part of online research registries to improve recruitment and screening for clinical trials.