Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Nov 1.
Published in final edited form as: Clin Neuropsychol. 2017 Jan 12;31(8):1375–1386. doi: 10.1080/13854046.2016.1275819

Convergent and criterion validity of the CogState computerized brief battery cognitive assessment in women with and without breast cancer

Sunita K Patel 1,1,2, Adrienne M Meier 1,1,2, Nathaniel Fernandez 1,1, Tracy T Y Lo 1,1,2, Colleen Moore 1,1, Nicole Delgado 1,1
PMCID: PMC5507751  NIHMSID: NIHMS853948  PMID: 28080264

Abstract

Objective

Computerized tests have increasingly garnered interest for assessing cognitive functioning due to their potential logistical and financial advantages over traditional ‘pencil and-paper’ neuropsychological tests. However, psychometric information is necessary to guide decisions about their clinical and research utility with varied populations. We explored the convergent construct validity and criterion validity of the CogState computerized tests in breast cancer survivors, a group known to present with mostly mild, subtle cognitive dysfunction.

Method

Fifty-three post-menopausal women (26 breast cancer survivors, 27 healthy controls) completed the CogState Brief Battery tests with passed performance checks, conceptually matched traditional neuropsychological tests, and a self-report measure of daily functioning, the Functional Activities Questionnaire.

Results

Significant positive correlations were found between the CogState Brief Battery tests and traditional neuropsychological tests, although the traditional tests specifically hypothesized to correlate with CogState tests did not reach statistical significance. ANCOVA results showed preliminary support for criterion validity, as the patient and control groups differed on the traditional test of working memory (Digits Backwards, p=.01), with a trend towards significance for the CogState test of working memory (One Back, p=.02), controlled for age, race, and mood.

Conclusions

The results provide preliminary support for further research to determine if the CogState tests are viable as screening tools to detect subtle cognitive differences between breast cancer survivors and healthy women. Our study was limited by the low base rate of cognitive impairment and small sample size. We recommend further research employing sufficiently powered sample sizes and a longitudinal, repeated measures study design.

In recent years, computerized cognitive tests have garnered increased interest for assessing cognitive function. These computerized cognitive tests offer potential logistical and financial advantages over traditional ‘pencil-and-paper’ neuropsychological tests. Advantages include standardization of test administration and scoring, increased accessibility for individuals in remote locations, facile randomization of test questions, and reduced personnel costs (Darby et al., 2014). Disadvantages include potential inaccuracies associated with a wider range of testing environments, limitations for individuals without computer familiarity, relatively narrow assessment of cognitive functions, and limited availability of validated computerized tests. Although a wide range of clinical research projects could benefit from the efficiency of computerized tests, few tests have been thoroughly validated for the mild, subtle cognitive impairments typically found in non-central nervous system (CNS) cancer.

CogState’s cognitive measures include up to 14 semi-automated assessment tests for individuals aged 6 to 90. The CogState tests have been used to evaluate cognitive function in Alzheimer’s disease (Grove et al., 2014), schizophrenia (Harvey et al., 2013), ADHD (Mollica, Maruff, & Vance, 2004), HIV (Winston et al., 2012), and cancer (Caine et al., 2016; Conklin et al., 2015, Fliessbach et al., 2010; Noll et al., 2013; Phillips et al., 2010). The CogState tests were developed to be sensitive to cognitive change and are increasingly used in longitudinal clinical trials of pharmaceutical interventions (Collie et al., 2007). The CogState tests can be used alone or can be grouped into batteries to measure specific areas of cognition (e.g., CogHealth, CogSport, etc.). Many of the CogState tests utilize playing cards as the testing stimuli, due to their familiarity, broad applicability across cultures, and lack of language-dependence (Maruff et al., 2009).

The utility and psychometric properties of the CogState tests have been investigated in several studies. The reported test–retest reliability estimates have ranged between .84 and .91 (Collie, Maruff, Darby & McStephen, 2003; Falleti, Maruff, Collie, & Darby, 2006). In one study of healthy individuals (n = 867), the mean difference between the CogState test-retest scores was approximately 2%, much lower than the 7 to 19% difference reported for traditional neuropsychological tests (Makdissi et al., 2001). In addition, practice effects have been described as negligible, based on repeated assessments with intervals greater than one week (Falleti et al., 2006).

The CogState tests have been found to reliably identify impairments in groups of adults with specific cognitive deficits (e.g., dementia, schizophrenia, and HIV/AIDS; Maruff et al., 2009). The CogState Brief Battery detected both the severe impairments characteristic of Alzheimer’s disease (AD; n = 44), and the more subtle deficits associated with mild cognitive impairment (MCI; n = 68), in a cross-sectional study of 653 community-dwelling Australian older adults (> 60 years old). The CogState performance differences between the MCI and AD groups persisted even after controlling for covariates such as depression. Based on these results, the authors recommended the CogState Brief Battery as a screening tool for the cognitive changes associated with Alzheimer’s disease (Lim et al., 2012).

Several studies have compared the CogState cognitive tests with traditional neuropsychological tests to examine discrepancies in the detection of subtle cognitive impairment (criterion validity). De Jager et al. (2009) evaluated whether the CogState tests were as sensitive to MCI as two well-validated traditional tests, the Hopkins Verbal Learning Test (HVLT) and the Mini-Mental Status Examination (MMSE). These tests were administered to 21 individuals with MCI and 98 cognitively healthy controls (HC). The participants also completed self-report questionnaires, including an Activities of Daily Living Scale. The authors reported that the maximum discrimination between controls and MCI individuals was 86% for the CogState tests, 90% for HVLT, and 65% for MMSE. The CogState test results were significantly correlated with HVLT and Activities of Daily Living. Overall, these results support more investigation of the CogState tests in multiple patient populations.

The goal of the present study was to extend validation of the CogState tests to a new population: BC survivors. Approximately 30% of survivors treated for non-CNS cancers show mild cognitive deficits (Wefel & Schagen, 2012). BC is the most prevalent of all cancers and treatment-related cognitive effects, particularly in memory and executive functioning, have been demonstrated repeatedly over the past two decades (Tannock, Ahles, Ganz, & Van Dam, 2004; Wefel & Schagen, 2012). Tannock et al. (2004) proposed that the CogState tests might facilitate screening for chemotherapy-induced cognitive changes in large, multi-center trials. However, few studies have validated the CogState tests for individuals with non-CNS cancer. One study (Vardy et al., 2006) showed equivocal results for chemotherapy-associated cognitive dysfunction in a population of 31 Canadian breast cancer survivors aged 31 to 65. This study compared performance on the CogState tests to a different computerized battery, and did not compare the results to performance on traditional cognitive tests.

The handful of studies that have reported the use of CogState tests with non-CNS cancer patients have used varied criteria to define cognitive impairment and most did not (a) compare performance on CogState tests with traditional neuropsychological tests, (b) include a HC group for comparison, or (c) evaluate associations with real-world functioning. In the current investigation, we explored the convergent construct validity and criterion validity of the CogState tests for women with BC. We also explored the association of CogState tests with self-report of daily functioning.

Methods

Participants

The participants were post-menopausal women enrolled in a larger, funded study on the association of biomarkers and cognitive functioning in breast cancer survivors at City of Hope Medical Center (Patel et al., 2015). For the current sub-study, IRB approval was obtained to recruit participants who returned for the final assessment on the longitudinal parent study. CogState computerized testing was performed after the traditional neuropsychological assessment, and other procedures associated with the parent study, were completed. Of the 76 women eligible for the CogState tests sub-study, 66 (87%) participated (n = 35 BC participants and n = 31 HC participants without cancer or other severe illnesses); the participant’s lack of sufficient time was the primary reason for non-participation. Both the BC and HC groups had participated in identical study procedures for the parent study, including three brief assessments with traditional neuropsychological measures at matched time intervals. Recruitment for the sub-study occurred at the fourth assessment, which was the final assessment on the parent study.

Procedures

CogState Computerized Tests

The CogState Brief Battery (15 minutes) was administered to participants in person using a laptop computer. A trained research assistant (RA) oriented the participants to each test using standardized procedures, which consisted of reading the instructions and rules for each subtest from the laptop screen and demonstrating how to respond using the mouse with their dominant hand. Verbal and written instructions urged the participant to “respond as fast and as accurately as possible.” The CogState Brief Battery involves a brief practice trial prior to each subtest. A distinctive beep sounded when a participant made an error or pressed an incorrect key (i.e., feedback) during the practice trial. The feedback during the CogState practice trials is analogous to feedback during practice trials for traditional tests, although the feedback is provided by the examiner in traditional tests, and by the computer in the CogState tests and supplemented by the examiner as needed. The participants were required to respond either ‘yes’ or ‘no’ for various tasks by either left- or right-clicking the mouse using their dominant hand, respectively.

The CogState Brief Battery comprises four tests (Maruff et al., 2009), which are described below. Age-based normative data for this battery were derived from healthy participants from ages 18 to 89. The CogState software records response speed (in milliseconds) and accuracy (number correct or incorrect) for each subtest. The software then scores the tests and automatically transforms both speed and accuracy into normalized distributions. The results are interpreted by an appropriately trained professional (Maruff et al., 2009).

Detection (DET) is a test of reaction time that assesses psychomotor processing speed. Participants quickly right-click on the mouse when the playing card (a joker) turns face up. The participants’ mean reaction time (speed, log10 transformed) for correct responses is the outcome. Lower scores indicate better performance.

Identification (IDN) is a reaction time test that assesses attention. Participants identify whether the joker card presented is red. Participants’ mean reaction time (speed, log10 transformed) for correct responses is the outcome. Lower scores indicate better performance.

One Back (ONB) assesses working memory. Participants identify whether the card currently face-up is identical to the previous card. All of the playing cards are used in this task. The arcsine-transformed square root of the proportion of correct responses (accuracy) is the outcome used in this study, with higher scores indicating better accuracy. Many studies use speed as the primary outcome measure for this test.

One Card Learning (OCL) assesses short-term visual learning and memory. Participants identify whether the card face-up has ever appeared before during the task. Six repeating playing cards are interspersed with eight non-repeating cards, for a total of 14 different cards across the 42 trials of this task. The arcsine-transformed square root of the proportion of correct responses (accuracy) is the outcome, with higher scores indicating better accuracy.

Traditional ‘Paper and Pencil’ Neuropsychological Tests

A trained RA administered the neuropsychological tests in person to participants, according to standardized procedures outlined in the developer’s manuals, and scored the responses according to the developer guidelines. For this study, the tests were selected a priori from the parent study’s larger battery to conceptually match the cognitive functions measured by the CogState tests, based on the developer’s manual version 6 and descriptions from previous studies (Maruff et al., 2009) as follows: (a) processing speed: the WAIS-IV Processing Speed Index (PSI, consisting of the Symbol Search and Digit Symbol Coding subtests) was matched with DET; (b) attention: the WAIS-IV Digit Span Forward (DSF) test was matched with IDN; (c) working memory: the WAIS-IV Digit Span Backward (DSB) test was matched with ONB; and (d) learning and memory: the Hopkins Verbal Learning Test-Revised immediate total recall score (HVLT-R; Brandt & Benedict, 2001) was matched with OCL. The raw scores were converted to standardized scores using published normative data, and scoring was double-checked by a second scorer for accuracy (i.e., inter-reliability). Higher scores indicate better performance.

Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV; Wechsler, 2008)

The PSI, DSF, and DSB subtests were used in the current study. The PSI consists of the Symbol Search and Digit Symbol Coding subtests. In Symbol Search, participants are instructed to quickly scan a row of symbols and determine whether it includes either of the two target symbols. In Digit Symbol Coding, participants write down the correct symbols in the blank boxes corresponding with the digits presented in a key at the top. DSF assesses attention. Participants are asked to repeat numbers in the order in which they were read, with each subsequent trial increasing in complexity by one numerical digit. DSB assesses working memory and requires participants to repeat numbers in reverse order.

Hopkins Verbal Learning Test-Revised (HVLT-R;Brandt & Benedict, 2001) is a standardized verbal learning and memory test, consisting of three immediate recall trials (total recall) as well as a long delay (delayed recall). Participants are asked to recall words from a list of 12 words that are read aloud. The score from the immediate recall phase of the task (total recall) represents the participants’ learning and memory performance.

Functional Activities Questionnaire (FAQ;Pfeffer, Kurosadi, Harrah, Chance, & Filos, 1982) is a 10-item self-report measure that assesses instrumental activities of daily living and complex cognitive/social functions related to managing finances, appointments, and events. Participants rate their function on a 4-point scale, where 3 = dependent, 2 = requires assistance, 1 = has difficulty but does by self, and 0 = normal. Total scores range from 0 to 30, with higher scores indicating greater dependence on others. Compared to other daily living questionnaires, the FAQ assesses higher levels of “instrumental” functioning and is considered a more appropriate tool for mild, rather than severe, cognitive deficits (Marson & Herbert, 2006).

Demographic and Health Questionnaire

As part of the parent study, the participants completed a questionnaire on demographic factors such as age, race/ethnicity, and education, as well as health-related factors such as Body Mass Index (BMI) and the presence of comorbid chronic health conditions.

Brief Symptom Inventory (BSI-18)

As part of the parent study, the participants completed a standardized, self-report measure to assess mood (Derogatis, 2001). The BSI consists of three subscales (depression, anxiety, somatic) and generates an overall score: Global Severity Index (GSI), which summarizes the overall level of emotional distress. Higher scores indicate higher distress.

Analytical Approach

The CogState performance scores were examined for failed performance “integrity” checks, which assess whether participants failed to understand the task or put forth sufficient effort. The recommended cut offs to estimate valid performance for the CogState tests are provided in the developers’ manual (CogState Ltd, 2011), and were derived from probabilistic estimates based on population-specific data, including data from healthy participants (A. Schembri, personal communication, July 13, 2016). In consultation with the CogState research team, we used the recommended cut offs most relevant to our population, which assessed whether: (a) DET accuracy was >80%; (b) IDN accuracy was >80%; (c) ONB accuracy was >70%; and (d) OCL accuracy was >50%. Data for individuals with one or more failures were omitted from analyses, using the conservative criteria provided by the developers (CogState Ltd, 2011).

The distributions of scores obtained on the CogState tests and traditional neuropsychological tests were checked for outliers and normality and was found to be normal, therefore no transformations were applied. For ease of interpretation, the cognitive scores were converted to z-scores, with higher scores representing better performance. Demographic and health characteristics (e.g., age, education, BMI, comorbidity, and mood) between the BC and HC groups were examined using t-tests and chi-square tests.

Analyses of convergent validity included partial Pearson correlations between scores obtained on the four CogState tests and the four traditional neuropsychological tests selected a priori as conceptual matches, controlled for any demographic or health characteristics that were different between the control and patient groups. We also evaluated correlations between tests that were not defined as conceptual matches.

To determine if the CogState tests were similar to traditional neuropsychological tests in assessing cognitive functioning, separate Analysis of Covariance (ANCOVA) models were used. These models evaluated differences between the BC and HC groups on cognitive functioning, as assessed by the traditional tests and by the CogState tests (criterion validity). Between-group analyses controlled for any demographic or health characteristics that were different between the BC and HC groups. For any group differences that emerged on the CogState and traditional neuropsychological tests, we examined the association with self-reported daily functional status using Spearman correlations, because the distribution of FAQ scores was skewed. All tests of statistical significance were two-sided with a significance level of p ≤ .01. We chose a significance threshold of .01, rather than a Bonferroni multiple comparisons correction, to guard against overly conservative corrections, which could increase the likelihood of Type II errors in this exploratory study. Bonferroni correction for the 14 tests of significance would place the significance threshold at p ≤ .003 (0.05/14).

Results

Each participant’s CogState test performance was evaluated using the embedded integrity checks provided by the CogState developers (CogState Ltd, 2011). Participants who did not meet the criteria were removed from the final analyses. Of the 66 consented participants for this sub-study, 13 (20%) failed at least one of the four CogState integrity checks. Investigation of trends for failed integrity checks revealed that 12 participants (18%) failed one integrity check, whereas one participant (1%) failed two integrity checks. An exploration of failed integrity checks by subtest revealed that four participants failed DET, one failed IDN, three failed ONB, and five failed OCL.

The final sample consisted of 53 female participants, (26 BC and 27 HC), who were predominantly White/Caucasian (68%), with a mean age of 62.98 (SD = 7.51), and mean education of 14.51 years (SD = 1.66). The BC participants were significantly younger than the HC participants (p = .04), and a significantly greater percentage were non-Caucasian. p = .03). In addition, the BC participants were significantly more emotionally distressed (p < .01). See Table 1. Although older age is associated with a BC diagnosis, with a significant increase in post-menopausal women, the average magnitude of the difference in our sample was only four years, and all of the women were post-menopausal. Furthermore, BC incidence rates are generally reported as lower in ethnic minority women than in White/Caucasian women, though this may partly reflect screening differences (Kohler et al., 2015). Therefore, these group differences (age, race) were judged to be random effects and controlled for statistically in subsequent analyses. Increased emotional distress, however, has consistently been associated with cancer status and is likely to be non-random (Bower, 2008) therefore, mood was also used as a covariate in the analyses to control for possible effects on cognitive performance.

Table 1.

Demographic and clinical characteristics

Variable HC (n = 27) BC (n = 26)
Age of patient at time of assessment, yrs, mean (SD)a* 65.07 (8.57) 60.81 (5.59)
Race/Ethnicityb*
 White/Caucasian, n (%) 22 (81.5%) 14 (53.8%)
 Non-White/Caucasian, n (%) 5 (18.5%) 12 (46.2%)
Years of education, mean (SD) 14.67 (1.55) 14.35 (1.79)
Body mass index, mean (SD) 28.79 (6.75) 29.19 (5.10)
Charlson Comorbidity Index, mean (SD) 0.59 (0.75) 0.81 (0.90)
Brief Symptom Inventory, mean (SD)c* 42 (7.6) 49 (10.5)
Functional Activities Questionnaire, mean (SD)d 0.44 (0.94) 1.77 (4.45)
Open in a new tab

HC=Healthy Controls; BC=Breast Cancer

a

t(51) = 2.14, p = .037

b

χ2 = 4.64, p = .031

c

t(51) = 2.45, p = .005

d

t(51) = −1.53, p = .135

*

p < .05

Convergent Validity

Significant positive partial correlations were observed between three of the four CogState tests and at least one traditional neuropsychological test. However, none of the four a priori conceptually matched test pairs met the p ≤ .01 criterion for significance (Table 2). CogState IDN (attention) was significantly associated with WAIS-IV PSI (processing speed); OCL (learning and memory) was significantly associated with two of the four traditional tests (digit span backwards and processing speed). Also, ONB (working memory) was significantly associated with WAIS-IV PSI (processing speed) and HVLT-R (learning and memory) but not with its matched traditional test (WAIS-IV DSB).

Table 2.

Partial correlations between cognitive domains assessed by CogState and traditional neuropsychological tests, controlled for age, race, and mood (n = 53)

CogState Tests Traditional Neuropsychological Tests
WAIS-IV PSI (Processing Speed) WAIS-IV DSF (Attention) HVLT-R Immediate Recall (Learning/Memory) WAIS-IV DSB (Working Memory)
Detection/DET (Processing Speed) .180 (p=.16) .224 (p=.08) .246 (p=.06) .156 (p=.23)
Identification/IDN (Attention) .449** (p<.001) .212 (p=.10) .187 (p=.15) .217 (p=.09)
One Card Learning/OCL (Learning and Memory) .354** (p<.01) .281 (p=.03) .285 (p=.03) .339** (p<.01)
One Back/ONB (Working Memory) .395** (p<.01) .058 (p=.66) .329** (p=.01) .158 (p=.22)
Open in a new tab
**

p ≤ .01.

Criterion Validity

The ANCOVA results showed significant differences between HC and BC participants on the traditional measure of working memory (WAIS-IV DSB; p = .01); differences on the CogState test of working memory did not reach the p = .01 criterion but trended toward significance (ONB; p = .02). In all cases, BC survivors scored lower than HCs (Table 3).

Table 3.

ANCOVA results for performance differences between healthy control (HC) and breast cancer (BC) groups assessed by traditional neuropsychological tests and CogState tests, controlled for age, race, and mood.

Outcome HC (n = 27) BC (n=26) F Df p ηp2
Mean SD Mean SD
Processing Speed
 WAIS-IV PSI 1.23 0.90 0.77 0.90 1.01 1,48 0.32 0.02
 CogState DET −0.55 0.74 −0.88 0.80 2.62 1,48 0.11 0.05
Attention
 WAIS-IV DSF 0.37 0.65 0.10 0.93 1.63 1,48 0.21 0.03
 CogState IDN −0.72 0.91 −1.08 0.97 1.44 1,48 0.24 0.03
Learning/Memory
 HVLT-R 0.66 0.84 −0.11 1.29 2.04 1,47 0.16 0.04
 CogState OCL −0.01 0.91 −0.19 0.89 0.54 1,48 0.54 0.01
Working Memory
 WAIS-IV DSB 0.63 0.71 0.00 0.76 6.70 1,48 0.01** 0.12
 CogState ONB 0.52 0.82 −0.22 1.05 6.23 1,48 0.02* 0.11
Open in a new tab
**

p ≤ .01

*

p ≤ .05.

Note. For ease of interpretation, all scores converted to Z scores with higher scores indicating better performance.

Self-reported Functional Status

Spearman’s rho correlations between FAQ and each of the two tests that suggested between-group differences (WAIS-IV DSB, and CogState ONB) were not significant (ρ = −.11, p = .30 ρ = −.04, p = .74; respectively). Notably, because most participants had no difficulties with daily function, the range of the FAQ scores was restricted, and may have impaired detection of correlations.

Discussion

Computerized cognitive tests have been proposed to improve the efficiency of large clinical trials (Tannock et al. 2004). However, these computerized tests have not been well validated in populations with subtle cognitive dysfunction, such as survivors of non-CNS cancer. Therefore, we explored the psychometric properties of the CogState computerized Brief Battery tests in women with and without BC. Specifically, we (a) examined the concordance between CogState tests and their a priori conceptually matched traditional neuropsychological tests (convergent validity); (b) investigated whether between-group differences on traditional neuropsychological tests would also be found on a priori conceptually matched CogState tests (criterion validity); and (c) examined the association of key tests with self-report of daily functioning. All of the analyses were controlled for group differences (age, race, and mood).

The CogState tests did not consistently correspond with their a priori conceptually matched tests, suggesting that there is limited support for convergent validity, as defined for this analysis. However, the comparison may have been limited by differences in the test modalities; the Cogstate Brief Battery tests are all visual, whereas all of the traditional tests in this study are auditory, except for PSI. The WAIS-IV PSI comprises two timed visual motor tasks and was correlated with most of the CogState tests, suggesting that comparison with traditional timed visual tests may have shown better convergence.

Of note, the two conceptually matched tests of working memory, CogState ONB and WAIS-IV DSB, were not correlated. Instead, WAIS-IV DSB was correlated with the CogState OCL test, which is a visual learning and memory test that also requires working memory to identify if a card has been presented previously. It is not clear why CogState ONB was better correlated with WAIS-IV PSI and HVLT-R than it was with WAIS-IV DSB. Although CogState ONB and WAIS-IV PSI are both timed visual motor tests, HVLT-R involves auditory/verbal processes and is not timed. Perhaps the correlations reflect shared variance with an underlying broad mental ability, similar to the “g” factor, which is thought to influence performance on tests of cognitive abilities that are seemingly unrelated (Sternberg & Grigorenko, 2002). Furthermore, three of the four CogState computerized tests showed significant positive correlations with at least one traditional, paper-and-pencil test, providing preliminary support for future analysis of convergent validity with traditional tests.

Analyses for criterion validity showed that after controlling for age, race, and mood, the CogState ONB working memory test showed trends (p = .02) toward detection of the subtle cognitive differences between our BC and HC groups, similar to the WAIS-IV DSB test (p = .01), a well-established traditional test of working memory. To confirm the presence of between-group differences in cognitive functioning, we conducted similar analyses using data from the final assessment of the parent study. After controlling for the same covariates, the BC survivors (n = 136) scored significantly lower than the HC group (n = 72) for WAIS-IV DSB (p < .01).

Limitations of our study include the use of a convenience sample with unequal prior exposure to the instruments. Specifically, the participants had taken the traditional tests four times prior to the CogState tests, and the fourth time occurred immediately prior to the CogState test administration in the same assessment session. As such, there may be practice effects on performance for the traditional measures but not the CogState tests. We also did not counterbalance administration of the CogState and traditional tests, and it is possible that women were more fatigued by the time they took the CogState tests.

Another limitation is that we did not assess the participants’ comfort level with computerized tests. A 20% failure rate is high and the reasons are unknown. It is also possible that the computerized feedback provided during the brief practice trials administered at the beginning of each subtest may not have provided sufficient training, particularly for older participants. However, we did not identify any systematic differences when we statistically compared various characteristics, including age, for those who failed the integrity checks compared to those who passed. Recent studies have tended to administer CogState tests twice, where the first administration is viewed as a familiarity trial to enhance participants’ understanding and preparation (Winston et al., 2012). This is particularly important for trials intended to examine change as the double administration also minimizes practice effects on subsequent assessments. It is possible that employing a similar procedure in this cross-sectional study may have reduced the high failure rate. In exploratory analyses, we re-evaluated CogState ONB performance without removing the participants who did not pass the integrity checks, and found that including all participants produced a similar trend toward significance (p = .03).

Despite the study limitations, the results from this exploratory investigation encourage further investigation of the CogState tests as potential screening tools for mild cognitive dysfunction. We recommend future research with larger sample sizes and longitudinal designs to better evaluate the stability of sensitivity across time. We also recommend examining the impact of practice effects, if any, on the relationships between measures and controlling for these effects. Longitudinal designs will also be important for assessing how the computerized screening tests perform under varied conditions, such as following interventions. Overall, it is possible that the CogState tests may be promising tools for clinical research, but further analysis is necessary to determine their sensitivity to the mild cognitive dysfunction associated with non-CNS cancer.

Acknowledgments

This work was supported in part by an American Cancer Society Research Scholar grant # 12-049-01-CPPB (S.P). Research reported in this publication includes work performed in the City of Hope Survey Core, which is supported by the National Cancer Institute of the National Institutes of Health under award number P30CA033572. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Editorial assistance was provided by Nancy Linford, PhD.

Footnotes

Conflict of Interest Statement: The authors have no financial disclosures for this manuscript.

References

  1. Bower JE. Behavioral symptoms in patients with breast cancer and survivors. Journal of Clinical Oncology. 2008;26(5):768–777. doi: 10.1200/JCO.2007.14.3248. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Brandt J, Benedict HB. The Hopkins Verbal Learning Test-Revised: Professional Manual. Lutz, FL: Psychological Assessment Resources; 2001. [Google Scholar]
  3. Caine C, Deshmukh S, Gondi V, Mehta M, Tome W, Corn BW, … Kachnic L. CogState computerized memory tests in patients with brain metastases: Secondary endpoint results of NRG oncology RTOG 0933. Journal of Neurooncology. 2016;126(2):327–336. doi: 10.1007/s11060-015-1971-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Cargin JW, Maruff P, Collie A, Masters C. Mild memory impairment in healthy older adults is distinct from normal aging. Brain and Cognition. 2006;60(2):146–155. doi: 10.1016/j.bandc.2005.10.004. [DOI] [PubMed] [Google Scholar]
  5. CogState Limited. CogState research manual version 6. Melbourne, Australia: 2011. [Google Scholar]
  6. Collie A, Darekar A, Weissgerber G, Toh MK, Snyder PJ, Maruff P, Huggins JP. Cognitive testing in early-phase clinical trials: Development of a rapid computerized test battery and application in a simulated Phase I study. Contemporary Clinical Trials. 2007;28:391–400. doi: 10.1016/j.cct.2006.10.010. [DOI] [PubMed] [Google Scholar]
  7. Collie A, Maruff P, Darby DG, McStephen M. The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test-retest intervals. Journal of the International Neuropsychological Society. 2003;9(3):419–428. doi: 10.1017/S1355617703930074. doi:10.10170S1355617703930074. [DOI] [PubMed] [Google Scholar]
  8. Conklin HM, Ashford JM, Di Pinto M, Vaughan CG, Gioia GA, Merchant TE, … Wu S. Computerized assessment of cognitive late effects among adolescent brain tumor survivors. Journal of Neurooncology. 2013;113(2):333–340. doi: 10.1007/s11060-013-1123-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Darby DG, Fredrickson J, Pietrzak RH, Maruff P, Woodward M, Brodtmann A. Reliability and usability of an internet-based computerized cognitive testing battery in community-dwelling older people. Computers in Human Behavior. 2014;30:199–205. doi: 10.1016/j.chb.2013.08.009. [DOI] [Google Scholar]
  10. De Jager CA, Schrijnemaekers AC, Honey T, Budge MM. Detection of MCI in the clinic: Evaluation of the sensitivity and specificity of a computerized test battery, the Hopkins Verbal Learning Test and the MMSE. Age and Ageing. 2009;38:455–460. doi: 10.1093/ageing/afp068. [DOI] [PubMed] [Google Scholar]
  11. Falleti MG, Maruff P, Collie A, Darby DG, McStephen M. Qualitative similarities in cognitive impairment associated with 24 h of sustained wakefulness and a blood alcohol concentration of 0.05% Journal of Sleep Research. 2003;12(4):265–274. doi: 10.1111/j.1365-2869.2003.00363.x. [DOI] [PubMed] [Google Scholar]
  12. Falleti MG, Maruff P, Collie A, Darby D. Practice Effects Associated with the Repeated Assessment of Cognitive Function Using the CogState Battery at 10-minute, One Week, and One Month Test-retest Intervals. Journal of Clinical and Experimental Neuropsychology. 2006;28(7):1095–112. doi: 10.1080/13803390500205718. [DOI] [PubMed] [Google Scholar]
  13. Fliessbach K, Rogowski S, Hoppe C, Sabel M, Goeppert M, Helmstaedter C, … Schlegel U. Computer-based assessment of cognitive functions in brain tumor patients. Journal of Neuro-oncology. 2010;100(3):427–437. doi: 10.1007/s11060-010-0194-9. [DOI] [PubMed] [Google Scholar]
  14. Franzen MD. Reliability and Validity in Neuropsychological Assessment. 2. Klewer Academic/Plenum Publishers; New York, NY: 2002. [Google Scholar]
  15. Grove RA, Harrington CM, Mahler A, Beresford I, Maruff P, Lowy MT, … Horrigan JP. A randomized, double-blind, placebo-controlled, 16-week study of the H3 receptor antagonist, GSK239512 as a monotherapy in subjects with mild-to-moderate Alzheimer’s Disease. Current Alzheimer Research. 2014;11(1):47–58. doi: 10.2174/1567205010666131212110148#sthash.lQmmoZ64.dpuf. [DOI] [PubMed] [Google Scholar]
  16. Harvey PD, Siu CO, Hsu J, Cucchiaro J, Maruff P, Loebel A. Effect of lurasidone on neurocognitive performance in patients with Schizophrenia: A short-term placebo- and active-controlled study followed by a 6-month double-blind extension. European Neuropsychopharmacology. 2013;23(11):1373–1382. doi: 10.1016/j.euroneuro.2013.08.003. [DOI] [PubMed] [Google Scholar]
  17. Kohler BA, Sherman RL, Howlader N, Jemal A, Ryerson AB, Henry KA, … Henley SJ. Annual report to the nation on the status of cancer, 1975–2011, featuring incidence of breast cancer subtypes by race/ethnicity, poverty, and state. Journal of the National Cancer Institute. 2015;107(6):djv048. doi: 10.1093/jnci/djv048. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Lim YY, Ellis KA, Harrington K, Ames D, Martins RN, Masters CL … AIBL Research Group. Use of the CogState Brief Battery in the assessment of Alzheimer’s disease related cognitive impairment in the Australian Imaging, Biomarkers, and Lifestyle (AIBL) study. Journal of Clinical and Experimental Neuropsychology. 2012;34(4):345–358. doi: 10.1080/13803395.2011.643227. [DOI] [PubMed] [Google Scholar]
  19. Makdissi M, Collie A, Maruff P, Darby DG, Bush A, McCrory P, Bennell K. Computerised cognitive assessment of concussed Australian Rules footballers. British Journal of Sports Medicine. 2001;35:354–360. doi: 10.1136/bjsm.35.5.354. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Marson D, Herbert KR. Functional assessment. In: Attix D, Welsh-Bhomer K, editors. Geriatric neuropsychology: Assessment and intervention. New York, New York: The Guilford Press; 2006. pp. 158–197. [Google Scholar]
  21. Maruff P, Collie A, Darby D, Weaver-Cargin J, Masters C, Currie J. Subtle memory decline over 12 months in mild cognitive impairment. Dementia Geriatric Cognitive Disorders. 2004;18:342–348. doi: 10.1159/000080229. [DOI] [PubMed] [Google Scholar]
  22. Maruff P, Thomas E, Cysique L, Brew B, Collie A, Snyder P, Pietrzak RH. Validity of the CogState Brief Battery: Relationship to standardized tests and sensitivity to cognitive impairment in mild traumatic brain injury, Schizophrenia, and AIDS dementia complex. Archives of Clinical Neuropsychology. 2009;24(2):165–178. doi: 10.1093/arclin/acp010. [DOI] [PubMed] [Google Scholar]
  23. Mollica CM, Maruff P, Vance P. Development of a statistical approach to classifying treatment response in individual children with ADHD. Human Psychopharmacology. 2004;19(7):445–456. doi: 10.1002/hup.624. [DOI] [PubMed] [Google Scholar]
  24. Noll RB, Patel SK, Embry L, Hardy KK, Pelletier W, Annett RD, … Barakat LP. Children’s Oncology Group’s 2013 blueprint for research: Behavioral science. Pediatric Blood & Cancer. 2013;60(6):1048–1054. doi: 10.1002/pbc.24421. [DOI] [PubMed] [Google Scholar]
  25. Patel SK, Wong AL, Wong FL, Breen EC, Hurria A, Smith M, … Bhatia S. Inflammatory biomarkers, comorbidity, and neurocognition in women with newly diagnosed breast cancer. Journal of National Cancer Institute. 2015;107(8):djv131. doi: 10.1093/jnci/djv131. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Pfeffer RI, Kurosadi TT, Harrah CH, Jr, Chance JM, Filos S. Measurement of functional activities in older adults in the community. Journal of Gerontology. 1982;37(3):323–329. doi: 10.1093/geronj/37.3.323. [DOI] [PubMed] [Google Scholar]
  27. Phillips KA, Ribi K, Sun Z, Stephens A, Thompson A, Harvey V, … Bernhard J. Cognitive function in postmenopausal women receiving adjuvant letrozole or maoxifen for breast cancer in the BIG 1–98 randomized trial. Breast. 2010;19(5):388–395. doi: 10.1016/j.breast.2010.03.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Sternber RJ, Grigorenko EL, editors. The general factor of intelligence: How general is it? Mahwah, New Jersey: Erlbaum; 2002. [Google Scholar]
  29. Tannock IF, Ahles TA, Ganz PA, Van Dam FS. Cognitive impairment associated with chemotherapy for cancer: Report of a workshop. Journal of Clinical Oncology. 2004;22(11):2233–9. doi: 10.1200/JCO.2004.08.094. [DOI] [PubMed] [Google Scholar]
  30. Vardy J, Wong K, Yi Q, Park A, Maruff P, Wagner L, Tannock IF. Assessing cognitive function in cancer patients. Support Care Cancer. 2006;14:1111–1118. doi: 10.1007/s00520-006-0037-6. [DOI] [PubMed] [Google Scholar]
  31. Wechsler D. Wechsler Adult Intelligence Scale-Fourth Edition: Technical and interpretive manual. San Antonio, TX: Pearson; 2008. [Google Scholar]
  32. Wefel JS, Schagen SB. Chemotherapy-related dysfunction. Current Neurology & Neuroscience Report. 2012;12(3):267–275. doi: 10.1007/s11910-012-0264-9. [DOI] [PubMed] [Google Scholar]
  33. Winston A, Puls R, Kerr SJ, Duncombe C, Li P, Gill JM, … Cooper DA. Dynamics of cognitive change in HIV-infected individuals commencing three different initial antiretroviral regimens: A randomized, controlled study. HIV Medicine. 2012;13(4):245–251. doi: 10.1111/j.1468-1293.2011.00962.x. [DOI] [PubMed] [Google Scholar]

RESOURCES