CogState computerized memory tests in patients with brain metastases: secondary endpoint results of NRG Oncology RTOG 0933

Abstract

Whole brain radiotherapy (WBRT) is associated with memory dysfunction. As part of NRG Oncology RTOG 0933, a phase II study of WBRT for brain metastases that conformally avoided the hippocampal stem cell compartment (HA-WBRT), memory was assessed pre- and post-HA-WBRT using both traditional and computerized memory tests. We examined whether the computerized tests yielded similar findings and might serve as possible alternatives for assessment of memory in multi-institution clinical trials. Adult patients with brain metastases received HA-WBRT to 30 Gy in ten fractions and completed Hopkins Verbal Learning Test-Revised (HVLT-R), CogState International Shopping List Test (ISLT) and One Card Learning Test (OCLT), at baseline, 2 and 4 months. Tests’ completion rates were 52–53 % at 2 months and 34–42 % at 4 months. All baseline correlations between HVLT-R and CogState tests were significant (p ≤ 0.003). At baseline, both CogState tests and one component of HVLT-R differentiated those who were alive at 6 months and those who had died (p ≤ 0.01). At 4 months, mean relative decline was 7.0 % for HVLT-R Delayed Recall and 18.0 % for ISLT Delayed Recall. OCLT showed an 8.0 % increase. A reliable change index found no significant changes from baseline to 2 and 4 months for ISLT Delayed Recall (z = −0.40, p = 0.34; z = −0.68, p = 0.25) or OCLT (z = 0.15, p = 0.56; z = 0.41, p = 0.66). Study findings support the possibility that hippocampal avoidance may be associated with preservation of memory test performance, and that these computerized tests also may be useful and valid memory assessments in multi-institution adult brain tumor trials.

Introduction

Formal measurement of neurocognitive functioning (NCF) is included in a subset of brain cancer trials to quantify impact of tumors and treatments on cognition, particularly memory. While there is no shortage of NCF tests, a handful are familiar to many investigators, as they originated as early as the 1940s and have appeared often in important recent trials. One distinct advantage of consistent use of a small number of tests is that doing so promotes efforts to standardize NCF activity across studies [1].

At the same time, strict adherence to a particular protocol may paradoxically discourage inclusion of NCF in trials. NCF endpoints are often viewed as burdensome for both patients and staff [2], yet NCF can be defined and measured in many ways, and while one study may require detailed examination of cognition to address questions of interest properly, another might require only selective examination or even use of a single proxy. Another unintended consequence of limiting trials to a particular subset of tests is that it may deter innovation within the choice of tests used for measurement. For example, the suitability of some NCF tests for repeated administration for purposes of detecting change over time may be questioned [3].

When NCF is included in a trial, often it serves as the primary endpoint, and memory, generally the auditory/verbal domain, is frequently the focus, as it is a major component of quality of life [4] and susceptible to early and sometimes severe deterioration due to tumors [5] and their treatments [6]. Paper and pencil tests are most often chosen for memory assessment in adult cancer patients; although, computerized tests are used often in some brain conditions where cognition is compromised, such as concussion [7] and Alzheimer’s disease [8]. Recently, computerized tests have begun to be incorporated into cancer studies [9, 10], including pediatric populations [11].

As part of NRG Oncology RTOG 0933 [12], a phase II study of patients with brain metastases who underwent whole brain radiotherapy (WBRT) but with conformal avoidance of the peri-hippocampal stem cell region (HA-WBRT), two computerized tests of memory were included as secondary endpoints, along with the Hopkins Verbal Learning Test-Revised (HVLT-R) [13], a traditional test of memory used often in adult cancer trials that measured the study’s primary endpoint. Of interest was whether the computerized tests yield findings similar to HVLT-R and whether they might serve as possible alternatives for assessment of memory, within the context of a multi-institution cooperative group trial.

Materials and methods

Protocol approval was received from the Institutional Review Board at each study site and informed consent obtained from each patient prior to participation. All sites met technology and training requirements to ensure ability to perform HA-WBRT and administer NCF tests properly.

Patients and intervention

As detailed in our primary publication [12], 113 patients were accrued (average monthly accrual was 5.9) from March 31, 2011, to November 1, 2012, with 13 patients excluded due to lack of protocol treatment or study criteria ineligibility, resulting in 100 analyzable patients (median age 61, 52 % female). All patients had brain metastases outside a 5 mm margin around the hippocampus. Patient selection and treatment parameters matched the criteria used in PCI-P-120-9801 [14], a phase III trial of patients who received WBRT (i.e., no conformal avoidance of the peri-hippocampal stem cell region) for brain metastases and who served as the pre-specified historical control for this study.

WBRT is associated with memory dysfunction in some patients [15, 16], as well as patient-reported quality of life [17]. The historical control identified a mean relative decline of 30 % from baseline to 4-month follow-up in HVLT-R Delayed Recall. The present study employed intensity-modulated radiotherapy (IMRT) techniques [18] to allow coverage of the planning target volume while conformally avoiding the hippocampal neural stem cell niche, an area implicated in new memory formation [19]. Treatment consisted of HA-WBRT to 30 Gy in 10 fractions.

NCF assessment

Memory assessment was conducted at baseline, 2 and 4 months. The 12-month data were not analyzed due to low completion (8–10 %, with more than 50 % loss due to death). Because HVLT-R Delayed Recall served as the measure for the primary endpoint, with statistical comparison to HVLT-R Delayed Recall findings from the historical control, the usual sequence of HVLT-R components was altered to conform to that study’s method, which consisted of 3 learning trials, followed immediately by the recognition component, a delay period of 20 min, and then delayed recall; this altered format has also been used in other phase III cooperative group research [20, 21].

Two computerized tests of memory were also included. The International Shopping List Test (ISLT) [22, 23] is a computer-administered and scored, 16-item (there are shorter versions) auditory/verbal list-learning task with psychometric properties including reliability and validity comparable to traditional list-learning tasks [24], but possibly with less susceptibility to proactive interference, a potential confound [25]. As with HVLT-R and other word-list tests, words (unseen by the patient) are read aloud by an examiner and patients immediately recall, in any order, as many words as possible from the list. For ISLT, timing of oral presentation of the words is cued by their appearance onscreen, and the examiner enters responses by clicking the list’s words on the computer’s screen (non-list words said aloud by the patient are indicated by clicking an “Other Word” button). There are 3 learning trials and one delayed recall trial that follows 20 min later. Because each list is pseudo-randomly generated from a pool of 128 words, 8 lists are possible without overlap. Content is drawn from food items common to specific geographic locations and languages. Instructions are available in multiple languages (including variants). A sample screen view of the test is shown in Fig. 1a.

Fig. 1.

a Sample screen from ISLT. b Illustration of presentation of OCLT stimuli

The One Card Learning Test (OCLT), which demonstrates good psychometric properties including reliability and validity [26–29], involves a visuoperceptual pattern separation paradigm [30] within a continuous visual recognition task [31], where standard playing cards are presented one at a time in the center of the computer’s screen. Pattern separation appears to be a key function of memory activity within the hippocampus and particularly the dentate gyrus as demonstrated by fMRI [32, 33]. As each card is presented, the patient must indicate whether that card has been presented previously in the task by pressing buttons representing “yes” or “no.” The software can generate essentially unlimited content sequences. Instructions are available in multiple languages (including variants) and OCLT requires no verbal response or delay period between portions of the test. This version of OCLT used 88 items. An illustration of the test is shown in Fig. 1b.

The sequence of administration in this protocol was HVLT-R, ISLT and OCLT (OCLT was administered during the 20-min period between ISLT Immediate and Delayed Recall) at baseline and 4 months, to allow for proper comparison to historical control. At 2-months follow-up, as a check for potential effects resulting from the administration sequence of the tests, their administration sequence was altered to ISLT, OCLT and HVLT-R. (Sequence of administration was not analyzed due to limited completion rates at 2 and 4 months.) At each testing session, there was a break of about 10 min with intervening activity between the two list-learning tasks to lessen the possibility of interference effects.

Statistical analyses

Raw test scores were used for statistical analyses. Patients served as their own controls. Mean relative decline from baseline for each test was assessed with the following formula: baseline—follow-up/baseline. ISLT and OCLT were further examined via a Reliable Change Index (RCI). RCIs may be calculated in a number of ways, but they all are a means of controlling the sources of error associated with repeated NCF assessments. The method chosen for this study, which uses the within-subject standard deviation, or mean square error from a linear regression model as the standard error of measurement, is described in detail elsewhere [34]. Associations between the 3 tests were evaluated using Spearman correlation coefficients. Due to the rates of patient deaths throughout the study, cases also were split into two groups to allow for comparisons between those who were alive at 6 months and those who had died by 6 months; these were evaluated with the Wilcoxon rank-sum test. Because the study involved multiple comparisons, which increases the likelihood of Type 1 error, significance level was set at 0.01, rather than 0.05, for all analyses.

Results

Completion

Of the 100 eligible patients, 92 and 89, respectively, completed HVLT-R, and ISLT and OCLT, prior to HA-WBRT. Completion for HVLT-R at 2 months was 53 (53 %) cases analyzed, with 23 not analyzed and 24 deaths; whereas, at 4 months the numbers were 42 (42 %), 17 and 41, respectively. For the CogState tests, at 2 months there were 52 (52 %) cases analyzed, with 24 not analyzed and 24 deaths, and at 4 months there were 34 (34 %), 25 and 41. For most tests, the largest number of non-analyzable cases was due to data not submitted.

Relative decline and RCI

At 4 months, the mean relative decline was 7.0 % (95 % confidence interval [CI] −4.7–18.7 %) for HVLT-R Delayed Recall and 18.0 % (95 % CI 5.5–30.5 %) for ISLT Delayed Recall. OCLT showed an 8.0 % increase (95 % CI −16.9–0.90 %). A reliable change index (mean = 0, SD = 1) found no significant change from baseline to 2 months and from baseline to 4 months for ISLT Delayed Recall (z = −0.40, p = 0.34; z = −0.68, p = 0.25, respectively) or OCLT (z = 0.15, p = 0.56; z = 0.41, p = 0.66, respectively).

Association among the tests

Spearman correlation coefficients are shown in Table 1. All baseline correlations among the 3 tests were significant (0.31–0.71; p ≤ 0.003). For the follow-up time points, associations between HVLT-R and ISLT Immediate Recall and Delayed Recall (both involve recall of newly presented material, as when taking an essay test in school) and HVLT-R Immediate Recognition and OCLT (both involve recognition of material where newly presented information must be differentiated from that which was not presented, as when taking a yes/no choice test in school) were emphasized.

Table 1.

Spearman correlation coefficients, 95 % confidence intervals, and significance values

CogState	HVLT-R
	Immediate Recall	Immediate Recognition	Delayed Recall
Baseline correlations
ISLT Immediate Recall	(n = 89)	(n = 89)	(n = 89)
	0.71	0.62	0.38
	(0.59, 0.80)	(0.47, 0.73)	(0.19, 0.55)
	<.0001	< .0001	0.0002
ISLT Delayed Recall	(n = 88)	(n = 88)	(n = 88)
	0.67	0.67	0.42
	(0.53, 0.77)	(0.53, 0.77)	(0.23, 0.58)
	<.0001	< .0001	< .0001
OCLT	(n = 89)	(n = 89)	(n = 89)
	0.44	0.38	0.31
	(0.25, 0.59)	(0.18, 0.54)	(0.11, 0.49)
	<.0001	0.0003	0.0030
2-month correlations
ISLT Immediate Recall	(n = 48)	(n = 49)	(n = 49)
	0.73	0.62	0.76
	(0.56, 0.84)	(0.41, 0.77)	(0.61, 0.86)
	<.0001	< .0001	< .0001
ISLT Delayed Recall	(n = 48)	(n = 49)	(n = 49)
	0.73	0.58	0.72
	(0.56, 0.84)	(0.36, 0.74)	(0.54, 0.83)
	<.0001	< .0001	< .0001
OCLT	(n = 47)	(n = 48)	(n = 48)
	0.22	0.45	0.25
	(−0.07, 0.48)	(0.19, 0.65)	(−0.04, 0.50)
	0.1328	0.0012	0.0909
4-month correlations
ISLT Immediate Recall	(n = 32)	(n = 32)	(n = 32)
	0.80	0.46	0.73
	(0.63, 0.90)	(0.13, 0.69)	(0.51, 0.86)
	<.0001	0.0087	< .0001
ISLT Delayed Recall	(n = 33)	(n = 33)	(n = 33)
	0.59	0.41	0.66
	(0.31, 0.78)	(0.07, 0.66)	(0.40, 0.82)
	0.0003	0.0187	< .0001
OCLT	(n = 33)	(n = 33)	(n = 33)
	0.14	0.46	0.19
	(−0.21, 0.46)	(0.14, 0.69)	(−0.16, 0.50)
	0.4273	0.0068	0.2804

At 2 and 4 months, Immediate Recall correlations between HVLT-R and ISLT were significant (0.73 and 0.80, p ≤ 0.0001), as were correlations between HVLT-R and ISLT Delayed Recall (0.72 and 0.66, p ≤ 0.0001). A relatively strong association may be expected between these 2 tests since both involve hearing a list of words and recalling them aloud. Correlations between HVLT-R Immediate Recognition and OCLT were .45 (2 months, p = 0.001) and .46 (4 months, p = 0.007); a relatively weaker association may be expected between HVLT-R and OCLT, as the two tests represent different NCF domains (auditory versus visuoperceptual), impart different stimuli (words versus playing cards) and involve different formats (yes/no recognition of words versus continuous visual recognition of standard playing cards).

Alive versus deceased at 6 months

Median survival was 6.8 months (95 % CI 4.8–10.9 months). As shown in Table 2, baseline Wilcoxon rank-sum tests found that 4 of the 6 memory test components differed for those who were alive at 6 months and those who had died by 6 months. Those alive at 6 months scored higher at baseline for HVLT-R Immediate Recall (median score of 25 for patients alive at 6 months versus 20 for patients deceased at 6 months, p = 0.008), ISLT Immediate Recall (27 vs. 23, p = 0.001) and Delayed Recall (10 vs. 7, p = 0.001), and OCLT (0.95 vs. 0.88, p = 0.001), suggesting that the sample may have varied in degree of illness at baseline. There was a trend toward significance for HVLT-R Delayed Recall (10 vs. 7, p = 0.022), but the relationship was not significant for HVLT-R Immediate Recognition (12 vs. 11, p = 0.10). Figure 2a displays HVLT-R and ISLT Immediate Recall scores at baseline, 2 and 4 months; the two groups appeared to diverge over time. Figure 2b displays HVLT-R and ISLT Delayed Recall scores across these time intervals; the two groups appeared to continue to differ over time. Figure 3a, b display HVLT-R Immediate Recognition and OCLT scores across the intervals, respectively.

Table 2.

Baseline raw score^* distributions and significance values by status at 6 months

	Alive at 6 months (n = 49)	Died by 6 months (n = 43)	P value§
ISLT Immediate Recall	(n = 47)	(n = 42)
Median	27	23	0.0012
Min–Max	8–40	0–35
ISLT Delayed Recall	(n = 47)	(n = 42)
Median	10	7	0.0005
Min–Max	0–16	0–13
OCLT	(n = 47)	(n = 42)
Median	0.9473	0.8826	0.0014
Min–Max	0.47–1.23	0.41–1.09
HVLT Immediate Recall	(n = 49)	(n = 43)
Median	25	20	0.0078
Min–Max	10–34	8–35
HVLT-Delayed Recall	(n = 49)	(n = 43)
Median	10	7	0.0223
Min–Max	2–12	0–12
HVLT-Immediate Recognition	(n = 49)	(n = 43)
Median	12	11	0.0945
Min–Max	0–12	4–12

^§P value from Wilcoxon rank-sum test

^*Higher scores reflect better performance for all tests. Scores for HVLT-R and ISLT are in whole words, with score ranges = 0–36 for HVLT-R Immediate Recall, 0–12 for HVLT-R Immediate Recognition and HVLT-R Delayed Recall, and 0–48 for ISLT Immediate Recall and 0–16 for ISLT Delayed Recall. Scores for OCLT are arcsine transformations of the square root of the proportion of correct responses and range from 0 to 1.57

Fig. 2.

a ISLT and HVLT-R Immediate Recall raw scores. b ISLT and HVLT-R Delayed Recall raw scores

Fig. 3.

a HVLT-R Immediate Recognition raw scores. b OCLT raw scores

Discussion

The present study supports the primary findings of NRG Oncology RTOG 0933 [12], which found that HA-WBRT is associated with preservation of performance on a memory test used often in adult cancer trials. It also provides preliminary evidence that two other memory tests, CogState computerized International Shopping List Test and One Card learning Test, may be useful and valid measures of memory in multi-institution adult brain tumor trials. Results are consistent with evidence from a validation study [35] that found 4 CogState computerized tests (including OCLT) administered to adult brain tumor patients (primarily glioblastoma) compared favorably with a traditional paper and pencil battery that included HVLT-R, suggesting that those tests may be reliable and valid measures with adult brain tumor populations. Further study of HA-WBRT is needed to determine whether aspects of cognition, including memory but also other domains such as executive functions, may benefit from the techniques.

The original HVLT [36] included only Immediate Recall and Immediate Recognition tasks to avoid a need for a delay period (e.g., 20 min) and minimize total task time. Nevertheless, delayed recall is a useful means of measuring memory (whereas immediate recall is often used to assess learning) and a delayed recall component was sometimes added by researchers, as it was when the test was formally revised and became HVLT-R [13]. In that revision, the recognition component was moved so that it followed delayed recall, which is the format used in most memory list tasks. To remain consistent with the format of the pre-specified historical control study, this study kept the recognition component between conclusion of Immediate Recall and beginning of Delayed Recall, which limits generalizability of these data to studies where HVLT-R Recognition followed Delayed Recall.

Compliance with NCF measurement points can be difficult for patients with brain tumors and may be less than 50 % in multi-institution studies [37]. In this study, completion rates at 4 months for all tests were below 50 %, an important limitation that complicates our understanding of the effects of treatment on memory. Compliance may have been especially challenging because patients were asked to undergo three memory tests at each session. The computerized tests showed a greater drop in compliance at 4 months than did HVLT-R, possibly due to their following HVLT-R in administration sequence; patients were asked to complete all tests at each testing session, but were urged, at minimum, to complete the HVLT-R since it served as the primary endpoint.

Previous research has shown that NCF decline correlates with tumor growth [38], and this study, as with others [39, 40], found that better memory at baseline was associated with longer survival. It is unclear why HVLT-R Immediate Recognition and Delayed Recall, unlike HVLT-R Immediate Recall, failed to maintain a performance differentiation between those who had died at 6 months and those who remained alive, when the difference was identified by both ISLT conditions and OCLT. One possibility is that it reflects an artifact of the study’s ordering of HVLT-R components. However, it is also possible that certain features of the computerized tests may have offered additional precision. For example, unlike HVLT-R, ISLT does not group words according to semantic categories, whereby content may be organized strategically by the patient to improve performance [41, 42]. Further, use of semantic clustering may increase between a first and a second session [43]. Unlike ISLT, HVLT-R content may be particularly susceptible since it includes emotionally-laden categories such as weapons (e.g., “bomb”) and alcoholic beverages (e.g., “bourbon”), and there is evidence that emotionally-arousing content is more readily semantically categorized [44]. OCLT uses neutral, universal content and does not present an opportunity to use strategy to improve performance. Further, it minimizes score range restrictions (word lists usually contain 12–16 words) by computing the arcsine transformation of the square root of the proportion of correct responses across a large number of items. Last, memory assessment in cancer trials has historically relied on auditory list-learning tasks and OCLT offers investigators the ability to assess visuoperceptual memory.

Potential advantages of computerized NCF tests

As with this study, since NCF measurement often suffers from data completion failure at follow-up time points, tests should obtain as much information as possible at each assessment session. Without adding to burden, software-based tests may automatically capture multiple scores, including accuracy, various error subtypes and speed (in milliseconds rather than whole seconds timed by hand), with some serving as potential covariates.

Computerized formats allow a research assistant (psychometricians are not required) to focus solely on proper test administration and may reduce sources of potential error by automating data capture and scoring. They also may include useful features such as direct data upload to real-time databases, nearly unlimited numbers of differing test forms and elimination of time delay periods (e.g., continuous recognition formats) between portions of a test such as required for immediate and delayed recall. Further, because statistical measures for differentiating change from stability need not require access to normative data sets, NCF tasks can be pre-configured to sample populations’ capabilities by a priori selection of task length, format or difficulty level. For example, Children’s Oncology Group [45] is studying use of computerized tests in pediatric populations where level of difficulty can be scaled from children to adults, offering potential for lifespan harmonization of NCF data. Perhaps surprising is evidence that even the elderly can enjoy computer-based testing and that some prefer it to paper and pencil tests [46].

Use of technology may promote internationalization of studies and allow for greater choice as to where data are collected, including remote login for test administration (e.g., a primary care clinic or even the patient’s home). By removing barriers to innovative trials design, protocols might include more frequent NCF sessions and detailed charting of performance curves. Studies have found online administration in patients’ homes to be practical with post-surgical cardiac [47], multiple sclerosis [48] and concussion patients [49]. Cancer patients appear to find interacting with technology to be the same or preferable to paper and pencil formats [35, 50].