Practice effects and amyloid deposition: Preliminary data on a method for enriching samples in clinical trials

Abstract

Clinical trials in Alzheimer’s disease are moving towards prevention studies in prodromal individuals with amyloid burden. However, methods are needed to identify individuals expected to be amyloid positive for these studies to be feasible and cost effective. The current study sought to determine whether short-term practice effects on cognitive tests can identify those with notable uptake on amyloid imaging. Twenty-five, non-demented older adults (15 cognitively intact, 10 Mild Cognitive Impairment) underwent amyloid imaging via ¹⁸F-flutemetamol and two cognitive testing sessions across one week to determine practice effects on a visual memory test. Results indicated that whereas ¹⁸F-flutemetamol uptake showed little association with baseline performance on a visual memory test (r=−0.04, p=0.85), it was significantly correlated with practice effects across one week on that same memory measure (r=−0.45, p=0.02), with greater uptake being associated with lower practice effects. The odds ratio of notable ¹⁸F-flutemetamol uptake was 5 times higher in individuals with low practice effects compared to high practice effects. Although these preliminary results need to be replicated in larger samples, short-term practice effects on cognitive tests may provide an affordable screening method to identify individuals who are amyloid positive, which could enrich samples for preventative clinical trials in Alzheimer’s disease.

Introduction

As clinical trials in Alzheimer’s disease (AD) and Mild Cognitive Impairment (MCI) progress, there will be increasing focus on preventative studies in prodromal individuals who are biomarker positive ¹. These studies may utilize markers from blood (e.g., APOE), cerebrospinal fluid (e.g., tau, beta-amyloid), or brain imaging modalities (e.g., magnetic resonance imaging, amyloid imaging) to identify potential participants ². For example, the Anti-Amyloid Treatment in Asymptomatic AD (A4) study (http://www.adcs.org/Studies/A4.aspx) is a secondary prevention trial aimed at preventing AD dementia in cognitively normal individuals who have amyloid deposition in their brains. However, current biomarkers in AD are far from definitive. Across multiple amyloid imaging agents, a relatively high percentage of cognitively normal older adults are amyloid positive (i.e., show uptake of an amyloid imaging tracer above some pre-determined cutoff suggestive of AD pathology); conversely, a sizeable minority of patients diagnosed with MCI and AD are amyloid negative (i.e., fall below the cutoff) ^3–5. Without some additional methods for enriching samples with those likely to be biomarker positive, these trials are likely to spend an excessive amount of resources (e.g., patient and staff time, costs of scans) on individuals who turn out to be biomarker negative ⁶, which could derail these preventive efforts.

Recent studies have identified some potential enrichment strategies. For example, in a large cohort of cognitive normal individuals, Mielke et al. ⁴ noted that older age and APOE e4 status best predicted amyloid positivity. Cognitive dysfunction, either with objective test scores or subjective complaints, has been linked to amyloid positivity ^{4, 7, 8}. Building on prior work, practice effects may be another method for enriching samples in these future clinical trials.

Practice effects are improvements in cognitive test performance that occur with repeated evaluation with the same or similar test materials ⁹. Although these improvements are routinely observed in cognitively intact individuals, their magnitude of improvement can be influenced by demographic factors, such as age ^{10, 11} and intellect ^{12, 13}. Some tests are more susceptible to practice effects than others ^{14, 15}. Some patient groups have diminished or absent practice effects on repeated testing ^16–18. Although practice effects have traditionally been viewed as a source of error in repeated assessments, they can provide clinically useful information. For example, practice effects can separate intact elders from those with milder cognitive impairments ^18–22. Prognostically, practice effects across short retest intervals (e.g., one week) have predicted cognitive outcomes across longer intervals (e.g., 6 – 12 months) ^{23, 24}. Practice effects have also been positively correlated with treatment response ^25–29.

To our knowledge, practice effects have not been studied against biomarkers of AD pathology, such as amyloid deposition. If practice effects were predictive of amyloid positivity, then they could be used as an affordable screening method to identify individuals who are likely to be amyloid positive, which could enrich samples for preventive clinical trials. Therefore, the purpose of this study was to examine the relationship between short-term practice effects and uptake of the investigational amyloid PET imaging agent ¹⁸F-flutemetamol in a sample of non-demented older adults. It was hypothesized that notable uptake of ¹⁸F-flutemetamol would be negatively correlated with short-term practice effects on a delayed recall memory task.

Methods

Participants

Twenty-five older adults (18 females/7 males, mean age=74.6 [6.8] years, mean education=16.1 [3.2] years) were enrolled in this study. These individuals were all recruited from senior centers and independent living facilities to participate in studies on memory and aging. All reported to be functionally independent in activities of daily living, and this was corroborated by a knowledgeable informant. Based on objective cognitive testing, the majority of these individuals were classified as cognitively intact (n=15), with the remainder characterized as MCI (n=10) ³⁰, exhibiting at least an amnestic profile. Exclusion criteria for this study included: history of neurological disease known to affect cognition (e.g., stroke, head injury with loss of consciousness of >30 minutes, seizure disorder, demyelinating disorder, etc.); dementia based on DSM-IV criteria; current or past major psychiatric illness (e.g., schizophrenia, bipolar affective disorder); 30-item Geriatric Depression Score >15; history of substance abuse; current use of cholinesterase inhibitors, other cognitive enhancers, antipsychotics, or anticonvulsant medications; history of radiation therapy to the brain; history of significant major medical illnesses, such as cancer or AIDS; and currently pregnant.

Procedures

The local institutional review board approved all procedures and all participants provided informed consent before data collection commenced. As part of a larger study, all participants completed a neuropsychological battery designed to characterize their cognitive status across several relevant domains. The battery contained a measure of visual memory, the Brief Visuospatial Memory Test – Revised (BVMT-R) ³¹. On this test, participants are given 10 seconds to study 6 different geometric designs in 6 different locations on an 8.5″ × 11″ card. Immediate recall is tested after each of 3 study trials. After 20 – 30 minutes, delayed recall of the designs and locations is tested. This Delay Recall trial on the BVMT-R, which ranged from 0 – 12 points with higher scores indicating better memory, was the primary cognitive variable in this study. As part of the larger study, the BVMT-R was repeated after approximately one week to quantify practice effects. The BVMT-R was chosen as the primary practice effects variable for multiple reasons. First, the BVMT-R seems to be a particularly challenging memory test, with geometric designs that are not easily verbally encoded. Second, since participants must also recall the correct location of designs, it is a paired-associates task, and these types of memory tasks have been shown to be particularly effective in patients with memory disorders. Third, as a novel memory task, we have previously found it to have large practice effects in older adults and these practice effects have predicted future cognitive functioning in prior studies ^{20, 23, 24}. Finally, visual spatial tasks, including those assessing memory, appear to be sensitive to various dementing conditions ^32–35. Although alternate forms are available for the BVMT-R, they were not used in this study to maximize practice effects.

Following completion of the cognitive battery, participants underwent ¹⁸F-flutemetamol PET imaging. ¹⁸F-flutemetamol was produced under PET cGMP standards and the studies were conducted under an approved IND. Imaging was performed 90 minutes after the injection of 185 mBq (5 mCi) of ¹⁸F-flutemetamol. Emission imaging time was approximately 30 minutes. Two different PET tomographs were used for the imaging, a GE Advance PET scanner or a GE Discovery DST PET/CT scanner. The two PET tomographs have substantially equivalent performance characteristics as determined by phantom brain studies. The images were acquired in 3D mode with measured attenuation correction utilizing Germanium rod sources (GE Advance) or with CT attenuation correction (GE DST PET/CT). The image reconstruction used for both PET tomographs was iterative, 30–32 subsets, loop filter of 2.0 and diameter of 25.6 cm. Using methods previously described ³⁶, a semi-quantitative global composite of standardized ¹⁸F-flutemetamol uptake value ratios in the cerebral cortex (SUVRs) was obtained normalized to cerebellar cortex.

Statistical analyses

Raw score on the Delay Recall trial of the BVMT-R was used in all analyses. To generate a practice effects score on this measure, a standardized regression-based prediction formula was developed using an independent sample of 167 non-demented older adults who were also administered the BVMT-R at baseline and one-week ²⁰. This methodology, which allowed us to predict one-week scores from observed baseline scores, is frequently used in the assessment of neuropsychological change ³⁷. Briefly, a stepwise linear regression model predicted one-week BVMT-R scores from baseline BVMT-R scores, years of education, and gender of the 167 subjects (age did not contribute to this model): F(3,166)=105.7, p<0.001, R²=0.66. All relevant variables met assumptions for this type of analyses (e.g., normality of residuals, collinearity, independence, heteroscedasticity). The resulting formula (0.278 + [BVMT-R at baseline * 0.725] + [education * 0.181] + [gender * 0.987]) was applied to the current sample to yield a predicted one-week BVMT-R score. The predicted one-week BVMT-R score was subtracted from the observed one-week BVMT-R score and divided by the standard error of the estimate of the regression equation (1.88) to yield a z-score that reflects how much the predicted score deviated from the observed score for each participant. Additional details of this practice effects score, including its calculation, can be obtained from the first author. In the primary analyses, Pearson correlations were calculated between the ¹⁸F-flutemetamol global composite and three BVMT-R Delay Recall scores: observed baseline, observed one-week, and practice effects z-score. An alpha value of 0.05 was used for these limited comparisons. Effect sizes (e.g., Cohen’s d) were calculated for statistically significant results. Secondary analyses examined correlations between ¹⁸F-flutemetamol and BVMT-R scores in the two subgroups (intact, MCI).

Results

No adverse events were reported during the injection, uptake time, or imaging studies with the investigational imaging agent ¹⁸F-flutemetamol. The mean global composite of SUVRs normalized to cerebellar cortex was 1.65 (SD=1.30, range=1.05 – 7.66). One participant was an outlier with a very high level of ¹⁸F-flutemetamol uptake (i.e., 7.66), and when this observation was removed, the mean global composite was 1.40 (SD=0.37, range=1.05 – 2.37). Including this outlier, 8 of the 25 scans were categorized as “positive” for ¹⁸F-flutemetamol uptake, using a cutoff of 1.56 ³⁶. Of these 8 scans with notable ¹⁸F-flutemetamol uptake, 4 were from participants categorized as cognitively intact and 4 were from participants categorized as MCI.

For the entire group, the mean cognitive performances on the BVMT-R Delayed Recall trial at baseline and one-week follow-up were in the average range, although there was some variability. Practice effects on this measure also fell in the average range, again with some variability. Relevant cognitive scores are presented in the Table 1.

Table 1.

Cognitive performances and correlations with ¹⁸F-flutemetamol global composite

	Total Sample (n=25)		Cognitively Intact (n=15)		MCI (n=10)
Cognitive measure	Mean (SD) (range)	r p	Mean (SD) (range)	r p	Mean (SD) (range)	r p
BVMT-R Delayed Recall
Baseline	6.7 (3.1) 1 – 12	−0.04 0.85	7.7 (2.7) 2 – 12	−0.01 0.99	5.3 (3.3) 1 – 10	0.14 0.69
One-week	9.1 (2.5) 3 – 12	−0.15 0.46	9.9 (1.7) 6 – 12	0.11 0.70	7.8 (3.1) 3 – 11	−0.03 0.93
Practice effects	−0.07 (0.77) −1.48 – 1.61	−0.45 0.02	0.10 (0.50) −0.93 – 0.89	−0.35 0.19	−0.48 (0.99) −1.48 – 1.61	−0.42 0.23

Note. MCI = Mild Cognitive Impairment. r = correlation with ¹⁸F-flutemetamol global composite. Brief Visuospatial Memory Test – Revised (BVMT-R) scores at baseline and one-week are raw scores. Practice effects BVMT-R scores are z-scores based on standardized regression-based change formula, with M=0, SD=1.

The correlations between ¹⁸F-flutemetamol uptake and the three BVMT-R scores for the entire sample are presented in Table 1. Briefly, ¹⁸F-flutemetamol uptake was not correlated with the Delay Recall trials of the BVMT-R at baseline (p=0.85) or one week (p=0.47); however, ¹⁸F-flutemetamol uptake was significantly associated with practice effects on this memory measure (r=−0.45, n=25, p=0.02) with a large effect size (d=1.1). When the outlier with a very high level of ¹⁸F-flutemetamol uptake was removed from these analyses, the results were similar (e.g., ¹⁸F-flutemetamol uptake and practice effects: r=−0.43, n=24, p=0.03).

In secondary analyses, when these analyses were re-examined based on cognitive status of the participants (i.e., intact vs. MCI), the results showed the same trends. These results are presented in Table 1. Additionally, neither age nor education significantly correlated with amyloid uptake values.

To further examine practice effects as an enriching strategy for clinical trials requiring amyloid positivity as an entrance criterion, each value was dichotomized. For, individuals with a global composite of ¹⁸F-flutemetamol SUVR of ≥1.56 were labeled amyloid positive and those with composites <1.56 were amyloid negative ³⁶. For practice effects, individuals with standardized regression based reliable change index z-score ≤ −0.50 were labeled as low practice effect and those with indexes >−0.50 were high practice effects. Using this dichotomization strategy, 8 individuals were amyloid positive (32%) and 17 were amyloid negative (68%); 9 had low practice effects (36%) and 16 had high practice effects (64%). As can be seen in Table 2, of those with notable ¹⁸F-flutemetamol uptake (i.e., positive amyloid scans), 5 had low practice effects (63%) and 3 had high practice effects (37%). Of those without notable ¹⁸F-flutemetamol uptake (i.e., negative amyloid scans), 4 had low practice effects (23%) and 13 had high practice effects (77%). Therefore, the odds ratio of having a positive amyloid scan was 5 times higher if the individual had low practice effects compared to high practice effects.

Table 2.

Practice effects and ¹⁸F-flutemetamol global composite

Notable 18F-flutemetamol uptake	Low practice effects	High practice effects
Yes	5	3
No	4	13

Note. Notable ¹⁸F-flutemetamol uptake: Yes = ¹⁸F-flutemetamol global composite ≥1.56, No = ¹⁸F-flutemetamol global composite <1.56; Low practice effects = reliable change index ≤ −0.50: High practice effects = reliable change index >−0.50.

Discussion

With clinical trials in AD moving towards preventative studies in prodromal individuals who are biomarker positive ¹, methods must be developed to enrich samples with those most likely to be biomarker positive or these trials will deplete valuable resources ⁶. Our preliminary results suggest that practice effects may be an affordable and non-invasive first-line screening option to enrich preventive trials. In 25 non-demented older adults, one-week practice effects on a visual memory measure significantly and negatively correlated with uptake of ¹⁸F-flutemetamol, an investigational amyloid imaging radioligand, so that lower practice effects were associated with greater ¹⁸F-flutemetamol uptake/amyloid deposition. When these variables were dichotomized, those having greater ¹⁸F-flutemetamol uptake/positive amyloid scan were 5 times more likely to have low practice effects compared to high practice effects. Therefore, future studies requiring amyloid positivity might initially screen with practice effects and funnel those with low practice effects into their more expensive imaging procedures. Due to the small sample size of this study, these results should be viewed as preliminary. However, if these findings are replicated in larger samples, then practice effects have the potential to reduce financial costs, personnel time, and participant involvement in those trials. For example, a single amyloid imaging scan may cost $4,000 or more, whereas the collection of practice effects is notably less expensive (e.g., 2 BVMT-R test forms = $6, 2 hours of research assistant time = $50). Given these numbers, practice effects could be collected on over 70 subjects for the same cost as a single amyloid scan. Despite their potential value in clinical trials, there are some challenges to implementing practice effects methodology as a screening measure. First, to quantify practice effects, we utilized standardized regression-based change formulas. Although this technique is frequently used in neuropsychology to assess long-term cognitive change (e.g., one year) ³⁷, it has not been widely used over shorter periods (e.g., one week). Therefore, we needed to generate change formulas from our own independent data ²⁰. The change formulas from the current study could be applied at other sites using the BVMT-R. We also chose the cutoff on our practice effects score based on the distribution within the current sample (e.g., two reasonably separated subgroups), but different cutoffs would have likely yielded different results. Second, since practice effects vary by specific test ¹⁵, it is not clear if this methodology would be as effective with a different memory test. In our sample, practice effects on the Delayed Recall trial of the BVMT-R proved to be effective in identifying those with amyloid positivity. The BVMT-R might be an ideal measure in mildly impaired subjects, as it is sufficiently challenging (e.g., recall of 6 different geometric designs in 6 different locations) to avoid ceiling effects on repeated testing. This particular measure has also been useful in predicting cognitive trajectory in these patients ^{23, 24}. Similarly, visual spatial cognition seems to be a particularly sensitive domain in various dementing conditions ^{32, 33}. Third, this enrichment strategy may be most useful in older adults who are cognitively intact or have early MCI, and it is unclear if would be as helpful in cases of late MCI or early AD. We have been experimenting with shorter retest intervals (e.g., within the same session) ³⁸, and these may be more beneficial in patients with more advanced disease. Fourth, it is not clear that practice effects on the BVMT-R would be as useful in predicting amyloid positivity using other radioligands (e.g., ¹¹C-Pittsburgh compound, other ¹⁸F agents). Despite some of these challenges, the potential benefits of using practice effects would seem to outweigh them.

Whereas the association between amyloid imaging agents and baseline cognitive functioning has been examined ^{4, 7, 8, 39}, to our knowledge, short-term practice effects have not been previously investigated in amyloid imaging studies. It is possible that practice effects are a more sensitive measure of cognitive integrity than traditional measures. For example, in this study, baseline memory was not significantly correlated with ¹⁸F-flutemetamol uptake, but practice effects on that measure was. Consistent with prior studies, lower levels of practice effects were associated with worse outcomes (e.g., diagnosis of cognitive disorder, cognitive decline, non-response to intervention) ^{20, 23–25}. Future studies should continue to examine practice effects as an index of brain health.

Although group data can be useful for research endeavors, it would also be useful to see the clinical potential of practice effects and amyloid imaging. Two of our 25 cases clearly demonstrate these findings (see Figure 1). Although both cases had similar demographic profiles, fell in the severely impaired range on baseline memory testing (e.g., 1^st percentile), and would likely be classified as amnestic MCI, Case A demonstrated large practice effects after one week on memory testing and also showed low ¹⁸F-flutemetamol uptake on imaging (indicated by absence of “hotter” colors such as red). Conversely, Case B showed small one-week practice effects and high ¹⁸F-flutemetamol uptake (indicated by presence of “hotter” colors such as red). If these findings can be replicated in larger samples, then practice effects might also be used to assist in clinical decision making about which patients should be recommended for expensive diagnostic technologies such as amyloid imaging.

Figure 1.

Practice effects and ¹⁸F-flutemetamol imaging in two subjects.

Note. Both subjects had similar demographic profiles (age, education, gender), both fell in the severely impaired range on baseline memory testing (e.g., 1^st percentile), and both would likely be classified as amnestic MCI. Case A demonstrated large practice effects after one week on memory testing and also showed low ¹⁸F-flutemetamol uptake on imaging (indicated by absence of “hotter” colors). Case B showed small one-week practice effects and high ¹⁸F-flutemetamol uptake (indicated by presence of “hotter” colors).

Despite the interesting findings, some limitations of this study should be acknowledged. First, as noted earlier, these results should be viewed cautiously as the sample size was small. Larger studies with a wider range of cognitive functioning (including practice effects) would better test this hypothesis. Secondly, the sample was relatively homogeneous (e.g., all Caucasian, highly educated, mostly female, healthy enough to complete a PET scan), and the ability to generalize these findings to a more diverse group is unknown. Third, structural imaging was not part of this research protocol. Thurfjell et al. ⁴⁰ has shown that combining ¹⁸F-flutemetamol and structural MRI can better characterize disease states and predict progression. We are starting to collect these structural imaging scans on this cohort and encourage future investigations to do the same. Similarly, APOE status was not determined in this cohort. Fourth, our sample was quite mild in their cognitive dysfunction, with most subjects being cognitively intact and a minority of them being classified as MCI. The current study focused on those who were “biomarker positive” for a suspected AD pathology, which is not the same thing as having AD dementia. As noted earlier, it is unclear if these findings would replicate in more impaired samples. However, since clinical trials appear to be moving towards earlier points in the disease spectrum, the relevance of these findings to more advanced cases might be less. Lastly, it should be reiterated that ¹⁸F-flutemetamol is an investigational amyloid imaging agent. Regardless of these limitations, the current study found notable relationships between amyloid-binding with ¹⁸F-flutemetamol and practice effects in non-demented community-dwelling older adults, and future examination of practice effects as a screening tool in preventative clinical trials in AD seems warranted.