Outcomes Assessment in Clinical Trials of Alzheimer’s Disease and its Precursors: Readying for Short-term and Long-term Clinical Trial Needs

Abstract

An evolving paradigm shift in the diagnostic conceptualization of Alzheimer’s disease is reflected in its recently updated diagnostic criteria from the National Institute on Aging-Alzheimer’s Association and the International Working Group. Additionally, it is reflected in the increased focus in this field on conducting prevention trials in addition to improving cognition and function in people with dementia. These developments are making key contributions towards defining new regulatory thinking around Alzheimer’s disease treatment earlier in the disease continuum. As a result, the field as a whole is now concentrated on exploring the next-generation of cognitive and functional outcome measures that will support clinical trials focused on treating the slow slide into cognitive and functional impairment.

With this backdrop, the International Society for CNS Clinical Trials and Methodology convened semi-annual working group meetings which began in spring of 2012 to address methodological issues in this area. This report presents the most critical issues around primary outcome assessments in Alzheimer’s disease clinical trials, and summarizes the presentations, discussions, and recommendations of those meetings, within the context of the evolving landscape of Alzheimer’s disease clinical trials.

INTRODUCTION

Alzheimer’s disease (AD) is a fatal neurodegenerative disease that causes progressive cognitive, neuropsychiatric, and functional deterioration of memory and of self. Given that more than 30 million people worldwide are affected by AD, and that this number is growing dramatically,^1–3 the search for effective treatments to prevent its onset, significantly delay its progression, or otherwise positively intervene in the disease course is an international research priority.^3–5

It has been over 10 years since the last of five drugs was approved for the treatment of AD.^6–13 All five approved therapies target neurotransmitter systems and enhance the function of surviving neuronal circuitry. Although worthwhile, their effectiveness has provided modest symptomatic improvement but has not reversed prior decline or slowed the progression of the underlying disease processes. Efforts have turned toward identifying a disease-modifying therapy for AD, but these efforts have yet to yield a successful intervention.^14–21 The lack of success in developing a disease-modifying therapy for AD may arise, in part, from methodological imprecision in the conduct of clinical trials. Recent advances have occurred to reduce this imprecision by 1) improving the diagnostic accuracy of subjects enrolled into trials, especially through the incorporation of biomarkers; 2) moving to secondary prevention trials to evaluate therapies; and 3) developing more sensitive clinical instruments, including composite, computerized, and performance-based measures to assess disease progression and treatment effects.

In past clinical trials, the diagnosis of AD was based largely on the National Institute of Neurological and Communicative Disorders and Stroke and the Alzheimer’s Disease and Related Disorders Association (NINCDS-ADRDA) criteria²² that relied on clinical assessments of cognitive and functional deficits and the concurrent exclusion of other etiologies of dementia. These criteria had important utility in helping to identify a relatively homogeneous cohort of patients to be studied, but were susceptible to imprecision since up to 20 percent of those with a clinical diagnosis of AD did not have evidence of AD neuropathology (by current criteria) on post-mortem examination^23–25 or on positron emission tomography (PET) imaging of brain amyloid burden during life.^26–29 Furthermore, the criteria did not allow AD to be diagnosed until symptoms were well established, presumably at a point when disease pathology was advanced.

It is now widely accepted that pathologic changes in AD begin decades before the emergence of the constellation of clinical symptoms that define dementia onset.^3,30–32 There is also an emerging consensus that intervention at the earliest stages of AD, prior to dementia onset and widespread irreversible neuronal damage, will be the most feasible scientifically and socioeconomically, and will have the most impact.

Technical advances in imaging and fluid biomarkers have furthered our understanding of the underlying disease processes, and now allow AD to be identified with some certainty before the onset of obvious clinical symptoms. These advances have been incorporated into new diagnostic criteria proposed by the National Institute on Aging-Alzheimer’s Association (NIA-AA)^33–36 and by the International Working Group (IWG).^37–39 Both sets of criteria hold that AD encompasses a continuum from preclinical to dementia, and they both aim to improve diagnostic specificity early in the continuum through the use of AD biomarkers. The criteria differ in that those proposed by the IWG capture both the prodromal and the more advanced stages of dementia within the same diagnostic framework, whereas those proposed by the NIA-AA classify three phases of disease: the first identified by the presence of biomarker abnormalities before the emergence of clinical symptoms,³⁶ the second by the presence of both mild clinical symptoms and biomarker abnormalities,³⁵ and the third by the presence of AD dementia.²² Application of either of these new criteria in therapeutic trials^40,41 opens up the possibility for more accurate identification of patients for treatment. In the interim and since the initial IWG criteria, further research in biomarkers associated with stages of AD (i.e., markers of progression vs. diagnosis) has prompted recent publication of the IWG-2 criteria.⁴²

As aspects of current practices in trial design and conduct improve, as outlined above, the regulatory framework around approval of new AD therapies with different mechanisms of action than the currently approved neurotransmitter-based therapies requires re-thinking. Additional work needs to be done in anticipation of a novel AD therapy that shows clear clinical benefit, including, whenever possible, performance-based assessments that underscore real-world functional benefit. Our companion paper⁴³ considers the state of the art regarding potential performance-based measures of functional capacity.

REGULATORY FRAMEWORK

Regulators, recognizing the pressing need for a shift in paradigm(s) when evaluating interventions for preclinical and predementia disease, have drafted guidance for AD drug approval. Historically, approval for drugs to treat AD has depended on dual primary endpoints assessing cognition and functioning; however, given the increasing emphasis on preclinical and prodromal AD trials, new definitions of what constitutes “functionally meaningful benefit” and “global decline” must be reexamined. Furthermore, as studies move into earlier intervention, the current divide between clinical trial design and registration requirements for “symptomatic” and “disease-modifying” therapies are likely to close.

By United States law, a clinical trial must establish clinical benefit to be the basis for an approval. Customarily, a clinically meaningful benefit is defined as a favorable, statistically robust effect on how a patient feels, functions, and/or survives.^44–46 This “clinical meaningfulness” could be reflected, for example, in an established “minimally important difference” between groups on a measure of functional ability/disability and/or a “responder” definition that is based on prior evidence that a certain degree of improvement on the selected scale represents a meaningful improvement for an individual. When the primary outcome does not directly measure daily function, additional evidence for a relationship between the impact of the drug on the treatment target and how the patient feels or functions is needed. Alternatively, a co-primary functional (e.g. ADCS-ADL) or global endpoint (e.g. ADCS-CGIC) could be used in addition to a cognitive outcome measure as is historically the case for AD trials.

Draft guidance was recently issued in the United States offering the US Food and Drug Administration’s (FDA) current thinking on the development of drugs for early-stage disease.⁴⁷ While their view on trial endpoints remains unchanged for dementia, they suggest that trials with individuals with early AD or late mild cognitive impairment (MCI) can theoretically use a single primary outcome measure if it can substantiate a treatment effect on both cognition and function, though no drug has been approved on such an outcome. The draft guidance newly acknowledges that the co-primary endpoint approach is impractical for predementia trials because of limited or relatively absent functional deficits. An effect solely on cognition may also be insufficient in preclinical cohorts, as detecting changes from no cognitive impairment or a relatively weak signal remains challenging. Nevertheless, in the case where functional deficits are essentially absent (e.g., early MCI or preclinical AD), an improvement on a cognitive measure may be sufficient for possible accelerated approval under Subpart H (i.e., an effect on an intermediate clinical endpoint that is reasonably likely to predict ultimate clinical benefit⁴⁸ [i.e., an effect on function]). This route would then require confirmatory post-marketing studies to establish clinical benefit.

The European Medical Agency (EMA) has also considered new guidance for trials in AD with concept papers, entitled “Need” and “No need” issued at the end of 2013.^49,50 The “No need” paper made the case that it is premature to rewrite the current, broadly worded guidance that accepts data-driven discussions and agreements with the EMA on a program by program basis. The “Need” paper holds that some clinical trials already focus on “prodromal” populations, and highlights the increasing emphasis on strategies aimed at earliest diagnosis, prognostication, and enrichment of clinical trials. The “Need” concept paper (as in the preceding 2009 guidance) explicitly acknowledges the “asymptomatic” disease stage. Version 2 of the “Discussion Paper on the Clinical Investigation of Medicines for the Treatment of Alzheimer’s Disease and Other Dementias,” released in October 2014, moved this discussion forward, and a new version of the EU draft AD guidance was released in January 2016.⁵¹

The 2016 draft of the EMA guidance recognizes potential challenges with a cognition-function co-primary requirement for pAD/MCI due to AD, including ceiling effects with existing tools and impact of compensation strategies. It also notes that creation of more sensitive and specific tools or novel cognition-function composites may be possible solutions, while stressing the need to assess both cognition and function in a comprehensive manner and demonstrate clinical relevance of effects. For preclinical AD, multiple approaches are described, such as diagnosis of dementia, significant cognitive decline, and change in cognitive function, including the use of more sensitive novel tools. Issues of a lack of tool validation, use of responder definitions and time-to-event analyses to support relevance, feasibility issues around trial duration and drop-out, and lack of reliable surrogates/need for lengthy follow-up to confirm relevant cognitive changes, are also discussed.

There is no doubt that regulatory conversations are moving the field in a much needed direction. Evolving therapeutic strategies that sequentially move from established regulatory paths for AD dementia to novel strategies for MCI due to AD/prodromal AD, and now to early MCI, take more direct aim at targeting the disease at its various stages. Building on this progress, consensus discussions focused around new principles for accepting the earliest predictive features of the AD in its preclinical state become crucial.

HEALTH ECONOMICS AND PAYERS

Consideration should also be given to the evidentiary needs of payers (i.e., coverage bodies and health technology assessment bodies) to support reimbursement. Currently, this includes using endpoint and statistical analyses to address the following questions: Is a new product better or safer than an existing product? How does its value compare to existing products? These considerations will continue to apply to therapeutic strategies aimed at AD. However, the next phase of AD drug development has prevention as its goal, and there are no precedents for judging value. In this context, what constitutes “meaningful benefit” is being debated across the research field and has yet to be resolved. Issues include determining the potential impacts on patient-reported outcomes, including quality of life, and on ethical, legal, and social issues.⁵² Finally, the field needs to be prepared to address the likely question from third party payers of “How do we value therapeutics that prevent the slide from ‘normal function’ to clinical decline?” It is likely that some clarity will ensue with regard to health economic criteria once there are new successful clinical trials for the prevention of Alzheimer’s disease and the totality of the evidence base can be evaluated.

COGNITIVE OUTCOMES: HISTORY AND PSYCHOMETRICS

The most frequently used cognitive outcome measure in clinical drug trials for mild to moderate AD is the Alzheimer’s Disease Assessment Scale-Cognitive, which was designed in 1984 to assess deficiencies in episodic memory and a number of other cognitive domains (e.g., language, orientation, praxis) that are universally affected by AD disease.^53,54 While the ADAS-Cog has proven utility in clinical trials of cholinesterase inhibitors in demented patients and has become standard in the field, there is concern that the test is not particularly sensitive to cognitive changes in very mild AD, MCI, or preclinical AD, and will not be appropriate for trials in these cohorts.^55–59 Indeed, in several recent clinical trials of cholinesterase inhibitors and other novel interventions in patients with MCI, the ADAS-Cog did not differentiate between treatment and placebo effects,^15–20 despite the introduction of supplementary items and use of 11-, 12-, 13- and 14-item versions.

Psychometric evaluation has shown that total scores on the ADAS-Cog are very low in MCI, often in the range of 11 to 12 (standard deviation [SD]=~4.0) out of a potential range 0 to 70 or 0 to 80 (potential range 0–70, 11-item; 0–80, 12-item including Delayed Word Recall).^60–62 This contrasts with average scores of 25 to 28 in mild-to-moderate AD cohorts entering treatment trials.^62,63 Although the ADAS-Cog performs adequately in terms of scaling assumptions, reliability, and validity, in the mild AD and MCI cohorts, the presence of large floor effects in mild AD and MCI cohorts imposes serious limitations on the instrument’s ability to detect change.⁶⁴ Furthermore, the response categories for some of the items do not work as intended, and this contributes to the inability of several subtests to discriminate subtle changes in cognition. The limited utility of the ADAS-Cog in mild AD, MCI, and less impaired states was further confirmed by a Rasch analysis⁶⁵ of the experimental measurement paradigm, Andrich.⁶⁶ Taken together, these findings indicate that the ADAS-Cog is not optimal for measuring cognitive decline in clinical trials targeting individuals in mild AD, MCI, or preclinical states.

Similar problems with floor and ceiling effects and inability to detect cognitive change in MCI and preclinical AD also occur with two other measures that are commonly used in AD clinical trials: the Mini-Mental State Exam (MMSE) and the Clinical Dementia Rating (CDR) scale sum of boxes (CDR-SB). In a large database of individuals with MCI (n=2,551) or various forms of dementia (n=4,796), MCI cases with a CDR global score of 0.5 had a mean CDR-SB score of 1.30 (SD=1.16) (out of a possible total of 18) versus a CDR-SB score of 0.11 (SD=0.36) in healthy elderly controls. In the same sample, MMSE scores were 27.2 (SD=2.3) for MCI cases (out of a possible total of 30) compared to 28.9 (SD=1.3) for the healthy elderly controls.⁶⁷ These findings suggest that the MMSE and CDR-SB lack sensitivity to reliably differentiate individuals with MCI from healthy individuals and could indicate they are not optimal outcome measures in clinical trials involving MCI or preclinical AD, though they do show sensitivity to decline in longer-term studies in these patients.^59,68 Indeed, such issues can be handled in current study design by sample size and trial duration, and improving outcomes should have the added benefit of increasing trial efficiency.

Discussion of this issue is expanded considerably by Harvey et al,⁴³ but the take-home point from the perspective of outcomes measurement and clinical trial design is that different assessments than the ADAS-COG, MMSE, and CDR-SB, and which are likely performance-based, will be required for clinical trials targeting individuals with MCI or preclinical AD.

PSYCHOMETRIC REQUIREMENTS FOR DETECTING CHANGE IN MINIMALLY IMPAIRED SAMPLES

There are now many more statistical, psychometric, and methodological tools available to validate potential cognitive outcome measures than those originally used to validate the ADAS-cog.⁵⁴ Input from statisticians and psychometricians is critical at all stages of scale development as new measurement strategies are developed for clinical trials across the expanded AD spectrum. This input should include 1) ensuring content validity (this may constitute establishing a sound conceptual basis in the case of a cognitive test, or ecological validity on the case of a performance-based outcome, and where concept elicitation is not feasible), 2) verifying the best question-answer combinations to suit the aspect of cognition or clinical function being assessed, 3) calculating sample size, 4) selecting models to evaluate various aspects of the scale, and 5) selecting among the various options for the final assessment tool and its accompanying analysis guidelines. Statisticians will need to work closely with neuropsychologists, epidemiologists, psychometricians, and clinical trial methodologists to develop outcomes that best reflect changing cognitive and functional status of the individuals assessed, and will need to contribute to understanding the clinical meaningfulness of these outcome measures.^69,70

A key consideration that has been poorly addressed by traditional psychometric methods of scale development is the need to ensure that scales are capable of measuring the range of cognitive performance that will be exhibited by the subjects at study entry and over time. This includes avoiding floor and ceiling effects and ensuring that there are enough possible responses within the metric in order that change can be visualized while minimizing possible false effects resulting from normal fluctuations in cognition. This latter consideration is particularly important when participants are selected for being apparently healthy (i.e., without clinically notable cognitive deficits at the time of study entry).

A central feature of outcomes assessments in clinical trials will be the ability to accurately identify very subtle declines and separate these from the expected stability in performance that would accrue from successful treatment in a prevention trial. In order to identify stability (expected in the case of successfully treated participants in the active treatment group) and decline (expected in some proportion of the placebo group), a scale needs to cover a wide range of functioning with gradations that enable precise measurement across the spectrum of disease severity, including those in the apparently healthy range. An important complication here is that some participants in these studies are not likely to develop a cognitive disorder (i.e., not all participants receiving placebo treatment will worsen). Thus, change will be detectable in only a subset of placebo-treated patients.

STATISTICAL STRATEGIES TO INCREASE SENSITIVITY IN EXISTING MEASURES

As the above discussion makes clear, detection of change in treatment trials of prodromal and preclinical AD has a number of highly specific requirements. Thus, it may take time to develop completely new scales to measure subtle changes in very early AD. However, treatment development efforts are ongoing even though existing measures are challenged in terms of detecting potential change in mildly impaired populations. To address these challenges, alternative strategies have been adopted to examine existing measures in nontraditional ways. Some options used to improve the usefulness of existing scales are 1) the creation of alternate and empirically derived composite outcomes that provide robust and sensitive combinations of existing items and 2) use of item-level analysis in the form of item-response theory (IRT) and Rasch analysis to understand the contribution of scale items that best assess an underlying latent variable (in this case, cognition or function). These strategies are not mutually exclusive, and multiple methods can be applied to the same set of items to identify the most robust way to identify treatment-related changes. Added benefits may be that these more robust outcome measures will reduce the estimated sample sizes for clinical trials and otherwise improve the efficiency of the analysis of trial data.^{55,64,65,71,72} However, a clear conceptual basis is needed prior to undertaking such work (e.g., a belief that items may be combined on the basis of best estimating a new unidimensional underlying construct such as disease progression). Without such clarity, concerns may persist regarding clinical meaningfulness and interpretation of such scores and difficulty in understanding what is being measured.

One approach that has been used to create composite endpoints is to identify and combine a reasonable set of items based on their face validity given our knowledge of the disease state and the concept we are attempting to estimate or measure (e.g., disease progression). Again, issues around content validity and conceptual basis should be considered here as mentioned above. Confirmatory factor analysis is often used to eliminate redundant items and to ensure that there are items that represent different levels of function within each domain of interest. Item selection can be challenging when cross-sectional data are used because 1) the disease is degenerative, and elements of functioning that represent the core disease process change over time, and 2) there is substantial measurement error because individuals with the disease manifest day-to-day variability in cognitive and functional performance, particularly when more severely impaired. The first point can be addressed by extracting items for the composite from longitudinal studies so that decline over time can be included as a method of confirming which items are relevant to disease progression (depending on the length of the trial from which the items were extracted and the expected amount of decline). The second point can be addressed by measuring the same domain with more than one assessment tool (i.e., with a construct validity approach) to potentially reduce measurement error. Once appropriate items are identified, they are combined in some way (e.g., summing across standardized scores for each item) to form the composite.

Another approach to the development of composite outcomes is to use principal component analysis (PCA) with all available items and let the results determine which items best represent a domain, as indicated by a high correlation with one of the factors identified by the model. These factors represent the empirical consensus of the items; thus, domains with more items are likely to emerge as primary factors in the model and under-defined domains may be error variance. As is true for the face-validity approach, different results might be obtained from a PCA depending on whether cross-sectional or longitudinal data are used in the analysis. Using baseline measures could yield a PCA solution with a first principal factor that includes both items that are sensitive to decline and items that are insensitive to decline. In contrast, the first principal component of an analysis of change scores across domains should reflect the group of items most sensitive to change over time regardless of their baseline factor structure.

A third approach to composite development focuses on maximizing sensitivity to decline by using the mean to SD ratio (MSDR) of change scores.⁵⁷ MSDR change scores are similar to standard scores in that scales with different measurement metrics can be combined. This approach is quite amenable to large-scale combination approaches, and original analyses were based on an exhaustive search (i.e., brute force) method where data from multiple clinical trials with multiple outcome measures were combined into a single database. The MSDR approach can also be implemented directly with a reduced rank regression model with time as the outcome. Development of a partial least-squares regression model that represents a compromise between principle components regression and reduced rank regression models allows simultaneous consideration of both the time factor and factors identified through convergence of the items.

Several companies have launched trials with new composite outcome measures that were developed through a combination of empirical (based on sensitivity of individual items) and construct validity (based on opinions of experienced clinicians and neuropsychologists) approaches.^58,71,74 Edland et al⁷¹ have taken a different tactic by attempting to optimize existing outcome measures. Using data from the Alzheimer’s Disease Neuroimaging Initiative, these investigators modeled data from a scale for activities of daily living (ADL), and reset the scoring paradigm using an IRT approach. They have shown that sample size estimates using this optimized outcome are reduced by approximately 17 to 20 percent.⁷¹ This approach makes the assumption that the targeted population will have a similar range of multivariate functioning as the “training set” (i.e., the ADNI cohort) used to adjust the scale. It also requires a very large database to serve as the training dataset. In addition, power calculations based on measures optimized in this manner include terms for the fixed and random variance, both of which are determined by the choice of outcome measure and the target population of the trial. Additional work with this measure using data from a clinical trial identified a learning effect during the first six months of the trial and suggested the need for a single-blind, run-in phase for three months. Including this run-in phase while not changing the overall 18-month length of the study reduced retesting effects and halved the sample size needed to show the same amount of decline on the outcome assessment.

Another way to possibly extract increased sensitivity from existing measures is to reanalyze existing trial data to gain a better understanding of the sources of variability in cognitive outcome measures that may mask true drug effects. One specific direction, for example, might be to remove items that contribute to error variance in outcome tests such as the ADAS-Cog and reassess drug-versus-placebo effects over time. It may be the case, for example, that subjective ratings of language and memory made by the examiner on the ADAS-Cog are susceptible to rater-to-rater and/or test session-to-test session variability that add noise to the signal of change over time. These subjective ratings are scored on a 0 to 5-point scale and can account for about 25 percent of the ADAS-Cog total score. It has been shown by Hobart et al^64,65 that the response categories for these subjective ADAS-Cog subtests could be made dichotomous without loss of information. It also possible that a version of the ADAS-Cog that includes only those items that are well-targeted to a population with mild cognitive changes (i.e., excluding items aimed at more severe illness) would provide a clearer signal of the cognitive impact of an intervention in early AD.

Regardless of the statistical approach used to improve sensitivity of outcome measures, consideration should be given to an evaluation of how well the assessment tool measures the latent trait under study. In the case of cognitive, functional, or clinical decline in AD, the measurement tool chosen should be capable of measuring the range of performance that will likely be exhibited by the people entering the specific study and the expected changes that will follow intervention. The gradation of measurement for each item should be sufficient to show any clinically meaningful change for every patient and, as such, should not be sparse.

DISCUSSION AND FUTURE DIRECTIONS

The design of prevention trials poses an enormous challenge and risk for sponsors. As clinical trials focus more on prevention of AD or intervention in very early disease stages, the assessment tools used to detect change will need to be matched to the status of the participants in the study. Furthermore, the psychometric properties of these assessment tools will need to be optimized for the study population. Newer statistical methods and novel cognitive and functional measures, as discussed by Harvey et al,⁴³ hold great promise in this regard as we move forward. While the majority of risk in any trial lies in whether the drug under study has the desired effect, failure to use appropriate tools for assessing cognitive and functional change adds the risk of missing interventions that work.

The use of unmodified outcome measures designed for more severe stages of illness in trials holds as great a risk as imprudent drug selection. Careful conceptualization of treatment targets and state-of-the-art understanding of the characteristics of the very early stages of AD are required to make progress. Regulatory guidance is being updated in concert with new developments in understanding treatment targets and populations of interest. While novel ways of using existing instruments hold some promise for detection of change in populations with very early AD, this effort will have to be paired with developing new assessment strategies.

ACKNOWLEDGMENT

This paper is a product of the “Cognitive Assessment of Early Alzheimer’s Disease in Clinical Trials” working group of the ISCTM. The executive committee of the society has endorsed this paper. The authors of the paper consist of individuals who presented at meetings, wrote and edited the manuscript, and approved its final content. Other individuals who attended these meetings contributed to the concepts and content of this paper. This paper does not reflect the opinions or endorsement of the employers of the authors or funding agencies that supported the research reviewed in this paper. Order of authorship, other than the workshop co-chairs, is presented in alphabetical order.