A blood test for Alzheimer’s disease: progress, challenges and recommendations

Abstract

Ever since the discovery of APOE ε4 around 25 years ago researchers have been excited about the potential of a blood test for Alzheimer's disease (AD). Since then researchers have looked for genetic, protein, metabolite and/or gene expression markers of AD and related phenotypes. However, no blood test for AD is yet being used in the clinical setting. We first review the trends and challenges in AD blood biomarker research, before giving our personal recommendations to help researchers overcome these challenges. While some degree of consistency and replication has been seen across independent studies, several high-profile studies have seemingly failed to replicate. Partly due to academic incentives there is a reluctance in the field to report predictive ability, to publish negative findings and to independently replicate the work of others. If this can be addressed then we will know sooner whether a blood test for AD or related phenotypes with clinical utility can be developed.

Progress

The identification of genetic markers such as APOE ε4 arguably represented the first step change in progress towards a blood test for late onset Alzheimer's disease (AD), as genetic markers can be measured from blood samples [1]. Since then 19 other significant markers of AD have been identified by a Genome-Wide Association Study (GWAS) [2]. These markers have been combined into a polygenic risk score with ~80k more weakly associated genetic markers achieving an Area Under the Curve (AUC) of 78% for prediction of AD. This compares with 72% achievable with just age, sex and APOE ε4 (the `co-variate only' model) [3]. In a smaller recent study (N ~ 1,600) some of the same authors have shown that the same risk score has an AUC of 84% for predicting pathologically confirmed cases [4], which if confirmed in larger studies may have enough clinical utility to justify the use of genome-wide genotyping in the clinic to aid diagnosis. Even an AUC of 78%, if validated further, may have some utility for recruitment of higher risk individuals to prevention trials [5]. Another promising approach has been to develop polygenic hazard scores to predict age of dementia onset, although the utility of this for the clinic or for enriching trials has not yet been fully assessed [6].

AD polygenic risk scores have also been shown to be associated with brain atrophy [7,8] and cerebrospinal fluid (CSF) amyloid beta [6,7], but to the best of our knowledge they have not yet been shown to be predictive of these phenotypes. In fact, two independent studies (N = 657 and N = 242) have shown that this genetic risk score does not appear to be predictive of amyloid or tau pathology measured from CSF [9,10]. A preliminary report that the polygenic hazard score is better able to predict elevated brain amyloid does not appear to test whether it improves upon a model using age, gender and APOE alone mean its clinical utility is uncertain [11]. If the negative findings are correct then this would be consistent with the idea that biomarkers must be optimised to specific use contexts, e.g. useful markers of AD diagnosis may differ from markers of pathology or progression. For this reason the literature has broadened out from the case-control design to encompass endophenotype designs which look for markers of brain atrophy [12], CSF pathology [13] or cognitive decline [14]. However, it should be noted these findings do not yet seem to have translated into accurate prediction models [15].

Following on from the early genetic work, proteomic researchers joined the search for blood biomarkers of AD. In a clear parallel to earlier genetics research two main approaches were taken: ‘candidate’ studies and ‘discovery’ studies. The AlzBiomarker project has performed a meta-analysis of association studies of ‘candidate’ blood (and CSF) markers with AD, showing that of all candidates in blood, total-tau appears to be the most associated with AD [16]. Predictive results from the use of blood total-tau are only provided for individual small-scale studies, so the reported sensitivity and specificity for predicting AD diagnosis is likely to be overly optimistic, e.g. 97% specificity and 91% sensitivity in Chiu et al., [17].

We have reviewed the 21 ‘discovery’ blood protein studies published between 2002 - 2014, using a wide-range of proteomics techniques and AD-related outcomes. A low consistency of biomarkers identified between studies was observed, but four candidate biomarkers were observed in studies utilizing five independent research cohorts: α-1-antitrypsin, α-2-macroglobulin, apolipoprotein E, and complement C3. When examined in a new dataset these proteins, when combined with age, sex and presence of APOE ε4, had an AUC of 82% for predicting AD diagnosis, versus 79% for the co-variate only model [18]. While superficially better than the genetic risk scores AUC (78% versus 82%), it should be noted that the genetic result is more trustworthy as it comes from a much larger study and shows a greater difference in predictive ability between co-variate only and biomarker models.

In another parallel to genetic research, proteomic studies have also looked-for biomarkers of endophenotypes of AD, which have been reviewed in Baird et al., [19] who see promise in this area, but acknowledge limited success in identification of a reproducible signature. More recently, Ovod et al., [20] have developed an assay for plasma amyloid beta 42 and 40, and used it to show that the ratio has an AUC of 89% for predicting amyloid positivity in a small cohort (N = 41). However, this work has not yet been replicated in independent cohorts.

Blood metabolite studies of AD are more novel, with high profile papers by Mapstone et al., [21] and Proitsi et al., [22]. Mapstone et al., [23] identified ten lipids which could predict conversion from Mild Cognitive Impairment to AD over 2 - 3 years with an AUC of 92%. Proitsi et al., [22] identified 24 metabolites which had an AUC of 71% for predicting AD diagnosis, in a considerably larger study (N = 277 versus N = 85).

Gene expression has also been explored as a potential source of blood biomarkers for AD. Little consistency has been seen in the genes selected by these various studies, leading Han et al., [23] to suggest that greater concordance might be seen at the pathway level. We demonstrated in Voyle et al., [24] a failure to replicate classifiers between independent sample sets, and that simple pathway level summaries of gene expression are no more predictive of AD. Endophenotype approaches have been explored in gene expression studies as well, for example Lunnon et al., [25] show that gene expression is predictive of brain atrophy. However, this work has not yet been replicated in independent cohorts.

More recently, we have been attempting to combine different modalities of biomarker to improve predictive ability. In Voyle et al., [26] we found that five metabolites could be used to predict amyloid positive individuals with 72% accuracy, rising to 79% when combined with levels of the protein fibrinogen gamma. This study was limited in size and requires replication in independent samples. In a similar vein, we showed a marginal improvement in the prediction of CSF amyloid beta levels using genetic risk from the large AD GWAS [2,3] and plasma tau levels (AUC 67% versus 66% for co-variate only model) [9]. Such a small improvement may be artefactual, and even if true is not likely to be useful by itself. Studies seeking to find multi-modal AD blood biomarkers and/or biomarkers of endophenotypes are likely to become more common but have been held back by the sample sizes available, which is a focus for improvement going forward.

Challenges and recommendations

Further progress towards a clinically useful blood test will be slow unless we acknowledge and learn from the limitations of our current approaches, therefore what follows is our personal view on key challenges and important recommendations for future AD blood biomarker research. We draw attention to limitations of existing studies not to dismiss them, but to point out room for improvement in the field and in our own research. What follows will seem obvious to some, but needs to be highlighted as a counter-point to the over-optimism of the field.

The quality of experimental design in this field is variable, although this has been improving over time. One of the most obvious aspects of this is in sample size of non-genetic AD biomarker discovery studies, for example a study seeking to find blood protein AD biomarkers in 2002 used only 18 research participants [27]. Ten years later Doecke et al., [28] achieved N ~ 1000. This seemed a positive trend, but unfortunately, we are not aware of any larger studies published in the five years that followed. In fact, many smaller-scale studies (N ~ 100) are still published, e.g. [20,21,26,29,30]. We should learn from the field of genetics, where small-scale candidate gene studies were plagued with replication issues [31] that were only solved by larger sample size and unbiased approaches. We recommend that this is tackled, in part, using samples from larger cohorts, such as UK Biobank [32] and the Precision Medicine Initiative [33], as well as cohort consortia such as the European Medical Information Framework - AD (http://www.emif.eu/) and Dementia Platform UK (http://www.dementiasplatform.uk).

Additional design considerations involve the appropriateness of the population used, this should be guided by the anticipated context of use of the potential blood tests. Most studies have sought to find a blood test that could be helpful in the diagnosis of Alzheimer's disease, however none have yet been performed in the primary healthcare population in which it would have greatest utility [34]. This has been in large part due to the challenges of recruiting research participants, and the priority given to large scale recruitment rather than to representativeness of populations relative to anticipated context-of-use. Similarly, studies have sought to find AD markers that could be used to identify asymptomatic patients with early signs of AD, but very few have been performed in that population [35,36,37,38]. Another problem of unrepresentative sample populations is that they may not reflect the prevalence of AD related phenotypes (e.g. amyloid positivity) in populations appropriate to the anticipated context-of-use, which could inflate the positive and negative predictive values.

Partly due to the history of this field, which was initially led by clinicians and lab biologists, the level of statistical rigour is understandably variable. One example is the focus of many papers on p-values instead of predictive ability (sensitivity/specificity/positive predictive value etc). Significant p-values, even if replicable, do not necessarily mean that a biomarker is useful for predicting AD. To do so requires a suitably large effect size and a good understanding of confounding factors (e.g. age, gender, APOE ε4, medication use). For readers who may struggle to interpret and understand predictive measures we heartily recommend Tze-Wey Loong's excellent visual explanation [39].

A critically important consideration is cross-validation, i.e. the assessment of predictive models in additional data not used in its construction. The data used in model construction is referred to as the training or in-sample dataset, whereas the independent data used for assessment is called the test or extra-sample data. It is important that predictive performance is reported from the test data, as results from training data can be artefactually better due to overfitting to noise [40]. This is equivalent in importance to blinding in clinical trials in the sense that it helps to protect results from the preconceptions of the researcher. This can be a major concern in studies which report predictive accuracy in training sets only, i.e. where no cross-validation has been performed. In some studies, k-fold cross-validation has been performed, in which the training data is repeatedly split into different training and test subsets and average performance in the test sets given. This is better than no cross-validation, but can still give overly generous predictive performance due to the train and test datasets sharing the same systematic noise, especially in small datasets and when k < 10 [41].

Despite seemingly performing cross-validation, researchers can often subconsciously and artefactually inflate predictive performance. The two most common examples of this are (1) when both train and test data are used for variable selection, for example when variables are selected for model inclusion by ranking the p-values from repeated univariate tests using all available data (and therefore not correctly holding out test data), and (2) by performing correct cross-validation, finding a poor result which does not get reported, and then trying a new model (new variables, or new modelling approach or formula), only reporting models which have good predictive performance. The former mistake is one that we have made ourselves in Kiddle et al., [29], the latter mistake is probably the most common which is evidenced in part by the relative absence of negative results published in this area. The absence of negative results is a form of reporting bias, leading to an overly optimistic impression of this field in the literature. While no evidence of reporting bias for ‘candidate’ blood protein markers of AD is seen in the AlzBiomarker meta-analysis, in the context of discovery and replication of blood biomarker panels we are only aware of the following relevant papers showing negative results, some emerging from our own group [9,24], some consistent with our negative findings [10], and others showing limited [42,43] or no replication [44] of previous high-profile positive results [21,45] when examined by independent researchers. It seems implausible that negative results are truly rare in AD blood biomarker research.

In terms of root causes, we are generally dis-incentivised to share negative or disappointing results, even if they are correct. However, the examples provided above demonstrate that it is simply not true that journals refuse to publish negative results, although we have certainly argued against peer reviewers who fail to see the value in doing so. In a worst case scenario negative results can be published on pre-print servers such as bioRxiv (http://biorxiv.org/). It may be worthwhile for the community to come together to generate a pre-registration platform, and to persuade journals to only publish pre-registered AD biomarker studies, so that groups failing to publish negative findings can be pressured into doing so. While not explicitly focused on blood markers, a very positive move in this direction has been challenges in which predictions are submitted before test data is released, notably the AD DREAM challenge [15], which generated a negative result, and the TADPOLE challenge which is currently open for submission until November 2017 (https://tadpole.grand-challenge.org/).

Another cause could be confirmation bias, the psychological phenomena where individuals seek out evidence that matches their pre-conceived ideas [46]. This can mean that if a researcher believes a blood marker of AD will be found they may disregard evidence to the contrary. This can lead to Hypothesising After Results Are Known, researchers testing many different hypothesis (or models) reporting only the positive results and ignoring the multiple testing problem relating to all the unreported negative results [47]. This has also been called the file drawer effect, meaning that many published results are false positives [48]. John Ioannidis has discussed this in his controversially titled paper – “why most published research findings are false” [49]. While this focuses on p-values, it is likely to apply to studies reporting predictive performance as well. Interestingly, the field of blood biomarkers for AD fits his criteria for risk of high false positive publication rate: (1) small studies, (2) small effect sizes, (3) greater number and lesser pre-selection of tested relationships, (4) flexibility in designs and analyses, (5) financial interests and prejudice, and (6) many teams involved in a scientific field in chase of statistical significance.

The testing of promising biomarkers by independent researchers is essential to progress them towards the clinic [34], however this is rare. This has therefore been a focus of our research [18,50]. Where this have been done for two high-profile studies [21,45] they show a complete failure to replicate [44] or a significantly less promising performance [42,43]. Less promising replications fit with the findings of John Ioannidis [51] who has studied the reasons that discovery results are typically inflated. In terms of clinical utility a high predictive performance is required, meaning that partial replications in well-designed studies almost always rule out the clinical utility of the marker.

Given all the above it is healthy to be sceptical about the potential of biomarkers in the current literature. It is a safe bet to assume that existing AD biomarker candidates at the very least require further validation and at worst are non-replicable, or are not of sufficiently high performance for clinical use. We hope this changes soon, but we believe that our recommendations may help the field to achieve this sooner.

While it is tempting to think that these problems may plague all blood biomarker research, this can be disproved by the example of blood protein markers of aging. Using proteomics techniques that have also been used in AD research we have discovered plasma proteins correlating highly with chronological age, that replicate strongly in an independent cohort [52], in another cohort in serum [50] and in at least two studies by independent researchers [53,54]. This shows that failures to replicate AD biomarkers cannot solely be blamed on the measurement technologies used, but it is certainly true that if a novel technology was able to detect a stronger AD related signal then it is more likely to be replicated.

Given the large reproducibility crisis in science [49], and in this field specifically, how do we improve the way we do research to increase the chance that a reproducible blood test can be found? John Ioannidis has provided general recommendations for researchers to improve the chance of true positive findings that we think are relevant: “large-scale collaborative research, replication culture, registration, sharing, reproducibility practices, better statistical methods, standardization of definitions and analyses, more appropriate (usually more stringent) statistical thresholds, and improvement in study design standards, peer review, reporting and dissemination of research, and training of the scientific workforce” [55]. Other recommendations from John Ioannidis regard institutional changes affecting the incentives for scientists which would be even harder to achieve.

As highlighted by John Ionnadis sharing and quality of reporting is important, and for that reason we strongly recommend that the field adopt the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guidelines [56]. Specifically for the reporting of pre-analytical variables, we recommend the advice provided by Sid O'Bryant et al., [57].

In terms of sharing we recommend that raw data should be shared as widely as possible, allowing independent statisticians to verify reproducibility of findings. This has been most successfully achieved through the Alzheimer's Disease Neuroimaging Initiative (https://ida.loni.usc.edu/), but is also being done to some extent by other cohorts including the Australian Imaging and Behaviour Longitudinal study of aging (https://ida.loni.usc.edu/) and AddNeuroMed (https://www.synapse.org/#!Synapse:syn4907804). Clinicians leading large data collection need to be won over by the benefits of data sharing, including by inclusion of appropriate authorships (e.g. of clinical consortia), to counter the recent high-profile accusation that data sharing leads to “research parasites” [58]. While data sharing is done to some extent, the field would be greatly improved by researchers also sharing analysis scripts wherever possible. This would allow errors to be spotted, would make reproducibility straightforward, and would greatly assist junior researchers to develop the coding skills that are increasingly important for modern biomedical research. Code sharing can be facilitated by websites such as GitHub (https://github.com/), which allow both private and public sharing of code, as well as version control. Care must of course be taken not to release data that is sensitive, including within the script itself, but we have shown that this can be achieved [59].

Our final recommendation is that the ultimate aim is prediction models useful in a given clinical context, and that we shouldn't be limiting ourselves to looking for markers in blood. Other variables, derived from routine clinical data such as electronic health records, wearables or cognitive tests may have more promise for this purpose. This is not to say analysis of these datasets is not without its own challenges. Electronic health records represent sparse, incomplete and often subjective representations of the disease state with important data often buried within free text narrative. The opportunities here are vast though, and although the datasets represent secondary use data, the data themselves are much larger and more representative of clinical settings than we are typically used to in blood biomarker studies, albeit less controlled in terms of co-variates and missing data. We have established research programmes in this area and through information and extraction toolkits such as Clinical Records Interactive Search [60], CogStack [61] and the KConnect programme (KConnect.eu), rolled out to multiple hospitals. We have used natural language processing to explore questions that include characterising trajectories of cognitive decline with a specific focus on identifying and validating associations, with medications for example [62]. The ubiquitious use of smartphones and wearables devices provides the opportunity for a more objective, continuous and pervasive phenotype, throughout the disease continuum from at risk, early diagnosis through to post diagnosis engagement, compliance and self-management. Such data provides the opportunity to augment our blood biomarker studies and clinical trials. We have established programmes such as the RADAR-CNS (RADAR-CNS.org), a major goal of which is to develop a generalised real-time streaming platform that will enable active (eg through questionnaires) and passive (eg. accelerometry and heart rate using sensors on wearables and devices) remote monitoring, tracking phenotypes such as function and cognition.

Conclusions

The field of AD blood biomarkers has expanded to include genetic, protein, metabolite and gene expression markers, as well as combinations of the above. While some consistency has been seen across independent studies, several high-profile studies have seemingly failed to replicate. At the same time, there appears to be a strong reporting bias for studies seeking to find a biomarker panel with few negative results published, making the true state of play impossible to assess. Will a clinically useful blood test for AD be developed? It is simply too early to say, but we will have a better chance if we can improve the design, analysis and reporting of studies. Many alternative markers exist within health records, from wearables or innovative cognitive tests, and these should also be explored.