Factors influencing accuracy of cortical thickness in the diagnosis of Alzheimer's disease

Abstract

There is great value to use of structural neuroimaging in the assessment of Alzheimer's disease (AD). However, to date, predictive value of structural imaging tend to range between 80% and 90% in accuracy and it is unclear why this is the case given that structural imaging should parallel the pathologic processes of AD. There is a possibility that clinical misdiagnosis relative to the gold standard pathologic diagnosis and/or additional brain pathologies are confounding factors contributing to reduced structural imaging classification accuracy. We examined potential factors contributing to misclassification of individuals with clinically diagnosed AD purely from cortical thickness measures. Correctly classified and incorrectly classified groups were compared across a range of demographic, biological, and neuropsychological data including cerebrospinal fluid biomarkers, amyloid imaging, white matter hyperintensity (WMH) volume, cognitive, and genetic factors. Individual subject analyses suggested that at least a portion of the control individuals misclassified as AD from structural imaging additionally harbor substantial AD biomarker pathology and risk, yet are relatively resistant to cognitive symptoms, likely due to “cognitive reserve,” and therefore clinically unimpaired. In contrast, certain clinical control individuals misclassified as AD from cortical thickness had increased WMH volume relative to other controls in the sample, suggesting that vascular conditions may contribute to classification accuracy from cortical thickness measures. These results provide examples of factors that contribute to the accuracy of structural imaging in predicting a clinical diagnosis of AD, and provide important information about considerations for future work aimed at optimizing structural based diagnostic classifiers for AD.

1. INTRODUCTION

Alzheimer's Disease (AD) is the most common form of dementia in older adults and is characterized by significant loss or decline in memory, problems in learning, and other cognitive abilities (Bäckman, Jones, Berger, Laukka, & Small, 2004; Burns and Iliffe, 2009; Carlesimo and Oscar‐Berman, 1992; Dubois et al., 2014; Holtzman, Morris, & Goate, 2011; McKhann et al., 1984, 2011; Querfurth and LaFerla, 2010). The current gold standard for a conclusive diagnosis of AD is through postmortem examination to identify the disease‐defining regional patterns of neurofibrillary tau tangles and amyloid plaque deformities in the brain (Arnold, Hyman, Flory, Damasio, & Van Hoesen, 1991; Braak and Braak, 1991; Brun and Gustafson, 1976; Hardy, 2006; Hyman et al., 2012; Montine et al., 2012; Selkoe and Hardy, 2016). Mild cognitive impairment (MCI) is considered as a potential intermediate stage between normal aging and AD. Individuals with an MCI diagnosis have an increased risk of developing AD and conversion rate is reported to be approximately 10%–15% per year (Petersen et al., 2001). Although MCI is known to be a clinically heterogeneous condition, clinical diagnosis of probable AD is assumed to be a more stable condition that can be achieved with reasonable accuracy relative to the pathologic gold standard (Joachim, Morris, & Selkoe, 1988; Lopez et al., 2000). However, clinical misdiagnosis relative to pathology can be substantial and depends on several factors. For example, a recent review across a total of 919 samples with at least one clinical visit and autopsy demonstrated that clinical diagnosis of probable and possible AD has a sensitivity ranged from 70.9% to 87.3% while specificity ranged from 44.3% to 70.8% relative to the pathological diagnosis (Beach, Monsell, Phillips, & Kukull, 2012). The implication of this is that there is a relative uncertainty in the clinical diagnosis of AD as compared to the neuropathological diagnosis. Additionally, clinical diagnoses are often achieved late in the disease process when substantial brain tissue damage has resulted in noticeable cognitive deficit and other symptoms. Early clinical diagnosis is likely to have greater inaccuracy relative to the neuropathologic criteria (which is why diagnosis of MCI due to AD in the absence of biomarkers is particularly difficult). Given potential for clinical misdiagnosis and the need for early diagnostics, great emphasis has been put towards neuroimaging approaches for robust diagnosis of AD, prior to symptom development.

Structural imaging procedures have been successful in AD diagnostics (Chetelat and Baron, 2003; Frisoni, Fox, Jack, Scheltens, & Thompson, 2010; Park and Moon, 2016; Teipel et al., 2013). This is due to the fact that structural atrophy mirrors patterns of the characteristic regional pathology of this condition (Jack, Petersen, O'brien, & Tangalos, 1992; Jack et al., 1997; Scheltens et al., 1992). Several prior studies have applied structural imaging procedures in the later as well as earlier, and “preclinical” (e.g., MCI) stages of the disease (Dickerson et al., 2009; Eskildsen et al., 2013; Fotenos, Snyder, Girton, Morris, & Buckner, 2005; Jack et al., 1999; Killiany et al., 2000, 2002; Scheltens, Fox, Barkhof, & De Carli, 2002). Given current goals of clinical trials for identifying individuals in the earliest stages of disease, structural imaging could be advantageous as an initial screen for molecular procedures such as positron emission tomography which is currently used but costly and invasive.

AD detection using machine learning and magnetic resonance imaging (MRI) is a promising area of research. Support vector machines (SVM) is the most widely used procedure in neuroimaging studies to classify AD (Aguilar et al., 2013; Cuingnet et al., 2011; Klöppel et al., 2008; Magnin et al., 2009; Salvatore, Battista, & Castiglioni, 2016; Schmitter et al., 2015; Vemuri et al., 2008; Wolz et al., 2011). Although relatively successful, review of prior work demonstrates that there seems to be a “hard” limitation to accuracy with most studies reporting a range between 80 and 90%. There are two likely explanations for this limitation. First, to our knowledge, no prior work has been performed using a gold standard pathologic diagnosis and therefore all studies will have some degree of clinical misdiagnosis that contributes detrimentally to both the training and testing phases of classification. Second, it is likely that biological variability due in part to aging and other health and disease factors contribute to each individual's pattern of cortical thinning in a manner that may confound classification. To our knowledge, no prior studies have examined how these many factors contribute to machine learning classification accuracy of AD from structural MRI, however, recent studies have uncovered the need to consider age as a confounding factor contributing to misclassification of individuals with AD (Dukart et al., 2011; Falahati et al., 2016). Little is currently known however about misclassification of AD from structural MRI when age is removed as a factor.

Current strategies for therapeutic clinical trials aim to identify individuals in the earliest stages of AD. Such a task should be greatly enhanced by structural imaging, yet it is critical to know factors that influence classification based on brain structure alone. Although it is possible to enhance classification through overfitting (e.g., including a range of variables in the classifier that are not thought to be directly related to AD pathology), this would in fact decrease the pathologic accuracy of the classification by including individuals with impairment due to other etiologies which would confound therapeutic trials aimed specifically at AD pathology. The goal of our work was to define factors that contribute to the known 10%–20% misclassification from structural MRI, as opposed to achieving the greatest possible accuracy in classification. Specifically, we aimed to identify demographic and biological factors that were most influential on the accuracy of classification of individuals with a clinical diagnosis of probable AD exclusively using regional cortical atrophy patterns given known links between cortical atrophy and regional patterns of AD histopathology. This work follows on recent efforts to optimally integrate biomarker information into the early and accurate diagnosis of individuals with pathologic AD (Falahati et al., 2016; Hwang et al., 2016; Jack et al., 2016; Landau et al., 2010; Mattsson et al., 2015; Palmqvist et al., 2015).

Structural imaging was achieved with measurements of cortical thickness based on MRI which has been shown to index pathology in AD (Bakkour, Morris, & Dickerson, 2009; Dickerson et al., 2009; Lerch et al., 2008; McDonald et al., 2009; Salat et al., 2011). Cortical thickness measurements are sensitive to subtle degenerative changes making this imaging marker an ideal feature for AD classification (de Vos et al., 2016; Eskildsen et al., 2013; Raamana et al., 2015; Wolz et al., 2011). Classification performance was examined relative to demographic factors and biomarkers in misclassified individuals to gain insight into potential causes of misclassification including age, sex, education, Mini Mental State Examination (MMSE), cerebrospinal fluid (CSF) tau and amyloid‐beta, Florbetapir amyloid PET positivity (F‐AV45), APOE4 genotype, scanner type, American National Adult Reading Test (ANART) score, and Rey Auditory Verbal Learning Test (RAVLT) scores. We additionally examined the influence of white matter hyperintensity (WMH) burden on classification given this common type of brain tissue alteration in both typical aging as well as AD. WMH are highly prevalent in older adults and in AD and are related to brain structural measures, and therefore may be an important factor related to classification accuracy (de Leeuw, 2001; Debette and Markus, 2010; Hopkins et al., 2006; Mortamais et al., 2013; Murray et al., 2005; Provenzano et al., 2013; Ylikoski et al., 1995). Such information could be useful in determining whether misclassification was due to potential biological variability, clinical misdiagnosis, and/or technical limitations of the procedures and this information could be used toward further improvement of structural imaging procedures for the diagnosis of AD and provide potential insight into mechanisms of variability in cognitive expression of AD.

2. MATERIALS AND METHODS

2.1. Dataset

We used the structural brain MRI scans from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset (http://adni.loni.usc.edu). The ADNI was launched in 2003 as a public private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging, positron emission tomography, other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment and early AD. For up‐to‐date information, see http://www.adni-info.org. A total of 406 subjects (269 controls and 137 AD) were considered in this study. Standard 3 T baseline T1‐weighted images were included from the ADNI data set.

Several studies have reported the effect of aging on global and regional brain changes in controls and AD (Lim, Zipursky, Murphy, & Pfefferbaum, 1990; Salat, Kaye, & Janowsky, 1999; Salat, Kaye, & Janowsky, 2001; Salat et al., 2004; Shear et al., 1995; Thompson et al., 1998). As noted previously, age‐related brain changes can potentially lead to misclassification of younger AD patients and older control individuals. We therefore created matched control and patient groups for age and other demographic factors. 100 subjects (50 Controls and 50 AD) matched for age, sex, and education were used in the surface based cortical thickness analysis and the remaining 306 subjects were used for classification. Subject demographics for the entire study and for the matched groups used for cortical thickness study are presented in Table 1.

Table 1.

Subject demographics from the ADNI dataset

	Entire study		Cortical thickness study
	Control	AD	Control	AD
Number of subjects	269	137	50	50
Sex (Female/Male)	149/120	58/79	25/25	25/25
Age, years (mean)	72.86	74.16	72.81	72.77
Education, years (mean)	16.58	15.79	16.98	16.58

2.2. Cortical thickness measurement

The FreeSurfer image analysis suite version 5.3.0 (http://surfer.nmr.mgh.harvard.edu) was employed to process the MRI data and compute cortical thickness measurements. The technical details of cortical reconstruction and volumetric segmentation performed with the Freesurfer are described in prior publications (Dale, Fischl, & Sereno, 1999; Dale and Sereno, 1993; Fischl, Sereno, & Dale, 1999a; Fischl, Sereno, Tootell, & Dale, 1999b; Fischl and Dale, 2000; Fischl, Liu, & Dale, 2001; Fischl et al., 2002, 2004a, 2004b; Han et al., 2006; Jovicich et al., 2006; Reuter, Rosas, & Fischl, 2010; Reuter, Schmansky, Rosas, & Fischl, 2012; Segonne et al., 2004). Briefly, this processing includes motion correction and averaging of multiple volumetric T1 weighted images (M. Reuter et al., 2010), removal of nonbrain tissue using a hybrid watershed/surface deformation procedure (Segonne et al., 2004), automated Talairach transformation, segmentation of the subcortical white matter and deep gray matter volumetric structures (Fischl et al., 2002, 2004a), intensity normalization (Sled, Zijdenbos, & Evans, 1998), tessellation of the gray matter white matter boundary, automated topology correction (Fischl et al., 2001; Segonne, Pacheco, & Fischl, 2007), and surface deformation following intensity gradients to optimally place the gray/white and gray/cerebrospinal fluid borders at the location where the greatest shift in intensity defines the transition to the other tissue class (Dale et al., 1999; Dale and Sereno, 1993; Fischl and Dale, 2000). Once the cortical models are complete, a number of deformable procedures can be performed for further data processing and analysis including surface inflation (Fischl et al., 1999a, 1999b), registration to a spherical atlas which is based on individual cortical folding patterns to match cortical geometry across subjects (Fischl et al., 1999a, 1999b), parcellation of the cerebral cortex into units with respect to gyral and sulcal structure (Desikan et al., 2006; Fischl et al., 2004a), and creation of a variety of surface based data including maps of curvature and sulcal depth. This method uses both intensity and continuity information from the entire three dimensional MR volume in segmentation and deformation procedures to produce representations of cortical thickness, calculated as the closest distance from the gray/white boundary to the gray/CSF boundary at each vertex on the tessellated surface (Fischl and Dale, 2000). The maps are created using spatial intensity gradients across tissue classes and are therefore not simply reliant on absolute signal intensity. The maps produced are not restricted to the voxel resolution of the original data thus are capable of detecting submillimeter differences between groups. Procedures for the measurement of cortical thickness have been validated against histological analysis (Rosas et al., 2002) and manual measurements (Kuperberg et al., 2003; Salat et al., 2004). Thickness measurements were mapped on the inflated surface of each participant's reconstructed brain (Dale et al., 1999; Fischl et al., 1999a, 1999b). This procedure allowed visualization of data across the entire cortical surface (i.e., both the gyri and sulci) without interference from cortical folding. Maps were subsequently smoothed using a circularly symmetric Gaussian kernel across the surface with a full‐width‐half‐maximum (FWHM) of 10 mm. Next, cortical maps were averaged across participants using a nonrigid high‐dimensional spherical averaging method to align cortical folding patterns (Fischl et al., 1999a, 1999b). This procedure provides accurate matching of morphologically homologous cortical locations among participants on the basis of each individual's anatomy while minimizing metric distortion, resulting in a mean measure of cortical thickness at each point on the reconstructed surface.

2.3. Statistical analysis

Classifier features were selected as regions showing statistical thinning in the AD group compared to the control group. Whole‐brain surface‐based general linear models were performed at each surface vertex (10,242 vertices per hemisphere). Resulting z‐statistic maps were thresholded at p < .05. Multiple comparison correction was then performed using a clusterwise procedure (Hagler, Saygin, & Sereno, 2006). Surface data were corrected for multiple comparisons with a threshold of p < .0001 for both left and right hemispheres.

2.4. Support vector machine classifier

Support vector machine is a commonly utilized supervised, multivariate classification method. The problem of AD detection using SVM was formulated as a binary classification problem. In brief, given an N observation samples {x _i, y_i}, where x _i = [x_i1, …, x_in] ∈ ℜⁿ and y_i ∈ [1, −1] is the coded group label. If the sample belongs to that of an AD patient then the coded group label y_i is defined as one (y_i =1), otherwise, sample belongs to control where the coded group label is (y_i = −1). The SVM classifier finds a hyperplane maximizing the margin between groups. More details on SVM can be found in (Cortes and Vapnik, 1995; Schölkopf and Smola, 2002). In this study, we used the SVM implementation publicly available in LibSVM (csie.ntu.edu.tw/∼cjlin/libsvm). To determine a suitable cost parameter C and kernel parameter γ of the nonlinear Gaussian function in the SVM classifier, the parameter values are optimized using a cross‐validation via grid‐search approach (Chang and Lin, 2011). The grid search was performed over the ranges C = 2⁻⁵, 2⁻², …, 2¹⁵, γ = 2⁻¹⁵, 2⁻¹³, …, 2⁵. The optimized set of parameters was then used to train the SVM classifier using the training set. The cortical thickness values obtained using surface based thickness analysis were used as features in the SVM and classifier was trained to predict the group membership on the remaining 306 subjects. A total of 22 features (cortical thickness regions) were used for classification. We used the 10‐fold cross‐validation to estimate the SVM classifier performance. During each fold, the classifier was developed using 90% of the subjects as training data and remaining 10% of the subjects as testing data. For better generalization, the classifier was tested for 100 different random combination of training and testing datasets.

3. RESULTS

3.1. Cortical thickness comparison between controls and AD

Cortical thickness measurements were obtained using FreeSurfer on 100 subjects (50 Controls and 50 AD) matched for age, gender, and education. As expected, group differences in cortical thickness between the controls and AD were robust and are demonstrated in Figure 1.

Figure 1.

(a) Surface maps of the cortical thickness differences between the controls and AD groups smoothed on the surface with an approximate Gaussian kernel of a full‐width‐half‐max (FWHM) of 10 mm (p < .05 uncorrected). (b) Surface maps of the cortical thickness differences after the correction for multiple comparisons (thresholded at p < .0001) [Color figure can be viewed at http://wileyonlinelibrary.com]

Regions exhibiting significantly reduced cortical thickness in AD compared to controls following cluster‐based multiple comparison correction based on the Freesurfer Desikan/Killiany parcellation atlas (Desikan et al., 2006) were entorhinal cortex, precuneus cortex, fusiform gyrus, banks of the superior temporal sulcus, inferior parietal cortex, supramarginal gyrus, superior frontal gyrus, inferior temporal gyrus, superior parietal cortex, superior temporal gyrus, middle temporal gyrus, rostral middle frontal gyrus, caudal middle frontal gyrus, and pars opercularis. Mean cortical thickness within each significant region was extracted from the remaining 306 subjects, and used as an input to the SVM classifier.

3.2. Performance of SVM classifier

To illustrate the performance of the SVM classifier, mean cortical thickness values for each of the significant regions of interest were extracted from the remaining 306 subjects and used as features. A total of 22 features (cortical thickness regions) were used for classification. We used the 10‐fold cross‐validation to obtain an unbiased estimate of the classifier performance. During each fold the classifier was developed using data from 90% of the subjects and tested using data from the remaining 10% of the subjects. For better generalization, the classifier was tested for 100 different random combination of training and testing datasets. The sensitivity, specificity, and accuracy were calculated to measure the performance of the SVM classifier. Sensitivity is defined as the proportion of true positives that are correctly identified by the test and specificity is defined as the proportion of true negatives that are correctly identified by the test. Accuracy is calculated as the proportion of true results (both true positives and true negatives) by the test. In our experiments, mean accuracy of 90.32% (standard deviation 4.02), mean sensitivity of 0.96 (standard deviation 0.03), and mean specificity of 0.76 (standard deviation 0.14) were obtained using the SVM classifier.

Hippocampal volume has been used in several prior classification studies and is very effective to achieving classification to similar degrees as other structural measures. We therefore performed a secondary classifier analysis incorporating this imaging information with the initial classifier. Inclusion of hippocampal volume (corrected for ICV) as a feature increased the accuracy by 2.43%; however, we did not include hippocampal volume as a feature in the final classifier. This is because the goal of the work was to utilize structural features that are believed to be most related to the primary pathologic features in AD for a greater potential ability to identify individuals with true AD pathology. Our prior work demonstrated that hippocampal volume statistically factors with white matter lesions (WMH) which are not considered a primary pathologic feature of AD (Coutu et al., 2016, 2017). Inclusion of hippocampal volume in the classifier may slightly improve accuracy with regard to clinical diagnosis, but this may not be concordant with the pathologic diagnosis of AD. We therefore did not include hippocampal volume to attempt to minimize vascular related variance in the classification.

We additionally performed a classification analysis using whole‐cortex surface measurements. Cortical thickness values from the whole‐cortex were used as features to train an SVM classifier with an accuracy of 89.27% (standard deviation 3.49), sensitivity of 0.97 (standard deviation 0.02), and specificity of 0.74 (standard deviation 0.08). Although in the high‐performance range, results indicate that cortical thickness could not perfectly classify clinical group membership using the SVM classifier.

3.3. Analysis to determine factors contributing to misclassification

The results obtained using SVM classifier demonstrate that ∼10% of the subjects are misclassified based on structural imaging compared to their clinical diagnosis based on thickness values alone. These findings show that the procedures are robust for a large majority of the sample and therefore additional information about failures would better inform future implementations of SVM. Follow‐up analyses examined potential factors that may contribute to any individual being misclassified. As noted, limitations in accuracy are potentially related to technical restrictions, incorrect clinical diagnosis or the confounding effects of demographic and/or health disparities rather than the method used for classification (Falahati, Westman, & Simmons, 2014). In the following analyses, we therefore compared variables of interest across four groups: controls correctly classified as controls (classified CN), controls classified as AD (misclassified CN), AD correctly classified as AD (classified AD), and AD classified as controls (misclassified AD).

Demographic information. Initial analysis demonstrated that groups differed with regard to age as expected from prior work. Specifically, older control individuals were more likely to be classified as AD. However, age was not the only factor contributing to misclassification. We therefore created age and sex matched subsets of the four groups classified CN, misclassified CN, classified AD, and misclassified AD for subsequent comparison.

Cortical thickness. Given that the SVM is based on regional patterns of cortical thinning, it was expected that groups classified as having AD would have generally thinner cortex than groups classified as control. However, it is unknown whether these results would be regionally selective or more global. Thus, although a somewhat circular analysis (given that the classifier was based on thickness values) we next compared age and sex matched groups using surface based thickness general linear models to better understand the spatial nature of the thickness patterns in the misclassified groups for illustrative purposes.

Surface maps of the cortical thickness differences between correctly classified controls and misclassified controls revealed reduced cortical thickness in misclassified controls in several cortical regions (Figure 2a) including inferior parietal cortex, superior frontal gyrus, medial orbital frontal cortex, inferior parietal cortex, superior temporal gyrus, superior frontal gyrus, superior parietal cortex, entorhinal cortex, precuneus, postcentral gyrus, middle temporal gyrus, pars orbitalis, precentral gyrus, and lateral occipital cortex. In contrast, thickness differences between correctly classified AD and misclassified AD revealed that correctly classified AD had significantly reduced thickness in several regions including overlap with most of the classifier regions. Thicker cortex in misclassified AD was found in supramarginal gyrus, middle temporal gyrus, insula, inferior temporal gyrus, precentral gyrus, lateral orbital frontal cortex, superior frontal gyrus, precentral gyrus, superior parietal cortex, lingual gyrus, inferior parietal cortex, pars triangularis, entorhinal cortex, middle temporal gyrus, lateral occipital cortex, fusiform gyrus, and pars orbitalis (Figure 2b).

Figure 2.

Surface maps of the cortical thickness differences between the classified and misclassified groups smoothed on the surface with an approximate Gaussian kernel of an FWHM of 10 mm [Color figure can be viewed at http://wileyonlinelibrary.com]

Cortical thickness comparison between misclassified control and correctly classified AD demonstrated that although the misclassified controls had reduced cortical thickness relative to correctly classified controls (Figure 2a), they still had significantly thicker cortex than correctly classified AD in typical AD pathology regions. The regions included entorhinal cortex, middle temporal gyrus, fusiform gyrus, inferior temporal gyrus, supramarginal gyrus, middle temporal gyrus, inferior parietal cortex, and parahippocampal gyrus (Figure 2c) suggesting that any pathology in those regions may be an earlier stage and/or a different pathophysiological process. The misclassified controls have thinning in classifiers regions (regions sensitive to AD pathology). Compared to correctly classified controls, it would seem that this is a more global degenerative effect, and compared to correctly classified AD, the AD sensitive regions are relatively more affected in AD compared to the misclassified controls. This suggests that addition of features of the proportion of change in AD regions compared to other non‐AD‐specific regions could be an important addition to the classification process. Misclassified AD compared to correctly classified controls, exhibited greater cortical thickness in selective regions overlapping the classifier regions (Figure 2d). The regions including parahippocampal gyrus, superior temporal gyrus, inferior parietal cortex, entorhinal cortex, middle temporal gyrus, precuneus cortex, and supramarginal gyrus potentially suggest an earlier stage of disease pathology. The misclassified AD compared to controls has a pattern of thinning that is similar to the AD classifier regions. In theory, these individuals should have been detected by the classifier. It is possible that these individuals are therefore in an earlier stage of pathology and may require additional features to be accurately classified.

We next examined a series of variables that are associated with a diagnosis or enhanced risk for Alzheimer's disease including MMSE score, CSF tau and amyloid‐beta, (18) F‐AV45, APOE4 genotype, scanner type, and WMH volume. We used WMH available through the ADNI database, calculated from fluid attenuated inversion recovery (FLAIR) and T1‐weighted images using a Bayesian segmentation method (DeCarli, Murphy, Teichberg, Campbell, & Sobering, 1996; DeCarli et al., 1999; DeCarli, Pauline, & Evan, 2013). The WMH volume was expressed as a portion of intracrainial volume (WMH/ICV × 100) to correct for head size. CSF tau was used to pick extreme subjects that were unlikely to have a pathologic diagnosis matching the clinical diagnosis to illustrate how the factors of interest contribute to clinical classification accuracy. A single misclassified control subject (ADNI subject Id: 127_S_5185) was selected based on having the greatest CSF tau levels in that group (levels suggestive of AD pathology) represented as red circle and a single misclassified AD subject (ADNI subject Id: 052_S_4959) was selected based on having the lowest CSF tau levels in that group (levels suggestive of lack of tau pathology) represented a green circle for example comparisons. These single subjects are not presented as representative of the group, but simply as examples of the individual variability within the misclassified group. Specifically, we aimed to examine in detail example individuals that were most likely to have biomarker data that was inconsistent with their clinical diagnosis as potentially salient examples of factors influencing structural imaging classifier accuracy. Plots showing each of these markers across the four groups are demonstrated in Figure 3.

Figure 3.

Plots showing the distribution of CSF biomarkers, amyloid imaging, white matter hyperintensity, cognitive, genetic factors, and scanner types. A single misclassified control subject (ADNI subject Id: 127_S_5185) is represented as red circle and a single misclassified AD subject (ADNI subject Id: 052_S_4959) is represented as green circle. The symbol * indicates significantly different. AD = classified AD, mAD = misclassified AD, CN = classified control, mCN = misclassified control [Color figure can be viewed at http://wileyonlinelibrary.com]

We also examined educational level, ANART score, and RAVLT scores to assess the verbal learning, memory performance, and premorbid verbal intelligence of the groups. Plots showing each of these factors across the four groups are demonstrated in Figure 4.

Figure 4.

Plots showing the education level, ANART, and RVALT scores. Single misclassified control subject (ADNI subject Id: 127_S_5185) is represented as red circle and a single misclassified AD subject (ADNI subject Id: 052_S_4959) is represented as green circle. The symbol * indicates significantly different. AD = classified AD, mAD = misclassified AD, CN = classified control, mCN = misclassified control [Color figure can be viewed at http://wileyonlinelibrary.com]

At the group level, CSF tau and amyloid‐beta were not significantly different between classified and misclassified subjects within each group type (in AD and in controls compared to misclassified within group). Analysis of WMH volume suggested that misclassified controls had a significantly greater WMH burden compared to correctly classified controls (p = .011, by two‐sample t test) (Table 2). We therefore performed an additional classifier analysis incorporating white matter hyperintensity features with the initial classifiers. An accuracy of 91%, sensitivity of 0.98, and specificity 0.75 was obtained using the SVM classifier. Results suggest that inclusion of WMH features doesn't have a major impact on classification performance; however, there are several ways that WMH could provide additional classification information including use of spatial properties which differ in individuals with AD compared to controls. Thus, although these factors differ in the misclassified groups to some degree, the degree of heterogeneity prevents useful simple incorporation into the SVM classification procedure although future work will explore optimal parameters necessary for such incorporation.

Table 2.

Two‐sample t test analysis between classified and misclassified subjects within each group type

	Classified controls vs misclassified controls, p value	Classified AD vs misclassified AD, p value
Tau	.728	.990
Amyloid‐beta	.564	.195
WMH	.011*	.298
MMSE	.045*	.272
Education	.777	.346
ANART	.001*	.056
RAVLT (immediate)	.048*	.054
RAVLT (learning)	.883	.042*
RAVLT (forgetting)	.969	.213
RAVLT (forgetting %)	.754	.253

Note. * Significantly different.

Analysis of MMSE score revealed significant difference between the correctly classified controls and misclassified controls (p = .045). The distribution of APOE4 (positive/negative for at least one e4 allele) and (18) F‐AV45 uptake (standardized uptake values ratio (SUVR) > 1.11: florbetapir positive otherwise negative) were not different between the groups. We examined the influence of scanner types on the groups and found that distribution of Siemens and Philips scanners were not different between the classified and misclassified subjects. The variations attributable to individual scanner types may have less effect on the groups except for a greater representation of GE scanner type in the misclassified control group as shown in Figure 3. Although these data provide information about trends at the group level, within group variability across measures suggest heterogeneity supporting additional analyses of individual subjects from the sample.

ANART score, typically used as an estimate of premorbid verbal intelligence and serves as a proxy of cognitive reserve (Katzman et al., 1988; Lo and Jagust, 2013; McGurn et al., 2004; Schmand, Smit, Geerlings, & Lindeboom, 1997; Stern et al., 1994; Stern, 2012). ANART score was significantly different between the misclassified controls and correctly classified controls (p = .001). Analysis of RAVLT score revealed immediate recall scores were significantly different between the correctly classified controls and misclassified controls (p = .048) and also RAVLT learning scores were significantly different between the correctly classified AD and misclassified AD (p = .042). The RAVLT forgetting score, percentage of forgetting, and education level were not significantly different between classified and misclassified subjects. In summary, the misclassified controls had higher ANART and poorer RAVLT performance relative to correctly classified controls. These results suggest that some portion of the misclassified controls may in fact be in early stages of impairment, potentially masked due to higher premorbid function. Misclassified AD may be in the earlier stages of impairment relative to correctly classified AD. Although the thinning pattern is indicative of early AD pathology, it should be noted that misclassified AD did not differ from correctly classified AD with regard to biomarker levels or cognition. It is unclear why this is, however, it is possible that the cortical thickness measures are more sensitively quantifiable than the cognitive or CSF values (i.e., a range of thickness values related to variation in pathology are linked to similar cognitive and CSF values). Alternatively, it is possible that the differences in thickness are linked to subtle differences in white matter lesion volumes, or similarly, that the misclassified AD are generally healthier in various other domains that may influence cortical thickness whereas the typical AD patient is less healthy generally.

At the individual subject level, the misclassified control (misclassified control with the highest CSF tau levels across all misclassified controls) had MMSE of 30, APOE4 positive, (18) F‐AV45 positive, 0.51% white matter hyperintesity, 132 pg/ml amyloid‐beta, and 156 pg/ml tau. One sample t test presented in Table 3 showed significant difference between the single misclassified control subject and correctly classified controls across several parameters including increased CSF concentration of tau (selected for this but in the misclassified group), decreased concentration of CSF amyloid‐beta, increased WMH, APOE4 and (18) F‐AV45 uptake positivity, all considered to be typical characteristic factors related to presence or risk for AD. The misclassified control had an education of 20 years, ANART total score (no of errors) of 2, RAVLT immediate recall score (total of 5 trails) of 51, learning score (6th trial) of 10, forgetting score (30 min delay) of 1, and percentage of forgetting 7.14%. RAVLT score in this individual suggested subtle impairment, however, this individual did not convert to MCI or AD during the course of ADNI participation (from baseline visit to 24 months visit). These consistent results indicates the possibility of clinical misdiagnosis relative to pathologically positive AD in certain individuals in the control group, likely due substantially to “cognitive reserve” based in high education and premorbid intelligence. The individual misclassified AD had lower hyperintensity burden, and greater MMSE than correctly classified AD. The individual had high education and premorbid intelligence, which likely contributed to some preserved cognitive capacity. Taken together, it is possible that this individual is impaired for reasons other than AD pathology.

Table 3.

One‐sample t test analysis between correctly classified group and misclassified single subject within each group type

	Misclassified single control subject	Misclassified single AD subject	Correctly classified control group vs misclassified single control subject, p value	Correctly classified AD group vs misclassified single AD subject, p value
Tau (pg/ml)	156	42	<.001	<.001
Amyloid‐beta (pg/ml)	132	119	<.001	<.001
WMH (%)	0.51	0.47	<.001	>.05
MMSE	30	25	<.001	<.001
Education	20	20	<.001	<.001
ANART	2	6	<.001	<.001
RAVLT (immediate)	51	26	<.001	<.001
RAVLT (learning)	10	1	<.001	>.05
RAVLT (forgetting)	1	5	<.001	<.001
RAVLT (forgetting %)	7.14	83.33	<.001	>.05

The misclassified AD subject had MMSE of 25, was APOE4 positive, (18) F‐AV45 positive, 119 pg/ml amyloid‐beta, and 42 pg/ml tau, but WMH of 0.47%. The misclassified AD had an education of 20 years, ANART score of 6, RAVLT immediate recall score of 26, learning score of 1, forgetting score of 5, and percentage of forgetting 83.33%. The misclassified AD remained fairly stable cognitively (memory loss prominent with poor insight) and functionally from baseline visit throughout their participation in ADNI (to 12 months visit). This example is somewhat inconsistent in having low tau but also low amyloid CSF values (tau levels would be expected to be high and amyloid low in AD). Findings suggest that given the high general prevalence of WMH in AD, patients with less WMH are likely to show altered patterns of cortical thinning compared to those who do not have WMH which may contribute to misclassification. Overall, the findings from the individual participant analyses demonstrate that individuals within the sample have clusters of demographic, biomarker, and neuropsychological profiles that are inconsistent with a match between the clinical diagnosis and likely existence of AD pathology and also highlight factors that modulate the efficacy of the structural classification.

4. DISCUSSION

There is strong interest in AD “diagnostics” using machine learning and structural MRI (Cho et al., 2012; Coupé et al., 2012; Davatzikos, Bhatt, Shaw, Batmanghelich, & Trojanowski, 2011; Falahati et al., 2014; Klöppel et al., 2015; Liu et al., 2012; Schouten et al., 2016; Westman et al., 2011a, 2011b; Westman, Muehlboeck, & Simmons, 2012; Zhang et al., 2011). These studies have demonstrated the obvious utility of structural imaging in this endeavor. Several studies have shown that hippocampal atrophy is an early indicator of AD (Jack et al., 1999; Killiany et al., 2002; Rana et al., 2017; Schröder and Pantel, 2016). The hippocampus has therefore been used in several prior classification studies (Chupin et al., 2009; Colliot et al., 2008; Frisoni et al., 1999) and is very effective to achieving classification to similar degrees as other structural measures. However, most prior studies aimed to enhance classification accuracy relative to clinical diagnosis through the optimal selection of classifiers for matching the clinical label. Little is currently known about factors that contribute to misclassification of AD purely from structural MRI which should mirror the primary pathologies of AD. The goal of this work was to determine factors that contribute to the mismatch between structural classification and clinical diagnosis through examination of a range of demographic, biological, and neuropsychological data. The overall findings demonstrate that in fact, although a clinical misclassification of ∼10% was found, the structural classifier may actually have greater accuracy for a pathologic diagnosis (which would be most useful for clinical trials and therapeutics directed towards primary AD pathology), and also that secondary pathologies (such as WMH) influence thickness values in regions overlapping typical AD pathology and therefore the accuracy of classification.

Cortical thickness measurements based on MRI has been shown to index pathology in AD (Bakkour et al., 2009; Dickerson et al., 2009; Lerch et al., 2008; McDonald et al., 2009; Salat et al., 2011). In this study, as expected based on the use of cortical thickness measures in the SVM classifier, whole surface contrasts of cortical thickness maps between classified controls and misclassified controls revealed reduced cortical thickness in misclassified controls compared to correctly classified controls as shown in Figure 2. However, the regional patterns of reduced thickness were somewhat distinct from regions showing strong effects for AD including the regions used in training the classifier and seemed to be more global in nature, at least at the group level.

The need for integration of biomarker information into the diagnosis of individuals with pathologic AD has been discussed extensively (Jack et al., 2016; Mattsson et al., 2015; Palmqvist et al., 2015). These consensus papers suggest that amyloid‐beta biomarkers, tau biomarkers, and biomarkers of neurodegeneration are necessary to identify early AD with high accuracy and would be useful to understand disease pathogenesis and expedite drug development. WMH are common type of brain tissue alteration in older adults, more prevalent in AD and are related to brain structural measures and therefore this tissue damage is an important factor related to diagnosis of AD. Several studies have investigated the association of WMH and cortical atrophy and generally found a higher degree of cortical atrophy among individuals with higher burden of WMH (Appelman et al., 2009; Capizzano et al., 2004; Godin et al., 2009; Raji et al., 2012). There is increasing evidence of WMH association with cognitive decline (Prins and Scheltens, 2015; Provenzano et al., 2013; Rieckmann et al., 2016). Recent studies have demonstrated interactions between WMH and cortical thickness and cognition (Jacobs, Clerx, Gronenschild, Aalten, & Verhey, 2014; Seo et al., 2012; Tuladhar et al., 2015). In this study, we found misclassified individuals differed from their correctly classified counterparts on several of these relevant measures. For example, misclassified controls, as a group, had a significantly greater WMH burden compared to correctly classified controls. The controls with more WMH are likely to show altered patterns of cortical thinning compared to those who do not have WMH which may contribute to misclassification. These findings suggest that WMH may be used as an additional biomarker for early and accurate diagnosis of AD. Ongoing studies are exploring the utility of WMH in classification of AD, however, given that our goal here was to classify based on features considered to be linked to the primary pathology of AD, the inclusion of WMH to better match the clinical diagnosis would not necessarily help in achieving the goal of a pathologic classification.

Education and premorbid verbal intelligence typically serves as a proxy of cognitive reserve. Prior studies have shown association of the cognitive reserve markers with a lower risk of AD and memory decline (Buckner, 2004; Katzman et al., 1988; Lo and Jagust, 2013; Murray et al., 2011; Stern et al., 1994; Stern, 2012). Premorbid intelligence assessed by ANART modifies the relationship between biomarkers of pathology and cognition in AD with individuals with high cognitive reserve having greater biomarker abnormalities than those with low cognitive reserve (Vemuri et al., 2011). This study demonstrates misclassified controls had a significantly greater ANART, RAVLT immediate recall, and MMSE score compared to correctly classified controls. Thus, cognitive factors have a compensatory function that acts to preserve a clinical state despite reduced cortical thickness in some control individuals. It is therefore possible that the structural MRI measures provide an accurate pathologic, but not clinical diagnosis.

One limitation of the current work is that the classification procedure relied on clinical diagnostic information for determination of features selected. Although the large majority of participants likely have a match of their clinical diagnosis with brain pathology, it is possible that some inaccuracies in diagnosis shifted performance of the classifier to some degree. Future investigations should consider creation of a classifier based on “high confidence” individuals with diagnoses that are strongly supported by genetic and CSF biomarker data for optimal weighting towards classification based on AD pathologic processes. In the current investigation, a single misclassified control subject was selected based on having the greatest CSF tau levels in that group and a single misclassified AD subject was selected based on having the lowest CSF tau levels in that group for example comparisons. These single subjects are not presented as representative of the group, but simply as an example of the individual variability within the misclassified group. Although we highlight two individuals as an example, full subject data for all subjects classified and misclassified is provided in Figures 3 and 4. These plots show trends in the data at the group level. As noted, there are a range of values for correctly classified and misclassified individuals. This suggests that there is more than one explanation for misclassification as we tried to touch on in the current work. A secondary procedure such as hierarchical clustering could potentially subclassify the misclassified individuals and provide insight to the types of misclassification that occur in the sample as follow up investigation to the current work.

The major conclusions from current work are as follows: (a) WMH have an important influence on classification accuracy, with individuals with a clinical diagnosis of AD being more likely to be classified as control when WMH volume is low, and control individuals being more likely to be classified as AD when WMH volume is high. (b) At an individual level, subjects in the ADNI sample have biomarker evidence of AD pathology while remaining relatively cognitively resilient and having a clinical diagnosis of being nondemented. (c) Individuals clinically diagnosed with AD but misclassified based on cortical thickness patterns may have a pathologic diagnosis inconsistent with AD. These findings demonstrate the need for integration of biomarker based diagnostic criteria as has been described in recent consensus papers and recent classification work (Jack et al., 2016; Mattsson et al., 2015; Palmqvist et al., 2015). Here we note the critical contribution of white matter lesions to biomarker interpretation. A more detailed analysis of individual level factors across many subjects in the sample will be important follow up work to the data presented here. Thus, patterns of cortical thinning alone cannot be the only features to be used in clinical classification schemes. In the absence of additional biomarker data, some information regarding premorbid function and WMH burden must likely be included in these procedures. Additionally, future investigation should be explicit regarding goals of clinical versus pathologic diagnostic accuracy. Future work will examine the degree to which subgroups of individuals can be determined from the misclassified individuals based on clustering of AD and demographic related parameters to better understand features classes that contribute to classification accuracy.

Belathur Suresh M, Fischl B, Salat DH. Factors influencing accuracy of cortical thickness in the diagnosis of Alzheimer's disease. Hum Brain Mapp. 2018;39:1500–1515. 10.1002/hbm.23922