Generalizability of findings from a clinical sample to a community-based sample: A comparison of ADNI and ARIC

Abstract

INTRODUCTION:

Clinic-based study samples, including the Alzheimer’s Disease Neuroimaging Initiative (ADNI), offer rich data, but findings may not generalize to community-based settings. We compared associations in ADNI to those in the Atherosclerosis Risk in Communities (ARIC) study to assess generalizability across the two settings.

METHODS:

We estimated cohort-specific associations between risk factors, cognitive test scores, and neuroimaging outcomes to identify and quantify the extent of significant and substantively meaningful differences in associations between cohorts. We explored whether using more homogenous samples improved comparability in effect estimates.

RESULTS:

The proportion of associations that differed significantly between cohorts ranged from 27% to 34% across sample subsets. Many differences were substantively meaningful (e.g., OR for APOE-4 on amyloid positivity in ARIC: OR=2.8, in ADNI: OR=8.6).

DISCUSSION:

A higher proportion of associations differed significantly and substantively than would be expected by chance. Findings in clinical samples should be confirmed in more representative samples.

1. INTRODUCTION

Generalizability is an oft-cited limitation of epidemiological studies, particularly when relying on clinic-based study samples.¹ The Alzheimer’s Disease Neuroimaging Initiative (ADNI) is an example of a clinical cohort that collects high quality imaging, biomarker, genetic, and clinical data.² ADNI data are shared publicly, and have allowed researchers to make important progress in many areas, particularly around biomarkers associated with presence and progression of Alzheimer’s disease (AD).^3,4 Because ADNI recruitment procedures were designed to obtain a sample emulating that of a clinical trial, the sample is generally healthy, predominantly white, and overwhelmingly well-educated.^5,6 Researchers using ADNI typically recognize and acknowledge the potentially limited generalizability of their findings,^1,6-10 but the extent of this limitation remains unclear.⁶ Given the richness and high quality of data from clinical cohorts, it is important to understand when findings in these samples are likely to generalize to target populations, and when representative sampling is necessary to make broader inferences.

The Atherosclerosis Risk in Communities (ARIC) study collects cognitive and imaging measures similar to those in ADNI. Importantly, ARIC is a community-based cohort, representing another target population of interest. Our goal was to compare associations between risk factors, cognitive outcomes, and neuroimaging outcomes in ADNI and in ARIC to examine the degree to which associations differ, and whether particular types of associations are more or less comparable. This may help identify whether and when associations estimated in clinic-based samples such as ADNI may generalize more broadly to a community-based setting, using ARIC as an example, in the context of dementia research.

2. METHODS

2.1. Study Samples and Eligibility Criteria

2.1.1. ADNI

The Alzheimer’s Disease Neuroimaging Initiative (ADNI; adni.loni.usc.edu) is a longitudinal study of adults aged 55-90 launched in 2003 as a public-private partnership. The primary goal of ADNI is to test whether magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early AD.

ADNI has enrolled approximately 1800 individuals to date across four phases: ADNI-1 (2004-2010), ADNI-GO (2009-2011), ADNI-2 (2011-2016), and ADNI-3 (2016-present). Participants were recruited from approximately 60 sites across the United States and Canada to meet a target number of persons who were cognitively normal, had MCI, or had mild AD. Recruitment was restricted to those with Hachinski Ischemic Score¹¹ ≤ 4 and Geriatric Depression Scale (GDS)¹² score ≤ 6, were in general good health, did not use of certain medications, and were able and willing to meet study requirements.⁵ After an initial screening visit; eligible participants return for a baseline visit and follow-up visits every 6 to 12 months.

For this cross-sectional analysis, we used data from the screening and baseline visits from any ADNI phase. Demographic data, APOE status, blood cholesterol, and triglyceride levels were collected during screening. Functional status and vital signs were collected at baseline. The Boston naming test¹³ and animal naming test ^14,15 were first administered at screening, and the Mini Mental State Examination (MMSE)¹⁶ and the word fluency test ^14,15,17 were first administered at baseline (word fluency was not administered in ADNI-1 and Boston naming was not administered in ADNI-3). PET scans using florbetapir AV-45 were administered at baseline for participants recruited in ADNI-GO, ADNI-2, or ADNI-3. 3 Tesla (3T) MRI exams were conducted for a subset of ADNI-1 participants and for all participants from ADNI-GO, ADNI-2, and ADNI-3 at either screening or baseline. Finally, participants were diagnosed at baseline with normal cognition, MCI, AD, or other dementia etiology by a study physician, based on full review of participants’ medical histories, clinical dementia rating (CDR), GDS, functional activity questionnaire (FAQ) scores, laboratory test results, and neuropsychological tests.¹⁸

2.1.2. ARIC

The ARIC cohort recruited 15,792 participants aged 45-64 at baseline from four U.S. communities in 1987-1989. In Forsyth County, NC, White and Black participants were selected using the 1980 census and a two-stage area probability sampling approach, stratified on race and income.¹⁹ In Jackson, MS, investigators randomly selected eligible Black residents from the Mississippi Highway Patrol database. ¹⁹ Minneapolis, MN investigators used the Hennepin County jury selection database to draw potential participants using a simple random sampling approach.¹⁹ In Washington County, MD, participants were randomly sampled from a 1975 health census and Department of Motor Vehicles database.¹⁹

In these cross-sectional analyses, we consider data from participants of ARIC Visit 5 (V5; 2011-2013), which was attended by 6,538 participants (5,765 participants had died, and 3,489 persons were unable or unwilling to attend V5; Table A.1). We define sociodemographic characteristics using data from ARIC baseline, and define all other variables of interest using data collected at V5 or the most recent annual follow-up call preceding V5. Cognitive status (normal, MCI, or dementia) was ascertained at V5 by first applying an algorithm using scores from cognitive and functional assessments, followed by final diagnosis by ARIC physicians and neuropsychologists (Table A.2).²⁰ All participants with evidence of cognitive impairment at V5, those with a prior ARIC brain scan, and a random sample of cognitively normal participants were invited for 3T MRI scans. Ultimately, 1,978 participants completed brain MRI; sampling weights allow weighting of the 3T MRI sample back to the full V5 sample. Of those with V5 3T MRI scans, 345 participants from Jackson, Washington County, and Forsyth County without renal failure, heavy alcohol use, or dementia completed PET scans.

2.1.3. Eligibility Criteria

We restricted both samples to Black and white persons 65+ years old given the small number of participants of other race in these samples and lack of persons under age 65 in the ARIC sample. We also excluded persons missing demographic data or dementia status, and ADNI participants with screening and baseline visit dates more than 12 months apart.

2.2. Risk Factors

Sociodemographic predictors included age, gender, self-reported race (white vs. Black), education (up to high school/GED vs. >high school/GED) and marital status (married vs. widowed/divorced/separated/never married). We also considered history of hypertension, measured systolic and diastolic blood pressure, blood cholesterol and triglycerides, APOE-E4 allele status (yes/no), and functional status as potential risk factors of interest. Because ADNI and ARIC used slightly different assessment of functional status,²¹ we created two functional status indicators (any vs. no functional limitation): one using 4 items worded identically or close to identically (FAQ4), and one using an additional 5 items worded similarly across studies (FAQ9).

2.3. Cognitive Performance

For cognitive outcomes, we focused on tests administered in a consistent manner to both ARIC and ADNI participants: animal naming (60 seconds), word fluency (letter F; 60 seconds), the MMSE, and the Boston naming test (30 items). We computed z-scores for animal naming and word fluency scores using the means and standard deviations from the pooled ADNI and ARIC data, and dichotomized MMSE scores (<25) and Boston naming (<25) scores based on commonly-used thresholds.¹⁵

2.4. Neuroimaging

We confirmed ADNI and ARIC PET scans used the same ligand (florbetapirAV-45²²) and were conducted similarly.^23-25 Each study published global florbetapir standardized uptake value ratios (SUVR) based on cortical regions of interest. In ADNI, these included parts of the frontal, anterior/posterior cingulate, lateral parietal, lateral temporal, and precuneus regions.²⁶ ARIC similarly used parts of the frontal, anterior/posterior cingulate, parietal, lateral temporal, and precuneus regions, and in addition included the occipital region.²³ These values were then standardized to a reference region (whole cerebellum in ADNI; cerebellum gray in ARIC). Following convention, we considered only dichotomized SUVR.^23,27-30 We used study-specific cutoffs (1.11 in ADNI, and 1.20 in ARIC)^23,26 because the PET protocols are not identical, and importantly, because study-specific cutoffs are used to define PET positivity in the literature. For MRI outcomes, we confirmed both studies used 3T MRI and processed imaging data using Freesurfer software.³¹ We used three relevant composite MRI imaging outcomes (AD signature ROI cortical region volume,³¹ frontal lobe volume, and temporal lobe volume). These outcomes were provided by ARIC and constructed from raw data for ADNI. All volumetric measures were divided by intracranial volume prior to use in analyses.

Details of between-cohort harmonization of measurement are available in Table A.2.

2.5. Statistical analyses

We pooled ADNI and ARIC data into a single dataset and ran logistic and linear models estimating (i) associations of risk factors with cognitive testing and PET/MRI outcomes, (ii) associations of PET/MRI outcomes with cognitive testing outcomes, and (iii) associations of PET outcomes with MRI outcomes. Each model included an indicator for cohort and its interaction with the predictor of interest, allowing for estimation of cohort-specific associations and statistical evaluation of between-cohort differences. Models adjusted for age, gender, education, race, and dementia status (dementia/no dementia), as well as their interactions with cohort to account for potential differences in confounding structure.

To understand how findings changed with increasing homogeneity, we repeated analyses in four subgroups defined by race and cognitive status: (i) all white and Black participants; (ii) all white participants; (iii) white and Black participants with normal cognition or MCI; (iv) white participants with normal cognition or MCI. Because PET scans were not administered for ARIC participants with dementia, analyses involving amyloid-β were limited to subgroups (iii) and (iv).

Next, to focus on associations that would be considered ‘positive findings’ in at least one cohort, we restricted comparisons to associations that had p<0.2, and more stringently to p<0.05, in at least one cohort. In addition to null hypothesis testing, we also computed the absolute difference in point estimates as a percentage relative to the ARIC point estimate, and we created forest plots of cohort-specific findings to examine differences qualitatively. Finally, for associations that differed significantly between cohorts, we compared estimated cohort-specific marginal expectations to better understand what was driving differences. Because our purpose was to illustrate the potential extent of the differences we may expect to find across the literature, we did not adjust for multiple comparisons.

2.5.1. Sensitivity analyses

To examine the sensitivity of findings to changes in sample selection, we re-ran all analyses limiting the sample to individuals with complete data across all predictors, cognitive test outcomes, and either MRI outcomes (“common MRI sample”), or PET imaging outcomes (“common PET sample”). We also re-ran all analyses involving the common MRI sample using ARIC MRI sampling weights designed to weight back to the full sample of ARIC V5 participants. Finally, we considered analyses including cohort-interactions with only the predictor of interest.

ARIC and ADNI participants provided consent at data collection, and procedures were approved by each study site’s institutional review boards. All analyses were conducted in SAS (v.9.4), R (v.4.0.0) and STATA (v.15).

3. RESULTS

Our eligible sample included 1,787 participants from ADNI and 6,445 participants from ARIC (Table 1). Differences in sample characteristics across ADNI and ARIC were statistically significant for all except for the proportions of participants scoring <25 on the MMSE and PET positive participants. Participants from ADNI were more likely to be male, highly educated, and APOE E4 positive. Remarkably, though more ADNI participants have MCI or dementia (60% vs. 27%), ADNI participants performed slightly better on cognitive testing (Table 1). These patterns were largely consistent across subsamples used in primary and sensitivity analyses.

Table 1.

Sample characteristics across all participants with non-missing data on age, gender, education, race, and dementia status

	ADNI		ARIC
Variable	N	Mean (SD) / N (%)	N	Mean (SD) / N (%)
Predictors
Age in years	1787	75.1 (6.0)	6445	75.8 (5.3)
Male	1787	983 (55%)	6445	2649 (41.1%)
Education: greater than HS/GED	1787	1516 (84.8%)	6445	3327 (51.6%)
Black	1787	72 (4.0%)	6445	1527 (23.7%)
Married	1780	1347 (75.7%)	6445	3954 (61.3%)
Cognitive status
Normal	1787	712 (39.8%)	6445	4734 (73.5%)
MCI	1787	762 (42.6%)	6445	1371 (21.3%)
Dementia	1787	313 (17.5%)	6445	340 (5.3%)
FAQ score ≥1 (out of 9 items)	1776	871 (49.0%)	2476	1700 (68.7%)
FAQ score ≥1 (out of 4 items)	1780	551 (31.0%)	2578	514 (19.9%)
Serum Cholesterol (mg/dL)	1331	192.8 (39.9)	6343	181.5 (42.0)
Triglycerides (mg/dL)	1321	139.0 (74.6)	6329	124.9 (60.1)
Hypertension history (ARIC up to visit 5)	1787	861 (48.2%)	6445	4937 (76.6%)
Systolic blood pressure (mm Hg)	1764	133.6 (16.8)	6413	130.6 (18.6)
Diastolic blood pressure (mm Hg)	1764	73.8 (9.2)	6413	66.3 (10.8)
At least one APOE4 allele	1716	772 (45%)	6236	1813 (29.1%)
Cognitive test outcomes
MMSE score < 25	1785	265 (14.8%)	6435	957 (14.9%)
Boston Naming score < 25	1451	369 (25.4%)	6196	2089 (33.7%)
Animal Naming score	1786	17.7 (6.1)	6343	16.0 (5.1)
Word Fluency score	1055	13.7 (4.8)	6251	11.3 (4.5)
MRI outcomes
AD signature region volume, standardized by intracranial volume	1025	0.04 (0.0)	1954	0.04 (0.0)
Frontal lobe volume, standardized by intracranial volume	1028	0.11 (0.01)	1954	0.11 (0.01)
Temporal lobe volume, standardized by intracranial volume	1026	0.07 (0.01)	1954	0.07 (0.01)
PET imaging outcomes
PET positive (SUVR > 1.11 in ADNI, and SUVR > 1.2 in ARIC) a	832	445 (53.5%)	344	176 (51.2%)

SD: Standard deviation; FAQ: Functional activities questionnaire; MMSE: Mini mental state examination; SUVR: standardized uptake value ratio

^aADNI cortical regions of interest include frontal, anterior/posterior cingulate, lateral parietal and lateral temporal regions, standardized by whole cerebellum reference region; ARIC cortical regions of interest include frontal, anterior/posterior cingulate, lateral parietal, lateral temporal, occipital and precuneus regions, standardized by cerebellum gray.

Not all participants had complete data on our predictors and outcomes of interest; thus analytical sample sizes ranged 493-1,786 in ADNI and 166-6,435 in ARIC, depending on the specific sub-sample and predictor-outcome combination examined (Table 2). In analyses considering all available observations for white and Black participants, 34% of all considered associations were significantly different between ARIC and ADNI (Table 2, Figures A.1a and A.1b). This proportion increased to 42% after restricting to associations with p-value<0.2 in at least one cohort, and to 50% after restricting to associations with p-value<0.05 in at least one cohort (Table 2, Figure 1). These proportions were lower when analyses were restricted to more homogenous subsamples. However, even after restricting to white participants with normal cognition or MCI – our most stringent restriction – the overall proportion of associations that were statistically different across cohorts was 30% overall (Table 2, Figures A.1a and A.1b); and 44% after restricting to associations with p-value<0.05 in at least one cohort (Table 2, Figure 1). Differences between cohorts were not clustered in particular risk factor-outcome association types (Figures A.2a and A.2b).

Table 2.

Proportion of associations that differ between ADNI and ARIC ^a in analyses using all available observations for each association

	All available associations		Associations with p<0.2 in at least one cohort		Associations with p<0.05 in at least one cohort
Sample (Ns)	Number of associations evaluated	Number (%) different associations	Number of associations evaluated	Number (%) different associations	Number of associations evaluated	Number (%) different associations
Proportion of associations that differ statistically between ADNI and ARIC (p<0.05)
White & Black, all cognition (ADNI N=630 to 1,786, ARIC N=1,517 to 6,435) b	103	35 (34.0%)	84	35 (41.7%)	64	32 (50.0%)
White, all cognition (ADNI N = 600 to 1,714, ARIC N=1,179 to 4,910) b, c	96	26 (27.1%)	73	26 (35.6%)	56	23 (41.1%)
White & Black, normal/MCI (ADNI N=515 to 1,474, ARIC N=255 to 6,069)	123	39 (31.7%)	104	39 (37.5%)	82	38 (46.3%)
White, normal/MCI (ADNI N=493 to 1,414, ARIC N=166 to 4,711) c	115	34 (29.6%)	90	34 (37.8%)	73	32 (43.8%)
Proportion of associations that differ qualitatively between ADNI and ARIC (>50% difference in point estimates)
White & Black, all cognition (ADNI N=630 to 1,786, ARIC N=1,517 to 6,435) b	103	51 (49.5%)	84	42 (50.0%)	64	31 (48.4%)
White, all cognition (ADNI N = 600 to 1,714, ARIC N=1,179 to 4,910) b, c	96	50 (52.1%)	73	41 (56.2%)	56	28 (50.0%)
White & Black, normal/MCI (ADNI N=515 to 1,474, ARIC N=255 to 6,069)	123	68 (55.3%)	104	57 (54.8%)	82	46 (56.1%)
White, normal/MCI (ADNI N=493 to 1,414, ARIC N=166 to 4,711) c	115	61 (53%)	90	52 (57.8%)	73	39 (53.4%)

^aModels adjusted for age, gender, education, race (where applicable), and dementia status (where applicable), as well as cohort interactions with each covariate

^bExcludes associations involving PET outcomes

^cExcludes associations involving race as a predictor

Figure 1.

Heatmap of P-values for statistical significance in differences across ADNI and ARIC estimated associations between risk factors and cognitive test/imaging outcomes, and between imaging outcomes and cognitive test outcomes; limited to associations with p<0.05 statistical significance independently in at least one study

NA: Association not applicable to sample

I: Association not significant to p<0.05 in either cohort

In primary analyses across subsamples defined by race and cognitive status, the proportion of associations with point estimates differing by >50% between ADNI and ARIC ranged from 50%-55% of all associations evaluated, with no evidence of reduced differences in subsamples with more homogenous composition (Table 2, Figures A.3a and A.3b). These proportions were similar when restricting to associations with p-value<0.2 in at least one cohort (range 50%-58%) and with p-value<0.05 in at least one cohort (range 48% - 56%; Table 2, Figure 2). Again, differences in point estimates between cohorts were not clustered within any particular risk factor-outcome association types (Figures A.4a and A4b).

Figure 2.

Heatmap of percentage differences in point estimates of associations^a between risk factors and cognitive test/imaging outcomes, and between imaging outcomes and cognitive test outcomes; limited to associations with p<0.05 statistical significance independently in at least one study

^a Percentage difference relative to ARIC estimates, computed as ∣ARIC estimate – ADNI estimate∣/∣ARIC estimate∣

NA: Association not applicable to sample

I: Association not significant to p<0.05 in either cohort

Many of the differences in estimated associations were substantively meaningful (Figures 3-4, Figure A.5). For example, the association between having any functional limitation (FAQ9≥1) and standardized animal naming score is −0.49 in ADNI, compared to −0.17 in ARIC (Figure 3), with differences driven by higher scores among those without a limitation in ADNI than in ARIC (Figure A.5a). Conversely, the association between a 10-year older difference in age and standardized AD signature ROI cortical region volume is −0.72 in ARIC, but just −0.42 in ADNI (Figure 3), with differences driven by larger volumes among younger individuals in ADNI vs in ARIC (Figure A.5e). Notably, having an APOE-4 allele was associated with 8.6 times the odds of being PET positive in ADNI, compared to 2.79 times the odds in ARIC (Figure 4), driven by higher probability of PET positivity among APOE-4 carriers in ADNI than in ARIC (0.78 vs. 0.58) (Figure A.5g). In some cases, estimated associations were significant in only one cohort. For example, APOE4 status was significantly associated with all four cognitive test outcomes in ADNI, but not in ARIC (Figures 3-4). Finally, in rare instances, the direction of estimated associations was inconsistent between ADNI and ARIC (Figures 3-4; Figures A.5b, A.5d, A.5f).

Figure 3.

Comparison of estimated associations in ADNI vs. in ARIC using all available observations ^a that are significant to p<0.05 in at least one cohort; continuous outcomes

^a for associations involving SUVR, this excludes observations of participants with dementia

Figure 4.

Comparison of estimated associations in ADNI vs. in ARIC using all available observations ^a that are significant to p<0.05 in at least one cohort; binary outcomes

^a for associations involving SUVR, this excludes observations of participants with dementia

^b MMSE (score < 25)

^c boston naming (score < 25)

^d SUVR (cohort cut-off)

Sensitivity analyses were consistent with our overall findings. Patterns in statistical differences were similar in analyses using the weighted and unweighted common MRI samples (Table A.3.a. and A.3.b.), and analyses excluding cohort interactions with covariates (Tables A.4-A.5). In the common PET sample, proportions of significantly different associations were lower and closer to what would be expected by chance, though this is likely due to loss of power from smaller sample sizes (ADNI n=485-507, ARIC n=162-248, (Table A.3.c.). The proportion of associations with point estimate differences exceeding 50% were marginally higher in analyses using the common MRI sample (Tables A.3.a and A.3.b.), and similar in analyses using the common PET sample (Table A.3.c.), as well as in analyses excluding cohort interactions with covariates (Tables A.4-A.5).

4. DISCUSSION

In this study assessing the generalizability of associations across ADNI and ARIC, we found that the proportion of associations that differ significantly and the proportion of associations that differ qualitatively across the two samples are substantial, much higher than would be expected by chance. Furthermore, many of these differences were substantively meaningful. Statistically significant differences were observed less frequently in samples of more homogenous participants defined by race and/or cognitive status, though differences remained higher than would be expected by chance alone. There were no apparent patterns in risk factors or outcomes for which associations were more or less likely to differ between cohorts, thus we are unable to make recommendations on which types of associations may be assumed to be generalizable with confidence.

Our study is the first, to our knowledge, to directly examine the extent to which findings observed in a clinical sample are comparable to those from a community-based cohort to better understand whether clinical sample-based findings are generalizable to other populations. The necessity for this work has long been recognized given ADNI’s ongoing contributions to research on relationships between cognitive measures, neuropsychiatric symptoms, biomarkers, risk factors, and AD pathology and progression,^4,6 and facilitation of efforts around early AD detection, improving diagnostic classification, monitoring of disease progression, and improving clinical trial efficiency. ^3,4,32 However, lack of generalizability of ADNI findings to other populations may critically limit the applicability of findings from ADNI. For example, given ADNI’s generally healthy and highly-educated population, certain findings from ADNI may be less relevant to more typical older adults, given evidence on the association between comorbidity burden and faster cognitive decline,^33-35 and potential modification of the association between biomarkers and cognitive decline by education.^36-39 Thus, while ADNI and similar studies remain an invaluable resource for AD research, findings should not be assumed to be directly applicable to other target populations. Efforts to confirm findings from ADNI and other clinic-based samples across different, less-selected population-based studies are warranted.^1,6,40

This study contributes to the long-standing debate around the necessity and value of recruiting representative samples for research. Many scholars would agree that the value of representative sampling depends on the research goals.^40-43 Specifically, the prevailing consensus is that while representative sampling is necessary for conducting descriptive studies^42,43 or for informing healthcare planning,⁴⁰ it should not generally be required for studies estimating associations between variables or estimating causal effects.^42-44 Our findings suggest that inferences based on highly-selected, clinical samples may not be relevant to the broader population, even if unbiased.

As we were careful to include only variables measured in comparable ways across studies, differences in estimated associations are likely due to imbalanced distribution of characteristics that modify associations of interest across cohorts. In this case, if we can identify and measure all characteristics that modify the association of interest and differ between ADNI and the target population, we may be able to apply transport estimators to produce unbiased estimates of associations in the target population.^45-50 However, the assumptions required for transport are strong and often untestable, and existence of unknown or unmeasured modifiers, as well as positivity violations in the distribution of modifiers would preclude use of transport algorithms for inference. We plan to examine the utility of transport algorithms in future work.

Our study has several strengths. We were deliberate in our selection of risk factors, cognitive test outcomes, and imaging outcomes to identify those measured similarly, or which we felt could be harmonized across cohorts (Table A.2). Additionally, we conducted various sensitivity analyses to ensure that our findings are robust.

Our work also has limitations. First, the sub-sample of ARIC participants that underwent PET imaging was small (N=345) and excluded those with dementia; we were unable to assess these associations across the full spectrum of cognitive status and had low power to detect interactions. Second, we assessed the comparability of ADNI findings to only one other cohort, which itself is not generalizable to the national population: though ARIC participants were initially recruited using random sampling, they were recruited from just four US communities from the late 1980s. However, the baseline characteristics of the overall V5 sample and the V5 MRI and PET subsamples are reasonably similar to those of the full ARIC cohort, suggesting they remain relatively representative of the original communities from which they were sampled (Table A. 1). ARIC has the rare advantage among community-based studies of having outcomes data comparable to ADNI. Without considering other comparable datasets, we are unable to make broad conclusions about generalizability of specific associations; researchers should test the comparability of associations of interest in other cohorts where comparable data are available. Third, the ADNI sample is predominantly white (96%), and most black ARIC participants were recruited from Jackson or Forsyth County. This conflates the effects of race and place, and raises questions about whether associations with race have comparable interpretation across samples. Furthermore, due to the small sample of Black participants in ADNI, we were unable to examine whether and how between-cohort differences may differ by race. Fourth, we cannot rule out differences due to imperfect harmonization, particularly in the Boston naming test, history of hypertension, FAQ9, and PET SUVR (Table A.2). Notably, there is no universal consensus about how summary/global cortical PET SUVR measures should be constructed; the differences between ADNI and ARIC reflect the variation in PET SUVR constructs that we may see elsewhere. As such, evaluation of between-cohort differences using the PET variables as originally constructed and made available for use in research by ADNI and ARIC is useful and valuable. Finally, because our goal here was only to assess the comparability of various cross-sectional associations, we do not recommend making any substantive inferences based on the sample of estimated associations presented here.

In summary, we found that a substantial proportion of estimated associations between predictors, cognitive performance, and neuroimaging outcomes differed between the ADNI and ARIC cohorts. Thus, findings from ADNI and similar, highly-selected clinical cohorts should not be assumed to be directly generalizable to other populations and confirmation of findings in clinical samples such as ADNI in more representative samples is necessary. Future efforts should evaluate the use of transportability algorithms to infer associations from ADNI or other clinical samples to other target populations.