Generalizability of cognitive results from clinical trial participants to an older adult population: Addressing external validity

Abstract

Introduction

Study inclusion criteria and recruitment practices limit the generalizability of randomized‐controlled trial (RCT) results. Statistical modeling could enhance generalizability of outcomes. To illustrate this, the cognition–depression relationship was assessed with and without adjustment relative to the target population of older women.

Methods

Randomized participants from four RCTs and non‐randomized participants from two cohorts were included in this study. Prediction models estimated probability of being randomized into trials from target populations. These probabilities were used for inverse odds weighting relative to target populations. Weighted linear regression was used to assess the depression–cognition relationship.

Results

There was no depression–cognition relationship in the combined randomized sample. After applying weights relative to a representative cohort, negative relationships were observed. After applying weights relative to a non‐representative cohort, bias of estimates increased.

Discussion

Quantitative approaches to transportability using representative samples may explain the absence of a‐priori established relationships in RCTs.

1. INTRODUCTION

Issues of limited generalizability arise when outcomes from a recruited sample do not represent findings observed or expected in a target population. ¹ , ² , ³ Limited generalizability issues can be potentially mitigated either through broad recruitment criteria or through statistical adjustment of data. Recruitment of diverse and socioeconomically representative populations for randomized‐controlled trials (RCTs) is often acknowledged ⁴ , ⁵ , ⁶ and strategies for such recommended. ⁷ , ⁸ , ⁹ These latter recommendations should facilitate greater generalizability of RCT data if recommended recruitment objectives are achieved.

Statistical approaches addressing selection bias of RCTs may extend inferences from a clinical trial cohort to a target population and potentially enhance generalizability of trial data. ² , ¹⁰ , ¹¹ The population participation model assumes that participants are randomized into a RCT with a known probability of selection, ² , ¹⁰ which under the superpopulation framework may be drawn from a logistic distribution. ¹⁰ , ¹² An inverse odds of sampling weights approach potentially operationalizes transportability of findings from a RCT cohort to a target population. ¹⁰ , ¹¹ , ¹²

Small sample sizes are a limiting factor of generalizability. ¹³ Recruiting a heterogenous population reflective of a target population is challenging regardless of the sample size of a RCT, and smaller clinical trials are at a greater disadvantage in this respect. It is common practice to harmonize studies with similar eligibility criteria, outcomes of interest and target population to increase heterogeneity of the pooled sample with the goal of expanding the generalizability of the clinical trial data. ¹⁴ While combining and harmonizing RCTs may broaden generalizability of data, selected cohorts for clinical trials may not be representative of target populations, and observational findings from populations may not reflect true associations in the target population.

RESEARCH IN CONTEXT

1. Systematic Review: The authors reviewed the literature using PubMed. Limited generalizability of randomized clinical trials is often acknowledged, but statistical approaches addressing generalizability are not widely implemented.

2. Interpretation: The present study describes one approach to minimize external bias and to expand generalizability of findings from a randomized cohort of postmenopausal women to non‐randomized target populations of older women.

3. Future Directions: This manuscript demonstrates a parametric framework for mitigating external bias and for expanding generalizability of randomized‐controlled trial (RCT) findings. Future studies could potentially use non‐parametric methods for estimation of probability of randomization and consider the issue of external bias and generalizability of RCT data under a causal estimation framework.

The objective of the present study is to demonstrate the utility of inverse odds of sampling weights in addressing issues of external validity. As a proof of concept, observational findings among postmenopausal women from four randomized studies conducted at the Atherosclerosis Research Unit at the University of Southern California (ARU) were transported to a target population from two non‐randomized observational studies. The a‐priori established relationship between depression and cognition, ascertained and validated in different community and clinical cohorts, ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ was assessed in the USC ARU cohort with and without weighting relative to the Health and Retirement Study (HRS) as a target population. HRS is a survey of a representative sample of US older adults born between 1931 and 1959 with oversampling of certain demographic groups. ²¹ , ²² As a secondary objective, the depression–cognition relationship in the ARU population was assessed weighted relative to Alzheimer's Disease Neuroimaging Initiative (ADNI) as a target population. ADNI is a clinical cohort collecting high quality biomarker data and designed to recruit participants according to eligibility criteria of Alzheimer's disease clinical trials. ²³ While ADNI is not a representative cohort, ²⁴ it is usually referenced for its contributions to Alzheimer's disease research. We aim to demonstrate that using a non‐representative sample as a target population will result in biased estimates of association.

2. METHODS

2.1. Study participants

The current study used four RCTs with a secondary trial outcome of cognitive performance conducted at the ARU ²⁵ , ²⁶ , ²⁷ , ²⁸ : the B‐Vitamin Atherosclerosis Intervention Trial (BVAIT, NCT00114400), Women's Isoflavone Soy Health (WISH, NCT00118846) trial, Early versus Late Intervention Trial with Estradiol (ELITE, NCT00114517), and the Nattokinase Atherothrombotic Prevention Study (NAPS, NCT02080520). Postmenopausal women were primarily recruited to the ARU RCTs. Similar eligibility criteria, biomarkers, and data collection procedures and instruments allowed for pooling and harmonizing data from the randomized cohorts of these four RCTs for post hoc exploratory analyses unrelated to the tested interventions. The nonrandomized target populations studied were from HRS and from ADNI.

2.2. BVAIT participants

BVAIT was a randomized, double‐blind, placebo‐controlled trial testing whether reduction of plasma total homocysteine (tHcy) levels with B vitamin supplementation reduces subclinical atherosclerosis progression assessed as carotid artery intima‐media thickness (CIMT). ²⁵ Participants included in the current analysis were postmenopausal women ≥40 years old with fasting tHcy ≥8.5 μmol/L and no clinical signs or symptoms of CVD. Exclusion criteria were fasting triglycerides > 5.64 mmol/L (500 mg/dL), diabetes mellitus or fasting serum glucose > 6.99 mmol/L (126 mg/dL), systolic blood pressure ≥160 mm Hg and/or diastolic blood pressure ≥100 mm Hg, untreated thyroid disease, creatinine clearance < 70 ml/min, life‐threatening illness with prognosis < 5 years, or > 5 alcoholic drinks daily. Men were included in the original trial but excluded from the current study.

2.3. WISH participants

WISH was a randomized, double‐blind, placebo‐controlled trial testing whether isoflavone soy protein reduces subclinical atherosclerosis assessed as carotid artery intima‐media thickness progression. ²⁶ Participants were postmenopausal women without vaginal bleeding > 1 year and serum estradiol < 20 pg/ml. Exclusion criteria were clinical signs, symptoms, or a personal history of CVD, diabetes mellitus or fasting serum glucose > 6.99 mmol/L (126 mg/dL), fasting triglycerides > 5.64 mmol/L (500 mg/dL), systolic blood pressure ≥160 mm Hg and/or diastolic blood pressure ≥110 mm Hg, untreated thyroid disease, serum creatinine > 2 mg/dL, life‐threatening illness with prognosis < 5 years, alcohol intake > 5 drinks/day or substance abuse, taking menopausal hormone therapy, or soy, nut, or related food allergies.

2.4. ELITE participants

ELITE was a randomized, double‐blind, placebo‐controlled trial that evaluated whether postmenopausal hormone therapy would reduce the progression of subclinical atherosclerosis when therapy was initiated soon after menopause (< 6 years) but not when therapy was initiated a long time after menopause (≥10 years). ²⁷ Exclusion criteria were indeterminate time‐since‐menopause, fasting plasma triglyceride level > 500 mg/dl (5.65 mmol/L), diabetes mellitus or fasting serum glucose > 140 mg/dl, serum creatinine > 2.0 mg/dl (177 mmol/L), diastolic blood pressure > 110 mmHg or systolic blood pressure > 160 mmHg, untreated thyroid disease, liver disease, life‐threatening disease with prognosis < 5 years, history of deep vein thrombosis or pulmonary embolism, history of breast cancer, current postmenopausal hormone therapy within 1 month of screening.

2.5. NAPS participants

NAPS was a randomized, double‐blinded, placebo‐controlled clinical trial that tested whether daily nattokinase supplementation would reduce progression of subclinical atherosclerosis. ²⁸ Eligible participants for the original trial included healthy men and women without clinical evidence of CVD. Specific inclusion criteria were age 55 years and older men or postmenopausal women (no uterine bleeding for > 6 months). Exclusion criteria included: clinical signs, symptoms or personal history of CVD, diabetes mellitus or fasting serum glucose > 140 mg/dL, plasma triglycerides > 500 mg/dL, uncontrolled hypertension (systolic blood pressure (BP) > 160 mmHg or diastolic BP > 110 mmHg), uncontrolled tachycardia or irregular heart rate (i.e., atrial fibrillation), untreated thyroid disease, renal insufficiency (serum creatinine > 2.0 mg/dL), life threatening illness with prognosis < 5 years, current use of lipid‐lowering medication, and current use of food supplements containing soy, soy protein, isoflavone or other phytoestrogens, sensitivity or allergy to soy or nuts, regular use of aspirin or other antiplatelet medication, use of anticoagulants, or bleeding diatheses or tendencies. Women from NAPS were selected for the prediction modelling to increase overall sample size and improve model accuracy.

2.6. HRS participants

HRS is a longitudinal survey of community‐dwelling persons over age 50 in which information on income and wealth, health, cognition, and use of healthcare services, work and retirement, and family connections are collected. ²¹ The selected population was based on a multi‐stage area probability design involving location based stratification and clustering with oversampling of Hispanic and Black households. ²¹ , ²² Women who completed the 2016 HRS interview, were previously enrolled in HRS, and did not live in a nursing home were enrolled in 2016 HRS biomarker study and were thus selected for the current study. Consistent with the ARU trial criteria, participants were excluded from the target population if they had diabetes mellitus, history of cardiovascular disease, or biomarker levels above the exclusion threshold of ARU trials. Participant demographics and background information, physical conditions such as blood pressure, diabetes, CVD, arthritis, and dyslipidemia, as well as fasting blood biomarkers of total and high‐density lipoprotein (HDL) cholesterol and glycosylated hemoglobin (HbA1c) were used to construct the representative target population.

2.7. ADNI participants

ADNI is a consortium of universities and medical centers in the United States and Canada established to develop standardized imaging techniques and biomarker procedures in cognitively normal individuals, individuals with mild cognitive impairment, and individuals with mild Alzheimer's dementia. ²³ Enrolled individuals were between 55 and 90 years of age (inclusive) and were required to have a study partner to provide an independent evaluation of functioning. All individuals were required to have a Hachinski Ischemic Score of less than or equal to 4; permitted medications stable for 4 weeks prior to screening; a Geriatric Depression Scale score of less than 6; a study partner with 10+ hours per week of contact either in person or on the telephone and who could accompany the participant to the clinical visits; visual and auditory acuity adequate for neuropsychological testing; good general health with no diseases precluding enrollment; six grades of education or work history equivalent; and ability to speak English or Spanish fluently. Women had to be sterile or 2 years past childbearing potential. Cognitively unimpaired women were selected for the current study, to match the women selected from the four ARU trials. Additionally, participants who took medications for diabetes mellitus, and had biomarker levels above ARU trials’ exclusion threshold were excluded from the present study. The protocols for all studies were approved by the Institutional Review Boards of the respective institutions, and all participants provided written informed consent.

2.8. Consent statement

The protocols for all studies were approved by the Institutional Review Boards of the respective institutions, and all participants provided written informed consent.

2.9. Depression assessment

All participants from the four ARU RCTs were surveyed for depressive symptoms using the Center for Epidemiological Studies‐Depression (CES‐D) measure. ²⁹ CES‐D is a 20‐item measure that queries persons to rate how often over the previous week they experienced symptoms associated with depression, such as restless sleep, poor appetite, and loneliness. Scores range from 0 to 60, with 16 serving as a cut‐point to identify individuals at risk for clinical depression. Baseline CES‐D measurement was used for the current study.

2.10. Cognitive assessments

All randomized participants from the four ARU RCTs underwent neuropsychological assessment by a single trained psychometrist using the same cognitive battery. This battery included 14 tests commonly administered to detect changes in executive function and verbal and visual memory. ³⁰ Test scores were normalized to have a mean 0 and standard deviation of 1. Domain‐specific task scores were summed, and inverse weighted by the inter‐test correlation matrix. Baseline cognitive assessment was used for the current study.

2.11. Apolipoprotein E (APOE) genotyping

APOE genotyping among ARU RCT participants was conducted using TaqMan Assay‐on‐Demand Genotyping Service (Applied Biosystems). ³¹ Genotyping in ADNI was performed using Illumina HumanOmniExpress BeadChip methodology. ³² Genotyping in HRS was performed using Illumina's Human Omni2.5‐Quad (Omni2.5) BeadChip methodology. ³³ Participants with at least one e4 isoform were treated as APOE‐e4 (APOE4) carriers.

2.12. Statistical methods

Cohorts were analyzed to identify common demographic, health, and biomarker characteristics collected in all studies. Variable distributions and frequencies among randomized participants and non‐randomized participants in HRS and ADNI were compared using effect size estimates (Cohen's D for continuous covariates and Cramer's V for categorical variables).

Inverse odds of sampling weights were used to transport findings from the randomized sample to the target populations. Weighted logistic prediction models were created to estimate the probability of being randomized into an ARU trial, assuming all participants came from the target population represented by HRS. HRS participants’ survey weights were used for the weighted logistic regression. ARU participants had assigned weight of 1. No weighting was applied for the ARU–ADNI prediction model. Shared demographic characteristics, health and biomarker measurements were treated as potential predictors for the preliminary model. Contingency tables and univariate logistic regressions with cohort (randomized vs. non‐randomized) as an outcome were used to determine whether a variable should be included in the primary prediction model (p‐value cutoff = 0.15). A primary main effects model was selected using best subsets regression; the model with the lowest Akaike information criterion was selected as the primary model. Linearity of continuous predictors was assessed using fractional polynomials. Non‐linear order terms were added to analyses for variables failing linearity assumptions. Interactions between predictors were considered, and interactions with p‐value ≤ 0.10 were added to the model. Model classification statistics used an empirical cut‐point following Liu's method. ³⁴ Area under the receiver operating characteristics (ROC) curve, as well as sensitivity and specificity, and accuracy at the cut‐point were reported for prediction models. Model‐predicted odds of being enrolled into a randomized trial were estimated and inverted to construct participant‐level weights for subsequent regression analyses using the ARU population.

The association between depression symptoms and global cognition in the ARU randomized cohort was assessed using unweighted linear regression without consideration of participation probabilities. The same association was assessed using weighted least squares regression to estimate the association adjusted for representativeness to the target population; regression weights were the inverse odds of trial participation selection from the logistic HRS/ARU and ADNI/ARU modeling. Age, education, race, and APOE4 carrier status were included as covariates. Regression assumptions were checked through statistical testing and visual inspection. Observations were assumed to be independent from each other a‐priori. Homoskedasticity and linearity assumptions were tested using a visual inspection of the regression residuals versus fitted values scatterplot. Normality assumption was tested using a combination of visual and quantitative inspection of the distribution of residuals.

3. RESULTS

3.1. Participant demographics

A total of 1329 randomized postmenopausal female participants (n = 164 enrolled in NAPS, n = 176 in BVAIT, n = 639 in ELITE, and n = 350 in WISH), and 2477 non‐randomized participants (n = 630 in ADNI, n = 1847 in HRS) were selected for the prediction modelling (Figure 1). Nine hundred and twenty participants with complete genotyping, demographic, and cognitive screening results were selected for subsequent weighted analyses. There were differences in all common demographic variables between the three cohorts (Table 1). ARU participants had higher BMI compared to participants in ADNI, and lower BMI compared to those surveyed for HRS. Similarly, participants in ARU trials had lower systolic blood pressure (SBP) and diastolic blood pressure (DBP) compared to participants in HRS and ADNI. Randomized participants were younger than non‐randomized participants, were more educated, and had higher total cholesterol. Additionally, randomized participants had lower proportions of current smokers, were more racially diverse compared to ADNI participants and less diverse compared to HRS participants (due to oversampling of Black and Hispanic households in HRS) and had higher proportion of participants who reported their marital status as single. Comparison of HRS participants as a cohort and as a survey with weighting based on their inclusion in a HRS biomarker substudy revealed differences in age, race, education, marital status, and total cholesterol; these differences, however, were not clinically meaningful (Table S1).

FIGURE 1.

Flow diagram of participant inclusion in the present study sample. Cognitively normal women from ADNI were included in the non‐randomized clinical cohort.

TABLE 1.

Participant demographics.

	ADNI	ARU	HRS	Effect size	Effect size
	(N = 630)	(N = 920)	(N = 27938256)	(ARU‐HRS)	(ARU‐ADNI)
BMI, kg/m2	26.3 (5.29)	27.3 (5.55)	28.34 (6.42)	0.16	0.19
SBP, mmHg	130 (16.5)	120 (16.3)	129.43 (62.27)	0.13	0.6
DBP, mmHg	72.0 (10.4)	75.8 (10.5)	83.9 (62.58)	0.13	0.36
Age, year	72.7 (7.24)	60.2 (8.31)	64.74 (10.66)	0.34	0.55
Race				0	0.24
Asian and other	15 (2.4%)	86 (9.3%)	1486614 (5.3%)
Black	33 (5.2%)	85 (9.2%)	2754104 (9.9%)
Hispanic	25 (4%)	140 (15.2%)	2405818 (8.6%)
White	557 (88.4%)	609 (66.2%)	21291720 (76.2%)
Education, years	15.3 (2.73)	16.5 (2.16)	13.2 (3.01)	2.29	0.5
Marital Status				0.01	0.31
Divorced	84 (13.3%)	245 (26.6%)	4483936 (16 %)
Married	398 (63.2%)	505 (54.9%)	14307450 (51.2%)
Widowed	120 (19.05%)	78 (8.5%)	3006904 (10.8%)
Separated	0 (0%)	14 (1.5%)	6091726 (21.8%)
Single	28 (4.4%)	78 (8.5%)	48240 (0.2%)
Smoking Status				0	0.26
Current	96 (15.2%)	44 (4.8%)	3815072 (13.7%)
Never	413 (65.6%)	508 (55.6%)	14682608 (52.6%)
Previous	121 (19.2%)	362 (39.6%)	9440576 (33.7%)
Total Cholesterol, mg/dL	209.6 (37.9)	223.4 (34.6)	204.8 (40.44)	0.36	0.39
HDL, mg/dL		64.7 (16.9)	62.07 (19.6)	0.05

Note: Mean and standard deviations of continuous variables and counts and frequencies of categorical variables are presented. Cohen's d was calculated for continuous variables and Cramer's V was calculated for categorical variables as estimates for effect sizes. HRS was treated as a survey (N = 1847 with weight associated with enrolling in biomarker substudy, representing a total of N = 27938256 participants).

Randomized participants with complete cognitive, APOE genotyping, and CES‐D questionnaires and estimated odds of selection based on prediction models using HRS and ADNI as reference populations were used for the final analyses. Mean (SD) CES‐D score was 9.64 (8.27). In the final sample, participants’ mean global cognitive composite was −0.03 (1.80), mean verbal memory composite was 0.04 (1.35), average executive function composite was −0.02 (1.34), and visual memory composite had a mean of 0.03 (1.10).

3.2. Prediction model

3.2.1. HRS cohort as the target population

Using an empirical cut point of 4.4*10⁻⁵ for the model‐predicted probability of enrolling in an ARU trial for classification (Table S2), the prediction model had sensitivity of 0.80 and specificity of 0.64. Area under the ROC curve was 0.82, and the model correctly classified 63.7% of participants. Inverse odds of selection for randomized ARU participants had a mean of 1.07 and SD = 3.57 (range = 0.0003–62.27).

3.2.2. ADNI cohort as the target population

Using an empirical cut point of 0.58 for the model‐predicted probability of enrolling in an ARU trial for classification (Table S3), the model predicting the probability of randomization into an ARU trial, assuming all participants came from ADNI, had a sensitivity of 0.86 and specificity of 0.77. Area under the ROC curve was 0.90, and the model correctly classified 83.1% of participants. Average inverse odds of selection were 0.24 (SD = 1.14., range = 0.00002–32.03).

3.3. Relationship between depressive symptoms and cognitive performance

3.3.1. Global cognition

After adjusting for age, education, race, and APOE4 carrier status, depressive symptoms (measured by CES‐D) were not associated with global cognition among ARU participants (β = −0.004, 95% confidence interval (CI) = (−0.01,0.003), p = 0.26, Table 2, Figure 1). The association was also not apparent when using ADNI as a target population (β = −0.005, 95% CI = (−0.01,0.004), p = 0.26, Table 2, Figure 2). There was a negative association between depressive symptoms and global cognition when weighting the association using HRS as a target population (β = −0.018, 95% CI = (−0.027, −0.01), p < 0.001, Table 2, Figure 2). The average global cognition score is 0.05 SD lower per SD increase in the CES‐D score.

TABLE 2.

Regression coefficients for global cognition and verbal memory as outcomes.

	ARU	Relative to	Relative to	ARU	Relative to	Relative to
	cohort	ADNI	HRS	cohort	ADNI	HRS
	Global cognition			Verbal memory
Intercept	2.87	3.11	4.1	2.29	2.8	2.68
	(2.00,3.75)	(2.13,4.08)	(3.32,4.88)	(1.60, 2.98)	(2.06, 3.54)	(2.08, 3.28)
Race
Asian	‐1.01	‐0.98	‐1.32	‐0.77	‐0.90	‐0.76
	(−1.37, −0.65)	(−1.67,−0.30)	(−1.93, −0.71)	(−1.06, −0.48)	(−1.42,−0.38)	(−1.23, −0.28)
Black	‐1.17	‐1.89	‐1.47	‐0.50	‐0.81	‐0.39
	(−1.5, −0.83)	(−2.27,−1.52)	(−1.75,−1.20)	(−0.76,−0.23)	(−1.11, −0.52)	(−0.6, −0.17)
Hispanic	‐1.14	‐0.78	‐0.71	‐0.76	‐0.89	‐0.42
	(−1.4, −0.85)	(−1.17,−0.39)	(−1.17,−0.25)	(−0.99, −0.53)	(−2.46, 0.69)	(−0.78, −0.07)
Other	‐1.01	‐0.11	‐1.89	‐1.01	‐0.03	‐1.71
	(−1.9, −0.04)	(−2.17,1.96)	(−2.84,−0.93)	(−1.77,−0.25)	(−0.04,−0.02)	(−2.45, −0.98)
Age, years	‐0.04	‐0.05	‐0.06	‐0.03	‐0.03	‐0.03
	(−0.05, −0.03)	(−0.06,−0.03)	(−0.07,−0.05)	(−0.04, −0.02)	(−0.04, −0.02)	(−0.04, −0.02)
Education
<8th grade	‐1.65	‐1.94	‐2.01	‐1.93	‐1.86	‐1.89
	(−3.91,0.61)	(−10.96, 7.09)	(−8.30,4.28)	(−3.72, −0.16)	(−8.73,5.01)	(−6.74,2.97)
Some high school	‐2.77	‐2.46	‐2.43	‐1.74	‐3.2	‐2.89
	(−3.99, −1.55)	(−2.95,−1.96)	(−3.83, −1.02)	(−0.55, −0.77)	(−3.58, −2.82)	(−3.98, −1.81)
High School	‐1.03	‐0.93	‐1.37	‐0.5	‐0.6	‐1.07
	(−1.5, −0.48)	(−1.27 ,−0.60)	(−2.03,−0.70)	(−0.93,−0.07)	(−0.85, −0.35)	(−1.59, −0.56)
Some college	‐0.31	‐0.16	‐0.76	‐0.34	‐0.34	‐0.62
	(−0.57,−0.05)	(−0.43,0.11)	(−1.02,−0.50)	(−0.55, −0.13)	(−0.55,−0.13)	(−0.82, −0.42)
Trade or business school	‐0.07	0.04	‐1.77	‐0.44	‐0.38	‐2.85
	(−0.65, 0.5)	(−0.53, 0.60)	(−2.29, −1.24)	(−0.89, 0.01)	(−0.81, 0.05)	(−3.25,−2.45)
Graduate or professional	0.38	0.75	‐0.04	0.07	0.08	‐0.23
	(0.13,0.63)	(0.48,1.02)	(−0.32, 0.23)	(−0.13,0.27)	(−0.13,0.28)	(−0.44, −0.01)
APOE4 carrier status	‐0.15	‐0.09	0.5	‐0.20	‐0.6	0.19
	(−0.3, 0.07)	(−0.31,0.13)	(−0.28, 0.73)	(−0.37,−0.03)	(−0.76,−0.43)	(0.01, 0.36)
CES‐D	‐0.003	‐0.005	‐0.02	‐0.004	‐0.02	‐0.01
	(−0.01, 0.003)	(−0.01,0.004)	(−0.03,−0.01)	(−0.009,0.002)	(−0.02. −0.01)	(−0.02,−0.01)

Note: Weighted regression coefficients and coefficients are presented for associations relative to ADNI and HRS cohorts.

FIGURE 2.

Coefficients for depressive symptoms–cognitive domain relationship. The relationships are adjusted for age, race, education, and APOE4 carrier status. Blue lines represent the association in ARU sample (unadjusted), orange lines represent the coefficient after weighting relative to ADNI, and green lines represent the coefficient of the relationship after weighting relative to the HRS cohort. Panel A represents the relationship between CES‐D and global cognition, B between CES‐D and verbal memory, C between CES‐D and executive function, and D between CES‐D score and visual memory.

3.3.2. Verbal memory

There was no apparent association between depressive symptoms and verbal memory composite score (β = −0.003, 95% CI = (−0.009,0.002), p = 0.19, Table 2, Figure 2) among ARU participants after adjusting for covariates. The association was negative (β = −0.017, 95% CI: (−0.023,−0.01), p < 0.001) after generalizing the randomized population to ADNI. Similarly, there was a negative association between depressive symptoms and verbal memory after generalizing the randomized population to HRS (β = −0.012, 95% CI: (−0.019, −0.006), p < 0.001, Table 2, Figure 2). This corresponds to an estimated 0.07 SD decrease in verbal performance per 1 SD increase in depressive symptom assessment score.

3.3.3. Executive function

After adjusting for covariates, depressive symptoms were associated with a decreased executive function composite score (β = −0.006, 95% CI: (−0.01, −0.001), p = 0.02) among ARU participants. The association was not apparent when weighting relative to demographic characteristics of ADNI (β = −0.003, 95% CI = (−0.008, 0.003), p = 0.34, Table 2, Figure 2). There was a negative association between depressive symptoms and executive function (β = −0.009, 95% CI = (−0.015, −0.004), p = 0.001, Table 3) when weighting relative to demographic characteristics of the HRS cohort. One SD increase in CES‐D score may decrease the estimated executive function score by 0.06 SD.

TABLE 3.

Regression coefficients for executive function and visual memory as outcomes.

	ARU	Relative to	Relative to	ARU	Relative to	Relative to
	cohort	ADNI	HRS	cohort	ADNI	HRS
	Executive function			Visual memory
Intercept	3.52	5.34	5.12	1.51	‐0.13	1.26
	(2.92, 4.14)	(4.7, 5.98)	(4.61, 5.63)	(0.93, 2.10)	(−0.79,0.53)	(0.8, 1.73)
Race
Asian	‐0.77	‐0.48	‐1.02	‐0.26	‐0.88	‐0.58
	(−1.02, −0.52)	(−0.93, −0.03)	(−1.42, −0.62)	(−0.50, −0.01)	(−1.33, −0.43)	(−0.95, −0.21)
Black	‐0.85	‐1.53	‐0.81	‐0.13	‐0.79	‐0.42
	(−1.08, −0.61)	(−1.77, −1.28)	(−0.99, −0.62)	(−0.36, 0.09)	(−1.04, −0.54)	(−0.58, −0.25)
Hispanic	‐1.12	‐0.74	‐0.81	0.004	0.05	0.18
	(−1.32, −0.91)	(−1.00, −0.48)	(−1.11, −0.50)	(−0.19, 0.20)	(−0.24, 0.33)	(−0.09, 0.46)
Other	‐0.71	‐0.04	‐0.46	0.25	0.47	‐0.57
	(−1.38, −0.04)	(−1.40, 1.32)	(−1.09, 0.17)	(−0.39, 0.90)	(−0.89, 1.83)	(−1.14, 0)
Age, years	‐0.05	‐0.08	‐0.07	‐0.03	‐0.01	‐0.02
	(−0.06, −0.04)	(−0.09, −0.07)	(−0.08, −0.07)	(−0.03,−0.02)	(−0.02, −0.0003)	(−0.03, −0.02)
Education
<8th grade	‐0.26	‐1.35	‐0.77	‐0.99	0.7	‐0.23
	(−1.83, 1.30)	(−7.28, 4.58)	(−4.9, 3.36)	(−2.50, 0.52)	(−5.23, 6.63)	(−3.98,3.53)
Some high school	‐2.79	‐2.46	‐2.21	‐0.36	1.39	0.72
	(−3.64, −1.94)	(−2.78, −2.13)	(−3.13, −1.29)	(−0.004, 0.76)	(0.84, 1.94)	(−0.53, 1.96)
High School	‐0.61	‐0.67	‐1.02	‐0.44	0.2	‐0.12
	(−0.99, −0.23)	(−0.89, −0.45)	(−1.45, −0.58)	(−0.80, −0.07)	(−0.02, 0.42)	(−0.51, 0.28)
Some college	‐0.27	‐0.4	‐0.73	‐0.05	0.48	0.14
	(−0.46, −0.09)	(−0.58, −0.22)	(−0.91, −0.56)	(−0.22, 0.13)	(0.3, 0.66)	(−0.01, 0.3)
>Trade or business school	‐0.45	‐0.67	‐1.53	0.38	1.23	‐0.61
	(−0.84, −0.05)	(−1.04, −0.3)	(−1.88, −1.19)	(−0.004, 0.76)	(0.86, 1.60)	(−0.93, −0.3)
Graduate or professional	0.29	0.15	‐0.19	0.19	0.94	0.39
	(0.12, 0.47)	(−0.03, 0.32)	(−0.37, −0.01)	(0.02, 0.35)	(0.77, 1.12)	(0.23, 0.55)
APOE4 carrier status	‐0.1	0.05	0.6	0.09	0.18	‐0.04
	(−0.25, 0.06)	(−0.09, 0.19)	(0.45, 0.75)	(−0.06, 0.23)	(0.03, 0.32)	(−0.18, 0.09)
CES‐D	‐0.006	‐0.003	‐0.009	0.002	0.01	‐0.003
	(−0.01, −0.001)	(−0.01, 0.003)	(−0.015, −0.004)	(−0.003, 0.006)	(0.01, 0.02)	(−0.008, 0.002)

Note: Weighted regression coefficients and coefficients are presented for associations relative to ADNI and HRS cohorts.

3.3.4. Visual memory

There were no depressive symptoms–visual memory relationships in ARU cohort (β = 0.002, 95% CI = (−0.003, 0.006), p = 0.4) and when weighting relative to HRS (β = −0.003, 95% CI = (−0.008, 0.002), p = 0.27, Table 3, Figure 2). There was a positive association between depressive symptoms and visual memory when weighted relative to ADNI ((β = 0.014, 95% CI = (0.008, 0.019), p < 0.001, Table 3, Figure 2).

4. DISCUSSION

The present study assessed generalizability of findings from randomized clinical trial participants (ARU) to an observational clinical cohort (ADNI) and to the general aging US population (HRS). We found that ARU participants were not representative of the general population (HRS), as evidenced by differences in covariate distributions and the lack of the established association between depressive symptoms and cognitive performance. However, weighting the non‐representative sample relative to a reference dataset with oversampling of Black and Hispanic households aligned the estimates with findings from larger community‐based samples and addressed the issue of generalizability in randomized studies. Moreover, we showed that the choice of an appropriate cohort for the target population is critical since the reference cohorts themselves may not be representative of a target population. We demonstrated this by separately harmonizing ARU and ADNI cohorts, and ARU and HRS cohorts by selecting common demographic variables between the cohorts. These harmonized datasets were used to estimate the inverse odds of ARU trial inclusion relative to the harmonized demographic characteristics, and estimated the depressive symptoms–cognition associations using the weights relative to ADNI and HRS.

A meta‐analysis combining evidence from 62 studies representing 21,544 participants showed that female sex was related to more severe depressive symptoms in dementia than male sex. ³⁵ Studies have identified late life depression as a modifiable risk factor for dementia, with a population attributable fraction for dementia risk of 3.9%. ¹⁵ Compared to APOE4 non‐carriers, APOE4 carriers with depressive symptoms were more susceptible to tau accumulation in the amygdala and entorhinal cortex among non‐demented participants of the Framingham Heart Study PET imaging substudy, identifying a potential mechanism linking depression, genetic risk factors, and cognitive decline. ³⁶ Additionally, racial differences were observed between depressive symptoms and cognition, identifying race as a confound and indicating the need for adjustment for race in the analyses. ³⁷ Moreover, depressive symptoms (measured with CES‐D) are associated with poor cognitive performance among HRS participants. ³⁸ Thus, lack of association in the ARU cohort may be attributed to the lack of representativeness in this cohort or to over‐representation of Black participants in HRS.

Generalizability theory (G‐theory) is a framework to generalize inferences from a randomized trial to a target population. ³⁹ Inverse odds of selection weighting (IOSW) approach can be used to enhance generalization, after harmonizing baseline covariate data of randomized participants with those of a non‐randomized sample. ¹¹ It has been shown that non‐nested trial participation – where the trial and target populations do not come from the same population (as in the case of ARU trial participants and ADNI and HRS as target populations) – can be modelled using a logistic distribution. ¹⁰ , ¹² This is crucial when externally valid estimates of associations or intervention effects are not guaranteed by trial design or when the external validity of the estimates is not quantified. ¹⁰ A possible reason for the lack of generalizability is selection bias. ¹ Indeed, the trial selection criteria ensured that ARU participants did not have clinical conditions prevalent in the general population (e.g., diabetes, heart disease), were motivated to complete multiple study visits, and were willing to consent to be randomized to receive the study treatment. IOSW approach tends to adjust for these differences, assigning a higher weight to participants that would be more representative of general population and penalizing participants with characteristics different from the those of population.

Overall, inverse probability weighting (IPW) of harmonized randomized and non‐randomized covariates generalizes findings from a specialized cohort to general populations and corrects for selection bias to provide regression estimates closer to true associations. This approach mirrors propensity score matching (PSM) approach for external controls. Both approaches first create a prediction model to estimate probability of being in the RCT. PSM then uses a matching algorithm to restrict the external (target population) to the trial population while IPW approach for generalizability uses the probabilities (propensities) to address external validity of the RCT data. The IPW approach assumes that the cohort from which results are transported is a subsample of the target population. When this assumption is not valid (in case of ARU and HRS/ADNI), inverse odds of selection weights are recommended for statistical adjustment. ¹¹ IOSW are IPWs multiplied by (1‐probability(selection into the transported cohort) and then multiplied by the ratio of the unconditional sampling probability to the unconditional non‐sampling probability.

Confounding is a major concern in all epidemiological studies. In the present study, we accounted for confounding at two stages. The first stage was the prediction of ARU trial participation using both ARU and target population data as a preparation for IOSW. Demographic and blood based cardiovascular biomarkers were candidate predictors due to cohort‐wise differences attributable to ARU trials' inclusion criteria. At the second stage, we used confounder adjustment to reduce bias of the estimates of the association of depressive symptoms on cognition. This step is necessary since IOSW adjusts for cohort‐wise differences and our prediction models do not account for the association of depressive symptoms with cognition. APOE4, age, education, and race are known confounders of this relationship; thus, they were included as covariates a priori. Additional potential confounders (such as income and marital status) were considered but they did not appreciably change the effect of depressive symptoms on cognition and thus were not included in the present study. ARU studies did not collect Alzheimer's disease biomarkers; thus, some unmeasured confounding may be unaccounted for in the present study. In the present study, depression–confounder interactions did not improve our models; thus, we did not include them in our analyses.

We made some assumptions to transport findings from ARU trials to populations of older women in the US represented by HRS and to cognitively normal women enrolled in ADNI. We assumed positivity, which requires non‐zero probability of being sampled into the ARU cohort for every combination of values of the observed confounders. To addess this assumption, we excluded participants in HRS and ADNI with biomarker levels above the thresholds for ARU trials and with known history of diabetes and cardiovascular diseases. We also assumed conditional exchangeability, that assumes random selection into the ARU cohort from reference populations within every combination of values of the observed confounders. Conditional exchangeability is not guaranteed but was assumed to hold based on common characteristics of the cohorts of interest (Table 1) and on a selection mechanism independent of the confounder distributions. Within the ARU cohort, direct confounder adjustment addrresses conditional conditional exchangeability across exposures of participants with different characteristic profiles. Additionally, IOSW addresses conditional exchangeability of ARU and the target population of participants. We also assumed consistency: since the “exposure” in terms of transportability is the selection into ARU cohorts, the mechanisms of hypothetical assignment for a participant are well defined. We also assume no measurement error, and that the parametric models are specified correctly.

Our study demonstrates the utility of covariate harmonization and prediction modelling to extend cross‐sectional associations from a highly selected clinical trial sample to a specialized clinical cohort (ADNI) and to a community‐based cohort (HRS). The presented analytical approach quantifies study generalizability when other approaches (i.e., replication studies) are limited. While efforts to recruit more diverse and representative samples for clinical studies should be encouraged, analytical approaches to enhance generalizability may be used to extend findings from non‐representative clinical samples currently available to researchers. Researchers should aim to use analytical approaches to generalize findings when the lack of generalizability is a significant limitation to their study. Future efforts should evaluate generalizability under a causal framework and infer associations from other randomized trials to different target populations.

CONFLICTS OF INTEREST STATEMENT

The authors declare no conflicts of interest. Author disclosures are available in the supporting information.

AUTHOR INFORMATION

Membership of the Alzheimer's Disease Neuroimaging Initiative is listed in the Acknowledgments.

Consortia

The Alzheimer's Disease Neuroimaging Initiative

Contributions

Conceptualization: V.A.; Implementation and computational methods: V.A. & W.J.M.; Paper writing: all authors; Paper review and supervision: W.J.M. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Dr Wendy J. Mack.

ADNI Collaborators

Principal Investigator Michael W. Weiner, MD University of California, San Francisco ATRI PI and Director of Coordinating Center Clinical Core Paul Aisen, MD University of Southern California Co PI of Clinical Core Ronald Petersen, MD, PhD Mayo Clinic, Rochester Executive Committee Michael W. Weiner, MD University of California, San Francisco Paul Aisen, MD University of Southern California Ronald Petersen, MD, PhD Mayo Clinic, Rochester Clifford R. Jack, Jr., MD Mayo Clinic, Rochester William Jagust, MD University of California, Berkeley John Q. Trojanowki, MD, PhD University of Pennsylvania Arthur W. Toga, PhD University of Southern California Laurel Beckett, PhD University of California, Davis Robert C. Green, MD, MPH Brigham and Women's Hospital/Harvard Medical School Andrew J. Saykin, PsyD Indiana University John C. Morris, MD Washington University St. Louis Richard J. Perrin, MD, PhD Washington University St. Louis Leslie M. Shaw, PhD University of Pennsylvania ADNI External Advisory Board (ESAB) Zaven Khachaturian, PhD Prevent Alzheimer's Disease 2020 (Chair) Maria Carrillo, PhD Alzheimer's Association William Potter, MD National Institute of Mental Health Lisa Barnes, PhD Rush University Marie Bernard, MD NIA Hector González University of California, San Diego Carole Ho Denali Therapeutics John K. Hsiao, MD NIH Jonathan Jackson, PhD Massachusetts General Hospital Eliezer Masliah, MD NIA Donna Masterman, MD Biogen Ozioma Okonkwo, PhD University of Wisconsin, Madison Laurie Ryan, PhD NIA Nina Silverberg, PhD NIA ADNI 3 Private Partner Scientific Board (PPSB) Adam Fleisher, MD Eli Lilly (Chair) Administrative Core—Northern California Institute for Research & Education (NCIRE / The Vererans Health Research Institute) Michael W. Weiner, MD University of California, San Francisco Diana Truran Sacrey, NCIRE / The Vererans Health Research Institute Juliet Fockler, University of California, San Francisco Cat Conti, BA NCIRE / The Vererans Health Research Institute Dallas Veitch, PhD NCIRE / The Vererans Health Research Institute John Neuhaus, PhD University of California, San Francisco Chengshi Jin, PhD University of California, San Francisco Rachel Nosheny, PhD University of California, San Francisco Miriam Ashford, PhD NCIRE / The Vererans Health Research Institute Derek Flenniken, NCIRE / The Vererans Health Research Institute Adrienne Kormos, NCIRE / The Vererans Health Research Institute Data and Publications Committee Robert C. Green, MD, MPH BWH/HMS (Chair) Resource Allocation Review Committee Tom Montine, MD, PhD University of Washington (Chair) Cat Conti, BA NCIRE / The Vererans Health Research Institute Clinical Core Leaders and Key Personnel Ronald Petersen, MD, PhD Mayo Clinic, Rochester (Core PI) Paul Aisen, MD University of Southern California (Core PI) Michael Rafii, MD, PhD University of Southern California Rema Raman, PhD University of Southern California Gustavo Jimenez, MBS University of Southern California Michael Donohue, PhD University of Southern California Devon Gessert, BS University of Southern California Jennifer Salazar, MBS University of Southern California Caileigh Zimmerman, MS University of Southern California Yuliana Cabrera, BS University of Southern California Sarah Walter, MSc University of Southern California Garrett Miller, MS University of Southern California Godfrey Coker, MBA, MPH University of Southern California Taylor Clanton, MPH University of Southern California Lindsey Hergesheimer, BS University of Southern California Stephanie Smith, BS University of Southern California Olusegun Adegoke, MSc University of Southern California Payam Mahboubi, MPH University of Southern California Shelley Moore, BA University of Southern California Jeremy Pizzola, BA University of Southern California Elizabeth Shaffer, BS University of Southern California Brittany Sloan, BA University of Southern California Biostatistics Core Leaders and Key Personnel Laurel Beckett, PhD University of California, Davis (Core PI) Danielle Harvey, PhD University of California, Davis Michael Donohue, PhD University of Southern California MRI Core Leaders and Key Personnel Clifford R. Jack, Jr., MD Mayo Clinic, Rochester (Core PI) Arvin Forghanian‐Arani, PhD Mayo Clinic Bret Borowski, RTR Mayo Clinic Chad Ward, Mayo Clinic Christopher Schwarz, PhD Mayo Clinic David Jones, MD Mayo Clinic Jeff Gunter, PhD Mayo Clinic Kejal Kantarci, MD Mayo Clinic Matthew Senjem, MS Mayo Clinic Prashanthi Vemuri, PhD Mayo Clinic Robert Reid, PhD Mayo Clinic Nick C. Fox, MD University College London Ian Malone, PhD University College London Paul Thompson, PhD University of Southern California School of Medicine Sophia I. Thomopoulos, BS University of Southern California School of Medicine Talia M. Nir, PhD University of Southern California School of Medicine Neda Jahanshad, PhD University of Southern California School of Medicine Charles DeCarli, MD University of California, Davis Alexander Knaack, MS University of California, Davis Evan Fletcher, PhD University of California, Davis Danielle Harvey, PhD University of California, Davis Duygu Tosun‐Turgut, PhD University of California, San Francisco Stephanie Rossi Chen, BA. NCIRE / The Vererans Health Research Institute Mark Choe, BS NCIRE / The Vererans Health Research Institute Karen Crawford, University of Southern California School of Medicine Paul A. Yushkevich, PhD University of Pennsylvania Sandhitsu Das, PhD University of Pennsylvania PET Core Leaders and Key Personnel William Jagust, MD University of California, Berkeley (Core PI) Robert A. Koeppe, PhD University of Michigan Eric M. Reiman, MD Banner Alzheimer's Institute Kewei Chen, PhD Banner Alzheimer's Institute Chet Mathis, MD University of Pittsburgh Susan Landau, PhD University of California, Berkeley Neuropathology Core Leaders and Key Personnel John C. Morris, MD Washington University St. Louis Richard Perrin MD Washington University St. Louis Nigel J. Cairns, PhD, FRCPath Washington University St. Louis–Past Investigator Erin Householder, MS Washington University St. Louis Erin Franklin, MS Washington University St. Louis Haley Bernhardt, BA, R. EEG T Washington University St. Louis Lisa Taylor‐Reinwald, BA, HTL Washington University St. Louis (ASCP) – Past Investigator Biomarkers Core Leaders and Key Personnel Leslie M. Shaw, PhD Perelman School of Medicine, University of Pennsylvania (co‐PI) John Q. Trojanowki, MD, PhD Perelman School of Medicine, University of Pennsylvania (co‐PI) Magdalena Korecka, PhD Perelman School of Medicine, University of Pennsylvania Michal Figurski, PhD Perelman School of Medicine, University of Pennsylvania Informatics Core Leaders and Key Personnel Arthur W. Toga, PhD University of Southern California (Core PI) Karen Crawford University of Southern California Scott Neu, PhD University of Southern California Genetics Core Leaders and Key Personnel Andrew J. Saykin, PsyD Indiana University School of Medicine (Core PI) Kwangsik Nho, PhD Indiana University School of Medicine Shannon L. Risacher, PhD Indiana University School of Medicine Liana G. Apostolova, MD Indiana University School of Medicine Li Shen, PhD UPenn School of Medicine Tatiana M. Foroud, PhD NCRAD/Indiana University School of Medicine Kelly Nudelman, PhD NCRAD/Indiana University School of Medicine Kelley Faber, MS, CCRC NCRAD/Indiana University School of Medicine Kristi Wilmes, MS, CCRP NCRAD/Indiana University School of Medicine Initial Concept Planning & Development Michael W. Weiner, MD University of California, San Francisco Leon Thal, MD – Past Investigator University of California, San Diego Zaven Khachaturian, PhD Prevent Alzheimer's Disease 2020 NIA John K. Hsiao, MD National Institute on Aging Part B: Investigators By Site Oregon Health & Science University: Lisa C. Silbert, MD Betty Lind, BS Rachel Crissey Jeffrey A. Kaye, MD, A – Past Investigator Raina Carter, BA – Past Investigator Sara Dolen, BS – Past Investigator Joseph Quinn, MD – Past Investigator University of Southern California: Lon S. Schneider, MD Sonia Pawluczyk, MD Mauricio Becerra, MD Liberty Teodoro, RN Karen Dagerman, MS Bryan M. Spann, DO, PhD – Past Investigator University of California – San Diego: James Brewer, MD, PhD Helen Vanderswag, RN Adam Fleisher, MD – Past Investigator University of Michigan: Jaimie Ziolkowski, MA, BS, TLLP Judith L. Heidebrink, MD, MS Lisa Zbizek‐Nulph, MS Joanne L. Lord, LPN, BA, CCRC – Past Investigator Lisa Zbizek‐Nulph, MS, CCRP Mayo Clinic, Rochester: Ronald Petersen, MD, PhD Sara S. Mason, RN Colleen S. Albers, RN David Knopman, MD Kris Johnson, RN Baylor College of Medicine: Javier Villanueva‐Meyer, MD Valory Pavlik, PhD Nathaniel Pacini, MA Ashley Lamb, MA Joseph S. Kass, MD, LD, FAAN Rachelle S. Doody, MD, PhD – Past Investigator Victoria Shibley, MS – Past Investigator Munir Chowdhury, MBBS, MS – Past Investigator Susan Rountree, MD – Past Investigator Mimi Dang, MD – Past Investigator Columbia University Medical Center: Yaakov Stern, PhD Lawrence S. Honig, MD, PhD Akiva Mintz, MD, PhD Washington University, St. Louis: Beau Ances, MD, PhD, MSc John C. Morris, MD David Winkfield, BS Maria Carroll, RN, MSN, GCNS‐BC Georgia Stobbs‐Cucchi, RN, CCRP–Past Investigator Angela Oliver, RN, BSN, MSG – Past Investigator Mary L. Creech, RN, MSW – Past Investigator Mark A. Mintun, MD – Past Investigator Stacy Schneider, APRN, BC, GNP – Past Investigator University of Alabama—Birmingham: David Geldmacher, MD Marissa Natelson Love, MD Randall Griffith, PhD, ABPP – Past Investigator David Clark, MD – Past Investigator John Brockington, MD – Past Investigator Daniel Marson, JD, PhD – Past Investigator Mount Sinai School of Medicine: Hillel Grossman, MD Martin A. Goldstein, MD Jonathan Greenberg, BA Effie Mitsis, PhD – Past Investigator Rush University Medical Center: Raj C. Shah, MD Melissa Lamar, PhD Patricia Samuels Wien Center: Ranjan Duara, MD Maria T. Greig‐Custo, MD Rosemarie Rodriguez, PhD Johns Hopkins University: Marilyn Albert, PhD Chiadi Onyike, MD Leonie Farrington, RN Scott Rudow, BS Rottislav Brichko, BS Stephanie Kielb, BS – Past Investigator University of South Florida: USF Health Byrd Alzheimer's Institute: Amanda Smith, MD Balebail Ashok Raj, MD – Past Investigator Kristin Fargher, MD – Past Investigator New York University: Martin Sadowski, MD, PhD Thomas Wisniewski, MD Melanie Shulman, MD Arline Faustin, MD Julia Rao, PhD Karen M. Castro, BA Anaztasia Ulysse, BA Shannon Chen, BA Mohammed O. Sheikh, MD – Past Investigator Jamika Singleton‐Garvin, CCRP – Past Investigator Duke University Medical Center: P. Murali Doraiswamy, MBBS, FRCP Jeffrey R. Petrella, MD Olga James, MD Terence Z. Wong, MD Salvador Borges‐Neto, MD – Past Investigator University of Pennsylvania: Jason H. Karlawish, MD David A. Wolk, MD Sanjeev Vaishnavi, MD Christopher M. Clark, MD – Past Investigator Steven E. Arnold, MD – Past Investigator University of Kentucky: Charles D. Smith, MD Gregory A. Jicha, MD, PhD Riham El Khouli, MD Flavius D. Raslau, MD University of Pittsburgh: Oscar L. Lopez, MD MaryAnn Oakley, MA Donna M. Simpson, CRNP, MPH University of Rochester Medical Center: Anton P. Porsteinsson, MD Kim Martin, RN Nancy Kowalski, MS, RNC Melanie Keltz, RN Bonnie S. Goldstein, MS, NP – Past Investigator Kelly M. Makino, BS – Past Investigator M. Saleem Ismail, MD – Past Investigator Connie Brand, RN – Past Investigator University of California Irvine IMIND: Gaby Thai, MD Aimee Pierce, MD Beatriz Yanez, RN Elizabeth Sosa, PhD Megan Witbracht, PhD University of Texas Southwestern Medical School: Brendan Kelley, MD Trung Nguyen, MD Kyle Womack, MD Dana Mathews, MD, PhD– Past Investigator Mary Quiceno, MD– Past Investigator Emory University: Allan I. Levey, MD, PhD James J. Lah, MD, PhD Ihab Hajjar, MD Janet S. Cellar, DNP, PMHCNS‐BC – Past Investigator University of Kansas, Medical Center: Jeffrey M. Burns, MD Russell H. Swerdlow, MD William M. Brooks, PhD University of California, Los Angeles: Daniel H.S. Silverman, MD, PhD Sarah Kremen, MD Liana Apostolova, MD – Past Investigator Kathleen Tingus, PhD – Past Investigator Po H. Lu, PsyD – Past Investigator George Bartzokis, MD – Past Investigator Ellen Woo, PhD – Past Investigator Edmond Teng, MD, PhD – Past Investigator Mayo Clinic, Jacksonville: Neill R Graff‐Radford, MBBCH, FRCP (London) Francine Parfitt, MSH, CCRC Kim Poki‐Walker, BA Indiana University: Martin R. Farlow, MD Ann Marie Hake, MD – Past Investigator Brandy R. Matthews, MD – Past Investigator Jared R. Brosch, MD Scott Herring, RN, CCRC Yale University School of Medicine: Christopher H. van Dyck, MD Adam P. Mecca, MD, PhD Adam P. Mecca, MD, PhD Susan P. Good, APRN Martha G. MacAvoy, PhD Richard E. Carson, PhD Pradeep Varma, MD McGill Univ., Montreal‐Jewish General Hospital: Howard Chertkow, MD Susan Vaitekunis, MD Chris Hosein, MEd Sunnybrook Health Sciences, Ontario: Sandra Black, MD, FRCPC Bojana Stefanovic, PhD Chris (Chinthaka) Heyn, BSC, PhD, MD, FRCPC U.B.C. Clinic for AD & Related Disorders: Ging‐Yuek Robin Hsiung, MD, MHSc, FRCPC Ellen Kim, BA Benita Mudge, BS Vesna Sossi, PhD Howard Feldman, MD, FRCPC – Past Investigator Michele Assaly, MA – Past Investigator St. Joseph's Health Care: Elizabeth Finger, MD Stephen Pasternak, MD Irina Rachinsky, MD Andrew Kertesz, MD – Past Investigator Dick Drost, MD – Past Investigator John Rogers, MD – Past Investigator Northwestern University: Ian Grant, MD Brittanie Muse, MSPH Emily Rogalski, PhD Jordan Robson M.‐Marsel Mesulam, MD – Past Investigator Diana Kerwin, MD – Past Investigator Chuang‐Kuo Wu, MD, PhD – Past Investigator Nancy Johnson, PhD – Past Investigator Kristine Lipowski, MA– Past Investigator Sandra Weintraub, PhD – Past Investigator Borna Bonakdarpour, MD– Past Investigator Nathan Kline Institute: Nunzio Pomara, MD Raymundo Hernando, MD Antero Sarrael, MD University of California, San Francisco: Howard J. Rosen, MD Bruce L. Miller, MD David Perry, MD Georgetown University Medical Center: Raymond Scott Turner, MD, PhD Kathleen Johnson, NP Brigid Reynolds, NP Kelly MCCann, BA Jessica Poe, BS Brigham and Women's Hospital: Reisa A. Sperling, MD Keith A. Johnson, MD Gad A. Marshall, MD Stanford University: Jerome Yesavage, MD Joy L. Taylor, PhD Steven Chao, MD, PhD Jaila Coleman, BA Jessica D. White, BA – Past Investigator Barton Lane, MD – Past Investigator Allyson Rosen, PhD – Past Investigator Jared Tinklenberg, MD – Past Investigator Banner Sun Health Research Institute: Christine M. Belden, PsyD Alireza Atri, MD, PhD Bryan M. Spann, DO, PhD Kelly A. Clark Edward Zamrini, MD – Past Investigator Marwan Sabbagh, MD – Past Investigator Boston University: Ronald Killiany, PhD Robert Stern, PhD Jesse Mez, MD, MS Neil Kowall, MD – Past Investigator Andrew E. Budson, MD – Past Investigator Howard University: Thomas O. Obisesan, MD, MPH Oyonumo E. Ntekim, MD, PhD Saba Wolday, MSc Javed I. Khan, MD Evaristus Nwulia, MD Sheeba Nadarajah, PhD Case Western Reserve University: Alan Lerner, MD Paula Ogrocki, PhD Curtis Tatsuoka, PhD Parianne Fatica, BA, CCRC University of California, Davis – Sacramento: Evan Fletcher, PhD Pauline Maillard, PhD John Olichney, MD Charles DeCarli, MD Owen Carmichael, PhD – Past Investigator Dent Neurologic Institute: Vernice Bates, MD Horacio Capote, MD Michelle Rainka, PharmD, CCRP Parkwood Institute: Michael Borrie, MB ChB T‐Y Lee, PhD Dr Rob Bartha, PhD University of Wisconsin: Sterling Johnson, PhD Sanjay Asthana, MD Cynthia M. Carlsson, MD, MS Banner Alzheimer's Institute: Allison Perrin, PhD Anna Burke, PhD – Past Investigator Ohio State University: Douglas W. Scharre, MD Maria Kataki, MD, PhD Rawan Tarawneh, MD Brendan Kelley, MD – Past Investigator Albany Medical College: David Hart, MD Earl A. Zimmerman, MD Dzintra Celmins, MD University of Iowa College of Medicine Delwyn D. Miller, PharmD, MD Laura L. Boles Ponto, PhD Karen Ekstam Smith, RN Hristina Koleva, MD Hyungsub Shim, MD Ki Won Nam, MD – Past Investigator Susan K. Schultz, MD – Past Investigator Wake Forest University Health Sciences: Jeff D. Williamson, MD, MHS Suzanne Craft, PhD Jo Cleveland, MD Mia Yang, MD– Past Investigator Kaycee M. Sink, MD, MAS – Past Investigator Rhode Island Hospital: Brian R. Ott, MD Jonathan Drake, MD Geoffrey Tremont, PhD Lori A. Daiello, Pharm.D, ScM Jonathan D. Drake, MD Cleveland Clinic Lou Ruvo Center for Brain Health: Marwan Sabbagh, MD Aaron Ritter, MD Charles Bernick, MD, MPH – Past Investigator Donna Munic, PhD – Past Investigator Akiva Mintz, MD, PhD – Past Investigator Roper St. Francis Healthcare: Abigail O'Connelll, MS, APRN, FNP‐C Jacobo Mintzer, MD, MBA Arthur Wiliams, BS Houston Methodist Neurological Institute: Joseph Masdeu, PhD Barrow Neurological Institute: Jiong Shi, MD, PhD Angelica Garcia, BS Marwan Sabbagh – Past Investigator Vanderbilt University Medical Center: Paul Newhouse, PhD Long Beach VA Neuropsychiatric Research Program: Steven Potkin, PhD Butler Hospital Memory and Aging Program: Stephen Salloway, MD, MS Paul Malloy, PhD Stephen Correia, PhD Neurological Care of CNY: Smita Kittur, MD – Past Investigator Hartford Hospital, Olin Neuropsychiatry Research Center: Godfrey D. Pearlson, MD – Past Investigator Karen Blank, MD – Past Investigator Karen Anderson, RN – Past Investigator Dartmouth‐Hitchcock Medical Center: Laura A. Flashman, PhD – Past Investigator Marc Seltzer, MD – Past Investigator Mary L. Hynes, RN, MPH – Past Investigator Robert B. Santulli, MD – Past Investigator Cornell University Norman Relkin, MD, PhD – Past Investigator Gloria Chiang, MD – Past Investigator Michael Lin, MD – Past Investigator Lisa Ravdin, PhD – Past Investigator Athena Lee, PhD

Supporting information

Supporting Information

Supporting Information

Aslanyan V, Pa J, Hodis HN, et al. Generalizability of cognitive results from clinical trial participants to an older adult population: Addressing external validity. Alzheimer's Dement. 2023;15:e12417. 10.1002/dad2.12417