Accounting for lack of representation in dementia research: Generalizing KHANDLE study findings on the prevalence of cognitive impairment to the California older population

Abstract

INTRODUCTION:

Most dementia studies are not population-representative; statistical tools can be applied to samples to obtain critically-needed population-representative estimates, but are not yet widely used.

METHODS:

We pooled data from the Kaiser Healthy Aging and Diverse Life Experiences (KHANDLE) study and the California Behavioral Risk Factor Surveillance Study (CA-BRFSS), a population-representative study. Using weights accounting for sociodemographic/health differences between KHANDLE and CA-BRFSS, we estimated cognitive impairment prevalence and age- and sex-adjusted racial/ethnic inequalities in California adults 65+ without prior dementia diagnosis.

RESULTS:

After weighting KHANDLE, the estimated cognitive impairment prevalence in California was 20.3% (95% confidence interval 17.8%–23.0%); unweighted prevalence was 24.8% (23.1%–26.6%). Inequalities (larger prevalences) were observed among Black and Asian groups versus whites.

DISCUSSION:

We employed a novel statistical approach to estimate population-representative cognitive impairment prevalence and inequalities. Such statistical tools can help obtain population-representative estimates from existing studies and inform efforts to reduce racial/ethnic disparities.

Introduction

Population-representative estimates of dementia and cognitive impairment, which includes persons at high risk for progressing to dementia [1], are needed to understand burden of disease and disparities [2,3]. However, dementia research study participants often differ from the general population on sociodemographic and health factors [2–4]. Participants are often predominantly white, highly educated, or have family history of dementia [2,4,5]. These and other potential differences may affect study outcomes, presenting a substantial obstacle to obtaining population estimates. This gap is especially salient when characterizing outcomes among minoritized racial/ethnic groups [3,5,6].

The Kaiser Healthy Aging and Diverse Life Experiences (KHANDLE) study, a recently-recruited cohort study in California, was designed to evaluate how race/ethnicity and lifecourse factors influence late-life brain health and cognitive decline. KHANDLE recruited approximately equal proportions of Asian, Black, Latino, and white participants, making the sample more diverse than most dementia research studies [3,5]. However, even in diverse samples, selection processes that lead to differences between participants and the general population may remain and, if unaccounted for, could affect generalizability (i.e., yield selection bias). Statistical tools such as weighting can be used to generalize study results to populations of interest [7,8]. We aimed to generalize from KHANDLE and obtain race/ethnicity-specific prevalence and inequality estimates for cognitive impairment in the population of Asian, Black, Latino, or white, English- and Spanish-speaking California adults 65+ without prior dementia diagnosis.

Methods

Data and participants

We obtained cognitive impairment prevalence estimates in KHANDLE and used the California Behavioral Risk Factor Surveillance Study (CA-BRFSS), a population-representative study, to generalize estimates to the California population. KHANDLE comprises community-dwelling older adults residing in the San Francisco Bay and Sacramento areas recruited between March 2017 and December 2018. Individuals eligible for KHANDLE were long-term members of Kaiser Permanente Northern California (KPNC), an integrated healthcare delivery system, were age ≥65 years on January 1, 2017, spoke English or Spanish, and participated in Kaiser Permanente multiphasic health checkup exams between 1964–1985. Random sampling was stratified by race/ethnicity and educational attainment to recruit approximately equal proportions of Asian, Black, Latino, and white participants with diversity in educational attainment. Exclusion criteria included: diagnosis of dementia or other neurodegenerative disease (frontotemporal dementia, Lewy body disease, Pick’s disease, Parkinson’s disease with dementia, Huntington’s disease) and presence of health conditions that would impede participation in study interviews (i.e. hospice activity in past 12 months, history of severe chronic obstructive pulmonary disease in past 6 months, congestive heart failure hospitalizations in past 6 months, and history of end stage renal disease or dialysis in past 12 months). KHANDLE enrolled 1,712 individuals; this analysis includes 1,709 participants who reported Black, Latino, Asian, or white race/ethnicity.

For our population-representative sample, we pooled data from the 2014–2018 CA-BRFSS. BRFSS is a national annual health survey that is representative of the population in each state [9]. Computer-assisted telephone interviews (English or Spanish) were conducted among California residents via landline and cell phones. Because proxy interviews were not permitted, we expect very few participants with moderate to severe dementia were included, mirroring KHANDLE’s exclusion of individuals with a dementia diagnosis. In this analysis, we restricted to participants who reported Black, Latino, Asian, or white race/ethnicity and were age ≥65 years (unweighted n=12,399). We applied sampling weights developed by the Centers for Disease Control to make CA-BRFSS representative of the California 65+ population (weighted n=5,317,953) [9–11]. Specifically, we calculated 2014–2018 pooled weights by multiplying each participant’s weight by the proportion of their respective year’s sample relative to the pooled sample [10].

Measures

Harmonized measures

To estimate weights accounting for sociodemographic and health-related differences between KHANDLE and the California 65+ population represented by CA-BRFSS, we harmonized variables across the datasets. We briefly describe the harmonization process below; Supplement 1 provides details.

Race/ethnicity

KHANDLE and CA-BRFSS asked participants to self-report race/ethnicity and allowed multiple responses. We derived a harmonized race/ethnicity summary variable as follows: participants reporting Latino ethnicity were categorized as Latino; otherwise, a primary race/ethnicity was assigned based on race reported (Black, Asian, or white). Participants who did not report Latino ethnicity and reported multiple races were assigned a primary race/ethnicity according to historical disenfranchisement in the U.S. (Black, Asian, then white). Due to small sample sizes for analysis in both samples, we excluded other races/ethnicities (e.g. American Indian/Alaska Native) and individuals with missing race/ethnicity.

Sociodemographic variables

Age was harmonized as a continuous variable top-coded at 90 years, and we created indicators for sex (male/female), marital status (yes/no married or living with a partner as if married), interview language (English/Spanish), and history of military service (yes/no). Educational attainment was defined by highest grade/degree completion: “less than high school graduate/GED”, “high school graduate/GED”, “technical/trade program or some college,” “college completion/graduate degree.” Per capita income was estimated and dichotomized at the median reported in CA-BRFSS (details in Supplement 1a).

Health-related variables

Self-rated health was assessed in KHANDLE and CA-BRFSS identically; we created an indicator for good health (“excellent/very good/good” versus “fair/poor”). Smoking history indicated whether participants currently smoked versus not. Serious vision impairment was defined using self-rated eyesight with glasses or contact lenses (poor or legally blind) for KHANDLE and using self-report of blindness or serious difficulty seeing, even with glasses for CA-BRFSS. The harmonized physical activity measure captured exercise in the last month (details in Supplement 1b). Finally, we created two dichotomous activities of daily living measures indicating difficulty walking/climbing stairs and dressing/bathing (details in Supplement 1c).

Cognitive impairment outcome

Cognitive impairment was available only in KHANDLE; hence the need to generalize results from KHANDLE to the population represented by CA-BRFSS. Cognitive impairment, defined as mild cognitive impairment or dementia, was determined in KHANDLE as reported previously [12,13]. Briefly, all participants were administered two cognitive test batteries: the Spanish and English Neuropsychological Assessment Scales (SENAS) [14] and the NIH Toolbox Cognitive Health Battery (NIHTB-CHB) [15–17]. A random sample within each racial/ethnic group (n=541) were selected to undergo a full clinical evaluation, including clinical neuropsychological testing and a clinical exam, and clinical diagnosis adjudicated by three senior clinicians. A screening algorithm was used to select remaining participants for clinical neuropsychological testing if they were classified as high probability of impairment based on NIHCTB-CHB measures, and those with abnormal clinical neuropsychological test results were referred for a clinical exam. Although KHANDLE exclusion criteria precluded prior dementia diagnosis, a small proportion (<1%) of participants with an adjudicated diagnosis were diagnosed with dementia; the majority of cognitive impairment was mild cognitive impairment (participants with adjudicated cognitive impairment described in Supplement 2). Among participants with an adjudicated diagnosis, a model was developed to predict cognitive impairment using adjudicated diagnosis as the dependent variable, and SENAS and NIHTB-CHB cognitive test scores, age, education, gender, race/ethnicity, and race/ethnicity by test score interactions as independent variables. This model was used to predict probabilities of cognitive impairment for participants who did not receive adjudicated diagnoses. Prevalence of cognitive impairment was taken as the sum of individual predicted probabilities (0 or 1 for individuals with adjudicated diagnoses) divided by the sample size (n=1,709).

Statistical analysis

Overview

We aimed to generalize estimates of prevalence and inequalities in cognitive impairment from KHANDLE to the “target population” of California represented by CA-BRFSS. The basic idea behind weighting, a multivariable form of standardization that allows for continuous variables [18], is to re-weight the study sample to mimic the population of interest. Broadly, generalizing from KHANDLE to CA-BRFSS required (a) understanding differences in characteristics of KHANDLE versus CA-BRFSS participants, (b) modeling probability of participation in KHANDLE based on these characteristics and constructing weights such that people underrepresented in KHANDLE compared to CA-BRFSS are given greater weight, and (c) applying those weights in KHANDLE to obtain estimates for the target population (Figure S1). We conducted each step stratified by race/ethnicity. We explain each step below and provide code (github.com/mayeda-research-group/KHANDLE-weighting) that can be adapted by readers to implement these methods.

Evaluating differences between KHANDLE and CA-BRFSS

We identified harmonized variables for which KHANDLE and CA-BRFSS differed by assessing covariate balance between KHANDLE and CA-BRFSS using race/ethnicity-specific standardized mean differences, calculated as (mean(KHANDLE_k)-mean(CA-BRFSS_k))/SD(CA-BRFSS_k) for each racial/ethnic group and variable k. Differences <0.25 were deemed to represent adequate covariate balance [19].

Estimating and evaluating weights

To model the selection process (i.e., differences between KHANDLE and CA-BRFSS), we developed participation weights (specifically, stabilized inverse odds of selection weights [8]), using an iterative process. First, we pooled KHANDLE and CA-BRFSS data and used logistic regression to estimate probability (propensity score) that each observation was from KHANDLE versus CA-BRFSS. Second, we used the propensity scores to calculate weights for KHANDLE participants, defined as inverse of the odds that a participant was included in KHANDLE given their covariate profile, multiplied by the unconditional odds of participating in KHANDLE [8]. Third, we applied weights to KHANDLE and re-assessed covariate balance. Fourth, we added variables to the logistic regression model to improve covariate balance as needed. The first logistic regression model adjusted for sociodemographic variables (race/ethnicity, age, sex, educational attainment, and interactions between race/ethnicity and all other variables). The final logistic regression model also included indicators for greater than median income per household member, mobility difficulties, interview language, and self-rated health, as well as interactions between race/ethnicity and these variables. Conceptually, observations with covariate combinations underrepresented in KHANDLE versus CA-BRFSS received relatively large weights and observations with covariate combinations overrepresented in KHANDLE versus CA-BRFSS received relatively small weights.

To evaluate final weight performance, we examined covariate balance between weighted KHANDLE and CA-BRFSS. We also used density plots to evaluate overlap in propensity scores in the two samples to assess whether there were large segments of the CA-BRFSS sample that would have little or no chance of being included in KHANDLE. Overlap in range of propensity scores indicates that the samples have enough similarity to generalize from KHANDLE to CA-BRFSS.

Applying weights to obtain estimates for the target population represented by CA-BRFSS

CA-BRFSS data were used to estimate weights in KHANDLE as described above; subsequent analyses required only KHANDLE data (weighted and unweighted). To generalize results from KHANDLE to CA-BRFSS, we used weighted KHANDLE data to estimate overall and race/ethnicity-specific prevalence of cognitive impairment in the California population of older adults without diagnosed dementia represented by CA-BRFSS.

We estimated population-representative age- and sex-adjusted racial/ethnic inequalities in cognitive impairment prevalence by standardizing race/ethnicity-specific prevalences in the weighted KHANDLE sample to the marginal age and sex distribution in CA-BRFSS and calculating prevalence ratios (PRs) and prevalence differences (PDs). Standardizing accounted for differences in age- and sex- distributions across racial/ethnic groups and facilitated estimation of racial/ethnic inequalities that would be observed in CA-BRFSS if all racial/ethnic groups had the same age and sex distribution.

Finally, for comparison, we estimated analogous unweighted results in the KHANDLE sample, including overall and race/ethnicity-specific prevalence of cognitive impairment and age- and sex-adjusted racial/ethnic inequalities standardized to the marginal age and sex distribution in KHANDLE.

Missing data

Missing data were multiply imputed with chained equations and predictive means matching [20]. Data on income was most frequently missing (12.9% KHANDLE, 20.1% CA-BRFSS); all other variables were missing ≤2.7% (KHANDLE) or ≤8.5% (CA-BRFSS). Details in Supplement 4. Separate imputation models were developed for KHANDLE and CA-BRFSS (40 imputed datasets each). Weight creation and estimation of cognitive impairment prevalence was conducted for pairs of imputed KHANDLE and CA-BRFSS datasets (m=40 pairs). Final estimates of cognitive impairment prevalence were calculated by averaging estimates across imputations.

To account for uncertainty due to use of weights and Rubin’s rules for combining variance across imputations, confidence intervals were calculated using bootstrapping [21,22]. We resampled and estimated overall and race/ethnicity-specific cognitive impairment prevalence and age- and sex-standardized prevalence ratios and differences 1,000 times from each of the 40 imputed KHANDLE datasets; confidence intervals were defined as the 2.5^th and 97.5^th percentiles of bootstrapped estimates [21]. Analyses were conducted in R version 4.0.2 using Twang and mice packages [23,24].

Results

Compared with CA-BRFSS, KHANDLE participants were slightly older, more racially/ethnically diverse (e.g., 25.9% vs. 6.0% Black), had higher education and income, and better health (Table 1). These differences were generally similar across racial/ethnic groups (Figure 1A, Table S4), although some larger differences were observed for Latinos (shown in blue). Details on final weight evaluation are in Supplement 6; importantly, propensity score distributions substantially overlapped between KHANDLE and CA-BRFSS. After applying weights, race/ethnicity-specific covariate balance was acceptable for almost all variables, including variables not included in the model that produced the weights (Figure 1B). The exception was ‘good’ or better self-rated health in Latino participants (Figure 1B), which remained slightly higher in KHANDLE than CA-BRFSS even after including that variable in the weight model.

Table 1.

Characteristics of Black, Latino, Asian, and white KHANDLE and CA-BRFSS (2014–2018) participants

Characteristic	KHANDLE N=1709	CA-BRFSS N=12,399
Race/ethnicity
Asian	24.3%	14.0%
Black	25.9%	6.0%
Latino	20.4%	18.6%
White	29.4%	61.4%
Age, mean (SD)	76.0 (6.8)	73.9 (6.9)
Male	40.6%	44.1%
Educational attainment
Less than high school	7.0%	16.1%
High school diploma/GED	9.8%	18.0%
Trade school/technical school/some college	35.0%	34.9%
College graduate or higher	48.2%	30.9%
Per capita income above BRFSS median	70.6%	49.8%
Interviewed in English	97.1%	89.4%
Married/living with partner	56.4%	54.0%
Military service	18.0%	19.8%
Health-related variables
Self-rated health “good” or better	81.2%	75.3%
Difficulty walking/climbing stairs	13.7%	26.7%
Blind or serious vision impairment	4.1%	6.8%
Difficulty dressing	2.0%	6.5%
Current smoker	3.0%	4.2%
Exercised in the last month	84.8%	75.8%

Note:Characteristics are averaged across 40 multiply imputed samples, and CA-BRFSS percentages are shown weighted to be representative of the CA population of adults 65+.

Abbreviations: BRFSS, Behavioral Risk Factor Surveillance System; KHANDLE, Kaiser Healthy Aging and Diverse Life Experiences; SD, standard deviation.

Figure 1.

Covariate balance (race/ethnicity-specific standardized mean differences) between Kaiser Healthy Aging and Diverse Life Experiences (KHANDLE) and California Behavioral Risk Factor Surveillance System (CA-BRFSS) shown for (A) KHANDLE unweighted, and (B) after applying weights to KHANDLE. Standardized mean differences shown are averages across 40 multiply imputed datasets

After weighting KHANDLE to generalize to CA-BRFSS, overall cognitive impairment prevalence was estimated as 20.3% (95% CI 17.8%–23.0%) (Figure 2 and Table S6); the unweighted KHANDLE estimate was 24.8% (23.1%–26.6%). Within racial/ethnic groups, the generalized (weighted) prevalence estimate was highest among Black participants (29.7% [23.6%–35.9%]), followed by Asians (25.0% [18.3%–33.7%]), whites (19.0% [15.7%–22.4%]), and Latinos (18.0% [12.9%–23.7%]). The generalized prevalence was substantially lower than the unweighted KHANDLE estimate among Latinos, slightly lower among Black and white participants, and slightly higher among Asians.

Figure 2.

Estimates of prevalence of cognitive impairment in Kaiser Healthy Aging and Diverse Life Experiences (KHANDLE) generalized to California Behavioral Risk Factor Surveillance System (CA-BRFSS) and unweighted KHANDLE, overall and by race/ethnicity

After generalizing to CA-BRFSS and age- and sex-standardizing, the largest inequalities (versus whites) were observed among Black individuals (PR=1.71 [1.35–2.16], and PD=13.3% [7.2%–19.7%]), followed by Asians (PR=1.46 [1.11–1.93], and PD=8.6% [2.1%–15.8%]) (Figure 3 and Table S7). Small Latino-white inequalities were observed, but estimates were imprecise (PR=1.14 [0.85–1.53], and PD=2.7% [−3.1%–9.0%]). Compared with inequality estimates in unweighted KHANDLE, the generalized Asian-white inequality was larger, and the Latino-white inequalities were slightly smaller. Black-white inequalities were similar in the unweighted and generalized estimates.

Figure 3.

Estimates of population-representative age- and sex-adjusted* racial/ethnic inequalities in cognitive impairment in KHANDLE generalized to CA-BRFSS: Panel A) prevalence ratios, and Panel B) prevalence differences

*Unweighted KHANDLE standardized to marginal age/sex distribution in KHANDLE; KHANDLE generalized to CA-BRFSS standardized to marginal age/sex distribution in CA-BRFSS

Discussion

We used an approach from the statistical and epidemiologic literature [8] novel in dementia research to generalize prevalence of cognitive impairment from KHANDLE, a diverse study of cognitive aging, to a target population of California older adults. Because KHANDLE excluded those with diagnosed dementia, our research question focused on a target population also without a dementia diagnosis (represented by CA-BRFSS, which prohibited proxy respondents). After weighting KHANDLE data, we estimated 20.3% prevalence of cognitive impairment in Asian, Black, Latino, and white English- and Spanish-speaking California adults 65+ without prior dementia diagnosis, with highest burden among Black individuals. We also estimated substantial inequalities (elevated prevalence of cognitive impairment) compared with whites in Black and Asian older adults, and small differences for Latinos.

It is increasingly recognized that dementia research study participants substantially differ from the general population [2,3,25], imposing barriers to obtaining population-representative estimates of dementia-related outcomes and inequalities. For example, analyses of National Alzheimer’s Coordinating Center (NACC) data suggest Black-white inequalities in diagnostic progression in NACC are not generalizable to the general population [25]. Our findings demonstrate that correction for selection processes are necessary to generalize from a study sample (here, KHANDLE) to a broader target population of interest (here, represented by CA-BRFSS). Selection into studies of cognitive aging such as KHANDLE may be driven by factors associated positively or negatively with cognitive health (e.g., older age, family history of dementia, comorbidities, concern about cognition, race/ethnicity, education, socioeconomic status) [26–29]. The direction of the difference between an estimate in a sample and the target population depends on the balance of opposing selection factors. In our analysis, although KHANDLE participants had higher levels of education and income than CA-BRFSS participants, the higher prevalence of cognitive impairment in unweighted KHANDLE versus KHANDLE generalized to CA-BRFSS likely reflects KHANDLE’s older age and oversampling of Black participants, a group with high cognitive impairment prevalence. Although cognitive impairment prevalence estimates depend on diagnostic assessment methods [13,30], the direction of change when generalizing from a sample to a target population is driven by the selection process into the study.

Our population-representative estimates of racial/ethnic inequalities in cognitive impairment among older adults without prior dementia diagnosis are the first of their kind, and are partially consistent with extant literature. Both our weighted (20.3%) and unweighted (24.8%) prevalence estimates of cognitive impairment are higher than the estimated prevalence of mild cognitive impairment for persons age 65+ from a prior systematic review (16.6%) [1]. Several important differences between our study and the meta-analysis that could contribute to our higher estimate include that KHANDLE is more racially/ethnically diverse than most studies, the meta-analysis did not correct for selection bias, and some studies included in the meta-analysis did not exclude people with diagnosed dementia from the denominator, as our study does. Our estimates of large Black-white inequalities and small Latino-white inequalities is consistent with evidence on dementia incidence and prevalence in these groups [29,31]. Our finding of higher cognitive impairment prevalence in Asians versus whites was unexpected given prior reports of lower age-adjusted dementia incidence rates among Asian versus white KPNC members, although no prior prevalence estimates exist [31]. These results could reflect several possibilities. First, racial/ethnic patterns in cognitive impairment could differ from dementia. Second, Asians have longer survival following dementia diagnosis [32] and may similarly have longer survival with cognitive impairment, which would increase prevalence in this group relative to other groups, independent of incidence. Finally, language fluency can influence cognitive test results: cognitive assessments were offered in English or Spanish; non-native English language among Asians may have resulted in higher estimated cognitive impairment in this group.

Generalizing estimates to a target population via weighting requires assumptions, most importantly that the weights account for the impact of the selection process on the outcome of interest [7,8]. However, neither understanding nor modeling the selection process is straightforward. One assessment of whether the selection process was adequately modeled is evaluating whether variables that affect the outcome have similar distributions (are balanced) in the weighted sample and the target population. In our analyses, we achieved good covariate balance on almost all available variables, including variables not included in the weights. This suggests that we successfully modeled the selection process into KHANDLE for measured variables, and likely closely-related unmeasured variables, for all racial/ethnic groups. However, it is possible that some important unmeasured variables predictive of cognitive impairment differ between weighted KHANDLE and our target population. For example, we did not have harmonized measures of nativity, education quality, family history of dementia, or comorbidities. The extent to which omission of these variables biases results depends on the extent to which weighting on the measured covariates accounts for these differences. For example, while comorbidities are not included in our weights, our weights include self-rated health and mobility difficulties, which are highly correlated with comorbidities [33,34]. Additionally, differences by race/ethnicity in covariate balance before weighting indicated that selection processes differed by race/ethnicity. Measures like nativity could be more important for Latino and Asian estimates than for Black and white estimates, especially because KHANDLE participation required long-term KPNC membership.

Variables measured with error, especially differential error across the sample and target population, could result in appearance of a well-modeled selection process without fully correcting for selection bias. In our analysis, most items were straightforward to harmonize between KHANDLE and CA-BRFSS, but measurement error in these variables, or imperfect harmonization for more complex variables like income, could leave residual differences between weighted KHANDLE and CA-BRFSS. Missing selection factors or imperfect harmonization may have larger consequences for inequality estimates (i.e. prevalence ratios and differences) if bias in generalized prevalence estimates occurs in opposite directions for racial/ethnic groups being compared (e.g. if generalized prevalence is overestimated among Asians and underestimated among whites).

The main limitations of our study relate to the challenges of understanding and modeling the selection process, including harmonization of measures. A key strength of our work is that KHANDLE is an unusually racially/ethnically diverse sample with robust cognitive assessments. This enabled precise estimation and generalization of cognitive impairment prevalence within racial/ethnic groups. Even with statistical tools, samples without representation of all groups in the target population cannot be generalized [7,8]. Additionally, our weighting approach is novel in dementia research. We anticipate that these methods will be useful for dementia researchers interested in drawing inferences about broader populations from study samples.

In conclusion, population-representative estimates in dementia research are important for understanding public health burden and disparities, but efforts are stymied by the non-representative nature of most samples [2,3]. We employed statistical tools to correct for such selection factors and allow estimation of prevalence of and inequalities in cognitive impairment in the population of California older adults without prior dementia diagnosis. Our results indicated that cognitive impairment is common for older adults and that there are substantial racial/ethnic inequalities in cognitive impairment. Even in KHANDLE, an unusually diverse study of cognitive aging, there were important differences between the sample and the general population of older adults. Establishment of cohorts that include diverse participants is an ongoing and critical step for the field, but statistical tools may still be required to generalize results. These tools are only as good as the data underlying the models [7,8]. In addition to greater diversity within cohorts, measuring factors that affect outcomes of interest and selection into studies will improve the ability to obtain population-representative estimates in dementia research, which in turn inform efforts to reduce disparities.