Combining direct and proxy assessments to reduce attrition bias in a longitudinal study

Abstract

Retaining severely impaired individuals poses a major challenge in longitudinal studies of determinants of dementia or memory decline. In the Health and Retirement Study (HRS), participants complete direct memory assessments biennially until they are too impaired to complete the interview. Thereafter, proxy informants, typically spouses, assess the subject’s memory and cognitive function using standardized instruments. Because there is no common scale for direct memory assessments and proxy assessments, proxy reports are often excluded from longitudinal analyses. The Aging and Demographics Memory Study (ADAMS) implemented full neuropsychological exams on a subsample (n=856) of HRS participants, including respondents with direct or proxy cognitive assessments in the prior HRS core interview. Using data from ADAMS, we developed an approach to estimating a dementia probability and a composite memory score based on either proxy or direct assessments in HRS core interviews. The prediction model achieved a c-statistic of 94.3% for DSM diagnosed dementia in the ADAMS sample. We applied these scoring rules to HRS core sample respondents born 1923 or earlier (n=5,483) for biennial assessments 1995-2008. Compared to estimates excluding proxy respondents in the full cohort, incorporating information from proxy respondents increased estimated prevalence of dementia by 12 percentage points in 2008 (average age = 89) and suggested accelerated rates of memory decline over time.

1. Introduction

In longitudinal studies of the determinants of dementia or memory decline, it is essential to retain severely cognitively impaired individuals in the sample. Selective attrition of impaired individuals may lead to underestimates of rate of cognitive change and incidence of impairment. Further, such selective dropout can bias estimated effects of risk factors for dementia and related outcomes. Few studies have optimal tools to retain severely impaired individuals.

In the Health and Retirement Study (HRS), participants complete direct memory assessments biennially until they are too impaired to complete the interview (1). Thereafter, proxy informants, typically spouses, report on the memory and cognitive function of study participants using validated instruments. Proxy reports are often excluded from longitudinal analyses (2-4), however, because no common scale for the direct and proxy assessments has been available. Although poor proxy assessments probably correspond with impaired memory functioning, there was no way to quantify the magnitude of impairment because proxy and direct assessments were never available for the same people at the same interview wave. To circumvent this problem, some prior studies dichotomize the cognitive outcome and class proxy respondents as “impaired,” but this dichotomization reduces statistical power (5, 6). Further, it is unclear what cut-point on the continuous direct assessments corresponds with the level of impairment in the proxy respondents. The Aging, Demographics, and Memory Study (ADAMS), which implemented full neuropsychological batteries and dementia diagnoses for a subsample of HRS participants, provides an opportunity to calibrate proxy and direct memory assessments onto the same scale. By applying this calibration to the full HRS cohort, we can calculate a predicted probability of dementia and a continuous memory score using data available for all HRS participants and examine how these outcomes changed over time in this nationally-representative cohort.

In this report, we use the ADAMS subsample to develop simple approaches to combining HRS proxy and direct memory assessments to estimate: 1) a dementia predicted probability score and 2) a continuous memory score. Our approach derives scores for all ADAMS participants and then uses regression models to calibrate the direct and proxy reports from the prior HRS core interview to predict the ADAMS outcomes. These scores can be used for longitudinal analyses of HRS core interviews. We then apply these prediction rules to estimate dementia and memory changes among the oldest respondents in the HRS cohort followed biennially from 1995 through 2008.

2. Methods

2.1. Samples

Data are from the HRS (the parent study) and ADAMS (a validation substudy). HRS is a nationally representative longitudinal study of US adults age 50+ and their spouses (1, 7-10). Enrollments occurred in 1992, 1993, 1998, and 2004, depending on birth year. Participants completed approximately biennial interviews about health, including brief cognitive assessments. Cohort members too ill to participate in core HRS interviews were retained in the sample via proxy interviews.

In 2001-2003, ADAMS was initiated, using a random sample of participants age 70+ in the 2000 or 2002 waves of HRS, with oversampling of individuals with poor cognitive test scores. In-person clinical assessments and comprehensive neuropsychological exams were conducted on 856 respondents. Technical details for ADAMS are available elsewhere (3, 11, 12).

2.2. Measures

2.2.1. HRS core biennial cognitive assessments

Brief cognitive assessments were conducted either in person or by telephone in each HRS core interview. An earlier randomization study showed no significant interview mode effect on participants’ cognitive performance (1). Memory was assessed with immediate and (approximately 5-minute) delayed recall of a list of ten common nouns read aloud. Participants also completed the Telephone Interview for Cognitive Status (TICS), which included serial-7 subtractions (13). The HRS interview schedule implements TICS only once for respondents under the age of 65 and biennially thereafter; the word recall assessments are completed biennially regardless of participant age. For respondents too impaired to participate in direct cognitive assessments (about 10% each wave), a proxy (approximately 70% spouses) was interviewed. Proxies rated subjects’ memory performance on a 5-point Likert scale (originally coded as: 1 = excellent to 5 = poor) and completed a 16-item version of the Jorm Informant Questionnaire for Cognitive Decline (IQCode) (14-16). IQCode summary score averaged all 16 items, so the possible range was 1 to 5.

2.2.2. ADAMS neuropsychological battery

All ADAMS participants were assigned dementia diagnoses based on DSM-III-R and DSM-IV (11, 17). The diagnosis was initially reached by a study geropsychiatrist, a neurologist, a neuropsychologist, and a cognitive neuroscientist based on in-home assessment, and later confirmed when the study geropsychiatrist reviewed available medical records. Of the 856 ADAMS participants, 795 (93%) completed direct memory assessments based on the CERAD Word-List Learning-, Delayed Recall and Recognition (18) and/or the Wechsler Memory Scale-Logical Memory(19). Both CERAD and Wechsler assessments produced an immediate and a delayed recall score and 722 participants completed all four assessments. We created an ADAMS composite memory score averaging the non-missing Z-scores on these 4 assessments.

Nearly all ADAMS participants (n=852) had clinical dementia ratings (CDRs) (18, 20, 21). CDR reflects the presence and severity of dementia as it affects functioning, and ranges from 0 (no dementia) to 5 (terminal dementia).

Because CDR is only available for ADAMS participants, this additional data can be used to recover information for the 61 ADAMS participants who did not complete direct memory assessments. To achieve this goal, we used linear regression to impute composite memory scores based on CDR, HRS cognitive scores, and basic demographics. Results were nearly identical when HRS cognitive scores were omitted from this second phase imputation model.

2.3. Analyses

Figure 1 displays the available data sources and our analytical steps. Participants in the full HRS cohort (n=20,316) were assessed biennially (either directly or via a proxy) and provided either directly-assessed or proxy-reported memory assessments. Among those, a subsample participated in ADAMS (n=856), and completed the comprehensive neuropsychological battery, which included many more detailed direct memory assessments and assigned a dementia diagnosis.

Figure 1.

Study and analytic flow chart

In analysis step 1, we built prediction models for dementia diagnoses and composite memory assessments using 745 non-Hispanic ADAMS participants. For the dementia model, we used logistic regression, with DSM dementia diagnosis from ADAMS interviews as the binary outcome (demented vs. not demented). We considered as predictors variables drawn from the HRS core interview immediately prior to the ADAMS assessment (i.e. 2000 or 2002, an average of 12.9 months earlier) including demographics (age, sex, and race) and cognitive assessments (immediate and delayed word recall, TICS, IQCode, and proxy memory assessment). The time lag between the HRS core interview and the ADAMS assessment was not significantly associated with dementia or memory scores in ADAMS and was therefore not included in the prediction models. Direct cognitive assessments in the HRS core interview were available for 76% of ADAMS respondents and proxy assessments for the remaining 24%. To retain participants regardless of whether they participated directly or via proxy in HRS core interviews, we adopted the missing-indicator method (22), and included a binary indicator variable for whether the interview was by proxy. For individuals who completed direct cognitive assessments, the proxy variables were set to zero. For individuals with proxy assessments, their direct assessments scores were set to zero. Note that we re-centered both IQCODE and proxy memory scores by subtracting 5 (the worst proxy assessment scores) in regression models so that the coefficient for the proxy indicator is interpretable as the difference in log odds of dementia between individuals with the worst performance on direct assessments and the worst possible proxy assessments. We restricted to non-Hispanics, because preliminary analyses indicated the correspondence between HRS core interview cognitive assessments and ADAMS neuropsychological battery scores differed for Hispanics. With few Hispanics in ADAMS (n=84), we could not derive stable estimates for Hispanics.

We tested interactions between demographics and each cognitive score. We also tested quadratic terms for the cognitive scores and age, and we only retained interactions and quadratic terms significant at p<.05.

We developed a continuous memory score using a similar approach. We used identical model development procedures as for dementia probability score, but with linear regression model predicting ADAMS composite memory score as the outcome and excluding TICS from the predictor list. We excluded TICS so the continuous memory score could be calculated for HRS participants under age 65, when TICS was not routinely assessed in the cohort given the very low prevalence of dementia at younger ages.

In analysis step 2, we applied the beta coefficients obtained from our ADAMS prediction models to HRS 1995 core interview respondents; we imputed their dementia probability scores and memory scores for each assessment wave from 1995 through 2008.

In analysis step 3, to provide evidence for the validity of our predicted scores, we computed the predicted dementia prevalence and average memory scores for 1995 to 2008 for non-Hispanic HRS cohort members born 1923 or earlier (72 or older in 1995, n=5,483). We plot changes in the prevalence of dementia and average memory for this cohort over time. We used this subsample of HRS, instead of all birth years, because it is the birth cohort with the longest continuous follow-up on the HRS core cognitive measures. In order to demonstrate the consequences of ignoring the proxy responses, we also computed predicted dementia probability and average memory scores for this HRS cohort when all proxy respondents were excluded.

Regression models and means were weighted by survey weights and executed in SAS 9.2. Code is available from the authors.

3. Results

Table 1 shows the beta coefficients predicting dementia diagnosis (left panel) and composite memory score (right panel) based on the brief direct and proxy cognitive assessments in the core HRS interviews (analysis step 1). The coefficient estimates for dementia diagnosis are on a logit scale; positive coeffient estimates are associated with increased dementia probability. Exponentiating these coefficients provides an odds ratio for dementia associated with a one unit change in the predictor variable; for example, the odds of dementia increased by a factor of exp(.095)=1.10 per additional year of age (for an individual with mean scores on the memory and TICS direct assessments). The final logistic regression model for dementia diagnosis achieved a c-statistic of 94.3%, which suggests good discrimination (23).

Table 1.

Regression coefficients predicting dementia diagnosis or memory score in non-Hispanic ADAMS participants, based on most recent core interview direct and proxy neuropsychological assessments and demographic (n=745)

	Dementia			Memory Score
	Logistic Regression Coefficientse	SE	P	Linear Regression Coefficients	SE	P
Predictor variablea
Intercept	4.608	1.459	.002	.422	.155	.007
Proxy Respondent (1=yes; 0=no)	1.889	2.069	.361	−1.388	.179	<.001
Word Immediate Recall	.933	.388	.016	.116	.026	<.001
Word Immediate Recall Squared	−.266	.070	<.001	-	-	-
Word Delayed Recall	−.797	.159	<.001	.024	.025	.331
TICSb	−1.075	.324	.001	-	-	-
TICS Squared	.043	.020	.036	-	-	-
IQCODEc (centered at 5)	2.220	1.119	.047	−.068	.070	.332
Proxy Memory Scored(centered at 5)	1.096	.423	.010	−.210	.083	.011
Male	−.854	.507	.092	−.132	.061	.031
Age (centered at 70)	.095	.025	<.001	−.067	.008	<.001
Black	−.695	.445	.118	−.398	.086	<.001
Word Delayed Recall*male	.543	.208	.009	-	-	-
Word Delayed Recall*age	-	-	-	.010	.002	<.001
IQCODE*male	1.551	.543	.004	−.446	.080	<.001
Proxy Respondent*Age	-	-	-	.039	.011	<.001

^aAll predictor variables were drawn from the most recently available HRS core interview. Quadratic and interaction terms that were not statistically significant predictors of one of the outcomes were dropped from the model for that outcome.

^bStandard TICS scoring includes immediate word recall, but we treated the immediate word recall as a separate item. We omitted items for “naming scissors” (because of low correlation with other items) and the second attempt at counting backwards from 20 (because of apparent inconsistencies in administration). TICS scores therefore ranged from 0 to 13.

^cProxies completed a 16-item version of the Jorm Informant Questionnaire for Cognitive Decline (IQCode). The IQCode assesses decline in several cognitively demanding tasks, such as recognizing familiar faces or understanding movie plots. For each task, proxies rank the participant on a 5-point scale (centered in this analysis to range from −4: much better to 0: much worse).

^dProxies rated the subject’s memory performance on a 5 point Likert scale (centered in this analysis to range from −4: excellent to 0: poor).

^eThese regression coefficients can be directly applied to estimate odds of dementia, based on individual age, sex, race, and either proxy assessments or performance on the direct neuropsychological assessments. Estimates are from a cohort and odds can be directly converted to probabilities.

The coefficient estimates for memory score are on a linear scale. Positive coefficient estimates are associated with improved memory score. The final linear regression model for composite memory score had an r² of 0.61 (Table 1).

Applying these scoring rules to estimate average dementia probability score and composite memory score (standardized using the 1995 mean and standard deviation) at each interview wave from 1995-2008 for non-Hispanic HRS cohort members born 1923 or earlier revealed consistent increases in dementia probability (Figure 2a) and declines in memory score (Figure 2b) across time, as expected. In Figure 2a, we also plotted predicted dementia prevalence rates excluding all proxy respondents. As expected, estimated dementia prevalence was much lower when all proxy respondents were deleted. The difference in prevalence rates between those including and excluding proxy respondents was .12 in 2008 with an average age of 89. We also contrasted the trajectory of average word list recall (excluding proxy assessments) with the trajectory of our imputed memory scores (based on both direct and proxy assessments). The standardized word recall list score with only the directly assessed respondents showed much slower average decline compared to the imputed memory score including both direct and proxy respondents (Figure 2b).

Figure 2.

Trajectories of cognitive change in the HRS 1995 cohort (n=5,483), contrasting estimates of dementia probability scores (2a) and memory functioning (2b) with (solid line) and without (dashed line) proxy respondents. Scores were calculated by applying the regression coefficients shown in Table 1 (noting that age is centered at 70 and white women comprise the reference category in Table 1) to derive a predicted odds of dementia or composite memory score for each individual in the weighted sample in each year, based on that individual’s characteristics and test scores in the core interview. For example, a 70 year old white woman who did not complete a direct assessment but had an IQCODE score of 4 and a proxy memory score of 5 would have predicted odds of dementia of (noting the recentering of the variables): odds=exp(4.608+1.889 + 2.220*(4−5) + 1.274*(5−5)) = 72.02; probability=65.62/(1 + 65.62) = .99 By comparison, a 70 year old white woman who completed the direct assessment and had an immediate word recall score of 7, a delayed recall score of 5 and a TICS of 11 would have a predicted odds of dementia score of: odds=exp(4.608 + 1.889*1 + .933*7 − .266*7*7 − .797*5 − 1.075*11 + .043*11*11) < .001; probability = .001 Models are based on non-Hispanic participants in the 1995 Core HRS interview, sample weighted to be representative of the non-institutionalized US population born 1923 or earlier (average age 79 at baseline). By 2008, 1,418 individuals remained in the sample.

4. Discussion

We provide an approach to combining information from the direct and proxy HRS cognitive assessments into common scales that can be used in longitudinal studies of cognitive decline. Deriving predicted dementia/memory outcomes based on these methods, instead of restricting to HRS respondents with direct cognitive assessments, can reduce the potential for attrition bias in analyses of HRS data. We showed that rate of increase in dementia prevalence and rate of memory loss are underestimated when proxy reports are omitted.

Attrition associated with cognitive impairment has been identified as a major potential bias in studies of cognitive aging (24-27), so this approach could improve estimates of rate of cognitive change and determinants of cognitive risk factors. Users of HRS data can directly apply our approach by using the beta coefficients in Table 1 to compute the imputed dementia probability score or memory score for both direct and proxy respondents. Users of other studies may also apply our approach to developing their own scores. The parameters estimated in our models may apply in other samples, but this should be directly assessed in other studies and study-specific parameters may be necessary. Many studies besides HRS already include both proxy and direct cognitive assessments, e.g., the Cache County Study (28), some waves of the Cardiovascular Health Study (29), and the Second Longitudinal Study of Aging (30), but these are often not analyzed simultaneously because of uncertainty about how to calibrate the proxy measures against the direct assessments. Finally, in future longitudinal data collection efforts, the value of cognitive assessments may be substantially enhanced by collecting information from proxies and conducting comprehensive neuropsychological assessments on a small subsample of individuals to bridge the proxy and direct reports.

Our approach to predicting dementia produces a continuous probability score, rather than a dichotomous case status. The probability score can be dichtomized if needed or analyzed as a continuous variable ranging from zero to one. The continuous probability score may be a more appropriate representation of the emergence of dementia and uncertainty about case status in individuals with incipient cognitive difficulties. Because of the restricted range of the dementia probability score, regression models using logit or probit links are appropriate when this score is the dependent variable. The continuous score has advantages and disadvantages compared to the case-prediction algorithms such as that previously developed by Langa et al (5) where all proxy respondents were classified as “impaired”. A major advantage of Langa’s approach is its simplicity, and for many purposes, including estimating dementia prevalence in large population-based samples, this approach is probably sufficient and possibly preferable. For longitudinal regression models, our more complex approaches may reduce misclassification, improve statistical power, and improve sensitivity to small cognitive declines. Although estimated scores cannot be interpreted on an individual level, they may be valuable for longitudinal population research. It should be noted that although our method of retaining severely impaired respondents in the analysis may reduce bias due to selective attrition, it is unlikely to eliminate bias. Individuals with neither direct nor proxy assessments are unlikely to be entirely at random, and selective survival may also bias effect estimates.

Our approach has some limitations. Applying scoring rules from ADAMS sample to HRS core interviews from 1995 to 2008 assumes the relationship between dementia/memory outcomes and predictors is invariant across waves. The invariance assumption may be tested in studies where the linking component (such as ADAMS in our case) is conducted in multiple waves. Further, in analysis using the imputed scores containing ADAMS participants, standard errors of parameter estimates may be inflated. A resampling technique such as the Jackknife (31) may be used to adjust for standard errors.

The problem faced in HRS is common in longitudinal aging studies: the very people who do not participate in cognitive assessments may be those experiencing the greatest cognitive decline. Our approaches provide a direct solution to researchers using similar samples of HRS respondents (i.e., non-Hispanics aged 70+), and a general method for users of other similar studies. The addition of comprehensive neuropsychological assessments on a small subsample of cohort members with repeated direct and proxy cognitive assessments provides an opportunity to substantially improve longitudinal analyses of cognitive aging.