Attrition in longitudinal data is primarily selective with respect to level rather than rate of change

Abstract

Objectives:

An important question in longitudinal research is whether the individuals who discontinue participation differ in their level of, or their change in, cognitive functioning relative to individuals who return for subsequent occasions.

Methods:

Performance in five cognitive domains was examined in nearly 5,000 participants between 18 and 85 years of age who completed between one and five longitudinal occasions.

Results:

Little or no differences in cognitive performance were apparent between young adults who did or did not return for subsequent longitudinal occasions. However, among adults above about 45 years of age, returning participants had higher levels of cognitive performance, but approximately similar magnitude of longitudinal change, as participants completing fewer occasions.

Conclusions:

These results suggest that generalizability of longitudinal comparisons may be restricted to individuals with relatively high levels of cognitive functioning, but that rates of cognitive change are nearly comparable for individuals completing different numbers of longitudinal occasions.

Longitudinal contrasts are essential for providing information about within-person change. However, both the generalizability and the validity of longitudinal comparisons can be compromised if the individuals who return for subsequent occasions differ in meaningful ways from the individuals who do not return. In fact, there are numerous reports that attrition in longitudinal studies is selective, usually in the direction of higher levels of functioning in participants who continue in the study compared to those who do not continue (e.g., Chatfield, et al., 2005; Rabbitt et al., 2008; Riegel et al., 1968; Salthouse, 2014a; Schaie et al., 1973; Van Beijsterveldt, et al., 2002). Although the relation of attrition on level of cognitive functioning is well established, less is known about the relation of attrition on change in cognitive functioning. A few studies have reported more negative cognitive change in participants who discontinued participation compared to participants who continued (e.g., Cooney et al., 1988; Siegler & Botwinick, 1979), but inconsistent results have also been reported (e.g., Kennison & Zelinski, 2005).

The lack of definitive information about the relation of attrition on cognitive change is unfortunate because the implications of attrition could be quite different depending on whether the effects are primarily on the level of cognitive functioning, or on the rate of change in functioning. That is, if the people who drop out of a longitudinal study have lower levels of cognitive functioning than people who continue, generalizability of the results may be limited to high-functioning individuals, but interpretations of the longitudinal age trends would not necessarily be affected. In contrast, inferences about the direction and magnitude of longitudinal change could be distorted if the individuals who drop out of a longitudinal study have more negative cognitive change prior to dropping out than the individuals who continue. Specifically, cognitive decline might be underestimated in longitudinal comparisons if the returning participants have less negative change than the participants who drop out.

A major goal of the current project was to investigate the relation between both level and change in cognitive functioning and the number of longitudinal occasions completed by the participants. In keeping with prior research, participants who returned for subsequent longitudinal occasions were expected to have higher levels of cognitive functioning on the first occasion than participants who did not return. Change in cognitive functioning was assessed in terms of slopes relating cognitive performance to the number of completed occasions. If attrition is selective with respect to cognitive change, these slopes would be expected to be more negative in participants who completed fewer occasions. In contrast, no relation between the slopes and number of completed occasions would be expected if attrition was unrelated to cognitive change.

The data for the current project were based on an on-going longitudinal study that began in 2001 (Salthouse, 2014b; Salthouse et al., 2008). New participants were recruited between 2001 and 2015, and prior participants were invited to return after intervals averaging about three years. Factors associated with the attrition from the first to the second occasion were discussed in Salthouse (2014a).

Method

Sample

Participants were recruited with newspaper advertisements, flyers, and referrals from other participants. The first participants were recruited in 2001, with new samples of participants recruited nearly every year through 2015. Returning participants were tested between 2004 and 2017, and were invited to return after an average of about 3 years from the previous occasion. The research was conducted in compliance with the Helsinki Declaration, and was approved by the University of Virginia Institutional Review Board for Social and Behavioral Sciences (Protocol # 2009–0108-00).

Table 1 contains characteristics of participants in three age groups (young: ages 18 to 44, middle: ages 45 to 64, and old: ages 65 to 85), who had completed between one and five longitudinal occasions. It is important to emphasize that there was no overlap in participants across groups, as none of the participants with only one occasion were also in the group with two occasions, none of the participants with either one or two occasions were included in the group with three occasions, etc.

Table 1:

Demographic characteristics of participants in three age groups who completed different numbers of longitudinal occasions

	Number of Occasions
	1	2	3	4	5	β	F / eta2
Age 18–44 (Young)
Number	1002	397	167	133	62	NA	NA
Prop. Female	.60	.67	.65	.71	.69	.07*	2.89 / .007
Years Educ.	15.1	15.1	15.2	15.0	14.8	−.01	0.36 / .001
T1 Age	28.7	30.3	32.5	33.2	34.7	.22*	21.96* / .048
T1 Health	2.1	2.1	2.2	2.1	2.1	.05	1.54 / .004
T1 MMSE	28.7	28.5	28.4	28.8	28.1	−.05	2.75 / .008
T1 Est. IQ	108.4	108.6	107.3	108.7	103.2	−.04	1.40 / .004
T1-Tn int.	NA	3.4	6.5	9.4	10.0	.73*	317.62* / .558
Age 45–64 (Middle)
Number	825	464	338	313	138	NA	NA
Prop. Female	.69	.69	.70	.74	.71	.03	0.97 / .002
Years Educ.	15.7	15.4	16.0	16.0	16.3	.07*	4.65* / .009
T1 Age	54.7	54.6	54.9	54.8	54.7	.00	0.16 / .000
T1 Health	2.2	2.1	2.2	2.1	2.0	−.06*	2.69 / .005
T1 MMSE	28.2	28.3	28.5	28.6	28.8	.09*	3.70* / .008
T1 Est. IQ	108.1	108.2	112.2	112.8	113.8	.14*	10.50* / .023
T1-Tn int.	NA	3.6	6.2	8.7	10.6	.77*	630.51* / .590
Age 65–85 (Old)
Number	454	318	180	124	70	NA	NA
Prop. Female	.58	.59	.57	.61	.64	.02	0.41 / .001
Years Educ.	15.8	16.0	16.1	16.6	16.3	.08*	2.13 / .007
T1 Age	74.0	73.6	73.0	71.0	69.1	−.23*	17.78* / .059
T1 Health	2.4	2.4	2.2	2.2	2.3	−.09*	2.72 / .009
T1 MMSE	27.7	28.1	28.5	28.5	29.0	.19*	9.40* / .038
T1 Est. IQ	106.5	109.5	111.5	112.6	115.8	.20*	9.90* / .041
T1-Tn int.	NA	3.1	5.7	8.0	10.4	.83*	501.10* / .686

Note:

^*p<.01.

Health is a self rating on a scale from 1 for “excellent” to 5 for “poor”. MMSE is the Mini-Mental State Exam (Folstein et al., 1975). Estimated IQ was based on a regression equation predicting full-scale IQ from three cognitive tests (Salthouse, 2014). T1-Tn int. is the number of years between the first (T1) and the last (nth) occasion. NA indicates that the estimate was not available. The column labeled β contains the standardized regression coefficient for the number of occasions effect in a regression analysis, and the column labeled F / eta² contains the results of the analysis of variance treating number of occasions as a categorical variable.

In the young group participants who had completed more occasions were older than those who had completed fewer occasions, but the reverse was the case in the old group. In the middle and old groups the participants completing more occasions had more years of education, higher estimated IQ scores, and among the old participants, higher MMSE scores, than participants completing fewer occasions. However, there were no differences in these characteristics across participants with different numbers of occasions in the young group. Although not reported in the table, the intervals between successive occasions (e.g., T1 to T2, T2 to T3, etc.) were shorter among participants completing more occasions.

Tests

A total of 16 cognitive tests, representing five cognitive domains, were administered in the same order to all participants. Episodic memory was represented by word recall, paired associates, and story memory tests. Perceptual speed ability was represented by a digit symbol substitution test, and pattern comparison and letter comparison tests. Reasoning was represented by a matrix reasoning test, a letter sets test, and a series completion test. Spatial visualization (space) ability was represented by a spatial relations test, a paper folding test, and a form boards test. Finally, vocabulary was represented by a provide-the-definition test, a picture naming test, and multiple-choice synonym and antonym tests. Details of the tests, including reliabilities and results of confirmatory factor analyses supporting the hypothesized ability structure, are reported in other publications (e.g., Salthouse, 2010; Salthouse et al., 2008).

All scores were converted to z-scores based on the mean and standard deviation of the distribution of scores on the first occasion completed by all participants. Composite scores were then created by averaging z-scores for the tests representing each cognitive domain. It is unlikely that the range of scores was restricted by measurement floor or ceiling effects because the means and standard deviations of the raw scores were not close to the minimum or maximum possible, and the aggregation of three or more separate scores into composite scores also served to minimize measurement artifacts.

Results

Figures 1 and 2 portray mean composite cognitive scores as a function of number of occasions in the three age groups. The memory scores are portrayed in Figure 1, and the speed, reasoning, space, and vocabulary scores are portrayed in the four panels of Figure 2. Within each cognitive domain there was relatively small variation in cognitive performance across number of occasions in the young group, moderately large variation in the old group, and an intermediate level of variation in the middle group. Furthermore, the relations between cognitive performance and number of occasions were generally positive in the young group, negative in the old group, and with the exception of speed, nearly flat in the middle group. Prior research comparing participants tested either once or twice. and comparing participants from the same birth cohort tested at different ages in different years, suggests that the positive relations in the young group are likely attributable to practice, or test experience, effects (Salthouse, 2010; 2018).

Figure 1 –

Mean composite z-score (and standard errors) for memory ability in adults in three age groups who had completed between 1 and 5 occasions. Successive occasions are separated by arbitrary units and do not correspond to actual intervals.

Figure 2 –

Mean composite z-scores (and standard errors) for speed, reasoning, space, and vocabulary in adults in three age groups who had completed between 1 and 5 occasions. Successive occasions are separated by arbitrary units and do not correspond to actual intervals.

The first set of analyses investigated the relation between scores on the first occasion and the number of completed occasions. In one of the analyses number of occasions was treated as a continuous variable and entered as a predictor of composite score in linear regression analyses. Because participants completing different numbers of occasions may have differed qualitatively and not merely quantitatively, analyses of variance were also conducted in which number of occasions was treated as a categorical variable. In order to investigate the role of demographic measures, in both types of analyses the relations between number of occasions and T1 composite scores were examined with and without demographic measures as covariates.

Inspection of the entries in Table 2 reveals that there were significant relations between number of completed occasions and T1 composite scores in the middle and old groups when no covariates were included in the analyses. However, the effects were much smaller, and sometimes not significant, when demographic variables were controlled. This pattern of results implies that large proportions of the relations between number of completed occasions and T1 composite scores were general, and associated with the controlled demographic variables, and not specific to particular cognitive domains.

Table 2:

Results of regression analyses (β) and analyses of variance (F and eta²) on first-occasion (T1) composite z-scores associated with number of completed occasions

	β		F / eta2
	No Cov.	Cov.	No Cov.	Cov.
Cognitive Domain
Age Group
Memory
Young	−.02	.03	0.46 / .001	2.00 / .006
Middle	.16*	.04	14.56* / .026	1.54 / .004
Old	.26*	.10*	21.48* / .070	4.39* / .019
Speed
Young	−.04	.04	1.80 / .004	1.96 / .006
Middle	.15*	.05	13.19* / .024	2.30 / .005
Old	.23*	.04	16.88* / .056	2.09 / .009
Reasoning
Young	−.07*	−.01	2.76 / .006	1.42 / .004
Middle	.17*	.03	15.94* / .029	2.16 / .005
Old	.22*	.03	14.58* / .050	1.15 / .005
Space
Young	−.10*	−.00	4.40* / .010	0.37 / .001
Middle	.15*	.03	13.45* / .025	1.45 / .003
Old	.21*	.07*	13.10* / .045	2.97 / .013
Vocabulary
Young	−.00	.02	0.64 / .001	1.23 / .004
Middle	.19*	.04*	21.76* / .039	2.37 / .005
Old	.21*	.04	13.10* / .044	1.29 / .006

Note:

^*p<.01.

Entries in the columns labeled No Cov. are from analyses with no covariates, and those in the columns labeled Cov. are from analyses with age, sex, self-rated health, years of education, and estimated IQ as covariates.

Mean slopes, based on least squares regression equations relating the composite cognitive score to number of occasions carried out for each individual participant, are summarized in Table 3. The slopes are based on the T1 and T2 scores in participants completing only two occasions, on the T1, T2, and T3 scores in participants completing only three occasions, etc. It can be seen that nearly all of the slopes were positive in the young adult group, negative in the old group, and close to zero in the middle group.

Table 3:

Slopes of change in composite z-score units per occasion in participants in three age groups with different numbers of completed occasions and results of regression analyses and analyses of variance.

	Number of Occasions
	2	3	4	5	β	F / eta2
Cognitive Domain
Age Group
Memory
Young	.14*	.07*	.07*	.04	−.07	1.81 / .009
Middle	.02	.04	.02	.02	.00	0.47 / .001
Old	−.22*	−.12*	−.13*	−.04	.11*	2.66 / .014
Speed
Young	.08*	.02	.05*	−.00	−.08	2.24 / .011
Middle	−.06*	−.04*	−.03*	−.05*	.01	0.29 / .001
Old	−.14*	−.09*	−.08*	−.07*	.04	0.45 / .002
Reasoning
Young	.11*	.09*	.06*	.05	−.05	0.77 / .004
Middle	−.02	.03	−.00	−.00	.00	1.18 / .003
Old	−.10*	−.07*	−.08*	−.05*	.06	0.79 / .004
Space
Young	.18*	.11*	.10*	.07*	−.11*	3.15 / .016
Middle	.06*	.05*	.03*	.03*	−.06	1.30 / .004
Old	−.03	−.04	−.03*	−.03	−.00	0.36 / .002
Vocabulary
Young	.09*	.08*	.07*	.09*	.01	0.18 / .001
Middle	.01	.00	.00	.00	.02	0.11 / .000
Old	−.14*	−.07*	−.05*	−.05*	.11	2.61 / .014

Note:

^*p<.01.

Covariates in the regression analyses and in the analyses of variance were the relevant composite score on the first occasion, age, sex, self-rated health, years of education, and estimated IQ.

Because the slope results were similar with and without covariates, only the regression and analysis of variance results with covariates, including T1 score to control for initial level of performance, less negative slopes with more occasions for vocabulary in the old group. In each of these cases the effects were most pronounced in participants completing only two occasions, and the standardized regression coefficients were not significantly different from zero in participants completing three or more occasions. In addition, the slope was small for memory in older participants completing five occasions, but as just noted, the regression analysis comparing slopes in participants completing three, four, or five occasions was not significant.

Discussion

A finding apparent in both figures and in each table was that the patterns of attrition were different at different ages. That is, relatively little selective attrition was evident in adults under 45 years of age, and when it occurred it was in the direction of lower, rather than higher, levels of functioning for participants who returned for subsequent occasions. However, there was clear selective attrition in adults over 65, with higher levels of performance in participants who returned for more subsequent occasions. Selectivity of attrition was intermediate between the young and old groups for adults between 45 and 64 years of age. Smaller selectivity of attrition at younger ages has also been reported by Riegel et al. (1968) and Schaie et al. (1973), and in a subset of the current participants by Salthouse (2014a).

Although selectivity of attrition varied with age, within the older adults it was evident in all cognitive domains in Table 2, and with the estimated IQ and MMSE measures in Table 1. Findings from earlier studies suggested that attrition varied across cognitive abilities (e.g., Cooney et al., 1988; Riegel et al., 1968; Siegler & Botwinick, 1979), but there was no evidence in the current study that selective attrition was restricted to a particular type of cognitive functioning.

As expected on the basis of prior research, the results in Figures 1 and 2 and in Table 2 indicate that performance on the initial occasion was higher for middle-aged and older participants who returned for subsequent occasions than for participants who did not return. Furthermore, there were nearly monotonic relations between performance on the initial occasion and the number of subsequent occasions completed. This finding, which has also been reported by Siegler and Botwinick (1979) and Siegler et al. (1982), indicates that one’s initial level of cognitive ability is an important predictor of how long an individual will remain in a longitudinal study.

A key question in the current study was whether individuals completing different numbers of occasions had different rates of cognitive change. The approximately parallel functions in Figures 1 and 2, and the small effect sizes in both the regression analyses and analyses of variance in Table 3, after controlling for first occasion performance, indicate that the patterns of cognitive change were fairly similar in participants who completed between two and five longitudinal occasions. Although the slopes in memory and vocabulary were slightly more negative for older adults who dropped out after two occasions than for older adults who completed three or more occasions, and were smaller in memory for older adults who completed five occasions, there were no significant slope differences among participants completing between three and five occasions. Most of the attrition effects can therefore be inferred to be on the level of functioning and not on rate of change.

The current results focusing on change across two or more occasions extend those of an earlier study (Salthouse, 2014a) comparing performance of participants who completed either one or two longitudinal occasions. Longitudinal changes in the prior study were examined in returning participants who had the same initial levels of performance as the non-returning participants, and between returning participants and non-returning participants whose second occasion scores were imputed based on their first occasion scores, and the first and second occasion scores of the returning participants. The major findings were that similar magnitudes of cognitive change were evident with both procedures. These results were interpreted as suggesting that selective attrition from the first to the second occasion is primarily related to the level, rather than the change, in cognitive functioning, which is consistent with the interpretation of the current results.

Several limitations of the study should be acknowledged. First, most of the participants reported themselves to be in very good or excellent health, and were high functioning in terms of estimated IQ and MMSE, which could restrict generalizability of the results. Second, the analyses of rates of change were relatively crude because they were based on comparisons of different people with each number of occasions. Third, in addition to level of performance, the participants completing different numbers of occasions may have differed in other respects that could have influenced the results. And fourth, the attrition effects in this study may have been underestimated if some participants might have subsequently returned for additional occasions. However, it is important to note that the pattern of results was very similar in two additional analyses. One involved participants initially tested no later than 2008, and who thus had at least nine years to have participated in additional assessments. The second supplementary analysis involved participants whose last occasion was 2014, and who therefore had three or more years to have participated again. The standardized regression coefficients in these analyses closely resembled those in Table 3 based on the complete sample of participants, which suggests that the results were not distorted by limited opportunities to have returned for additional assessments. Despite the limitations, the study also has important strengths, such as moderately large samples of adults across a wide age range who completed between one and five longitudinal occasions spanning an interval averaging over 10 years, multiple cognitive tests representing five different cognitive domains, and a consistent pattern of results across two different analytical methods.

To summarize, the results of this study confirm earlier findings that at least among adults over about 45 years of age, individuals who drop out of a longitudinal study tend to have lower levels of performance in several different types of cognitive functioning. Moreover, the differences on the initial occasion were monotonically related to the number of longitudinal occasions subsequently completed, which suggests that continued participation in a longitudinal study is partially influenced by one’s initial level of cognitive functioning. However, with the exception of more negative change in memory and vocabulary from the first to the second occasion among older participants completing only two occasions, the rates of longitudinal change were not significantly different across participants completing different numbers of longitudinal occasions. Selective attrition therefore appears to threaten the generalizability of longitudinal comparisons more than the validity of the age-cognition relations derived from those comparisons.