All data collection and analysis methods have limitations: Reply to ^and

Abstract

The commentaries on my article contain a number of points with which I disagree, but also several with which I agree. For example, I continue to believe that the existence of many cases in which between-person variability does not increase with age indicates that greater variance with increased age is not inevitable among healthy individuals up to about 80 years of age. I also do not believe that problems of causal inferences from correlational information are more severe in the cognitive neuroscience of aging than in other research areas; instead, I contend that neglect of these problems has led to confusion about neurobiological underpinnings of cognitive aging. I agree that researchers need to be cautious in extrapolating from cross-sectional to longitudinal relations, but I also note that even longitudinal data are limited with respect to their ability to support causal inferences.

I appreciate the thoughtful remarks of Rabbitt (2011) and Raz and Lindenberger (2011), and I welcome the opportunity to respond to them. Because of space limitations, I am not able to address all of the issues raised in the commentaries, but in the following I have attempted to respond to what appear to be the most important substantive issues on which we may disagree. The initial section of my target article (Salthouse, 2011) used results from two recent projects to illustrate major characteristics of cognitive aging. In both projects there were nearly linear age trends in the cross-sectional means and longitudinal changes in measures of cognitive functioning, but in each case the between-person variance was approximately constant across most of adulthood. Based on these findings, and others cited in the article, I stated that “age-related differences in mean performance can occur without concomitant increases in between-person variability” (p. XX) The relation of age to variability was not a major focus in the article, but it is relevant to the interpretation of moderation analyses because to the extent that variance systematically increases with increased age in healthy adults, stronger brain-cognition relations at older ages might be a statistical consequence of greater variance rather than a reflection of the emergence of new, or stronger, brain-cognition relations.

Both commentators objected to my suggestion that variability does not inevitably increase with age (Rabbitt, 2011; Raz & Lindenberger, 2011). I agree that an expectation of greater variance at older ages is intuitively plausible and that this finding has been reported in a number of studies. However, because there are cases where mean age-related declines occur without appreciable age-related increases in between-person variance, I suggested that the relation was not inevitable. In addition to the results from the two projects portrayed in the figures, several additional studies were cited, including cross-sectional data from the nationally representative samples used to establish the norms for various cognitive test batteries (described in Salthouse, 2010), and longitudinal data from several different studies. In all of the cited studies negative age trends were reported in measures of level or change in cognitive functioning, with small to non-existent age-related increase in variance. A similar pattern is also evident in scatterplots of various brain structure variables, as several studies cited in Salthouse (2011) contain figures indicating nearly linear negative age trends in the average value with little or no increase in variance in samples of healthy adults up to about 80 years of age. Among these are studies of total or regional brain volume (e.g., Abe, et al. 2008; Allen, et al. 2005; DeCarli, et al. 2005; Fotenos, et al. 2005; 2008; Good, et al. 2001; Kennedy et al. 2009; Lemaitre et al. 2010; Sowell, et al. 2003; Walhovd, et al., 2009; Zimmerman, et al. 2006), studies of cortical thickness (e.g., Ecker, et al. 2009; Lemaitre, et al. 2010; Salat, et al. 2004), and studies of white matter integrity based on diffusion tensor imaging (e.g., Abe, et al. 2008; Charlton, et al. 2006; Grieve, et al. 2007; Hsu, et al. 2008; Michielse, et al. 2010; Rovaris, et al. 2003; Salat, et al. 2005; Stadlbauer, et al. 2008; Sullivan & Pfefferbaum, 2006; Voineskos, et al. 2010; Westlye, et al. 2010). The research findings regarding age-related increases in variability are clearly mixed, perhaps in part because of the inclusion of individuals in early stages of dementia or in the period of terminal decline in some studies, but the existence of studies such as those cited above are consistent with the claim that age-related decreases in the means can occur without inevitable age-related increases in variance in samples of relatively healthy adults between about 20 and 80 years of age.

Raz and Lindenberger (2011) noted that the discussion in Salthouse (2011) emphasized linear age relations whereas age relations are sometimes nonlinear. I completely agree that nonlinear age relations can occur and that in some respects they are more interesting than linear trends. Unfortunately, nonlinear trends may not be detected when, as in many of the studies reviewed, the age range of the research participants is restricted, or only extreme groups of young and old adults are compared. Nevertheless, nonlinear age relations could be accommodated in most analytical methods, and none of the substantive points in the article depend on the specific form of the age relations.

The commentators had markedly different views about the usefulness of mediation-like procedures with cross-sectional data. Rabbitt (2011) noted several examples from his own work in which they were informative, as when control of the variance in measures of brain volume reduced the relation between measures of balance and of cognition. In contrast, Raz and Lindenberger (2011) argued strongly against mediation analyses of cross-sectional data, particularly when they are used to make inferences about longitudinal changes. I agree that cross-sectional mediation analyses can be misleading if they are used as a proxy for longitudinal relations, and many researchers have clearly interpreted them in this fashion. However, it is worth considering whether it is the analytical methods that are the problem, or it is the inferences about developmental phenomena based upon results of the methods.

Because the term mediation seems to imply change over time, in that if X mediates the Y-Z relation then X is often assumed to occur intermediate in the temporal sequence between Y and Z (Kraemer,et al. 2008), mediation analyses can invite inferences about temporal relations that may not be justified. However, mediation techniques can also be viewed as a type of variance partitioning, similar to other statistical control methods such as various forms of multiple regression and partial correlation, and these procedures can be useful even if none of the variables involve a temporal dimension. That is, a discovery that the relation between A and C is reduced when variance in B is controlled, but that there is little change in the A-B relation when variance in C is controlled, or in the B-C relation when variance in A is controlled, is likely to be informative even if all variables are static and have no temporal connotation. To illustrate, consider the interpretations of the preceding pattern if A referred to Attention Deficit / Hyperactivity Disorder (ADHD), B referred to Behavior problems, and C referred to Cooperation with peers. Rather than merely indicating that ADHD was associated with low levels of peer cooperation, a pattern such as that outlined above would be consistent with an interpretation that behavior problems may be largely responsible for the relation between ADHD and peer cooperation. Whether information of this type is relevant to the ultimate causes of the phenomenon could be debated, but it can often serve to distinguish among alternative interpretations of relations observed at a particular point in time. Just as we should resist inappropriate extrapolation of results beyond the situations to which they apply, we should also resist universal rejection of analytical procedures that can be informative when their limitations are recognized.

I very much agree with the general point that researchers need to be cautious when making inferences about longitudinal changes from cross-sectional differences. The distinction between difference and change is often ignored, even in the titles of articles in which the term “change” appears and all of the data are based on cross-sectional differences. It is therefore worth emphasizing that whether comparable patterns are evident in cross-sectional and longitudinal data is an empirical question, and not a logical necessity. In fact, two recent projects from my laboratory revealed one situation (with the Connections variable) in which speed and fluid cognitive abilities had similar patterns of correlations with both cross-sectional differences and longitudinal changes (Salthouse, 2011), and another situation (with the MMSE variable) in which reasoning and memory abilities had somewhat different patterns of correlations with cross-sectional differences and with longitudinal changes (Soubelet & Salthouse, in press).

Raz and Lindenberger (2011) suggested that multivariate longitudinal studies and experimental interventions are better alternatives than cross-sectional studies for reaching causal inferences. Although longitudinal data clearly contain more information than cross-sectional data because temporal information about change is included in addition to the cross-sectional information about differences, it is important to consider exactly how longitudinal data could be informative about causes. As I noted in the article, relations between the early change in one variable and the later change in another variable (i.e., lead-lag relations) can be more informative than simple correlations among the changes. However, lead-lag relations can be difficult to interpret in terms of causality without considerable knowledge (or strong assumptions) about the timing of critical events, including the onset of the change in the leading variable, the time until the leading change reaches a critical value, and the interval between the critical value in the leading variable and the onset of change in the lagged variable. I am not suggesting that these problems are insurmountable, but rather that at the current time we have very little information relevant to the timing of critical events in either brain or cognitive variables. Another limitation of much of the existing research is that in many of the studies all of the participants have been adults over about 60 years of age, and correlations among changes occurring during a period when both variables have been changing for years or decades are likely to be less informative about causal sequences than correlations obtained during a period when one or both variables are just beginning to change. I agree that longitudinal data are generally preferred over cross-sectional data, but I also believe that it is important to recognize that the mere existence of correlations among changes provides a limited basis for inferring causality because the relevant data are still observational, relations among changes after a prolonged period of change may not be informative about relations at earlier periods in time when the causal sequences were first initiated, and even lagged relations may reflect pre-existing characteristics of the individuals rather than direct causal influences.

Although for the reasons mentioned above I suspect that there are limits on how much can be learned about causes of aging from only studying older adults, I was puzzled by the implication that I believe that studying young adults would necessarily be informative about brain-cognition relations in aging, or that it is “a sufficient basis for understanding the brain basis of cognitive aging” (Raz & Lindenberger, 2011). My view is that brain-cognition relations might be fruitfully studied in young adults only if powerful tests of moderation failed to find different B-C relations at different ages. However, as I noted in the article, the existing results on moderation are weak and inconsistent, and therefore very little is currently known about the relation between age and B-C relations. It is unfortunate that so few studies with either cross-sectional or longitudinal data have included large samples of adults across a wide age range because moderation analyses could be very informative about the timing, and age-specificity, of brain-cognition relations.

In conclusion, I appreciate the time and efforts of the commentators because their remarks provided an opportunity for clarification of my position, and highlighting of differences in perspectives. The major points of the article were that despite much research examining interrelations of aging, measures of brain structure and measures of cognitive functioning, only tentative conclusions are currently possible regarding the neuroanatomical substrates of age-related cognitive decline. As I stated in the article, all data collection and analytical methods have limitations, and strong causal inferences are exceedingly difficult when the critical variable of age cannot be experimentally manipulated. Because no method is without limitations, my suggestion was that confidence in conclusions will likely be greatest when the results are found to converge across multiple methods involving different sets of assumptions and different types of data. If only cross-sectional data are available then some types of mediation analyses involving decomposition of covariances will often be more informative about the cross-sectional interrelations among the variables than simple correlations, particularly if alternative interpretations are considered. Longitudinal data are essential for drawing conclusions about changes occurring within the individual, but because they are observational rather than experimental, it is important to recognize that even longitudinal data are limited with respect to their ability to identify causes, and the limitation is probably even greater when all of the observations are obtained at some unknown point after the critical relations originated.

The commentators had quite different perspectives on what researchers in this area should do in the future, as Rabbitt (2011) suggested that the only option was to “Keep calm and carry on”, whereas Raz and Lindenberger (2011) argued that “no practical reason can justify the continuation of business as usual”. My view lies between those of the commentators, as I feel that greater progress can be achieved if researchers were to recognize potential limitations of all data collection and analysis methods, and attempt to base conclusions on results from multiple methods of data collection and analysis whenever possible.