A squared standard error is not a measure of individual differences

Fisher et al. (1) investigate the congruence between intraindividual correlations and cross-sectional correlations using six empirical datasets. While others have emphasized that there is no mathematical law dictating that these correlations should be the same (2–4), empirical studies are imperative to determine whether or not they differ in practice. Therefore, we greatly appreciate this timely and valuable contribution.

However, Fisher et al. (1) also raise a concern regarding the variability in correlations, which we believe is ill informed. Specifically, they state, “the variance around the expected value was two to four times larger within individuals than within groups. This suggests that literatures in social and medical sciences may overestimate the accuracy of aggregated statistical estimates.” We argue that this comparison is fundamentally flawed, because the two variances they compare are designed to represent inherently different phenomena.

The first variance they consider is based on estimating a correlation per person (over time), and then taking the variance of this intraindividual correlation across persons (right-hand margin in Fig. 1). This variance can be considered an estimate of a random effect, representing true individual differences in the intraindividual correlation.

Fig. 1.

Each cell in the data matrix contains a bivariate observation for person i at occasion t, with i = 1, …, N and t = 1, …, T. The right-hand margin contains intraindividual correlations per person, r_i.. The variance of this consists of (i) the variance of the true intraindividual correlations, ρ_i. (i.e., random effect), and (ii) the variance of sampling error, r_i.−ρ_i.. The latter goes to zero as the number of occasions, T, increases. The bottom margin contains the cross-sectional correlation estimates per occasion, r_.t. Under the assumption of stationarity (ρ_.t = ρ_..), the variance of this consists of sampling error variance only and will go to zero as the number of individuals, N, increases.

The second variance they consider is based on estimating the correlation per occasion (over persons), and then computing the variance of this cross-sectional correlation across occasions (bottom margin in Fig. 1). This latter variance can be thought of as a proxy of the squared SE of the cross-sectional correlation estimate. To see this, recall that a SE reflects the variability in parameter estimates when drawing independent samples of the same size, from the same population, infinitely many times (5). Hence, even though the samples used here are not independent (i.e., they consist of the same individuals at different occasions), the second variance the authors compute is clearly based on the same logic that underlies the squared SE of the cross-sectional correlation estimate.

Two comments can thus be made about comparing these variances. First, a SE is not supposed to measure variability across individuals; rather, it is a measure of variability in the parameter estimate from sample to sample. Second, the SE of a correlation estimate is a function of the correlation and the sample size (6) and will go toward zero as the number of individuals increases; in contrast, the variability of the intraindividual correlation is a function of both sampling variance (which will go to zero as the number of time points increases) and the random effect (which is independent of the number of individuals and time points, see Fig. 2).

Fig. 2.

The effect of sample size (number of persons, N, and number of time points, T) on the distributions of (i) intraindividual correlation estimates, r_i. (pink), and (ii) cross-sectional correlation estimates, r_.t (blue). For simplicity, the cross-sectional correlation and the average within-person correlation are both 0.5; the dashed density represents the random effect (i.e., true individual differences in intraindividual correlation). As sample size increases, sampling variance goes to 0 so that (i) the pink distribution coincides with the true dashed density, and (ii) the blue distribution becomes increasingly narrow. The var(r_i.)/var(r_.t) ratio changes from 2.62 to 5.21 to 17.

To be clear, the authors emphasize that the two variances they compare are not the same. However, we argue that this is not surprising at all: A squared SE is not an estimate of a random effect and should not be interpreted as meaning to reflect such. Hence, the take-home message is not that we may be overestimating the accuracy of aggregated statistical estimates, but that we should interpret measures in terms of what they are meant to represent.