Assessing the reliability of within-person change over time: A dynamic factor analysis approach

Researchers have long been interested in the reliability of change, but to date there have been few methods for assessing reliability of change in the context of daily diary designs. Cranford and colleagues (2006) proposed a method to estimate the reliability of change using Generalizability Theory (GT), but their approach assumes that the loadings and error variances of individual items are constant across both individuals and items. We present a new method for assessing reliability in the daily diary context that does not make such assumptions. Our method uses results from dynamic factor analysis (DFA) to compute McDonald’s (1999) omega for each subject. A simulation study indicated that the proposed method provides accurate estimates of the reliability of within-person change across a range of loading/error combinations. When the equal item loadings assumption of GT was violated, the DFA method provided accurate estimates whereas the GT method underestimated the true reliability. We illustrated the methods with empirical data from a daily diary study of 221 individuals who were preparing for a stressful examination. Individuals rated their moods twice a day on 5 subscales over 44 days. In this case the DFA method and the GT method yielded similar estimates; however, the estimates from the DFA method were slightly larger. A further advantage of the DFA method is that it revealed the autoregressive nature of the repeated measurements, which in the empirical example involved lag-1 and lag-2 effects. The autoregressive component of the latent factor variance was substantial, ranging from 21% of the variability in anger ratings to 50% of the variability in anxiety ratings. On the individual level, persons who had larger systematic autoregressive variability over time were those with higher reliability estimates.