Questionnaires and task-based measures assess different aspects of self-regulation: Both are needed

While Enkavi et al.’s (1) examination of the reliability of self-regulation dependent variables (DVs) from online assessments is an important addition to the field, their conclusion that “survey DVs are more appropriate for individual differences analyses [than behavioral tasks]” (p. 5476) is likely overstated. Existing research indicates that task-based constructs of self-regulation 1) are highly reliable at the latent variable level and 2) predict variance in real-world behavior that is 3) independent of variance predicted by questionnaires. Thus, both questionnaires and task-based measures are important to understand self-regulation and associated behavioral outcomes.

First, the psychological constructs assessed by self-regulation questionnaires and behavioral tasks are not interchangeable. Enkavi et al.'s (1) finding of low associations between self-report and task-based measures of self-regulation is consistent with prior reports of their separability (2–5). Several explanations have been proposed for this lack of convergent validity, including low reliability, low validity of self-reports or tasks (6), and differences in the psychological constructs they measure (4, 5). While these explanations are not mutually exclusive, the last possibility is supported by work demonstrating that this separability is evident 1) when using highly reliable latent variables (factors that extract shared variance across multiple tasks) for both self-report and task constructs, 2) at the genetic level, and 3) in self-report and task-based measures’ abilities to predict independent portions of variability in real-world outcomes ranging from psychopathology-related behaviors to academic performance (2–5).

Second, task-based constructs of self-regulation, particularly executive functions, have high stability when examined at the level of latent variables. In contrast, individual tasks, as emphasized by Enkavi et al. (1), tend to have low reliability and suffer from task impurity, issues which have been discussed for quite some time in the literature on individual differences in executive functions (see ref. 7). Task impurity arises because executive function tasks require control of lower-level processes such as sensorimotor functions, so performance on an individual task reflects variation in these processes as well as executive processes of interest. While difference scores between conditions (e.g., incongruent vs. congruent trials) aim to “subtract out” these lower-level processes, reducing impurity, they are not a panacea: Even when they are reliable, difference scores show low correlations across tasks (e.g., ref. 8). Fortunately, administering tasks in relatively controlled settings and using latent variables or factor scores considerably alleviates these problems. For example, latent variables for executive functions show high stability, with test−retest correlations ranging from 0.86 to 1.0 across a 6-y interval from late adolescence to early adulthood (8) and from 0.97 to 0.98 across a 6-y interval in middle age (9). Although Enkavi et al. acknowledge the increased reliability of factor scores, they do not discuss the literature documenting associations with real-world behavior at the latent and task levels (e.g., refs. 3, 7, and 10).

Since both self-reports and tasks can yield reliable measures of individual differences that independently predict behavior, both are relevant and valuable as measures of self-regulation. In light of this evidence, Enkavi et al.’s (1) conclusion is too hasty. Abandoning task-based measures would considerably slow scientific progress.