Reply to Friedman and Banich: Right measures for the research question

In their Letter to the Editor, Friedman and Banich (1) suggest we (2) “overstate” the higher suitability of dependent variables (DVs) derived from surveys for individual difference analyses. We appreciate this opportunity for a continued discussion regarding the measurement of self-regulation. However, their critiques (1) do not provide evidence against the higher reliability of survey DVs, which is central to our conclusions. Instead, they levy criticisms addressed in another of our papers (3).

For instance, the authors (1) argue that task DVs could provide value for individual difference analyses via latent variables. Indeed, we confirmed latent variables underlying task DVs are more reliable compared to individual task DVs (2, 3). Reliabilities of individual DVs, on the other hand, are comparably low in our data and the findings listed in their letter: In ref. 4, they range from 0.21 to 0.56; in ref. 5, they range from 0.29 to 0.76; and, in our battery, they range from 0.24 to 0.72. Notably, neither ref. 4 nor ref. 5 mentions the reliability of survey DVs in comparison. This suggests that task DVs are only viable for individual difference analyses when combined through latent variable modeling, unlike survey DVs which are sufficiently reliable in isolation, at least for the probed self-regulation measures. Critically, numerous studies in the literature report associations between single-task DVs and other individual characteristics instead of collecting multiple measures necessary for latent variable modeling.

Friedman and Banich (1) also argue another point that our papers confirm: Self-report surveys and cognitive tasks measure separate constructs (3). While this implies that tasks cannot be replaced by surveys, it does not suggest that task DVs (or their underlying latent constructs) are equally useful as individual difference measures. In contrast to findings the authors list (6, 7), we find that ref. 3 survey DVs predict some variance in real-world outcomes, while task DVs predict little to none. While multiple reasons for this disparity are possible (see ref. 3 for a full discussion), surveys captured psychological constructs relevant for the measured outcomes, while tasks did not. Importantly, this conclusion remains unchanged when predicting behavioral outcomes using scores derived from latent task constructs, contrary to Friedman and Banich’s (1) suggestion. Thus, even while latent task constructs are stable across individuals, they still are inadequate for understanding a range of outcomes of scientific and societal interest.

Being primarily cognitive neuroscientists ourselves, we have no intention of abandoning task measures completely. Yet, our analyses are a sober reminder of the importance of choosing the right measure for the right research question. As we explain with the example of the Stroop effect in our paper and as ref. 8 details as well, task DVs are designed to minimize between-subject variance, which limits their usefulness for individual difference analyses. Through its open availability, our work can serve as a test bed for researchers to optimize tasks DVs for individual difference analyses. As it currently stands, however, our conclusion is neither “overstated” for survey DVs nor dooming for task DVs. It is comparative, realistic, nuanced, and cautions against common practices in the field.