Côté et al. (1) provided evidence that economic inequality moderates the effect of income on generosity. In their study, individuals with higher household income were less generous in a dictator game than poorer individuals only if they resided in a US state with comparatively large economic inequality. We questioned this finding because we did not find any evidence for the postulated moderation effect of economic inequality across three studies (ref. 2; for similar replication failures see ref. 3). However, our studies were conceptual rather than direct replications as we used different measures of generosity (charitable donations, behavior in a trust game, and volunteering) and also included non-US samples.
Côté and Willer (4) subsequently conducted three additional studies to test whether the postulated moderation effect could at least be confirmed under the narrower circumstances that existed in the original study (i.e., assessment of generosity in a dictator game using a US sample). Two of the reported analyses indeed seemed to confirm the original finding. However, a closer look revealed that these two analyses deviated from the analysis used in the original study without further justification. In the Amazon Mechanical Turk 2016 Study, personal income was analyzed (rather than household income for which the moderation was not statistically significant); in the Human Cooperation Laboratory Study, participants were excluded if they failed a comprehension check (whereas no participants were excluded from the original analysis). Thus, it is not possible to rule out that these two analyses were optimized to find statistically significant evidence for the postulated moderation.
To settle this issue, Côté and Willer (4) conducted a preregistered direct replication study of Côté et al. (1) with sufficient statistical power. We wholeheartedly applaud them for this step. A clear commitment to a particular analysis before the data are observed addresses the concerns raised above and improves the interpretability and credibility of findings (5, 6). This is particularly important when researchers are attempting to replicate their own research, as the temptation to analyze the data until they “find” the anticipated effect (7) might be particularly strong. The results of this preregistered study were crystal clear: The original finding was not replicated, that is, state-level inequality did not moderate the effect of household income on generosity in a dictator game (P = 0.40). Furthermore, there were also no moderation effects for charitable donations (P = 0.73) or volunteering (P = 0.86), which confirmed our findings (2).
Case closed? Not quite. Côté and Willer (4) do not seem to be convinced by the results of their preregistered study because they finally conducted an internal meta-analysis of the original study, the nonpreregistered studies, and the preregistered study, which supposedly showed mixed evidence. However, internal meta-analyses can be completely invalidated by even very small doses of selective reporting (8). Because we cannot rule out selective reporting for the original and the nonpreregistered studies, we have to conclude that this internal meta-analysis is uninformative, especially when compared with preregistered replications (8–10). Thus, we have to conclude that there is no evidence that economic inequality moderates the effect of income on generosity.
Footnotes
The authors declare no competing interest.
References
- 1.Côté S., House J., Willer R., High economic inequality leads higher-income individuals to be less generous. Proc. Natl. Acad. Sci. U.S.A. 112, 15838–15843 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Schmukle S. C., Korndörfer M., Egloff B., No evidence that economic inequality moderates the effect of income on generosity. Proc. Natl. Acad. Sci. U.S.A. 116, 9790–9795 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.von Hermanni H., Tutić A., Does economic inequality moderate the effect of class on prosocial behavior? A large-scale test of a recent hypothesis by Côté et al. PLoS One 14, e0220723 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Côté S., Willer R., Replications provide mixed evidence that inequality moderates the association between income and generosity. Proc. Natl. Acad. Sci. U.S.A. 117, 8696–8697 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Nosek B. A., Ebersole C. R., DeHaven A. C., Mellor D. T., The preregistration revolution. Proc. Natl. Acad. Sci. U.S.A. 115, 2600–2606 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Lakens D., The value of preregistration for psychological science: A conceptual analysis. https://psyarxiv.com/jbh4w/ (18 November 2019).
- 7.Simmons J. P., Nelson L. D., Simonsohn U., False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). [DOI] [PubMed] [Google Scholar]
- 8.Vosgerau J., Simonsohn U., Nelson L. D., Simmons J. P., 99% impossible: A valid, or falsifiable, internal meta-analysis. J. Exp. Psychol. Gen. 148, 1628–1639 (2019). [DOI] [PubMed] [Google Scholar]
- 9.Kvarven A., Strømland E., Johannesson M., Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nat. Hum. Behav., 10.1038/s41562-019-0787-z (2019). [DOI] [PubMed] [Google Scholar]
- 10.van Elk M., et al. , Meta-analyses are no substitute for registered replications: A skeptical perspective on religious priming. Front. Psychol. 6, 1365 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]