Evidence that publication bias contaminated studies relating social class and unethical behavior

Piff et al. (1) presented provocative evidence across seven experiments that people of a higher social class were more likely to engage in unethical behavior. A striking characteristic of the reported findings was the consistency of the results across different definitions of social class and measurements of unethical behavior. The multiple replications might appear to provide strong evidence for the claim, but the analysis here shows that they actually indicate the findings are unbelievable.

All the findings in the work of Piff et al. (1) are based on statistical analyses of behavior. Support for their claim was found by rejecting a null hypothesis that indicated no difference in unethical behavior as a function of social class. Because of random sampling, an experiment can only reject the null hypothesis with a probability, called power, that depends on the sample size(s), design, and effect size of the experiment. Repeated tests should reject the null hypothesis with a frequency that reflects the underlying power of the experiments. Table 1 shows the statistical properties of the key findings that related unethical behavior to social class, with each experiment’s test statistic converted to a standardized effect size called Hedges g. The effect size was then used to estimate the power of the experiment. Most of the power values are close to one half because these experiments just barely rejected the null hypothesis based on the standard criteria in experimental psychology (P < 0.05). The probability that all seven findings would reject the null hypothesis is the product of the power values (2, 3), which is only 0.02.

Table 1.

Statistical properties of the experiments in Piff et al. (1)

Sample size(s)	Effect size (g)	Observed power
274	0.120	0.51
152	0.166	0.53
105	0.199	0.52
64, 65	0.557	0.88
108	0.228	0.65
195	0.141	0.50
90	0.209	0.50

The low probability indicates that the findings are internally inconsistent. Given the small effect sizes estimated by the seven experiments, one would expect to reject the null hypothesis approximately four times (the sum of the power values) rather than the observed seven. The low probability of the experimental findings suggests that the data are contaminated with publication bias. Piff et al. (1) may have (perhaps unwittingly) run, but not reported, additional experiments that failed to reject the null hypothesis (the file drawer problem), or they may have run the experiments in a way that improperly increased the rejection rate of the null hypothesis (4). This conclusion would hold even if one imagined that the true effect sizes were 20% larger than what was reported.

Whatever its source, the presence of a publication bias means that the findings in Piff et al. (1) do not provide useful information about the claimed effect. It remains an open question whether unethical behavior is related to social class, and only new experiments that are free from bias will be able to address the issue. If the effect of social class on unethical behavior turns out to be real, then the findings of Piff et al. (1) almost surely overestimate its magnitude (5).