Significance of attitudinal experiments

Experiments involving environmental manipulations targeting psychological traits, like attitudes or personality, are a powerful tool in social science. However, where the benefits offered by experimental research designs are typically well laid out, the pitfalls of comparing control groups and treatment groups receive scant attention, if recognized at all. The recent field experiment by Enos (1), showing that intergroup contact strengthens exclusionary attitudes, provides a showcase for discussing three main pitfalls. These pitfalls can be labeled as (i) effect sizes, (ii) effect duration, and (iii) subject selection, and have far-reaching implications for the real-life significance of results of attitudinal experiments, sufficient to warrant a different conclusion to the results presented by Enos (1).

The effect-sizes pitfall is most basic and concerns the potential triviality of the found differences between the control group and the treatment group. The (non)triviality of such differences can be assessed using standard effect-size measures, which come with established criteria (e.g., ref. 2). Cohen’s d, which considers differences in group means normalized by sample SDs, is most popular and has thresholds for small (d ≥ 0.20), moderate (d ≥ 0.50), strong (d ≥ 0.80), and negligible (d < 0.20) effects. Applying these criteria, the normalized mean differences reported by Enos (1) (0.330, 0.201, and 0.082) are small at best and not close to even moderate effects.

The effect-duration pitfall concerns the longevity of effects found, whether they are fleeting or enduring. Enos (1) reports that effect sizes of intergroup contact waned greatly over time, so that their proexclusionary effect was “considerably stronger” after 3 d of contact compared with 10 d of contact. Hence, although the exclusionary effect of intergroup contact was never strong to begin with, we may also question its significance for actual ethnically diverse communities, as these typically involve people living close to each other for prolonged periods.

What I call the “subject-selection” pitfall seems least recognized and concerns the possibility that the sample used for the analysis was unrepresentative in a way that inflated effect sizes. Enos (1) considered an unrepresentative sample of white subjects living in almost all-white suburban communities, making subject selection a matter of concern. The reasoning is as follows: Exclusionary attitudes manifest themselves to no small extent in residential segregation, meaning that people prefer to have neighbors of the same ethnicity. White subjects living in predominantly white communities, then, have already shown above-average exclusionary attitudes, which—in turn—is expected to affect their attitudinal response to intergroup contact. Specifically, stronger prior exclusionary attitudes are likely to strengthen individuals’ initial response to intergroup contact, thus inflating effects found compared with effects that would have been obtained in a population sample comprising subjects with representative prior exclusionary attitudes.

Enos (1) concludes that exposure to intergroup contact causes a strong exclusionary shift. However, given the above pitfalls, a more grounded conclusion is that even in a sample of segregated suburbanites, the exclusionary effect of intergroup contact was small to negligible and waned considerably with a modest prolongation of said contact. More generally, experimental trait research could benefit a lot from attending to the above pitfalls as a matter of course.