Reply to van Hoorn: Pitfalls of narrow interpretations of significance

van Hoorn (1) argues that the conclusions reached in my report (2) are unwarranted because of pitfalls associated with (i) effect sizes, (ii) effect duration, and (iii) subject selection. None of these concerns should properly be labeled as pitfalls.

van Hoorn (1) uses terminology from Cohen (3) for standardized units to argue that the effects measured after 3 d are “small” or “negligible.” Importantly, these descriptions offered by Cohen are not absolute measures of substantive importance, only qualitative guidelines for a researcher estimating effect sizes for power analyses. An effect should not be labeled “small” without a consideration of the substantive topic at hand (4). Before judging the substantive significance of an attitude change, it should be ascertained how such a change may affect behaviors of interest, such as the treatment of immigrants or voting.

In my paper (2) I report effects that appear to wane between a 3-d and 10-d period; however, these differences should be interpreted cautiously given that in only one of three cases does the difference achieve even marginal levels of conventional statistical significance. However, if this is the true effect of extended exposure to treatment, a “pitfall” is surely the wrong label. If an effect decreases with prolonged contact, this is something that should be understood. Testing for the effects of over-time exposure was one of the primary purposes of the experimental design in my study (2). Only considering effects as significant if they reach some arbitrary level of duration would result in many important phenomena being treated as insignificant.

van Hoorn (1) is conflating representativeness with substantive significance. To understand substantively significant effects of contact with new social groups, a researcher should use subjects that have not yet been exposed to that group, rather than seeking a sample that is representative of a larger population. Additionally, the median proportion Hispanic in the Census Tracts used in my study (2) were within two percentage points of the median Census Tract in the United States, making the population both typical and also substantively important for understanding the real-world effects of immigration. Furthermore, the sample used in my study (2) counters van Hoorn’s (1) claim that these subjects likely hold above-average exclusionary attitudes: as reported in table 2 of ref. 2, the subjects were far more politically liberal than the general US population. The claim that stronger prior exclusionary attitudes are likely to strengthen an individual’s initial response is also undermined by the results reported in table 1 of ref. 2, where the exact opposite occurs: the attitudes with higher pretreatment exclusionary levels were actually less responsive to the treatment.

An important feature of the design of my study (2) is that it involved exposure to only two putative immigrants for perhaps only minutes per day in the midst of potential distractions from a busy train platform. As such, these results are likely smaller than the results that may be obtained from the nonexperimental process of immigration involving far more than two immigrants and more thorough exposure. The design of my study (2) can be considered an opportunity to more broadly test the effects of this important phenomenon.