Reply to Sharp et al.: Psychological targeting produces robust effects

Sharp et al. (1) suggest that our original findings refute rather than support the effectiveness of psychological targeting (2, 3). We respectfully disagree for the following reasons.

Sharp et al. (1) state that “If psychological targeting worked reliably, it should have been 100% effective in all five experiments” and that “by random chance, it would have been effective in 50% of them in two or three of the five experiments.” These arguments misunderstand the principles of statistical inference under which hypothesis tests operate. First, experiments do not require a 100% success rate to generate reliable effects. University degrees produce higher levels of income (4), but that does not mean that 100% of people with a university degree earn more than those without. Second, the chance of rejecting the null hypothesis is not 50%, but rather depends on the confidence level of the statistical tests performed. In the case of our study, the chance of rejecting the null hypothesis in five out of six tests (interaction effects on clicks and conversions in studies 1 and 2, main effects on clicks and conversions in study 3) at a 5% α-level is, in fact, as low as 1.78 × 10⁻⁶ [calculated as $\frac{a^m\times {\left(1-a\right)}^{n-m}\times n!}{m!\left(n-m\right)!}$ with a = α-level, n = number of effects tested, and m = number of significant effects (5)]. As our significance levels were much lower in most cases, this represents a conservative estimate. Finally, there was no single case in which the effect went against our hypothesis.

Sharp et al. (1) continue by stating that “psychological targeting was effective in only two [out of five] of the experiments.” This statement misrepresents the original results because (i) the number of significant effects on clicks, defined by Sharp et al. (1) as separate main effects for each target group, were three, not two (only study 1 produced null results on clicks); (ii) we studied interactions, not main effects, in studies 1 and 2; and (iii) it selectively focuses on click-through rather than conversion rates. The authors justify this focus by writing that “conversions [. . .] occurred after click-through, and so are explained by self-selection effects out of the control of the experimenters.” This proposition is misguided. First, there is no reason why self-selection effects (other than the effects of our experimental manipulation) should have resulted in conversion outcomes that are perfectly aligned with our hypothesis. All interaction results on conversions support the effectiveness of psychological targeting. What else could have caused these interactions? Second, the ultimate success metric for businesses is not clicks but conversions, making the authors’ focus on clicks questionable not only from a research perspective but also from an applied marketing perspective.

Finally, Sharp et al. (1) identify the “failure to rule out differences in creative quality” as a main problem of our original results. Ad creatives matter, no doubt. This is why we controlled for the main effects of ad creatives in all our analyses, finding the effects of psychological targeting to be robust. Compared with pretesting ads for overall quality as suggested by Sharp et al. (1) (it is virtually impossible to control for all ways in which ads differ from one another), controlling for main effects of ad creatives provides a more rigorous approach to factoring out their influence.