Reply to Maier et al., Szaszi et al., and Bakdash and Marusich: The present and future of choice architecture research

In our article “The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains” (1), we quantitatively review more than 200 empirical studies on behavior change interventions that apply a choice architecture approach. Our metaanalysis reveals that the overall effect of choice architecture interventions on behavior, uncorrected for publication bias, is Cohen’s d = 0.43, a value that is largely consistent with the results reported in other recent analyses of the effectiveness of choice architecture interventions in smaller subsets of the literature (2–5).

Our analyses further point to a publication bias in the literature that favors the reporting of successful implementations of choice architecture interventions, as indicated by visual inspection of the funnel plot and an Egger’s test of b = 2.10, 95% CI [1.31, 2.89], t(332) = 5.22, P < 0.001. Additional sensitivity analyses suggest an attenuation of the overall effect size of interventions to d = 0.31 under moderate publication bias and d = 0.08 under severe publication bias. We conclude that “in the light of these findings, the true effect size of interventions is likely to be smaller than estimated by our meta-analytic model due to the overrepresentation of positive effect sizes in our sample” (p. 4).

The three letters by Maier et al. (6), Szaszi et al. (7), and Bakdash and Marusich (8) underline the issue of publication bias in the literature and present alternative methods to identify and correct for this bias. Similar to our sensitivity analyses, they report smaller effect sizes or even null effects when applying robust Bayesian metaanalysis (6), Andrews–Kasy, weighted average of adequately powered studies (WAAP), or trim and fill methods (7) to the data. The letters moreover point out that the high heterogeneity among the effect sizes of interventions is problematic, and Szaszi et al. state that “we need to understand when and where some nudges have huge positive effects and why others are not able to repeat those successes” (p. 1).

We were grateful to see that our work has sparked an intense discussion about the effectiveness of choice architecture in the scientific community, as expressed not only in these letters but also in the number of discussions and commentaries published on social media. The objective of our article was not to write the “final word” on choice architecture interventions but to provide an overview of the current state of the empirical literature, to enable interested researchers to run alternative analyses by providing documented raw data and code, and to point out knowledge gaps and potential problems in the literature to stimulate future research.

We agree that the publication bias observed in the current choice architecture literature is problematic. Publication bias is not unique to research on choice architecture. Addressing it, however, is especially important in a field that seeks to design and test behavior change interventions that might have a high societal impact. Given the attention choice architecture research is attracting, the field may be especially susceptible to the methodological issues that give rise to publication bias, placing disproportionate emphasis on seemingly high intervention effects. More importance needs to be given to appropriately powered studies and open science practices such as preregistered reports and the publication of null results to avoid a skewed distribution of published choice architecture studies. We highlight this point in our discussion, where we make the call for future research to “place special emphasis on appropriate sample size planning and analysis standards when evaluating choice architecture interventions” (p. 8).

We also could not agree more with Szaszi et al.’s (7) assertion that we need to better understand the heterogeneity of effect sizes of choice architecture interventions. Indeed, as we state in our paper, “more research is needed to identify factors that may explain the variability in effect sizes above and beyond those investigated here” (p. 8). While the corpus of studies coded for this metaanalysis did not contain enough information to test for the potential impact of many theoretically interesting moderators, future research should systematically examine contextual, sociodemographic, and psychological factors that may explain differences in the impact of choice architecture interventions, and the psychological mechanisms underlying their effects on decision-making.

Choice architecture research is a relatively young field that comes with some methodological “growing pains.” In our analysis, we empirically identify publication bias and unexplained heterogeneity as two highly problematic issues that need to be addressed to improve the research standards of the field in the future, and to increase the role behavioral interventions can play in addressing some of our most pressing societal challenges.