Nudging: Progress to date and future directions

Abstract

Nudges influence behavior by changing the environment in which decisions are made, without restricting the menu of options and without altering financial incentives. This paper assesses past empirical research on nudging and provides recommendations for future work in this area by discussing examples of successful and unsuccessful nudges and by analyzing 174 articles that estimate nudge treatment effects. Researchers in disciplines spanning the behavioral sciences, using varied data sources, have documented that many different types of nudges succeed in changing behavior in a wide range of domains. Nudges that automate some aspect of the decision-making process have an average effect size, measured by Cohen’s d, that is 0.193 larger than that of other nudges. Our analyses point to the need for future research to pay greater attention to (1) determining which types of nudges tend to be most impactful; (2) using field and laboratory research approaches as complementary methods; (3) measuring long-run effects of nudges; (4) considering effects of nudges on non-targeted outcomes; and (5) examining interaction effects among nudges and other interventions.

1. Introduction

In 2008, Richard H. Thaler and Cass R. Sunstein published Nudge: Improving Decisions about Health, Wealth, and Happiness. The authors argued that managers and policy makers can help individuals make wiser choices by subtly altering the features of the environments in which those individuals make decisions. For example, changing the language used to describe the options in a menu, the format in which the options are presented, or the process by which the options are selected can alter individuals’ choices in domains ranging from health care to personal finance to environmental conservation. Importantly, to qualify as “nudges,” these “choice architecture” strategies do not mandate or forbid options, and they do not meaningfully change the financial incentives associated with various options. Rather, nudges tap into the psychology of decision making and gently guide individuals to different outcomes. In this paper, written a little more than ten years after the publication of Nudge, we provide an assessment of the academic literature evaluating the impact of nudge techniques, and we highlight challenges that this literature will need to confront going forward.

The insight that small changes to the decision-making environment can significantly alter choices has its foundations in a long literature in behavioral economics. Going back at least as far as the work of Herbert A. Simon on bounded rationality (Simon, 1955) and the work of Daniel Kahneman and Amos Tversky on heuristics and biases (Kahneman & Tversky, 1972, 1979; Tversky & Kahneman, 1973, 1974, 1981), social scientists have recognized that individuals face limitations on their ability to process information. Because of these limitations, individuals form many judgments using mental shortcuts that can lead to systematic decision-making errors. Subsequent work in behavioral economics incorporated these psychological phenomena into theoretical models of individual decisions (e.g., Bordalo, Gennaioli, & Shleifer, 2012; Bushong, Rabin, & Schwartzstein, 2019; Gabaix, 2014; Koszegi & Rabin, 2006; Laibson, 1997; O’Donoghue & Rabin, 1999, 2001; Shefrin & Thaler, 1988; Thaler & Shefrin, 1981). Additional work in this area also documented the empirical relevance of such psychological factors for economic outcomes in both laboratory and field settings (e.g., Angeletos, Laibson, Repetto, Tobacman, & Weinberg, 2001; Beshears & Milkman, 2011; Busse, Pope, Pope, & Silva-Risso, 2015; Camerer, Babcock, Loewenstein, & Thaler, 1997; Camerer & Lovallo, 1999; Chetty, Looney, & Kroft, 2009; DellaVigna & Malmendier, 2006; Fehr & Goette, 2007; Lacetera, Pope, & Sydnor, 2012; Lerner, Small, & Loewenstein, 2004; Loewenstein & Prelec, 1992; Malmendier & Tate, 2005; Milkman & Beshears, 2009; Odean, 1998, 1999).¹

Building on this large literature that investigates deviations from the neoclassical economic model of decision making, the idea that choice architecture strategies can and should be used to influence behavior relies on two further implications of the behavioral economics perspective. First, the impact of psychological biases on behavior implies that individuals’ choices cannot necessarily be relied upon, in the spirit of Afriat (1967), to reflect individuals’ normatively relevant preferences. This gap between revealed and normative preferences provides a rationale, beyond standard market failure considerations, for well-intentioned managers and policy makers to intervene in individuals’ decision making (Beshears, Choi, Laibson, & Madrian, 2008a). Second, the role of psychological factors in decision making represents not only a challenge but also an opportunity for improving economic outcomes. Managers and policy makers can augment their traditional toolkit of financial incentives with nudges that harness psychological factors in the service of promoting wiser decisions. The literature on nudges is thus a natural extension of prior work in behavioral economics, but it is also distinct from past research in that it focuses on how to apply behavioral economics ideas to important practical problems with the goal of impacting outcomes.

For one example of how to apply behavioral economics ideas in this way, consider automatic enrollment in retirement savings plans. Based on the insights that people are sometimes inattentive (Gabaix, 2019), tend to procrastinate when it comes to taking actions with short-run costs but long-run benefits (Laibson, 1997; O’Donoghue & Rabin, 1999), are often reluctant to switch away from an option because of an aversion to giving up its benefits (Samuelson & Zeckhauser, 1988), and are attracted to options that are perceived to be the social norm (Cialdini & Goldstein, 2004) or are perceived to be endorsed by a trusted authority (McKenzie, Liersch, & Finkelstein, 2006), a manager who hopes to increase employee savings in a firm’s retirement plan may wish to change employees’ default enrollment status in the plan from non-participation to participation (Thaler, 1994). This nudge changes the enrollment process from an opt-in mechanism (which implements a default contribution rate of zero to the plan for employees who do not actively elect to participate) to an opt-out mechanism (which implements a default contribution rate that is strictly positive for employees who do not actively elect an alternative). This change in the default dramatically increases the fraction of employees contributing to the plan (Beshears et al., 2006, 2008b, 2018; Choi et al., 2002, 2004; Madrian & Shea, 2001). Better yet, retirement savings can be further boosted by automatically escalating employees’ contribution rates at future points in time, such as the beginning of each year (Thaler & Benartzi, 2004).

Many public and private organizations have embraced choice architecture approaches to influencing decisions. Adoption of nudge techniques by these organizations has been driven in part by the quantitative evidence demonstrating the potential for nudges like automatic enrollment to cheaply and effectively change behavior. Adoption has also been bolstered by the argument that nudges do not restrict the choice set of well-informed individuals but do help individuals who would otherwise have difficulty selecting and implementing beneficial options from the choice menu (Camerer, Issacharoff, Loewenstein, O’Donoghue, & Rabin, 2003; Thaler & Sunstein, 2003, 2008). Continuing the savings example, automaticity and defaults are now widely applied in the context of retirement plans. In the United States, the Pension Protection Act of 2006 promoted the adoption of automatic enrollment in defined contribution plans (Beshears, Choi, Laibson, Madrian, & Weller, 2010), and a 2016 survey conducted by the Plan Sponsor Council of America (2018) found that 60% of the 401(k) plans in the sample used automatic enrollment. In several states, including California, Illinois, and Oregon, certain employers that do not sponsor qualified retirement plans are required to automatically enroll their employees in state-administered Individual Retirement Accounts (Center for Retirement Initiatives, 2019). Automatic enrollment also features prominently in national-level retirement systems in the United Kingdom, New Zealand, and Turkey. Beyond the domain of savings, the Organisation for Economic Co-operation and Development reports that 202 organizations around the world apply nudge tactics to public policy (OECD Research, 2018), and multinational firms such as Google, Merck, Swiss Re, and Deloitte have internal groups of employees responsible for incorporating choice architecture techniques into organizational processes.

The enthusiasm for nudge tactics among practitioners has been accompanied by burgeoning interest in nudge tactics among scholarly researchers. In this paper, we use two approaches to discuss the scholarly literature on nudging to date and to highlight important challenges for research in this area to consider in the future. First, we describe individual research articles that serve as examples illustrating key issues. This discussion is far from a comprehensive literature review, but it serves to make our points concrete. Second, we build and analyze a data set of 174 articles that estimated nudge treatment effects. These articles comprise the full set of articles in Elsevier’s Scopus database that (1) presented new data evaluating the effect of a nudge, (2) cited one of three seminal works on nudging (Camerer et al., 2003; Thaler & Sunstein, 2003, 2008), and (3) had at least ten citations as of July 2019. This data set represents only a fraction of the universe of research on nudging and is not a perfectly representative sample, but the selection criteria suggest that the data set is unlikely to generate a misleading impression of the portion of the literature receiving citations.

We begin our assessment of the empirical literature on nudging by exploring nudge treatment effect estimates across academic disciplines, domains of application, research settings (field observation versus other approaches), and types of nudges (those that automate some aspect of the decision-making process versus those that do not, as well as nudge categories defined by Beshears and Gino (2015)). For each category along each of these dimensions, we calculate the fraction of treatment effect estimates associated with the category and perform a p-curve analysis (Simonsohn, Nelson, & Simmons, 2014) for estimates that fall in the category. For all categories that have enough treatment effect estimates to permit meaningful analysis, we find that nudges have a statistically reliable effect on behavior. Thus, whether we consider the literature on nudging overall or category by category, the finding that nudges change outcomes is not purely an artifact of statistically unsound research practices or publication bias in favor of statistically significant estimates. However, this result does not rule out the possibility that our database captures a set of nudge treatment effect estimates that reflects some degree of publication bias. Comparisons of average treatment effects across categories are subject to the caveat that the severity of publication bias may vary by category, potentially causing average treatment effects in our sample to be differentially inflated relative to true average treatment effects.

We discuss five issues that future research on nudging should devote more attention to. First, we recommend that researchers investigate which types of nudges tend to have larger effects on outcomes. Such guidance would be useful to managers and policy makers who must select from a large number of possible nudge techniques when seeking to change behavior. In exploratory regressions analyzing our data set of nudge treatment effect estimates, we find suggestive evidence that nudges that automate some aspect of the decision-making process tend to have larger and more robust effects than nudges that do not automate some aspect of the decision-making process, but further work is needed to examine this issue. Second, researchers should increasingly use field-based and laboratory-based approaches as complementary methods to investigate why and in which situations nudges change outcomes. Third, researchers should place greater emphasis on studying the extent to which nudges lead to cumulative long-run effects on outcomes. Nudges may have long-run effects because they induce changes in habits, because they prompt investments in durable capital (e.g., physical capital or organizational capital as embedded in systems and processes), or because they are applied repeatedly over time. Fourth, researchers should put more effort into measuring the effects of nudges on non-targeted outcomes, as such unintended consequences can partially or even completely offset the intended effects of nudges on targeted outcomes. Fifth, in applications, nudges often represent only one part of a multi-pronged approach to changing behavior, so researchers should increase focus on the interaction effects among nudges and traditional interventions such as financial incentives. Depending on the circumstances, various interventions may be substitutes or complements for one another, and the distinction is critical for designing effective packages of interventions.

Overall, the literature on nudging has been a success, as the empirical evidence indicates that applying behavioral science in this way to solve managerial problems and to advance policy objectives can indeed change behavior. Still, we advocate a more ambitious approach for the next generation of nudging research. Instead of merely asking whether choice architecture strategies can change the action that a person takes in a particular situation (the resounding answer is that they can), future research should go much further by asking which types of choice architecture strategies are most impactful at changing outcomes in an enduring and comprehensive fashion in important contexts.

The paper proceeds as follows. In Section 2, we describe the construction of our data set of nudge treatment effect estimates. In Section 3, we use the data set and a series of example articles to assess the nudge literature to date. We explain our recommendations for future nudge research in Section 4. Section 5 contains our general discussion and conclusions.

2. Data set construction

In this section, we briefly describe the process for constructing our data set capturing information on past empirical research on nudging. The Online Appendix provides further details.

2.1. Selection of articles

To build our data set, we first identified all of the articles in Scopus, Elsevier’s abstract and citation database, that cite at least one of three foundational works on nudging: “Regulation for Conservatives: Behavioral Economics and the Case for ‘Asymmetric Paternalism’” (Camerer et al., 2003), “Libertarian Paternalism” (Thaler & Sunstein, 2003), and Nudge: Improving Decisions about Health, Wealth, and Happiness (Thaler & Sunstein, 2008). In order to focus on articles that have some degree of influence on the scholarly conversation, we limited the list of articles to those that had at least ten citations in Scopus as of July 1, 2019. We then examined each article in the resulting list of 1052 articles to determine whether it reported at least one treatment effect estimate for a nudge intervention based on novel data (as opposed to a re-analysis of previously reported data). In judging whether or not an intervention qualifies as a nudge, we relied on the definition articulated by Thaler and Sunstein (2008, p. 6): a nudge is an intervention that changes “people’s behavior in a predictable way without forbidding any options or significantly changing their economic incentives. To count as a mere nudge, [an] intervention must be easy and cheap to avoid. Nudges are not mandates.” This process narrowed the list to 174 articles.

Fig. 1 shows the distribution of publication year for the 174 articles in our data set. The book Nudge, which was cited more frequently in our data set than the other two foundational works on nudging, was first published in 2008, so the publication year for articles in our data set is concentrated in the years following 2008. The distribution tails off in the later years because a later publication date leaves less time for an article to accumulate at least ten citations by July 1, 2019.

Fig. 1.

Distribution of articles in the data set by year of publication.

2.2. Determination of the number of nudge treatment effect observations

The unit of observation in our data set is the nudge treatment effect estimate. A given article may include multiple experimental conditions, measure multiple outcome variables, and work with multiple populations in multiple settings. Our data set includes separate observations to capture the distinct nudge treatments embedded in a single article. For example, an article comparing two nudge treatments to a control group (using one outcome variable and one study population in one research setting) is represented by two observations in our data set. When an article examines multiple outcome variables, populations, or settings, we follow the authors of the article in determining how to record the results. If the authors report results aggregated across these dimensions, our data set records an observation for the aggregated treatment effect. Otherwise, our data set records multiple observations for the disaggregated treatment effects. This process generated 965 nudge treatment effect observations. See the Online Appendix for details.

2.3. Variables recorded in the data set

For each observation in our data set, we recorded the following variables:

• What academic discipline is associated with the article containing the treatment effect estimate (e.g., economics)?

• In what domain of application was the treatment effect estimated (e. g., health care)?

• In which type of research setting was the treatment effect estimated (e.g., field experiment)?

• What is the nature of the outcome variable (e.g., a summary measure of a series of actions in a field setting)?

• Did the nudge treatment automate some aspect of the individual’s decision-making process?

• To which categories and subcategories of nudges does the treatment belong, in the taxonomy of Beshears and Gino (2015)?

• Did the researchers collect follow-up data to measure the treatment effect over a longer time horizon?

• If the researchers collected follow-up data, did the treatment effect persist over the longer time horizon?

• Did the researchers also measure the treatment effect of the nudge on an outcome variable that could offset the treatment effect on the focal outcome variable recorded in this observation? If so, was there such a variable in the same domain as the focal outcome variable? Was there such a variable in a different domain from the focal outcome variable?

• Did the article test whether there is an interaction effect between the nudge and another intervention?

• Is the outcome variable continuous or dichotomous?

• What was the size of the treatment effect, as measured by Cohen’s d?

• For the test of the hypothesis that the treatment effect was zero, was the p-value less than 0.10? Was the p-value less than 0.05? What was the exact p-value?²

• Did the nudge backfire?

The Online Appendix describes the details of these variables, but several variables require further commentary here.

The variables capturing the research setting and the nature of the outcome variable are closely related to each other. In our analysis, we focus on the research setting and not the nature of the outcome variable.

To develop a taxonomy of nudges, Beshears and Gino (2015) build on the idea, popularized by Kahneman (2011), that humans have two basic modes of thinking that they use to make decisions. System 1 thinking is fast and intuitive, but it is prone to errors because it relies on mental shortcuts that can sometimes be led astray. System 2 thinking is slow and deliberative, but it is more likely to reach well-considered conclusions. Beshears and Gino (2015) argue that nudges can alter an individual’s decisions by triggering system 1 (that is, by eliciting an intuitive reaction that leads to a different choice), by engaging system 2 (that is, by prompting a momentary pause during which the individual engages in a more reflective cognitive process), or by bypassing both systems (that is, by removing the individual from some aspect of the decision-making process).³ Each of these categories of nudges has subcategories, which we describe in Sections 3.4.2–3.4.4. We allow a given nudge to fall in more than one category or subcategory to recognize that a nudge may operate through multiple mechanisms. Note that the set of nudges that automate some aspect of the decision-making process is larger than the set of nudges that bypass both systems because some nudges that trigger system 1 and some nudges that engage system 2 also involve automaticity. For example, a nudge might automate the process of filling out a registration form for a service, but the individual might still be required to submit the form to sign up for the service. This nudge triggers system 1 by simplifying the process of implementing the decision to sign up. The nudge does not bypass both systems because it has not removed the individual from the process of making the decision to sign up. Thus, the variable recording whether the nudge involves automaticity is not redundant with the variable recording the nudge category.

To calculate Cohen’s d for a continuous outcome variable, we divide the estimated treatment effect size in the natural units of the outcome variable by the mean of the standard deviation of the outcome variable across the experimental conditions. When the outcome variable is dichotomous, we use the arcsine transformation ($2^{*}\text{arcsine}\sqrt{p_1}-2^{*}\text{ arcsine }\sqrt{p_2}$) to calculate Cohen’s d, where p₁ and p₂ are the proportions of successes in the two experimental groups (see, for example, Chernev, Böckenholt, & Goodman, 2015).⁴ Many articles do not report sufficient information to calculate Cohen’s d, so out of the 965 nudge treatment effect observations in our data set drawn from 174 research articles, Cohen’s d is non-missing for only 507 observations drawn from 101 articles. We perform an ordinary least squares regression for which the outcome variable is an indicator for having a missing Cohen’s d. The explanatory variables are (a) either an indicator for whether the nudge involves automaticity or indicators for nudge categories in the Beshears and Gino (2015) taxonomy; (b) indicators for academic discipline; (c) indicators for domain of application; (d) an indicator for whether the research setting involves field observation (either via a field experiment or via observational analysis of a natural experiment involving field data); and (e) an indicator for whether the outcome variable is dichotomous. The indicators for academic discipline are jointly statistically significant, as are the indicators for domain of application, suggesting that scholarly norms within a subfield are important determinants of whether researchers report the necessary information for calculating Cohen’s d. The indicator for whether the outcome variable is dichotomous is also statistically significant, but this pattern is not surprising because the data requirements for calculating Cohen’s d for dichotomous variables are less stringent. The indicator for whether the nudge involves automaticity and the indicator for whether the research setting involves field observation are not significant, and the indicators for nudge categories are not statistically different from each other.

Finally, it is natural to ask whether the analysis recorded in a given observation in our data set was pre-registered. We follow the methodology of Schäfer and Schwarz (2019) to determine whether the analyses captured in our data set were pre-registered, and we find that none of them were. Thus, it is important to keep in mind that the effect size estimates in our data set may overstate true effect sizes (DellaVigna & Linos, 2020; Schäfer & Schwarz, 2019). Furthermore, differences in effect size estimates across the nudges in our data set may be due to differential overstatement of true effect sizes, perhaps driven by differential severity of publication bias. However, for some important comparisons of nudge treatment effect estimates (e.g., the comparison of nudges that involve automaticity versus those that do not), there is little reason to expect differential overstatement of true effect sizes. In addition, the possibility of overstated effect sizes does not call into question the conclusion that nudges affect behavior. When we conduct p-curve analyses (described in Section 3), the evidence indicates that the conclusion that nudges impact outcomes is not purely the result of publication bias.

3. Assessing the empirical literature on nudging

In Tables 1–4, we analyze our data set by breaking down the nudge treatment effect estimates into categories along several different dimensions: academic discipline, domain of application, research setting, and type of nudge. For each category that we consider, we report the percentage of nudge treatment effect estimates in the data set that are associated with that category. In this calculation, each observation receives a weight proportional to the inverse of the total number of observations associated with the same article. The calculation thereby weights each article equally. We prefer this weighting scheme because it does not place greater weight on an article merely because it measures a larger number of outcome variables.

Table 1.

Nudge research by academic discipline.

	Percentage of effects associated with this discipline	Mean Cohen’s d	Percentage of effects with p < 0.05	Percentage of effects with p < 0.10	Percentage of p-curves with evidential value (p < 0.05)
All disciplines	100%	0.405	61.0%	65.6%	100%
Economics and finance	21.8%	0.201	62.4%	66.1%	100%
Economics	21.3%	0.201	61.3%	65.2%	100%
Finance	0.6%	-	-	-	-
Environmental science	8.6%	0.480	65.4%	67.1%	100%
Marketing / consumer behavior	18.4%	0.377	48.7%	54.8%	100%
Medicine	11.5%	0.377	64.1%	68.3%	100%
Psychology and cognitive science	27.0%	0.526	62.4%	69.1%	100%
Psychology	24.1%	0.531	60.9%	68.4%	100%
Cognitive Science	2.9%	-	-	-	-
Public health	6.3%	0.435	69.5%	69.5%	98%
Miscellaneous	6.3%	0.490	66.0%	70.0%	100%
Computer science	0.6%	-	-	-	-
Engineering	0.6%	-	-	-	-
Law	1.2%	-	-	-	-
Management	1.2%	-	-	-	-
Political science	0.6%	-	-	-	-
Public administration	0.6%	-	-	-	-
Transportation	1.7%	-	-	-	-

Percentage of effects associated with this discipline is a weighted mean, with an observation’s weight proportional to the inverse of the number of observations associated with the same article. Mean Cohen’s d and percentage of effects with p < 0.05 (p < 0.10) are weighted means, with an observation’s weight proportional to the inverse of the number of non-missing observations associated with the same article and with the discipline named in the leftmost column. For the p-curve analysis, we randomly sample one observation per article and calculate the p-value for the null hypothesis that the random sample does not have evidential value. We repeat this procedure for 50 independent random samples and report the fraction of samples for which p < 0.05. We only report mean Cohen’s d, the percentage of effects with p < 0.05, the percentage of effects with p < 0.10, and the percentage of p-curves with evidential value if the discipline is associated with at least ten articles.

Within a category that we consider, we also report the mean Cohen’s d, the percentage of effects with p < 0.05, and the percentage of effects with p < 0.10. Here, each observation within a category receives a weight proportional to the inverse of the total number of non-missing observations in that category associated with the same article.

We also perform a p-curve analysis (Simonsohn et al., 2014) for each of the categories that we consider. A p-curve analysis is a statistical test that is applied to a collection of null hypothesis statistical tests. In our case, the inputs to the p-curve analysis are the p-values from the tests of the null hypothesis that the nudge treatment effect is zero. If the null hypothesis were true for all of the nudge treatment effects in the category under consideration, we would expect the p-values less than 0.05 to be uniformly distributed over the interval from zero to 0.05. The p-curve analysis formally tests the hypothesis that the distribution is uniform. If this hypothesis is rejected and the nudge treatment effect p-values less than 0.05 are close to zero more frequently than they are close to 0.05, we conclude that there is evidence for the existence of a nudge treatment effect not equal to zero.⁵ In order to weight articles equally in the p-curve analysis and to use independent p-values as inputs to the p-curve algorithm, we implement the following procedure. Among observations within a category, we randomly select one observation per article and enter the resulting subset of p-values into the p-curve algorithm.⁶ We record whether the algorithm rejects (with p < 0.05) the null hypothesis that the collection of p-values represents a body of research that does not have evidential value. We repeat this procedure for 49 additional independent random draws of one observation per article, and we report the percentage of these 50 random draws for which the algorithm rejects the null hypothesis of no evidential value.

In Tables 1–4, we always report the percentage of nudge treatment effect estimates associated with a given category. However, we only report the results of the other calculations when the category is associated with at least ten research articles. In this section, we primarily discuss the percentages of nudge treatment effect estimates associated with various categories because we wish to establish that many different types of nudges have been shown to be effective in a wide range of academic disciplines, domains of application, and research settings. We defer cross-category comparisons of Cohen’s d, the likelihood of an effect with p < 0.05, and the likelihood of an effect with p < 0.10 to Section 4, in which we report the results of regression analyses. Every category for which we conduct the p-curve analysis has 86% or more of its associated p-curves rejecting the null hypothesis of no evidential value, with most categories having 100% of their associated p-curves rejecting that null hypothesis. Thus, we do not discuss the p-curve results category by category. We simply note here that the presence of nudge treatment effects is statistically robust.

3.1. Nudge research by academic discipline

Table 1 shows that the literature evaluating nudge interventions is truly multidisciplinary. The most represented academic discipline in our data set is psychology, which is associated with 24.1% of nudge treatment effect estimates. Economics is the second-most represented discipline, with 21.3% of estimates. The category encompassing marketing and consumer behavior follows close behind with 18.4%. Medicine, environmental science, and public health also have substantial representation. In each of the categories with at least ten associated research articles, mean Cohen’s d falls in the range 0.20–0.53, and the percentage of effects with p < 0.05 (p < 0.10) falls in the range 49–70% (55–70%).

3.2. Nudge research by domain of application

In Table 2, we see that the effect of nudge interventions on behavior has also proved to be robust across many domains of application. In our data set, 33.9% of the nudge treatment effect estimates are applied to health-related decisions, and 24.1% are applied to decisions related to the environment. Financial decision-making and prosocial behavior are also well represented. Among domains associated with at least ten research articles, mean Cohen’s d falls in the range 0.21–0.50, and the percentage of effects with p < 0.05 (p < 0.10) falls in the range 52–70% (60–76%).⁷

Table 2.

Nudge research by domain of application.

	Percentage of effects associated with this domain	Mean Cohen’ s d	Percentage of effects with p < 0.05	Percentage of effects with p < 0.10	Percentage of p-curves with evidential value (p < 0.05)
Environment	24.1%	0.421	66.6%	71.3%	100%
Finance	7.4%	0.316	70.1%	76.1%	98%
Health	33.9%	0.416	59.3%	62.5%	100%
Exercise	1.7%	-	-	-	-
Health care	10.9%	0.286	57.3%	59.5%	96%
Healthy eating	19.5%	0.504	59.7%	62.0%	100%
Miscellaneous health	1.7%	-	-	-	-
Prosocial behavior	6.9%	0.213	51.8%	66.4%	86%
Miscellaneous	27.7%	0.504	59.0%	62.2%	100%
Crime / criminal justice	2.3%	-	-	-	-
Development	0.6%	-	-	-	-
Education	1.1%	-	-	-	-
Labor	0.6%	-	-	-	-
Other	23.1%	0.554	59.1%	63.0%	100%

Percentage of effects associated with this domain is a weighted mean, with an observation’s weight proportional to the inverse of the number of observations associated with the same article. Mean Cohen’s d and percentage of effects with p < 0.05 (p < 0.10) are weighted means, with an observation’s weight proportional to the inverse of the number of non-missing observations associated with the same article and with the domain named in the leftmost column. For the p-curve analysis, we randomly sample one observation per article and calculate the p-value for the null hypothesis that the random sample does not have evidential value. We repeat this procedure for 50 independent random samples and report the fraction of samples for which p < 0.05. We only report mean Cohen’s d, the percentage of effects with p < 0.05, the percentage of effects with p < 0.10, and the percentage of p-curves with evidential value if the domain is associated with at least ten articles.

3.3. Nudge research by research setting

Table 3 splits our data set into two types of research settings: observation in the field (via field experiments or observational analyses of natural experiments) and observation in other environments (via laboratory experiments, online experiments, or surveys). These two types of settings are roughly equally prevalent, with observation in the field accounting for 44.0% and observation in other environments accounting for 56.0% of the nudge treatment effect estimates. Among treatment effect estimates associated with field observation, mean Cohen’s d is 0.32, and the percentage of effects with p < 0.05 (p < 0.10) is 66.0% (70.0%). Among treatment effect estimates not associated with field observation, mean Cohen’s d is 0.48, and the percentage of effects with p < 0.05 (p < 0.10) is 60.1% (65.3%).

Table 3.

Nudge research by research setting.

	Percentage of effects associated with this setting	Mean Cohen’ s d	Percentage of effects with p < 0.05	Percentage of effects with p < 0.10	Percentage of p-curves with evidential value (p < 0.05)
Field observation (field experiment or observational study)	44.0%	0.320	66.0%	70.0%	100%
Laboratory experiment, online experiment, or survey	56.0%	0.481	60.1%	65.3%	100%

Percentage of effects associated with this setting is a weighted mean, with an observation’s weight proportional to the inverse of the number of observations associated with the same article. Mean Cohen’s d and percentage of effects with p < 0.05 (p < 0.10) are weighted means, with an observation’s weight proportional to the inverse of the number of non-missing observations associated with the same article and with the setting named in the leftmost column. For the p-curve analysis, we randomly sample one observation per article and calculate the p-value for the null hypothesis that the random sample does not have evidential value. We repeat this procedure for 50 independent random samples and report the fraction of samples for which p < 0.05.

3.4. Nudge research by type of nudge

In Table 4, we present statistics summarizing nudge treatment effect estimates associated with different types of nudges. Because one of the issues that we highlight in Section 4 is the importance of determining which types of nudges are particularly impactful relative to others, this subsection goes into detail regarding our definitions of different categories and subcategories of nudges.

Table 4.

Nudge research by type of nudge.

	Percentage of effects associated with this category	Mean Cohen’ s d	Percentage of effects with p < 0.05	Percentage of effects with p < 0.10	Percentage of p-curves with evidential value (p < 0.05)
Nudges that use automaticity	15.3%	0.521	72.7%	78.6%	100%
Nudges that do not use automaticity	84.7%	0.385	58.2%	62.7%	100%
Nudges that trigger system 1	72.1%	0.468	63.2%	66.7%	100%
By arousing emotions	29.3%	0.326	54.9%	59.7%	100%
By harnessing biases	20.0%	0.515	64.3%	65.0%	100%
By simplifying the process	24.4%	0.539	71.4%	73.9%	100%
Nudges that engage system 2	41.6%	0.346	60.9%	65.1%	100%
By encouraging joint evaluation	-	-	-	-	-
By creating opportunities for reflection	22.4%	0.329	65.0%	68.7%	100%
By prompting planning	1.9%	-	-	-	-
By inspiring broader thinking	14.7%	0.396	49.5%	54.2%	100%
By increasing accountability	1.2%	-	-	-	-
By emphasizing disconfirming evidence	-	-	-	-	-
By using reminders	2.6%	-	-	-	-
Nudges that bypass both systems	13.8%	0.546	69.9%	77.1%	100%
By setting the default	13.2%	0.546	68.8%	76.3%	100%
By making automatic adjustments	0.6%	-	-	-	-

Percentage of effects associated with this category is a weighted mean, with an observation’s weight proportional to the inverse of the number of observations associated with the same article. Note that percentages add up to more than 100% because a given nudge can belong to more than one category and more than one subcategory. Mean Cohen’s d and percentage of effects with p < 0.05 (p < 0.10) are weighted means, with an observation’s weight proportional to the inverse of the number of non-missing observations associated with the same article and with the category named in the leftmost column. For the p-curve analysis, we randomly sample one observation per article and calculate the p-value for the null hypothesis that the random sample does not have evidential value. We repeat this procedure for 50 independent random samples and report the fraction of samples for which p < 0.05. We only report mean Cohen’s d, the percentage of effects with p < 0.05, the percentage of effects with p < 0.10, and the percentage of p-curves with evidential value if the discipline is associated with at least ten articles.

3.4.1. Nudges that do and do not use automaticity

First, we divide nudge treatments into those that do and those that do not automate some aspect of an individual’s decision-making process. An example of a nudge that involves automation is changing the default enrollment status in a defined contribution retirement savings plan from non-participation to participation. If an individual does not actively indicate a desire to contribute to the plan or not contribute to the plan, this nudge automatically implements a strictly positive contribution amount on behalf of the individual. Previous research has documented that this nudge generates large increases in retirement plan participation rates (Beshears et al., 2006, 2008b; Choi et al., 2002, 2004; Madrian & Shea, 2001).

Table 4 reveals that 15.3% of the nudge treatment effect estimates in our data set involve a nudge that uses automaticity. Among this subset of treatment effect estimates, mean Cohen’s d is 0.521, and the percentage of effects with p < 0.05 (p < 0.10) is 72.7% (78.6%). Among the subset of treatment effect estimates involving nudges that do not use automaticity, mean Cohen’s d is 0.385, and the percentage of effects with p < 0.05 (p < 0.10) is 58.2% (62.7%). While the effects of nudges are statistically robust regardless of whether or not they automate some aspect of the decision-making process, there is suggestive evidence that automaticity produces larger effects. We explore this possibility in more depth in Section 4.1.

3.4.2. Nudges that trigger system 1

We next examine nudges that operate by triggering system 1—that is, nudges that change behavior by invoking a fast, intuitive reaction. In Table 4, we see that 72.1% of the treatment effect estimates in our data set involve a nudge that triggers system 1, at least in part (recall that a given nudge may simultaneously trigger system 1, engage system 2, and bypass both systems). Mean Cohen’s d for these treatment effect estimates is 0.468, and the percentage of effects with p < 0.05 (p < 0.10) is 63.2% (66.7%).

Beshears and Gino (2015) list three different techniques for triggering system 1: arousing emotions, harnessing biases, and simplifying the decision-making process.

3.4.2.1. Nudges that arouse emotions.

The first technique for triggering system 1 is to arouse emotions. For example, Beshears, Choi, Laibson, Madrian, and Zeldes (2014) study framing effects in annuity purchases. Life annuities transform a lump sum payment into a steady stream of monthly income that lasts for the rest of the annuitant’s life. When a hypothetical annuity purchase decision is framed as a choice between more versus less guaranteed income, survey respondents annuitize more of their wealth compared to when the decision frame emphasizes the implications of annuities for flexibility and control over asset allocation and the timing of spending. Of course, having “more guaranteed income” and having “less flexibility” are two ways of framing the same fundamental feature of annuities, namely the steady stream of payments that they provide. Emphasizing one lens or the other invokes different emotional reactions that sway an individual’s annuity purchase decisions. Table 4 shows that 29.3% of the treatment effect estimates in our data set are associated with a nudge that arouses emotions.

3.4.2.2. Nudges that harness biases.

The second technique for triggering system 1 is to harness biases. Nudges that harness biases change behavior by tapping into psychological processes that are known to influence decision making in a systematic fashion. For example, Beshears, Dai, Milkman, and Benartzi (2019) test a retirement savings nudge that harnesses the “fresh start effect,” the tendency of individuals to initiate the pursuit of virtuous goals at moments that represent the beginning of a new time period (Dai, Milkman, & Riis, 2014). When individuals received mailings offering the opportunity to increase retirement plan contributions at a future point in time that was framed as a new beginning (the recipient’s birthday, New Year’s, or the first day of spring), they contributed more compared to when the future point in time was framed as a control temporal landmark and compared to when the future point in time was discussed without reference to a temporal landmark. Table 4 shows that 20.0% of the treatment effect estimates in our data set are associated with a nudge that harnesses biases.

3.4.2.3. Nudges that simplify the process.

The third technique for triggering system 1 is to simplify the process by which decisions are made. By making it easy to select a particular option from the choice menu, a nudge can take advantage of the tendency of system 1 to gravitate towards options that involve little up-front effort to implement. For example, Beshears, Choi, Laibson, and Madrian (2013) apply this technique to increase retirement savings plan participation. The typical plan enrollment process is somewhat complex because it often involves selecting both a contribution rate and an asset allocation for those contributions. When the process is simplified to be a “yes or no” choice of whether to enroll with a pre-selected contribution rate and pre-selected asset allocation, plan participation rates increase by 10–20 percentage points. Here, simplification makes people more likely to take action. In Table 4, we see that 24.4% of the treatment effect estimates in our data set are associated with a nudge that simplifies the decision-making process.

3.4.3. Nudges that engage system 2

We now turn to nudges that operate by engaging system 2—that is, nudges that change behavior by prompting individuals to initiate a more methodical decision-making process than would have otherwise taken place. Table 4 shows that 41.6% of nudge treatment effect estimates in our data set involve a nudge that engages system 2. Mean Cohen’s d for these treatment effect estimates is 0.346, and the percentage of effects with p < 0.05 (p < 0.10) is 60.9% (65.1%).

Beshears and Gino (2015) identify seven techniques for engaging system 2, and five of these techniques are represented in our data set: creating opportunities for reflection, prompting planning, inspiring broader thinking, increasing accountability, and using reminders.⁸

3.4.3.1. Nudges that create opportunities for reflection.

The first technique for engaging system 2 is to create opportunities for reflection. Individuals have limited attention, so in some cases it is possible to promote wise decision making by simply encouraging individuals to pause for a moment to consider the options available to them. For example, consider the case of a company that was looking to increase the number of its employees who were receiving maintenance prescription medications for chronic conditions, such as high cholesterol, by mail delivery instead of by in-person pick-up at the pharmacy. Mail delivery is more cost-effective both for the employer and for the employees. The company changed from an opt-in policy, under which employees could elect to receive mail delivery but would otherwise obtain prescriptions by in-store pick-up, to an active choice policy, under which employees were eligible for the prescription drug plan only if they indicated affirmatively whether they wished to use the mail delivery service or wished to pick up prescriptions at the pharmacy. The financial incentives associated with mail delivery and in-store pick-up were unchanged, but the active choice policy called attention to the decision at hand and increased the percentage of employees choosing home delivery from 6% to 42%, creating cost savings for the employer and its employees of approximately $700,000 per year (Beshears, 2016c; Beshears, Rooney, & Sanford, 2016a, 2016b; Beshears et al., in press-b). In Table 4, we see that 22.4% of the treatment effect estimates in our data set are associated with a nudge that creates opportunities for reflection.

3.4.3.2. Nudges that prompt planning.

Other nudges engage system 2 by prompting individuals to create plans for implementing valuable actions in the future. The process of thinking through the details of how to enact plans embeds those plans more firmly in memory, raises awareness of possible logistical hurdles (which are then more likely to be resolved), and creates a personal commitment from which deviation is undesirable, and all of these factors increase the likelihood of following through on intentions (Beshears, Milkman, & Schwartzstein, 2016). Consistent with this reasoning, when an employer offers free workplace influenza vaccination clinics to employees, prompting employees to write down the date and time when they plan to visit a clinic increases vaccination rates (Beshears, 2016a, 2016b; Milkman, Beshears, Choi, Laibson, & Madrian, 2011). Similar results obtain in the domain of preventive cancer screenings (Dai et al., 2012; Milkman, Beshears, Choi, Laibson, & Madrian, 2013). Table 4 indicates that 1.9% of the treatment effect estimates in our data set are associated with a nudge that prompts planning.

3.4.3.3. Nudges that inspire broader thinking.

Some nudges engage system 2 by encouraging individuals to reconsider the lens through which they are viewing a problem and to contemplate the consequences of different possible actions within a broader context. A broader decision-making frame tends to incorporate factors that might not otherwise be considered, such as long-run implications and possible unintended consequences. For example, McKenzie and Liersch (2011) demonstrate that encouraging employees at a company to think about what their future retirement savings plan balance will be causes 41% of them to report greater interest in saving more, whereas encouraging employees to think about their current plan balance causes only 27% of them to report greater interest in saving more. In Table 4, we see that 14.7% of the treatment effect estimates in our data set are associated with a nudge that inspires broader thinking.

3.4.3.4. Nudges that increase accountability.

Another technique for engaging system 2 is to increase the extent to which individuals feel accountable for their actions. Even if individuals are holding themselves accountable to some internally generated standard of behavior, and not being held accountable to a standard imposed by an external party, their desire to be consistent with their self-professed values and to honor their promises to themselves can change their behavior. In an experiment conducted at a hotel, guests who were invited during the check-in process to commit to reusing bathroom towels and were given a symbolic pin to wear for making such a commitment were more likely to reuse towels than guests who were invited to make a generic commitment to environmentally friendly behavior or who were not given a symbolic pin (Baca-Motes, Brown, Gneezy, Keenan, & Nelson, 2013). See Aleksovska, Schillemans, and Grimmelikhuijsen (2019) for a review of the literature on accountability. Table 4 shows that 1.2% of the treatment effect estimates in our data set are associated with a nudge that increases accountability.

3.4.3.5. Nudges that use reminders.

A final technique for engaging system 2 is to remind individuals of the opportunity to take a particular action. When individuals intend to implement a behavior but have not yet found a convenient moment for following through, or when individuals have simply forgotten of their intentions, a reminder can bring their intentions back to the top of mind and prompt them to take action. For example, Altmann and Traxler (2014) show that among patients of a dentist who were due for a check-up, a postcard reminder increased both the likelihood of making an appointment and the likelihood of completing an appointment by approximately ten percentage points. For another example, placing a free workplace influenza vaccination clinic in a location that employees pass regularly in the course of their day-to-day activities increases vaccination rates relative to placing the clinic in a nearby location that employees do not pass regularly, as encountering the clinic on the way to some other activity reminds employees to obtain a vaccination (Beshears, Choi, Laibson, Madrian, & Reynolds, 2016). In Table 4, we see that 2.6% of the treatment effect estimates in our data set are associated with a nudge involving a reminder.

3.4.4. Nudges that bypass both systems

When a nudge bypasses both systems, it takes individuals’ actions as inputs and changes how those actions translate into outcomes. Table 4 shows that 13.8% of nudge treatment effect estimates in our data set involve a nudge that bypasses both systems. Mean Cohen’s d for these treatment effect estimates is 0.546, and the percentage of effects with p < 0.05 (p < 0.10) is 69.9% (77.1%).

Beshears and Gino (2015) identify two subcategories of nudges that bypass both systems: nudges that set the default and nudges that make automatic adjustments.

3.4.4.1. Nudges that set the default.

In Section 1, we discussed automatic enrollment in employer-sponsored retirement savings plans, an example of a nudge that sets the default. By changing the default contribution rate from zero to a strictly positive percentage of pay, automatic enrollment changes what happens when an employee does not actively indicate a preferred contribution rate. Without automatic enrollment, such an employee does not become a plan participant; under automatic enrollment, such an employee does start contributing to the plan. The nudge bypasses both systems in the sense that many individuals passively accept the default without using either system 1 thinking or system 2 thinking to consider their options, and automatic enrollment translates this action (or, more accurately, inaction) into plan participation instead of non-participation as the outcome. Past research has documented that automatic enrollment generates large increases in savings plan participation rates (Beshears et al., 2006, 2008b; Choi et al., 2002, 2004; Madrian & Shea, 2001). In a meta-analysis of nudges that set the default, Jachimowicz, Duncan, Weber, and Johnson (2019) find that these nudges influence outcomes across a wide range of settings. In Table 4, we see that 13.2% of the treatment effect estimates in our data set are associated with a nudge that sets the default.

3.4.4.2. Nudges that make automatic adjustments.

Whereas a nudge that sets the default changes the outcome that is implemented when an individual does not actively select an option, a nudge that makes automatic adjustments changes the outcome that is implemented when an individual does make an active choice. For example, consider a firm that offers a traditional before-tax retirement savings account to its employees. When employees contribute money out of their paychecks to such an account, the contributions are tax-deductible in the year they are made, and subsequent withdrawals from the account are taxable. Now consider what happens if the firm introduces a “Roth” savings account as one of its retirement offerings. Contributions to a Roth account are not tax-deductible in the year they are made, but withdrawals from the account are not taxable. If employees use rules of thumb such as “save 10% of income” and ignore the tax treatment of savings, the Roth account increases savings relative to the before-tax account because a given percentage of gross income contributed to a Roth account, which is not taxed at withdrawal, translates into greater after-tax savings than the same percentage of gross income contributed to a before-tax account, which is taxed at withdrawal. Consistent with this idea, Beshears, Choi, Laibson, and Madrian (2017b) analyze eleven companies that introduced Roth retirement accounts and find no evidence that these accounts changed total contributions. Thus, the Roth accounts bypassed both systems because many employees seemed not to change their system 1 thinking or system 2 thinking; instead, they elected to contribute the same percentage of gross income, without regard for the tax treatment of contributions. The Roth accounts made automatic adjustments to employee outcomes in the sense that the same percentage of gross income contributed to a Roth account translated into higher effective (after-tax) savings. Table 4 indicates that 0.6% of the treatment effect estimates in our data set are associated with a nudge that makes automatic adjustments.

4. Future challenges for the empirical nudge literature

Having documented that there is statistically robust evidence in favor of the efficacy of nudges across multiple disciplines, domains of application, research settings, and types of nudges, we turn to a discussion of challenges that the empirical nudge literature must confront if it is to further expand its influence on managerial practice and policy design.

4.1. Comparing categories of nudges based on their impact

To begin our exploration of the categories of nudges that tend to have the largest impact on outcomes, we use our data set of nudge treatment effect estimates and study the correlates of a nudge’s effect size as measured by Cohen’s d. Recall that we cannot calculate Cohen’s d for all of the observations in our data set because the necessary information is not always available, but we can calculate Cohen’s d for 507 treatment effect estimates reported in 101 articles.

Columns 1 and 2 of Table 5 report the results from ordinary least squares regressions with Cohen’s d as the outcome variable. In column 1, the explanatory variable of interest is an indicator for whether the nudge automates some aspect of the decision-making process. The control variables are indicators for the academic discipline of the article from which the treatment effect estimate is drawn (see Table 1), indicators for the domain of application (see Table 2), an indicator for whether the research setting involves field observation (either via a field experiment or via observational analysis of a natural experiment involving field data), and an indicator for whether the outcome variable for which Cohen’s d is calculated is dichotomous (because the calculation of Cohen’s d differs for continuous versus dichotomous variables; see Section 2.3). In the regression, each observation has a weight proportional to the inverse of the number of observations in the regression associated with the same article. This procedure gives each article equal weight in the regression. Standard errors are clustered by article. We find that nudges involving automaticity are associated with a Cohen’s d that is 0.193 larger (p < 0.05). To put the magnitude of this estimate in context, Fig. 2 shows predicted values and associated 95% confidence intervals from the regression holding all right-hand-side variables fixed at their means except for the indicator for whether the nudge involves automaticity, which takes a value of either zero or one. With other variables held at their means, the predicted Cohen’s d for nudges that involve automaticity is approximately 50% larger than the predicted Cohen’s d for nudges that do not involve automaticity.

Table 5.

Regression analysis of effect size and statistical significance.

	Cohen’s d	Cohen’ s d	Indicator for p < 0.05	Indicator for p < 0.05	Indicator for p < 0.10	Indicator for p < 0.10
Nudge uses automaticity	0.193*	-	0.139*	-	0.155**	-
	(0.083)	-	(0.066)	-	(0.058)	-
Nudge triggers system 1	-	0.476**	-	0.828**	-	0.802**
	-	(0.095)	-	(0.126)	-	(0.150)
Nudge engages system 2	-	0.421**	-	0.764**	-	0.779**
	-	(0.135)	-	(0.138)	-	(0.160)
Nudge bypasses both	-	0.658**	-	0.815**	-	0.915**
systems	-	(0.204)	-	(0.182)	-	(0.191)
Research setting uses	−0.113	−0.107	0.006	0.028	0.002	0.012
field observation	(0.100)	(0.100)	(0.063)	(0.065)	(0.064)	(0.065)
Dichotomous outcome	−0.125	−0.103	-	-	-	-
	(0.093)	(0.095)	-	-	-	-
Indicators for disciplines	yes	yes	yes	yes	yes	yes
Indicators for domains	yes	yes	yes	yes	yes	yes
p-value for hypothesis “triggers system 1” = “engages system 2”	-	0.484	-	0.368	-	0.727
p-value for hypothesis “triggers system 1” = “bypasses both systems”	-	0.269	-	0.926	-	0.365
p-value for hypothesis “engages system 2” = “bypasses both systems”	-	0.140	-	0.710	-	0.279
R-squared	0.187	0.177	0.068	0.062	0.070	0.061
Number of papers	101	101	174	174	174	174
Number of observations	N = 507	N = 507	N = 965	N = 965	N = 965	N = 965

This table reports the results of ordinary least squares (OLS) regressions with the outcome variable in the column heading and the explanatory variables in the leftmost column. When a nudge belongs to more than one category, the variables recording whether the nudge triggers system 1, whether the nudge engages system 2, and whether the nudge bypasses both systems take fractional values (1/2 and 1/2, or 1/3 and 1/3 and 1/3). The second, fourth, and sixth regressions omit a constant term. An observation’s weight in the regression is proportional to the inverse of the number of observations associated with the same article. Standard errors are clustered by article. The symbols ⁺, *, and ** indicate statistical significance at the 10%, 5%, and 1% levels, respectively.

Fig. 2.

Predicted Cohen’s d for nudges that do and do not use automaticity. This figure shows predicted values and 95% confidence intervals from an ordinary least squares (OLS) regression for which the outcome variable is Cohen’s d. The right-hand-side variables are an indicator for whether the nudge uses automaticity, indicators for academic discipline, indicators for domain of application, an indicator for whether the research setting involves field observation, and an indicator for whether the outcome variable is dichotomous. An observation’s weight in the regression is proportional to the inverse of the number of observations associated with the same article. Standard errors are clustered by article. For the predictions, all right-hand-side variables are held fixed at their means except for the indicator for whether the nudge uses automaticity, which takes a value of either zero or one.

In column 2 of Table 5, we replace the indicator for whether the nudge automates some aspect of the decision-making process with variables capturing the extent to which a nudge falls into the three major categories identified by Beshears and Gino (2015), namely nudges that trigger system 1, nudges that engage system 2, and nudges that bypass both systems.⁹ The control variables, weighting procedure, and clustering of standard errors are the same as in column 1. Nudges that bypass both systems tend to have a higher Cohen’s d than nudges that trigger system 1 and nudges that engage system 2, but these differences are not statistically significant.

Columns 3 and 4 of Table 5 use the same regression specifications as columns 1 and 2, respectively, except for two changes. First, the outcome variable is an indicator for whether the estimated treatment effect is statistically significant at the 5% level. Second, we drop the indicator for whether the treatment effect is estimated for a dichotomous variable, which was only relevant for the Cohen’s d regressions because the calculation of Cohen’s d differed for continuous versus dichotomous variables. Columns 5 and 6 of Table 5 are identical to columns 3 and 4, respectively, except the outcome variable is an indicator for whether the estimated treatment effect is statistically significant at the 10% level. Columns 3–6 indicate that nudges involving automaticity are associated with a 13.9 percentage point (15.5 percentage point) higher likelihood of having a treatment effect that is statistically significant at the 5% (10%) level, but there are not systematic differences in the likelihood of a statistically significant treatment effect across the three major categories of nudges.

The results in Table 5 are robust to alternative specifications. The results in columns 1 and 2 are nearly identical if we winsorize Cohen’s d at the 1st and 99th percentiles. Also, for our baseline regressions, Cohen’s d is treated as missing and dropped from the sample if the estimated nudge treatment effect was the opposite of the predicted direction and either statistically significant or marginally statistically significant. However, the results in columns 1 and 2 are similar if we include those observations in the regression sample either coded as having a positive value for Cohen’s d or coded as having a Cohen’s d of zero. In addition, when the nudge treatment effect is estimated for a dichotomous outcome variable and the mean of the outcome variable for the control group is unknown, our baseline regressions assume that the treatment effect is symmetric around 0.5 for the purposes of calculating Cohen’s d. The results in columns 1 and 2 are similar if we instead drop these observations from the regression sample. We have also explored alternative methods of weighting observations. If we assign each observation a weight proportional to the inverse of the squared standard error of the Cohen’s d estimate, some observations receive extremely high weights, but if we winsorize those weights at the 20th and 80th percentiles, the results in columns 1 and 2 are similar.¹⁰ We view equal weighting of the nudge treatment effect observations in our data set as inappropriate because of the disproportionately high weight assigned to articles that include a large number of small-sample studies. Nonetheless, the results in Table 5 are similar if we pursue this strategy, except that the coefficient estimate on the indicator for using automaticity in column 1 is no longer statistically significant or marginally statistically significant (even though the coefficient estimate of 0.160 is close to the coefficient estimate of 0.193 in Table 5). Finally, in columns 3–6 of Table 5, logistic regressions deliver the same conclusion as ordinary least squares regressions.

Overall, the results in Table 5 suggest that nudges involving automaticity have a larger impact, both as measured by Cohen’s d and as measured by the likelihood of a statistically significant effect, than nudges that do not involve automaticity. This finding aligns with prior theorizing that automating some aspect of the decision-making process, without relying on individuals to act differently, tends to be more powerful for driving changes in outcomes than encouraging individuals to change their decision-making process, which requires attention and effort (Beshears & Gino, 2015). Of course, our conclusions here are tentative because they are based only on correlational evidence. While we control for some potential confounding factors, the type of nudge that was applied in a given setting was not randomly assigned, so the relationships that we find may be driven by omitted variables, such as how difficult it is to change individuals’ behavior in that setting. Moreover, while it is likely that certain types of nudges are consistently more impactful than other types of nudges regardless of context, a critical ingredient for impact may also be the match between the type of nudge and the nature of the behavior to be changed.

It is also important to note that nudges may be beneficial even if they do not exert an effect on the targeted outcome variable. Nudges that simplify the choice environment sometimes have this property. For example, in an experimental study of mutual fund investment decisions, a nudge that simplified the presentation of information regarding the funds did not change participants’ investment choices, but participants needed less time to reach decisions of the same quality, suggesting that the nudge was valuable (Beshears, Choi, Laibson, & Madrian, 2011). Finally, when judging the desirability of a nudge, it is not only the nudge’s effect size that is important, but also the cost of implementing the nudge. We return to this point in Section 4.5.

Notwithstanding the caveats above, the evidence in Table 5 suggests that there are important differences in impact across various categories of nudges. Developing a better understanding of this heterogeneity would be a valuable pursuit for the empirical literature on nudging.

4.2. Using field research and laboratory research as complementary methods

Field research and laboratory research have both played major roles in the empirical literature on nudging. As indicated in Table 3, 44.0% of the nudge treatment effects in our data set were estimated using field experiments and the observational analysis of field data, and 56.0% of effects were estimated using laboratory-style methods (laboratory experiments, online experiments, and surveys). We argue that research on nudging is most compelling when it uses field and laboratory approaches as complementary methods for understanding why and under what circumstances nudges influence behavior.

Field and laboratory approaches are complementary because they build on each other. Laboratory techniques are often deployed more quickly and easily, so they are well suited for exploring the viability of new nudge interventions that have not been tested previously. Research in the field can then build on laboratory findings by focusing on nudges that have shown promise in laboratory settings. This step is important because even when evidence from laboratory experiments documents that a given nudge can alter individual decisions, evidence from the field may reveal that the size of the effect of the nudge diminishes in field settings, perhaps to the point where it is too small to be of practical use or too small to detect. In some cases, the discrepancy between the laboratory and field results may be due to the fact that laboratory experiments can amplify the effect of a nudge by stripping away relevant contextual factors, which would have otherwise also influenced the decision at hand. Field research, on the other hand, incorporates those contextual factors, and those factors may drown out the effect of the nudge by drawing attention away from the stimuli by which the nudge is delivered or by triggering alternative ways of thinking about the choice that compete with the decision-making process promoted by the nudge.

To be clear, it is not a critique of laboratory techniques to point out that they tend to strip away contextual factors and are less suited for measuring the size of nudge effects in field settings. The objective of laboratory-style research is often to demonstrate the existence of an effect, not to estimate the size of an effect, and the demonstration of an effect in the absence of contextual factors strengthens the case for the generalizability of the effect across contexts. Furthermore, when field data indicate that a nudge has a small effect or no effect (or even has the opposite of the intended effect) in a particular context, it is often logistically challenging or prohibitively costly to collect additional field data that might shed light on the psychological mechanism behind the result, in part because many nudges deployed in field settings operate through multiple mechanisms simultaneously (Hauser, Gino, & Norton, 2018). Laboratory approaches, which tend to have more carefully controlled experimental manipulations and more targeted measurement of key constructs, are valuable for understanding the decision-making processes that underlie variation in nudge effects across domains and for developing predictions regarding the situations in which a nudge will or will not have the desired effect. We emphasize that the combination of field and laboratory approaches is necessary for providing guidance to practitioners who use nudges to address managerial and public policy problems.

The literature on financial returns aggregation and investment portfolio decisions is one example of the importance of using both field and laboratory methods. Some laboratory studies demonstrate that showing individuals aggregated returns, such as returns aggregated over a long time horizon instead of a short time horizon, increases individuals’ willingness to invest in risky assets (Gneezy & Potters, 1997; Thaler, Tversky, Kahneman, & Schwartz, 1997). These findings are consistent with the hypothesis that individuals are myopically loss averse—they evaluate the outcomes of risky investments within a narrow frame, such as a short time window, and when they decide whether or not to invest in a risky asset, a possible loss of a given size within this frame is weighted more heavily than a gain of the same size within this frame (Benartzi & Thaler, 1995). Under this hypothesis, aggregating returns widens the frame, and for most common risky asset return distributions, a wider frame reduces the likelihood of experiencing a loss, which in turn increases willingness to invest in risky assets. Despite the laboratory evidence indicating that an aggregation nudge can change risk-taking behavior, Beshears, Choi, Laibson, and Madrian (2017a) show that the nudge has no effect in a setting that uses mutual funds instead of laboratory gambles and a one-year experimental period instead of a short laboratory session. This evidence suggests that embedding the nudge within a field context drowns out the effect of the nudge. To better understand the mechanisms underlying this result, Beshears et al. (2017a) conduct follow-up laboratory experiments and document that changes to the return distribution or to the amount of time between the moment when a portfolio is chosen and the moment when returns are viewed eliminates the effect of the nudge. These findings do not invalidate the hypothesis that myopic loss aversion drives portfolio choices, but they do suggest that nudges based on this hypothesis are unlikely to impact portfolio choice in field settings.

The literature on nudges that provide information about peer behavior is another example of the importance of using both field and laboratory methods. Field experiments have demonstrated that telling individuals that a particular behavior is common among their peers can make those target individuals more likely to engage in the behavior themselves. This result has been documented in domains including residential energy conservation (Allcott, 2011), contributions to an online community (Chen, Harper, Konstan, & Li, 2010), towel reuse in hotels (Goldstein, Cialdini, & Griskevicius, 2008), voting (Gerber & Rogers, 2009), job choice (Coffman, Featherstone, & Kessler, 2017), food consumption (Sparkman & Walton, 2017), tax compliance (Halls-worth, List, Metcalfe, & Vlaev, 2017; Bott, Cappelen, Sørensen, & Tungodden, 2020), and fare evasion at train stations (Ayal, Celse, & Hochman, in press). In the context of an employer-sponsored retirement savings plan, however, Beshears, Choi, Laibson, Madrian, and Milkman (2015) show that telling certain non-participating individuals about the high participation rates of their peers reduces the target individuals’ likelihood of enrolling in the plan, and that this effect is stronger when the participation rate among peers is (plausibly exogenously) higher. The effect is driven by individuals who have low incomes compared to their peers, suggesting that the nudge backfires because it interacts with individuals’ concerns regarding relative economic standing, triggering feelings of discouragement and thereby lowering plan enrollment rates. Additional work, including work in laboratory environments, documents that peer information nudges may have no effect or may backfire because individuals’ preferences may not depend on perceived social norms, because disclosing the low prevalence of an undesirable behavior may unintentionally make the behavior seem acceptable, because the reference group whose behavior is reported may be interpreted as dissimilar from the target individuals, or because individuals may misremember the peer information in self-serving ways (Bicchieri & Dimant, 2019; Dimant, van Kleef, & Shalvi, 2020).¹¹ Thus, the combination of field and laboratory evidence provides a nuanced understanding of when and how managers and policy makers might successfully use peer information nudges.

4.3. Studying the long-run effects of nudges

In our data set of nudge treatment effects, 17 out of the 174 articles collect follow-up data to estimate at least one treatment effect over a longer time horizon than the initial time horizon used to examine the impact of the nudge;¹² 24 out of the 174 articles estimate at least one treatment effect for an outcome variable that measures the cumulative impact of a series of actions in a field setting (one type of outcome variable that captures a long-run effect); and 36 out of the 174 articles fall in either of the first two categories. Thus, at most 21% of articles attempt to assess the long-run effect of a nudge. Future empirical research on nudging should devote more attention to whether and how nudges exert long-run effects, as nudging is more compelling as a managerial and policy tool if it can generate long-lasting changes in outcomes instead of only short-run effects.

One particularly intriguing area for future research is the mechanisms by which nudges can help individuals develop habits that lead to sustained changes in behavior. Habits are patterns of behavior characterized by repeated automatic performance of an action or set of actions in response to a routinely occurring cue to act (Wood & Runger, 2016). Unfortunately, in many situations, encouraging individuals to create entirely new routines for engaging in a desirable behavior is unlikely to succeed in promoting habit formation because such new routines are frequently disrupted by competing demands on individuals’ time and attention (Beshears, Lee, Milkman, Mislavsky, & Wisdom, 2019). Success may require devising routines that dovetail with existing recurring events in individuals’ lives, but this hypothesis deserves further study.

Given the challenges of helping individuals develop beneficial habits, another promising direction for future research is to explore how nudges can prompt individuals to make one-time, up-front investments that generate a long series of future changes in outcomes. For example, Brandon et al. (2017) document that a nudge showing individuals how their home energy consumption compares to their neighbors’ consumption induces investment in energy-saving technologies, such as energy-efficient appliances. These one-time investments lead to persistent reductions in energy use.

Of course, even if a nudge has a short-run effect but not a long-run effect, it may nonetheless be valuable if it can be applied repeatedly and continues to have a short-run effect each time it is applied. Some past work has identified nudges that, upon repeated application, have an effect each time they are applied (see, e.g., Allcott & Rogers, 2014; Altmann & Traxler, 2014; Beshears et al., 2013), but additional work on this issue would be valuable.

4.4. Measuring the effect of nudges on non-targeted outcomes

Out of the 174 articles in our data set of nudge treatment effects, only 12 measure the effect of the nudge on an outcome variable that could offset the treatment effect on the focal outcome variable and that is in the same domain as the focal outcome variable. Only three articles measure the effect of the nudge on an outcome variable that could offset the treatment effect on the focal outcome variable and that is in a different domain from the focal outcome variable. There are strong theoretical reasons to believe that nudges can impact outcomes other than the targeted outcomes. Nudges often operate by changing the choice architecture of one decision-making setting to prompt individuals to change their behavior in that setting. Nudges do little to shift the fundamental costs and benefits of various possible courses of action, so outside of the setting with altered choice architecture, individuals may engage in behavior that compensates for the changes induced by the nudge.

Investigations of nudge effects on non-targeted outcomes do not always find evidence of compensatory behavior. Beshears, Choi, Laibson, Madrian, and Skimmyhorn (2019) study the effects of automatic enrollment into an employer-sponsored retirement savings plan on both targeted and non-targeted outcome variables. Consistent with prior research, they find that automatic enrollment increases contributions to the savings plan, the targeted outcome. One concern with automatic enrollment is that it might also have the unintended consequence of increasing financial distress if individuals have more retirement savings but diminished financial resources for repaying debt. However, Beshears, Choi, Laibson, et al. (2019) do not find that automatic enrollment increases financial distress or credit card and other non-secured debt.¹³

In other situations, however, nudges have effects on non-targeted outcomes that undermine their effects on targeted outcomes. For example, Wisdom, Downs, and Loewenstein (2010) study interventions designed to decrease caloric intake at a fast-food sandwich chain. One nudge intervention decreased the convenience of ordering high-calorie sandwiches and thereby reduced the likelihood with which participants selected high-calorie sandwiches. An unintended consequence was that the nudge simultaneously increased the caloric content of the side dishes and drinks selected by participants, perhaps due to participants’ feelings that their virtuous sandwich choices should be rewarded with indulgent side dish and drink choices. The increase in caloric intake from side dishes and drinks entirely offset the decrease in caloric intake from sandwiches.

The dearth of attempts to gauge the effects of nudges on non-targeted outcomes is a glaring omission in the empirical nudge literature, as such unintended consequences can partially offset, entirely eliminate, or even reverse the benefits that nudges deliver on a targeted dimension. In addition to documenting effects on non-targeted outcomes, future research on nudging should attempt to understand the circumstances under which nudging does and does not induce compensatory behavior that undermines the intended impact of the nudge.

4.5. Placing a nudge in the context of other nudges and other interventions

Managers and policy makers often deploy multiple interventions simultaneously. Empirical research on nudging must therefore help managers and policy makers understand how a given nudge fits within a broader constellation of strategies for changing behavior. Towards this end, Benartzi et al. (2017) point out that an important metric by which nudges should be judged is their impact relative to their cost. These authors show across four policy domains that relative to traditional interventions such as financial incentives, nudges tend to deliver greater impact on targeted behaviors per dollar spent. Going forward, evaluations of nudge interventions should place greater emphasis on calculating the costs associated with nudges in order to facilitate comparisons for managers and policy makers who are choosing among interventions.

Furthermore, it is important for managers and policy makers to have evidence on the interactive effects among nudge interventions and other interventions. Out of the 174 articles in our data set of nudge treatment effect estimates, 48 articles (27.6%) feature at least one test of the interaction of a nudge with some other intervention (whether the other intervention is a nudge or not). More research along these lines is needed to provide guidance as to the optimal mix of interventions for achieving managerial and policy objectives.

In some cases, nudges act as complements to other interventions. For example, a nudge that streamlines the process of applying for financial aid for attending college is complementary to existing financial aid programs and can increase college enrollment (Bettinger, Long, Oreopoulos, & Sanbonmatsu, 2012). In other cases, nudges are substitutes for other interventions. For example, in employer-sponsored retirement savings plans, both automatic enrollment and employer matching contributions (contributions from the employer that are deposited in employee accounts contingent upon employees’ own contributions) are intended to increase plan participation. These two interventions are likely to be substitutes. In the absence of automatic enrollment, an employer match of $0.25 per dollar of employee contributions is estimated to increase plan participation rates by approximately 10 percentage points relative to not offering a match (Papke, 1995; Basset, Fleming, & Rodrigues, 1998).¹⁴ When a plan automatically enrolls employees, an employer match of $0.25 per dollar of employee contributions is estimated to increase plan participation rates by no more than 5–6 percentage points relative to not offering a match (Beshears, Choi, Laibson, & Madrian, 2010), an increase that is substantial but smaller than the increase in the absence of automatic enrollment.

An understanding of the extent to which interventions serve as complements or substitutes for each other is a valuable input to managerial and policy decisions. For example, in the case of retirement savings plans, employers with automatic enrollment may wish to direct financial resources away from employer matching contributions, a substitute for automatic enrollment, towards other programs that promote retirement security, such as employer contributions to employee accounts that are not contingent on employee contributions.

5. General discussion and conclusions

In this paper, we discussed examples of past research on nudging and analyzed a data set of 174 articles in the empirical literature on nudging in order to assess progress to date in this literature and to highlight key challenges for future research on the topic. We documented that many types of nudges, as studied by scholars in several different academic disciplines examining data in field and laboratory settings, have succeeded in changing behavior in a wide range of domains of application. We argued that future research on nudging should place greater emphasis on (1) seeking to identify the types of nudges that tend to be most impactful, (2) using field-based methods and laboratory-based methods as complementary approaches, (3) examining the long-run effects of nudges, (4) considering the effects of nudges on non-targeted outcomes, and (5) studying nudges within the context of a suite of nudges and other interventions designed to change a particular set of behaviors. Many organizations have started to use nudging as a technique for changing behavior, but in order for nudging to take its place alongside traditional interventions (such as financial incentives) in the standard toolkit of managers and policy makers, research on nudging will need to devote much more attention to the five issues above.

As discussed in Section 4, results from the handful of research articles that have begun to address the five issues outlined above suggest that some nudges that are currently considered powerful methods for changing behavior may prove to be less effective than previously thought. For example, nudges that have a large short-run impact may have a negligible long-run impact, necessitating repeated exposure to the nudge in order to generate a meaningful cumulative impact. Nudges that succeed in changing a targeted behavior may simultaneously induce offsetting changes in non-targeted behaviors.

Even when nudges are deemed effective after they have been evaluated in a more comprehensive fashion, it is important to note that their impact on outcomes is often modest. Consider two of the most widely known nudges: automatic enrollment in an employer-sponsored retirement savings plan and personalized reports comparing the recipient’s energy consumption to the energy consumption of the recipient’s neighbors. When Beshears, Choi, Laibson, et al. (2019) evaluated the implementation of automatic enrollment at a large employer, they estimated that automatic enrollment increased cumulative retirement plan contributions during the first four years after an employee’s hire date by only 4.1% of first-year annualized salary, a measurable change but not one that on its own can eliminate the risk that a household will experience a drop in its standard of living at retirement (Munnell, Hou, & Sanzenbacher, 2018). Similarly, Allcott (2011) estimated that personalized reports containing information about neighbors’ electricity utilization reduced energy consumption by 2%, a notable effect but not one that on its own can reduce carbon emissions to the point where the consequences of global climate change are entirely mitigated.

The observation that the effects of nudges tend to be modest does not imply that psychological factors are unimportant in determining behavior. To the contrary, the fact that it is difficult to change behavior in, for example, the retirement savings domain is consistent with a pervasive role for psychological factors—a nudge that boosts savings in one narrow context is easily swamped by the operation of psychological factors driving overspending in all other contexts that an individual encounters. Nor does the observation that the effects of nudges tend to be modest imply that managers and policy makers should abandon the choice architecture approach to influencing behavior. No single policy or intervention should be expected to resolve a major societal problem on its own—substantial progress requires many policies and interventions with modest effects all pushing in the right direction. Furthermore, nudges may have modest effects, but they also have small costs of implementation. On the basis of impact per unit of cost incurred, nudges are in fact highly valuable tools for changing behavior (Benartzi et al., 2017).

The modest effect sizes of most nudges do imply, however, that it would be unwise to ignore the possibility of implementing other types of interventions merely because a nudge has been deployed to address a given problem (Bhargava & Loewenstein, 2015; Hagmann, Ho, & Loewenstein, 2019; Loewenstein & Chater, 2017). In moving beyond nudges, of course, managers and policy makers must be prepared to accept interventions that fall short of the nudge definition of influencing behavior without limiting choice or meaningfully changing financial incentives. In some cases, even outright prohibition of certain options may be appropriate. Consider the case of restrictions on pre-retirement withdrawals from retirement savings accounts. In the United States, such withdrawals are widely available in exchange for payment of a 10% tax penalty, and they are available penalty-free under special circumstances. In other countries, such as the United Kingdom, such withdrawals are prohibited entirely except in extreme situations (Beshears, Choi, Hurwitz, Laibson, & Madrian, 2015). If the population includes both individuals who have self-control problems that cause them to undersave and individuals who do not have self-control problems (Augenblick, Niederle, & Sprenger, 2015; Beshears et al., 2020), simulations indicate that a policy prohibiting pre-retirement withdrawals leads to large welfare gains for the individuals with self-control problems and very small welfare losses for the individuals without self-control problems, a tradeoff that a policy maker may be willing to make (Beshears, Choi, Clayton, et al., 2019).

When managers and policy makers contemplate the range of possible interventions they might implement, they should consider how the interventions affect consumer welfare, a challenging question when individuals’ actions do not necessarily reflect their best interests (see Allcott & Kessler, 2019; Beshears et al., 2008a; Bernheim & Rangel, 2009; Bernheim, Fradkin, & Popov, 2015). A further challenge is that while nudges are expressly intended as tools for improving individual welfare, managers and policy makers who embrace this role for nudges must recognize that profit-maximizing firms may have an incentive to exploit individuals’ biases for their own gain (see Baker & Wurgler, 2013; Beshears, Gino, Lee, & Wang, 2016; Heidhues & Koszegi, 2018; and Malmendier, 2018). The designs of interventions should anticipate and account for the ways in which such firms may undermine efforts to improve individual welfare. A complete discussion of formal welfare analysis and strategic interactions with profit-maximizing firms is beyond the scope of this paper, but we briefly highlight a key ethical tension that emerges when contrasting nudge interventions and traditional interventions such as financial incentives. On one hand, traditional interventions may be attractive because they are transparent in their attempts to influence individuals’ decisions, whereas nudge interventions, especially those that trigger system 1 or bypass both systems, influence individuals’ decisions in ways that those individuals may not fully recognize and understand. On the other hand, nudge interventions may be attractive because they help individuals who would otherwise have difficulty making wise decisions and because they do not restrict the choices of individuals who make wise decisions on their own (Camerer et al., 2003; Thaler & Sunstein, 2003, 2008). Traditional interventions, in contrast, often restrict the choices of the latter group in order to help the former group. This tension is one that managers, policy makers, and society at large must grapple with.

In summary, the empirical literature on nudging has established that choice architecture techniques can succeed in changing behavior in many managerial and policy-relevant settings. This paper has outlined future directions that research in this area should pursue in order to make nudges part of the standard toolkit of managers and policy makers.