Priming Behavior: A Meta-Analysis of the Effects of Behavioral and Nonbehavioral Primes on Overt Behavioral Outcomes

Abstract

Past meta-analyses of the effects of priming on overt behavior have not examined whether the effects and processes of priming behavioral or nonbehavioral concepts (e.g., priming action through the word go and priming religion through the word church) differ, even though these possibilities are important to our understanding of concept accessibility and behavior. Hence, we meta-analyzed 351 studies (224 reports and 862 effect sizes) involving incidental presentation of behavioral or nonbehavioral primes, a neutral control group, and at least one behavioral outcome. Our random-effects analyses, which used the correlated and hierarchical effects model with robust variance estimation (Pustejovsky & Tipton, 2021; Tanner-Smith et al., 2016), revealed a moderate priming effect (d = 0.37) that remained stable across behavioral and nonbehavioral primes and across different methodological procedures and adjustments for possible inclusion/publication biases (e.g., sensitivity analyses from Mathur & VanderWeele, 2020; sensitivity analyses from Vevea & Woods, 2005). Although the findings suggest that associative processes explain both the effects of behavioral and nonbehavioral primes, lowering the value of a behavior weakened the effect only when the primes were behavioral. These findings support the possibility that even though both types of primes activate associations that promote behavior, behavioral (vs. nonbehavioral) primes may provide a greater opportunity for goals to control the effect of the primes.

Will a reminder of money, God, sex, or exposure to an evocative image of the American flag affect behavior? If so, through which psychological mechanisms? Will priming behavioral concepts such as achieve or action affect behavior through the same mechanisms as priming money or national symbols? Considering the effects of priming money or God along with the effects of priming achieve and action brings up important considerations. On the one hand, values and attitudes associated with broad concepts can influence behavior (Ferguson, 2007; Fishbein & Ajzen, 1974; Hepler & Albarracín, 2014). For example, exposure to the concept of money has been shown to reduce prosocial behavior (Capaldi & Zelenski, 2016; Vohs et al., 2006), and presentation of the color red has been shown to impair intellectual performance (Bertrams et al., 2015; Maier et al., 2008). On the other hand, William James, one of psychology’s earliest founding fathers, described how the (conscious) mental representation of a behavior, not a color, causes behavior to be enacted (Bargh & Chartrand, 1999; James, 1890). But are there any differences in the effects of behavioral or nonbehavioral primes, and what are the mechanisms underlying each type of influence?

Despite the theoretical importance of priming effects, their reliability is often questioned. Failed replications (e.g., Corker et al., 2020; Doyen et al., 2012; Harris et al., 2013) have raised skeptical voices like:

So it all seems pretty clear. I have no reason to believe in this effect (i.e., money-priming effect). And, to the extent it is happening, the effect could vary: it could be positive in some scenarios, negative in others, large in some places, small in others, etc. No evidence for any sort of universal effect; the explanations all devolve into contextual stories, which tell us nothing more than what we already knew, which is that lots of factors influence individual behavior and attitudes.

(Andrew, 2016, para. 5)

Therefore, answers to the question will priming concepts such as money or national symbols exert the same effect as priming behavioral concepts such as achieve or action? are best provided with methods that can statistically assess the robustness of the effect.

One approach to systematically examining a scientific phenomenon is to estimate the effect meta-analytically, thus circumventing the limitations of specific paradigms and study conditions as well as capitalizing on a vast literature to draw conclusions (Albarracín et al., 2018; Borenstein et al., 2001; Glass, 1976). A summary of the existing meta-analyses of priming effects appears in Table 1. Among these synthesis efforts, a meta-analysis by DeCoster and Claypool (2004) showed that trait priming affects impressions of other people. For example, priming participants with hostility-related words such as rude and stab increases perceptions of how hostile somebody is (Bargh & Pietromonaco, 1982). In a related area, a meta-analysis by Van den Bussche et al. (2009) showed that semantic priming, in this case subliminal, affected the semantic processing and interpretation of subsequent stimuli. For example, being presented with the word cat facilitated subsequent processing of the word dog (Marcel, 1983). More relevant to the subject of this article, other meta-analyses have examined the effect of priming on behavior. A meta-analysis by Shariff et al. (2016) showed that priming religion (e.g., through the word God) promoted prosocial behavior, and a meta-analysis by Lodder et al. (2019) showed that priming money (e.g., through the presentation of banknotes) influenced not only how people thought of money but also task performance and prosocial behavior. A meta-analysis by Weingarten et al. (2016a) demonstrated that priming behaviors (through words such as run and make) had robust effects on a variety of behaviors (e.g., anagrams, reaction times, food consumption, and product choices), and a meta-analysis by Chen et al. (2021) showed that priming achievement (e.g., through words like win and success) influences organizational behaviors such as job performance, creativity, persistence, and unethical behavior. Weingarten et al. (2016a) also considered the degree to which the observed effects were direct, associative, or goal-mediated. They concluded that valued primes elicited stronger effects, particularly in the absence of a satisfaction opportunity, suggesting goal mediation.

Table 1.

Summary of Existing Meta-Analyses of Priming Effects

Short reference	Type of priming	Type of behavioral outcome	Mean effect size [95% CI]	k	Heterogeneity indice (s)	Findings and conclusions about publication bias
DeCoster and Claypool (2004)	Trait priming	Impression formation	Assimilation: d = 0.35 [0.30, 0.41]	45	Assimilation: Qw = 81.07 (p < .001)	Not assessed
			Anchoring: d = −0.51 [−0.63, −0.39]	11	Anchoring: Qw = 29.14 (p = .001)
			Correction: d = −0.68 [−0.82, −0.55]	9	Correction: Qw = 33.18 (p < .001)
Van den Bussche et al. (2009)	Semantic priming	Semantic interpretation	d = 0.80 [0.60, 1.00]	88	Var(between-study) = 0.15 Var(between-condition) = 0.10	Nonsignificant correlation between sample size and effect size
Shariff et al. (2016)	Religious priming	Prosocial behavior	d = 0.40 [0.34, 0.46]	92	Qb(df = 24) = 74.42	Trim-and-fill and p-curve reflected significant religious priming after adjustment
Lodder et al. (2019)	Money priming	Prosocial behavior, performance, persistence, and so forth	d = 0.31 [0.26, 0.36]	246	I2 = 81.3%, τ2 = 0.117	Significant effects for published and unpublished reports but not preregistered ones
Weingarten et al. (2016a)	Behavioral priming	Performance, food consumption, enacted choice of product, etc.	d = 0.35 [0.29, 0.41]	352	I2 = 62.5%	Trim-and-fill and p-curve reflected true effects of behavioral priming
Chen et al. (2021)	Achievement priming	Performance, persistence, creativity, (un)ethical behavior	d = 0.44 [0.36, 0.52]	34	Not reported	Not assessed

Note. d = Cohen’s d; k = number of effect sizes included; CI = confidence interval; Q_w = within-factor homogeneity; p = statistical significance; Var = estimation of variance; Q_b = between-factor heterogeneity.

Despite these important past meta-analytic efforts (see Table 1), no prior synthesis has shed light on the question of whether priming behavioral or nonbehavioral concepts is more effective at shaping behavior. Moreover, none of the prior efforts has been sufficiently comprehensive to investigate this question. Shariff et al.’s (2016) and Lodder et al.’s (2019) meta-analyses included only research on priming nonbehavioral concepts, whereas Weingarten et al. (2016a) included only word primes that were closely connected to a behavior or a goal (i.e., behavioral primes). Although Chen et al.’s (2021) meta-analysis included both behavioral and nonbehavioral primes, they specifically focused on supraliminal achievement priming, which led them to consider a small set of priming studies (i.e., only 23 studies and 34 effect sizes). In contrast, a comprehensive meta-analysis to address the differences between behavioral and nonbehavioral primes must include both of these types of primes and provide a comprehensive look at the literature. Filling this gap was one of the objectives of the present meta-analysis.

Besides the degree of coverage necessary to address our question, one common concern about past meta-analyses of priming is insufficient examination of publication bias and other inclusion biases (Vadillo et al., 2016; van Elk et al., 2015). Despite converging evidence about the robustness of different types of priming effects from multiple meta-analyses (Chen et al., 2021; Lodder et al., 2019; Weingarten et al., 2016a), the methods to assess different types of inclusion biases have become ever more sophisticated and require regular reexamination of the evidence (Furuya-Kanamori et al., 2020; Lin & Chu, 2018; McShane et al., 2016; Schuch et al., 2016). Consider the following criticism of Weingarten et al.’s (2016a) meta-analysis:

A meta-analysis on behavioral priming effects suggests a meta-analytic effect size of d = 0.35. … As you would expect, all publication bias alarm bells go off in the study. Regrettably, the bias detection is not state of the art. E.g., after trim-and-fill, authors conclude this analysis is “suggesting a significant effect after accounting for publication bias.”

(Lakens, 2020)

In addition to reexamining inclusion bias, replicating meta-analyses is also important (Bakker et al., 2012; John et al., 2012; Lakens et al., 2016; Moher et al., 2009; Smaldino & McElreath, 2016; Valentim, 2019). In the spirit of contributing such a replication and answering a new theoretical question, we meta-analyzed 867 effect sizes¹ obtained from 230 published and unpublished articles conducted in the United States and internationally. Priming methods included various forms of subliminal and supraliminal presentation of verbal or visual stimuli that were either closely connected to a goal or behavior (e.g., succeed) or more broadly evocative without denoting any behavior (e.g., God). Our main objective was to estimate the average size of the priming effect on behavioral measures, both across and within behavioral and nonbehavioral primes and as a function of different methodological features (e.g., verbal or visual modality of priming, see also Chen et al., 2021; use of funnel debriefing, see also Weingarten et al., 2016a; and social desirability). Furthermore, we compared the effectiveness of behavioral and nonbehavioral primes and tested factors that could theoretically moderate effects through mediation of goal activation (e.g., goal value and expectancy, delay, and opportunity for satisfaction) to better understand the impact and underlying mechanisms of behavioral and nonbehavioral primes. We also aimed to assess the presence of publication/selection bias with multiple methods.

Priming Behavioral and Nonbehavioral Concepts

In 1890, William James coined the term ideomotor action to refer to the possibility that consciously representing a behavior may spontaneously activate the behavior in a person. It was not until 1996 that Bargh, Chen, and Burrows, building on James’ notion, proposed that priming could influence human behavior in an automatic manner (Bargh et al., 1996). Bargh et al. (1996) tested their theory with three experiments. In the first experiment, embedding words related to politeness within a sentence-scrambling task lengthened the time participants waited before interrupting an experimenter who was ostensibly distracted in a conversation with a prior participant. In the second experiment, words related to the concept elderly, also embedded in a sentence-scrambling task, slowed down the speed at which participants walked. In the third experiment, subliminal exposure to faces of Black males made participants react more aggressively when they were asked to restart the study due to an ostensible computer error. Taken together, these experiments provided support for the notion that exposure to a concept could lead to behaviors associated with that concept.

Despite the impact of Bargh et al.’s (1996) experiments on priming research and on social psychology more generally, the effects of priming behavioral concepts (e.g., rude and polite) have never been distinguished from the effects of priming nonbehavioral concepts (e.g., elderly and black people). Bargh et al. (1996) primed the behavioral concept politeness in the first experiment but used nonbehavioral concepts associated with stereotypes in the second (i.e., elderly) and third experiments (i.e., black people). Moreover, in the following decades, social psychologists proceeded to prime concepts that were not connected to behaviors, including God, money, sex, and the U.S. flag, all of which were shown to have influences on what participants did. As mentioned, God priming has been shown to reduce self-interest and thus promote prosocial behaviors like helping others and making charitable donations (Shariff et al., 2016; Shariff & Norenzayan, 2007). Money priming has been shown to weaken interpersonal connection and thus reduce prosocial behaviors as well (Capaldi & Zelenski, 2016; Lodder et al., 2019; Vohs et al., 2006). Sex primes have been shown to motivate positive relational behaviors such as self-disclosure and self-sacrifice (Gillath et al., 2008) but promote (e.g., in males) aggressive behaviors sometimes (Mussweiler & Förster, 2000). The national flag has been shown to increase conservative, confirmatory strategies of information selection (Scherer, 2014), as well as behaviors that benefit the well-being of a country (e.g., paying taxes; Chan, 2019).

Before addressing the question of whether the effects of behavioral and nonbehavioral primes vary, we define behavioral primes as introducing stimuli that are closely connected to a behavior or a goal concept and can thus provide clear behavioral guidance for an upcoming task. Behaviors are often primed verbally, by presenting adverbs, verbs, or nouns, which are useful to convey a specific behavioral routine or clear manner of conduct, or visually, by presenting stimuli that directly depict a behavior or goal achievement. As one example of verbal primes, Bargh et al. (1996) primed the concept rude with adverbs such as impolitely and bluntly. As examples of visual primes, Foulk et al. (2016) primed the concept rude by letting the participants witness the experimenter speaking to a participant in a rude manner, and Bipp et al. (2017) primed the concept achievement with a picture of a man standing on the top of a mountain. In all these cases, the primes were well poised to provide clear guidance to participants’ upcoming behavior in the experimental situation. For example, in Bargh et al.’s (1996) first experiment, either the word rudely or the word politely was introduced before measuring the time participants took to interrupt the researcher who was ostensibly talking to a previous participant. In this case, rudely or politely could easily guide participants’ behavior (i.e., accelerating the interruption) after the prime.

We define nonbehavioral primes as stimuli that are not closely connected to a specific behavior or goal and thus cannot provide direct behavioral guidance during the following task. For example, Shariff and Norenzayan (2007) primed the concept God with words such as spirit, divine, and sacred and then measured participants’ money allocation in a dictator’s game as a measure of prosocial behavior. Visually, Caruso et al. (2017) primed the concept money by showing participants a faint image of $100 bills in the background of the instruction screen. In these cases, the concepts of God and money did not provide direct guidance but could evoke relevant associations that could still influence subsequent performance.

Despite the absence of past research directly comparing the effects of behavioral and nonbehavioral primes, similar distinctions have been made and deserve attention. To begin, behavioral primes are concrete concepts that can be subsumed under larger, more abstract concepts (Devine, 1989). Accordingly, activating the abstract concepts may activate all the subcomponents (i.e., all-or-none logic, Anderson, 1982; Hayes-Roth, 1977), including concrete behavioral concepts capable of eliciting actual behavior. Although our conceptualization dovetails well with Devine’s, we do not believe that the distinctive feature of behavioral and nonbehavioral concepts is level of concreteness/abstraction. Rather, concreteness/abstractness is orthogonal to the distinction between behavioral and nonbehavioral concepts. For example, Barack Obama is more concrete than Black people but neither provides direct guidance on how to act. Similarly, winning the game and graduating are more concrete than achieve but each can provide guidance on how to act. Therefore, although concrete behavioral concepts (e.g., help) can be components of larger nonbehavioral abstract concepts (e.g., religion), the behavioral and nonbehavioral distinction cannot be reduced to the distinction between concrete and abstract concepts.

Social psychology also provides hints as to whether the two types of primes may differ. An all-or-none logic (Anderson, 1982; Devine, 1989; Hayes-Roth, 1977) operates when an abstract concept activates its concrete subcomponents. Consequently, priming nonbehavioral concepts like religion (e.g., through the word church) should exert similar effects on behaviors as priming behavioral concepts like prosociality (e.g., through the word help). After all, past research has shown that activating nonbehavioral concepts associated with a stereotype can activate that stereotype as well as behaviors associated with it (Devine, 1989). For example, in an experiment from Devine (1989), participants who were primed with words stereotypically associated with Black people (e.g., athletic, musical, and jazz) became more vigilant during a subsequent task that required tracking stimuli that could appear at unpredictable times and on varying locations of a computer screen.

Differences between the two types of primes are, however, possible. On the one hand, clearer and more specific behavioral goals may yield better performance (Latham & Piccolo, 2012), and specific intentions are well known to predict specific behaviors better than are general intentions (Aizen et al., 2019; Ajzen & Fishbein, 1980). Consistent with this possibility, Chen et al.’s (2021) meta-analysis showed that context-specific primes (e.g., priming achievement with a picture of an employee making a phone call in a call center and measuring job performance in a call center; Latham & Piccolo, 2012) have stronger effects than more general primes (e.g., priming achievement with a picture of a woman winning a race and measuring job performance in a call center; Latham & Piccolo, 2012). Presumably then, behavioral primes, which are more specific and proximal to behavior, may have a stronger impact than nonbehavioral ones.

On the other hand, more diffuse, general primes may activate more abstract identification of behaviors (i.e., action identification theory; Vallacher & Wegner, 1989), which gives people more leeway to justify a behavior, thus increasing the likelihood of executing the behavior. Moreover, more diffuse, general primes simply have wider semantic associations and may thus activate more concepts than may specific primes. For instance, a reminder of money may bring to mind the concepts of business, success, extravagance, and competition; feelings of achievement and enjoyment; and the behaviors of shopping, spending, investing, traveling, and gambling. Similarly, a reminder of sex may activate the positive concepts of romance, happiness, love, and commitment, as well as the negative concepts of dominance, violence, and jealousy. Therefore, relative to behavioral concepts, nonbehavioral ones could have stronger effects on behavior by activating the accessibility of multiple concepts.

A broad umbrella of associations may produce stronger priming effects in two ways. First, a broader concept may flexibly impact responses that are adaptive to different situations (Koestler, 1968). According to the situated inference model of priming (Loersch & Payne, 2011), the same primes often allow for different responses to a question raised by the task or experimental situation. For example, in a series of studies conducted by DeMarree and Loersch (2009), participants primed with the African American stereotype (vs. not) expressed more aggression (i.e., choosing a more powerful hot sauce as punishment for their partner) and those primed with a Buddhist monk (vs. not) expressed less aggression (i.e., choosing a less powerful hot sauce as punishment). However, these effects were only present when participants were instructed to think about themselves as opposed to their best friends. These studies thus showed that the same prime may elicit different responses depending on contextual factors and the participant’s interpretation of the connection between the primed concept and the task at hand (see also Albarracín et al., 2011; Senay et al., 2010). In the context of our analysis, nonbehavioral primes may provide more associations that, being applicable to more contexts, may influence behavior more than behavioral primes do.

Another reason why nonbehavioral primes may have stronger effects is that a nonbehavioral prime is a less obvious source of influence than is a behavioral concept. As is well established, people who identify an external source of influence often attempt to counter the influence (Bargh & Hassin, 2022; Brehm, 1966; Herr et al., 1983; Higgins et al., 1985; Lombardi et al., 1987; Schwarz & Clore, 1983; Sparrow & Wegner, 2006; Wegener & Petty, 1995; Weinberger & Stoycheva, 2020). In the context of priming, people who detect a priming attempt might try to suppress acting in response to the prime. Thus, a prime is more likely to influence behavior when people do not realize that they are being primed and instead attribute the concept to their own internal thought processes (Albarracín et al., 2011; Loersch & Payne, 2011). In this light, a person who is asked to unscramble a sentence containing the word achievement might perceive a connection between the prime and an intellectual task immediately following the prime. In contrast, a person who is presented with the letter “A” might not detect a connection with the subsequent request to perform an intellectual task. Likewise, a person who is asked to draw through a labyrinth with the reminder of money might have difficulties in identifying and avoiding the influence of the money prime.

Psychological Mechanisms Underlying Behavioral and Nonbehavioral Priming

Social psychologists have always been interested in understanding the mechanisms of behavioral influences (Ajzen & Fishbein, 1980; Fishbein & Ajzen, 2010; Glasman & Albarracín, 2006; Zanna et al., 1980) and debates over the mechanisms of priming are important for psychology as a whole (Bargh et al., 1988, 1996; Bargh & Gollwitzer, 1994; Dijksterhuis & Aarts, 2010; Srull & Wyer, 1979; Weingarten et al., 2016b). One key process distinction concerns a direct, perception–behavior link, as opposed to a goal-mediated account of priming (Bargh et al., 2001; Bargh & Gollwitzer, 1994; Chartrand & Bargh, 1999; Dijksterhuis & Bargh, 2001; Weingarten et al., 2016a, 2016b). Even though both accounts assume ultimate influences on behavior, the perception–behavior link resembles a conditioned response in which a stimulus can trigger behavior without the person ever forming a goal (e.g., Bargh et al., 1996). As mentioned, the idea of conscious ideation leading to behavior was first proposed by William James (see also Fiske, 1992) and then extended to stimuli that could influence behavior outside of conscious awareness (Dijksterhuis & Bargh, 2001). This perception–behavior link account (Bargh et al., 1996; Dijksterhuis & Bargh, 2001) assumes that the residue of conscious thought (e.g., priming and casual exposure to a behavior) activates motor associations that are semantically connected to that thought and have the potential to influence behavior. Encountering a stimulus may, however, also instill a goal previously activated in the presence of the same stimulus (Bargh & Gollwitzer, 2001). That is, when a stimulus and a goal have been coactivated a sufficient number of times, an association is likely to be formed. Once this association is in place, the stimulus may prime the goal, and the goal may in turn yield a behavioral response (Bargh et al., 2001; Cohn et al., 2014; Fitzsimons & Bargh, 2003). Both of these mechanisms could underlie the effects of behavioral and nonbehavioral primes.

Relative to perception–behavior effects, goals offer flexibility and better reflect people’s motivations and values (Bargh et al., 2001). Whereas the perception–behavior link requires pairing the same stimulus and the same response to ultimately produce priming, a goal permits more flexible responses to novel contexts. Accordingly, priming a cooperation goal has been shown to lead to more cooperation even when people have not encountered the exact task in the past (Bargh et al., 2001). Also, the effects of primes often depend on current or chronic goals and values of the person who is primed. For example, priming a brand of iced tea increases iced tea consumption among participants who are thirsty, but not among participants who are not (Strahan et al., 2002; Veltkamp et al., 2011). Furthermore, achievement primes produce better problem-solving among participants who chronically value achievement but have a counterproductive effect among those who do not (Hart & Albarracín, 2009).

In this article, we were interested in studying if the effects of behavioral and nonbehavioral primes involve both perception–behavior effects and goal mediation. Although the perception–behavior link and goal activation are likely to exist on a continuum rather than as distinct entities (for similar notions, see Bargh, 1994), goal mediation may be inferred from responses to (a) goal value, (b) goal expectancy, and (c) satisfaction. With respect to value, valuing an end state is part of the definition of a goal. Therefore, valuing the behavior may lead people to act more consistently with their goal than not valuing the behavior. For example, Förster et al. (2005) conducted an experiment in which participants received either $1 or $0.05 to find a target in a search task. As shown by the study findings, while the goal was active, the accessibility of concepts related to the goal was higher for participants in the higher value condition (i.e., $1 condition). Accordingly, a prime may have stronger effects on behavior when people value the goal as well as weaker effects when people devalue a goal. For example, Weingarten et al.’s (2016a) meta-analysis showed that priming effects were significantly weaker when goal value was lower (d = −0.056, 95% CI [−0.248, 0.136], k = 23) compared to a control value condition (d = 0.326, 95% CI [0.270, 0.386], k = 293). This effect implies that presence of a disincentive (e.g., perceiving high inequality in a dictator game, Zhu, 2012), for example, may motivate people to consciously control and suppress their behavior even if the behavior did not require a goal to be activated (Bargh, 1994).

Goal expectancy (Förster et al., 2005; Locke & Latham, 2002) is defined as the perceived probability of achieving a goal and is closely related to task difficulty such that more difficult tasks have lower expectancy of success than easier tasks. A higher goal expectancy is often linked to a higher motivation to achieve the goal because people are prone to pursue objectives they believe they can achieve (Förster et al., 2005; Locke & Latham, 2002). An early meta-analysis of task difficulty and goal pursuit (Wood et al., 1987) found that easier tasks led to more goal activation and goal pursuit than did difficult tasks, thus supporting the proposed positive link between goal expectancy and behavior. However, other work suggests that it is difficult goals that strengthen goal pursuit (Heath et al., 1999; Locke & Latham, 1990; Stajkovic et al., 2006). Unsurprisingly, Weingarten et al.’s (2016a) meta-analysis failed to find associations between priming effects and goal expectancy, suggesting the need for further research in this area.

With respect to the temporal dynamic of priming effects, a mere perception–behavior link would predict that each encounter with the stimulus should increase activation of the concept (Bargh et al., 1996). In contrast, a goal account sometimes assumes that an opportunity for satisfaction may eliminate priming effects (Bargh et al., 2001; Cesario et al., 2006; Förster et al., 2005). For example, Albarracín et al. (2008) conducted an experiment in which they primed participants with either an action goal or an inaction goal and then counted the number of thoughts about a text participants generated as the behavioral outcome. In this study, participants were randomly assigned to complete an active task (i.e., doodling) or an inactive task (i.e., resting) in between the priming task and the behavioral measurements. This study showed that those primed with an action goal were more active in the thought-generation task when they were assigned to rest (vs. doodle) in between the priming task and the dependent measure. Likewise, those primed with an inactive goal were more inactive in the thought-generation task when they were assigned to doodle in between the priming task and the dependent measure. However, meta-analytic evidence on the effect of temporal delay on priming effects is mixed. On the one hand, Weingarten et al.’s (2016a) meta-analysis found that priming effects persisted more in the absence of an intervening opportunity for goal satisfaction. On the other hand, Chen et al.’s (2021) meta-analysis failed to replicate this pattern. Moreover, recent studies (Möschl et al., 2019) showed that intention deactivation after completion is modulated by a multitude of factors (e.g., the specific paradigm) that either foster a rapid deactivation or lead to continued retrieval of completed intentions. These mixed findings thus call for further meta-analytic efforts in this area.

In this meta-analysis, we used goal value, goal expectancy, and satisfaction opportunity to test for the perception–behavior link and goal mediation accounts of goal priming. That is, in addition to observing if priming behavioral or nonbehavioral concepts triggered a stronger or weaker behavioral effect, we examined the role of goal value, goal expectancy, and satisfaction opportunity for each type of prime. If priming behavioral and nonbehavioral concepts both influence behavior via the perception–behavior link, goal value, and perhaps goal expectancy, and satisfaction opportunity may exert little effect. In contrast, if priming behavioral concepts heightens goals more than does priming nonbehavioral concepts, then goal value, goal expectancy, and/or satisfaction opportunity may only exert effects when behavioral concepts are primed.

The Current Meta-Analytic Review

This meta-analysis sought to estimate the mean effect size of priming behavioral and nonbehavioral concepts and to quantify inclusion bias in each of these sets and the literature as a whole. We began with the 323 effect sizes in Weingarten et al.’s (2016a) database and proceeded to expand the search. First, rather than only including studies that primed behavioral concepts with words, as Weingarten et al. (2016a) did, we included both behavioral and nonbehavioral primes, as well as primes presented either verbally or visually (see also Chen et al., 2021). We obtained an effect size to represent the effect of a prime on consistent behavior, such as polite behavior in response to a politeness prime (Bargh et al., 1996) or aggressive behavior in response to a sexual prime (Mussweiler & Förster, 2000). We then estimated the weighted mean effect and its heterogeneity, both across the board and separately for behavioral and nonbehavioral primes. Inclusion bias was carefully analyzed following best practices and multiple methods. We then analyzed the possible moderating effects of goal value, goal expectancy, and presence of a satisfaction opportunity (i.e., presence of a relevant filler task) in studies priming behavioral and nonbehavioral concepts. We also used moderator analyses to explore the effect of different priming characteristics, such as prime content (e.g., achievement, money, religion, and stereotypes) and prime modality (e.g., visual vs. verbal), different task characteristics (e.g., social desirability of the outcome), and design features such as type of inclusion of covariates, exclusion of participants, different control primes, and presence of funneled debriefing (e.g., Ciani & Sheldon, 2010). The current meta-analysis was preregistered on Open Science Framework (OSF), and the protocol can be found at https://osf.io/e2z6u.

Method

Literature Search

Our literature search process used Weingarten et al.’s (2016a) meta-analysis as a starting point and relied on similarly thorough processes. The databases we searched included PsycInfo; ProQuest Dissertations and Theses; the Reproducibility Project Open Science Framework; https://PsychFileDrawer.Org; Communication Abstracts; Advances in Consumer Research, which as the proceedings of the Association for Consumer Research; the Foreign Doctoral Dissertations Database of the Center for Research Libraries (http://www.crl.edu); PubMed; the Education Resources Information Center; and the Databases of the Institute of Psychology Information for the German-Speaking Countries (http://www.zpid.de). The general search for all databases included the logic (prime OR priming OR primed) AND (behavior OR goal OR action OR motivation). The searches of APA PsycInfo, ProQuest Dissertations and Theses, the Reproducibility Project Open Science Framework, https://PsychFileDrawer.Org, Communication Abstracts, and Advances in Consumer Research also included NOT (“semantic prim!”) NOT (“affect! prim!”). The PsycInfo search added the logic AND me.exact (“Empirical Study”) AND pop.exact (“Human”) to narrow the search to empirical reports with human participants. Article searches were performed iteratively in 2014, 2017, 2018, and 2020, with the last search conducted in February 2020. We also made requests for unpublished data to 320 authors and sent requests to the list hosts of the Society for Personality and Social Psychology, the Society for Consumer Psychology, and the Society for Experimental Social Psychology.

Inclusion Criteria

To be eligible, studies must meet the following inclusion criteria.

Incidental Priming Rather Than Overt Directions

Studies must activate concepts through incidental priming. They must not provide direct behavioral instructions, which involve explicitly telling participants how they should behave on the measured outcomes (e.g., Brunyé & Taylor, 2009). For example, a study measuring the length of time spent working on a puzzle after completing a scrambled sentence task designed to prime achievement (vs. neutral) was eligible. In contrast, a study measuring the completion of a puzzle after being explicitly told to work on the puzzle until completion was not. Regular instructions that guide participants through the task and do not directly influence the outcome measures (e.g., asking participants to walk down a certain hallway when measuring the walking speed) would not lead to study exclusion, but an instruction telling the participants to walk at their normal walking speed would lead to exclusion if walking speed was the outcome measure.

Have a Controlled Experimental Design

Studies must involve an experimental manipulation in which participants were randomly assigned to priming or control conditions. An example would be a study in which participants were randomly assigned to a creativity priming condition (through a creativity task) or a control condition (i.e., no creativity task; Sassenberg et al., 2017).

Presence of a Nonopposite Control Group

To assess the effect of the prime relative to a neutral baseline, studies must include a control prime that is not the semantical opposite of the experimental prime. For example, Chartier et al. (2020) was excluded because it compared the effect of an action prime and an inaction prime on performance in a test similar to the Scholastic Aptitude Test. Similarly, Zhong and Liljenquist (2006) and Fayard et al. (2009) were both excluded because they compared the effect of an ethical prime and an unethical prime on the likelihood of taking antiseptic wipes. Eligible, nonopposite control groups involved neutral word primes, neutral reading passages or neutral imagination task primes, nonsense word primes (e.g., a nonsense word like gub), unrelated goal primes (e.g., sexual arousal primes in the experimental condition and happiness primes in the control condition; Maner et al., 2007), and no control task.

Presence of Eligible Prime

Studies must have a prime that was presented as either words, visual images, or a writing/reading/imagination task (e.g., imagining counting banknotes; Mok & De Cremer, 2016).

A Behavioral Dependent Variable

Outcome measures must assess enacted behavior (e.g., task performance, amount of money donated), instead of intentions or other self-report measures. Additionally, outcome measures could not be measures of the accessibility of the primed concept (e.g., an Implicit Association Test) even though such measures are performance-based. For example, the flag priming studies reported in Klein et al. (2014) were excluded because their dependent variable was a political attitude rather than a behavior. When it was unclear whether a measure represented accessibility of a concept or enacted behavior, the research team discussed it to reach consensus.

Adequate Statistics

Studies must present adequate statistics for calculating an effect size (e.g., Ms and SDs/SEs, F statistics, t statistics). If adequate statistics were missing, we contacted the authors for the original data and only excluded the studies if we got no responses.

Based on the above inclusion criterion, we included 81² out of the 84 reports originally included in Weingarten et al. (2016a) and 149 new reports. All 230 reports were included, and the moderators of interest were coded.

Moderator Coding

In addition to calculating effect sizes, we coded variables that could potentially moderate the priming effect, including (a) prime type (i.e., behavioral and nonbehavioral concept), (b) manipulation of goal value, (c) manipulation of goal expectancy, (d) delay/opportunity for satisfaction, (e) proportion of experimental primes over the total stimuli presented, (f) priming modality (i.e., verbal or visual), (g) content of prime, (h) abstractness of prime (dropped due to low interrater reliability), (i) liminality, (j) social desirability of the outcome, (k) type of dependent measure, (l) task flexibility (dropped due to low interrater reliability), (m) type of control group, (n) presence of a funneled debriefing, and (o) presence of a task prior to priming. We also recorded descriptive characteristics of the study, including (a) year, (b) country, and (c) source type (published article, dissertation/thesis, or working article/unpublished data). Part of our data came directly from Weingarten et al.’s (2016a) meta-analysis. As reported in their article, the interrater reliability was sufficient for all coded variables (κs > .6, αs > .8). Our team in charge of coding moderators not assessed by Weingarten et al. (2016a) as well as all moderators in new reports (i.e., 229 studies and 525 effect sizes) consisted of two main authors who have received adequate training in meta-analysis coding and achieved sufficient interrater reliability³ for most variables (κs > .6, αs > .8). The exceptions were abstractness of prime and task flexibility (κ < .6), which were therefore excluded from all analyses. The interrater reliability for each moderator can be found in Table 2.

Table 2.

Descriptive Statistics

Variable	Type of statistic	Summary statistics
	Descriptive characteristics
Year (α = 1)	M (SD)	2009.82 (6.27)
	Mdn	2010
Country (κ = 1)	U.S. count (%)	493 (57.19)
	Non-U.S. count (%)	369 (42.81)
Publication status (κ = 1)	Published count (%)	664 (77.03)
	Unpublished count (%)	198 (22.97)
	Theoretical moderators
Type of priming (κ = 0.80)	Behavioral count (%)	497 (57.66)
	Nonbehavioral count (%)	365 (42.34)
Goal value (κ = 1)	Nonmanipulated count (%)	769 (89.21)
	Lower count (%)	8 (0.93)
	Higher count (%)	85 (9.86)
Goal expectancy (κ = 1)	Nonmanipulated count (%)	783 (90.84)
	Lower count (%)	35 (4.06)
	Higher count (%)	44 (5.10)
Delay/satisfaction (κ = 1)	No delay count (%)	708 (82.13)
	Delay without satisfaction count (%)	124 (14.39)
	Delay with satisfaction count (%)	30 (3.48)
	Prime characteristics
Proportion of prime (α = 1)	M (SD)	0.84 (0.23)
	Mdn	1
Modality of priming (κ = 1)	Verbal priming count (%)	662 (76.80)
	Visual priming count (%)	200 (23.20)
Content of prime (κ = 1)	Achievement, intelligence, or efficacy count (%)	260 (30.16)
	Common behaviors count (%)	206 (23.90)
	Money, marketing, or finance count (%)	66 (7.66)
	Morality, God, or prosociality count (%)	110 (12.76)
	Motivation count (%)	68 (7.89)
	Sex, gender, or romantic behavior count (%)	40 (4.64)
	Stereotype count (%)	112 (12.99)
Liminality (κ = 1)	Subliminal count (%)	136 (15.78)
	Supraliminal count (%)	726 (84.22)
	Task characteristics
Social desirability (κ = 0.65)	Neutral count (%)	583 (67.63)
	Negative count (%)	37 (4.29)
	Positive count (%)	242 (28.08)
Dependent measure (κ = 1)	Performance count (%)	299 (34.69)
	Other count (%)	563 (65.31)
	Other design features
Control type (κ = 1)	Neutral word count (%)	493 (57.19)
	Other count (%)	369 (42.81)
Funneled debriefing (κ = 0.72)	Absent count (%)	536 (62.18)
	Present count (%)	326 (37.82)
Task before priming (κ = 1)	Absent count (%)	605 (70.19)
	Present count (%)	257 (29.81)

Note. α = Cronbach’s alpha; κ = Cohen’s kappa.

Theoretical Moderators

Priming Behavioral and Nonbehavioral Concepts.

Primes were categorized as concerning (a) behavioral concepts if the priming stimuli provided directional guidance for the following task or (b) nonbehavioral concepts if the priming stimuli did not provide directional guidance for the following task. More specifically, primes were categorized as priming behavioral concepts if (a) they directly primed a behavior or a goal (e.g., to run, to win, to be fast) or (b) they primed a trait or a value that provided directional guidance for the following task (e.g., priming equality and measuring inequality of profits during a game, Ganegoda et al., 2016). Primes were categorized as nonbehavioral if (a) they represented people or stereotypes of groups of people (e.g., athletes, professors, elderly people), (b) they primed a trait or a value that did not provide directional guidance for the following task (e.g., priming cuteness and measuring indulgence choice, Scott & Nenkov, 2016), (c) they primed an institution, object, or entity connected to a value without clear directional implications for the following task (e.g., God, national flag), or (d) they primed a common object without directional implications for the following task (e.g., money, food, and cigarettes).

Manipulation of Goal Value.

We coded goal value into three categories: (a) no manipulation, (b) higher goal value (e.g., offering a greater monetary reward or preselecting participants who value the goal), or (c) lower goal value (e.g., preselecting participants who do not value the goal). For example, the low-achievement motivation condition in Hart and Albarracín (2009) was coded as lower in goal value because this group of participants was selected to have lower achievement motivation. Correspondingly, the high-achievement motivation condition in Hart and Albarracín (2009) was coded as higher in goal value because this group of participants was selected to have higher achievement motivation. Another example is Seitchik and Harkins’ (2014) research, in which goal value was manipulated upward by telling participants that their performance would be evaluated by others (vs. no manipulation).

Manipulation of Goal Expectancy.

We coded whether the studies involved objective manipulations of goal expectancy into three categories: (a) no manipulation, (b) higher expectancy (i.e., the goal is easier to attain in one condition), or (c) lower expectancy (i.e., the goal is more difficult to attain in one condition). The manipulation of goal expectancy could involve changes to the objective difficulty of the task. For example, Stajkovic et al. (2006) manipulated the difficulty of the task by asking participants to List 4 or 12 uses of a commonly used object (e.g., a wire coat hanger). Other studies manipulated goal expectancy by simply altering participants’ perceptions of their likelihood of attaining the task, without manipulating actual difficulty. For example, Capa et al. (2011) primed the goal of studying as well as the presence or absence of positive words to manipulate participants’ perceptions of goal attainability.

Delay/Satisfaction Opportunity.

We coded for whether the study involved delay and opportunities for goal satisfaction between the prime task and the behavioral measurement. All studies were coded into one of the three categories: (a) no delay (i.e., no filler task between prime task and behavioral measurement), (b) delay without satisfaction opportunity (i.e., inclusion of a filler task that was not relevant to the primed goal), or (c) delay with satisfaction opportunity (i.e., inclusion of filler task that was relevant to the primed goal). For example, the research by Van Tongeren et al. (2018) was coded as involving a delay without satisfaction opportunity because their studies primed participants with the concept of superhero and then included an irrelevant personality scale as the filler task between the prime and the measure of helping behavior. The studies by Lowery et al. (2007) were coded as involving a delay with satisfaction because the researchers primed intelligence and then gave participants a practice exam before an actual statistics exam (i.e., the dependent measure). In this case, the practice exam acted as a filler task, which was highly relevant to the primed goal, and therefore provided an opportunity for goal satisfaction.

Exploratory Moderators

Prime Characteristics.

We coded the content of each prime into seven broad categories: (a) achievement, intelligence, or efficacy; (b) common behaviors (i.e., action, inaction, diet, or socializing); (c) money, marketing, or finance; (d) morality, God, or prosociality; (e) motivation (e.g., priming hedonics motivation, Ramanathan & Menon, 2002); (f) sex, gender, or romantic behavior; and (g) stereotypes. For example, a fairness prime was put in the morality category because fairness can be viewed as a dimension of moral judgment (Zdaniuk & Bobocel, 2013), and a jealousy prime was put in the sex category because the emotion of jealousy is most often seen in romantic relationships (Maner et al., 2007).

We also coded whether each prime entailed (a) verbal stimuli (e.g., words, statements, and writing or reading tasks) or (b) visual images (e.g., pictures and imagination tasks). Verbal primes could involve (a) scrambled sentence tasks, (b) anagrams, (c) lexical decision tasks, or (d) writing or reading prompts designed to evoke a goal. Visual primes could involve (a) foveal presentation, (b) parafoveal presentation, or (c) imagination tasks. When a prime entailed both verbal and visual stimuli (e.g., a magazine with both texts and images, a no-smoking sign with both the image and the text no smoking), we coded modality based on which aspect was dominant. For example, Papies and Hamstra (2010) primed a dieting goal with a poster showing a weekly recipe. Although posters were usually considered visual stimuli, this particular prime was coded as verbal priming because the verbal information (i.e., the recipe) was more salient than the visual features (i.e., the graphs, the color, etc.). As another example, Boyland et al. (2017) primed a dieting goal with a television commercial. Although the commercial included both verbal and visual information, we coded it as visual given that the visual modality is more dominant for television.

The primes were also coded based on their liminality and dosage. Primes could be either (a) subliminal (e.g., parafoveal images) or (b) supraliminal (e.g., word completion tasks). The proportion of primes was coded by calculating the ratio of prime-related stimuli to total stimuli during the priming task. The numbers of each type of stimuli were usually explicitly reported in the articles. Hence, if a study reported priming achievement with the words win, bread, chair, goal, and window, the proportion of prime stimuli was recorded 0.4 because two of the five words (i.e., win and goal) are primes, whereas the others are fillers.

Task Characteristics.

We coded the nature and social desirability of the behavioral measure. Common types included task performance with scoreable answers (e.g., anagrams), persistence on a task, reaction time, consumption of food or drink, enacted choices regarding products, spending, donations, and volunteering. Social desirability was coded into three categories: (a) neutral outcomes (e.g., chocolate consumption; Taylor et al., 2014), (b) socially desirable outcomes (e.g., amount of donation; Gasiorowska et al., 2016), or (c) socially undesirable outcomes (e.g., cheating behavior; Kleinlogel et al., 2018).

Other Design Elements.

We also coded for type of control condition, the use of funneled debriefing in a study, and presence of a task prior to priming. Common categories of control groups included (a) neutral word controls (e.g., neutral words like hat when the prime was win; Hart & Albarracín, 2009), (b) nonsense controls (e.g., religion primes through words like God vs. nonsense words like gub; Lin et al., 2016), (c) priming unrelated goals (e.g., sexual arousal prime vs. happiness prime; Maner et al., 2007), (d) neutral reading or neutral imagination tasks (e.g., thinking about normal daily activities as a control when the experimental priming involved thinking about indulgent activities; Salerno et al., 2014), and (e) no-task controls (e.g., Dijksterhuis & Van Knippenberg, 1998). We also coded for whether a study included funneled debriefing to evaluate participants’ awareness of the true purpose of the experiment. This method involves starting with the most abstract questions and then gradually funneling down to more specific questions to test participants’ suspicion (e.g., Ciani & Sheldon, 2010). Last, we also coded for whether the study asked the participants to complete any task prior to the priming task. An example of such task could be found in Milyavsky et al. (2012), where the researchers asked the participants to do a coloring task prior to the priming task as a pretest and did the same task for a second time after the priming task as the behavioral measurement.

Descriptive Characteristics.

In addition to the methodological factors described above, we also recorded descriptive characteristics including (a) year, (b) country, and (c) source type (published article, dissertation, working article or unpublished data, etc.), which allowed us to determine if a report was published or unpublished.

Assessment of Study Quality

Following National Institute of Health’s study quality assessment tools for controlled intervention studies (https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools), we assessed the overall quality of our included studies with respect to three major factors, including use of random assignment, participants’ blindness to the hypotheses and condition assignment, and signs of p-hacking (Wicherts et al., 2016).

Use of Random Assignment

Use of random assignment was one of our inclusion criteria. Therefore, all included studies involved random assignment. Use of randomization was thus adequate in all included articles.

Blindness to the Hypotheses and Assignment

With regard to whether participants were blind to the hypotheses and assignment, 37% of included studies used funneled debriefing at the end of the study to evaluate participants’ awareness of the study’s purpose and hypotheses. When this procedure is used, any participants aware of the true purpose of the experiments or the hypotheses is excluded from data analyses, thus retaining participants blind to the study hypotheses. As for studies without funneled debriefing, it was hard to determine whether participants remained totally blind to the hypotheses. However, the priming effect is relatively subtle effect, especially when the primes were delivered subliminally. Therefore, the chance that participants became aware of the hypotheses was relatively low in our included studies.

Signs of p-Hacking

Conducting research involves a variety of choices in designing a study, collecting and analyzing data, and reporting results, and the flexibility and arbitrariness of these choices constitute researchers’ degrees of freedom, which can lead to potential p-hacking (Simmons et al., 2011; Wicherts et al., 2016). To assess likelihood of p-hacking among the included studies, we counted the number of articles that (a) reported preregistration of the studies, (b) included covariates (i.e., control variables) in their analyses, and (c) reported excluding participants from analyses. To further investigate the influences of these differences in the reported effect sizes, at the suggestion of an anonymous reviewer, we ran exploratory moderator analyses for each of these three factors and then conducted our key analyses both controlling and not controlling for them.

Meta-Analytic Strategy

Calculation of Effect Sizes and Effect Size Variances

The research team coded eligible articles and calculated an effect size from each eligible group comparisons (i.e., one priming group vs. one control group) for each behavioral measure. For example, four effect sizes were obtained from Study 3 in Da-Costa (2015) because the study involved two different primes (i.e., a performance prime vs. a control prime and a mastery prime vs. a control prime) and two dependent measures (i.e., time spent on task and number of attempts on the task). Effect sizes were calculated as (M₁ − M₂)/SD_pooled. We recorded a positive effect when the prime produced a more prime-consistent response than the control group. Usually, it resulted from M_Priming − M_Control. However, when the prime was expected to reduce a behavior (e.g., morality priming was expected to inhibit unethical behavior, Welsh & Ordóñez, 2014), the sign of the effect size was reversed. If there was not enough information to calculate effect sizes directly from means and standard deviations or proportions (e.g., standard deviations not reported), we derived the effects sizes from t tests, F statistics, or confidence intervals. Hedges and Olkin’s (1985) factor was applied to all effect sizes to correct for small sample bias. We followed the formula in Lipsey and Wilson (2001) to calculate the variances for between-subject effects and then weighted the effect sizes by their inverse variances using the Metafor package in R. We used random-effects models throughout due to the significant heterogeneity observed in the data. To ensure the accuracy of the extracted effect sizes, a second coding team extracted the data for effect sizes. The effect sizes obtained in this fashion were then compared with those obtained from the first team. Any discrepancies were examined to identify which extracted effect size was correct and the data were analyzed before and after effect sizes were corrected. The pattern and significance of the key analyses changed minimally.

Weighted Mean Effect Sizes

Weighted mean effect sizes were first estimated following Hedges and Olkin’s (1985) methods. However, including all eligible effect sizes from each study minimizes data loss but violates the assumption of statistical independence. Specifically, effect sizes from the same study likely share contributors of variance (e.g., location of the lab, ambient temperature, disposition of the experimenter), and effect sizes from the same sample may have even more similarities because they belong to the same participants. Two methods to this problem are robust variance estimation (i.e., RVE model, Tanner-Smith et al., 2016) and multilevel modeling (i.e., MLM model, Assink & Wibbelink, 2016). On the one hand, the RVE model is better at estimating fixed effects (e.g., estimation of mean effect) while accounting for the dependency among correlated effect (CE) sizes but is limited to dealing with heterogeneity at multiple levels. On the other hand, the MLM is better at dealing with effects within a hierarchical structure and providing a hypothesis testing tool for heterogeneity parameters but is weaker at dealing with dependent effect sizes coming from the same studies (Tanner-Smith et al., 2016). A recent method (Pustejovsky & Tipton, 2021), the correlated and hierarchical effects (CHE) model, relies on RVE and combines the strengths of the two models above to jointly address correlated effects and a hierarchical structure. Therefore, we relied on the CHE model under RVE to obtain the weighted mean effect of priming but also reported results using the traditional CE model under RVE and MLM.

Inclusion Bias Analyses

Following recommendations on best practices in inclusion bias analyses (Lin et al., 2018; McShane et al., 2016), we examined publication bias and small-study bias using multiple methods, including (a) introducing publication status as a moderator; (b) introducing publication year as a moderator to test for the decline effect (Pietschnig et al., 2019); (c) funnel plots; (d) Egger’s intercept tests, with the modified measure of precision as suggested in Pustejovsky and Rodgers (2019); (e) precision-effect test and precision-effect estimate with standard errors (PET–PEESE), with the modified measure of precision; (f) sensitivity analyses from Vevea and Woods’s (2005) selection models; and (g) sensitivity analyses from Mathur and VanderWeele (2020), as explained below. We did not use p-curve and p-uniform methods as they do not perform as well as the original Hedges selection method approach in the high-heterogeneity settings that characterize meta-analyses in the social and behavioral sciences (McShane et al., 2016).

1. One intuitive way to examine publication bias is to test publication status as a moderator of effect sizes and determine whether effects sizes retrieved from unpublished reports are lower than those from published reports. If publication bias led to nonsignificant effects going to the “file drawer,” then published effects should be significantly stronger than the unpublished effects. This comparison gives a quick and intuitive sense of the bias, but neither quantifies nor corrects it.

2. One common phenomenon observed in literatures plagued with publication bias is that the effect sizes tend to decrease over time (Pietschnig et al., 2019). As stated in Fanelli et al. (2017), small, early, and well-cited studies tend to produce inflated effects and those published later tend to regress to an lower estimation if the initial studies were indeed overestimating the effect. Therefore, the time trend of effect sizes can also provide insights into the biases in a field. However, like analyses of publication status, analyses of time trends can neither quantify nor correct the bias.

3. Funnel plots show the distribution of effect sizes around their mean against their standard error. In theory, the observed effect sizes should be a representative sample of an underlying normal distribution of possible effect sizes. If this assumption is true, then the shape of the funnel plot should resemble a funnel such that more precise studies (i.e., those with smaller variances) are close to the mean effect size in the middle of the plot and less precise studies funnel away from the average effect. However, if publication bias has eliminated nonsignificant and negative effects, the funnel should take an asymmetrical shape. When the bias is one-tailed, this asymmetrical shape should show a hole on the bottom left quadrant. An example of a biased funnel plot appears in Figure 1. When the bias is two-tailed, this asymmetrical shape may show a hole on both the bottom left and right quadrants and include effects that are both negative and positive in sign. All in all, however, funnel plots serve only as an exploratory tool because they do not provide a quantitative analysis of publication bias.

4. Egger et al.’s (1997) regression test provides a quantitative assessment of the asymmetry of the effect size distribution by regressing each effect size on its precision. The significance of the slope (i.e., the coefficient on the measure of precision) indicates the funnel plot asymmetry. However, this method has been criticized for lacking intuitive interpretation (Rothstein et al., 2006), inflating Type I error under certain circumstances (e.g., when summary estimate is the natural log of odds ratio; Peters et al., 2006), and lacking a tool to correct for the bias. To address the inflating Type I error problem, Pustejovsky and Rodgers (2019) suggested that one can use a modified measure of precision that is not correlated with the effect size. As shown by the simulation reported in Pustejovsky and Rodgers (2019), an Egger’s test with the usual measure of precision had inflated Type I error rate, but the modified measure of precision corrects the problem. Therefore, in this article, we report the Egger test results with the modified measure of precision.

5. PET–PEESE is a method to estimate an effect size after ruling out the small-study effect (Stanley & Doucouliagos, 2014). PET uses a linear model and reestimates the effect size after controlling for the square root of the variances, whereas PEESE uses a curvilinear model and reestimates the effect size after controlling for the variances. This method has been criticized for inflating Type I error when the data set has large heterogeneity (Stanley, 2017). However, it has the advantage that it can be combined with MLM and RVE. When combined with RVE, it functions similarly as Egger’s sandwich test (Rodgers & Pustejovsky, 2021), which tests for the effect size distribution asymmetry using the RVE to correct for effect-size dependency.

6. Selection methods assume that the probability of publication depends on the p value of its effect size, leading to different chances that different p values will be published and therefore included in a meta-analysis (Hedges & Vevea, 1996; Vevea & Woods, 2005). As recommended in McShane et al. (2016), selection methods should be used to explore the range of estimates, instead of obtaining a single estimate. The Vevea and Wood’s sensitivity analyses allow researchers to estimate the lower boundary of the effect by calculating an adjusted effect size assuming moderate or severe selection biases of different shapes. A two-tailed selection bias assumes that studies are being placed in the file drawer if they produce neither a significantly negative effect nor a significantly positive effect, whereas a one-tailed selection bias assumes that studies are being placed in the file drawer if they fail to produce a significant positive effect. These methods perform consistently the best with the high heterogeneity that characterizes the behavioral sciences (McShane et al., 2016). However, these methods do not account for the statistical dependence among effect sizes. An alternative sensitivity analysis, which was recently proposed by Mathur and VanderWeele (2020), provides adjusted effect sizes assuming different levels of bias severity, or η, defined as the number of times an affirmative (vs. a nonaffirmative) study is likely to be published. This method also estimates the minimum η needed to nullify an observed effect. In this article, we applied and reported results using both of these new methods.

Figure 1. An Example of a Biased Funnel Plot.

Moderator Analyses

The best methods for moderator analyses when studies generate multiple effect sizes are RVE and MLM. As mentioned above, MLM provides a better tool to assess effects within a hierarchical structure and estimate heterogeneity at each level, but it does not model the dependence between correlated effects sizes coming from the same source (i.e., the same sample, the same study from the same lab). In contrast, RVE models dependence among correlated effects but is limited when dealing with heterogeneity at multiple levels of hierarchical data. Moreover, RVE has low power for moderators with uneven distributions (e.g., having 500 cases in one level but only 10 cases in another level; Tanner-Smith et al., 2016). As shall be seen, although we do have correlated effects coming from the same sample or study, that circumstance is rare. We include a total of 862 effect sizes, coming from 351 studies and 545 samples, indicating that most studies contain only one or two samples, and most samples only contain one or two effect sizes. Additionally, the distribution of our key moderators (i.e., goal value, goal expectancy, and opportunity for satisfaction) is highly uneven. Therefore, as preregistered, the dominant hierarchical structure of data and the uneven distribution of key moderators led us to use MLM (Assink & Wibbelink, 2016), instead of RVE, for all our moderator analyses. In these moderator analyses, we nested effect sizes (the Level 1 variable) within the sample from which they were obtained (the Level 2 variable), which were themselves nested within the study from which they came (the Level 3 variable), thus allowing a random intercept for each study, each sample, and each effect. Moderators were each introduced as dummy-coded variables, and the significance of the regression coefficient of each dummy-coded level was used to determine what levels of the moderators were influential. When testing interaction terms, we used model comparison to test whether the model fit was significantly better with or without the interaction term.

For exploratory moderator analyses, each moderator was entered into the regression analyses as a single predictor. For theoretical moderator analyses, we also wanted to control for effects from descriptive and methodological factors. We thus included all descriptive and methodological factors as covariates in all theoretical moderator analyses.⁴

Transparency and Openness

This article has carefully followed the Transparency and Openness Promotion Guidelines. Our efforts include but are not limited to (a) citing all data, program code, and methods in the text and the reference list; (b) sharing the data on which conclusions are based on OSF; (c) sharing the R codes needed to reproduce analysis results on OSF; (d) sharing the relevant research materials (e.g., original coding sheet) on OSF; (e) carefully following the meta-analysis reporting standard; (f) preregistering the study design and the main hypotheses on OSF; and (g) preregistering the analysis plan on OSF. The preregistered protocol of the current research can be found at https://osf.io/e2z6u. All replication materials, including the datafiles, the analyses syntax (in R), and the coding sheets can be found at https://osf.io/vhbgf/.⁵

Results

Descriptive Statistics

A summary of the descriptive statistics appears in Table 2. As can be seen, reports ranged in date from 1983 to 2020 and came from published and unpublished sources. Most included reports were journal articles (77%) published in the United States (57%) around 2010. Other reports were from a wide range of countries, such as Canada, Netherlands, German, Italy, England, France, China, and Japan, each covering less than 10% of the total reports. Among all effect sizes, 497 effects (58%) involved priming of behavioral concepts, whereas 365 (42%) involved priming of nonbehavioral concepts. Most included studies did not involve a manipulation of goal value (89%), a manipulation of goal expectancy (91%), or a filler task between priming task and measured behavior (82%). Therefore, the asymmetry in these moderators deemed RVE inadequate for moderator analyses. In terms of content, 30% primed achievement, intelligence, or efficacy; 24% primed common behaviors; 8% primed concepts relevant to money, marketing, or finance; 13% primed morality, God, or prosociality; 8% primed motivation; 4% primed sex, gender, or romantic behavior; and 13% primed stereotypes. Moreover, 77% studies presented the priming stimuli verbally, 84% used supraliminal priming, 28% concerned a socially desirable behavior, 4% concerned a socially undesirable behavior, 35% examined a performance type of behavioral outcome, 57% compared the priming condition with a neutral word control, 38% performed a funneled debriefing as an awareness check, and 30% had a task prior to the priming task. On average, 84% of the words or trials of the priming tasks were primes as opposed to fillers. Among the 224 included reports, three reported preregistration of their studies, 70 included covariates in their analyses, and 41 reported excluding participants from analyses.

Overall Average Effect Size

We included 224 reports containing 862 effect sizes (see Figure 2, for a flowchart included in the preferred reporting items for systematic reviews and meta-analyses). Following Weingarten et al. (2016a), we removed outliers (k = 6) of d above absolute 2.5 from all analyses, in addition to reporting the average effect size including those outliers as a sensitivity analysis. Due to significant heterogeneity, Q (df = 855) = 3,846.71, p < .001, we used random-effect models in all analyses. As shown in Table 3, across analyses that used different statistical models to deal with the dependence stemming from the inclusion of multiple effect sizes from the same study or from the same sample (i.e., CHE under RVE, CE under RVE, or MLM), results indicated a moderate effect size with substantial heterogeneity. Thus, these results clearly suggest the presence of a moderately sized priming effect with substantial variability across studies and contexts but still present across different methodological procedures and in analyses with or without outliers.

Figure 2. Flow of Reports in This Systematic Review.

Note. During the initial screening process, we did not record the specific reason of exclusion for each article. Therefore, we did not have the exact count of number of articles excluded during each phase or for each reason. To obtain the best estimation we could, we later rescreened a subsample of 400 articles from January 2017 to September 2017 and recorded the specific reason of exclusion for this subsample. In the current flowchart, all the numbers with an asterisk (*) are estimated counts made based on the percentage of articles excluded due to each reason in the 400-article subsample.

Table 3.

Mean Effect Size Estimators and Heterogeneity Indices

Parameter name	CHE under RVE	CE under RVE	MLM	CHE under RVE (including outliers as sensitivity analyses)
Weighted mean effect and 95% CI	0.368 [0.334, 0.402]	0.409 [0.374, 0.444]	0.366 [0.332, 0.400]	0.370 [0.336, 0.404]
Number of studies	351	351	351	351
Number of samples	545	545	545	545
Number of effects	856	856	856	862
Heterogeneity indices
I2	77.8%	68.6%	66.7%	79.4%
τ2 (study level)	<.001	.104	.001	<.001
ω2 (sample level)	.030		.052	.023
σ2 (effect level)	.130		.064	.161

Note. CHE = correlated and hierarchical effects model; RVE = robust variance estimation; CE = correlated effects model; MLM = multilevel modeling; CI = confidence interval; I² = overall amount of heterogeneity/overall amount of variance; τ² = estimation of between-study heterogeneity; ω² = estimation of between-sample heterogeneity; σ² = estimation of between-effect heterogeneity.

Effects of Priming Behavioral and Nonbehavioral Concepts

An important goal of this meta-analysis was to compare the behavioral effect of priming behavioral versus nonbehavioral concepts. Hence, we first ran a multilevel metaregression of predicting the priming effect size on whether the study primed a behavioral or nonbehavioral concept. Our results revealed no difference (B = 0.05, SE = 0.03, p = .14) between the effects of priming nonbehavioral concepts (d = 0.394, 95% CI [0.327, 0.462], k = 361) and the effects of priming behavioral concepts (d = 0.344, 95% CI [0.300, 0.388], k = 495) on behavior. This initial result provides a hint that the processes involved in the two types of priming are largely overlapping.

Inclusion Bias Analyses

To gauge inclusion bias, we first analyzed all effects and then conducted separate analyses for priming behavioral and nonbehavioral concepts. Tables 4 and 5 summarize the publication, time, and inclusion bias analyses.

Table 4.

Estimation of Mean Effect Size Adjusted by Different Publication Bias Methods

Data set	Published versus nonpublished effects			Year of report	Unadjusted (CE under RVE, with modified precision)	Adjusted by selection methods (assuming severe two-tailed biases)	Adjusted by selection methods (assuming severe one-tailed biases)
	Published d [95% CI]	Unpublished d [95% CI]	Qb	B (SE)	d [95% CI]	d	d
All effects (k = 856)	0.39 [0.36, 0.43]	0.26 [0.18, 0.34]	10.14**	−0.003 (0.003)	0.42 [0.38, 0.46]	0.27	0.03
Behavioral primes (k = 495)	0.38 [0.33, 0.43]	0.23 [0.12, 0.33]	8.10**	−0.003 (0.004)	0.39 [0.34, 0.44]	0.24	0.02
Nonbehavioral primes (k = 361)	0.40 [0.35, 0.46]	0.32 [0.19, 0.45]	1.54	−0.003 (0.004)	0.46 [0.41, 0.52]	0.28	0.07

Note. k = number of effects; d = Cohen's d; CI = confidence interval; Q_b = heterogeneity between groups; B = unstandardized regression coefficient; SE = standard error; CE = correlated effects model; RVE = robust variance estimation. Due to the small value of the Bs and SEs for analyses on year of report, we reported three digits after decimal point.

^**p < .01.

Table 5.

Estimation of Mean Effect Size Adjusted by Mathur and VanderWeele’s (2020) Sensitivity Analyses (Two-Tailed)

Data set	Assuming different bias severity
	η = 2	η = 5	η = 100	η = 10,000
	d [95% CI]	d [95% CI]	d [95% CI]	d [95% CI]
All effects (k = 856)	0.27 [0.24, 0.30]	0.20 [0.18, 0.23]	0.15 [0.12, 0.17]	0.14 [0.12, 0.17]
Behavioral primes (k = 495)	0.26 [0.22, 0.30]	0.20 [0.16, 0.23]	0.14 [0.11, 0.18]	0.14 [0.11, 0.17]
Nonbehavioral primes (k = 361)	0.29 [0.25, 0.34]	0.22 [0.8, 0.26]	0.15 [0.11, 0.19]	0.15 [0.10, 0.19]

Note. η = The number of times more likely an affirmative study is to be published than a nonaffirmative study; k = number of effects; d = Cohen’s d; CI = Confidence interval.

Analyses of All Effects

Publication Status as a Moderator

We first ran a multilevel regression of our effect size with publication status as the predictor to determine if published and unpublished reports differed from each other. As shown in Table 4, published reports had significantly larger effect sizes than unpublished ones (Q_b = 10.14, p < .01), suggesting publication bias. However, as can be seen, the priming effect was still significant among unpublished reports.

Testing for the Decline Effect

We also regressed effect sizes on publication year using a multilevel regression to assess the trend over time. The results showed no trend over the years (B = −0.003, SE = 0.003, p = .377).

Funnel Plot

We visually inspected the funnel plot of the effect sizes against their standard error, which appears in Figure 3. As explained previously, if the distribution of effect sizes was unbiased, the plot should resemble a funnel, with studies with greater errors (assessed as smaller sample sizes) displaying greater variability on both sides of the mean effect (Sterne et al., 2006). As shown in Figure 3, the funnel plot seemed mostly symmetrical (compare with Figure 1) to the naked eye. However, statistical methods are of course more appropriate ways of determining bias.⁶

Figure 3. Funnel Plot of All Effects (d < Absolute 2.5, Variance < 0.5).

Egger’s Regression

As explained previously, merely inspecting funnel plots leads to an imperfect assessment of bias. Hence, we ran Egger’s test of asymmetry (Egger et al., 1997). Following suggestions from Pustejovsky and Rodgers (2019), we ran the Egger’s test with the modified measure of precision. The coefficient of precision was 1.15 (SE = 0.40, p = .004), which suggested significant asymmetry in the distribution of effect sizes.

PET–PEESE

We next applied the PET–PEESE method (Stanley & Doucouliagos, 2014) to adjust the effect size estimate by removing the small-study effect. We followed the guidelines provided by Stanley (2017) to first apply PET. PET was significant (B = 0.99, SE = 0.23, p < .001), suggesting a bias caused by small studies reporting large effects. After adjustment, the effect was still significant (d = 0.152, p < .001, 95% CI [0.025, 0.279], k = 856, I² = 65.4%). Because PET still suggested a nonzero true effect, we then performed PEESE. PEESE was significant as well (B = 1.63, SE = 0.42, p < .001), also suggesting a bias caused by small studies reporting large effects. Because PEESE analyses combined with RVE function similarly as the Egger’s Sandwich test (Rodgers & Pustejovsky, 2021), the significance of the variance term in the PEESE analysis also suggested asymmetry in the distribution of effect sizes after considering the dependencies among effects. After controlling for this bias, the average effect size was adjusted down to 0.291 (p < .001, 95% CI [0.217, 0.364], k = 856, I² = 66.7%) but remained statistically significant. Our PET–PEESE analyses showed evidence of small-sample bias, but the behavioral priming effect was robust to these biases.

Sensitivity Analyses

Last, we applied two different sensitivity analyses to obtain a bias-corrected estimate of our effect size. We first applied Vevea and Woods’s (2005) selection methods, which have been deemed best at estimating different types and magnitudes of bias and are most resilient to heterogeneity (McShane et al., 2016). As explained before, a two-tailed selection bias assumes that studies go to the file drawer if they produce neither a significantly negative effect nor a significantly positive effect, whereas a one-tailed selection bias assumes that studies go to the file drawer if they fail to produce a significant positive effect. Because, as shown in Figure 3, the effects in our data set go from negative to positive, assuming a two-tailed selection bias was more appropriate (Vevea & Woods, 2005). As shown in Table 4, even when assuming a severe two-tailed selection bias (see Table 1 in Vevea & Woods, 2005), the adjusted effect dropped to only 0.27, which was still a small-to-medium effect and remained statistically significant. We also report adjusted effects assuming one-tailed selection bias in Table 4 (i.e., d = 0.03, p > .05), for interested readers, though we believe that two-tailed bias assumption was more appropriate with our data set, as explained above. We then applied Mathur and VanderWeele’s (2020) sensitivity analyses, which has the advantage of being able to deal with dependent effects. We clustered our effects by the study from which they came and found that the priming effect could not be explained away (i.e., to adjust the effect estimate to be nonsignificant) even assuming the most severe magnitude of bias. As shown in Table 5, even considering an extreme level of biases (i.e., η = 10,000), the adjusted priming effect remained statistically significant. Similarly, we reported results assuming a one-tailed bias in Supplemental Table 1. Even assuming a severe one-tailed bias, the adjusted effect was still significant. Considering that Mathur and VanderWeele’s (2020) sensitivity analyses were more appropriate than Vevea and Woods’s (2005) ones given its ability to model dependency among correlated effects, our analyses thus showed that the priming effect was robust even to the most severe suppositions of bias.

Analyses of Bias Separating the Effects of Behavioral and Nonbehavioral Primes

We also aimed to examine the inclusion bias separately for the effects of priming behavioral and nonbehavioral concepts. Therefore, we plotted the funnel plot for each data set (shown in Figure 4) and reran the above analyses within each data set. As shown in Table 4, publication status was a significant predictor for the effect of priming both behavioral and nonbehavioral concepts, providing some evidence of publication bias in each case. When applying PET–PEESE, the pattern for nonbehavioral priming was similar to the overall pattern, such that both PET and PEESE were significant and the adjustments of PEESE still revealed a significant priming effect d = 0.334 (p < .001, 95% CI [0.229, 0.439], k = 361, I² = 66.7%). For behavioral priming, however, PET was significant and after PET’s adjustment, the priming effect was no longer significant d = 0.100 (p < .001, 95% CI [−0.072, 0.273], k = 495, I² = 65.3%), suggesting a possibility that there is a not a nonzero true effect for priming behavioral concepts, but the evidence is inconclusive given the many criticisms of PET–PEESE (Stanley, 2017). These results were then paired with results from other publication analyses to further understand the potential biases and their influence on the priming effect. The slope in Egger’s regression (with the modified measure of precision) and the adjustment of Vevea and Wood’s methods were similar across behavioral and nonbehavioral primes, suggesting potential biases. However, Mathur and VanderWeele’s methods suggested robust effect for priming both behavioral and nonbehavioral concepts, even assuming the most severe publication bias. Altogether, these analyses confirmed some level of inclusion bias in both behavioral and nonbehavioral priming experiments but suggested a robust and significant priming effect across the board.⁷

Figure 4. Funnel Plots of Subsets of Different Prime Type (d < Absolute 2.5, Variance < 0.5).

Note. (A) Funnel plot of effects of priming behavioral concepts and (B) funnel plot of effects of priming nonbehavioral concepts.

Moderator Analyses

Exploratory Moderator Analyses

We examined whether characteristics of the primes and the tasks moderated the effect of priming to better understand the boundary conditions. Exploratory moderators were tested through simple multilevel metaregressions. The results appear in Table 6 and show that prime modality, prime content, social desirability, and type of control group were all significant moderators.⁸ As shown, the priming effect was stronger when the prime was delivered visually, when money or stereotypes were primed, when the behavior was socially desirable, or when the prime condition was compared to a control group other than presentation of neutral words (e.g., nonsense control, no-task control, neutral reading or neutral imagination task control, or priming unrelated goals). When broken down into specific types of control groups, the priming effect was strongest when the priming condition was compared with nonsense words control. Similar to the results from Weingarten et al. (2016a), study location, dosage of primes, liminality, dependent measure type, task prior to priming, and presence of funneled debriefing did not emerge as significant moderators of the priming effect.

Table 6.

Exploratory Moderator Analyses

Moderators	d [95% CI]	Qb	k	I 2	τ2
Descriptive statistics
Country		1.65	856	65.9%	<0.0001
United States	0.34 [0.30, 0.39]		487
Non-United States	0.39 [0.32, 0.46]		369
Prime characteristics
Proportion of prime		2.02	856	66.5%	0.0004
Priming modality		5.05*	856	64.9%	<0.0001
Verbal	0.34 [0.31, 0.38]		656
Visual	0.43 [0.36, 0.51]		200
Content of prime		12.71*	856	62.1%	0.0007
Achievement, intelligence, or efficacy	0.30a [0.24, 0.36]		260
Common behavior	0.35 [0.26, 0.44]		206
Money, marketing, or finance	0.47b [0.34, 0.60]		66
Morality, God, or prosociality	0.38 [0.26, 0.49]		110
Motivation	0.43 [0.29, 0.56]		66
Sex, gender, or romantic behavior	0.46 [0.34, 0.57]		40
Stereotype	0.44b [0.33, 0.56]		108
Liminality		1.04	856	66.0%	0.0004
Subliminal	0.41 [0.32, 0.50]		136
Supraliminal	0.36 [0.26, 0.45]		720
Task characteristics
Social desirability of outcome		12.41**	856	63.7%	0.0017
Neutral	0.33a [0.28, 0.37]		582
Socially undesirable	0.48 [0.32, 0.65]		37
Socially desirable	0.45b [0.37, 0.53]		237
Dependent measure category		0.01	856	65.5%	0.0011
Task performance	0.37 [0.31, 0.43]		298
Others	0.36 [0.29, 0.44]		558
Other design features
Neutral word control		7.50**	856	65.8%	<0.0001
Yes	0.32 [0.28, 0.37]		487
No	0.42 [0.35, 0.49]		369
Control type (specific)		11.20*	856	62.7%	<0.0001
Neutral words	0.32a [0.28, 0.37]		487
Nonsense words	0.56b [0.38, 0.73]		31
No task	0.41 [0.31, 0.52]		99
Neutral reading or imagination	0.45 [0.30, 0.60]		48
Priming an unrelated goal	0.36 [0.23, 0.49]		67
Others	0.40 [0.30, 0.50]		124
Funneled debriefing		0.01	856	65.6%	0.0012
Yes	0.37 [0.30, 0.44]		322
No	0.36 [0.32, 0.41]		534
Task prior to the priming		3.15^	856	66.3%	0.0013
Yes	0.32 [0.25, 0.39]		252
No	0.39 [0.35, 0.43]		604

Note. d = Cohen’s d; CI = 95% confidence interval; Q_b = heterogeneity between groups; k = number of effect sizes included; I² = overall amount of heterogeneity/overall amount of variance; τ² = estimation of true amount of between-study heterogeneity. Within each moderator, ds that have different subscripts are statistically significantly (p < .05) different from each other (e.g., d with subscript a is statistically significantly different from d with subscript b).

^*p < .05.

^**p < .01.

^{^}p < .10.

To examine the influences of potential p-hacking procedures on the effect size, we also tested whether the effect size was moderated by preregistration status, inclusion of covariates, and exclusion of participants during data analyses. These results appear in Table 7. As shown, relative to nonpreregistered studies, preregistered research reported significantly smaller effects, which were null though variable, but inclusion of covariates and exclusion of participants did not inflate the effect size. The results concerning covariates and participant exclusion suggest that the observed priming effect could not be explained solely by these certain aspects of researchers’ degrees of freedom. The null effect found among preregistered studies was consistent with the findings from Lodder et al. (2019) but, given its small k, was not sufficient evidence to invalidate the priming effects. However, we later included these covariates in supplementary models of our theoretical moderators.

Table 7.

Additional Moderator Analyses for Assessing Study Qualities

Moderators	d [95% CI]	Qb	k	I 2	τ2
Preregistration		10.14**	856	66.4%	<.0001
Yes	0.02 [−0.20, 0.24]		13
No	0.37 [0.34, 0.41]		843
Including covariates		0.16	856	64.7%	0.0008
Yes	0.38 [0.30, 0.45]		275
No	0.36 [0.32, 0.40]		581
Excluding participants		0.98	856	65.9%	0.0004
Yes	0.41 [0.31, 0.50]		132
No	0.36 [0.32, 0.40]		724

Note. d = Cohen’s d; CI = 95% confidence interval; Q_b = heterogeneity between groups; k = number of effect sizes included; I² = overall amount of heterogeneity/overall amount of variance; τ² = estimation of true amount of between-study heterogeneity.

^**p < .001.

Theoretical Moderator Analyses

One major objective of the current meta-analysis was to test the mechanisms of priming behavioral and nonbehavioral concepts. Although our initial analysis found that the two were similar in strength, we wanted to examine the possibility of a behavior–perception link and goal mediation for both behavioral and nonbehavioral concepts. Therefore, using model comparison techniques, we then examined whether goal value, goal expectancy, and delay/opportunity for satisfaction affected each type of priming differently (i.e., an interaction effect) by testing whether inclusion of each of these interaction term significantly improved model fit. For example, when testing the significance of the interaction term between goal value and prime type, we compared (a) the model that only includes goal value, prime type, along with all the exploratory moderators as control (see the notes under Table 8) with (b) the above model plus the interaction term between goal value and prime type. If the full model had significantly better model fit, then we concluded on a significant interaction effect between goal value and prime type, which would lead us to run a no-intercept model with mean-centered control variables to obtain estimates of average effect size for each group, to better understand the interaction effect.

Table 8.

Theoretical Moderator Analyses Controlling for Control Factors

Predictor	B (SE)	k	K	I 2	τ2
Goal value and prime type interactions		856	351	58.6%	<.0001
Priming behavioral concepts when goal value is lower	−0.45 (0.19)*
Priming behavioral concepts when goal value is unchanged	0.38 (0.02)***
Priming behavioral concepts when goal value is higher	0.40 (0.06)***
Priming nonbehavioral concepts when goal value is lower	0.43 (0.29)
Priming nonbehavioral concepts when goal value is unchanged	0.37 (0.03)***
Priming nonbehavioral concepts when goal value is higher	0.18 (0.10)^
Control factors
Priming modality (verbal or visual)	0.09 (0.05)^
Source (journal article or others)	−0.16 (0.05)***
Year of report (1983–2020)	−0.01 (<0.01)**
Country (United States or others)	0.03 (0.03)
Proportion of prime (0–1)	−0.03 (0.08)
Priming concept (achievement or others)	0.08 (0.04)^
Liminality (subliminal or supraliminal)	−0.03 (0.05)
Social desirability of outcomes (desirable or undesirable)	0.07 (0.02)***
Dependent variable category (performance or others)	−0.07 (0.04)^
Control type (neutral control or others)	0.07 (0.04)^
Presence of funneled debriefing (present or not)	0.07 (0.04)^
Task prior to priming (yes or no)	−0.07 (0.04)^

Note. B = unstandardized regression coefficient; SE = standard error; k = number of effect sizes included; K = number of studies included; I² = overall amount of heterogeneity/overall amount of variance; τ² = estimation of true amount of between-study heterogeneity. This no-intercept model simultaneously entered the interaction term between goal value and prime type, as well as mean-centered priming modality, publication status, year of report, country, proportion of prime, content of prime, liminality, social desirability of outcome, type of dependent variable, type of control, presence of funneled debriefing, and presence of task prior to priming as covariates. The regression coefficient for each combination of level of goal value and prime type should be interpreted as an estimated mean effect size for that combination level when all other covariates were at their mean level. Mean estimates were obtained from no-intercept models.

^*p < .05.

^**p < .01.

^***p < .001.

^{^}p < .10.

Goal Value

If behavioral primes stimulated greater goal activation involvement, whereas nonbehavioral primes operated through the behavior–perception link, we may see an interaction between priming type and goal value, such that goal value may moderate the effect of priming of behavioral concepts more than that of priming of nonbehavioral concepts. Accordingly, our analyses found a significant interaction between goal value and prime type, ${\chi }_{\text{difference}}^2\left(2\right)=9.99,p=0.007$. As shown by the average effect sizes in Table 8, the priming effect was weaker when the goal value was lower than when it was not manipulated or when it was manipulated to be higher. However, goal value had no effect whatsoever when nonbehavioral concepts were primed. We also ran the above analyses additionally controlling for preregistration status, inclusion of covariates, and exclusion of participants during data analyses. After introducing those controls, the interaction term between goal value and prime type was still significant, ${\chi }_{\text{difference}}^2\left(2\right)=10.62,p=0.005$. As shown in Table 9, the pattern of the estimated mean effect sizes was also similar.

Table 9.

Theoretical Moderator Analyses Controlling for Additional Control Factors

Predictor	B (SE)	k	K	I 2	τ2
Goal value and prime type interactions		856	351	58.1%	<.0001
Priming behavioral concepts when goal value is lower	−0.47 (0.119)*
Priming behavioral concepts when goal value is unchanged	0.38 (0.02)***
Priming behavioral concepts when goal value is higher	0.39 (0.06)***
Priming nonbehavioral concepts when goal value is lower	0.47 (0.29)
Priming nonbehavioral concepts when goal value is unchanged	0.38 (0.03)***
Priming nonbehavioral concepts when goal value is higher	0.18 (0.10)^
Control factors
Priming modality (verbal or visual)	0.08 (0.05)^
Source (journal article or others)	−0.17 (0.05)***
Year of report (1983–2020)	−0.01 (<0.01)*
Country (United States or others)	0.01 (0.04)
Proportion of prime (0–1)	−0.04 (0.08)
Priming concept (achievement or others)	0.09 (0.04)*
Liminality (subliminal or supraliminal)	−0.03 (0.05)
Social desirability of outcomes (desirable or undesirable)	0.06 (0.02)**
Dependent variable category (performance or others)	−0.06 (0.04)
Control type (neutral control or others)	0.07 (0.04)
Presence of funneled debriefing (present or not)	0.07 (0.04)^
Task prior to priming (yes or no)	−0.08 (0.04)*
Preregistration status (yes or no)	−0.27 (0.12)*
Inclusion of covariates (yes or no)	0.02 (0.04)
Exclusion of participants (yes or no)	0.04 (0.05)

Note. B = unstandardized regression coefficient; SE = standard error; k = number of effect sizes included; K = number of studies included; I² = overall amount of heterogeneity/overall amount of variance; τ² = estimation of true amount of between-study heterogeneity. This no-intercept model simultaneously entered the interaction term between goal value and prime type, as well as mean-centered priming modality, publication status, year of report, country, proportion of prime, content of prime, liminality, social desirability of outcome, type of dependent variable, type of control, presence of funneled debriefing, presence of task prior to priming, preregistration status, inclusion of covariates, and exclusion of participants as covariates. The regression coefficient for each combination of level of goal value and prime type should be interpreted as an estimated mean effect size for that combination level when all other covariates were at their mean level. Mean estimates were obtained from no-intercept models.

^*p < .05.

^**p < .01.

^***p < .001.

^{^}p < .10.

Goal Expectancy

Similar to goal value, if the effect of priming behavioral concepts operated through goal activation and the effect of priming nonbehavioral concepts operated through the behavior–perception link, priming type and goal expectancy could interact. However, we did not find a significant interaction between goal expectancy and prime type, ${\chi }_{\text{difference}}^2\left(2\right)=1.79,p=0.41$.

Satisfaction Opportunity

If the effect of priming behavioral concepts operated through goal activation and the effect of priming nonbehavioral concepts operated through the behavior–perception link, priming type and satisfaction opportunity could interact, such that an opportunity for satisfaction may decrease the effect of priming behavioral concepts but not nonbehavioral ones. However, we did not find a significant interaction between delay/opportunity for satisfaction and prime type, ${\chi }_{\text{difference}}^2\left(2\right)=5.30,p=0.07$.

Discussion

Overview of Findings

Despite a controversy that has now spanned several decades, the existence and authenticity of the priming effect has been supported by several recent meta-analyses of the behavioral effects of priming (e.g., Chen et al., 2021; Weingarten et al., 2016a). Although each of these meta-analyses made significant contributions to the field, each of them has had limitations based on their inclusion criteria, methodological features, and objectives. Our meta-analysis stands as the most comprehensive review of priming of behavioral outcomes, including 862 effect sizes stemming from 224 reports. Our study is also the first to distinguish and systematically examine the mechanisms of priming behavioral or nonbehavioral concepts, in a meta-analysis or in the literature more generally. Moreover, our study made extensive methodological innovations that neither Chen et al. (2021) or Weingarten et al. (2016a) implemented. Specifically, we used Stanley and Doucouliagos’ (2014) PET–PEESE and the new sensitivity analyses from Mathur and VanderWeele (2020) to gauge inclusion bias, as well as the novel CHE model under RVE (Pustejovsky & Tipton, 2021) to deal with effect size dependence.

Our analyses revealed a moderate priming effect on overall behavior: $d=0.368,I^2=77.8\mathrm{\%}$ (see Table 3). These results showed that the priming effect was overall robust and that most of the variance in the observed effects is explained by heterogeneity between studies, which may be random or due to unaccounted factors such as study context. Our publication analyses showed some degree of bias, but Mathur and VanderWeele’s (2020) sensitivity analyses, Vevea and Woods’s (2005) sensitivity analyses (assuming two-tailed selection biases), and Stanley and Doucouliagos’ (2014) PEESE analyses revealed significant priming effects even after small-study bias was considered (see Table 4). Moreover, the significant priming effect was present for both behavioral and nonbehavioral concepts and controlling for bias did not remove the effect in either case. Our analyses also showed that priming effects were present even in unpublished reports. All these findings thus add to the evidence base of the effects of priming on overt behaviors.

Controversies and skepticism about whether the priming effect on behavior is real have now existed for several decades. Our meta-analysis, which compiles the largest number of priming studies to date, provides solid evidence that the priming effect is a real psychological phenomenon that remains robust when using the most advanced bias-detection methods. Our inclusion/publication bias analyses, which use cutting-edge techniques (e.g., Mathur & VanderWeele, 2020; Pustejovsky & Rodgers, 2019; Stanley & Doucouliagos, 2014), showed that the effect remained significant after considering different types of biases or even assuming the most severe type of publication bias, suggesting that priming effect is not a bubble created solely by publication bias. We also looked for potentially questionable research practices (e.g., excluding data and adding covariates) but found little evidence that these practices impact the reported effect sizes, further adding to the robustness of the priming effect. It is noteworthy, nonetheless, that researchers may not always transparently report their questionable research practices, and more p-hacking strategies, such as selective reporting of dependent variables or having multiple treatment conditions, may come into play. Thus, more work is needed to examine the role of a wider range of questionable research practices on the priming effect and all other phenomena in psychology and beyond, taking into consideration the reporting transparency.

One finding that is worth attention, however, is that the priming effect was no longer significant among preregistered reports ($d=0.02$), which is consistent with Lodder et al.’s (2019) findings ($d=0.01$). However, these results are from a very limited number of effect sizes ($k=13$) and are therefore not decisive, if informative at all. Preregistration is a relatively novel procedure in psychology, and most of the included priming studies were conducted before this practice was popularized. Also, most of the preregistered studies in our meta-analysis are replication studies. As noted by Bryan et al. (2019), replication researchers also have degrees of freedom in deciding how they want to replicate the original studies. Such degrees of freedom can make it easy to come up with false-negative replication results, and such failures to replicate have been argued to be more easily published than successful replications, creating a reverse publication bias that favors null effects among replication studies (Kirkegaard, 2020; Neuroskeptic, 2012). The bottom line is that several replication failures by themselves are not necessarily indications that the effect is not “real.” The replication failures may be due to differences in the methods, the spatial and historic context, the sample, and replication degrees of freedom, the impact of which needs to be addressed through future meta-analysis and well-powered registered experiments.

Besides synthesizing the research on behavioral priming, another important goal of the current meta-analysis was to compare the effectiveness of behavioral and nonbehavioral primes. Although many believe that priming effects could be stronger when the priming task provides clear and specific behavioral guidance, our meta-analysis did not show a significant difference between behavioral and nonbehavioral primes. One possibility is that, as suggested by Devine’s (1989) logic, behaviors can be automatically primed in response to both behavioral and nonbehavioral concepts. The perception–behavior effect thus appears to dominate, except for evidence of greater behavioral control when the value of a behavior is lowered.

Another important contribution of our research was to test goal activation versus the behavior–perception link as the mechanisms involved in priming behavioral and nonbehavioral primes. Habit researchers (e.g., Wendy Wood; Wood & Rünger, 2016) and goal researchers (e.g., Henk Aarts; Aarts et al., 2007) have debated whether behavioral priming needs goal activation to occur, and our analyses speak directly to that debate. Even though behavioral and nonbehavioral concepts were equally effective, the effect of priming behavioral concepts decreased drastically when goal value was manipulated downward. Thus, even though heightening value and other markers of goal mediation had no effect, it appears that removing value acts like a disincentive that motivates individuals to control their behavior. These findings were consistent with Macrae and Johnston’s (1998) work, which showed that behavioral priming effects can be eliminated when inhibitory cues are present in the environment, and Bargh and Hassin’s (2022) reports that conflict with current goals can act as a disincentive and limit behavioral priming effects.

Last, our exploratory moderator analyses indicated that priming effects were stronger when the behavior was socially desirable than when it was neutral. The behavioral tasks included in lab studies are eminently social and are thus influenced by social norms. People often follow social norms (Cialdini et al., 1990), as a result of which the normative component in socially desirable outcomes may elicit stronger behavioral responses. Our analyses also found stronger priming effects when the priming condition was compared to a control condition that used nonsense words (e.g., words like “gub” as a control for the religious prime “God,” Lin et al., 2016). Nonsense words may be an ideal, content-free control that enables researchers to best visualize the effect of primes. Last, consistent with the findings from Chen et al. (2021), our analyses also showed larger effects for visual priming. Visual stimuli often contain more information and are thus able to activate more associations and promote larger behavioral changes.

Furthermore, our analyses of the content of primes showed that the priming effect was robust across a variety of contents, which dovetails well with findings from prior meta-analyses. For example, we found a $d$ of 0.47 for money, marketing, or finance primes, which was slightly larger than the $d$ of 0.31 in Lodder et al. (2019). We found a $d$ of 0.38 for morality, God, or prosociality priming, which was comparable to the $d$ of 0.40 in Shariff et al. (2016). We found a $d$ of 0.34 for behavioral priming, which was comparable to the $d$ of 0.40 in Weingarten et al. (2016a). Finally, we found a $d$ of 0.30 for achievement priming, which was not too far off from the $d$ of 0.44 in Chen et al. (2021).

Altogether, our analyses provided strong support for the existence of a priming effect by showing its robustness across different contents and contexts and when adjusted by different methods, which assumed different types and severity of biases. Moreover, we partially replicated Weingarten et al.’s (2016a) findings in the case of behavioral priming, finding effects that could be suppressed when value was lowered. Finally, we identified social desirability as a possible moderator of the priming effect. These findings increase our understanding of the boundary conditions of the behavioral priming effect and provide important contexts for future studies and replications.

Limitations and Future Directions

Despite the comprehensiveness and novelty of the current meta-analysis, there are limitations to consider. First, because our meta-analysis included a comprehensive list of priming types and behavioral outcomes, our data set had high heterogeneity. According to Stanley (2017), high heterogeneity within the data set might lower the accuracy of some statistical methods, such as the PET–PEESE adjustments, by inflating its false alarm ratio. Therefore, although the adjusted effect by PET was no longer significant for priming behavioral concepts, this finding should be interpreted with caution because PET might have overly adjusted the effect due to high heterogeneity (Stanley, 2017). It was reassuring though, to see that the adjusted effect given by PEESE, remained significant even though it might have been overadjusted. Future studies and meta-analyses should continue to understand the heterogeneity of the priming effect and implement new inclusion bias methods appropriate for the distribution of our effects.

Related to the previous limitation, due to the high heterogeneity of effects in the current data set, we were unable to run network meta-analysis. Therefore, we could not directly compare the effectiveness of behavioral and nonbehavioral primes with each other. Instead, we could only compare the effectiveness of the two types of priming with respect to a control condition. Thus, the findings regard to the relative effectiveness of behavioral and nonbehavioral primes should be interpreted in relation to a control comparison. Future experiments, however, may directly compare the effectiveness of behavioral and nonbehavioral primes and thus reexamine our conclusions.

It is also noteworthy that meta-analysis is a correlational method, particularly when it comes to moderator analyses. Therefore, all the findings and conclusions made in this meta-analysis are correlational and do not demonstrate cause–effect relations. To better test the effects of priming behavioral and nonbehavioral concepts, or the effects of goal value, goal expectancy, and satisfaction opportunity, large-scale, preregistered experimental studies need to be conducted. Ideally, one could experimentally manipulate the prime to be a nonbehavioral or behavioral concept and test the effect of each of the moderators discussed in our article.

Closing Remarks

The field of priming research has gradually shifted its attention from the debate over the existence of priming effect to more refined questions, such as the underlying mechanism behind each type of priming. In this second era of the field, our article may serve a transitional function, to take stock and raise novel questions for future research. Our synthesis revealed a moderate effect of the incidental priming of behavioral and nonbehavioral concepts on behavioral outcomes, which remained significant after controlling for selection bias and methodological factors. Moreover, our synthesis showed a small difference between using behavioral and nonbehavioral primes, such that the effect of behavioral primes can be suppressed more than the effect of nonbehavioral ones. This goal mediation, however, appears against that suggests mere activation of associations as responsible for the effects of all concepts on behavior. We hope that our findings will reassure the field of the robustness of this long-debated psychological phenomenon, provide guidance in the selection of priming methods for future studies, and inspire new research on the important impact of concept accessibility on human behavior.

Supplementary Material

Supplemental materials: https://doi.org/10.1037/bul0000374.supp

Public Significance Statement.

This meta-analysis revealed a moderate effect of priming of behavioral and nonbehavioral concepts on behavioral outcomes. Furthermore, according to these findings, even though behavioral and nonbehavioral primes share much in common, behavioral primes allow for goal-mediated control of the priming effect to a greater extent than nonbehavioral ones.