Efficiently exploring the causal role of contextual moderators in behavioral science

Significance

Behavioral science interventions can address societal problems. Often, however, little is known about the contextual factors that must be in place for them to work reliably. We developed a method to investigate this efficiently. Applying this method, we found that the benefits of a student’s growth mindset depended on whether the teacher supported the growth mindset both through their messages (i.e., what they say) and through their structural opportunities (i.e., their policies). These findings provide a roadmap for understanding behavioral science effects more broadly and can thereby lead to more reliable guidance for policy and practice.

Abstract

Behavioral science interventions have the potential to address longstanding policy problems, but their effects are typically heterogeneous across contexts (e.g., teachers, schools, and geographic regions). This contextual heterogeneity is poorly understood, however, which reduces the field’s impact and its understanding of mechanisms. Here, we present an efficient way to interrogate heterogeneity and address these gaps in knowledge. This method a) presents scenarios that vividly represent different moderating contexts, b) measures a short-term behavioral outcome (e.g., an academic choice) that is known to relate to typical intervention outcomes (e.g., academic achievement), and c) assesses the causal effect of the moderating context on the link between the psychological variable typically targeted by interventions and this short-term outcome. We illustrated the utility of this approach across four experiments (total n = 3,235) that directly tested contextual moderators of the links between growth mindset, which is the belief that ability can be developed, and students’ academic choices. The present results showed that teachers’ growth mindset-supportive messages and the structural opportunities they provide moderated the link between students’ mindsets and their choices (studies 1 to 3). This pattern was replicated in a nationally representative sample of adolescents and did not vary across demographic subgroups (study 2), nor was this pattern the result of several possible confounds (studies 3 to 4). Discussion centers on how this method of interrogating contextual heterogeneity can be applied to other behavioral science interventions and broaden their impact in other policy domains.

Behavioral science interventions (e.g., “nudges,” “wise” interventions) shift behavior by changing people’s interpretations of information or situations, without large-scale, costly restructuring of the environment. These interventions have shown promise for addressing longstanding policy problems, such as inequalities in education, wealth, and health, at low cost (see refs. 1–3). However, meta-analyses indicate that their effects are also highly heterogeneous (see refs. 4–6), which means that they work in some contexts but not others (see ref. 7). For example, the popular Opower descriptive norms intervention, which informed people when they used more energy than similar neighbors, showed the strongest effects in communities that favored proenvironmental issues (8, 9). Similarly, an intervention to subtly frame voting as a reflection of the self (“being a voter” vs. “voting”) was ineffective in very low-turnout elections, where people felt only weakly obligated to vote (10–13).

Variation in behavioral science intervention effects is not inherently problematic or surprising. The effects of an intervention that only shifts people’s interpretations will necessarily depend on whether the context allows or invites them to act on those shifted interpretations (14). Unexplained heterogeneity, however, indicates significant limitations of theory and reduces policy implications. It suggests that we do not understand the mechanisms through which an intervention works and that we are not equipped to use the behavioral insight to know when and under what circumstances interventions will reliably solve problems (see ref. 7). Conversely, when equipped with precise knowledge of contextual moderators, investigators can more precisely scale the intervention where it is most likely to work or shape environments to make them more conducive to the intervention.

For example, if we understood the contexts that made people most likely to act on the Opower norms intervention, then researchers would have several options. They might decide to intervene only in contexts where the intervention is likely to be effective (e.g., proenvironmental districts) or they might refine the intervention so that it works more broadly (e.g., tailor the framing of the intervention to different populations). They might even try to alter the null-results settings themselves to make them more hospitable for the intervention (e.g., messages to increase proenvironmental attitudes). That is, one might enrich the context (the “soil”) in a way that would allow the intervention (the “seed”) to take root and grow to fruition (see ref. 15).

Any of these responses to heterogeneous effects could involve a significant investment of scientific resources, however. Such investments could be ill-advised if they were not rooted in strong causal evidence. In the present research, we propose and illustrate an efficient method for simulating and assessing the causal moderating effect of contextual variables in behavioral intervention studies so that heterogeneity-informed behavioral science can proceed more effectively. This method involves crafting scenarios that build on available evidence (e.g., rich observational, survey, and/or experimental research) to represent different moderating contexts. Next, researchers can randomly present the scenarios to participants and assess whether the link between the variables directly targeted by an intervention (e.g., perceived energy-usage norms, identification as a “voter”) and short-term correlates of typical intervention outcomes (e.g., intentions to conserve energy, plans to vote) are enhanced or weakened. This method is therefore a valuable bridge between high-quality heterogeneity evidence in initial studies and new, large, field experiments that directly target contexts.

The Case of Growth Mindset

Here, we use our proposed method to better understand contextual moderators of the links between students’ growth mindsets and their academic choices. A growth mindset is the belief that academic ability is not fixed but can be improved with effort, good strategies, and mentoring or support from others (see refs. 16 and 17). A fixed mindset—the opposite of a growth mindset—is the belief that academic ability is fixed and cannot be changed. A recent randomized controlled trial (RCT) revealed noteworthy, but heterogeneous, effects of an intervention that taught students the growth mindset in a large, multicontext, nationally representative sample of ninth-grade students (18, 19). The intervention improved students’ math grades chiefly when their math teachers also reported more of a growth mindset (19), perhaps because these teachers were more likely to provide support for students’ intervention-induced growth mindsets.

Growth mindset is an informative case study because its links to positive outcomes have been established in a broad body of high-quality correlational and experimental research, which has supported the same or similar conclusions (see refs. 16–18, 20, and 21). Growth mindset beliefs have been associated with academic outcomes such as performance and advanced course-taking in the long term (16, 18, 20), as well as learning-oriented choices in the short term—the focal outcome of the present research—such as the selection of more challenging academic tasks (18, 22). Further, like other behavioral science interventions (e.g., descriptive norms interventions to conserve electricity, invoking the self to promote voter turnout), its effects were stronger in more favorable contexts (18, 19).

Previous research on growth mindset moderators has been limited, however, because precise measures of the context were lacking (see ref. 23), and measured variables, in any case, cannot support causal inferences. That is, identifying teachers’ mindset beliefs as a moderator does not reveal what it is about teaching practices that encourage students to act on their growth mindset beliefs. This means that researchers do not currently know which practices to try targeting in future interventions aimed at teachers.

Here, we used the scenario study approach to help build a bridge from initial, but reliable, identification of a measured moderator (teachers’ mindset beliefs) to later, policy-relevant evidence that could be obtained from field studies that manipulate relevant contextual factors (e.g., teacher practices) (Fig. 1). Note that without this approach, researchers might presume a causal moderating role of teachers’ growth mindset beliefs alone and perhaps attempt to intervene with teachers accordingly (Fig. 1A). For example, such an intervention might focus primarily on changing teachers’ beliefs through persuasion but miss out on changing high-leverage practices. In fact, past teacher-focused interventions that took a beliefs-heavy and practices-light approach were found to be ineffective (e.g., ref. 24). In the proposed method, by contrast, (Fig. 1B), researchers would acknowledge that we do not yet know what it was about teachers that fostered growth mindset effects. We would then posit candidate teacher practices, manipulate them, and test their moderating effect on the link between growth mindset beliefs and short-term outcomes such as learning-oriented choices. This could inform a subsequent teacher-level RCT that might be administered alongside the student growth mindset intervention.

Fig. 1.

Two models for interrogating measured contextual moderators. In the naive model (A) researchers would conduct individual-level RCTs and find measured context-level moderators that were significant. Then, they would presume a causal interpretation of these measured moderators (without actually testing this interpretation) and conduct a subsequent field experiment accordingly. For example, having identified teacher mindset beliefs as a moderator of growth mindset intervention effects, researchers might conduct a subsequent RCT manipulating teachers’ growth mindset beliefs. In the proposed model (B) researchers would test the causal interpretation of measured moderators prior to attempting to conduct a subsequent RCT. They would do this by crafting well-informed scenarios that manipulate aspects of a hypothetical context (based on available, and ideally ecologically valid, evidence) and test whether these scenarios moderate the individual-level effects in large experiments. Only after establishing a probable causal role of particular moderators would the researchers attempt to conduct a subsequent field experiment. For example, having identified particular teaching practices as a moderator of student growth mindset effects, they might encourage teachers to use these practices, or they might modify the student growth mindset intervention to encourage students to use their growth mindset beliefs even when these beliefs are not clearly supported by the teacher’s practices.

The Present Research.

Here, we used the scenario study approach (Fig. 1B) to experimentally evaluate whether growth versus fixed mindset teaching practices moderate student growth mindset effects. Drawing on past observational (e.g., ref. 25), survey (e.g., ref. 26), and experimental research (e.g., refs. 27 and 28), we identified two relevant categories of teacher practices: verbal messages in support of a growth mindset and structural opportunities for students to act on their growth mindsets, such as formal grading policies (see ref. 29). We then crafted depictions of the teacher practices, faithful to the observational findings, which varied in fixed versus growth mindset messages and opportunities. In four experiments, we tested whether randomly assigned descriptions of teachers moderated the association between students’ growth mindset beliefs and the choices that are typically linked with a growth mindset, such as selecting more challenging coursework (18, 22). Two final studies ruled out possible confounds, showing that both messages and opportunities needed to be aligned (study 3) and that a teacher’s warm demeanor was not a replacement for authentic growth mindset practices (study 4).

This research makes several contributions. First, it provides the first known causal test of teacher-level contextual factors that may moderate growth mindset effects. Second, it generates specific, concrete, and well-founded recommendations for future teacher-level growth mindset RCTs. Finally, it shows how behavioral intervention research, in general, might acquire causal knowledge of important contextual sources of heterogeneity to inform future research.

Analytic approach.

Throughout the paper, we report estimates from Bayesian Causal Forest (BCF) models. BCF is a machine-learning algorithm that has emerged as a leading method to identify moderation without drawing spurious conclusions (30, 31). BCF is conservative and assumes that the effect of any variable is modest and not moderated. This prior belief shrinks the variability of the effect size across groups, such that only strong evidence will lead the model to identify moderation. In addition, BCF flexibly incorporates covariates and freely allows them to interact with one another to the extent that they can reliably predict variance in the outcome. Therefore, BCF removes the researcher degrees of freedom associated with covariate selection by allowing the model to learn how to incorporate the covariates without human intervention.

We present standardized coefficients (βs) from BCF, the interval of the posterior distribution from the 10th to 90th percentile, and the estimated probability that a given effect is different from zero (i.e., the proportion of draws from the posterior distribution in which the coefficient was different from zero in the expected direction). Note that our reporting of posterior probabilities, rather than P values, is consistent with calls to abandon “all-or-nothing” significance thresholds inherent in frequentist analyses and to instead report probabilities that a hypothesis is true as a continuous measure (refs. 32 and 33, see refs. 12, 18, 19, and 34 for other examples of published research that employs this approach to hypothesis testing) and also avoids common misinterpretations of P values (35). Following our preregistered standards (see https://osf.io/ncxtm), any effect below a 75% posterior probability (i.e., interquartile range contains 0) is not interpreted as meaningful, and any posterior probability above that is reported continuously (32, 33), with higher posterior probabilities indicating greater confidence.

Results

Study 1: Growth Mindset Moderation by Teachers’ Messages and Opportunities.

Participants were adolescents (n = 1,094) in grades 7 through 12 in the United States (preregistration: https://osf.io/us57r) who completed an online survey experiment. In all studies, we focused on measures of participants’ growth mindset beliefs, rather than manipulations of them. An advantage of measuring pre-established growth mindset beliefs is that this avoids potential demand effects that could come from being asked to apply newly acquired beliefs immediately after a persuasive intervention. Participants first reported their growth mindset beliefs in terms of the established construct of a growth mindset motivational framework (see ref. 36): their belief that intelligence is malleable and tightly associated “meanings” and action tendencies that grow out of the underlying malleability belief and that can either help or hinder performance in school (e.g., the belief that effort is a positive thing that helps to grow intelligence, rather than a negative thing that indicates a lack of ability) (see refs. 16 and 37).

Next, participants were randomly assigned to one of two conditions: a growth mindset teacher condition (n_Growth = 570) or a fixed mindset teacher condition (n_Fixed = 524). They read an interview with a math teacher about what the teacher would say to their students on the first day of class. In the growth mindset teacher condition, the teacher provided growth mindset-supportive messages (e.g., “I believe that all students can learn and do well in the class, no matter where they started out”) and structural opportunities (e.g., “If you show improvement in your exam grades over the course of the term, I’ll raise your final grade”). In the fixed mindset teacher condition, the teacher provided fixed mindset messages (e.g., “If you struggle in this class, remember that not everybody is a ‘math person’ who can easily learn and apply the right facts and techniques for problems”) and structural opportunities (e.g., “When you turn in assignments, whatever grade you get will be final”). (Note that in study 3, we vary messages and opportunities separately in a crossed experimental design.) An analysis of a manipulation check (SI Appendix) showed that the manipulation altered students’ perception of the teacher’s support for the growth mindset as expected.

After reading the teacher’s speech, participants reported whether they would make learning-oriented choices in the teacher’s class. Our focal outcome was their choice of a challenging (rather than easy) assignment in that teacher’s class. This measure of learning-oriented challenge-seeking behavior has been used in previous growth mindset research and has been validated as a predictor of consequential choices, including a selection of challenging (rather than easy) math courses a year later (18, 22). We hypothesized that participants’ growth mindset motivational frameworks would predict these learning-oriented choices only when the teacher’s messages and opportunities supported a growth mindset and not when they supported a fixed mindset.

As predicted, participants’ growth mindset motivational frameworks translated into learning-oriented choices only when the hypothetical teacher provided messages and opportunities that supported the growth mindset. In the growth mindset teacher condition, participants who reported more of a growth mindset motivational framework were more likely to choose a challenging hypothetical assignment that could facilitate learning rather than an easy review assignment, β = 0.11 [0.06, 0.17], pr(β > 0) = 0.99. However, this association was eliminated in the fixed mindset teacher condition, β = −0.02 [−0.07, 0.03], pr(β > 0) = 0.71 (probability of difference in βs between conditions = 0.99) (Fig. 2). Participants also reported on a number of secondary outcomes, such as their achievement goals (agreement with items like “If I were in this teacher’s class, it would be more important for me to learn than to get the best grades”), feelings of being respected by the teacher (agreement with “In this class, this teacher would treat me with respect”), and anticipated shame if they were to experience failure in the teacher’s class (agreement with “If I were in this teacher’s class, I would probably feel embarrassed if I got a problem wrong on the board in front of my peers,” reverse scored). These also showed the same interaction effects as this behavioral proxy outcome (SI Appendix).

Fig. 2.

Pattern of results on learning-oriented choices in study 1. Panel A displays the estimated choice of the challenging (vs. easy) assignment as a function of motivational framework and condition, estimated from an unconditional ordinary least-squares regression model. The density plot below shows the distribution of motivational framework. Panel B displays the association between motivational framework and assignment choice (standardized coefficients) as a function of condition, estimated in a BCF model. Boxes display the interquartile range, whiskers display the interval from the 10th to 90th percentile of the posterior distribution, and points represent draws from the posterior distribution outside of that range. Panel C displays the posterior distribution of the difference in the association between each condition and the fully fixed teacher condition (i.e., the interactions), with a dashed line at zero.

Study 2: Replication in a Nationally Representative Panel.

Opportunities to learn in US public schools are highly different across race and social class groups, as are stereotypes about students’ intellectual abilities, either of which could make a teacher’s growth mindset practices more or less credible to students in different contexts or from different backgrounds. Indeed, in some cases, students’ motivational frameworks have shown stronger or weaker associations with outcomes across social class and geographic groups (38–40). Although study 1’s sample was large, it was not ideal for an interrogation of possible heterogeneity of effects across subgroups because it did not use a random selection strategy that produced a sample that was representative of different subgroups (see ref. 41 for an explanation). Therefore, in study 2, we replicated the experiment in a nationally representative panel of adolescents (n = 803; growth mindset teacher condition, n = 411, fixed mindset teacher condition, n = 392) (preregistration: https://osf.io/69urj).

This study’s results replicated the findings from study 1. A growth mindset framework predicted choice of the challenging assignment positively and more strongly in the growth mindset teacher condition relative to the fixed mindset teacher condition (probability of difference in βs between conditions = 0.94; Fig. 3 A and B). Surprisingly, we did not find meaningful heterogeneity in these results across participant gender, underrepresented racial/ethnic minority status, income, or parent education (probability of the growth mindset framework × condition × subgroup interaction < 0.70 for each subgroup comparison). Thus, the negating effect of the fixed mindset teacher on the association between participants’ growth mindset beliefs and learning-oriented choices generalized across demographic subgroups in the United States (Fig. 3C). This result demonstrates the generalizability of the moderation effect. It may also suggest the importance of clarity and consistency in teachers’ growth mindset messaging. Note that our manipulations were clear about the classroom’s mindset culture, but if mindset messages and opportunities are more ambiguous in real classroom contexts, then students may derive different meanings from them depending on their identities or backgrounds.

Fig. 3.

Pattern of results on learning-oriented choices in study 2. Panel A displays the association between motivational framework and assignment choice as a function of condition, estimated in a BCF analysis. Boxes display the interquartile range, whiskers display the interval from the 10th to 90th percentile of the posterior distribution, and points represent draws from the posterior distribution outside of that range. Panel B displays the posterior distribution of the difference in the association with teacher type condition (i.e., the interaction), with a dashed line at zero. Panel C shows that this pattern did not vary by gender, underrepresented racial/ethnic minority (URM) status, income, or parent education.

Study 3: Were Both Messages and Structural Opportunities Necessary?

In studies 1 and 2, our manipulation of the teachers’ mindset-related practices included both messages and structural opportunities in the same manipulation. However, this did not allow us to distinguish whether it was teachers’ words or deeds that supported (vs. undermined) students’ growth mindset effects. In study 3, we tested the possibility that there were differential contributions of both messages and opportunities. Indeed, inconsistency between messages and opportunities may lead adolescents to hold ambivalent perceptions of whether their growth mindset beliefs are truly welcome. For example, opportunities to earn points for correcting one’s mistakes may be misinterpreted as a safety net for students who “lack natural ability” if not accompanied by growth mindset-supportive messages. Likewise, a teacher who verbally expresses valuing improvement over initial performance might seem hypocritical if they do not provide opportunities for students to be rewarded for improving.

To assess these possibilities, we conducted a new experiment in a large, national (although not representative) sample of adolescents (n = 1,082) (preregistration: https://osf.io/ykb4p). The procedure was similar to studies 1 and 2, except that participants were randomly assigned to one of four conditions in a 2 (messages: fixed vs. growth) × 2 (opportunities: fixed vs. growth) design. There were 272 participants in the fixed messages/fixed opportunities condition, 271 participants in the growth messages/fixed opportunities condition, 273 participants in the fixed messages/growth opportunities condition, and 266 participants in the growth messages/growth opportunities condition.

Results showed that both messages and opportunities mattered and jointly contributed to the association between participants’ growth mindset motivational frameworks and their choice of the challenging (vs. easy) assignment. Consistent with studies 1 and 2, the association was stronger and more positive in the growth messages/growth opportunities condition than in the fixed messages/fixed opportunities condition (probability of difference in βs between conditions = 0.98). The association in the fixed messages/growth opportunities and growth messages/fixed opportunities conditions was also more positive than in the fixed messages/fixed opportunities condition (probability of difference in βs between conditions > 0.95 for each pairwise comparison) but weaker and less positive than the association in the growth messages/growth opportunities condition (probability of difference in βs between conditions > 0.77 for each pairwise comparison). Thus, a growth mindset framework predicted learning-oriented choices the most when teachers’ messages and opportunities were both aligned with a growth mindset, the least when they were both aligned with a fixed mindset, and moderately when teachers’ messages and opportunities were not aligned (Fig. 4).

Fig. 4.

Pattern of results on learning-oriented choices in study 3. Panel A displays the estimated choice of the challenging (vs. easy) assignment as a function of motivational framework and condition, estimated from an unconditional ordinary least-squares regression model. The density plot below shows the distribution of motivational framework. Panel B displays the association between motivational framework and assignment choice (standardized coefficients) as a function of condition, estimated in a BCF model. Boxes display the interquartile range, whiskers display the interval from the 10th to 90th percentile of the posterior distribution, and points represent draws from the posterior distribution outside of that range. Panel C displays the posterior distribution of the difference in the association between each condition and the fully fixed teacher condition (i.e., the interactions), with a dashed line at zero.

In real educational settings, a teacher’s growth and fixed mindset-supportive practices are likely to be somewhat mixed, much like the two new conditions in this study that mixed fixed and growth mindset-supportive practices together. Here, we found that the mixed conditions could reduce the association between students’ growth mindset frameworks and their learning-oriented choices. However, these same data also have an optimistic implication: growth-mindset messages or opportunities, even in the presence of fixed-mindset practices, can still strengthen this association relative to uniformly fixed mindset practices. Overall, then, this study advances the relevance of the present research to actual classrooms by showing that both fixed and growth mindset-supportive practices can be important when, as in real-world settings, the messages and opportunities are more ambiguous to students.

Study 4: Was the Teacher’s Warm Demeanor Sufficient to Support Students’ Growth Mindsets?

In studies 1 to 3, growth mindset teachers’ messages and practices may have conveyed a warmer demeanor than those of fixed mindset teachers. That is, a teacher who expresses the idea that students can develop their abilities may naturally tend to be perceived as warmer than one who expresses the idea that students’ abilities are fixed. However, the warmth of the teachers’ demeanor itself is not theorized to be the active ingredient that creates an environment that supports students’ use of a growth mindset. Therefore, the goal of the present study was to rule out teacher warmth as a possible explanation for the effects in studies 1 to 3.

Participants in this study were adolescents (n = 256) (preregistration: https://osf.io/zrxem). The procedure was similar to studies 1 and 2, except that participants were randomly assigned to a firm fixed mindset teacher condition (n_Firm = 125) or a warm fixed mindset teacher condition (n_Warm = 131). The speech in the firm fixed mindset teacher condition was identical to that in the fixed mindset teacher condition in studies 1 and 2. The speech in the warm fixed mindset teacher condition communicated a fixed mindset but also a warm demeanor. For example, in the warm fixed mindset teacher interview, the teacher was reported to say, “If you’re struggling, I want you to know that I’m not going to overwhelm you and push you beyond what you can do. […] if math isn’t your thing, I don’t want you to stress out about it.” The manipulation successfully altered students’ perceptions of the teacher’s warmth (SI Appendix). This design allowed us to test whether simply communicating a warm disposition toward students (a common misunderstanding of growth mindset teaching) could produce the same pattern of results as authentic growth mindset messages and opportunities.

Simply making a fixed mindset teacher’s demeanor warmer did not increase participants’ willingness to enact a growth mindset motivational framework (by making more learning-oriented choices) to the extent that a growth mindset-supportive teacher did. The association between participants’ growth mindset frameworks and their choice of the challenging (vs. easy) assignment did not meaningfully differ between the warm and firm fixed mindset teacher conditions (probability of difference in βs between conditions = 0.73). In addition, when we combined the present data with the data collected from the nationally representative panel (study 2), as preregistered, we found that participants’ growth mindset frameworks were more associated with their learning-oriented choices in the growth mindset teacher condition from study 2 as compared to the warm fixed mindset teacher condition in the present study (probability of difference in βs between conditions = 0.93) (Fig. 5). Thus, this study showed that when a teacher’s messages and opportunities supported a fixed mindset, even if the teacher’s demeanor was warm, they nevertheless nullified the links between students’ growth mindsets and their learning-oriented choices.

Fig. 5.

Pattern of results on learning-oriented choices in study 4. Panel A displays the association between motivational framework and assignment choice (standardized coefficients) as a function of condition, estimated in a BCF model. Boxes display the interquartile range, whiskers display the interval from the 10th to 90th percentile of the posterior distribution, and points represent draws from the posterior distribution outside of that range. Panel B displays the posterior distribution of the difference in this association between each condition and the growth mindset teacher condition from study 2 in the nationally representative panel (i.e., the interactions), with a dashed line at zero.

Assessing Everyday Learning-Oriented Choices: Comfort with Academic Risk-Taking.

The focal outcome in this research was an established measure of students’ choice of a challenging hypothetical assignment (18, 22). However, another outcome that was also preregistered was students’ comfort with academic risk-taking (agreement with items like “If I were in this teacher’s class, I would feel comfortable […]”: “raising my hand when I am confused,” “going to the teacher for help after class”). In contrast to the assignment choice outcome, comfort with academic risk-taking captured a subjective feeling of comfort rather than a discrete choice of tasks. The results for this outcome were identical to those for choice of the challenging assignment across the four studies—the fixed versus growth mindset teacher moderated the link between students’ motivational framework and their comfort with academic risk-taking—although in a few cases, there were slight differences in the simple effects (see SI Appendix for full set of results). Overall, when considering a growth mindset teacher, adolescents were more likely to feel comfortable acting on their growth mindset beliefs by taking more intellectual risks in class.

Discussion

Growth mindset interventions have shown meaningful but heterogeneous effects (18, 19), like most other behavioral science interventions. How can this heterogeneity be leveraged to advance theory and practical application? In the present research, we introduced a method for examining heterogeneity in order to gain confidence in causal contextual moderators prior to embarking on large, costly, and under-informed multisite field experiments.

Our research demonstrated a stronger link between students’ growth mindset beliefs and their learning-oriented choices when the teacher conveyed that they supported a growth mindset through their messages and opportunities. This link was substantially attenuated when teachers conveyed a fixed mindset. This research therefore provides the clearest evidence to date for an important cause of growth mindset heterogeneity: teachers’ messages and opportunities. This is an advance because it reveals insights into a widely-debated topic among both scientists and practitioners: growth mindset heterogeneity (see ref. 42). Notably, our conclusions were bolstered by the use of preregistration and conservative Bayesian analyses (all studies), replication in a nationally representative sample (study 2), and studies that ruled out potential confounds (studies 3 and 4). All of this was critical for addressing the possibility that heterogeneity findings are actually the result of researcher degrees of freedom that lead to spurious results.

This research also provided more specific guidance for research and practice in the future. It did so by suggesting that the teacher’s messages and structural opportunities needed to be aligned with a growth mindset for students to fully profit from their growth mindset frameworks (study 3) and by showing that a teacher’s warm demeanor was not sufficient to overcome the growth-suppressing effects of fixed mindset messaging (study 4). An implication of this finding for future teacher-focused interventions is that teachers could be encouraged to consistently afford a growth mindset not only in what they say but also in their class policies. In addition, teachers should avoid using “false” growth mindset practices (such as being nice or giving diffuse encouragement) as a replacement for authentically communicating and enacting the belief that all students can grow and learn.

The present causal evidence for the role of teachers in producing heterogeneity in growth mindset effects therefore sets the stage for the next era of intervention research that manipulates both student mindsets and teachers’ practices, as a means of producing stronger effects on longer-term outcomes, such as academic performance (see ref. 43). This could build on an evidence base that has accumulated over the last several years, mostly in organizational and higher education settings (e.g., refs. 44–49), which has shown that growth mindset-supportive environments are associated with less psychological vulnerability, increased trust, and better student performance. Combined with evidence from the current studies, this body of research will allow researchers to more confidently design field experiments that change teachers’ beliefs and practices in an effort to enhance student mindset effects.

The methodology used in this study lays the groundwork for behavioral science experiments that make heterogeneity a primary interest. Such experiments are urgently needed because many, if not most, prominent findings cannot be relied upon to work predictably (e.g., refs. 50–52). This has led leaders such as Michael Hallsworth, the managing director of the Behavioural Insights Team in the Americas, and his colleagues, to lament that when the reliability of initial results is questionable, then “we have wasted resources […] that could have been allocated more profitably,” and that “it could damage the trust policy-makers and the public have in behavioral science” (53). The present paper offers a useful step on the path to a heterogeneity-attuned science of human attitudes and behavior.

We caution that our scenario method is best used to understand contextual moderators when the following conditions are met. First, the scenario must be able to evoke the contextual moderator, causing the participant to vividly and authentically imagine the real-life situations that are important to the theory. Second, it must be possible to measure a choice at immediate posttest that is a reasonable proxy for the outcome of interest in an intervention and that is not biased due to demand or social desirability effects. These two conditions will help to ensure that the scenario study resembles real-world contexts and outcomes to the extent possible and thereby increase the likelihood that results will be informative for understanding these contexts. Note, however, that our method may be less informative when there is only one plausible interpretation of a moderator. For example, if a nudge involving the simplification of a form (e.g., ref. 54) is only found to be effective among people who have not yet filled out the form, our method will do little to further clarify this pattern of moderation.

What could our method look like for other behavioral science phenomena? Consider three illustrative examples.

The first is the purpose for learning intervention, which is a short reading-and-writing exercise that teaches the lay theory that school can be a place to pursue personally important, self-transcendent goals, such as learning how to make a difference in others’ lives (55). One experiment found that the purpose intervention more powerfully changed adolescents’ performance on an assignment when teachers wrote a note that supported the “purpose” lay theory, expressing the teacher’s belief that the student has the “potential to get an interesting job and make people's lives better one day, if [they] develop [their] skills on assignments like this one” (56). But before researchers conduct a large experiment that trains teachers to write those notes, it would be worthwhile to use the method proposed here to explore potential moderators. For example, what are the different messages and structural opportunities that teachers can provide to support students’ orientation toward a purpose for learning or that can encourage them to see a larger role for themselves in contributing to the solution of societal problems? Multiarm scenario experiments could test different combinations of messages and opportunities and zero in the contextual supports that have the best chance of causally enhancing the benefits of the self-transcendent purpose for learning on students’ behavioral engagement with learning tasks (see ref. 55, study 4 for such a measure of engagement). This could lead to concrete guidance for ambitious field experiments and then for practitioners.

A second example is the Opower descriptive norms experiment discussed at the outset. Descriptive norms reduced energy consumption more effectively in contexts with higher proenvironmental attitudes (9). Before limiting the use of the intervention to only certain districts, or designing campaigns to shift environmental attitudes, it would be important to determine what the relevant aspects of the proenvironmental districts were that led the Opower intervention to be more effective. Scenario experiments could vary how the researchers describe aspects of a neighborhood that can convey proenvironmental attitudes to participants. Such scenario experiments could then test which aspects of proenvironmental neighborhoods enhance or reduce the effects of descriptive norms manipulations (like the Opower intervention). This could help policymakers determine how to create local conditions in which low-cost descriptive norms interventions might reduce energy usage, while also shedding light on the mechanisms of descriptive norms effects.

The two examples discussed above focus on features of the context that can support the message of a behavioral science intervention (see ref. 15). But what if there is no direct message? Some interventions are designed to “shore up the self” to make people less susceptible to threats or challenges, and in these cases, our method could be useful as well. For example, in self-affirmation interventions, participants complete a writing exercise that causes them to reflect on personally important values (e.g., refs. 57–59), and this exercise reduces the impact of information that threatens their identities (see ref. 60). Scenario experiments could help researchers more precisely identify threatening contexts that would benefit from self-affirmation interventions, while also shedding light on the ways that an environment could be changed to reduce the presence of identity threat and thereby reduce the necessity for affirmation interventions in that context.

In conclusion, heterogeneity in behavioral interventions is here to stay, and so it is important to examine it systematically and efficiently. The method we offered here is one important part of an ecosystem of studies that can do this (Fig. 1), building a bridge between large field experiments with measured moderators and future context-level interventions that enhance the individual-level interventions. This method has already led to new, practical, and theoretically informative insights into growth mindset research, and we suspect that it can do the same for many other behavioral phenomena in the future.

Materials and Methods

Each of the four studies was approved by the Institutional Review Board at the University of Texas at Austin. Participants in all studies provided informed consent or assent electronically before participating in the research.

Study 1 Materials and Methods.

Data for this study were collected from adolescent students in partnership with the Character Lab Research Network (N = 1,094). Participants were between 13 and 19 y of age (M_age = 15.12 y). The sample was 52% girls and 53% underrepresented racial/ethnic minority (URM; i.e., Black, Hispanic/Latinx, Native American, Alaska Native, Native Hawaiian, or Pacific Islander). We did not collect data on parent education in this study.

We used a between-subjects design in which participants were randomly assigned to one of two conditions: a growth mindset teacher condition (n_Growth = 570) or a fixed mindset teacher condition (n_Fixed = 524).

Participants were introduced to a study in which they would provide their thoughts on a math teacher. Participants completed a questionnaire in which they reported their growth mindset motivational framework. Next, participants read an interview with a math teacher about what they would say to their students on the first day of class, and they were told that they would subsequently provide their opinions about this teacher.

The content of the teacher’s first-day speech depended on the condition. In the growth mindset teacher condition, the teacher described their approach to teaching the class in a way that provided verbal growth mindset-supportive messages (e.g., “I believe that all students can learn and do well in the class, no matter where they started out”) and class policies that provided opportunities for students to act on a growth mindset (e.g., “If you show improvement in your exam grades over the course of the term, I’ll raise your final grade”).

In the fixed mindset teacher condition, the teacher described their approach to teaching the class in a way that provided verbal fixed mindset-supportive messages (e.g., “Students who do the best at the beginning of the year are typically the same ones who do well at the end”) and class policies that did not provide opportunities for students to act on a growth mindset (e.g., “You cannot make up for problems missed on previous tests, so make sure you’re prepared for each test”).

After reading the teacher’s description of the class, participants were asked to provide their feedback on the teacher. Participants reported their perceptions of the teacher’s mindset beliefs and affordances for the growth mindset (manipulation checks). Participants then reported the likelihood that they would engage in three types of learning-oriented behaviors if they were to take this teacher’s class: their comfort with taking academic risks in the class (such as raising their hand when confused) and their choice of a difficult (vs. easy) assignment. Critically, the assignment choice measure assessed a behavior that was never explicitly raised in the manipulation.^* See SI Appendix for the full set of items and manipulations, as well as a detailed description of the BCF models.

Study 2 Materials and Methods.

Data for this study were collected via a special adolescent research competition held by Time-sharing Experiments for the Social Sciences (TESS). Studies chosen by TESS are typically fielded with a nationally representative panel of adults collected by NORC at the University of Chicago, called the AmeriSpeak panel. However, for this special competition, adults from the AmeriSpeak panel were emailed and asked to invite their adolescent children to participate. The final sample included 803 adolescents between 13 and 17 y of age (M_age = 15.21 y). The sample was 53% girls and 37% URM. In addition, 46% of participants had at least one parent with a bachelor’s degree. Even accounting for nonresponse, probability sample panels such as AmeriSpeak generalize quite well to the population of Americans (61).

We used a between-subjects two-cell design in which participants were randomly assigned to one of two conditions: a growth mindset teacher condition (n_Growth = 411) or a fixed mindset teacher condition (n_Fixed = 392).

The procedure for this study was identical to that of study 2, except, due to space restrictions, we included fewer shorter scales for each outcome (see SI Appendix for details). See SI Appendix for the full set of items and manipulations, as well as a detailed description of the BCF models.

Study 3 Materials and Methods.

Data for this study were collected from adolescent students recruited through Dynata (N = 1,082). We did not retrieve demographic data for this study. However, for reference, we report data from another adolescent sample collected by Dynata using the same recruitment methods in study 4. Demographic descriptive statistics are reported there.

We used a between-subjects design in which participants were randomly assigned to one of four conditions in a 2 (messages: fixed vs. growth) × 2 (opportunities: fixed vs. growth) design. There were 272 participants in the fixed messages/fixed opportunities condition, 271 participants in the growth messages/fixed opportunities condition, 273 participants in the fixed messages/growth opportunities condition, and 266 participants in the growth messages/growth opportunities condition.

The procedure for this study was identical to those of studies 1 and 2, except that rather than only reading an interview with a math teacher, participants also read an interview with one of the math teacher’s former students. The math teacher interview was used to manipulate messages that communicated the teacher’s support for the growth or fixed mindset. The former student interview was used to manipulate information about the teacher’s provision of opportunities to act on the growth mindset. For example, the student interview included the question: “Could you get points back on homework assignments if you didn’t get something right the first time?” In the fixed opportunities conditions, the former student said, “No, you couldn’t get points back on assignments, even if you tried hard and fixed your mistakes,” whereas in the growth opportunities conditions, the former student said, “Yes, you could get points back on assignments if you went back and fixed your mistakes.” See SI Appendix for the full set of items and manipulations, as well as a detailed description of the BCF models.

Study 4 Materials and Methods.

Data for this study were collected from adolescent students recruited through Dynata (N = 256). Participants were between 13 and 18 y of age (M_age = 15.28 y), 47% of participants were identified as girls, 25% were URM, and 44% had at least one parent with a bachelor’s degree.

The procedure for this study was identical to those of studies 1 and 2, except that instead of contrasting a teacher with growth vs. fixed mindset-supportive practices, we contrasted two teachers whose practices supported a fixed mindset. In the firm fixed mindset teacher condition (n_Firm = 125), the teacher’s interview was identical to that of the fixed mindset teacher in studies 1 and 2. In the warm fixed mindset teacher condition (n_Warm = 131), the teacher was portrayed as supporting a fixed mindset but as also communicating a warm demeanor. In particular, the teacher communicated a fixed mindset by holding low standards for struggling students and expressing a desire not to overwhelm students and to ensure that they felt good about themselves, even if they were failing to master the material. For example, in the warm fixed mindset teacher interview, the teacher was reported to say, “If you’re struggling, I want you to know that I’m not going to overwhelm you and push you beyond what you can do. […] if math isn’t your thing, I don’t want you to stress out about it.” See SI Appendix for the full set of items and manipulations, as well as a detailed description of the BCF models.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Anonymized [Anonymized data (studies 1 to 4)] data have been deposited in OSF (10.17605/OSF.IO/RMF9B).