Closing Achievement Gaps with a Utility-Value Intervention: Disentangling Race and Social Class

Abstract

Many college students abandon their goal of completing a degree in STEM when confronted with challenging introductory-level science courses. In the U.S., this trend is more pronounced for underrepresented minority (URM) and first-generation (FG) students, and contributes to persisting racial and social-class achievement gaps in higher education. Previous intervention studies have focused exclusively on race or social class, but have not examined how the two may be confounded and interact. This research therefore investigates the independent and interactive effects of race and social class as moderators of an intervention designed to promote performance, measured by grade in the course. In a double-blind randomized experiment conducted over four semesters of an introductory biology course (N = 1040), we tested the effectiveness of a utility-value intervention in which students wrote about the personal relevance of course material. The utility-value intervention was successful in reducing the achievement gap for FG-URM students by 61%: the performance gap for FG-URM students, relative to CG-Majority students, was large in the control condition, .84 grade points (d = .98), and the treatment effect for FG-URM students was .51 grade points (d = 0.55). The UV intervention helped students from all groups find utility value in the course content, and mediation analyses showed that the process of writing about utility value was particularly powerful for FG-URM students. Results highlight the importance of examining the independent and interactive effects of race and social class when evaluating interventions to close achievement gaps and the mechanisms through which they may operate.

Many students start college intending to pursue a career in science, technology, engineering or math (STEM), but too many abandon this goal after introductory courses, either because they perform poorly, lose interest, feel uncomfortable in the course, or some combination thereof. Some groups are at greater risk for these problems. For example, African Americans, Hispanics, and Native Americans together constitute 26% of the U.S. population, but only 9% of STEM professionals, and 11% of STEM degree recipients in 2008 (National Science Board, 2012). Another group that struggles in college is first-generation (FG) students, those for whom neither parent obtained a 4-year college degree, compared to continuing-generation (CG) students, who have at least one parent with a 4-year degree. FG students constitute roughly 20% of students in American universities and represent a potentially large STEM talent pool, yet they drop out of college at a higher rate (28–35%) than CG students (17%; Chen, 2005; Radford, Berkner, Wheeless, & Shepherd, 2010). If we wish to increase the number of students in science and maximize the chances of discovering talent, it is critically important to promote motivation and performance for underrepresented ethnic minority (URM) and FG students in introductory science courses, which act as a gateway to STEM careers (Ferrini-Mundy, 2013).

Differences in academic performance between URM and majority students are referred to as racial achievement gaps, whereas differences between FG and CG students are referred to as the social-class achievement gap, because parental education is a proxy for socioeconomic status (Fiske & Markus, 2012; Jackman & Jackman, 1983; Pascarella & Terenzini, 1991; Snibbe & Markus, 2005). These achievement gaps can be attributed to a number of economic and social factors such as poverty, quality of schools, economic resources, and academic preparation (Bowen, Kurzweil, & Tobin, 2005; Pascarella & Terenzini, 1991), but they may also reflect psychological factors to the extent that FG and URM students feel less engaged in their classes, feel stigmatized, or worry about whether they belong in the course or at the university (Johnson, Richeson, & Finkel, 2011; Ostrove & Long, 2007).

Brief Interventions

In recent years a number of social-psychological interventions have been developed to help at-risk students in introductory college classes, with striking effects (Yeager & Walton, 2011). These brief interventions are powerful because they are psychologically precise and focus on a specific problem (e.g., identity threat, disengagement) at critical time points (e.g., freshman orientation, gateway courses; Walton, 2014). Some interventions, such as the values affirmation and belonging interventions (Cohen, Garcia, Apfel & Master, 2006; Walton & Cohen, 2011) are domain-general, and focus on students’ self-beliefs and sense of belonging in college. Others, such as the utility-value intervention (Hulleman & Harackiewicz, 2009), are domain-specific and address factors such as task values or interest in a particular course. These interventions have been used to help different groups of students. Some interventions focus on students with a history of poor performance (Hulleman, Godes, Hendricks, & Harackiewicz, 2010), whereas others have targeted either racial gaps (e.g., Cohen et al., 2006; Sherman et al., 2013; Walton & Cohen, 2011) or social class gaps in academic achievement (e.g., Harackiewicz, Canning, et al., 2014; Stephens, Hamedani, & Destin, 2014).

In this research, we tested two brief writing interventions hypothesized to help underrepresented students in an introductory college biology course: the values affirmation (VA) and utility-value (UV) interventions, which work at different levels of domain specificity and address different problems. The VA intervention targets one obstacle to student achievement, namely, identity threat. Students can feel threatened when they are aware of stereotypes about their group or worry about whether they “fit in” at college, and this can undermine their performance (Cohen, Purdie-Vaughns & Garcia, 2012; Ostrove & Long, 2011; Stephens, Fryberg, Markus, Johnson, & Covarrubias, 2012). When individuals affirm their personal values in a threatening environment, however, they can reestablish a perception of personal integrity and worth (Cohen & Sherman, 2014). For example, Cohen et al. (2006) found that a brief writing exercise in which students reflected on their core personal values reduced the gap in grades between African American and European American students by almost 40%. More recently, Harackiewicz, Canning, et al. (2014) found that the same VA technique worked to close the social-class achievement gap in college biology. In contrast, the UV intervention has not previously been tested with respect to racial or social class gaps, but has been shown to help students with a history of poor performance achieve higher grades in their classes (Harackiewicz & Hulleman, 2010).

Utility Value Interventions

The UV intervention targets different psychological processes critical to student achievement: perceived value of and engagement in coursework. It is a curricular intervention in which students write short essays about the personal relevance of course material. For example, a student might write an essay about how what she learned about animal physiology informs her future workout plans, explaining the basic principles of muscle potential and relating them to her life. These course assignments help students discover connections between course topics and their lives. The intervention is based in Expectancy-Value Theory (Eccles & Wigfield, 2002), which argues that individuals choose to take on challenging tasks – such as declaring a STEM major or persisting in a difficult biology course – if they (1) value the task, and (2) expect that they can succeed at the task (based on self-beliefs). Beliefs about the self (e.g., “I am very good at science”) and beliefs about the value of the task (e.g., “Biology is an important field”) are both critically important in predicting academic motivation. However, it may be more feasible to influence students’ subjective task values than their expectations for success (Pajeres, 1996). Eccles (2009; Eccles et al., 1983) identified four types of subjective task values: intrinsic value – the perceived importance of a task because of its inherent enjoyment or interest; attainment value – the perceived importance of a task for an individual’s identity and self-worth; utility value – the perceived importance or usefulness of a task for accomplishing future goals; and cost value – the perceived negative aspects of engaging in a task (e.g. time consumption). Expectancy-Value Theory posits that an increase in any of these values (except cost) will lead to greater motivation toward an academic task.

Of these three positive task values, Eccles and colleagues consider utility value to be the most “extrinsic” because it extends beyond the task itself to connections between that task and other tasks, activities, or goals (Wigfield & Eccles, 1992), and it may therefore be the task value most responsive to external intervention. In educational contexts, a student finds utility value in a topic if they believe it is useful and relevant beyond the immediate situation, for other goals or aspects of their life. Correlational research documents that when students perceive utility value in their courses, they work harder, develop more interest, and perform better (Brophy, 1999; Harackiewicz, Durik, Barron, Linnebrink-Garcia, & Tauer, 2008; Hulleman, Durik, Schweigert, & Harackiewcz, 2008; Wigfield, 1994). Moreover, recent experimental research indicates that it is possible to promote perceived utility value with simple interventions that ask students to write about the relevance of course topics to their own life or to the life of a family member or close friend (Canning & Harackiewicz, 2015; Hulleman et al., 2010; Yeager et al., 2014). This leads students to discover connections between course topics and their lives, in their own terms.

Discovering these connections helps students appreciate the value of course work, and can promote a deeper level of cognitive engagement (Harackiewicz & Hulleman, 2010). In other words, the intervention works by changing how students think about course topics. In addition, we know from experimental laboratory work (Canning & Harackiewicz, 2015) that self-generated utility value information (i.e. utility value connections that students identify for themselves) is more powerful than externally communicated utility value information (as might be produced, for example, when a professor tells students that material is important and relevant). The key is having students work to find the utility value for themselves, which is facilitated through writing assignments that are central to the intervention.

The utility-value intervention can promote academic performance by fostering student engagement with course content, and by helping students find personal value in the material. This intervention works best for students who doubt their competence and for those with a history of poor performance. Students who struggle in classes or doubt their competence are at greater risk for disengagement with course content (Durik, Hulleman & Harackiewicz, 2015). However, Hulleman and Harackiewicz (2009) found that their UV intervention raised interest and grades for 9^th grade science students who had low performance expectations in the course, relative to students in a control group who wrote summaries of course topics. Hulleman et al. (2010) found that a UV intervention promoted interest in an introductory psychology class for students who had performed poorly on early exams, relative to a control group. They also showed that the UV intervention promoted students’ perceptions of utility value in the psychology course, and that this increase in perceived utility value then increased interest, intention to major, and grades.

Can Utility-Value Interventions Close Achievement Gaps?

Although the UV intervention has proven effective in increasing motivation and academic performance for high school and college students who had low success expectancies and/or low performance early in the course, it has not previously been implemented to close racial or social-class achievement gaps. We hypothesize that both URM and FG students will benefit from a UV intervention relative to majority and CG students, in much the same way that students with a history of low performance benefit from these interventions. The UV intervention might prove effective for all students with a history of poor performance, whether URM, FG or not, in which case its efficacy would be attributed to its power to engage students who struggle academically (Durik, Hulleman & Harackiewicz, 2015), regardless of ethnicity or generational status. Alternatively, the intervention might have additional power for URM and FG students. It is important to test whether UV intervention effects are moderated by prior performance, URM status, or FG status, in analytic models that include all three potential moderators (i.e., prior performance, URM status, and FG status), to determine whether the UV intervention has unique potential for URM and/or FG students.

Indeed, there is reason to hypothesize that the UV intervention might promote performance for underrepresented students in STEM courses, over and above the positive effects documented for low-performing students in previous research. Goal congruity (Diekman, Brown, Johnston, & Clark, 2010), cultural mismatch (Stephens et al., 2012) and identity-based motivation (Oyserman & Destin, 2010) theories all suggest that congruity between a person’s identity, culture, or goals and the educational context can serve as a powerful motivational resource, and conversely, that a mismatch between personal goals and the educational context can lead to disengagement. Mismatch problems may be most acute in college STEM courses because the culture of science can be incongruent with the goals of at-risk students (Diekman, Clark, Johnston, Brown, & Steinberg, 2011).

Smith, Thoman and colleagues have found that communal goals such as working with, forming social connections with, or helping others (Pöhlmann, 2001) are more frequently endorsed by URM college students, and that these types of goals are often perceived as inconsistent with the culture of science (Brown, Smith, Thoman, Allen, & Muragishi, 2015; Smith, Chech, Metz, Huntoon, & Moyer, 2014; Thoman, Brown, Mason, Harmsen, & Smith, 2015). Research also suggests that FG students are more likely to have interdependent or “other-focused” goals in college (Harackiewicz, Canning et al., 2014; Piff, Kraus, Côté, Cheng, & Keltner, 2010; Stephens et al., 2012). Indeed, many researchers have noted that communion – a trait which reflects a greater emphasis on working with or helping others – is higher among ethnic minorities (Markus & Connor, 2013), and that Latino, Native American, and African American cultures especially emphasize helping members of their own communities (Fryberg & Markus, 2007; Harper, 2005, Smith et al., 2014, Torres, 2009). This analysis suggests that URM and FG students may struggle to stay engaged in STEM courses that do not seem to share their communal or “other-focused” goals.

We hypothesize that a UV intervention may help underrepresented students find connections to important communal goals in STEM courses even if the courses do not emphasize communal themes in their curriculum (Brown et al., 2015). In other words, the UV intervention may provide underrepresented students the opportunity to make course content congruent with their own goals. By giving students the opportunity to connect science content to their own lives in their own terms, they may be better able to identify the relevance of course topics to their personal goals, and become more motivated to engage with science content. Deriving meaningful connections between science and communal goals may be particularly important for students endorsing such goals, and thus the UV intervention may be uniquely powerful for underrepresented students.

Disentangling Race and Social Class in Intervention Research

Both URM and FG students face disadvantages in college; however, there may be some challenges specific to URM students, and not FG students (e.g., racial discrimination), and some that are unique to FG students (e.g., fewer educational/ financial resources). Moreover, because race and social class are increasingly correlated in American society (Duncan & Murnane, 2011; Reardon, 2011), some students (those who are both URM and FG) may experience both sets of challenges. Thus students may face challenges related to their URM status, their FG status, or both, and disadvantages may be compounded for FG-URM students (Jack, 2014). A number of social-psychological intervention studies have addressed either racial gaps (e.g., Cohen et al., 2006; Sherman et al., 2013; Walton & Cohen, 2011) or social class gaps in academic achievement (e.g., Harackiewicz, Canning, et al., 2014; Smeding, Dumas, Loose, & Régner, 2013; Stephens, Hamedani, & Destin, 2014). Given that social class and ethnicity are often confounded, it is not always clear whether interventions are addressing racial or social-class achievement gaps, or both. For example, Stephens and colleagues (2014) tested a difference-education intervention (in which incoming students learned about how different educational backgrounds can impact college experiences) that improved performance for FG students, yet 38% of the FG students were also African American or Latino. Although the researchers controlled for ethnicity in their analyses, they did not consider the interactions between race and social class.

Similarly, many interventions to close racial achievement gaps have not considered the impact of social class. In the Cohen et al. (2006) study testing a VA intervention with African American students, students’ social class (or parents’ educational attainment) was not reported nor was it included in analyses. In another VA study, Sherman et al. (2013) noted that virtually all of the Latino students were receiving lunch assistance (whereas few of the white students were), and that the racial gap was thus largely redundant with the social-class achievement gap. This overlap of race and social class in some populations makes it especially difficult to identify the groups for whom interventions are most effective.

It should be possible, however, to examine the independent and interactive effects of race and social class with a diverse sample and adequate statistical power. One goal of the current research is to disentangle intervention effects associated with race from those associated with social class in a large-scale intervention study with college students in an introductory biology course. To explore these possibilities, we collected baseline measures of prior academic performance, high-school poverty rate, performance expectancies for the biology class, science background, psychological experiences, and motivation. Consideration of these variables may help us identify the psychological processes most relevant for interventions with particular groups of students (Walton, 2014), and help interpret differences in responsivity to social-psychological interventions among students who may have intersecting identities.

Intersectionality

Intersectionality is a theoretical approach that simultaneously considers multiple categories of identity, difference, and disadvantage, such as gender, race, social class, sexual orientation, disability, and religion (Cole, 2009). This approach, originally based in critical race theory and feminist theory (Few-Demo, 2014), is critical of researchers’ tendency to consider a social category, such as African Americans, to be homogeneous, when in fact members of that category vary substantially on other dimensions such as social class and gender. In the research reported here, we consider the intersection of race and social class.

Ferree (2010) noted that an approach is labeled intersectional if it considers multiple dimensions of inequality “and considers how they interactively define the identities and experiences … of individuals and groups” (p. 428). There is some debate among intersectionality researchers regarding whether, when intersectionality is combined with quantitative methods, it involves testing for multiple main effects of the intersecting dimensions, or whether it should involve statistical interactions between intersecting dimensions (Ferree, 2010). In the case of the intersection of race and social class, the main-effects approach can detect the additive effects of race and social class, thereby disentangling these effects, which are so often confounded. The statistical interaction approach, which is consistent with Ferree’s definition above, examines whether certain combinations of race and social class (such as URM and FG status) have especially potent effects. In the research reported here, we test for both additive and interaction effects. Moreover, because intersectionality researchers have rarely used experimental methods, we present a novel test of intersectionality by testing a three-way interaction between the intervention and two intersecting person factors, race and social class.

The Present Study

We tested two interventions, one that has proven to be effective for reducing some achievement gaps in middle-school and college classes (the VA intervention), and one that has proven to be effective for students with a history of low academic performance (the UV intervention). We address two primary questions about these interventions: 1) Can the UV intervention reduce achievement gaps? 2) Can the VA intervention be paired with a UV intervention? In other words, can a domain-general, identity-based intervention (VA) and a domain-specific, curriculum-based intervention (UV) be combined in a college biology course, and how might they work together? Such a combination might be additive or synergistic, with each boosting the effectiveness of the other, but these two interventions have not previously been combined, and it is unclear whether two writing-based interventions can be implemented effectively in a single course.

In this study, we focused on closing achievement gaps in an introductory biology class, and tested UV and VA interventions in a 2 x 2 crossed design. Our design and a large sample allowed us to disentangle the independent and interactive effects of race (URM or Majority) and generational status (FG or CG) in moderating the effectiveness of the interventions. Although we focus mainly on social-psychological variables, we also included a social-structural variable, percent free/reduced lunch at the students’ high schools, as a proxy for poverty at both the school and neighborhood level.

Method

We implemented the UV and VA interventions in an introductory biology course in a double-blind, randomized experiment at a large Midwestern university, across four semesters, in the first course of a two-course sequence. This foundational course, offered in both Fall and Spring semesters, is a prerequisite for 34 undergraduate biomedical majors at this university (e.g., biochemistry, neuroscience, nursing, zoology) and a critical gateway course for premedical preparation and further study in the biological sciences. Over these four semesters, 2378 students were enrolled in this course, of whom 8% were underrepresented minority students (URM) and 21% were first-generation students (FG; i.e., no parent/guardian obtained a 4-year college degree).¹ All consenting URM and FG students enrolled in the course were included in this study, as well as a randomly selected subset of continuing-generation (CG) Majority students (CG-Majority students in this sample were 82% White, 18% Asian or Asian American). The remaining CG-Majority students in the course received comparable assignments but were not included in the study. Of the 1060 undergraduate students who were eligible for this study across four semesters, 1040 (417 male, 623 female) completed the course and gave consent for access to their academic records (6 students did not consent, and 14 dropped the course). Participants were 423 CG-Majority, 427 FG-Majority, 126 CG-URM (51 African American, 61 Hispanic, 14 Native American), and 64 FG-URM (26 African American, 35 Hispanic, 3 Native American) students.

The 15-week biology course covered 3 units: cellular biology, genetics, and either evolutionary biology (in semesters 1 and 2), or animal physiology (in semesters 3 and 4). Students met three times per week for 50-minute lectures. Between one and three lecture sections were offered each semester (for a total of 8 lecture sections across the 4 semesters). In addition to lectures, students attended a 3-hour laboratory, led by a graduate teaching assistant, once each week. There were approximately 15–40 laboratory sections in each semester. Students also attended a 50-minute recitation each week, led by a different graduate teaching assistant.

Interventions

Students were blocked on URM and FG status, gender, and lecture section and then randomly assigned to one of 4 conditions in a fully crossed 2 x 2 (UV intervention x VA intervention) experimental design. Instructors and teaching assistants were blind to experimental condition.

UV Intervention

Students completed either three UV or three control assignments. These writing assignments were fully integrated into the course and were presented as a course assignment from the instructors. Three weeks prior to each unit exam, course instructors emailed the assignment to each student. Students were given five days to complete each essay and turned them in via an online course management site. In both conditions, students were asked to:

Select a concept or issue that was covered in lecture and formulate a question. Select the relevant information from class notes and the textbook, and write a 1–2 page essay.

The utility value (UV) assignment varied slightly across the four semesters, but all UV assignments asked students to answer their question using course material and discuss the relevance of the concept or issue to their own life or to the lives of others:

Write an essay addressing this question and discuss the relevance of the concept or issue to your own life. Be sure to include some concrete information that was covered in this unit, explaining why this specific information is relevant to your life or useful for you. Be sure to explain how the information applies to you personally and give examples.

In contrast, the control assignment instructed students to address their question by summarizing course material:

Select the relevant information from class notes and the textbook, and write a 1–2 page response to your question. You should attempt to organize the material in a meaningful way, rather than simply listing the main facts or research findings. Remember to summarize the material in your own words.

Biology graduate students were hired to grade each assignment on scientific merit and to ensure that students followed directions. Although some graduate student graders were also teaching assistants, graders were never assigned to grade essays of students who were in their laboratory or recitation sections. The fact that assignments and graded feedback were turned in via a course management site (and not during lecture or lab time) ensured that all teaching assistants remained blind to their students’ conditions. Grader feedback and essay grades were provided to each student a few days before the unit exam. Each assignment was worth 0.6% of the final grade in the course. 1034 students completed the first assignment, 1017 students completed the second essay assignment, and 1006 students completed the third essay assignment (95% of students completed all three assignments).

VA Intervention

The VA intervention was administered in laboratory sessions early in the semester, and students wrote about personal values, as in previous research (Harackiewicz, Canning et al., 2014; Miyake et al., 2010). Students in the VA condition were instructed to write about why two or three values, selected from a list, were important to them. Students in the control condition were instructed to choose the two or three values that were least important to them, and to write about why other people might hold those values. Full methodological details regarding the implementation of VA are reported by Authors (2014).

Baseline Measures

In the second week of the course, a questionnaire was administered in laboratory sections, with questions about attitudes about biology and demographic information. All questionnaire items were answered on a 7-point scale ranging from “not at all true” to “very true” or “not at all” to “a lot,” unless otherwise noted. Scale scores represent the mean of constituent items. Missing data (less than 1% on each measure) were handled by multiple imputation (Rubin, 1987).

Attitudes about biology

Biology background was measured with three items (for each of three topics covered in the course: “I have a strong background in [cellular biology, genetics, and evolution or animal physiology],” α = .84). Belonging uncertainty (Walton & Cohen, 2011) was measured with two items (“When something bad happens, I feel that maybe I don’t belong at University X,” “Sometimes I feel that I belong at University X, and sometimes I feel that I don’t belong at University X,” α = .83). Competence valuation (Harackiewicz & Sansone, 1991) was measured with two items (“It is important to me to do well in this course,” “I want to do well in Introductory Biology,” α = .71). Desire to contribute to society was measured with one item (“I want to study biology because I want to make a contribution to society”). Confidence about performance was measured with three items (“I am confident that I will do well in Introductory Biology,” “I expect to get a good grade in this course,” “I am confident that I can obtain a final grade of B or better in this course,” α = .82). Interest in biology was measured with five items (“I’m really looking forward to learning more about biology,” “Biology fascinates me,” “I think the field of biology is very interesting,” “I’m excited about biology,” “To be honest, I just don’t find biology interesting,” reversed, α = .93). Perceived utility value was measured with four items (“The material we are studying in this course is useful for everyone to know,” “This class is important to my future,” “I think what we are learning in Introductory Biology 151 is important,” “The study of biology is personally important to me,” α = .74).

Motives for attending college

We administered a shortened version of Stephens and colleagues’ (2012) scale in which students were asked to indicate which of 10 items characterized their reasons for completing their college degree (checking as many as were relevant). Half the items referred to independent motives (i.e., “Become an independent thinker,” “Learn more about my interests,” “Prepare for a future career,” “Expand my understanding of the world,” and “Expand my knowledge of the world”), whereas the other half referred to interdependent motives (i.e., “Help my family out after I’m done with college,” “Give back to my community,” “Provide a better life for my own children,” “Show that people with my background can do well,” and “Be a role model for people in my community”). Independent and interdependent motives were measured by counting how many of the motives in each category were selected. In addition, we constructed a measure of helping motives by counting how many of three interdependent motives that demonstrate the instrumentality of a college education for helping others (“Give back to my community,” “Help my family out after I’m done with college,” “Provide a better life for my own children”) were selected.

Prior GPA and High-School Poverty Rate

We obtained students’ GPA from prior semesters from university records (n = 978). We also obtained information about the high school each student attended, in terms of the percentage of students who received financial assistance for school meals (percent free or reduced lunch) at those schools, as an indicator of poverty at the school or neighborhood level (n = 979). However, given that some students were freshmen or transfers, and that high-school information was not available for all students, we were missing some data on both measures. Thus, we used multiple imputation (Rubin, 1987) to create a measure of prior GPA and Free/ Reduced Lunch (FRL) for all students.

Outcome Measures

Coding of articulated utility value and essay length

The utility value and control writing assignments were coded for the level of utility value articulated in each essay. Research assistants coded the assignments on a 0–4 scale based on how specific and personal the utility value connection was to the individual. A “0” on this scale indicates no utility; a “1” indicates general utility applied to humans generically; a “2” indicates utility that is general enough to apply to anyone, but is applied to the individual; a “3” indicates utility that is specific to the individual; and a “4” indicates a strong, specific connection to the individual that includes a deeper appreciation or future application of the material. Utility value scores from the three essays were summed to create an overall measure of articulated utility value. Inter-rater reliability with this coding rubric was high, with two independent coders providing the same score on 91% of essays. Disagreements were resolved by discussion. Research assistants also recorded the number of words in each essay so that we could test whether students wrote more in UV conditions, or if certain groups of students wrote more, possibly reflecting higher levels of task engagement.

Biology course grade

Course instructors provided final course grades at the end of the semester (4.0 scale: A = 4.0, AB = 3.5, B = 3.0, BC = 2.5, C = 2.0, D = 1.0, F = 0). Grading standards were consistent across sections and semesters.

Results

Overview

Table 1 presents descriptive statistics and intercorrelations for all measures. Although students were randomly assigned to condition at the student level, we used hierarchical linear modeling (HLM) to account for the nested structure of the data (students nested within eight lecture sections, across four semesters; Raudenbush & Bryk, 2002). We tested a two-level random-intercept model in which students were nested within the eight lecture sections, taking into account the interdependencies in the data – between students in the same lecture section – by estimating within and between component variance. The intraclass correlation coefficient was small; lecture sections accounted for only 1.67% of the variance in biology course grade. Although this analysis demonstrated that the nesting of students would not have a large effect on our analyses compared to multiple regression models, we modeled the nesting structure so that accurate standard errors would be obtained. We present comparisons of regression and HLM results for the primary analyses in Tables 2 and 3. Analyses with HLM and regression yielded consistent results. Regression results are reported here so that we can report effect sizes (betas).

Table 1.

Zero-Order Correlations and Descriptive Statistics

	1	2	3	4	5	6	7	8	9	10	11	12	13	14
1. Percent Free/Reduced Lunch	--
2. Prior GPA	−.18**	--
3. Confidence about Performance	−.04	.14**	--
4. Interest in Biology	.00	−.03	.33**	--
5. Perceived Utility Value	−.02	−.03	.26**	.73**	--
6. Biology Background	−.05	−.09**	.28**	.22**	.19**	--
7. Belonging Uncertainty	.10**	−.16**	−.22**	−.07*	−.05	−.10**	--
8. Competence Valuation	−.01	.06*	.21**	.31**	.37**	.03	−.03	--
9. Desire to Contribute to Society	.01	.03	.26**	.54**	.56**	.16**	−.08*	.29**	--
10. Helping Motives	.13**	−.09**	.06	.05	.11**	−.03	.00	.17**	.15**	--
11. Independent Motives	.01	.02	.08**	.16**	.12**	−.05	−.09**	−.01	.08**	.15**	--
12. Articulated Utility Value	−.06*	.06	−.01	−.02	.01	−.06*	−.03	−.01	.03	.01	−.02	--
13. Average Essay Length	−.03	.20**	.06*	.11**	.08**	−.02	.00	.07*	.05	.02	.02	.15**	--
14. Biology Course Grade	−.21**	.58**	.23**	.08*	.05	.08**	−.12**	.06	.10**	−.13**	−.01	.07*	.27**	--
Mean	26.66	3.14	5.80	5.80	5.78	4.16	3.22	6.82	5.60	2.12	3.62	5.57	485.93	2.78
SD	16.94	0.52	0.87	1.05	0.90	1.32	1.79	0.40	1.49	0.92	1.55	4.07	102.61	0.81
N	1040	1040	1040	1040	1040	1040	1040	1040	1040	1040	1040	1040	1040	1040

^*p < .05,

^**p < .01.

Table 2.

Effects of URM status, FG status, and Gender on Baseline Measures

	HLM			Regression
	γ	F (df)	p	β	t (df)	p
Prior GPA
URM	−0.16	15.62 (1, 1035)	.000	−0.13	3.96 (1035)	.000
FG	−0.16	15.86 (1, 1035)	.000	−0.16	3.99 (1035)	.000
URM x FG	0.01	0.06 (1, 1035)	.813	0.01	0.23 (1035)	.816
Gender	0.06	3.60 (1, 1034)	.058	0.06	1.91 (1035)	.057
% Free/Reduced Lunch
URM	0.27	45.39 (1, 1035)	.000	0.21	6.75 (1035)	.000
FG	0.34	72.85 (1, 1035)	.000	0.34	8.55 (1035)	.000
URM x FG	0.14	11.41 (1, 1035)	.002	0.14	3.38 (1035)	.001
Gender	0.00	0.01 (1, 1035)	.670	0.00	0.08 (1035)	.935
Confidence about Performance
URM	−0.03	0.44 (1, 1035)	.509	−0.02	0.66 (1035)	.511
FG	−0.02	0.32 (1, 1035)	.572	−0.02	0.56 (1035)	.579
URM x FG	0.04	0.08 (1, 1035)	.773	−0.01	0.29 (1035)	.771
Gender	0.25	65.10 (1, 1034)	.000	−0.24	8.08 (1035)	.000
Interest in Biology
URM	0.06	1.88 (1, 1035)	.170	0.05	1.37 (1035)	.170
FG	0.05	1.35 (1, 1035)	.246	0.05	1.16 (1035)	.245
URM x FG	0.02	0.19 (1, 1035)	.661	0.02	0.44 (1035)	.661
Gender	0.00	0.01 (1, 1035)	.910	0.00	0.11 (1035)	.910
Perceived Utility Value
URM	0.07	2.56 (1, 1035)	.110	0.05	1.60 (1035)	.109
FG	0.02	0.23 (1, 1035)	.632	0.02	0.48 (1035)	.631
URM x FG	0.00	0.00 (1, 1035)	.992	0.00	0.01 (1035)	.992
Gender	−0.08	7.14 (1, 1035)	.008	0.08	2.68 (1035)	.008
Biology Background
URM	−0.09	4.45 (1, 1035)	.035	−0.07	2.11 (1035)	.035
FG	−0.02	0.20 (1, 1035)	.655	−0.02	0.45 (1035)	.654
URM x FG	−0.02	0.26 (1, 1035)	.613	−0.02	0.51 (1035)	.613
Gender	−0.04	1.89 (1, 1035)	.169	0.04	1.38 (1035)	.169
Belonging Uncertainty
URM	−0.01	0.12 (1, 1035)	.732	−0.01	0.34 (1035)	.732
FG	0.08	3.86 (1, 1035)	.050	0.08	1.97 (1035)	.049
URM x FG	0.01	0.08 (1, 1035)	.775	0.01	0.29 (1035)	.774
Gender	−0.08	5.86 (1, 1035)	.016	0.08	2.42 (1035)	.016
Competence Valuation
URM	0.08	4.12 (1, 1035)	.043	0.07	2.03 (1035)	.042
FG	0.01	0.09 (1, 1035)	.761	0.01	0.31 (1035)	.761
URM x FG	−0.02	0.13 (1, 1035)	.714	−0.02	0.37 (1035)	.714
Gender	−0.13	15.82 (1, 1035)	.000	0.12	3.98 (1035)	.000
Desire to Contribute
URM	0.08	3.47 (1, 1035)	.063	0.06	1.87 (1035)	.062
FG	0.01	0.06 (1, 1035)	.808	0.01	0.24 (1035)	.807
URM x FG	−0.03	0.57 (1, 1035)	.493	−0.03	0.69 (1035)	.492
Gender	−0.05	2.25 (1, 1035)	.134	0.05	1.50 (1035)	.134
Helping Motives
URM	0.20	22.71 (1, 1035)	.000	0.15	4.77 (1035)	.000
FG	0.21	26.08 (1, 1035)	.000	0.21	5.12 (1035)	.000
URM x FG	0.09	5.27 (1, 1035)	.022	0.09	2.30 (1035)	.022
Gender	0.08	7.50 (1, 1035)	.006	0.08	2.74 (1035)	.006
Independent Motives
URM	0.05	1.49 (1, 1035)	.223	0.04	1.25 (1035)	.210
FG	−0.00	0.00 (1, 1035)	.971	−0.00	0.03 (1035)	.978
URM x FG	0.07	2.82 (1, 1035)	.093	0.07	1.69 (1035)	.091
Gender	−0.09	9.00 (1, 1034)	.003	−0.09	2.98 (1035)	.003

Note. URM = Underrepresented Minority (URM = +1, Majority = −1), FG = First-generation (FG = +1, Continuing-generation = −1). Gender (Female = +1, Male = −1).

Table 3.

Effects of the UV Intervention, URM Status, and FG Status on Biology Course Grade

	HLM			Regression
	γ	F (df)	p	β	t (df)	p
UV Intervention	0.06	4.74 (1, 1027)	.030	0.08	2.26 (1028)	.024
URM	−0.14	25.54 (1, 1025)	.000	−0.13	5.05 (1028)	.000
FG	−0.02	0.82 (1, 1025)	.364	−0.03	0.85 (1028)	.393
URM x FG	0.00	0.01 (1, 1024)	.911	0.00	0.06 (1028)	.950
UV x URM	0.04	2.75 (1, 1023)	.097	0.06	1.69 (1028)	.092
UV x FG	0.01	0.11 (1, 1026)	.737	0.01	0.24 (1028)	.808
UV x URM x FG	0.07	6.33 (1, 1023)	.012	0.08	2.45 (1028)	.015
Confidence about Performance	0.12	36.95 (1, 1025)	.000	0.15	6.16 (1028)	.000
Prior GPA	0.44	456.36 (1, 1025)	.000	0.55	21.32 (1028)	.000
UV x Confidence	0.02	1.46 (1, 1024)	.227	0.03	1.27 (1028)	.203
UV x Prior GPA	−0.04	4.12 (1, 1026)	.043	−0.05	2.01 (1028)	.045

Note. UV = Utility Value intervention condition (UV = +1, Control = −1), URM = Underrepresented Minority (URM = +1, Majority = −1), FG = First-generation (FG = +1, Continuing-generation = −1).

Background Differences for URM and FG Students

We tested the main effects of URM status (Majority = −1, URM = 1), FG status (CG = −1, FG = 1), and their interaction for each baseline measure, controlling for gender (female = 1, male = −1) on all baseline measures.² Table 2 presents the full regression and HLM results for all background variables.

Prior GPA and high-school poverty rate

On prior GPA, there were independent negative effects for both URM and FG status, β = −0.13, p < .001, and β = −0.16, p < .001, respectively, indicating both a racial achievement gap (Cohen’s d = 0.27) and a social-class achievement gap (d = 0.31), which when considered together, indicate that FG-URM students had the lowest prior university GPAs (Figure 1, Panel A). On high-school poverty rate (FRL), independent positive effects for both URM and FG status, β = 0.21, p < .001, and β = 0.34, p < .001, respectively, revealed that URM students attended more impoverished schools than Majority students, and that FG students attended more impoverished schools than their CG peers. Furthermore, the URM by FG interaction was also significant, β = 0.14, p = .001. FG-URM students attended the most impoverished high schools and, by implication, lived in the most impoverished neighborhoods (See Figure 1, Panel B).

Figure 1.

Prior GPA (Panel A) and high-school poverty rate (indicated by the percentage of students who received free or reduced-priced lunch; Panel B) as a function of URM (Majority or URM) and generational status (CG or FG). Error bars are 95% confidence intervals.

Attitudes about biology and motives for attending college

There were no independent or interactive effects of URM or FG status on confidence about performance, interest in biology, perceived utility value, or independent motives for attending college, p > .100. There was a negative effect of URM status on biology background, β = −0.07, p = .035, a positive effect of FG status on belonging uncertainty, β = 0.08, p = .049, as well as positive effects of URM status on competence valuation, β = 0.07, p = .042, and desire to contribute, β = 0.06, p = .062. On helping motives, we found independent positive effects for both URM and FG status, β = 0.15, p < .00 and β = 0.21, p < .001, respectively, as well as a URM by FG status interaction, β = 0.09, p = .022, indicating that FG-URM students had the strongest motivation to help their families and communities.³ In fact, 80% of FG-URM students selected all three helping motives as reasons for attending college. Figure 2 presents these variables, standardized around the overall sample mean, for the four groups: CG-Majority, FG-Majority, CG-URM, and FG-URM. Considered together, these results show a unique pattern of challenges (higher high-school poverty rate, lower prior GPA, weaker perceived biology background, higher belonging uncertainty) and positive motivations (higher levels of competence valuation, desire to make a contribution, and helping motives) for the FG-URM students in this class.

Figure 2.

Standardized baseline measures as a function of URM (Majority or URM) and generational (CG or FG) status.

Gender Differences

We found significant effects of gender on five of eleven baseline measures. Females reported lower levels of confidence about performance, β = −0.24, p < .001, and higher levels of perceived utility value, β = 0.08, p = .008, competence valuation, β = 0.12, p < .001, belonging uncertainty, β = 0.08, p = .016, and helping motives as reasons for attending college, β = 0.08, p = .006, than male students. The effect of gender on prior GPA was not significant β = 0.06, p = .057, but women had slightly higher prior GPAs than men (d = 0.10). In contrast, men performed slightly better than women in this class (d = −0.16), but this difference was also nonsignificant, p = .22.

Course Performance

The primary outcome measure was course grade. Preliminary analyses revealed that there were no significant effects of the VA intervention for any group (i.e., there was no main effect of VA, no VA x URM status interaction, no VA x FG status interaction, and no VA x URM x FG interaction), and no significant interactions of VA with UV (these statistical tests are reported below). Therefore, we collapsed across VA condition for the analyses reported here, resulting in a two-cell UV vs. control design. We used confidence about performance as a covariate to control for baseline performance expectations, and prior GPA as a covariate to control for prior performance. The basic model, which was tested for biology course grade and related variables, included 11 terms: the main effects of the UV intervention, URM status, and FG status, 3 two-way interactions (UV intervention x URM status, UV intervention x FG status, and URM status x FG status), 1 three-way interaction (UV intervention x URM status x FG status), as well as the 2 covariates (confidence about performance and prior GPA), and 2 two-way interactions between the UV intervention and each of the covariates.⁴ Including both covariates and their interactions with the UV intervention allowed us to test whether the effects found here are consistent with previous studies that found the UV intervention was particularly effective for students with low performance expectations (Hulleman & Harackiewicz, 2009) and students with low grades (Hulleman et al., 2010). Table 3 presents the results for course performance in regression and HLM.

For course grade, there was a negative effect for URM status, β = −0.13, p < .001, showing that Majority students obtained higher grades in the class than URM students. There were also positive effects of confidence about performance, β = 0.15, p < .001, and prior GPA, β = 0.55, p < .001, indicating that confident students and students with higher prior GPAs performed better in the course than students with lower confidence and students with lower prior GPAs, respectively.

A significant UV main effect, β = 0.08, p = .024, showed that the UV intervention improved performance slightly for all students, on average (M = 2.81, 95% CI [2.74, 2.87]), relative to control (M = 2.76, 95% CI [2.68, 2.83]), d = 0.06. In addition, course performance was somewhat higher for URM students in the UV condition relative to control, compared to majority students, β = 0.06, p = .092 for the two-way interaction of UV with URM status (Figure 3, Panel A). The performance gap for URM students was substantial in the control condition, d = .60, p < .001. The treatment effect for URM students was .20 grade points (d = 0.23), resulting in a 40% reduction in the racial achievement gap.

Figure 3.

Course performance as a function of treatment condition (UV or control) and (A) URM status (Majority or URM) and as a function of (B) both URM and generational status (CG or FG). Error bars are 95% confidence intervals.

However, this two-way interaction effect was qualified by a significant three-way interaction between the UV intervention, URM status, and FG status, β = 0.08, p = .015. The UV intervention was most effective for FG-URM students (Figure 3, Panel B). The performance gap for FG-URM students, relative to CG-Majority students, was large in the control condition, .84 grade points, d = .98, p < .001. The treatment effect for FG-URM students was .51 grade points (d = 0.55), resulting in a 61% reduction in the achievement gap for these students.

UV replication analyses

The inclusion of the interactions between the UV intervention and both confidence about performance and prior GPA in our basic model allowed us to test for replication of prior utility value research, and we found evidence for partial replication. Hulleman and Harackiewicz (2009) found that a UV intervention was most effective for students with low performance expectations. This finding was not replicated in our sample, β = 0.03, p = .203. However, there was a significant negative interaction between the UV intervention and prior GPA, β = −0.05, p = .045, indicating that the UV intervention was most effective for students who had the lowest prior GPAs. This finding replicates previous research (Hulleman et al., 2010), and suggests that the UV intervention can help students with a history of poor performance, in addition to helping FG-URM students.

Testing the effects of high-school poverty rate

Thus far we have focused on one important aspect of social class, first-generation student status. However, social class encompasses both generational status and poverty levels. In order to test high-school poverty as a predictor of academic performance and determine whether UV effects were moderated by high-school poverty, we added several new terms to our basic model. Specifically, we added the main effect of high-school percent free/reduced lunch (FRL), 2 two-way interactions with FRL (UV Intervention x FRL and URM status x FRL), and the three-way interaction between the UV intervention, FRL, and URM status. In this model, there was a significant main effect of FRL on course performance, t(1024) = 3.30, p = .001, β = −.10, showing that students from poorer high schools obtained lower grades in the course. However, there were no significant interactions of FRL with either URM status or the UV intervention, p’s > .30. In contrast, the positive main effect of the UV intervention remained significant, t(1024) = 1.93, p = .05, β = .07, as well as the interaction between the UV intervention and prior GPA, t(1024) = 2.00, p = .05, β = −.05. Most important, the UV x FG x URM interaction remained significant, t(1024) = 2.19, p = .03, β = .08, with FRL controlled.⁵ These analyses suggest that although high-school poverty and generational status both influenced academic performance, the UV intervention was effective for FG-URM students even when FRL was taken into account.

Mediation Analyses

To understand the motivational processes underlying the effects of the UV intervention, we first examined the content of students’ essays in terms of articulated utility value, with the basic model described above. As expected, students in the UV condition articulated more utility value (i.e., made more personal connections to curricular content) in their essays (M = 8.76, 95% CI [8.57, 8.95]) than controls (M = 1.56, 95% CI [1.44, 1.69]), β = 0.87, p < .001. This important manipulation check indicates that our curricular intervention was successful in encouraging students to make personal connections with the course material in their writing assignments. There was also a significant effect of prior GPA, β = .05, p = .001, and a significant UV x GPA interaction, β = 0.06, p < .001; students with higher prior GPAs articulated more utility value in their essays, particularly in the UV condition. However, there were no significant interactions with FG or URM status (p > .40), suggesting that the effect of the intervention was comparable for all groups, in terms of helping students articulate the utility value of course topics.

Next, we examined the length of students’ essays with the same regression model. There was a main effect of prior GPA, β = 0.19, p < .001; students with higher prior GPAs wrote longer essays. In addition, students wrote longer essays in the UV condition (M = 506 words, 95% CI [476.99, 534.49]) compared to control (M = 479, 95% CI [469.23, 488.39]), β = 0.11, p = .008, suggesting that the UV intervention promoted engagement with curricular content, averaged across all students. A three-way interaction between the UV intervention, URM and FG status, β = 0.12, p = .004, revealed that FG-URM students showed the largest intervention effect, writing longer essays in the UV condition (M = 506 words, 95% CI [476.99, 234.49]) than in control (M = 433, 95% CI [389.10, 476.32]), suggesting that they became particularly engaged with this assignment. Compared to their peers in the control condition, FG-URM students in the UV condition wrote, on average, about 73 more words in their essays (Figure 4).

Figure 4.

Average essay length as a function of treatment condition (UV or control), URM status (Majority or URM), and generational status (CG or FG). Error bars are 95% confidence intervals.

Considered together, these analyses suggest that students in the utility value condition made more personal connections to the course material in their essays (as requested by the utility value assignment) than in control conditions, but that FG-URM students became particularly involved in the process of writing about utility value, writing longer essays in the intervention condition. A critical question is whether essay length (an indicator of cognitive engagement) mediated the effects of the UV intervention on performance for FG-URM students, who wrote more words and who benefited most from the intervention, in terms of course performance.

Moderated mediation analysis

We tested the indirect effects of the UV intervention on course grade through essay length as a function of URM and FG status, in a test of moderated mediation (Hayes, 2013, Model 12). Table 4 shows the effects of UV intervention, URM status, FG status, and essay length on course grade and the indirect effect for each student group (i.e., CG-Majority, FG-Majority, CG-URM and FG-URM). The confidence interval testing the indirect effect of the UV intervention through essay length does not include zero for FG-URM (95% CI: [0.010, 0.076]) and CG-Majority (95% CI: [0.004, 0.028]) students.⁶ Thus we can conclude that essay length partially mediated the positive effect of the UV intervention on course grade for these two groups of students, with the largest indirect effect for FG-URM students, who wrote longer essays and benefitted most from the UV intervention.

Table 4.

Essay Length as a Mediator of the UV Intervention Effect

Predictor	B	SE	t	p
Essay Length
UV Intervention	0.11	0.04	2.66	.008
URM Status	−0.06	0.04	1.36	.174
FG Status	0.02	0.04	0.47	.642
URM x FG	0.00	0.04	0.03	.980
UV x URM	0.06	0.04	1.37	.171
UV x FG	0.05	0.04	1.08	.279
UV x URM x FG	0.12	0.04	2.91	.004
Confidence about Performance	0.04	0.03	1.24	.214
Prior GPA	0.19	0.03	6.08	.000
UV x Confidence	0.01	0.03	0.20	.839
UV x Prior GPA	0.01	0.03	0.23	.819
Course Grade
Essay Length	0.11	0.02	5.45	.000
UV Intervention	0.05	0.03	1.84	.067
URM Status	−0.13	0.03	4.89	.000
FG Status	−0.03	0.03	0.94	.345
URM x FG	0.00	0.03	0.06	.952
UV x URM	0.04	0.03	1.48	.140
UV x FG	0.00	0.03	0.06	.952
UV x URM x FG	0.05	0.03	1.98	.048
Confidence about Performance	0.12	0.02	6.03	.000
Prior GPA	0.42	0.02	20.22	.000
UV x Confidence	0.03	0.02	1.26	.208
UV x Prior GPA	−0.04	0.02	2.08	.038

Mediator	Student Status	Boot indirect effect	Boot SE	Boot LLCI	Boot ULCI
Essay Length	CG-Majority	0.01	0.01	0.004	0.028
Essay Length	FG-Majority	0.00	0.01	−0.014	0.008
Essay Length	CG-URM	0.00	0.01	−0.016	0.018
Essay Length	FG-URM	0.04	0.02	0.010	0.076

Exploratory Text Analyses

To investigate essay content more fully, we used Linguistic Inquiry and Word Count (LIWC) text analysis to determine whether the content of the essays varied as function of the UV intervention and URM or FG status. LIWC software calculates the degree to which people use different categories of words (e.g., personal pronouns, social processes, cognitive mechanisms) in their writing by calculating how many words from each LIWC dictionary appear in each document. (Pennebaker, Booth, & Francis, 2007).

Utility value assignments are designed to encourage students to write differently, in terms of both style and content, than when they are simply summarizing the course material (as in the control writing assignments). For example, UV assignments are designed to promote personal writing, which is expected to contain more references to the self and/or other people (i.e., social language) and use more “everyday” language than typical scientific writing. In addition, the process of making connections between course material and their own lives is expected to deepen students’ cognitive processing and increase engagement with the material, which should also be reflected in the content of students’ essays.

We selected nine LIWC dictionaries that might capture the style and content of UV writing. We hypothesized that the UV assignment would prompt the use of more personal pronouns and shorter words (i.e., simpler words and less technical vocabulary), as indexed by the LIWC personal pronouns (e.g., I, us, your) and longer words (> 6 letters) dictionaries. We used the social processes dictionary (e.g. advice, discuss, encourage), and its three subcategories – family words (e.g. mom, daughter, brother), friend words (e.g. friend, buddy, neighbor), and human words (e.g. adult, baby, boy)—to explore the social content of students’ essays. These dictionaries assess relationships and communication with others (Tauscik & Pennebaker, 2010). Finally, to explore content related to cognitive involvement, we chose the cognitive mechanism dictionary (e.g. cause, conclude, explain) and two subcategories—insight words (e.g. consider, idea, understand) and causal words (e.g. because, effect, hence). The insight dictionary assesses active learning, encoding, and understanding, and the causal dictionary connotes attempts to explain causes and effects (Klein & Boals, 2001; Pennebaker & King, 1999; Pennebaker & Stone, 2003).

We examined the number of words used from each of the nine LIWC categories as a function of the UV intervention, URM status, and FG status. In addition to the 11 terms of our basic model, our model for these exploratory text analyses also controlled for gender and the UV x gender interaction, because previous research using LIWC has found gender differences in word usage (e.g., Newman, Groom, Handelman, & Pennebaker, 2008). For these analyses we were interested in the main effect of the UV intervention, to determine if students wrote differently in UV essays, compared to control essays. We were also interested in the UV x URM x FG interaction, to explore whether any treatments effects differed for FG-URM students. The regression results are shown in Table 5.

Table 5.

Exploratory Text Analysis

	Personal Pronouns			Six-Letter Words			Words per Sentence			Social Words			Family Words
	β	t	p	β	t	p	β	t	p	β	t	p	β	t	p
UV Intervention	0.82	34.12	.000	−0.16	4.02	.000	0.11	2.58	.010	0.53	14.40	.000	0.37	9.43	.000
URM	0.02	0.95	.342	−0.02	0.64	.520	−0.03	0.92	.361	0.05	1.56	.118	0.02	0.77	.443
FG	0.00	0.13	.900	0.02	0.36	.718	−0.08	1.93	.053	0.08	2.09	.037	0.05	1.27	.205
URM x FG	0.00	0.13	.895	0.02	0.49	.627	−0.06	1.46	.145	0.05	1.41	.158	−0.01	0.19	.852
UV x URM	−0.01	0.23	.815	0.07	1.80	.073	0.09	2.02	.043	0.04	1.02	.307	0.01	0.23	.817
UV x FG	0.02	0.96	.340	0.02	0.37	.712	−0.02	0.38	.703	0.07	1.94	.053	0.07	1.67	.095
UV x URM x FG	0.02	0.95	.343	0.07	1.69	.092	0.00	0.04	.972	0.11	3.07	.002	0.08	2.02	.043
Confidence	0.02	0.82	.413	0.01	0.33	.740	−0.02	0.45	.653	−0.02	0.73	.464	−0.02	0.63	.530
Prior GPA	−0.03	1.66	.098	0.25	8.08	.000	−0.01	0.15	.879	0.03	0.89	.375	0.06	1.84	.066
UV x Confidence	−0.01	0.51	.610	−0.03	0.83	.410	0.01	0.32	.747	0.01	0.50	.619	−0.03	1.00	.317
UV x Prior GPA	0.04	2.25	.025	−0.04	1.29	.198	0.04	1.21	.225	0.01	0.41	.682	0.02	0.72	.472
Gender	−0.01	0.49	.624	0.03	0.93	.354	−0.10	2.95	.003	−0.04	1.38	.169	0.09	2.84	.005

	Friend Words			Human Words			Cognitive Mechanism Words			Insight Words			Causal Words
	β	t	p	β	t	p	β	t	p	β	t	p	β	t	p
UV Intervention	0.11	2.61	.009	0.25	5.98	.000	0.21	5.03	.000	0.33	8.18	.000	0.01	0.22	.825
URM	0.04	1.13	.258	0.02	0.66	.512	−0.03	0.93	.352	0.04	1.36	.174	0.02	0.51	.611
FG	−0.09	2.15	.032	−0.03	0.65	.514	0.05	1.11	.267	0.03	0.60	.546	0.04	0.96	.337
URM x FG	−0.04	0.88	.380	−0.02	0.56	.576	0.05	1.13	.259	0.03	0.76	.450	0.03	0.69	.493
UV x URM	−0.02	−0.35	.727	0.00	0.10	.923	0.02	0.35	.724	0.03	0.68	.495	0.02	0.40	.688
UV x FG	0.03	0.68	.497	0.02	0.59	.558	0.05	1.18	.237	0.07	1.62	.105	0.00	0.06	.953
UV x URM x FG	0.02	0.41	.680	0.02	0.52	.607	0.10	2.33	.020	0.11	2.83	.005	0.05	1.10	.271
Confidence	0.04	1.12	.265	−0.04	1.15	.251	0.04	1.35	.178	0.03	0.83	.408	0.05	1.52	.129
Prior GPA	0.00	0.05	.960	0.00	0.01	.996	0.11	3.31	.001	0.03	0.84	.400	0.13	3.84	.000
UV x Confidence	0.00	0.09	.925	−0.06	1.89	.060	0.02	0.60	.552	−0.04	1.17	.242	−0.05	1.54	.125
UV x Prior GPA	0.04	1.30	.193	0.02	0.55	.583	0.01	0.25	.802	0.04	1.21	.226	−0.04	1.14	.255
Gender	0.07	2.27	.024	0.04	1.24	.216	−0.05	1.59	.112	−0.03	0.96	.339	−0.04	1.14	.256

Note. df = 1027. UV= Utility Value intervention condition (UV =+ 1, Control = −1), URM = Underrepresented Minority (URM = +1, Majority = −1), FG = First-generation (FG =+ 1, Continuing-generation = −1). Confidence = Confidence about performance. Gender (Female= +1, Male = −1).

As expected, students in the UV condition used more personal pronouns, β = 0.82, p < .001, and fewer long words (> 6 letters), β = −0.16, p < .001. In addition, UV essays contained more social words, β = 0.53, p < .001, family words, β = 0.37, p < .001, friend words, β = 0.11, p = .009, and human words, β = 0.25, p < .001, than control essays. There were also more cognitive mechanism words, β = .21, p < .001, and insight words, β = .33, p < .001, in UV essays, suggesting that students’ writing in the UV condition was more characteristic of active cognitive engagement in the course content (Pennebaker & Francis, 1996; Pennebaker & King 1999), but there were no significant effects on causal words. Considered together, these results indicate that the utility value essays contained more personal writing and social themes, as well as greater evidence of cognitive engagement than control essays.

Interestingly, significant UV x URM x FG interactions on social words, β = .11, p = .002, family words, β = .08, p = .043, cognitive mechanism words, β = .10, p = .02, and insight words, β = .11, p = .005 (see Figures 5 and 6) indicated that FG-URM students, who wrote longer UV essays and benefitted most from the UV intervention, wrote essays that contained more social words, as well as more words indicative of cognitive involvement with the course material. Consistent with our finding that FG-URM students are especially oriented towards helping their families and communities, their UV essays contained more family words (but not more friend or human words, suggesting a relative emphasis on family).⁷

Figure 5.

Essay content reflecting social processes, as indicated by Social Words (Panel A) and Family Words (Panel B) as a function of treatment condition (UV or control), URM status (Majority or URM), and generational status (CG or FG). Error bars are 95% confidence intervals.

Figure 6.

Essay content reflecting cognitive involvement in course material, characterized by Cognitive Words (Panel A) and Insight Words (Panel B), as a function of treatment condition (UV or control), URM status (Majority or URM), and generational status (CG or FG). Error bars are 95% confidence intervals.

Moderation Analyses

Another way to gain insight into the dynamics of the UV intervention is to examine moderation by individual-difference variables. Our primary model revealed that the UV intervention was effective for students with low prior performance, as has been shown in previous research (Hulleman et al., 2010). Given that the intervention had a positive effect for FG-URM students controlling for this interaction with prior GPA, our results suggest that the efficacy of the UV intervention for FG-URM students is not completely explained by the efficacy of the UV intervention for students with low prior GPA.

To explore whether UV effects might be moderated by the background variable of helping motives, we tested whether the UV intervention was effective, more generally, for students who endorsed helping motives as a reason for attending college, regardless of prior GPA, ethnicity or generational status. Given that there was so little variance in helping motives among FG-URM students (80% of them selected all three helping motives as reasons for attending college), however, we could not test models that included these students. We therefore excluded FG-URM students in this analysis in order to test whether the UV intervention improved performance for other students who endorsed helping motives. Accordingly, we tested a variation of the basic model: in this exploratory model, we did not include the terms testing the FG x URM and UV x FG x URM interactions (because there were no FG-URM students in this analysis) and we controlled for the effect of gender and the UV x gender interaction (because gender was a significant predictor of helping motives in the baseline analyses). This resulted in a model with 13 terms: the main effects of the UV intervention, helping motives, URM status, and FG status, and 3 two-way interactions (UV intervention x URM status, UV intervention x FG status, and UV intervention x helping motives), as well as 3 covariates (confidence about performance, prior GPA, and gender), and their two-way interactions with the UV intervention.⁸

Of interest in this model was whether helping motives predicted performance or moderated the effects of the UV intervention. We found a negative effect of helping motives, β = −0.09, p = .001, indicating that students who selected more helping motives as reasons for attending college received lower grades in the course. In this model, the main effect of the UV intervention was not significant, β = −0.01, p = .813 (but recall that FG-URM students are omitted from the analysis), but we found a positive interaction between the UV intervention and helping motives, β = 0.05, p = .047, indicating that the UV intervention was most effective for students who endorsed more helping motives. Notably, the UV x Prior GPA interaction was somewhat reduced, but still nearly significant, β = −0.04, p = .094, indicating that the helping moderation observed in this model was independent of the prior GPA moderation documented earlier. This result supports our hypothesis that the UV intervention can be particularly effective for students who are motivated by a desire to help others.

Failure to Replicate Previous Values Affirmation Finding

In a previous study in the same context, we had tested the VA intervention alone (i.e., without implementing the UV intervention) in a single semester, and found that it improved performance for FG students, relative to CG students (Authors, 2014). This effect did not differ according to URM status, and the VA intervention did not affect performance for URM students. In the present study, in which the UV and VA interventions were both implemented across four semesters, the positive effect of VA for FG students was not replicated. Table 6 presents the model testing the UV intervention crossed with the VA intervention and shows that there were no significant effects of VA, and that all the UV effects reported previously remained significant when VA terms were included in the model. Here we consider four possible reasons for this non-replication: (1) the earlier finding could have been a false positive; (2) continued administration of the VA exercise over many semesters may have reduced its effectiveness; (3) the size of achievement gaps may have been different in the earlier study than in the current one; (4) students had more writing assignments in the current study than in the earlier study, and the UV intervention, which involved more writing, could have outweighed the effects of the VA intervention.

Table 6.

Effects of the VA Intervention, the UV Intervention, URM Status, and FG Status on Biology Course Grade

	HLM			Regression
	γ	F (df)	p	β	t (df)	p
UV Intervention	0.06	4.52 (1, 1017)	.034	0.07	2.21 (1018)	.027
URM	−0.14	24.81 (1, 1016)	.000	−0.13	4.98 (1018)	.000
FG	−0.03	0.89 (1, 1015)	.346	−0.03	0.89 (1018)	.375
URM x FG	0.00	0.01 (1, 1015)	.943	0.00	0.02 (1018)	.983
UV x URM	0.04	2.56 (1, 1013)	.110	0.06	1.63 (1018)	.103
UV x FG	0.01	0.13 (1, 1016)	.715	0.01	0.28 (1018)	.778
UV x URM x FG	0.07	6.52 (1, 1013)	.011	0.08	2.49 (1018)	.013
Confidence about Performance	0.12	36.86 (1, 1015)	.000	0.16	6.16 (1018)	.000
Prior GPA	0.44	452.68 (1, 1015)	.000	0.55	21.25 (1018)	.000
UV x Confidence	0.02	1.17 (1, 1014)	.280	0.03	1.15 (1018)	.252
UV x Prior GPA	−0.04	4.03 (1, 1017)	.045	−0.05	1.99 (1018)	.047
VA Intervention	−0.01	0.07 (1, 1013)	.788	−0.01	0.25 (1018)	.801
VA x URM	−0.02	0.36 (1, 1015)	.551	−0.02	0.61 (1018)	.539
VA x FG	−0.05	2.93 (1, 1013)	.087	−0.06	1.89 (1018)	.072
VA x URM x FG	−0.04	2.37 (1, 1015)	.124	−0.05	1.58 (1018)	.115
UV x VA	−0.00	0.00 (1, 1014)	.949	−0.00	0.04 (1018)	.969
UV x VA x URM	0.02	0.44 (1, 1014)	.506	0.02	0.68 (1018)	.498
UV x VA x FG	0.04	2.52 (1, 1014)	.113	0.06	1.67 (1018)	.096
UV x VA x URM x FG	0.01	0.06 (1, 1016)	.808	0.01	0.29 (1018)	.772
VA x Confidence	0.01	0.31 (1, 1014)	.577	0.01	0.54 (1018)	.592
VA x Prior GPA	−0.00	0.00 (1, 1016)	.950	−0.00	0.12 (1018)	.907

Note. UV = Utility Value intervention condition (UV = +1, Control = −1), URM = Underrepresented Minority (URM = +1, Majority = −1), FG = First-generation (FG = +1, Continuing-generation = −1). VA = Values Affirmation (VA = +1, Control = −1).

Although it is possible that the earlier finding was a false positive, the results were consistent with previous work in college courses (Miyake et al., 2010), as well as a conceptual replication of several earlier studies in different contexts (Cohen et al., 2006; Cohen & Sherman, 2014). Indeed, the VA intervention has proven effective at addressing achievement gaps in various educational contexts. However, there have been some previous failures to replicate these effects (e.g., Dee, 2015; Kost-Smith et al., 2015), and it is important to explore other factors that might account for non-replication.

Although the VA intervention was administered with methods identical to those employed in this same context by Authors (2014), it is possible that the continued administration of VA (and UV) interventions over a three-year period in the same course led to some communication between students or teaching assistants that weakened the impact of the interventions in this study. This issue may be more critical in the case of “stealth” interventions such as the VA intervention, however, since communication about the intervention can undermine treatment efficacy (Cohen et al., 2012; Cohen & Sherman, 2014). Curricular interventions such as the UV intervention may be more robust to such effects. We monitored the administration of both interventions carefully, and detected no problems, but in a large course with multiple sections of large lectures, many instructional staff, and several discussion and laboratory sections, there may have been undetected factors that detracted from implementation efficacy (Cohen et al., 2012).

Another possibility is that achievement gaps differed between the samples. We examined achievement gaps in course grades in the control condition of each study, to examine performance in the biology class for students who received neither intervention (25% of students in the current study, n = 226) or did not receive a VA intervention (50% of students in the Authors study, n = 396). We conducted 2x2 regression analyses of the effects of FG status, URM status, and their interaction, on course grade, controlling for gender and confidence about performance, in the control group of each sample. Table 7 shows course performance for all four groups of students in the control conditions of each study. In the Authors (2014) sample, we found a significant effect for FG status, β = −0.30, p < .000, and a significant effect for URM status, β = −0.14, p = .006. In the current sample, we found that the FG effect was not significant, β = −0.07, p = .417, but that the URM effect was, β = −0.19, p = .005. These analyses indicate that the performance difference between Majority and URM students in the control condition (i.e., the racial achievement gap in this course) was comparable in the two studies (mean grade-point difference of .38 and .36, respectively), but that the difference in performance between FG and CG students (i.e., the social-class achievement gap in this course) in the control condition was much larger in the Authors (2014) study than in the current study (mean grade-point difference of .48 and .08, respectively). This occurred mainly because FG controls in the current study performed considerably better than those in the previous one; in addition, CG students in the current study performed slightly less well than those in the previous study. It is likely that VA is more effective when achievement gaps are larger, and previous research substantiates this view (Hanselman, Bruch, Gamoran, & Borman, 2014).

Table 7.

Course Performance in the Control Group Compared with Authors (2014) Sample

Biology Course Grade (Control Group Only)
	Current Sample		Authors (2014)
	Mean (SD)	N	Mean (SD)	N
All CGs	2.77 (.79)	121	2.86 (.69)	319
All FGs	2.69 (.87)	105	2.38 (.85)	77
Mean Social Class Gap (CG – FG)	0.08		0.48***
All Majority	2.80 (.82)	183	2.80 (.74)	363
All URMs	2.44 (.81)	43	2.42 (.79)	33
Mean Race Gap (Majority – URM)	.36**		.38**
CG-Majority	2.85 (.76)	93	2.87 (.69)	298
CG-URM	2.50 (.85)	28	2.67 (.66)	21
FG-Majority	2.75 (.88)	90	2.45 (.84)	65
FG-URM	2.33 (.75)	15	2.00 (.85)	12

Note. CG = Continuing-generation, FG = First-generation, URM = Underrepresented Minority.

^**p < .01.

^***p < .001.

It is also possible that differences in the educational context account for differences in treatment efficacy. Although the structure and content of the biology course was constant across the two studies, there may have been differences in the way the course was taught by different instructors, and this may have influenced how the VA intervention worked. In fact, one important feature of the course differed by design: in the present sample, three writing assignments (the UV/control essays) were added to the curriculum. Thus students were required to do more writing in the biology course in the present study compared to the same course in the Authors (2014) and this may have changed students’ experiences of the course. It is possible that the VA intervention was less effective because the course was more writing-intensive or seemed to place a greater emphasis on personal writing. Given that the UV and the VA interventions both involve personal writing, it also seems possible that the extensive writing in the three UV essay assignments outweighed the brief VA writing exercises conducted in laboratory sessions in the present study, thereby dampening the effect of the VA intervention. This suggests that it may not work to combine different types of writing interventions in a single semester (Walton, Logel, Peach, Spencer, & Zanna, 2014).

Our results, considered together with other non-replications of VA effects (Dee, 2015; Kost-Smith et al., 2012), suggest that although values affirmation can have powerful positive effects, it is sensitive to contextual and sample differences in ways that we do not yet fully understand (Walton, 2014; Yeager & Walton, 2011). More research is needed to identify factors that moderate the effectiveness of the VA intervention in college contexts, and future research should also examine how VA interventions work in conjunction with other interventions.

Discussion

There have been many attempts to address achievement gaps in high school and college classes, but researchers have typically focused on just the racial achievement gap or just the social class gap (Cohen et al. 2006; Harackiewicz, Canning et al., 2014, Sherman et al., 2013, Stephens et al., 2014; Walton & Cohen, 2011). To our knowledge, this is the first intervention study designed to close achievement gaps that has also been designed to disentangle the effects of race from the effects of social class. Our findings illustrate the importance of analyzing the separate and interactive effects of URM and FG status when evaluating interventions intended to close achievement gaps. Focusing on one gap to the exclusion of the other overlooks the challenges and motivational patterns unique to each group. We found that our UV intervention was particularly helpful for a unique group of underrepresented students–FG-URM students—who came into this course with the lowest grades, highest levels of high-school poverty, and weakest biology background. In other words, the UV intervention was most effective for the students who began the course with the most disadvantages.

On average, the UV intervention promoted performance for all students. Thinking about how course topics relate to their own life, or the lives of others, made biology more relevant to students, and helped them stay engaged in their coursework. Students wrote longer essays when thinking about utility value, and this may reflect greater engagement with course topics (Elliot & Klobucar, 2013). The assignment may also have promoted deeper cognitive processing of the material, facilitating learning (Harackiewicz, Tibbetts et al., 2014), and this is supported by the text analysis findings. Students in UV conditions used more cognitive mechanism words, especially insight words, which Pennebaker and King (1999) argue reflect active thinking and learning. Considered together, these results contribute to a growing body of work showing that curricular interventions based on psychological principles can promote active engagement and performance in college science classes (Deslauiers, Schelew, & Wieman, 2011; Haak, HilleRisLambers, Pitre, & Freeman, 2011).

For interventions to close achievement gaps, however, the effects must be strongest for disadvantaged groups, and we found the largest UV intervention effect for FG-URM students in this study. This result highlights the potential of the UV intervention to address important achievement gaps in college classes. Moreover, content analyses revealed that writing about utility value was particularly effective for FG-URM students. Although all students articulated more utility value in UV conditions, the process of articulating utility value was particularly effective for FG-URM students. They wrote longer essays in the UV condition, and mediation analyses revealed that this increased engagement with course material contributed to their increased performance in the course. Moreover, LIWC text analyses revealed that their essays contained more words indicative of active thinking, which may help account for their improved performance in these conditions (Klein & Boals, 2001).

First-generation Underrepresented Minority Students

What accounts for the greater responsivity of FG-URM students to the UV intervention? Our analyses provide some insight into the mechanisms at work for these students. One possibility is that the intervention was most effective for FG-URM students because UV interventions are most effective for students with lower prior grades (Hulleman et al., 2010), and FG-URM students had the lowest prior GPA among the categories of students. Notably, the positive effect of the UV intervention for FG-URM students remained significant after accounting for the effect of prior GPA and for the positive effects of the intervention for students with lower prior grades. This suggests that the UV intervention may work to promote engagement and close achievement gaps for multiple reasons that go beyond the issue of prior performance.

For example, the UV intervention may also work by helping students connect course material to important personal goals. Our moderation analysis, in which we tested whether individual differences in helping motives moderated the effects of the UV intervention, revealed that the UV intervention was more effective in improving performance for students who endorsed helping motives. This suggests that the UV intervention may be particularly impactful for FG-URM students, who endorsed more helping motives for attending college than any other group of students.

Consistent with previous research, FG-URM students in this study were highly motivated to succeed in the course, and they were particularly motivated by communal goals, such as helping others, contributing to society, or giving back to their families and communities (Smith et al., 2014; Stephens et al. 2012). From the perspective of both Stephens’ cultural mismatch theory and Diekman’s goal congruity theory, the UV intervention may have been especially powerful for FG-URM students because it gave them the opportunity to explore connections between the course content and their communal goals, thereby alleviating a sense of goal incongruity or cultural mismatch. Indeed, the LIWC text analyses revealed that FG-URM students in the UV condition were most likely to include content related to social processes, and family words in particular, in their essays. Discovering and appreciating the utility value of curricular content for their communal values and goals may, in turn, have increased their engagement with curricular content (as evidenced by their longer essays and more frequent use of cognitive engagement words in UV conditions), and inspired them to work harder in the course. Rather than addressing a deficit, the UV intervention may work for these students by relating the curriculum to students’ strengths–their positive motivations for studying biology.

In contrast, the UV intervention did not help FG-Majority students, and our baseline analyses offer some insight into why our intervention was more effective for some groups than others. Our findings suggest that FG students (both majority and URM) tended to feel more uncertainty about belonging, and we would not expect a UV intervention, which is specific to curricular content, to impact a student’s more general sense of belonging in college. For these students, targeting their uncertainty about belonging might prove more effective (Ostrove & Long, 2007; Walton & Cohen, 2011). Interventions that target social belonging processes, such as Walton and Cohen’s (2011) belonging intervention, focus on students’ experiences or sense of fit in the academic environment (Walton, 2014), and might be more effective with FG students. Indeed, Stephens et al. (2014) found that a difference-education intervention in which freshmen FG students heard stories about positive FG student experiences was effective in helping FG students adjust to college and perform better in in their classes.

Different interventions address different psychological processes, and it is important to consider the motivational patterns of the students targeted by interventions. Given that different interventions may not work well together (Walton et al., 2014), it is especially important for intervention researchers to understand the characteristics of the group they hope to help, in order to identify the most effective interventions to implement with that particular group. Only by considering the independent and interactive effects of race and social class were we able to identify the unique background and motivational profile of FG-URM students and document the full potential of the UV intervention to promote performance for the underrepresented students most at risk in this course.

The Intersection of Race and Social Class

We tested both the additive and interactive effects of race and social class, as well as the three-way interaction of race, social class, and the UV intervention, on academic and motivational variables and found evidence for both the main effects and the interactional approach to intersectionality. We measured eleven baseline variables, including prior GPA, high-school poverty rate, confidence, baseline interest, and helping motives, and found a mixed pattern of results. On some measures (e.g., baseline interest, confidence) we found no group differences; on others, we found a single main effect (e.g., URM students reported lower levels of biology background than majority students and FG students reported higher levels of belonging uncertainty than CG students). On other measures, we found two main effects but not an interaction (e.g., prior GPA), supporting the additive model of intersectionality. On the measures of high-school poverty and helping motives, however, we found two main effects and a significant interaction, supporting the interactive model. The interaction effect on high-school poverty, for example, indicates that FG-URM students came from more impoverished schools and neighborhoods than either their race or generational status would predict. The interaction effect on helping motives suggests that FG-URM students are particularly motivated to help others and give back to their communities. Only by measuring a number of baseline variables were we able to gain a nuanced understanding of differences between student groups and intersecting categories. These findings indicate the power of this statistical approach to identify important intersectionalities, and may contribute to development of more effective interventions in the future.

With respect to the primary outcome of course grade, we were able to explore additive and interactive models of intersectionality in moderating treatment effects. On course performance, we found a main effect of race, but not FG status, nor a URM x FG interaction. Perhaps most interesting, the three-way interaction with the intervention was significant (UV x URM x FG, Table 4, Figure 3), demonstrating that intersectionality can apply to the effects of an intervention. That is, the intervention was differentially effective as a function of both the race and social class of the students and was particularly beneficial for FG-URM students.

This same three-way interaction was also found for essay length; the intervention had a strong effect for FG-URM students, who became particularly engaged in the utility value writing assignments. In the variables derived from LIWC analyses, this three-way interaction was found for social-process words, family words, cognitive-mechanism words, and insight words, in all cases showing the pattern that the intervention was especially potent for FG-URM students. Overall, then, the findings point to the importance of considering the intersection of social categories in intervention research. Whereas prior gap-closing intervention studies have focused on single social categories, our results suggest that this approach may neglect important within-group differences that only an intersectional approach can fully address. By considering students’ intersecting identities, intervention researchers can develop a more nuanced understanding of both the populations they seek to help and the effectiveness of their interventions.

Implications for Expectancy-Value Theory

In this study we attempted to identify variables that may account for differential effects of the UV intervention, and our findings suggest that the relationships are complex. We found that the UV intervention was moderated by social class, race, prior GPA and helping motives. The UV intervention implemented in an introductory biology course was effective for FG-URM students (a doubly disadvantaged category of students; Jack, 2014), for students with low prior GPAs (an academic background category), and for students with high helping motives (a motivational category). These findings extend the range of groups who can benefit from a UV intervention, and contribute to our understanding of how UV interventions work, and to Expectancy-Value Theory more generally.

By testing both prior performance and baseline performance expectations (confidence about performance) as moderators of the UV intervention effects in the same model, we were able to test which variable was more important in this context. We found that prior performance was the more critical moderator of UV effects, whereas performance expectancies did not moderate the effects of the UV intervention. This helps us understand the Expectancy side of the Expectancy-Value model (Eccles, 2009). Prior research has employed one moderator or the other; in laboratory studies with new learning activities, for example, researchers typically test self-reported expectancies as the moderator, because there is no history of prior performance to consider (Canning & Harackiewicz, 2015; Shechter, Durik, Miyamoto, & Harackiewicz, 2001). In classroom studies, researchers have used either self-reported performance expectations (Hulleman & Harackiewicz, 2009) or prior exam performance (Hulleman et al., 2010) to assess expectancies. Expectancies are of course shaped by prior performance experiences (Eccles, 2009, Pajeres, 1996), but one measure reflects performance history and the other is more forward-looking. In an introductory college course, however, students may not be able to gauge how well they will do in the course (in fact, we found no group differences in confidence at baseline, despite eventual group differences in course grades), and thus prior performance may be the more relevant moderator in this context. Therefore to determine which expectancy variables are important moderators of UV intervention effects, it is important to consider both self-reported expectancies and prior performance in any given context.

Furthermore, our mediation and content analyses provide some insight into UV mechanisms, and here we address the Value side of the Expectancy-Value model. Our results help us understand how writing about utility value may help to promote performance–by increasing engagement in the writing assignments, by providing opportunities to write differently (i.e., including more personal and social content), and by encouraging active thinking about course topics. Although were able to show that engagement in the writing assignment (measured in terms of essay length) mediated course performance, most of our other analyses were more exploratory, and highlight a need for future research. One important question concerns the connections between these mechanisms. For example, in writing UV essays, do students first find personal examples and then become more engaged with course content, or does greater engagement help students make more personal connections? Are these separate processes, and how might we measure them more objectively? It is difficult to conduct fine-grained process analyses in a large-scale classroom intervention study, and future research may require controlled laboratory studies to advance our understanding of the motivational mechanisms that underlie the effectiveness of the intervention. Overall, this research contributes to a better understanding of how students think about value in science classes, and illuminates the Value side of Expectancy-Value Theory, which is understudied compared with the Expectancy side. The data show the power of utility value in enhancing students’ academic achievement, and highlight new areas for future research.

Strengths and Limitations

Although our sample was large enough to disentangle the effects of racial and generational status, sample sizes of different racial/ethnic groups were not large enough to explore differences between racial groups. In particular, we considered African American, Latino and Native American participants together in a single URM category, and were not able to separate these groups in analyses because of limited sample sizes. Our findings point to the importance of considering intersectionality in social science research, and it will be particularly important for future research to address intersectionality on a more specific level (e.g., the intersection of African American and FG identities). Different interventions may be more or less effective for different racial and cultural groups, and these are important issues to pursue in future research.

Similarly, the use of first-generation college student status as a measure of social class captures an important aspect of social class in the higher education context, but a dichotomous measure does not capture the full range of parental educational levels that may contribute to students’ experiences in college. Furthermore, FG status does not capture all aspects of social class that may contribute to achievement gaps. For example, limited financial resources and the need to work part- or even full-time while attending college are major barriers for low SES students (Pascarella, Pierson, Wolniak, & Terenzini, 2004) and we would not expect a psychological intervention to address these financial barriers. Indeed, by including high-school free/reduced lunch (as a proxy for poverty at the school and neighborhood levels) as a predictor, we found that high-school poverty had an independent negative effect on performance, over and above the effects of FG and URM status. Nonetheless, the effects of FG and URM status remained, as well as the positive effect of the UV intervention for FG-URM students. These issues will be important to address in future research with more direct measures of income and poverty level, as well as ways to capture high school quality directly.

Finally, although we inferred that FG-URM students in our sample perceived an incongruity or mismatch between their motivations and values and the norms of science, we did not collect measures of students’ perceptions of incongruity or mismatch. We hypothesized that the UV intervention provided FG-URM students the opportunity to make course content congruent with their own values and goals, and indeed, the treatment effects on social process and family words for FG-URM students suggests that the content of their essays was congruent with communal goals. However, we were not able to measure whether the UV intervention changed perceptions of incongruity, and thus our analysis remains speculative. If the UV intervention is particularly powerful because it helps FG-URM students resolve a mismatch between their communal goals and their perceptions of scientific fields, it will be important to measure these perceptions as well as the psychological experience of mismatch in future research.

Conclusions

Racial and social-class achievement gaps are a major issue in higher education. When these gaps occur in introductory science courses, they can lead URM and FG students to abandon STEM majors and careers. Our utility-value intervention successfully reduced the achievement gap for URM students by 40% and for FG-URM students by 61%. Moreover, the intervention was designed for implementation within the curriculum in large-lecture classes, making it practical for widespread implementation at low cost. This simple curricular change–asking students to reflect on personal connections to course material in course assignments–could help students remain engaged in introductory courses and ultimately widen the STEM pipeline.

Note. UV = Utility Value (UV = +1, Control = −1), URM = Underrepresented Minority Status (URM = +1, Majority = −1), FG = First-Generation Status (FG = +1, Continuing-generation = −1).