Abstract
One way to encourage performance and persistence in STEM fields is to have students write about the utility value (UV) or personal relevance of course topics to their life. This intervention has been shown to increase engagement and performance in introductory courses. However, questions remain about the longevity of the effects and how best to implement the intervention in terms of dosage and timing. We tested a UV intervention in the first semester of a two-semester introductory biology sequence. For each of three units across the semester, students (N = 577) were randomly assigned to receive either a UV writing assignment, in which they explained why course material was useful to them personally, or a control assignment, in which they summarized course material. This fully crossed design tested the effect of UV dosage level (0, 1, 2, or 3 UV assignments) as well as the effect of timing (e.g., UV first, control first). We found that students exposed to any dosage of UV earned higher grades in the course, were more likely to enroll in the second course of the biology sequence, and were less likely to abandon their STEM major than students who did not receive any UV assignments. In terms of timing, students with a history of poor performance benefitted from writing a UV essay in the beginning of the semester, whereas higher-performing students benefitted from a UV essay at the end of the semester. Recommendations for practice are discussed.
Keywords: academic motivation, task values, utility value, interventions, retention
High attrition rates in science, technology, engineering, and mathematics (STEM) fields have long been a global concern (OECD, 2015). A recent report indicated that if the United States is to remain competitive in the global economy, it will need 1 million more STEM professionals over the next decade than it is currently projected to produce (PCAST, 2012). Similar shortages in STEM professionals exist in France, Germany, and the United Kingdom (OECD, 2013). To increase the global STEM workforce, it will be imperative to retain more college students in STEM majors; however, almost half (48%) of students who initially major in a STEM field switch into a non-STEM field (Bettinger, 2010; Chen & Soldner, 2013; Goulden, Frasch, & Mason, 2009; Higher Education Research Institute, 2010). STEM students typically decide whether they will stick with their major within the first two years of college and especially after taking introductory STEM courses (Seymour & Hewitt, 1997; Strenta, Elliott, Adair, Matier, & Scott, 1994; Watkins & Mazur, 2013). A number of factors contribute to STEM attrition including uninspiring introductory courses and low grades in “weed-out” classes, indicating that a positive experience in a single introductory course can have an impact on student persistence in STEM majors (Gasiewski, Eagan, Garcia, Hurtado, & Chang, 2012; Ost, 2010; Seymour & Hewitt, 1997).
One way to increase performance and retention in introductory courses is to highlight the value or relevance of the course material. When students encounter subjects that do not seem important, they may become disengaged, lacking the motivation to do well. One type of value that has proven to be a powerful predictor of persistence and performance is utility value—the perception that a task is useful or important for present or future goals (Harackiewicz, Durik, Barron, Linnenbrink-Garcia, & Tauer, 2008; Hulleman, Durik, Schweigert, & Harackiewicz, 2008; Wigfield & Cambria, 2010). For instance, a student may find a lesson on genetics to be important for their personal goal of becoming a doctor or because it helps them understand a parent's disease. By increasing perceptions of utility value in introductory STEM courses, educators may have the opportunity to influence not only performance in the course, but also subsequent course enrollment because of the importance of these courses for future goals (e.g., majoring in biology, becoming a doctor). If educators can help students find the utility value (UV) of course material, they may be able to promote achievement and retention in introductory STEM courses that act as gateways to successive careers.
Utility-Value Interventions
Our approach is based in Eccles' expectancy-value theory (Eccles & Wigfield, 2002), which posits that expectancies for success and subjective task values together determine choice, persistence, and performance. Eccles identified four types of task values (Eccles, 2009; Eccles et al., 1983): intrinsic (the value of engaging in the task because it is interesting or enjoyable), utility (the perceived importance or usefulness of the task for accomplishing present or future goals), attainment (the perceived importance of the task for one's self concept or identity) and cost (the negative aspects of engaging in the task, such as lost time and opportunities). A large body of research has examined the role of success expectancies or perceived competence (e.g., “I'm really good at math and science”) in promoting interest and performance in STEM fields (Fredricks & Eccles, 2002; Gaspard, Dicke, Flunger, Schreier, et al., 2015; Jacobs, Lanza, Osgood, Eccles, & Wigfield, 2002; Watt, 2004). Less focus has been placed on subjective task values, which may be more amenable to intervention, and utility value in particular has proven responsive to external intervention (Harackiewicz, Tibbetts, Canning, & Hyde, 2014; Tibbetts, Harackiewicz, Priniski, & Canning, 2016).
Recent experimental research has focused on promoting perceptions of utility value with the intent of facilitating learning, interest, and persistence, based on substantial correlational research showing that perceptions of utility value predict academic achievement, effort and interest (Hulleman et al., 2008; Wigfield & Cambria, 2010). One relatively simple UV intervention instructs students to write about the relevance or utility value of course topics to their own life (Hulleman & Harackiewicz, 2009). The UV intervention as implemented in previous research (as well as the current study) targets both relevance and utility value. Although these constructs are theoretically distinguishable—relevance is defined as a personally meaningful connection to the individual, whereas utility value is a type of relevance focused on personal usefulness (see Priniski, Hecht, & Harackiewicz, in press)—in practice, similar interventions grounded in the same theoretical model have been called relevance interventions or utility-value interventions (e.g., Hulleman, Godes, Hendricks, & Harackiewicz, 2010; Hulleman & Harackiewicz, 2009; Gaspard, Dicke, Flunger, Brisson, et al., 2015). Students may write about relevance or personal usefulness (or both) and research to date does not distinguish between these processes. Regardless, a goal of these interventions is to help students perceive and articulate the utility value of the material they are learning. In line with this goal and with the theoretical underpinnings of the intervention, we call the intervention a utility-value intervention.
The utility-value writing intervention has been shown to work best for high school students who doubt their competence and for college students with a history of poor performance (Canning & Harackiewicz, 2015; Gaspard, Dicke, Flunger, Brisson, et al., 2015; Harackiewicz, Canning, Tibbetts, Priniski, & Hyde, 2016; Hulleman et al., 2008; Hulleman et al., 2010; Hulleman & Harackiewicz, 2009; Hulleman, Kosovich, Barron, & Daniel, 2016). For example, Hulleman and Harackiewicz (2009) implemented a UV intervention for high school students by having students write about the personal relevance of their science schoolwork. They found that students with less confidence in their science class reported higher interest and improved their grades in the UV condition, relative to a control condition, whereas confident students made no significant gains with the intervention. Hulleman et al. (2010) implemented the same intervention in a college introductory psychology class and found that students who had performed poorly on an early exam and who wrote about UV reported more interest in the course at the end of the semester when compared to those in the control group. More recently, a UV writing intervention promoted performance for underrepresented students in a college introductory biology course (Harackiewicz, Canning, et al., 2016). The UV intervention was successful in reducing the achievement gap for underrepresented minority students by 40% and for underrepresented minority students who were also first-generation college students by 61% (Harackiewicz, Canning, et al., 2016). Harackiewicz, Canning, et al. (2016) also found that the UV intervention was most effective for students who had the lowest prior GPAs, indicating that the intervention helped students with a history of poor performance, in addition to helping underrepresented minority students.
Across studies the UV intervention has been found to be most effective for students who are in most need of intervention. In laboratory studies and studies with younger students, an important moderator of intervention effects is success expectancies (confidence about performance), such that the intervention is most effective for students with low confidence; whereas in studies with college students, prior performance is a more critical moderator, such that the intervention is most effective for students with a history of poor performance. Indeed, Harackiewicz, Canning, et al. (2016) tested both performance (prior GPA) and success expectancies as moderators of a UV intervention implemented in a college biology course and found that the intervention effects were moderated by performance and not by success expectancies.
Retention Outcomes
UV writing interventions have been shown to be effective at increasing task engagement and academic performance within a single course or specific context; however, it is unknown whether intervention effects can extend over time. Little research has investigated the impact of UV interventions on retention, course-taking, and major persistence. Expectancy-value theory suggests that when students perceive utility value in their course, they are by definition finding value for a future situation or personal goal. Perceived utility value for one task or topic (e.g., a single introductory course) should facilitate persistence in achieving future goals. Therefore, UV writing interventions have the potential to impact retention outcomes, such as remaining in a major or persisting in a difficult sequence of introductory courses (Harackiewicz, Smith, & Priniski, 2016). However, existing studies of UV writing interventions have not investigated the possibility of long-term impacts beyond the course in which the intervention was implemented. To address this gap in the research, we investigate whether a UV writing intervention can influence outcomes such as course taking and major persistence.
Intervention Implementation: Dosage and Timing
UV writing interventions have already proven successful at increasing a number of positive outcomes; however, important questions remain about how best to implement the intervention. First, what is the optimal dosage for these writing assignments? The number of writing assignments has varied across studies. In high school science courses, the dosage ranged from 1-8 assignments over the course of a semester (Hulleman & Harackiewicz, 2009). In high school math courses, the primary intervention was followed by two short reinforcement assignments, 1 and 2 weeks after the intervention (Gaspard, Dicke, Flunger, Brisson, et al., 2015). In college classes, Hulleman et al. (2010) implemented a UV writing intervention twice during the second half of the semester (weeks 10 and 12 of a 15-week semester). Hulleman et al. (2016) also administered two doses of the writing intervention, but did so in the first half of the semester (weeks 4 and 8 of a 15-week semester). Additionally, UV writing assignments have been implemented three times during a semester, once for every major unit in the course (Harackiewicz, Canning, et al., 2016). It is unclear how many writing assignments are necessary for the intervention to be most effective.
Based on expectancy-value theory, we might expect that the more times a student is given the opportunity to discover the utility value of course material, the more likely they are to perceive value and thus exhibit greater motivation. In one study, Hulleman et al. (2016) found that the self-reported frequency with which students made connections between the course material and their lives was positively related to perceptions of utility value and continued interest in the course. In other words, students may benefit from multiple opportunities to make UV connections with the material, and frequent UV writing assignments could be a powerful tool to create those opportunities. Therefore, it is possible that the optimal dosage involves administering UV writing assignments repeatedly, for each unit or topic of the course. Considering that the UV intervention is a curricular assignment based in course content, students may need to be prompted to think about the utility for each unit or topic of the course. However, this has not been tested systematically, and it is possible that a smaller number of UV writing assignments could be just as effective. For example, a single UV assignment might be powerful because of its novelty in a science class. An alternative hypothesis is that too many UV writing assignment might be unnecessary (e.g., if a single assignment is sufficient to promote value) or even become overly burdensome to students and faculty, reaching a point of diminishing returns.
Practically speaking, it is important to investigate whether multiple UV writing assignments is the best practice, considering the time that instructors invest in grading these assignments and providing written feedback for students. In past implementations, the UV essay assignments were graded for accuracy of scientific content and students received feedback in the form of individualized written commentary (Harackiewicz, Canning, et al., 2016). If this intervention is to be scaled up and implemented broadly, it needs to be efficient and cost effective. The UV exercise is costly in terms of time commitment, both for students (who write the essays) and for instructors (who grade them); however, this cost is justified if students benefit from the intervention. It is important to know the minimum number of UV exercises required to produce maximum effectiveness.
Another implementation question relates to the timing of the writing assignments: Should educators incorporate UV assignments at the beginning or end of semester? It is not well understood when UV information is most effective. Research from the field of intervention science suggests that interventions should be administered at critical times in a student's development to elicit recursive processes, such as the beginning of the school year or the beginning of a difficult course (Cook, Purdie-Vaughns, Garcia, & Cohen, 2012; Harackiewicz & Priniski, in press; Yeager & Walton, 2011). Through recursive processes, a well-timed intervention initiates small psychological effects, such as perceptions of utility value, that then gain momentum over time, eventually leading to adaptive academic behaviors. In that regard, the UV intervention promotes deeper engagement with course material, and setting this process into motion early could change the way students approach subsequent material.
The timing of the UV intervention may be even more critical for students who doubt their competence or for those with a history of poor performance. Early implementation of the UV intervention may be critical for low-performing students to increase engagement and performance in the course, whereas implementing the UV intervention later in the semester might be critical for course-taking and major decisions. Implementing the intervention early in the semester may facilitate learning and provide a reason for students to engage with the course material. For instance, a single UV assignment, administered early in the semester, may establish a “habit of mind” so that students continue to make UV connections with the course material without needing to write more UV essays (Hulleman et al., 2016). On the other hand, students may need more time to absorb the course material before UV information can influence course-taking and major decisions, and therefore, assigning UV writing assignments later in the semester may be more effective for these outcomes. Thus, timing effects may depend on two things: students' prior achievement, with early intervention hypothesized to be more effective for students who struggle, and the outcome measure, with early intervention hypothesized to be more effective for performance in the course, and later intervention more important for decisions about the future. The existing literature contains examples of UV interventions implemented early in the semester (Hulleman et al., 2016), later in the semester (Hulleman et al., 2010), and throughout the semester (Harackiewicz, Canning, et al., 2016; Hulleman & Harackiewicz, 2009), but no study has directly compared these implementation strategies to test whether the timing of the intervention moderates its effectiveness. The design and large sample of the current study allowed us to test timing and dosage effects for UV interventions.
The Present Study
We tested a UV writing intervention in the first semester of a two-semester introductory biology course. We address two primary questions about the UV intervention: (a) Can the UV intervention increase performance as well as retention outcomes, such as course-taking and deciding to remain in a STEM field? and (b) What is the optimal timing and dosage for the UV writing intervention? Research to date is not conclusive regarding the number of UV essays to assign in a course, or whether this intervention is more effective earlier or later in a course. We varied the number (dosage level) and sequence (timing) of UV writing assignments, by varying whether the first, second, and third essay assignments were UV or control. This design was fully crossed, resulting in a 2 (First Essay: UV vs. Control) by 2 (Second Essay: UV vs. Control) by 2 (Third Essay: UV vs. Control) design. These eight experimental conditions allowed us to test the effect of UV dosage level (0, 1, 2, or 3 UV essays) as well as the effect of timing (by comparing groups who write UV assignments early in the semester with those who write them later).
To address our first research question, we tested the efficacy of the UV intervention averaged across dosage and timing. We hypothesized that the UV intervention would be effective at increasing performance as well as retention in the biology course sequence and persistence within a STEM major. We also hypothesized that increased grades in the course would mediate the effect of the intervention on retention and STEM major persistence. In other words, we expected that the UV intervention would lead students to continue in the biology course sequence and remain in a STEM major by first increasing their grades in the course. To address our second research question, we conducted three separate analyses to determine the optimal dosage and timing of the UV intervention. First, we tested the fully crossed 2 (First Essay: UV vs. Control) by 2 (Second Essay: UV vs. Control) by 2 (Third Essay: UV vs. Control) model. By testing the fully crossed model, we were able to examine effects of timing and dosage simultaneously. In terms of optimal dosage, we hypothesized that the UV intervention would be most effective at full dosage (three times throughout the semester or one UV assignment for each unit of the course). In terms of optimal timing of the intervention, however, our hypotheses were more exploratory, and we also examined the effects of timing in more fine-grained follow-up analyses. Specifically, we focused on a subset of the design where we could examine the effect of a single dose of UV writing, administered at different time points, and hypothesized that a single UV writing assignment would be more effective at the beginning of the semester compared to the end of the semester.
In all analyses we included one moderator (prior GPA), and two covariates: gender and initial interest in biology. Prior GPA was chosen as a moderator because prior research has shown that the UV intervention was more effective for students with a history of poor performance (Hulleman & Harackiewicz, 2009; Hulleman et al., 2010; Hulleman et al., 2016). We hypothesized that the UV intervention would be more effective at increasing grades for students with lower prior GPAs as in previous research. In terms of timing, we hypothesized that early implementation of the UV intervention and multiple dosage of the interaction would be most beneficial for low-performing students. In this biology course men typically outperform women, so it was important to control for this effect in the analyses. Additionally, it is important to control for baseline performance when examining treatment effects on course performance and baseline interest when examining treatment effects on course-taking and major decisions.
Methods
We implemented the UV intervention in a double-blind, randomized experiment in three sections of an introductory biology course at a large Midwestern university. The course is the first of a two-semester sequence required for 34 undergraduate majors in the biological and medical sciences (e.g., zoology, biochemistry, neuroscience, nursing). This important gateway course covers three units—cellular biology, genetics, and evolutionary biology—and consists of lectures, laboratory sections, and discussion sections, for a total of 5 credits. The course is team-taught, with a different professor for each topic and teaching assistants leading the laboratory and discussion sections. Of the 598 students eligible to participate in the study, 577 students (247 male, 330 female) completed the course and gave consent for access to their academic records (3 students did not consent; 18 dropped the course).1 Participants were 86% White and 14% Asian/Asian American. Preliminary analyses revealed that there were no significant differences between White and Asian/Asian American students or any interactions with the UV intervention on any outcome measure. Therefore, White and Asian/Asian American students were combined for analyses. The researchers' Institutional Review Board approved this study.
Utility-Value Intervention
Students were blocked on gender and randomly assigned to one of 8 conditions testing dosage and timing of the UV intervention. For each of the 3 five-week units of the course, students were assigned a UV or control writing assignment, in a fully crossed 2 (First Essay: UV vs. Control) by 2 (Second Essay: UV vs. Control) by 2 (Third Essay: UV vs. Control) design. Both the UV and control assignments instructed students to:
Select a concept or issue that was covered in lecture and formulate a question. Select the relevant information from class notes and the textbook, and write a 1-2 page essay.
The UV assignment then instructed students to answer their question using course material and were given the choice to discuss the relevance of the concept or issue to either their own or others' lives:
Write a 1-2 page essay addressing this question and discuss the relevance of the concept or issue to your own life. Be sure to include some concrete information that was covered in this unit, explaining why this specific information is relevant to your life or useful for you. Be sure to explain how the information applies to you personally and give examples.
OR
Write a 1-2 page letter to a family member or close friend, addressing this question and discuss the relevance of this specific concept or issue to this other person. Be sure to include some concrete information that was covered in this unit, explaining why the information is relevant to this person's life, or useful for this person. Be sure to explain how the information applies to this person and give examples.
Consistent with previous implementations of the UV intervention in college classes (Hulleman et al., 2010; Harackiewicz, Canning, et al., 2016), we gave students the choice in UV conditions to discuss how course content is relevant to either their own or others' lives, hypothesizing that students who have a difficult time connecting the material to their own life may be able to generate utility value for someone else's life more easily than their own. The control assignment instructed students to answer their question using course material:
Select the relevant information from class notes and the textbook, and write a 1-2 page response to your question. You should attempt to organize the material in a meaningful way, rather than simply listing the main facts or research findings. Remember to summarize the material in your own words.
The UV or control assignments were assigned three weeks before each unit exam. Students were given five days to complete the homework assignment and turn it in to a dropbox on the course management website. Biology graduate students graded the assignments for scientific content, writing quality, and whether students followed directions. To ensure that instructors and graduate teaching assistants remained blind to experimental condition, the assignments were emailed to students by course coordinators, and any graduate students who served as both teaching assistants and graders in the course were assigned to grade essays for a different section, so that no teaching assistant ever graded their own students' essays.
Grades and grader feedback were provided to the students a few days before the unit exam. Each essay was worth 0.6% of the students' final grade. 574 students completed the first assignment, 567 students completed the second assignment, and 555 students complete the third assignment, with 550 students (95.3%) completing all three assignments.
Baseline Measures
Questionnaire measures of attitudes about biology were collected in laboratory sections during the second week of the course. Questionnaire items were answered on a 7-point Likert-type scale (1 = “not at all true,” 7 = “very true” or 1 = “not at all,” 7 = “a lot”), unless otherwise noted. Scale scores represent the mean of constituent items. Missing data (less than 1% on each measure) were imputed using SPSS Missing Values software (version 24) with ten imputed data sets (Rubin, 1987).
Initial interest in biology
Interest in biology was measured with five items (“I'm really looking forward to learning more about biology,” “Biology fascinates me,” “I think the field of biology is very interesting,” “I'm excited about biology,” “To be honest, I just don't find biology interesting,” reversed, α = .93). Questionnaire measures of interest typically include items that measure value as well as positive affect (Harackiewicz, Durik, Barron, Linnenbrink-Garcia, & Tauer, 2008; Renninger & Hidi, 2011; Schiefele, 1991); however, given that the UV intervention targets values explicitly, we focus on affective processes in order to separate interest from value in our measurement. For example, we did not include two items from an earlier measure of initial interest (“I think what we will study in this course will be important for me to know,” and “I think what we will study in this course will be worthwhile to know;” Harackiewicz et al., 2008).
Prior performance
We obtained students' prior college GPA from university records (n = 533). For students with missing records (e.g., transfer students, first-year students), we used students' self-reported college GPA from the baseline questionnaire (n = 29). Cases with missing data on all prior performance measures (n = 15) were imputed using SPSS Missing Values software (version 24) with ten imputed data sets (Rubin, 1987).
Outcome Measures
Biology course grade
Course instructors provided final course grades at the end of the semester, on a 4.0 scale (A = 4.0, AB = 3.5, B = 3.0, BC = 2.5, C = 2.0, D = 1.0, F = 0). Grading standards and procedures were criterion based and consistent across sections (M = 2.94, SD = .72).
Continuation to the second course of the introductory biology sequence
We assessed students' course taking in the following semester, using course records, to determine whether they continued on to the second course of the introductory biology sequence. Though it is possible (but quite rare) for students to take the second course nonconsecutively, both semesters are required for biological and medical science majors. Failure to enroll in the second course would exclude students from 34 biological and medical majors and is therefore a preliminary indicator that a student is moving away from the biomedical sciences.
STEM major persistence
Self-reported college major was measured at the beginning and end of the semester. If students had not formally declared a major (n = 270; 46.8% of the sample), they were asked to list the major(s) they were considering. Majors were coded as STEM or non-STEM by two coders with an interrater reliability of 95%. For the purposes of this study, majors in the physical, biological and natural sciences, engineering, computer science, and mathematics were coded as STEM majors. Majors in the social sciences and humanities were coded as non-STEM. We compared students' reported major at the beginning of the semester to their major at the end of the semester and classified students into one of four categories: remained in non-STEM major, changed from non-STEM to STEM major, remained in STEM major, changed from STEM to non-STEM major. Since the majority of students entered the course with a STEM major (92.5%), we focus primarily on whether students remained in STEM or changed their intended major to a non-STEM field at the end of the semester, as a measure of STEM major persistence. Eleven students did not report their major at either the beginning and/or the end of the semester and were therefore excluded from analyses on this variable.
Articulated utility value
The UV and control writing assignments were coded for the level of utility value articulated in the essay. Research assistants coded the assignments on a 0-4 scale based on how specific and personal the UV connection was to the individual (i.e., the student or a close friend or family member for students who chose to write a letter in UV conditions). A “0” on this scale indicates no utility or relevance; a “1” indicates general utility or relevance applied to humans generically; a “2” indicates utility or relevance that is general enough to apply to anyone, but is applied to the individual; a “3” indicates utility or relevance that is specific to the individual; and a “4” indicates a strong, specific connection to the individual that includes a deeper appreciation or future application of the material. UV scores from the three essays were averaged to create an overall measure of articulated utility value, with higher scores indicating higher-quality UV connections. Inter-rater reliability with this coding rubric was high, with two independent coders providing the same score on 91% of essays. Disagreements were resolved by discussion.
Results
Analytic Strategy
Table 1 presents descriptive statistics and intercorrelations for all measures. Although students were nested within three lecture sections, randomization to condition occurred at the student level. The intraclass correlation coefficient was small; lecture sections accounted for only 2.4% of the variance in biology course grade. Indeed, hierarchical linear model (HLM) and regression yielded consistent results. Therefore, all analyses reported here were conducted with regression so that we can report effect sizes (betas).
Table 1. Zero-Order Correlations and Descriptive Statistics for Major Variables.
| 1 | 2 | 3 | 4 | 5 | 6 | |
|---|---|---|---|---|---|---|
| 1. Prior GPA | -- | |||||
| 2. Articulated Utility Value | .10* | -- | ||||
| 3. Biology Course Grade | .60** | .10* | -- | |||
| 4. Continuation to Second Course | .15** | .04 | .28** | -- | ||
| 5. STEM Major Persistence | .09 | .04 | .14** | .22** | -- | |
| 6. Initial Interest in Biology | -.01 | -.04 | .11** | .21** | .03 | -- |
|
| ||||||
| Range | .86-4 | 0-4 | 0-4 | -- | -- | 1-7 |
| M or % | 3.27 | 1.74 | 2.94 | 78.3% | 92.5% | 5.82 |
| SD | 0.54 | 1.00 | 0.72 | -- | -- | 1.00 |
| N | 577 | 577 | 577 | 577 | 534 | 577 |
Note. GPA = grade-point average. STEM = science, technology, engineering, and math.
p < .05,
p < .01.
The primary outcome measures were the final grade in the biology class, continuation in the introductory biology sequence, and STEM major persistence across the semester. Articulated utility value was examined as a check of the UV manipulation. We used OLS regression for the continuous outcome measure (biology course grade and articulated utility value) and logistic regression for the dichotomous outcome measures (continuation into the second course and STEM major persistence). Significant effects are described below, and full regression results are displayed in Table 2.
Table 2. Effects of the UV Intervention on Course Grade, Continuation, and STEM Major Persistence.
| Predictor | Biology Course Grade | Continuation to Second Course | STEM Major Persistence | ||||||
|---|---|---|---|---|---|---|---|---|---|
|
| |||||||||
| β | t(571) | p | B | Wald | p | B | Wald | p | |
| UV vs. Control | .09 | 2.63 | .009 | .30 | 4.55 | .033 | .51 | 4.44 | .035 |
| Prior GPA | .62 | 13.27 | .000 | .25 | 3.46 | .063 | .44 | 4.39 | .036 |
| UV × Prior GPA | -.02 | -0.34 | .734 | .17 | 1.50 | .221 | -.08 | 0.15 | .696 |
| Gender | .10 | 3.03 | .003 | -.13 | 1.45 | .229 | .16 | 0.58 | .445 |
| Initial Interest in Biology | .11 | 3.39 | .001 | .52 | 25.40 | .000 | .18 | 0.75 | .387 |
Note. UV = utility-value; UV vs. Control (control = -1, UV conditions = 1); Gender (female = -1, male = 1); GPA = grade-point average.
Test of Intervention Efficacy
First, we tested the efficacy of the UV intervention averaged across dosage and timing. We tested the main effect of the UV intervention (control = -1, UV = 1), collapsing across 7 UV conditions and controlling for gender (female = -1, male = 1), prior performance, and initial interest in biology. We also tested the interaction of the UV intervention with prior performance, allowing us to test for replication of prior UV research that found that the UV intervention was particularly effective for students with low grades (Harackiewicz, Canning, et al., 2016; Hulleman et al., 2010). In preliminary analyses, we tested initial interest in biology and gender as moderators of the UV intervention, however there were no interactions with the UV intervention on any dependent variable for either moderator, therefore, we trimmed these interactions from the final model.
Manipulation Check: Articulated Utility Value
In order to test whether the UV intervention led participants to articulate more utility-value connections in their essays than those in the control condition, we examined articulated utility value, which was measured on a 4-point scale assessing the quality of the utility value articulated in an essay. As expected, students in the UV conditions made higher-quality UV connections (i.e., made more personal and more specific connections to curricular content) in their essays (M = 1.96, SD = .91) than those in the control condition (M = .45, SD = .40), t(571) = 14.78, β = 0.52, p < .001.2 Indeed, the majority of students in the control condition (88%) made no personal UV connections in any of their essays. This important manipulation check indicates that the UV intervention was successful in encouraging students to make personal connections with the course material in their writing assignments. Gender, initial interest in biology, Prior GPA, and the interaction between the UV intervention and prior GPA, were not significant.
Biology course grade
We found that the UV intervention increased course grades (M = 2.96, SD = .71), compared to the control condition (M = 2.77, SD = .75), β = .09, p = .009, indicating that students who wrote a UV essay at any point in the semester received on average a higher grade in the course than students in the control group. There was a positive effect of gender, β = .10, p = .003, indicating that men performed better than women in this course. Prior GPA, β = .62, p < .001, and initial interest in biology, β = .11, p = .001, were significant predictors of biology course grade, indicating that students with higher prior GPAs and higher interest at the beginning of the course received higher grades in the course. The interaction between the UV intervention and prior GPA was not significant.
Continuation to the second course
We tested the same model used for biology grade, but used binary logistic regression to examine whether students enrolled in the second course of the biology sequence. Students in the UV condition were more likely to enroll in the second course of the sequence, Wald = 4.55, B = .30, p = .033. We found that 79.80% of students who wrote about UV at least once continued to the next semester, compared to only 69.50% in the control condition. Initial interest in biology significantly predicted whether students enrolled in the second course of the sequence, Wald = 25.40, B = .52, p < .001. Prior GPA was a marginal predictor of enrollment in the second course of the sequence, Wald = 3.46, B = .25, p = .063; however, this effect did not reach significance. The interaction between the UV intervention and prior GPA was not significant and there was no effect of gender.
STEM major persistence
We used binary logistic regression to test whether the UV intervention influenced students' STEM major decisions at the end of the semester using the same model as above. Our primary interest is whether students persisted in a STEM major across the semester, and so we excluded students who were not STEM majors when they started the class. Therefore, this analysis is conducted with the 534 students (93%) who initially reported a STEM major at the beginning of the semester. Among these students, we found that students in the UV conditions were more likely to persist in their STEM major, Wald = 4.44, B = .51, p = .035 (see Table 3). That is, 96.1% of students in the UV conditions remained in a STEM major from the beginning to the end of the semester, compared to 89.2% of students in the control condition. This suggests that the UV intervention was successful at retaining students in STEM disciplines across the semester, relative to the control condition. Prior GPA was a significant predictor of STEM major persistence, Wald = 4.39, B = .44, p = .036. The interaction between the UV intervention and prior GPA was not significant and there was no effect of gender or initial interest in biology.
Table 3. Reported major from beginning to end of semester.
| Control | UV Conditions | Total | |
|---|---|---|---|
| Started in Non-STEM Major | |||
| Remained in Non-STEM | 2 (40.0%) | 19 (70.4%) | 21 |
| Changed from Non-STEM to STEM | 3 (60.0%) | 8 (29.6%) | 11 |
| Total | 5 | 27 | 32 |
| Started in STEM Major | |||
| Remained in STEM | 66 (89.2%) | 442 (96.1%) | 508 |
| Changed from STEM to Non-STEM | 8 (10.8%) | 18 (3.9%) | 26 |
| Total | 74 | 460 | 534 |
Note. UV = utility value; STEM = science, technology, engineering, and math
Mediation Analyses
We used Hayes' (2013) bootstrapping procedure with PROCESS software, which allowed us to test biology course grade as a mediator of the UV intervention effect on enrollment in the second course of the sequence and STEM major persistence (see Figure 1). We hypothesized that the UV intervention would increase grades in the course, leading students to be more likely to enroll in the second course of the sequence and persist in their STEM major. Results based on 5,000 bootstrapped samples indicate that the indirect effect via biology course grade equaled .08, 95% CI [.0172, .1781] for enrollment in the second course and .06, 95% CI [.0007, .1685] for STEM major persistence. The confidence intervals of the indirect effects do not include zero; thus we can conclude that biology course grade partially mediated the positive effect of the UV intervention on continuation to the second course of the sequence and STEM major persistence.
Figure 1.
Mediation model showing the effect of the utility-value intervention on continuation to the second course in the sequence and STEM major persistence, as mediated by biology course grade. Values represent unstandardized coefficients, and values inside parentheses represent standard errors. Regression analyses include gender, initial interest in biology, and prior GPA as covariates. Along the lower path, the values above the arrow show the total effect of the utility-value intervention, and the values below the arrow show the direct effect. *p < .05, **p < .01, ***p < .001.
We also tested whether articulated utility value (measured before the end of the course) explained the UV intervention effect on course grades, continuation, and STEM major persistence; however, these indirect effects were not significant. The indirect effect via articulated utility value equaled .02, 95% CI [-.0252, .0604] for biology course grade, -.06, 95% CI [-.2465, .1260] for continuation to the second semester, and -.16, 95% CI [-.6122, .2710] for STEM major persistence.
Test of Intervention Timing and Dosage: Crossed Design
We found that the UV intervention increased grades, promoted continuation in the biology course sequence, and promoted retention for students in STEM majors across the semester; however, an important question remains about how best to implement UV interventions: What is the optimal timing and dosage of the UV assignments? To address this research question, we tested whether the UV assignments were more effective at different points in the semester and whether there was value added by assigning more UV essays in the course. Utilizing the 2 (First Essay: UV vs. Control) by 2 (Second Essay: UV vs. Control) by 2 (Third Essay: UV vs. Control) fully crossed design, we tested all interactions between the three treatment main effects (control = -1, UV = 1; for each of the three essays). We also tested all interactions with prior GPA, allowing us to examine whether replication of prior UV research (Harackiewicz, Canning, et al., 2016; Hulleman et al., 2010) depends on the timing and dosage of the intervention. Gender (female = -1, male = 1) and initial interest in biology were included as covariates. The final model included 18 terms: the main effects of the first, second, and third essay, prior GPA, six two-way interactions, four three-way interactions, one four-way interaction, and two covariates (see Table 4 for descriptive statistics by experimental condition and Table 5 for full regression results). The primary outcome measures for the test of timing and dosage were articulated utility value, the final grade in the biology class and continuation in the introductory biology sequence (given that 92.5% of students who reported majoring in STEM at the beginning of the semester persisted with a STEM major by the end of the semester, we did not have enough power to test the fully crossed model on STEM persistence). To interpret significant interactions with prior GPA, predicted values were generated for individuals one standard deviation below and above the mean.
Table 4. Descriptive Statistics for Course Grade, Continuation, and Articulated Utility Value by Experimental Condition.
| Biology course grade | Continuation to second course | Articulated utility value | ||||
|---|---|---|---|---|---|---|
|
| ||||||
| Utility Value Dosage | Unit for Utility Value Assignment | Label | N | M (SE) | % | M (SE) |
| 0 | None | CCC | 82 | 2.78 (0.06)a | 69.98jl | 0.42 (0.07)w |
|
| ||||||
| 1 | 1st | VCC | 71 | 3.00 (0.07)bc | 85.37km | 1.26 (0.07)x |
| 2nd | CVC | 70 | 2.97 (0.07)bc | 81.84jkm | 1.26 (0.07)x | |
| 3rd | CCV | 71 | 2.98 (0.07)bc | 84.40km | 1.39 (0.07)x | |
|
| ||||||
| 2 | 1st & 2nd | VVC | 71 | 2.91 (0.07)abc | 64.32l | 2.27 (0.07)y |
| 1st & 3rd | VCV | 67 | 2.83 (0.07)ab | 75.67jkl | 2.16 (0.08)y | |
| 2nd & 3rd | CVV | 69 | 2.97 (0.07)bc | 75.61jkl | 2.15 (0.07)y | |
|
| ||||||
| 3 | All | VVV | 76 | 3.05 (0.06)c | 89.80m | 3.14 (0.07)z |
Note. Predicted values from the regression equation, at the mean of each covariate (prior GPA, interest, gender). Posthoc comparisons using LSD are shown using superscripts; means/percentages with different superscripts are different at the p < 0.05 level. C = Control. V = Utility Value.
Table 5. Effects of Intervention Timing and Dosage on Course Grade, Continuation, and Articulated Utility Value.
| Biology course grade | Continuation to second course | Articulated utility value | |||||||
|---|---|---|---|---|---|---|---|---|---|
| β | t(559) | p | B | Wald | p | β | t(559) | p | |
| Essay 1 (E1) | .01 | 0.30 | .765 | .02 | 0.03 | .867 | .44 | 17.32 | .000 |
| Essay 2 (E2) | .05 | 1.55 | .122 | -.02 | 0.03 | .860 | .44 | 17.35 | .000 |
| Essay 3 (E3) | .03 | 0.97 | .331 | .25 | 4.38 | .036 | .45 | 17.57 | .000 |
| E1 × E2 | -.01 | 0.16 | .877 | -.00 | 0.00 | .995 | .05 | 2.07 | .039 |
| E1 × E3 | -.04 | 1.19 | .233 | .04 | 0.14 | .708 | -.01 | -0.22 | .824 |
| E2 × E3 | .02 | 0.48 | .631 | .10 | 0.69 | .405 | -.01 | -0.50 | .621 |
| E1 × E2 × E3 | .09 | 2.78 | .006 | .48 | 16.96 | .000 | .00 | 0.01 | .99 |
| Prior GPA | .61 | 18.45 | .000 | .41 | 14.84 | .000 | .04 | 1.53 | .126 |
| E1 × GPA | -.01 | 0.33 | .742 | -.05 | 0.22 | .641 | -.01 | -0.43 | .665 |
| E2 × GPA | -.02 | 0.50 | .619 | .06 | 0.35 | .555 | .05 | 1.84 | .066 |
| E3 × GPA | .02 | 0.56 | .573 | .23 | 4.48 | .034 | .05 | 1.92 | .056 |
| E1 × E2 × GPA | .04 | 1.13 | .258 | .11 | 1.10 | .295 | .01 | 0.35 | .730 |
| E1 × E3 × GPA | .02 | 0.72 | .471 | -.02 | 0.03 | .861 | .00 | 0.16 | .874 |
| E2 × E3 × GPA | -.06 | 1.80 | .072 | -.07 | 0.38 | .538 | .03 | 1.09 | .275 |
| E1 × E2 × E3 × GPA | -.02 | 0.47 | .641 | .12 | 1.23 | .267 | -.01 | -0.28 | .780 |
| Gender | .10 | 2.89 | .004 | -.16 | 2.06 | .152 | -.03 | -1.08 | .281 |
| Initial Interest in Biology | .12 | 3.48 | .001 | .56 | 27.06 | .000 | .02 | 0.71 | .478 |
Note. Essay 1, Essay 2, and Essay 3 (control = -1, utility value = 1); Gender (female = -1, male = 1); GPA = grade-point average.
Manipulation Check: Articulated Utility Value
As expected, we found a treatment effect for each time point: first essay, β = .44, p < .001, second essay, β = .44, p < .001, third essay, β = .45, p < .001, indicating that students writing a UV essay at any time point, made higher-quality UV connections in their essays, on average, compared to writing a control essay at each time point. This pattern of findings suggests an additive effect of UV dosage, such that three doses might prove optimal. Indeed, posthoc analyses using LSD indicate that students made higher-quality UV connections, on average, when they received three doses of UV, compared to one or two doses of UV (see Table 4). In order to examine dosage effects on articulated utility value, it is important to consider that some students wrote more UV essays than others, given our experimental design, therefore, we report supplementary analyses below to examine these effects in greater detail.
There was a significant two-way interaction with the first and second essay, β = .05, p = .039, indicating that there was a synergistic effect of writing a UV essay for the first two units of the course. In other words, students made higher-quality UV connections when they wrote a UV essay for the first two units (Ŷ = 2.70), compared to writing a UV essay for either the first (Ŷ = 1.70) or second unit (Ŷ = 1.70) only, or a control essay for both units (Ŷ = 0.92). There was also a marginal interaction between the third essay and prior GPA, β = .05, p = .056, indicating that students with high prior GPAs made somewhat higher-quality UV connections in UV essays for the third unit of the course (Ŷ = 2.29), compared to students with low prior GPAs (Ŷ = 2.11), though this difference was relatively small. Gender, initial interest in biology, and Prior GPA were not significant.
Biology course grade
We tested the fully crossed model to examine whether the UV assignments were more effective at different points in the semester. First, we examined the UV condition main effects for essay 1, essay 2, and essay 3, to determine whether there were unique benefits to receiving a UV assignment at any of the three time points. None of the treatment main effects were significant, p > .10. In other words, the UV assignments were not differentially effective at the three time points. However, there was a significant three-way interaction between the three UV condition terms, β = .09, p = .006. To probe the meaning of the three-way interaction, we conducted posthoc comparisons using LSD (see Table 4) and discuss overall patterns below. There was a positive effect of gender, β = .10, p = .004, indicating that men performed better than women in this class. Prior GPA, β = .61, p < .001, and initial interest in biology, β = .12, p = .001, were also significant predictors of biology course grade. There were no significant interactions with prior GPA.
Continuation to the second course
In contrast to the results for course grade, the results for continuation revealed an effect of timing. There was a positive main effect of UV condition for the third essay, Wald = 4.38, B = .25, p = .036, indicating that students who wrote a UV essay in the final unit of the course were more likely to enroll in the second biology course than students who wrote a control essay in the final unit of course. However, this effect was qualified by a significant two-way interaction between the third essay and prior GPA, Wald = 4.48, B = .23, p = .034, suggesting that UV implemented during the final unit of the course was most effective for promoting enrollment for high-performing students and may have decreased enrollment for students with very low (more than 1 standard deviation below the mean) prior GPAs (see Figure 4).
Figure 4.
Probability of continuing to the second course as a function of essay 3 (Control or Utility Value) and prior GPA.
There was also a significant three-way interaction between the three UV condition terms, Wald = 16.96, B = .48, p < .001. To probe the meaning of the three-way interaction, we conducted posthoc comparisons using LSD (see Table 4) and discuss general patterns below. There were positive main effects of prior GPA, Wald = 14.84, B = .41, p < .001, and initial interest in biology, Wald = 27.06, B = .56, p < .001, indicating that students with higher prior GPAs and higher interest at the beginning of the course were more likely to enroll in the second course. There was no effect of gender.
Posthoc analyses of the three-way interaction on course grades and continuation
For performance, a dosage pattern emerged among the UV conditions based on posthoc comparisons using LSD (Table 4). In general, both three doses as well as a single dose of UV (at any time point) promoted performance, compared to the control condition (see Figure 2). Although, the condition in which students received the full dosage of UV had the highest overall mean, it was not more effective than any of the three conditions in which students received a single dose of UV. In other words, both single doses of UV and the full dosage of UV were more effective for performance than two doses of UV. This pattern was not predicted, but may reflect positive effects of novelty with a single dose and the predicted positive effect of three doses administered throughout the semester.
Figure 2.
Biology course grade as a function of experimental condition. Condition labels reflect the order in which students completed the writing assignment (C = Control, V = Utility Value); therefore, CVC refers to the condition in which students wrote a control essay in the first unit, a utility value essay in the second unit, and a control essay in the third unit. Error bars represent 95% confidence intervals.
The pattern was similar for continuation to the second course. A single dose of UV (administered either at the beginning or the end of the semester) and three doses of UV promoted continued enrollment in the biology sequence (see Figure 3). Indeed, 85.92% of students who were assigned one UV essay at the beginning of the semester, 84.51% of students who were assigned one UV at the end of the semester, and 88.16% of students who were assigned all three UV essays continued to the next semester, compared to only 69.51% in the control condition. Although the condition in which students received the full dosage of UV had the highest continuation rate, it was not more effective at promoting continued enrollment than a single dose of UV administered either at the beginning or the end of the semester. For both performance and continuation, this pattern suggests that either one novel dose of UV or three consistent doses of UV were particularly effective, whereas two doses were not.
Figure 3.
Percentage of students who continued to the second course of the biology sequence as a function of experimental condition. Condition labels reflect the order in which students completed the writing assignment (C = Control, V = Utility Value); therefore, CVC refers to the condition in which students wrote a control essay in the first unit, a utility value essay in the second unit, and a control essay in the third unit. Error bars represent 95% confidence intervals.
Mediation analyses
We used Hayes' (2013) bootstrapping procedure with PROCESS software, which allowed us to test a mediation model similar to Figure 1, with biology course grade as a mediator of the three-way UV interaction on continuation to the second course of the sequence. We hypothesized that the full dosage of the UV intervention would increase grades in the course, leading students to be more likely to enroll in the second course of the sequence. Results based on 5,000 bootstrapped samples indicate that the indirect effect via biology course grade equaled .06, 95% CI [.0164, .1268]. The fact that zero falls outside this interval indicates significant partial mediation.
We also tested whether articulated utility value (measured before the end of the course) explained the three-way UV interaction on course grades and continuation; however, neither indirect effect was significant. The indirect effect via articulated utility value equaled = .00, 95% CI [-.0115, .0161] for biology course grade, and = -.00, 95% CI [-.0043, .0026] for continuation to the second course.
Supplementary Analyses
Supplementary analysis of dosage effects on articulated value
In order to examine dosage effects on articulated utility value, it is important to consider that some students wrote more UV essays than others, given our experimental design. Students who received one or two UV assignments also received one or two control assignments by design, in which we would expect lower levels of articulated value, and it is therefore important to examine the average level of utility value articulated in UV essays. We conducted an internal analysis of the UV essays to examine whether the dosage of UV assignments contributed to the quality of the UV connections students articulated in their essays. Specifically, we averaged the level of articulated utility value in all UV essays written by each student, but excluded the control essays. For instance, if a student was assigned a UV essay for the first two units and a control essay for the third unit, we averaged the level of articulated utility value in just the first two essays. We tested whether the number of UV assignments students received (1, 2, or 3) predicted the average level of articulated utility value for the UV essays written. These analyses exclude the 82 participants who wrote three control essays, and thus no UV essays. We also tested the interaction with prior GPA, and included gender (female = -1, male = 1) and initial interest in biology as covariates.
We found that number of UV essay assignments significantly predicted the average level of articulated utility value in UV essays, t(489) = 2.03, β = .09, p = .043 (M1UV = 2.87, SD1UV = 1.22; M2UVs = 2.99, SD2UVs = .95; M3UVs = 3.14, SD3UVs = .67). In other words, students articulated higher quality UV connections if they wrote more UV essays, suggesting that practice helps students find better UV connections. There was a significant effect of gender, t(489) = 2.35, β = -.11, p = .019, indicating that the average level of articulated utility value in UV essays was higher among women than among men. Initial interest in biology was also a significant predictor of the average level of articulated utility value, t(489) = 1.99, β = .09, p = .047. Prior GPA and the interaction with number of UV essays were not significant.
To test whether there were “spillover” effects of the UV essay assignments, we examined whether the dosage of UV assignments predicted the level of articulated utility value in all control essays written by each student, excluding all UV essays. For instance, if a student was assigned a UV essay for the first two units and a control essay for the third unit, the student might spontaneously generate UV connections in the control essay. Using the same model as above, we tested whether the number of UV assignments students received (0, 1, or 2) predicted the average level of articulated utility value for the control essays written. These analyses exclude the 76 participants who wrote three UV essays, and thus no control essays.
We found that number of UV essay assignments was a marginal predictor of the average level of articulated utility value in control essays, t(495) = 1.91, β = .08, p = .057 (M0UV = .45, SD0UV = .40; M1UV = .50, SD1UV = .57; M2UVs = .59, SD2UVs = .81). Though this effect did not reach statistic significant, it was in the hypothesized direction. In other words, students who wrote more UV essays articulated somewhat more UV connections in their control essays, on average, indicating that writing UV essays may increase the likelihood of making UV connections in subsequent control essays. Gender, initial interest in biology, Prior GPA and the interaction with number of UV essays were not significant.
Supplementary analysis of timing
To examine the effect of timing more directly, we conducted an exploratory analysis in a 3-cell subset of the experimental design. We identified the three UV conditions in which students received a single UV assignment, combined with two control assignments (UV first, UV second, or UV third). There was not enough variability in STEM major persistence within these three cells to test persistence as an outcome. Therefore, we examined the effects of timing on course grade, continuation to the second course, and articulated utility value. We hypothesized that writing a UV essay at the beginning of the semester (UV first) would be more effective than writing a UV essay at the end of the semester (UV third), especially for low-achieving students. To test our hypothesis, we used two dummy variables to examine the effects of the three conditions on course grade and continuation. The first dummy variable tested our hypothesis directly and compared students who wrote a UV essay in the first unit, followed by two control essays, versus students who wrote a UV essay in the third unit, preceded by two control essays (UV first = 1, UV third = 0). The second dummy variable compared students who wrote a UV essay in the second unit versus students who wrote a UV essay in the third unit (UV second = 1, UV third = 0). We interacted the two dummy variables with prior GPA and included gender (female = -1, male = 1) and initial interest in biology as covariates.3
There was a significant two-way interaction on course grade between the first dummy variable and prior GPA, t(204) = 2.15, β = -.17, p = .033. This shows that students with low prior GPAs earned higher grades in the course when they wrote about UV in the first unit (Ŷ = 2.68), compared to writing a UV essay in the third unit (Ŷ = 2.49). The opposite was true for students with high prior GPAs, in that writing a UV essay in the third unit was more beneficial (Ŷ = 3.51) than writing a UV essay in the first unit (Ŷ = 3.33). This interaction was in the same direction for continuation, though the effect did not reach statistical significance, Wald = 3.47, B = -.86, p = .063. There were no significant main effects or interactions on articulated utility value. As noted in the primary analyses, prior GPA was a significant positive predictor of biology course grade, t(204) = 8.03, β = .77, p < .001, and continuation to the second course in the biology sequence, Wald = 7.85, B = .98, p = .005, and initial interest in biology was a significant positive predictor of continuation, Wald = 4.96, B = .50, p = .026. There was no effect of gender on course grade as in the primary analyses, or any other outcome, possibly due to the reduced sample.
Taken together, these analyses suggest that students with a history of poor performance may benefit from writing a UV essay in the beginning of the semester, whereas high-performing students may benefit from writing a UV essay at the end of the semester. This is consistent with the effect of the third essay. We tested timing in two different ways, once with the fully crossed model and once with a subset of the design that examined timing across the three UV conditions in which students received a single UV assignment. Both tests of timing reveal that a UV assignment at the third time point was especially important for students with high prior GPAs.
Summary of Dosage and Timing Effects Across Measures
In our dosage analysis we found that three doses of the UV intervention was most beneficial for making high-quality UV connections, yet a single dose was just as effective as three doses on performance and continuation. In other words, either 1 novel UV assignment or 3 consistent UV assignments implemented throughout the semester were effective. In contrast, in our timing analysis we found that the results for a single dose are complex, because the timing is crucial for students depending on their prior performance, and the results differ by outcome measure. One dose of UV implemented early increased performance for low performers, whereas one dose of UV implemented late increased continuation for high performers. Across all analyses, we found that two doses were not effective for performance or continuation. Considered together, we conclude that full dosage (three UV assignments) is optimal to obtain the highest level of articulated UV, the beneficial early (first UV) effect on performance for low performers, and the beneficial late (3rd UV) effect on continuation for high performers.
Discussion
The results presented here indicate that a UV writing intervention implemented in an introductory course can impact performance, continued enrollment, and the retention of students in STEM majors. To our knowledge, this is the first study to demonstrate the downstream implications of a UV writing intervention. We found that the UV intervention increased performance in the course, as well as persistence in the two-semester biology sequence. Connecting course material to personal goals and values influenced whether students decided to enroll in the second course of the sequence and this process was partially mediated by their grade in the course. In other words, earning a good grade in this foundational course provided the springboard for enrolling in the second course. In addition to increasing grades and subsequent enrollment, our results suggest that the UV intervention also increased STEM major persistence across the semester. Students were more likely to report staying in their STEM major at the end of the semester in UV conditions, compared to the control group. Although these effects were relatively small in magnitude, it is worth noting that the positive effects of the UV intervention were robust, given that prior performance was controlled in all analyses. Moreover, the control condition was designed to provide pedagogical value (i.e., students were integrating and summarizing course material); even a small increase in performance, retention, and major persistence relative to this active control is a noteworthy improvement and has real-life implications for students' academic and professional careers.
Expectancy-value theory posits that perceptions of utility value in one context can have downstream implications for other contexts or future goals (Eccles & Wigfield, 2002); however, this has not been tested experimentally. Our results show that a UV manipulation can not only improve grades in the immediate context, but also improve retention outcomes that are related to students' personal goals, such as sticking with their intended STEM major. This finding supports the notion that the connection between UV and future goals is borne out in college contexts, such that a UV intervention can impact important decisions that students make about their academic and career paths.
Implications for Intervention Dosage
The design of this study and the large sample size allowed us to test effects of dosage and timing, in order to determine how best to implement the intervention. We found that assigning a single dose of UV in any unit was as effective as three UV essays consistently throughout the semester (once for each unit of the course) at increasing performance and continuation to the second course essay. Although students assigned the maximum dosage had the highest overall mean grades and continued enrollment rates, assigning only one UV essay had similar effects. In other words, a single UV assignment might be powerful because of its novelty and three UV assignments might be powerful because of their consistency.
One hypothesis is that students need practice with the UV writing assignment in order for it to be effective, and consistency when implementing UV assignments might facilitate this. It could be that after the first time students attempt to articulate how the course material is relevant or useful to them, they are better able to generate personal connections a second or third time. Our analysis of articulated utility value was consistent with this possibility. Posthoc comparisons and our internal analysis of the UV essays revealed that there was an additive effect of the UV writing assignments on articulated utility value, such that the more UV essays students wrote, the higher the quality of their articulated utility-value connections, on average. The first UV writing assignment may prime students to start thinking about utility value for other units in the course and the second and third assignments act as intervention boosters. In other words, once students write a UV essay, it may be easier to write additional UV essays, and thus they can generate higher-quality UV connections in subsequent essays.
Interestingly, although the UV intervention promoted articulated utility value (as intended) it did not mediate course grades or the long-term impact of the intervention. It is possible that students may think about or engage with the material differently as a result of repeatedly writing UV essays, resulting in increased performance in the class, and it may not necessarily be the quality of UV connections that drives performance. For instance, the UV assignments may set into motion other psychological processes and behaviors that influenced performance, such as greater engagement with course materials more broadly, more class participation, or feeling a sense of purpose. It will be important in future research to explore the underlying psychological changes and behaviors that occur as a result of this intervention that were not assessed in the current study.
An important feature of the UV intervention, as implemented in the current study, is that students received feedback from graders. Indeed, if UV assignments are to be integrated into course curricula at scale, it is important to test them in the way that writing assignments are typically utilized in science classes, which usually involves grading for adherence to instructions and content mastery. With such feedback, students can have more than one opportunity to revise their essay strategy. For instance, if in the first UV essay, a student makes only a superficial connection to the material (e.g., “carbohydrates are important because humans need them to survive”), rather than a well-developed, personal connection to the material (e.g., “As a runner, I can use what I learned about carbohydrates in this class to create my diet plan and optimally train for a marathon”), the grader can encourage the student to generate more specific and personal connections. The benefits of this correction process may not occur if the student only completes one or two essay assignments and therefore, multiple essay assignments might be needed to optimize this process. Though the UV exercise requires instructors to invest time in grading the essays, we believe this cost is justified given that students benefit from the intervention not only in terms of increased course grades, but also by persisting in the course sequence and STEM fields more broadly. It is unknown whether feedback on the assignment is an essential part of the intervention or whether simply completing the assignment thoughtfully is sufficient for its impact on student motivation and performance; therefore, it is possible that the time investment could be reduced by limiting instructor feedback. This is an important question, given that the cost of grading may be a deterrent for some departments to add UV writing to their curricula, and it should be addressed in future research.
Implications for Intervention Timing
In our primary analyses of intervention efficacy, we did not replicate previous research that showed that the UV intervention was most effective for students with a history of poor performance (Hulleman et al., 2010; Harackiewicz et al., 2016). Instead, it wasn't until we conducted a detailed analysis of intervention timing that we discovered when interactions with prior performance were likely to emerge. We found that students with high prior GPAs were more likely to continue to the second course if they wrote a UV essay at the end of the semester. Previous tests of the UV writing intervention had not found positive effects for high-performing students, but this may be due to the fact that no study has examined outcomes that extend beyond the course in which the intervention was implemented. In this study, we examined continuation and found a positive effect for high-performing students.
It is possible that the timing of the third essay coincided with the timing of when students were deciding which courses to take the following semester. Most students in this course were eligible to enroll in courses for the spring semester in the last four weeks of the fall semester and the third writing assignment was implemented during this period. High-performing students may need less support from a UV intervention administered repeatedly throughout the semester and may benefit most from a UV writing assignment when it is implemented at a critical time in the decision-making process. The third UV essay may have given these students that extra “nudge” to enroll in the next course. Interestingly, the third UV essay may have been detrimental for students with very low prior GPAs, in that they were less likely to continue to the second course after writing a UV essay in the final unit of the course. This is consistent with the notion that enhancing value with students who do not feel competent could actually lead to decreases in long-term motivation (e.g., Canning & Harackiewicz, 2015; Trautwein et al., 2012). Indeed, expectancy-value theory suggests that when students have higher value for an academic domain, but low expectations of being able to succeed in that domain, the value will only do so much to push them to continue – but if they have high value for the domain paired with high expectations to succeed in the domain, they will be more likely to continue (Eccles & Wigfield, 2002; Nagengast et al., 2011). Our results support this positive interaction between expectancies (prior performance) and value (utility value) on continued motivation. Students with high expectancies (those with high prior performance) benefitted more from an injection of value (the UV intervention) at the end of the semester when they were making their enrollment decisions.
In contrast, when we compared early vs. late implementation of the UV intervention, students with a history of poor performance needed the UV intervention early in the course in order for it to affect their performance in the course. Our internal analysis of intervention timing suggests that if only one UV assignment is given, students with a history of poor performance benefit more in terms of performance when the UV intervention is implemented early in the semester, whereas high-performing students benefit more when the UV intervention is implemented in the last (vs. first) unit of the course. Thus, we tested timing in two different ways, once with the fully crossed model and once with a subset of the design that examined timing across the three UV conditions in which students received a single UV assignment. Both tests of timing reveal that a UV assignment at the third time point was especially important for students with high prior GPAs, whereas the analysis in which we compared only a single dose suggests that the first time point is especially important for increasing performance for students with low prior GPAs.
This is consistent with prior research, in that the UV intervention was most effective for lower performing students, but only when they complete a UV assignment early in the semester (Hulleman et al., 2016). Our results indicated that the timing of the UV intervention may be particularly important for students, depending on their prior performance, and this has implication for replicating past research. Because the UV intervention can facilitate the learning process, students with a history of poor performance may benefit most in terms of performance from receiving the intervention early in the semester, to start them off on the right foot. In contrast, students with a history of high performance (who are already more likely to perform well in the course) may benefit most from an intervention at the end of the semester, to give them the extra push of motivation to do well in the “home stretch.”
Recommendations for Practice
Our results suggest that there is not a one size fits all approach to implementing the UV intervention. If educators are concerned about students making high-quality UV connections, assigning three doses of UV is optimal. If educators are concerned about increasing performance for students struggling in the course, assigning one essay early in the semester may be most effective. If educators are concerned about retention, assigning one essay at the end of the semester may be most effective, but only for students with high prior performance. Naturally, a compromise might be to assign two essays, once at the beginning and once at the end; however, we found that this condition was not more effective than the control on either performance or continuation. Therefore, the safest approach in our view is to implement the intervention multiple times throughout the semester. In the present study that meant implementing the intervention three times, once for each unit of the course. Three doses of UV were necessary to obtain the highest levels of articulated value, the beneficial early (first UV) effect on performance for low performers, and the beneficial late (3rd UV) effect on continuation for high performers. This recommendation is consistent with previous research suggesting that students may benefit from multiple opportunities to make UV connections with the material (Hulleman et al., 2016).
Limitations and Future Directions
One limitation of the current study is the lack of diversity in the sample. As noted earlier, the sample in this study is entirely White and Asian/Asian American. Studies using convenience samples in non-diverse contexts can reveal basic processes and intervention mechanisms that may apply to all students (Harackiewicz & Priniski, in press; Schwartz, Cheng, Salehi, & Wieman, 2016). More critically, it is important to test whether the results on retention outcomes found here hold for other, more diverse populations. Previous research has shown that the UV intervention, at full dosage (i.e., three per semester), can help close achievement gaps for underrepresented minority and first-generation college students (Harackiewicz, Canning, et al., 2016), suggesting that the UV intervention might provide a powerful tool for increasing representation of underrepresented ethnic minority and first-generation college students in STEM. The results of current study, conducted in the same context, are consistent with previous research, in terms of course grades, and go beyond previous research by showing that these increases in performance can lead to increases in the likelihood of persistence in STEM. Harackiewicz and colleagues (2016) found that the UV intervention effects on course performance were strongest for underrepresented minority and first-generation college students, so it is possible that these groups would also exhibit the strongest effects on persistence. This possibility will need to be tested with a large, diverse sample.
Another limitation of this study is that the timing of the writing assignments is confounded with the topic of the unit in which the writing assignment was implemented. The UV writing assignments were administered once for each unit of the course (cellular biology, genetics, and evolutionary biology). Therefore, it is unclear if the effects of timing we found are due to the unit in which the intervention was implemented or the timing of the intervention within the semester. For instance, it could be that writing about evolutionary biology in itself was beneficial for students with high prior performance, or as we hypothesize, it could be that the third writing assignment, regardless of its topic, was beneficial for these students due to its timing. In order to tease apart timing and topic, the unit of the course would need to be counterbalanced, which is typically not feasible in a large introductory course. This is an area for future research.
In addition, it will be important in future research to explore the underlying psychological changes and behaviors that occur as a result of this intervention that were not possible to measure in the current study. Future studies should include process measures directly after each implementation of the utility value essays. It is possible that students may experience an increase in perceived utility value, interest, or confidence as a result of the intervention. To examine the mechanism, process measures need to be collected directly after each essay assignment and before course grades are determined, which was not feasible in the current study. These issues will need to be addressed further in laboratory studies where perceptions can be measured closer in time to the intervention. Moreover, future researchers should consider other moderating variables of the intervention, such as literacy and writing skills and the ability to articulate utility value. As implemented in the current study, the UV writing assignments require a high level of conceptual work and writing skills, which might prove more challenging for students with less academic preparation. It is unknown whether the ability to articulate utility value is an integral part of the intervention or whether simply completing the assignment thoughtfully is sufficient for its impact. This question should be addressed in future research.
Finally, this study only measured students' intentions to major in STEM as a preliminary indicator of persistence, not actual degree completion. Our results suggest that the UV intervention made it more likely that students would continue majoring in STEM fields at the end of a foundational STEM course; however, a follow-up study is warranted to determine if students remained in STEM fields 2-3 years later and through graduation. By intervening with students early in their STEM training, it may be possible to keep them on track for completing a degree in STEM. Continuation within a foundational program of introductory STEM courses is just one of many steps required to improve STEM retention.
Conclusion
Retaining students in STEM fields is a national priority. Our research suggests that a UV writing intervention can increase performance, STEM course-taking, and intentions to major in STEM fields. By helping students appreciate the relevance and utility value of course material, we create the potential to improve STEM retention rates and increase the number of STEM professionals. Consistent with a growing body of research (Gasiewski et al., 2012; Ost, 2010; Seymour & Hewitt, 1997), we found that a positive experience in a single introductory course can have downstream implications for student persistence in STEM majors.
Educational Impact And Implications Statement.
In the current study, we manipulated implementation features of a utility-value (UV) intervention (dosage and timing), in which students wrote about the personal relevance and usefulness of course topics in an introductory biology sequence for biomedical majors. We found that any exposure to the UV intervention increased course performance, continuation to the second course of the biology sequence, and persistence within a STEM major. We compared one, two, and three doses of the intervention and found that students who received one or three doses benefitted the most from the intervention. Furthermore, in terms of timing, students with a history of poor performance earned higher grades when they received a single dose of the intervention early in the semester, whereas higher performing students were more likely to continue to the second course if they wrote a UV essay at the end of the semester. This study addresses key implementation questions for educators who use this intervention in their own courses and is the first study to suggest that the positive effects of UV interventions may extend beyond a single semester.
Acknowledgments
This research was supported by the National Institutes of Health (Grant R01GM102703). Elizabeth Canning was supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-1256259. Elizabeth Canning, Stacy Priniski, Cameron Hecht, and Yoi Tibbetts were supported by grants from the Institute of Education Sciences, U.S. Department of Education, through Award # R305B090009 and #R305B150003 to the University of Wisconsin-Madison. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education or the National Institutes of Health. We thank Kerstin Krautbauer; course coordinators Jean Heitz, Kerry Martin, Brian Parks, and Carlos Peralta; and all the research assistants who helped to implement this study.
Footnotes
There were 241 other students enrolled in this course who were not eligible to participate in this study: 161 first-generation (FG) majority college students, 50 underrepresented minority (URM) students, and 30 students who were both FG and URM. These students participated in a different study, with a different experimental design, intended to test the efficacy of the UV intervention at full dosage to close achievement gaps (Harackiewicz, Canning, et al., 2016). It is important to note that all students in this class were treated identically (same assignments, grading structure, and course content); the only difference between the sample presented here and the one in Harackiewicz, Canning, et al. (2016) is which analytic design the students were assigned to. Harackiewicz, Canning, et al. (2016) employed a 2-cell UV intervention design across 4 semesters and the current study employed an 8-cell UV design in a single semester. Harackiewicz, Canning et al. (2016) found that the UV intervention improved grades for FG-URM students. In that study, the UV intervention was crossed with a values-affirmation intervention, but there were no significant effects of the VA intervention. In the current study, an 8-cell UV intervention design was crossed with the same VA intervention (treatment vs. control), resulting in a 16-cell design. Full methodological details regarding implementation and testing of the VA intervention in this class are reported by Harackiewicz, Canning, et al. (2014) and Harackiewicz, Canning, et al. (2016). However, there were no significant effects of the VA intervention or any interactions with the UV intervention in the present study, and we therefore collapsed across VA condition for all analyses reported here. In addition, we tested the efficacy of the UV intervention in the control condition of the VA intervention, and found comparable effects of the UV intervention on grades, continuation, and STEM major persistence.
We also tested whether articulated value differed between the two types of UV assignment (i.e., when students chose to write about utility for themselves vs. for others) on the first essay. We found that students' assignments contained somewhat higher-quality utility-value connections when they wrote about utility value for others than when they wrote about utility value for themselves, p = .001. However, the difference between the two UV assignments (.43) was smaller than the difference between UV assignments and control assignments (2.57).
In a separate analysis, we compared the three UV conditions in which students wrote two UV assignments (UV first & second, UV first & third, UV second & third), however no main effects or interactions with prior GPA were significant.
Contributor Information
Elizabeth A. Canning, Indiana University
Judith M. Harackiewicz, University of Wisconsin-Madison
Stacy J. Priniski, University of Wisconsin-Madison
Cameron A. Hecht, University of Wisconsin-Madison
Yoi Tibbetts, University of Wisconsin-Madison.
Janet S. Hyde, University of Wisconsin-Madison
References
- Bettinger E. To be or not to be: Major choices in budding scientists. In: Clotfelter CT, editor. American Universities in a Global Market. Chicago: University of Chicago Press; 2010. pp. 68–98. [Google Scholar]
- Canning EA, Harackiewicz JM. Teach it, don't preach it: The differential effects of directly-communicated and self-generated utility–value information. Motivation Science. 2015;1(1):47–71. doi: 10.1037/mot0000015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chen X, Soldner M. STEM Attrition: College students' paths into and out of STEM fields statistical analysis report. NCES 2014-001. National Center for Education Statistics. 2013 Retrieved from http://eric.ed.gov/?id=ED544470.
- Cook JE, Purdie-Vaughns V, Garcia J, Cohen GL. Chronic threat and contingent belonging: Protective benefits of values affirmation on identity development. Journal of Personality and Social Psychology. 2012;102(3):479–496. doi: 10.1037/a0026312. [DOI] [PubMed] [Google Scholar]
- Eccles J. Who am I and what am I going to do with my life? Personal and collective identities as motivators of action. Educational Psychologist. 2009;44(2):78–89. doi: 10.1080/00461520902832368. [DOI] [Google Scholar]
- Eccles J, Adler TF, Futterman R, Goff SB, Kaczala CM, Meece JL, Midgley C. Expectations, values and academic behaviors. In: Spence JT, editor. Perspective on achievement and achievement motivation. San Francisco, CA: Freeman; 1983. pp. 75–146. [Google Scholar]
- Eccles J, Wigfield A. Motivational beliefs, values, and goals. Annual Review of Psychology. 2002;53(1):109–132. doi: 10.1146/annurev.psych.53.100901.135153. [DOI] [PubMed] [Google Scholar]
- Fredricks JA, Eccles JS. Children's competence and value beliefs from childhood through adolescence: Growth trajectories in two male-sex-typed domains. Developmental Psychology. 2002;38(4):519–533. doi: 10.1037/0012-1649.38.4.519. [DOI] [PubMed] [Google Scholar]
- Gasiewski JA, Eagan MK, Garcia GA, Hurtado S, Chang MJ. From gatekeeping to engagement: A multicontextual, mixed method study of student academic engagement in introductory STEM courses. Research in Higher Education. 2012;53(2):229–261. doi: 10.1007/s11162-011-9247-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gaspard H, Dicke AL, Flunger B, Brisson BM, Hafner I, Nagengast B, Trautwein U. Fostering adolescents' value beliefs for mathematics with a relevance intervention in the classroom. Developmental Psychology. 2015;51(9):1226–1240. doi: 10.1037/dev0000028. [DOI] [PubMed] [Google Scholar]
- Gaspard H, Dicke AL, Flunger B, Schreier B, Häfner I, Trautwein U, Nagengast B. More value through greater differentiation: Gender differences in value beliefs about math. Journal of Educational Psychology. 2015;107(3):663–677. doi: 10.1037/edu0000003. [DOI] [Google Scholar]
- Goulden M, Frasch K, Mason MA. Staying competitive: Patching America's leaky pipeline in the sciences. Berkeley: University of California, Berkeley Center on Health, Economic, & Family Security and the Center for American Progress; 2009. [Google Scholar]
- Harackiewicz JM, Canning EA, Tibbetts Y, Giffen CJ, Blair SS, Rouse DI, Hyde JS. Closing the social class achievement gap for first-generation students in undergraduate biology. Journal of Educational Psychology. 2014;106(2):375–389. doi: 10.1037/a0034679. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Harackiewicz JM, Canning EA, Tibbetts Y, Priniski SJ, Hyde JS. Closing achievement gaps with a utility-value intervention: Disentangling race and social class. Journal of Personality and Social Psychology. 2016;111(5):745–765. doi: 10.1037/pspp0000075. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Harackiewicz JM, Durik AM, Barron KE, Linnenbrink-Garcia L, Tauer JM. The role of achievement goals in the development of interest: Reciprocal relations between achievement goals, interest, and performance. Journal of Educational Psychology. 2008;100(1):105–122. doi: 10.1037/0022-0663.100.1.105. [DOI] [Google Scholar]
- Harackiewicz JM, Priniski SJ. Improving student outcomes in higher education: The science of targeted intervention. Annual Review of Psychology. doi: 10.1146/annurev-psych-122216-011725. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Harackiewicz JM, Smith JL, Priniski SJ. Interest matters: The importance of promoting interest in education. Policy Insights from the Behavioral and Brain Sciences. 2016;3(2):220–227. doi: 10.1177/2372732216655542. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Harackiewicz JM, Tibbetts Y, Canning EA, Hyde JS. Harnessing values to promote motivation in education. In: Karabenick S, Urden T, editors. Motivational interventions, advances in motivation and achievement. Vol. 18. Bingley, UK: Emerald Group PublishingBingley, UK; 2014. pp. 71–105. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hayes AF. Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. Guilford Press; 2013. [Google Scholar]
- Higher Education Research Institute. Degrees of success: Bachelor's degree completion rates among initial STEM majors. Los Angeles: Author; 2010. [Google Scholar]
- Hulleman CS, Durik AM, Schweigert SB, Harackiewicz JM. Task values, achievement goals, and interest: An integrative analysis. Journal of Educational Psychology. 2008;100(2):398–416. doi: 10.1037/0022-0663.100.2.398. [DOI] [Google Scholar]
- Hulleman CS, Godes O, Hendricks BL, Harackiewicz JM. Enhancing interest and performance with a utility value intervention. Journal of Educational Psychology. 2010;102(4):880–895. doi: 10.1037/a0019506. [DOI] [Google Scholar]
- Hulleman CS, Harackiewicz JM. Promoting interest and performance in high school science classes. Science. 2009;326(5958):1410–1412. doi: 10.1126/science.1177067. [DOI] [PubMed] [Google Scholar]
- Hulleman CS, Kosovich JJ, Barron KE, Daniel DB. Making connections: Replicating and extending the utility value intervention in the classroom. Journal of Educational Psychology. 2016 doi: 10.1037/edu0000146. [DOI]
- Jacobs JE, Lanza S, Osgood DW, Eccles JS, Wigfield A. Changes in children's self-competence and values: Gender and domain differences across grades one through twelve. Child Development. 2002;73(2):509–527. doi: 10.1111/1467-8624.00421. [DOI] [PubMed] [Google Scholar]
- Nagengast B, Marsh HW, Scalas LF, Xu MK, Hau KT, Trautwein U. Who took the “×” out of expectancy-value theory? A psychological mystery, a substantive-methodological synergy, and a cross-national generalization. Psychological Science. 2011;22(8):1058–1066. doi: 10.1177/0956797611415540. [DOI] [PubMed] [Google Scholar]
- OECD. OECD skills outlook 2013: First results from the survey of adult skills. OECD Publishing; 2013. [Google Scholar]
- OECD. Education indicators in focus. OECD Publishing; 2015. [Google Scholar]
- Ost B. The role of peers and grades in determining major persistence in the sciences. Economics of Education Review. 2010;29(6):923–934. doi: 10.1016/j.econedurev.2010.06.011. [DOI] [Google Scholar]
- President's Council of Advisors on Science and Technology (PCAST) Engage to excel: Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Washington, DC: 2012. [Google Scholar]
- Priniski SJ, Hecht CA, Harackiewicz JM. Making learning personally meaningful: A new framework for relevance research. Journal of Experimental Education. doi: 10.1080/00220973.2017.1380589. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Renninger KA, Hidi S. Revisiting the conceptualization, measurement, and generation of interest. Educational Psychologist. 2011;46(3):168–184. doi: 10.1080/00461520.2011.587723. [DOI] [Google Scholar]
- Rubin DB. Multiple imputation for nonresponse in surveys. New York, NY: Wiley; 1987. Retrieved from . [DOI] [Google Scholar]
- Schiefele U. Interest, learning, and motivation. Educational Psychologist. 1991;26:299–323. [Google Scholar]
- Schwartz DL, Cheng KM, Salehi S, Wieman C. The half empty question for socio-cognitive interventions. Journal of Educational Psychology. 2016;108(3):397–404. doi: 10.1037/edu0000122. [DOI] [Google Scholar]
- Seymour E, Hewitt NM. Talking about leaving: Why undergraduates leave the sciences. Boulder, CO: Westview Press; 1997. [Google Scholar]
- Strenta AC, Elliott R, Adair R, Matier M, Scott J. Choosing and leaving science in highly selective institutions. Research in Higher Education. 1994;35(5):513–547. [Google Scholar]
- Tibbetts Y, Harackiewicz JM, Priniski SJ, Canning EA. Broadening participation in the life sciences with social–psychological interventions. CBE-Life Sciences Education. 2016;15(3):es4. doi: 10.1187/cbe.16-01-0001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Trautwein U, Marsh HW, Nagengast B, Lüdtke O, Nagy G, Jonkmann K. Probing for the multiplicative term in modern expectancy–value theory: A latent interaction modeling study. Journal of Educational Psychology. 2012;104(3):763–777. doi: 10.1037/a0027470. [DOI] [Google Scholar]
- Watkins J, Mazur E. Retaining students in science, technology, engineering, and mathematics (STEM) majors. Journal of College Science Teaching. 2013;42(5):36–41. [Google Scholar]
- Watt HMG. Development of adolescents' self-perceptions, values, and task perceptions according to gender and domain in 7th- through 11th-grade Australian students. Child Development. 2004;75(5):1556–1574. doi: 10.1111/j.1467-8624.2004.00757.x. [DOI] [PubMed] [Google Scholar]
- Wigfield A, Cambria J. Students' achievement values, goal orientations, and interest: Definitions, development, and relations to achievement outcomes. Developmental Review. 2010;30(1):1–35. doi: 10.1016/j.dr.2009.12.001. [DOI] [Google Scholar]
- Yeager DS, Walton GM. Social-psychological interventions in education: They're not magic. Review of Educational Research. 2011;81(2):267–301. doi: 10.3102/0034654311405999. [DOI] [Google Scholar]