Student Attitudes Contribute to the Effectiveness of a Genomics CURE

ABSTRACT

The Genomics Education Partnership (GEP) engages students in a course-based undergraduate research experience (CURE). To better understand the student attributes that support success in this CURE, we asked students about their attitudes using previously published scales that measure epistemic beliefs about work and science, interest in science, and grit. We found, in general, that the attitudes students bring with them into the classroom contribute to two outcome measures, namely, learning as assessed by a pre- and postquiz and perceived self-reported benefits. While the GEP CURE produces positive outcomes overall, the students with more positive attitudes toward science, particularly with respect to epistemic beliefs, showed greater gains. The findings indicate the importance of a student’s epistemic beliefs to achieving positive learning outcomes.

INTRODUCTION

The Genomics Education Partnership (GEP) provides students with a course-based undergraduate research experience (CURE) (1). A growing body of literature describes the features of successful CUREs. For example, Auchincloss and colleagues outlined a structure for a CURE that includes the use of scientific practices, opportunities for discovery, broadly relevant or important work, collaboration, and iteration (2). Linn and colleagues added that CURE instructors should guide students in developing scientific practices while helping them expand their content knowledge and supporting them as they develop their own identities as scientists (3). Evidence suggests that CUREs provide a number of benefits, including increasing retention of students in science, technology, engineering, and mathematics (STEM) (4). It has been suggested that CUREs might also create more inclusive environments (5). In support of this notion, Hanauer et al. provided evidence that students involved in the Science Education Alliance program SEA-PHAGES show “gains correlated with persistence relative to those in traditional laboratory courses regardless of academic, ethnic, gender, and socioeconomic profiles” (6). Perhaps equally important, CUREs welcome students into the STEM community by engaging them in scientific research.

Students in the GEP are engaged in the careful annotation of groups of genes important for a biological function or system, working with recently sequenced organisms whose genomes have not yet been examined in detail. Pertinent genome data are displayed on a custom version of the University of California Santa Cruz Browser, including evidence of homology compared to a well-annotated reference species, results from ab initio gene finders, RNA-sequencing (RNA-seq) data, and any other available data. Students are responsible for verifying the presence of a functional gene, creating a model of the gene’s intron/exon structure, and, in some cases, also inferring the likely transcription start site. Additional explorations can include a search for functional annotation, analysis of repetitive sequences in the area, or synteny (the order of genes along a chromosome) compared to the reference. While the various lines of evidence generally agree as to whether a gene is present, they are often contradictory in the details; students, in resolving these issues, learn that while there is no “correct answer,” there is a defensible answer (a gene model) based on the available evidence. Each gene is analyzed independently by two or more students, and the results are reconciled by experienced students and pooled for use in a study establishing the pattern of evolution of that group of genes, addressing questions of biological interest. Thus, students are exposed to the scientific process, have opportunities for discovery, participate in broadly relevant work (as demonstrated by the student authors on GEP papers in the scientific literature [7, 8]), have opportunities for collaboration both within the classroom and across the network, and can iterate their work, meeting the definitions of a CURE outlined by Auchincloss et al. (2). For additional details on the mechanics of managing the collaborative structure of the GEP, see refs. 9 and 10. Annual assessment since 2006 demonstrates that GEP students not only learn about genes and genomes (as measured by a pre- and postquiz) but also gain a greater understanding of how science is done and feel that they have improved their requisite skills, as shown by a pre- and postsurvey (9, 10). In the present study, we use these same two outcome measures.

There is now significant literature reporting that students benefit from a research experience and that such experiences increase both retention in STEM majors and graduation rates (4, 5). Because most traditional research laboratory experiences are built on a mentor-and-apprentice model, there is limited capacity for large numbers of students to gain research experience in this fashion (11). Further, it has been shown that minority, first-in-family, and low-income students often fail to seek out such opportunities because they lack awareness of the benefits of research (5). CUREs provide one solution to the need to provide research experiences for all STEM students and are growing in popularity. In this study, we sought to better understand how students’ attitudes influence their success in a CURE setting.

In How People Learn (12), the editors, drawing from the developmental work of Piaget and Vygotsky, assert that a student’s preexisting knowledge contributes to learning as much as faculty pedagogy does. Preexisting knowledge includes “a range of prior knowledge, skills, beliefs, and concepts that significantly influence what they notice about the environment and how they organize and interpret it.” Students “construct new knowledge and understandings based on what they already know and believe.” Individual differences in prior beliefs include a positive or negative attitude toward science, an interest in science or a science career, and the tendency to persist at a task. Some researchers have explored individual differences by observing background variables, such as gender, age, first in family to attend college, etc., as ways to stratify results. For example, Hanauer et al. (6) examined the impacts of the SEA-PHAGES program (a CURE offered to first-year students) across a variety of subject variables, including prior academic success (i.e., grades), ethnicity, gender, and socioeconomic status. They found that a diverse range of students showed comparable gains on measures of science learning, persistence, and identity. Looking at similar variables, Rodenbusch et al. (4), in summarizing the results of the Freshman Research Initiative (FRI) program at the University of Texas, found that participating students showed universal gains in the probability of graduating with a STEM degree compared to nonparticipating students, noting “the outcomes of participating in the full FRI program were the same regardless of students’ gender, race/ethnicity, and first-generation in college status.” Similarly, the GEP reported that the institutional characteristics of its member schools showed no impact on student outcomes (10). These findings are reassuring from the perspective that an undergraduate research experience benefits students from diverse backgrounds (13). It therefore appears that a variety of student demographic characteristics are compatible with student success in a CURE. However, individual student attitudes may also influence success. Some studies have examined ownership of research work using the Project Ownership Survey (14), while others have asked about student perceptions of biology laboratories using the Laboratory Course Assessment Survey (15). In this study, we sought to explore the impact of student attitudes on our two outcomes measures, quiz scores and self-reported benefits, using a variety of existing tools, as described below.

A recent review of articles defining and measuring interest in biology notes that “as a construct…interest is complex” (16). Hidi and Renninger (17) distinguish between situational interest, evoked by activities that capture the interest of many individuals and that may be transient, with personal interest, a relatively stable trait (see also reference 18). They propose a four-phase developmental model: initial situational interest, maintained situational interest, emerging personal interest, and well-developed personal interest (17). This model can help categorize the variety of results published in the CURE outcomes literature. Initial interest may result from a first experience with research (19, 20). The middle phases of interest may describe the increased interest in research activities or the persistence in pursuing science within the undergraduate experience (21, 22). Finally, development of personal interest may be related to the pursuit of a STEM graduate degree and/or a STEM career (4, 23).

To assess the link between student attitudes and our CURE’s effectiveness, we used a battery of self-reported measures with respect to attitudes about science. There are few scales that have been described to assess interest as a factor in learning. As one aim of the study, we wanted to determine which scales were most useful for exploring the link between interest and learning. First, we included two published scales that measure positive and negative perceptions of science, comprising items related to feelings of personal agency and epistemological conceptions about science, originally developed by Wenk (24, 25). The personal agency that results in productive engagement with science is expressed by agreement with positive statements, such as “I can do well in science courses” and “Even if I forget the facts, I’ll still be able to use the thinking skills I learn in science.” Conversely, misconceptions regarding the nature of scientific thought and process are expressed by agreement with negative statements, such as “Creativity does not play a role in science” and “If an experiment shows that something does not work, the experiment was a failure.” We hypothesize that student interest grows as a result of success in science courses and other experiences with science, while student interest is dampened by misconceptions. Second, we adapted a published survey of student interests, the Test of Science-Related Attitudes (TOSRA). The TOSRA survey is an instrument that includes scales measuring the respondent’s enjoyment of science, interest in science beyond the classroom, and career interest in science (26–28). Third, the Grit scale (29, 30) aims to measure persistence and determination. Duckworth et al. (30) define grit as “perseverance and passion for long-term goals.” Persistence, while not strictly correlated with interest in science per se, has been identified as a contributing variable to success in a science program or career (31).

Here, we asked students to report their attitudes toward science, their interest in science, and their persistence using the above measures and then grouped their responses to make comparisons to two outcome measures, a subject matter quiz and a self-reported benefits survey. We hypothesized that positive attitudes toward science enhance performance on outcome measures and negative attitudes constrain outcomes, that higher interest in science also supports better outcomes, that higher Grit scores enhance outcomes, and that taking part in the CURE experience changes attitudes. We found that scores on the positive and negative perceptions of science scales had the most predictive value, that, in general, more positive attitudes about science are correlated with better outcomes on the performance measures, and that student attitudes were unaffected by the GEP CURE.

MATERIALS AND METHODS

Information was gathered from GEP students over two academic years, 2015 to 2016 and 2016 to 2017. Other data from this same period have been used to analyze the role of student frustration, problem solving approaches, and iteration in the learning process (32).

GEP schools and faculty

The demographics of the participating schools and students are detailed in our recent paper (32). In brief, the GEP curriculum has been used widely, including in community colleges (7%), primarily undergraduate institutions (80%) and research universities (13%), and in majority- and minority-serving, small and large, and residential and commuter schools. A third of the participating schools are minority-serving institutions (>30% minority students). Faculty using GEP materials range from those participating in a short introduction to genome annotation (usually ~10 h of instruction and hands-on investigation) to those spending all or part of a semester immersed in this research experience (over 36 h in a given course). There were 54 institutions (with one faculty member per institution) in the data set represented here.

Student data collection

All participating students were asked to complete a voluntary course survey and quiz before and after using GEP materials. After informed consent was obtained, students could opt out of any or all questions. Approval to conduct assessment for scholarly purposes was obtained by each participating GEP faculty member from their Institutional Review Board. Confidentiality was maintained by using encryption to eliminate identification of individual students. These unidentified responses were aggregated at Washington University in St. Louis and made available for analysis to D.L. The sections of the student surveys used for this study are described below and in Supplement 1 in the supplemental material.

Demographic information was collected from students in the precourse survey (described in detail in reference 32). Of an estimated 3,300 students eligible, 2,115 students participated (64%); within the participating group, women (61%), Pell-grant eligible (42%), first-in-family to college (23%), and minority (34%) students were well represented (Table S1). The students in the current study were mostly upper-class science majors (41% seniors, 30% juniors, 19% sophomores, and 10% first years). We received 2,037 complete precourse surveys and 1,307 complete postcourse surveys; 1,903 students took the precourse quiz, and 1,272 students completed the postcourse quiz. Data loss due to students missing the precourse quiz or the postcourse quiz or using incorrect or no identification resulted in a smaller number of matched data sets (“complete samples”) than overall participation (~700 complete samples; Table S2). Only complete samples were used in the analysis shown here.

Outcome measures

The first outcome measure was a subject matter quiz on the process of gene annotation and eukaryotic gene structure. Students were asked to complete a quiz of 20 multiple choice items both before and after instruction to determine if the students exhibited learning gains as a result of participating in the GEP research project. Questions assess a range of skills, from mastery of basic terminology and concepts to more complex cognitive skills, including data analysis and evaluation. The quiz was created and validated by a number of subject matter experts. Earlier iterations of the quiz have been shown to differentiate between genomics students and comparison groups of students and captured significant score increments from pre- to postquiz (9, 10). Prequiz sensitization effects were avoided by using two equivalent quizzes. Students were assigned to either quiz at random for the prequiz and the other for the postquiz. We continue to use these quizzes, so copies have not been included. In the current study, as with past work, the students in aggregate have higher scores on the postquiz than on the prequiz.

The second outcome measure consisted of self-reported benefits of the genomics experience. These items appeared in the postcourse survey only. Twenty of the self-reported benefit items are of special interest. These items, highlighted in orange in the supplemental materials (Supplement 1) are occasionally characterized as the Survey of Undergraduate Research Experiences (SURE) items (9, 10), as they were first introduced in a study of primarily summer undergraduate research experiences (33–35). Students in summer research experiences and the GEP CURE have shown similar scores over the years. The items reflect the benefits of an authentic research experience as described by experts (36). They have been shown to have internal consistency, to differentiate courses with CUREs from comparison courses (1), and to increase with increased instructional time (10). The self-reported benefits are each scaled 1 to 5, with 1 meaning little or no gain and 5 meaning very large gain. An average score (total scores divided by 20) was calculated for each respondent.

Attitude measures

All attitude measures appeared on both the precourse and the postcourse surveys.

Positive and negative perceptions. The survey contains items to assess positive perceptions (personal agency to do science) and negative perceptions (epistemological misperceptions) of science originally developed by Wenk (25) and modified by Lopatto (35) (Supplement 1, positive statements in green and negative statements in purple). These items first appeared in the Classroom Undergraduate Research Experiences survey (35) and have since been analyzed in depth (37). Each item on these perception scales was evaluated on a 1 (strongly disagree) to 5 (strongly agree) scale. Total scores summed from the five items constituting positive perceptions and total scores from the six items constituting negative perceptions were used for analysis.

Test of Science-Related Attitudes (TOSRA). The survey contains selected items from the TOSRA questionnaire (26–28), using elements from three of the question sets available. These are “Interest in Science Beyond the Classroom” (interest; including items such as “I do or would like to belong to a science-related club” and “I dislike reading books and online articles about science in my leisure time”), “Enjoyment of Science” (enjoyment; including items such as “Science courses are fun” and “Science courses bore me”), and “Career Interest in Science” (career; including items such as “I am planning to seek a job in science, genomics, medicine, or engineering” and “I would dislike a job in a science laboratory”; Supplement 1, text highlighted in yellow, blue, or pink, accordingly). The original TOSRA items were adapted for the present study as shown in Supplement 2. The items were evaluated on a scale of 1 (strongly disagree) to 5 (strongly agree). Items that expressed a negative opinion were reverse scored. Total scores for each category were summed and used for analysis.

Grit scale. We used a popular 12-item Grit scale that included items such as “Setbacks do not discourage me” and “I am a hard worker” (Supplement 1, text in red). Responses were scored on a scale from 1 (“not like me at all”) to 5 (“very much like me”). Negatively phrased items were reverse scored. Average scores (totals divided by 12) were used for analysis.

Plan for analysis

Data collection resulted in two outcome measures, annotation quizzes and self-reported benefits. The annotation quiz followed a pretest-posttest plan; changes in scores were evaluated with a related-groups t test with a significance level of 0.05. The self-reported benefits were posttest only.

The measures of student attitudes, positive perception of science, negative perception of science, TOSRA, and Grit, followed a pretest-posttest plan. For each measure, we examined the interitem reliability of the scale items as well as the correlations between measures. Each measure was tested with related-groups t tests to explore pretest-posttest differences.

The impact of student attitudes on the two outcome measures was evaluated by creating grouping variables to permit inferential testing on the two outcomes. We divided the student data into groups defined by the attitude measures. For positive and negative perceptions of science and grit, we created four quartiles for each and evaluated the outcome measures relative to these four groups of students. For the TOSRA scores, Fraser suggested that scores from the scales could be combined to create a profile for each student (27). We therefore calculated a median for each of the three scales (enjoyment, interest, and career) and assigned a code of “1” to an individual’s score below the median and a code of “2” to a score above the median. These assigned codes were then summed, yielding a profile code ranging from 3 (below the median on all three scales) to 6 (above the median on all three scales) for each respondent; the profile code was then used to create four groups (i.e., profile scores of 3, 4, 5, and 6). Following the creation of the groups, the impact of the attitude measures on the two outcomes was evaluated with a series of one-way between-groups analysis of variance (ANOVA) tests. The results of these analyses and relevant post hoc tests are presented in the figure captions for the convenience of the reader.

RESULTS

To examine the students’ contribution to effectiveness of a genomics CURE, we used two outcome measures, quiz scores and self-reported benefits, to examine student gains. These outcome measures have been extensively described in previous GEP publications (9, 10, 32). We compared these outcomes to student attitudes using several different scales previously described in the literature, including positive and negative perceptions of science, TOSRA, and Grit. Outcome measures from these data sets were similar to those previously observed (1, 10). A comparison of the precourse and postcourse quiz scores showed that the postcourse mean (mean = 9.7; standard deviation [SD] = 3.6) was significantly higher than the precourse mean (mean = 6.6; SD = 3.1; paired difference t = 24.9, degrees of freedom [df] = 703, P < 0.01). Approximately 80% of the students, scored higher on the postquiz than on the prequiz, reflecting a general pattern of improvement. The quiz means resemble those of previous studies (Fig. 3 in reference 10). The second outcome measure, the self-reported benefits, also resembled the pattern of previous studies (Fig. 1 in reference 1 and Fig. 1 in reference 10). We also confirmed that students in this study who spent more time on the GEP project had higher outcomes, as was previously observed (10) (data not shown).

FIG 3.

TOSRA scores differentiate outcome measures. The groups used here are based on scores from three TOSRA scales, Enjoyment of Science (“enjoyment,” median score of 21 of 25 possible points), Interest in Science Beyond the Classroom (“interest,” median score of 19 of 25 possible points), and Career Interest in Science (“career,” median score of 28 of 35 possible points). To generate a student TOSRA profile, in each case a student receives a 1 for a score below the median or a 2 for a score above the median. Thus, the profile group labeled 3 scored lower than the median on all three scales, while the profile group labeled 6 scored higher than the median on all three scales. The postcourse TOSRA profiles were used to analyze the postcourse quiz scores (A) and the postcourse mean self-reported benefits (B). Finally, mean scores from the two extreme TOSRA profile groups were compared to each other with respect to the individual items on the survey of self-reported benefits (C). (A) A one-way between-groups ANOVA test was performed using the quiz scores grouped by TOSRA profiles. The result yielded a significant difference between groups (F = 17.6; df = 3, 574; P < 0.001; r² = 0.08). A Tukey HSD test for pairwise differences indicated that the mean postcourse quiz scores for the students with the three higher TOSRA profile scores (4, 5, and 6) were all significantly higher than the mean postcourse quiz scores for students with the lowest TOSRA profile score (P < 0.05) but not different from each other. Error bars represent 2 SEM. (B) A one-way between-groups ANOVA test was performed using the self-reported benefit means grouped by TOSRA profiles. The result yielded a significant difference between groups (F = 5.1; df = 3, 629; P < 0.003; r² = 0.02). A Tukey HSD for pairwise comparisons indicates that the mean self-reported benefits for students with a profile score of 6 were significantly higher than those for students with a profile score of 3 or 4 (P < 0.05) but not significantly higher than the mean for students with a profile score of 5. Error bars represent 2 SEM. (C) Mean self-reported benefits for students who scored below the median on all three aspects of the postcourse TOSRA survey (resulting in an overall TOSRA score of 3) are shown as red triangles. Mean SURE scores for students who scored above the median on all three aspects of the TOSRA survey (resulting in an overall TOSRA score of 6) are shown as green diamonds. Higher self-reported benefits are reported by the latter group for most items on the survey.

FIG 1.

Positive perceptions of science scores differentiate outcome measures. (A and B) The postcourse positive perceptions of science scores were grouped into quartiles to examine the postcourse quiz scores (A) and the postcourse self-reported mean benefits derived from the student survey (B). For each figure, the x axis shows the four quartiles with the range of perceptions scores shown below the bars. (A) A one-way between-groups ANOVA test was performed using the mean quiz scores grouped by positive perceptions quartiles. The result yielded a significant difference between groups (F = 39.1; df = 3, 984; P < 0.001; r² = 0.10). Pairwise comparisons analyzed via the Tukey honestly significant difference (HSD) test showed that the mean quiz scores for students in quartiles 3 and 4 did not differ from each other, but both were significantly higher than quiz scores for students in quartiles 1 and 2. The mean quiz scores for quartile 2 were significantly higher than the mean quiz scores for quartile 1 (P < 0.05). Error bars represent 2 standard errors of the mean (SEM). (B) A one-way between-groups ANOVA test was performed using the mean self-reported benefits from the postcourse survey grouped by positive perceptions quartiles. The result yielded a significant difference between groups (F = 51.5; df = 3, 1,040; P < 0.001; r² = 0.13). Pairwise comparisons made with the Tukey HSD test indicated that the mean self-reported benefits for students in quartile 4 were significantly higher than those for students in the other three groups. The means for students in quartiles 3 and 2 were higher than those for quartile 1 students (P < 0.05), but groups 2 and 3 did not differ from each other. Error bars represent 2 SEM.

Positive and negative perceptions of science measures

To assess feelings of personal agency and epistemological conceptions about science, the positive and negative perceptions of science items were presented to students before and after their experience with the genomics CURE. Descriptive information for all of the scales is shown in Table 1. To explore the credibility of scales, we calculated Cronbach’s α, a metric of interitem consistency (Table 1). The values of α, all approximately 0.7 or greater, indicate an acceptable level of interitem consistency for the perceptions scales. Correlations between precourse and postcourse perceptions scales are shown in Table 2. Comparisons (related groups t tests) of the precourse data to the postcourse data indicated no significant increase for the positive perceptions scale (precourse mean = 21.6 and SD = 2.34 versus postcourse mean = 21.0 and SD = 3.11) or decrease for the negative perceptions scale (precourse mean = 13.7 and SD = 3.53 versus postcourse mean = 14.8 and SD = 4.56; Table 1). Thus, for these two measures, it appears that the GEP CURE did not influence positive or negative perceptions of science.

TABLE 1.

Interitem reliability of pre- and postexperience student measures of attitude and interest: Cronbach’s α score

Scalea	No. items	α b	Meanc	SD	Median	No. of observations
Precourse survey
Positive perceptions	5	0.71	21.6	2.34	22	2,083
Negative perceptions	6	0.68	13.7	3.53	13	1,989
TOSRA enjoyment	5	0.76	21.4	3.08	22	2,032
TOSRA interest	5	0.71	19.4	3.45	20	2,036
TOSRA career	7	0.80	29.2	4.66	30	2,012
Grit	12	0.79	3.6	0.52	3.7	1,785
Postcourse survey
Positive perceptions	5	0.84	21.0	3.11	21	1,342
Negative perceptions	6	0.82	14.8	4.56	14	1,329
TOSRA enjoyment	5	0.69	20.6	3.25	21	1,316
TOSRA interest	5	0.73	19.4	3.56	19	1,303
TOSRA career	7	0.73	27.9	4.8	28	1,286
Grit	12	0.78	3.6	0.54	3.6	1,220

^aThe perceptions of science scales are presented as the sum of the item scores (1 to 5 with negative items reversed). The TOSRA scales are presented as the sum of the item scores (1 to 5 with negative items reversed). Following common usage (30), the Grit scale is presented as an average of answers to each item (on a scale of 1 to 5 with negative items reversed).

^bCronbach’s α is a measure of interitem consistency, typically considered acceptable if the value exceeds 0.7 (43).

^cThere are no significant differences between preexperience and postexperience means for students involved in the GEP CURE for any of these measures. We note that student scores in the precourse survey data are quite high; for example, on the positive perceptions scale, the maximum score would be 25 (5 on each of the 5 Likert scale questions); thus, a mean of 21.6 is 87% of the maximum score.

TABLE 2.

Spearman rank-order correlations between measures of attitudes toward and interest in science for the precourse survey, postcourse survey, and precourse to postcourse survey data^a

Scale	Grit	TOSRA enjoyment	TOSRA interest	TOSRA career	Positive perceptions	Negative perceptions
Correlations between precourse survey scales
Grit	1	0.295	0.282	0.28	0.24	−0.17
TOSRA enjoyment		1	0.66	0.69	0.55	−0.43
TOSRA interest			1	0.68	0.48	−0.4
TOSRA career interest				1	0.48	−0.35
Positive perceptions					1	−0.41
Negative perceptions						1
Correlations between postcourse survey scales
Grit	1	0.29	0.33	0.32	0.27	−0.28
TOSRA enjoyment		1	0.67	0.76	0.5	−0.42
TOSRA interest			1	0.71	0.57	−0.45
TOSRA career interest				1	0.49	−0.41
Positive perceptions					1	−0.43
Negative perceptions						1
Pre- to postcourse correlationsb
Grit	0.68	0.21	0.22	0.13	0.18	−0.12
TOSRA enjoyment	0.13	0.61	0.47	0.42	0.32	−0.34
TOSRA interest	0.14	0.51	0.7	0.49	0.4	−0.37
TOSRA career interest	0.11	0.5	0.49	0.62	0.35	−0.33
Positive perceptions	0.13	0.37	0.42	0.32	0.52	−0.38
Negative perceptions	−0.1	−0.26	−0.3	−0.2	−0.28	0.62

^aEach correlation is significantly different from 0 at the P < 0.05 level of significance.

^bPrecourse surveys are the columns, and postcourse surveys are the rows.

TOSRA measures

To measure different aspects of student interest, we used a slightly modified TOSRA survey (26–28). The three scales used were “Interest in Science Beyond the Classroom” (interest), “Enjoyment of Science” (enjoyment), and “Career Interest in Science” (career). The items presented to the student are shown in Supplement 1 in the supplemental material. Fraser (27) reported that TOSRA scales are correlated with each other. Descriptive information about the interest scales is shown in Table 1. The three scales were correlated but not entirely redundant (Table 2). To investigate the credibility of the three scales, Cronbach’s α was calculated for each scale both pre- and postcourse (Table 1). The student responses on the precourse and the postcourse TOSRA survey questions yielded no significant changes in the three scales (Table 1). Thus, as for the perceptions of science scales, the values obtained before and after the CURE showed that enjoyment, interest, and career attitudes were unchanged.

Grit scale

To examine the effect of persistence, the 12-item Grit scale was presented to the students both before and after instruction. Descriptive information for the Grit data are shown in Table 1. An analysis of interitem consistency yielded acceptable values of Cronbach’s α (Table 1). The correlation between the precourse scores and the postcourse scores was significant (ρ = 0.68; Table 2). A comparison of precourse scores (mean = 3.6; SD = 0.52) with postcourse scores (mean = 3.6; SD = 0.54; Table 1) with a related-groups t test showed no statistically significant changes. Again, the GEP experience did not affect grit.

Evaluating the outcome measures with respect to the attitude measures

Next, we sought to assess the impact of student attitudes on the postcourse outcome measures. This task was complicated by features of the data that departed from the usual assumptions of parametric statistics. It might be expected that correlational procedures, such as the Pearson bivariate correlation coefficient or a multiple linear regression model, be applied to the relation between the outcomes and the attitude measures. Although the distributions of the two outcome measures were fairly symmetrical (Table S3), most of the attitude measures, with the exception of the Grit data, were skewed. The positive perceptions and three TOSRA scales each showed a negative (left-tailed) skew, while the negative perceptions measure showed a positive (right-tailed) skew. Furthermore, it might be argued that some of the attitude measures did not meet a strict definition of a measurement scale, that is, as an interval or ratio scale. The relations between measures were often monotonic but not linear, as would be expected for an analysis using parametric correlational techniques. These challenges to data analysis were met by simplifying the analysis and using nonparametric correlation coefficients. The results are shown in Fig. 1–4.

FIG 2.

Negative perceptions of science scores differentiate outcome measures. (A and B) The postcourse negative perceptions of science scores were grouped into quartiles to examine the postcourse quiz scores (A) and the postcourse mean self-reported benefits derived from the student survey (B). For each figure, the x axis shows the four quartiles with the range of negative perceptions scores shown below the bars. (A) A one-way between-groups ANOVA test was performed using the postcourse quiz scores grouped by negative perceptions quartiles. The result yielded a significant difference between groups (F = 54.0; df = 3, 955; P < 0.001; r² = 0.14). A Tukey HSD pairwise comparison analysis showed that the mean postcourse quiz score for students in quartile 4 (highest score for negative perceptions) was significantly lower than the mean quiz scores for each of the other three groups. The mean for quartile 3 students was significantly lower than the means of each of quartiles 2 and 1, which in turn differed from each other (P < 0.01). Error bars represent 2 SEM. (B) A one-way between-groups ANOVA test was performed using the self-reported benefits from the postcourse survey grouped by negative perceptions quartiles. The result yielded a significant difference between groups (F = 3.14; df = 3, 1,010; P < 0.05; r² = 0.01). Tukey HSD pairwise comparisons indicated that the mean self-reported benefits score for quartile 3 students was significantly lower than the mean self-reported benefits score for quartile 1 students (P < 0.05). Other differences were not significant. Error bars represent 2 SEM.

FIG 4.

Postcourse quiz scores are not differentiated by Grit score, while self-reported benefits are differentiated by Grit score using the postcourse survey data. (A and B) The postcourse Grit scores were divided into 4 quartiles to permit the comparisons of postcourse quiz scores (A) and comparisons of the postcourse self-reported benefits derived from the student survey (B). Following general practice, the Grit scores are represented as an average score per item (range 1 to 5). The range of Grit scores within a quartile is shown above the quartile label at the bottom of the figures. Error bars represent 2 SEM. (A) A one-way between-groups ANOVA test was performed using the quiz scores grouped by Grit quartiles. The analysis indicated no significant differences. (B) A one-way between-groups ANOVA test was performed using the mean self-reported benefits grouped by Grit quartiles. The result indicates a significant difference between groups (F = 12.4; df = 3, 842; P < 0.001; r² = 0.04). A Tukey HSD test for pairwise differences indicated that the students in the highest Grit quartile group had mean self-reported benefits scores significantly higher than those of the other three groups (P < 0.03). In addition, the mean for group 3 was higher than the mean for group 1 (P < 0.01). Error bars represent 2 SEM.

Figure 1A and B shows two outcome measures relative to the positive perception quartiles. In Fig. 1A, the two upper quartile groups yielded significantly higher quiz scores (means = 11.0 and 11.1 for quartile 3 [Q3] and Q4, respectively) than the lower two groups, and the lowest quartile group had significantly lower quiz scores (mean = 8.2) than the other quartile groups. In Fig. 1B, the highest quartile group reported significantly higher benefits (mean = 4.1) than any other group, while the lowest quartile group reported significantly lower benefits (mean = 3.3) than any other group. These results suggest that possessing a higher level of positive perceptions of science supports higher outcome measures following the genomics CURE.

Figure 2A and B illustrates how the outcome measures align with the negative perceptions quartiles. In Fig. 2A, the groups with higher levels of negative perceptions had significantly lower postcourse quiz scores. In Fig. 2B, the results for self-reported benefits are not orderly. Generally, these results suggest that reporting a high level of negative perceptions of science may correlate with learning while not affecting self-reported benefits.

Figure 3A to C illustrates how the two outcome measures align with the TOSRA profile created from the three TOSRA scales (enjoyment, interest, and career). Figure 3A shows the quiz score means for each profile group. The mean for students with a profile of 3 (mean = 8.4), in which students scored below the median on all three TOSRA scales, was significantly lower than the means of the other profile groups. The mean self-reported benefits, illustrated in Fig. 3B, showed a statistically significant difference between the lowest profile (a score of 3; mean = 3.5) and the highest profile (a score of 6; mean = 3.8). The differences in self-reported benefits between these two groups are shown in Fig. 3C. The figure shows the means for each of the 20 self-reported benefits, comparing the means for students with the lowest TOSRA profile (students with a score of 3) with those of the highest TOSRA profile (students with a score of 6), showing that the benefits for this group are consistently higher for most of the individual items. Taken together, the results indicate that a profile representing a higher interest in science correlates with greater success as measured by the outcomes.

Figure 4A and B shows how the two outcome measures map onto the four quartiles of the Grit scale. Figure 4A reveals an apparent difference between the lowest quartile (mean = 9.9) and the highest quartile (mean = 10.5); however, the difference is not statistically significant. Figure 4B shows the mean self-reported benefits for each quartile of the Grit scale. Here, the mean for quartile 4 (mean = 3.9) is significantly higher than the means of the other quartile groups. Thus, higher Grit scores align with self-reported benefits but not with quiz scores.

We had expected to collect the attitude measures into a multivariate model that would show the relative contributions of positive perceptions, negative perceptions, TOSRA, and Grit to predicting the outcome measures; however, the violations of the assumptions of parametric statistics and the ambiguity of the attitude scales (e.g., ordinal versus interval) suggested caution. Although we calculated the overall correlation between the attitude scores and the outcome measures (Table S4), we also explored the use of a partial correlation technique, that is, a correlation between an attitude measure and an outcome measure with the effects of other attitude measures removed based on the Spearman nonparametric correlation coefficient (38). When we calculated the correlations using the partial correlation technique, in this more restrictive instance, we found that the positive perceptions scale correlated with the quiz scores (ρ = 0.26, P < 0.05), the positive perceptions scale correlated with self-reported benefits (ρ = 0.28, P < 0.05), and the negative perceptions scale correlated with quiz scores (ρ = −0.27, P < 0.05). No other significant correlations were observed. Taken together, while the TOSRA and Grit scales show some correlation with student outcomes in some instances, the positive and negative perceptions scores are better at predicting student outcomes, showing that positive perceptions of science can enhance both student outcomes as measured by learning gains and perceived benefits.

DISCUSSION

Our investigation focused on student attitudes that may support the effectiveness of the GEP CURE and, by extension, other CUREs. Our analysis focused on the relation of the postcourse attitude measures to our two outcome measures, quiz scores and self-reported benefits, because no changes were observed in any of the attitude measures from the precourse to the postcourse. We found evidence for a relationship between each of the postcourse attitude measures and the outcome measures, but because these are correlated with each other, we looked at each individually. We conclude that the positive and negative perceptions of science measures show a relationship to the outcome measures when the effect of the other measures (TOSRA profile and Grit) are controlled through partial correlation.

We hypothesized that the GEP experience would alter student attitudes, but this was not the case (Tables 1 and 2). Because most life science students have self-selected for interest in science and the GEP project is implemented primarily in courses for science majors, this lack of effect may be due in part to “ceiling effects.” For example, an examination of the TOSRA scores on interest in science shows that the GEP students report ~80% of the maximum value on the precourse survey (Table 1) compared to general student populations, which report 50–65% of the maximum value (27, 39). While there is no standard benchmark for the TOSRA scales, GEP students display relatively high scores on science-related attitudes in the precourse survey, which do not significantly change as a result of the experience. In other words, to place our findings within the Hidi and Renninger model (17), GEP students tend to have a well-developed personal interest in science prior to their GEP experience.

The Grit scores were less informative for our study; they showed less correlation with the other attitudes studied here (Table 2), no change pre- and post-CURE (Table 2), and, with the exception of only those with the highest Grit scores (Fig. 4), no significant predictive power concerning student outcomes. Grit scores may correlate with the level of educational attainment when the age of the participants is controlled (29, 30), suggesting that greater ability to persevere on a task may facilitate educational success. Grit scores may also increase with age and experience. Indeed, we found that biology majors in their junior or senior year scored slightly higher on the Grit scale than less-experienced students (first-year and sophomore students; data not shown). However, Duckworth describes a developmental change in grit over a span of years rather than the short interval studied here (at most, a semester), so the observed lack of change is not surprising (29).

Of the attitude measures we used, we find the positive and negative perceptions of science (24, 25) to be the most promising outcome predictors, while the TOSRA and Grit scales were less informative and have been removed from more recent iterations of the pre- and postcourse surveys. Our data show that higher outcomes are associated with higher positive perceptions scores, suggesting that faculty would do well to explicitly address issues around personal agency. In particular, faculty should encourage students to articulate both the goals of the project and their individual goals and then encourage actions and behaviors that will help them achieve those goals (40). Further, our data also show that lower outcomes are associated with higher negative perceptions scores, again suggesting that faculty might explicitly discuss misconceptions about science and the scientific process with their students. This is likely the more difficult discussion, as data suggest that misconceptions about science are often entrenched and can be unaffected by current instruction (summarized in 41). Repeated discussions are therefore important in order to counteract inaccurate preconceptions. While our study is rooted in our specific genomics CURE, we suggest that addressing such issues will have effects beyond our CURE.

The students in this study were primarily junior and senior life science majors. Because these students are already highly interested in science, or, in other words, at the end of the process described by Hidi and Renninger (17), we see little change in these interest measures over the course of the GEP experience. This experience (one semester or less) may not be long enough to elicit further change but, in addition, suggests that student interest may have been solidified by earlier experiences before they encountered GEP. If this is the case, then it argues for earlier interventions, including access to research experiences, to foster interest, perhaps as early as middle or high school. In support of this idea, a small study by Harrison et al. found that first-year college students engaged in the SEA-PHAGES program expressed more interest in science careers and graduate education as a result of the experience (20). Alternatively, courses that span multiple semesters (including the SEA-PHAGES program [42] and the Freshman Research Initiative [4]) might also serve to foster and reinforce interest, although the earlier intervention and the longer intervention ideas are not mutually exclusive. We have also found that higher outcome measures are achieved when students spend more time on the GEP CURE (9, 10, 32).

In other work from GEP, we have found that students often encounter roadblocks when attempting to devise a workable gene model and, in so doing, experience frustration (32). Students who are supported by faculty, teaching assistants, and peers are more likely to persevere through the difficulties and ultimately succeed, a process we have termed formative frustration. Similarly, we have found that reminding students that they are engaged in a process that will provide new knowledge and that they are part of a larger community of researchers also correlates with higher outcome measures (Lopatto et al., unpublished data). In summary, optimizing student learning gains from a CURE may therefore involve the joint effects of agency, scientific thinking, formative frustration, and community support. Better outcomes might be achieved by interventions early in the course or earlier in a student’s career that either clarify the nature of science and scientific thought, emphasizing the process over the accumulation of facts, or reinforce a sense of agency for the individual student. These findings may be of utility for other efforts to engage students in research.