Computer-Based and Bench-Based Undergraduate Research Experiences Produce Similar Attitudinal Outcomes

Abstract

Course-based undergraduate research experiences (CUREs) have the potential to improve undergraduate biology education by involving large numbers of students in research. CUREs can take a variety of forms with different affordances and constraints, complicating the evaluation of design features that might contribute to successful outcomes. In this study, we compared students’ responses to three different research experiences offered within the same course. One of the research experiences involved purely computational work, whereas the other two offerings were bench-based research experiences. We found that students who participated in computer-based research reported at least as much interest in their research projects, a higher sense of achievement, and a higher level of satisfaction with the course compared with students who did bench-based research projects. In open-ended comments, similar proportions of students in each research area expressed some sense of project ownership as contributing positively to their course experiences. Their comments also supported the finding that experiencing a sense of achievement was a predictor of course satisfaction. We conclude that both computer-based and bench-based CUREs can have positive impacts on students’ attitudes. Development of more computer-based CUREs might allow larger numbers of students to benefit from participating in a research experience.

INTRODUCTION

Recent reports (American Association for the Advancement of Science, 2011; President’s Council of Advisors on Science and Technology [PCAST], 2012) recommend engaging students in meaningful scientific research, ideally early in their academic careers, as a way to promote a student’s science capacity, self-efficacy, and identity (Seymour et al., 2004; Lopatto, 2007; Russell et al., 2007). However, typical research apprenticeships, in which students work in a principal investigator’s lab, are not often available for all students—especially early in their undergraduate careers—a fact that may exacerbate barriers to full participation by underrepresented groups in science (e.g., women and individuals from certain minority groups; Bangera and Brownell, 2014). To address these challenges, many educators have turned to curricular solutions, in which scientific research is integrated into courses (PCAST, 2012; National Academies of Sciences, Engineering, and Medicine, 2015). In many cases, research projects become all or part of a laboratory experience, and students conduct discovery-based research that is broadly relevant (Auchincloss et al., 2014). A course-based undergraduate research experience (CURE) engages students in scientific practices via an investigation in which the outcome is unknown both to the students and the instructor (Auchincloss et al., 2014; Ballen et al., 2017). A growing body of evidence points to the utility of CUREs in increasing student motivation, confidence, interest, and retention in science disciplines (Olimpo et al., 2016; Rodenbusch et al., 2016; Corwin et al., 2018).

To date, CUREs have been developed that engage students in a variety of research experiences: bench-based experiences that require students to conduct experiments in a laboratory setting (e.g., Kowalski et al., 2016; Sarmah et al., 2016); field-based experiences in which students investigate research questions and collect data beyond the laboratory (e.g., Kloser et al., 2013; Thompson et al., 2016); or computer-based experiences in which students use computer programs to explore research questions with previously collected data (e.g., Shaffer et al., 2010; Brown, 2016). CUREs that combine two of these different types of experiences have also been implemented at several universities (e.g., Jordan et al., 2014).

While course designers may develop CUREs to meet specific objectives (Cooper et al., 2017), practical realities—such as money, space, and time—will dictate what is possible in a given curriculum. And, presumably, different types of CUREs have different affordances and constraints. For example, bench- and field-based CUREs may be perceived as more authentic than investigations involving previously collected data (“data mining”); if this perception of authenticity is key to generating successful outcomes (such as a student’s developing identity as a scientist), then computational investigations will be less desirable than other models. Also, bench- and field-based experiences involve hands-on interaction with biological organisms; this may be more appealing to students than the abstract engagement that occurs with computer-based labs. Further, bench-based projects may develop laboratory skills that are useful in advanced courses, summer research opportunities, and so on. However, data-mining research can be inexpensive (or free!), easily compressed into shorter time periods (as opposed to requiring a full semester), and devoid of laboratory hazards (such as toxic compounds and sharp tools) and—in the case of field research—natural threats such as inclement weather, dangerous animals, and vector-borne diseases. Further, computer-based data analysis skills are valuable in the current job market (De Mauro et al., 2017), either in isolation or in combination with field- and lab-based skills.

At our institution, we aim to engage all introductory biology students in a course-based research experience to meet goals that are departmental (e.g., persistence in the discipline), institutional (e.g., understanding of the tools and techniques used in biology), and societal (e.g., scientific literacy and appreciation for the processes of science). In our introductory courses for biology majors, we collaborate with principal investigators on-site to develop CUREs that are collaborative, discovery based, and broadly relevant. Because multiple research projects are running synchronously, and because these projects may be bench-based or computational, we were able to examine the effects of these different experiences on student attitudes. Our research questions were

1. How do students’ perceptions of course satisfaction, interest in their projects, and feelings of achievement at the end of the course vary as a function of CURE type?

2. How do students’ reflections about the course itself compare across CURE types?

Specifically, we assessed the relative attitudinal effects of two different types of CUREs. Two of our semester-long research projects involved live organisms (the bacterium Pseudomonas fluorescens and the vertebrate Danio rerio, the zebrafish; Table 1) and one involved databases of DNA sequences from the human gut microbiome. We were particularly interested in students’ feelings of course satisfaction and their sense of ownership of the research project, because both these metrics have been associated with an expressed intent to persist in science and the current course of study (Aitken, 1982; Seymour, 2002; Hanauer et al., 2016). We also analyzed students’ sense of achievement and interest in their research projects across the computer- and bench-based projects. It has been shown that students’ self-reports of personal gains and their attitudes toward an undergraduate experience are significantly correlated with their sense of satisfaction (Weston and Laursen, 2015). Essentially, we wondered whether students in the computer-based research project would derive the same benefits as their peers involved in the bench-based research experiences.

TABLE 1.

Similarities and differences between the computational and wet-lab research areas

Similarities: all students

meet weekly in sections of 16–25 students (all from the same research area) for 2-hour class/discussion led by a graduate TA; postdoctoral research mentor visits class periodically and meets with groups outside class as needed to guide projects develop their own research questions as a group, with advice from research mentor and TAs, and submit a project plan work on project 4–6 hours per person per week; coordinate shared work with group members; record work in shared electronic lab notebook create and present a group poster describing their project write an individual research paper

Differences between research areas
Bioinformatics	Pseudomonas	Zebrafish
Computational: ecology of the human gut microbiome	Bench lab: microbial physiology, experimental evolution	Bench lab: environmental toxicology, developmental biology
Work anywhere, anytime on personal computer	Work in dedicated lab space (open 56–60 hours/7 days per week) with undergraduate TAs and other staff
Data from publicly available data sets collected by other researchers	Collect data in lab using published procedures or self-developed protocols
Troubleshoot code, data subsetting, and analytical strategies	Troubleshoot experimental procedures and adapt or compensate for mistakes
Substantial data visualization and use of statistical tests	Limited data, little statistical analysis
Questions limited by available data and metadata, difficulty of analytical strategy	Questions limited by practical constraints (time, amount of work, feasibility, cost)

METHODS

Participants

Study participants were undergraduates (typically sophomores) enrolled in the second semester of a stand-alone introductory biology laboratory course sequence for majors at a large midwestern university. Data were collected across three semesters of this course, from Spring 2016 through Spring 2017. As part of the admissions process, demographic data (ACT, gender, first-generation college student status, and underrepresented minority [URM] status) were collected by the university. Because these demographic and psychosocial factors have been associated with student academic performance by us and others (e.g., for URM status, see Ballen and Mason, 2017; for gender, see Cotner and Ballen, 2017; Koester et al., 2016; for ACT, see Noble and Sawyer, 2002), they were used to establish whether there was equivalency between the three laboratory groups.

Course Description and Structure

Students participated in a two-semester laboratory course sequence. During the first semester, students experienced short (1- to 2-week) inquiry-based research experiences to introduce them to each of the three research areas (two bench-based: Zebrafish and Pseudomonas; and one computer-based: Bioinformatics). Following this introduction, students completed a survey in which they were asked to rank their research preferences (e.g., “I would love to work in this research area,” “I’m willing to work in this research area,” or “I definitely don’t want this research area”). Students were assigned a choice score to reflect whether they received their top choice of research area. Approximately 61% of students were assigned to their first-choice project (choice score of 3) and a further 12% (who had more than one first-choice project) were assigned to one of two equally interesting projects (choice score of 2). Approximately 9% of students were assigned to their second-choice project (choice score of 1). Students who did not complete the survey (17%, choice score of 0) were assigned to the Pseudomonas project. During the remainder of the first semester, students in each research area participated in a distinct series of laboratory experiences to learn useful skills for that area; they also read relevant literature, developed potential research questions, and wrote project proposals.

In the second semester, students registered for a course section based on their research area in the first-semester course. Sections were led by a graduate student teaching assistant (TA), often with help from an undergraduate TA. Each section consisted of between 16 and 25 students from the same research area; sections met weekly at different times, so students primarily interacted with other students in their own research areas. Within each section, students formed three- to six-person groups based on common research questions and conducted their investigations together. Students scheduled their own project work according to their availability and the needs of their projects. They recorded their work in electronic lab notebooks shared with their group members. Postdoctoral research mentors visited each section periodically during the semester and were available by email or by appointment to discuss research projects, answer questions, and provide support. Each student group presented a poster near the end of the semester, and each student wrote a research paper describing his or her project (due at the end of the semester). Course features are summarized in Table 1.

Student research projects varied in approach and methodology between different research areas within the second-semester course. Bioinformatics students worked with data from existing published data sets; their projects typically involved comparing diversity metrics between subpopulations of subjects to evaluate hypotheses they had developed (e.g., whether level of caffeine intake is correlated with relative levels of particular bacterial phyla). In contrast, Pseudomonas and Zebra­fish students designed experiments to collect their own data; because of time limitations, their data sets were less extensive than those available to Bioinformatics students and, consequently, data analysis was simpler. Pseudomonas projects used existing bacterial strains or new strains that students isolated in directed selection experiments or through adaptive radiation in static cultures, as described in Rainey and Travisano (1998). Typical projects evaluated hypotheses about the relative fitness of strains with different adaptations or the effects of environmental manipulations (such as a reduced external oxygen level) on the properties of different strains. Zebrafish projects typically addressed hypotheses about the effects of particular environmental contaminants on early embryonic development (e.g., eye formation or immune system development) or on various aspects of adult fish behavior (e.g., related to anxiety, withdrawal, or foraging).

End-of-Course Survey

Survey Administration.

At the end of the second-semester laboratory course, all enrolled students (N = 785) were invited (by email and through the course website) to respond to a survey soliciting their opinions about the course experience. A small amount of credit (0.2% of the final grade) was offered for their participation. Students were informed that identifying information would be used to align survey responses with institutional data, and they were assured that their responses would be anonymized and only used for research after final course grades were submitted. Survey responses were collected across three semesters (Spring 2016, Fall 2016, Spring 2017). ACT score data were available for 730 students, who represent the sample in this analysis. For those students, the overall survey response rate was 86%. Comparable rates of participation were observed across the three semesters (Spring 2016 = 85%, Fall 2016 = 93%, Spring 2017 = 79%) and across the three laboratory experiences (Bioinformatics = 85%, Zebrafish = 87%, Pseudomonas = 86%). The University of Minnesota’s Institutional Review Board determined the survey, along with survey administration and the use of institutional data, to be exempt from review.

Survey Instrument and Survey Analysis.

This survey was designed by the course director (C.K.) with the intent of gathering data to improve students’ experiences in subsequent semesters. The survey questions covered a range of topics, including student perceptions about what elements of the course were useful, how much time they spent on the course, their attitudes about the course (including positive and challenging aspects of the experience), and self-reported levels of confidence in particular skills. Three metrics were examined across the three laboratory experiences:

Course satisfaction. This measurement consisted of a single item: “How satisfied are you with your overall experience in Biol 3004?” Students could respond with “very satisfied,” “satisfied,” “dissatisfied,” very dissatisfied.”

Interest in research project. This measurement consisted of one item from the project ownership survey described by Hanauer and Dolan (2014): “My research project was interesting.”

Sense of achievement. This measurement consisted of one item from the project ownership survey described by Hanauer and Dolan (2014): “The findings of my research project gave me a sense of personal achievement.”

For course satisfaction, a four-point scale was used each semester. For sense of achievement and interest, there was a four-point scale in Spring of 2016, then a five-point scale in Fall of 2016, then back to a four-point scale in Spring of 2017. The changes in scaling were the result of department-level decisions in survey administration. The different scales were adjusted to a single scaled score: −1 (strongly disagree), −0.5 (disagree), 0 (neutral), 0.5 (agree), 1 (strongly agree). This scale was chosen rather than one that eliminated the neutral response entirely, because there were 24% neutral responses for the sense of achievement item for Fall 2016. In addition, prior research reports conflicting results about the inclusion of a neutral result on the positive and negative ends of the scale (Worcester and Burns, 1975; Velez and Ashworth, 2007); therefore, we decided not to assume a positive or negative response for the neutral responders and to leave the scale as a five-point scale. The mean across semesters for each project would be least altered by the scale conversion by maintaining the neutral response within a five-point scale and assigning common values to the common Likert values. The −1 to 1 scale has the additional advantage of conferring meaning to the numerical representation, with negative numbers representing negative responses and positive numbers representing positive responses. This conversion should not affect the reported comparison between research areas, because survey data were obtained for each research area in all three semesters. However, to confirm that conversion to this common scale did not significantly affect the data, the means across the two semesters with four Likert responses versus the one with five Likert responses were compared. The means did not differ significantly for either interest or sense of achievement (U = 1.5, p = 0.14, and U = 0.05, p = 0.96, respectively).

Data Analysis

To compare sense of achievement, interest, and course satisfaction by ACT score, we grouped students into low- and high-ACT bins. Students were placed into the low-ACT group if their ACT scores were below the mean ACT score for survey participants across all three semesters and into the high-ACT group if their ACT scores were above this mean. Mean ACT score for the low group was 26; for the high group it was 30. For comparison, students who earn a 26 on the ACT have a 65% probability of earning a 3.0 or higher first-year grade point average (GPA), while students who earn a 30 on the ACT have an 84% probability of earning a GPA within this range in their first year (Noble and Sawyer, 2002).

To assess equivalence of research project groups, we conducted chi-square tests on categorical items such as gender. Analysis of variance was used for ACT scores. To assess the significance of differences between research areas on interest, sense of achievement, and course satisfaction, we used Kruskal-Wallis or Mann-Whitney U nonparametric tests across student scores, with a Dunn-Bonferroni comparison used for post hoc pairwise project comparisons. To determine contributors to course satisfaction, we conducted a stepwise multiple linear regression analysis (both forward and backward) using course satisfaction as the dependent variable. Multiple combinations of independent variables were tested, as shown in Table 5 later in this article.

TABLE 5.

Linear regression analysis of factors contributing to course satisfaction

	Modela
Variables	1	2	3b	4b
Sense of achievement	β = 0.32	β = 0.33	β = 0.32	β = 0.33
	t = 7.4	t = 7.6	t = 6.4	t = 7.6
	p < 0.001	p < 0.001	p < 0.001	p < 0.001
Interest	β = 0.28	β = 0.23	β = 0.22	β = 0.23
	t = 4.8	t = 4.9	t = 4.5	t = 4.9
	p < 0.001	p < 0.001	p < 0.001	p < 0.001
Project type(bench based = 0;computer based = 1)	β = 0.14	Not included	β = 0.15	Not included
	t = 3.5		t = 3.5
	p < 0.001		p < 0.001
Choice score	NS	NS	NS	β = 0.09
				t = 2.2
				p = 0.03
ACT score	NS	NS	NS	NS
Gender (male = 0, female = 1)	NS	NS	NS	NS
First-generation status	NS	NS	NS	NS
R	0.52	0.51	0.52	0.51
Significance	F(3, 483) = 61	F(2, 484) = 83	F(3, 397) = 49	F(3, 397) = 46
	p < 0.001	p < 0.001	p < 0.001	p < 0.001

^aNS, not significant.

^bStudents who did not make a project choice were excluded.

Qualitative Analysis of Students’ Reflections on the Course

As part of the survey, students were asked, “What are some things you particularly liked about the course as a whole?” The unstructured responses for two semesters (Spring of 2016 and Fall of 2016) were coded independently by two undergraduate students majoring in biology. In the initial round of coding, themes were generated via a grounded approach (Glaser and Strauss, 1967) by both coders based on a subset of the responses (Table 2). Definitions for each theme were clarified through discussion with two faculty members (C.K. and A.S.) to reach a consensus; following the discussion, the rest of the responses were coded according to these definitions. Approximately 18% of the survey responses were coded by both coders (67 of 381 responses), with an average reliability measure of Cohen’s kappa equal to 88 (very good) (Altman and Altman, 1991). Each student’s response could include multiple themes; the average number of themes per response was 1.55. For final analyses, we focused on six common themes that appeared in at least 5% of all student responses; 39 of the 381 responses did not mention one of these common themes, and we excluded 15 other responses for which the coders did not agree on a theme, resulting in a final set of 327 student responses.

TABLE 2.

Coding categories for students’ reflections on what they liked about the course

Theme	Brief description	Example
Ownership–agency	Comment suggests a sense of ownership of the project.	“I liked how we got to come up with our own research question and design our own experiment.”
Group work	Comment suggests student appreciated teamwork.	“I also thought that I made some really good connections with members of my group and other zebrafish groups.”
Learn techniques	Comment suggests learned or gained confidence in techniques required for class or for research in general.	“I think this class greatly improved my understanding of statistics and my programming skills, especially in R.”
Real research	Comment talks about how experience relates to real research.	“It gave us a taste of what research is really like. We got to experience first hand the whole process from start to finish.”
Course structure	Comment talks about how course was structured.	“I like that it was a project based course which we later presented our results to others.”
Helpful TA	Comment suggests student appreciated help	“I also really enjoyed interacting with the undergrad lab TAs! They were all awesome and helpful.”

To gain further insight into students’ sense of ownership of their projects, we undertook a second round of coding using more specific themes representing aspects of project ownership that were developed based on conceptions frequently mentioned by students (Table 3). As in the first round, both coders evaluated a sample of responses, definitions for each theme were discussed and clarified, and then the coders evaluated all the remaining student responses. The average Cohen’s kappa was 90 (very good) based a subset of the responses scored by both coders (∼15%; 58 of 381 responses); six responses were excluded from further analysis, because the coders did not agree on a theme. Of the 381 student responses, 174 mentioned one or more of the four ownership themes. In both rounds of coding, coders were blind to the identity and to the research area of survey respondents.

TABLE 3.

Coding categories for students’ comments related to ownership

Theme	Brief description	Example
Choice of research topic	Comment suggests student appreciated choice of research question or project area.	“I liked that I had the option to conduct research on anything that interested me.”
Design own experiments	Comment suggests student enjoyed designing his or her own experiment(s).	“I liked being able to design and carry out my own experiment.”
Responsible for own project	Comment suggests student felt a sense of responsibility for the project.	“Making our own project and doing research on our own in the lab was very cool and was a good experience.”
General independence	Comment speaks positively about independence or freedom.	“I really liked the aspect of being put in a research group where you get the freedom to do what you want but there are people around to provide guidance.”

RESULTS

Equivalence of Project Groups

The project groups were assessed for equivalence on ACT score, gender, first-generation status, and URM status. Only ACT score was shown to be significantly different between the projects (F(2, 615) = 6.6, p = 0.001; Table 4; Hedges g for Bioinformatics and Zebrafish = 0.5, medium effect size).

TABLE 4.

Statistical tests for differences in demographic and psychosocial factors across project groups

	Bioinformatics (98)	Pseudomonas (333)	Zebrafish (187)	Test statistic
	Mean (SD)	Mean (SD)	Mean (SD)	F value (df  )	p value
ACT	29.6 (3.0)	29.0 (3.1)	28.0 (3.7)	6.6 (2, 615)	0.001

	Proportion (n)	Proportion (n)	Proportion (n)	χ2 (df  )	p value
Gender (F)	0.68 (67)	0.59 (197)	0.64 (119)	3.04 (2)	0.22
First generation	0.18 (18)	0.17 (56)	0.18 (33)	0.09 (2)	0.96
URM	0.08 (8)	0.11 (36)	0.11 (21)	0.71 (2)	0.70

Effect of ACT on Interest, Sense of Achievement, and Course Satisfaction

Students who scored below the mean ACT score for all survey participants reported significantly higher interest (U = 13.2, p < 0.001), sense of achievement (U = −3.5, p < 0.001), and course satisfaction (U = −2.7, p = 0.007), compared with students who scored above the mean (Figure 1).

FIGURE 1.

Mean interest (A), sense of achievement (B), and course satisfaction (C) scaled scores by ACT bins (Low ACT = students who scored below the mean ACT score for survey participants across all three semesters; High ACT = students who scored above the mean ACT score for survey participants across all three semesters). Error bar is standard error of the mean for the survey item. (n = 616; *, p < 0.05; **, p < 0.01; ***, p < 0.001.)

Semester of Lab Course Does Not Affect Interest, Sense of Achievement, or Course Satisfaction

This analysis was conducted across three semesters (Spring and Fall of 2016 and Spring of 2017). For the course as a whole, across the three semesters, there was no significant difference in student interest (H(2, 616) = 4.2, p = 0.12), sense of achievement (H(2, 616) = 4.0, p = 0.14), and course satisfaction (H(2,616) = 4.8, p = 0.09).

Interest in Project Varies Significantly by Research Area

There was a significant difference in students’ reporting of interest in their projects across the three different research areas (H(2, 614) = 17, p < 0.001; Figure 2A). A Dunn-Bonferroni post hoc pairwise comparison revealed that students working in bioinformatics reported significantly higher interest than students working on Pseudomonas (M_B = 0.61, M_P = 0.47; 9.0, p = 0.008) but expressed about the same interest as students working on zebrafish (M_Z = 0.62; 5.3, p = 0.06).

FIGURE 2.

Mean interest (A), sense of achievement (B), and course satisfaction (C) scaled score by research area. Error bar is standard error of the mean across students. (Statistical comparisons were each wet-bench research area against the computer-based research area; n = 616; *, p < 0.05; ***, p < 0.001.)

Bioinformatics Students Report a Significantly Higher Sense of Achievement

There was a significant difference in students’ feelings of achievement across research areas (H(2, 614) = 8.7, p = 0.01; Figure 2B). Bioinformatics students reported a higher sense of achievement than students working with Pseudomonas (M_B = 0.51, M_P = 0.35; Dunn-Bonferroni pairwise comparison, 54, p = 0.02) or zebrafish (M_Z = 0.36; Dunn-Bonferroni pairwise comparison, 55, p = 0.03).

Course Satisfaction Varies by Research Area

There was a significant difference in students’ satisfaction with the course across research areas (H(2, 614) = 23, p < 0.001; Figure 2C). Bioinformatics students had significantly higher course satisfaction than Pseudomonas students (M_B = 0.48, M_P = 0.22; Dunn-Bonferroni pairwise comparison, 82, p < 0.001) and Zebrafish students (M_Z = 0.36; Dunn-Bonferroni pairwise comparison, 46, p = 0.049).

Course Satisfaction Is Not Dependent on Choice of Project

To evaluate what factors might contribute to students’ satisfaction with the course, we conducted a simple stepwise linear regression analysis (forward and backward), with student course satisfaction as the dependent variable and choice score, interest, sense of achievement, ACT score, project type (bench based = 0; computer based = 1), gender, and first-generation status as possible predictors. Choice score, gender, ACT score, and first-generation status were not significant predictors of course satisfaction. The best-fitting model (R = 0.52, F(3, 483) = 61, p < 0.001) is

where β1 = 0.32 (t = 7.4, p < 0.001), β2 = 0.28 (t = 4.8, p < 0.001), and β3 = 0.14 (t = 3.5, p < 0.001). To ascertain whether some of the variables were being dropped from the regression equation because of overlapping specification, we performed additional analyses (Table 5). When students who did not indicate their research preferences were excluded from the analysis, choice score became a significant contributor to course satisfaction, but the contribution was minor (β = 0.09, indicating an effect that is almost negligible).

Choosing a Project Did Not Make a Significant Difference to Students’ Attitudes

Students who were placed in the Pseudomonas research area because they failed to complete the survey about their research preferences were compared with students who selected the Pseudomonas project. There was no significant difference in interest (Kruskal-Willis χ²(1, 286) = 0.22, p = 0.64), sense of achievement (Kruskal-Willis χ²(1, 286) = 2.5, p = 0.12), or course satisfaction (Kruskal-Willis χ²(1, 286) = 0.22, p = 0.83).

Students’ Reflections on the Course-Based Research Experience

To gain further insight into what factors were contributing to students’ satisfaction with their research experiences, we analyzed their open-ended responses to a survey question asking what they particularly liked about the lab course.

In their comments, Bioinformatics students frequently mentioned features of the course structure (33 of 57 students; 58%) (Figure 3). For example, they said they enjoyed carrying out a research project instead of more-scripted lab work and giving a poster presentation (“I like that it was a project based course which we later presented our results to others.”). They also commented on how helpful their TA was (13 of 57 students; 23%); on learning research techniques, particularly coding (11 of 57 students; 19%); and on working in groups (11 of 57 students; 19%).

FIGURE 3.

Common themes in students’ positive comments about their course-based research experience. Percentage of students whose comments included at least one of the six common themes is shown for each research area; responses could include multiple themes, so percentages sum to greater than 100%. N = 327 students (57 Bioinformatics, 185 Pseudomonas, 85 Zebrafish).

Pseudomonas students commented frequently on course structure as well (80 of 185 students; 43%), but also frequently mentioned a sense of ownership and agency with regard to their research (71 of 185 students; 38%); only 12% of Bioinformatics students’ responses (10 of 57 students; 18%) were coded as related to ownership/agency. Learning research techniques was only rarely mentioned by Pseudomonas students (13 of 185 students; 7%), but quite often they mentioned helpful TAs (36 of 185 students; 19%) and that their project work gave them insights into how research is really carried out (37 of 185 students; 20%); they sometimes wrote positively about group work as well (21 of 185 students; 11%).

Zebrafish students, like Pseudomonas students, frequently wrote about ownership of their research project (41 of 85 students; 48%) and course structure (30 of 85 students; 35%). They also commented fairly often on how their experience related to real research (16 of 85 students; 19%). Zebrafish students commented infrequently about group work and helpful TAs (9 of 85 students; 11% each) and rarely about learning new research techniques (5 of 85 students; 6%).

When students described what they particularly liked about the course, their comments about course structure frequently stated that the course structure fostered a sense of independence and ownership of the research project. (These responses were not included in the ownership/agency category during the first round of coding, because they primarily focused on the course rather than the students themselves.) To further explore the different ways in which students expressed their sense of ownership, we carried out a second round of coding wherein all the student responses were recoded using categories representing different aspects of ownership (Figure 4). Approximately 46% of the student responses (174 of 381) mentioned one or more of the four ownership themes. The percentage of students who made comments falling in one or more of these categories was not significantly different between research areas: 35% for Bioinformatics students (22 of 62), 49% for Pseudomonas students (106 of 218), and 48% for Zebrafish students (46 of 95; F(1, 343) = 1.4, p = 0.25).

FIGURE 4.

Common themes related to ownership from students’ positive comments about their course-based research experience. Percentage of student responses about ownership that included at least one of the four ownership themes is shown for each research area; responses could include multiple themes, so percentages sum to greater than 100%. N = 174 students (22 Bioinformatics, 106 Pseudomonas, 46 Zebrafish).

Bioinformatics and Zebrafish students commented frequently that they appreciated being able to choose their research topic (50% [11 of 22] and 39% [18 of 46], respectively, of the students who mentioned some aspect of ownership). Pseudomonas students also commented positively on having a choice of research topics (26 of 106 students; 25%) and frequently stated that they appreciated the opportunity to work independently in the lab (39 of 106 students; 39%). Bioinformatics and Zebrafish students commented fairly often on the level of independence as well (27% [6 of 22], and 20% [9 of 46] of responses, respectively). Across all three research areas, 27–30% of responses suggested that students felt a sense of responsibility for their own projects (6 of 22, 29 of 106, and 14 of 46 responses mentioning some aspect of ownership for Bioinformatics, Pseudomonas, and Zebrafish, respectively). Students in all three areas sometimes wrote positively about getting to design their own experiments (14–22% of responses; 3 of 22, 21 of 106, and 10 of 46 responses for Bioinformatics, Pseudomonas, and Zebrafish, respectively).

DISCUSSION

We examined how students felt about their experiences in three different research spaces to determine whether students who participated in a computer-based research experience reported similar attitudes compared with students who participated in a bench-based research experience. We specifically evaluated students’ perceptions of course satisfaction, interest in their projects, and sense of achievement at the end of the course; in addition, we coded their responses to an open-ended question to compare their reflections on their different experiences. We found that participating in the Bioinformatics project resulted in similar interest as one of the bench lab projects (Zebrafish) and higher interest than the second bench lab project (Pseudomonas). Moreover, participation in the computer-based experience led to greater feelings of course satisfaction and sense of achievement than either bench lab project. Recently, similar work at our institution—investigating student perceptions of a computer-based short-term research experience for nonmajors—indicated that computer-based research experiences were seen as authentic types of scientific inquiry (Ballen et al., 2018).

Because students essentially were able to self-select into research areas, the higher positive attitude outcomes associated with participation in computer-based research may have been caused by associated demographic or psychosocial attitudes of the population of students who chose Bioinformatics work. Although we cannot speak to possible differences of psychosocial attitudes, we evaluated several measures of the student populations in each research area and found them to be generally similar to one another. The only dimension on which students in the computer-based area differed significantly from students in the bench-based areas was in ACT scores, with students in the Bioinformatics group having higher ACT scores. However, when we examined interest, sense of achievement, and course satisfaction by high- and low-ACT score bins across research areas, students who scored below the mean on the ACT reported significantly higher interest, sense of achievement, and course satisfaction. This result suggests that the high interest, sense of achievement, and course satisfaction reported by the Bioinformatics students is not due to their higher average ACT scores. Moreover, because it is students in the low-ACT group who are reporting higher interest, sense of achievement, and course satisfaction, this result suggests that participation in course-based undergraduate research experiences may be particularly beneficial for academically at-risk students, often characterized by lower ACT scores.

Experiencing a sense of achievement was the biggest predictor of how satisfied students were with the course. This result suggests that one benefit of the course-based research experiences may be that students experience overcoming difficulties, either mastering a new skill (such as coding in R) or troubleshooting experimental difficulties (such as figuring out how to measure a dependent variable, or dealing with the death of organisms they are working with). Interestingly, ACT score does not have a significant influence on course satisfaction when interest, sense of achievement, and project type (bench-based or computer-based) are included as predictors in the regression analysis, suggesting that the effect of ACT score may be mediated through either interest or sense of achievement.

Students who participated in the Pseudomonas bench-based research area reported lower course satisfaction than students who participated in either of the other two research areas. The fact that students who did not indicate their research preferences were placed into the Pseudomonas area might have contributed to the lower level of satisfaction with the course, but when we compared Pseudomonas students who had not completed the survey with those who had, we found no significant differences in the reported attitudes of the two groups. In addition, we found that whether students worked in their top-choice research area was not a significant predictor of satisfaction at the end of the course, except when students who had not indicated their preferences were excluded from the regression analysis; in that case, choice score was a minor but significant contributor to satisfaction, though its effect pales in comparison to the effects of interest and sense of achievement.

Our analysis of students’ open-ended responses to a question asking what they liked about the course supports the idea that Bioinformatics students’ higher sense of achievement contributed to their satisfaction. Bioinformatics students’ comments about learning a technique were approximately two times more prevalent than those comments were for Zebrafish and Pseudomonas students. While Zebrafish and Pseudomonas students did experience learning of laboratory bench techniques, this learning occupied less of the instructional time compared with the learning of a programming language for the Bioinformatics students. It may be that Bioinformatics students’ higher sense of achievement compared with Zebrafish and Pseudomonas students reflects a feeling of having learned a useful skill.

The analysis of students’ comments revealed some other notable differences in their perceptions of the various course-based research experiences. For example, bench-based students were more than three times more likely to state that their projects gave them an opportunity to do “real research” as compared with Bioinformatics students. This difference may reflect students’ limited perceptions about the nature of scientific research. It may be that Bioinformatics students did not perceive computer-based data analysis to answer research questions as reflective of what occurs in “real” science research groups. This difference in comments suggests an opportunity for a course intervention to expose students to different types of scientific research, perhaps via exposure to primary investigators who use a variety of tools to answer scientific questions.

Our analysis of students’ comments also suggested that students’ sense of project ownership, or the way in which they experienced ownership, varied across the different research areas. When addressing what they liked about the course, the proportion of comments made by Bioinformatics students about feeling ownership of their projects was two to three times lower than for either of the bench-based project types when ownership was defined strictly as students talking about their feelings of agency. However, the fact that Bioinformatics students had to rely on data available in previously published work, rather than being able to generate their own, may have decreased their sense of agency. When student comments were recoded to include statements that encompassed both students’ feelings of agency and elements of the course structure that could promote that sense of agency, a broader sense of ownership emerged. Four common themes recurred: 1) a sense of independence, 2) being able to choose their own research projects, 3) the ability to design their own research approaches, and 4) their sense of responsibility to a project. When all of these themes were included in the analysis, there was no significant difference between the groups in the proportion of students mentioning ownership as a positive feature of the course.

These themes that emerged from students’ written responses to what they liked about the course are reflective of Wiley’s conceptual definition of ownership as responsibility (Wiley, 2009). These themes relate to the role students have in the project, including both stated feelings of responsibility and student concepts of the projects as being the result of their own endeavors (in choosing and designing the projects independently). The themes that arose in our students’ comments differ from those discovered during interviews conducted by (Hanauer et al., 2012), in which students were asked more specific questions about what they did for their research and why, their experiences during the project, and likely future directions. This difference in themes likely reflects the different contexts in which students were asked about their research, with students more likely to respond on a personal level to a question that asked about what they liked, as compared with answering questions focused on specific aspects of the project.

Limitations

Additional research is required to evaluate how much our findings can be generalized across different institutional and course contexts, because features of our course design or study population may have impacted the conclusions we infer from our results. For example, we placed students into research areas based on their stated preferences as much as possible, rather than assigning them randomly. Students in the Bioinformatics area were not significantly different from students in other research areas for any of the three factors we examined, apart from ACT score; however self-selection may have resulted in some undetected differences between the student populations in each area. Further measures of psychosocial attitudes, students’ interests, or preexisting skills would be needed to evaluate whether students who preferred the Bioinformatics area shared any distinct traits in comparison with students in the bench-based research areas. Another limitation is that our student population has a relatively high mean ACT score; populations with a broader range of ACT scores might respond somewhat differently to the research experiences we offered. With respect to the project options offered, our Bioinformatics research area specifically focused on human microbiome data, potentially allowing students to ask questions relevant to their own life circumstances, whereas other types of computational research experiences might not have the same level of “human interest” appeal. Finally, it should be noted that our students were exposed to several research areas with varied features; having multiple options, and the nature of the available choices, could impact student perceptions of their subsequent research experiences.

Our findings suggest that experiencing a sense of achievement is a predictor of course satisfaction. However, we did not measure students’ research achievements, so we cannot say that participating in the computational research area led to greater gains than participating in bench-based research. In addition, we can only speculate about some of the many factors influencing students’ perceptions of their experiences; we do not have data to address underlying causes behind their reported attitudes.

Future Directions

In conclusion, it appears as though students show equal or greater benefits in terms of interest, sense of achievement, and course satisfaction, and express an equal appreciation for ownership of their projects, when they participate in a computer-based rather than a bench-based CURE experience. Course satisfaction and ownership have been shown to be predictors of students’ persistence in science (Aitken, 1982; Hanauer et al., 2016), and our data suggest that interest and sense of achievement are predictors of course satisfaction. Thus, it seems that both computer- and bench-based contexts for CUREs can have positive impacts on student attitudes associated with persistence in science.

Given the results presented here, there is a need for development of more computer-based CUREs, either as hybrid research experiences with a field-based or bench-based context or as stand-alone research projects. Ongoing scientific research is incorporating more computer-based investigation of data beyond genomics (e.g., in systems biology; Bartocci and Lió, 2016), but published computer-based CUREs focus on investigating genomic comparisons (e.g., Shaffer et al., 2010; Jordan et al., 2014). Exposing students to computer-based research experiences has the added potential to better prepare them for future research endeavors by teaching computer skills and by expanding their definition of what “real research” might encompass. Further, research using existing data sets is particularly well suited to the online learning environment, an area in need of CURE development.

Our results also indicated that students who enter the university with lower ACT scores reported higher attitudinal outcomes from participating in a course-based research experience. Given that our student population has high mean ACT scores, further research in broader contexts is needed to determine whether CUREs are truly more beneficial for academically disadvantaged students.

In terms of equity, implementation of computer-based CUREs has the potential to expand CUREs to institutions where the resources involved in bench-based and field-based CUREs are not accessible. Many of these institutions serve populations that are generally underrepresented in science. Our results suggest that, in terms of attitudinal outcomes associated with persistence in a course of study or increased intent to pursue scientific research, computer-based CUREs are a suitable—and possibly desirable—alternative to bench-based CUREs. These findings should be welcomed by many of our colleagues, who struggle with the financial, space, and time constraints associated with some bench-based CUREs. By using databases as fodder for meaningful scientific inquiry, we can not only provide our students training in sought-after skills, we can introduce these developing scientists to a powerful source of information and a key tool of contemporary investigators.