Comparing the Impact of Course-Based and Apprentice-Based Research Experiences in a Life Science Laboratory Curriculum

Abstract

This four-year study describes the assessment of a bifurcated laboratory curriculum designed to provide upper-division undergraduate majors in two life science departments meaningful exposure to authentic research. The timing is critical as it provides a pathway for both directly admitted and transfer students to enter research. To fulfill their degree requirements, all majors complete one of two paths in the laboratory program. One path immerses students in scientific discovery experienced through team research projects (course-based undergraduate research experiences, or CUREs) and the other path through a mentored, independent research project (apprentice-based research experiences, or AREs). The bifurcated laboratory curriculum was structured using backwards design to help all students, irrespective of path, achieve specific learning outcomes. Over 1,000 undergraduates enrolled in the curriculum. Self-report survey results indicate that there were no significant differences in affective gains by path. Students conveyed which aspects of the curriculum were critical to their learning and development of research-oriented skills. Students’ interests in biology increased upon completion of the curriculum, inspiring a subset of CURE participants to subsequently pursue further research. A rubric-guided performance evaluation, employed to directly measure learning, revealed differences in learning gains for CURE versus ARE participants, with evidence suggesting a CURE can reduce the achievement gap between high-performing students and their peers.

INTRODUCTION

Large public universities face a unique challenge—to educate a sizeable and diverse student body using student-centered instructional strategies that are scalable, effective, and sustainable. This capacity issue creates a barrier to providing all science majors at large public universities meaningful engagement in authentic research (20, 38, 63). Authentic research investigations mirror the entire research process, which includes portraying how hypotheses are generated, how data are collected and analyzed to address research questions and make evidence-based conclusions, and how research discoveries are communicated (54). More than a decade of reports (3, 32, 42, 44) cite numerous studies documenting the positive outcomes that students realize as a result of their immersion in an authentic research experience (8, 22, 26, 28, 29, 37, 40, 47, 50, 56, 59). Collectively, these studies have empowered institutions to explore ways to effectively integrate research experiences into and throughout the college curriculum (9, 21, 24, 31, 61). One strategy involves an entire class in the investigation of a research question of broad importance to the scientific community (15, 60). Referred to as a course-based undergraduate research experience, or CURE, this scalable approach has been embraced by instructors at a variety of colleges and universities (11, 27, 39), as well as at the national level with multi-institution programs (30, 51, 52).

Corwin Auchincloss and colleagues (6) summarize several benefits of CUREs over apprentice-based research experiences (AREs) beyond expanding capacity, including the ability to reach all students, not just a few self-selected students who are highly motivated to seek co-curricular research opportunities. By contrast, CUREs utilize the time students spend in class, thus supporting the participation of a broad range of students, including, for instance, those who have jobs or longer commutes to and from campus that occupy much of their time outside of class. By mitigating factors that prevent some groups of students from participating in research, CUREs represent an inclusive and equitable means of integrating research into the undergraduate curriculum (8). When introduced early in the undergraduate curriculum, CUREs can positively affect the academic trajectory (22, 30) and career interests (26) of students. When integrated as a capstone experience in the senior year, CUREs can facilitate integration and application of concepts students have learned throughout college (33). Offering CUREs at different stages in the curriculum provides multiple entry points for students to engage in research. This consideration is especially important for students who transfer into four-year institutions. Course-based undergraduate research experiences offered during the third year can involve these undergraduates in authentic research as they acclimate to their new learning environment.

This study describes a research-based laboratory curriculum implemented in 2010 at the University of California, Los Angeles (UCLA). It was created as a framework for immersing large numbers of upper-division undergraduate Life Science majors into an assortment of authentic research experiences, wide-ranging enough to accommodate the diverse research interests of over 250 undergraduates per year, yet tailored to the unique motivations and competencies of individuals. This research curriculum was designed to engage third- and fourth-year life science students transitioning into their major area of study. The timing benefits students whether admitted as freshmen or transfer students. In addition, a third-year experience occurs early enough in the curriculum to allow inspired students to explore further research opportunities before graduating.

Research-based curriculum description

Referred to as the Competency-Based Research Laboratory Curriculum (CRLC), this laboratory program is comprised of courses that build upon the competencies described in the Scientific Foundations for Future Physicians (5) and Vision and Change (3) reports. The CRLC was designed to provide in-depth research opportunities for all majors in two life science departments at UCLA. A schematic of the curriculum is presented in Figure 1, with a full description of the program provided in Appendix 1.

FIGURE 1.

Competency-based research laboratory curriculum (CRLC) for Life Science majors. Course requirements for each path are enclosed in separate gray boxes stemming from arrows labeled Path 1 and 2. Path 1 is comprised of course-based undergraduate research experiences (CUREs), and Path 2 is based on an apprenticeship model (designated AREs). The light blue boxes in Path 1 (left panel, AL) and the top pair of red (AR) and gray (AS) boxes in Path 2 denote courses students enroll in during the first of two 10-week terms. The teal boxes in Path 1 (right panel, BL) and the bottom pair of red (BR) and gray (BS) boxes in Path 2 delineate courses student take during the second 10-week term. Both paths are preceded by requisite lower-division courses. Bridging the way to Path 2 is a 10-week Research Acquaintance term, in which students “try out” research with a faculty mentor before deciding to apply to enroll in Path 2. Note that there was a fifth set of Path 1 courses offered during the first two years of the CRLC, but the second course was discontinued and the first course instead became an elective for major credit. Although this set of courses is omitted from the diagram, the final dataset for the surveys in this study includes these participants’ responses as long as they completed both the AL and BL courses.

Briefly, upon completion of requisite lower-division core courses (“Introductory Biology Courses and Lab”), students fulfill their departmental major’s laboratory requirements by one of two research paths (Fig. 1). Termed a bifurcated curriculum, the CRLC allows student cohorts to select different options for delivery of parallel research experiences. Each pair of Path 1 AL and BL courses together make up a course-based undergraduate research experience (CURE), in which students experience the process of scientific discovery in a research team comprised of three or four students. All four CUREs focus on different research projects (Table 1), providing choices meant to accommodate the diverse interests of undergraduates in the two participating departments. Path 2, synonymous with an apprentice-based research experience (ARE), is intended for students who participate in a “Research Acquaintance” experience and become interested in fulfilling their degree requirements by continuing their independent research project with the same faculty mentor. Unlike Path 1, an application process is a requisite for entry into Path 2. The Path 2 proposal instructions, faculty mentoring agreement, and faculty mentor assessment guidelines are provided in Appendices 2–4.

TABLE 1.

Overview of Path 1 research projects: course-based undergraduate research experiences (CUREs).

Microbiology AL/BL: Microbiome Diversity in the Rhizosphere
Students isolate new microorganisms or novel DNA sequences from terrestrial environments (48). Students identify bacteria using standard microbiological and bioinformatics techniques followed by phylogenetic analysis with nucleotide sequences. Students investigate and compare microbial community diversity profiles from the soil surrounding the roots of different plants. 16S rRNA gene sequences are deposited into a class database, some of which are later published in NCBI Genbank.
Virology AL/BL: Bacteriophage Discovery and Genomics Investigations
Students discover a unique bacterial virus and characterize viral particles by electron microscopy. Students investigate environmental factors affecting the viral life cycle and virus/host interactions. Students use bioinformatics tools to investigate genome organization and predict the functions of virus genes. Genomes are published in NCBI Genbank.
Cell & Molecular Biology AL/BL: Sea Urchin Development and Genome Evolution
Students use RT-PCR and whole mount in situ hybridization to determine the expression pattern of genes from the annotated sea urchin genomea. Students use phylogenetic analysis to probe the evolutionary history of assigned genes using protein sequences. Student findings are deposited into a class database.
Plant-Microbe Ecology AL/BL: Characterizing an Essential Interface Supporting Plant Life
Students look for evidence suggesting beneficial plant-microbe interactions by inoculating plants with potential nitrogen-fixing bacteria (diazotrophs). They measure increases in plant growth by biomass accumulation and the ability to fix nitrogen. To investigate possible mechanisms underlying plant growth promoting effects, students use bioinformatics tools to explore the diazotroph genomes, annotating genes such as those involved in nitrogen fixation and phytohormone production. Some student discoveries lead to published papers.

^aSea Urchin Genome Project: http://www.spbase.org/SpBase/

NCBI = National Centre for Biotechnology Information; RT-PCR = reverse transcription polymerase chain reaction.

A research program built using backwards course design

What unifies this bifurcated configuration of research-oriented courses, irrespective of path, are the student learning outcomes (SLOs) described in Table 2. Following the principles of backwards course design (1, 62), research activities were designed to support student learning and the development of research skills. Assessments were intended to gauge achievement of the learning outcomes. Several SLOs emphasize higher-order cognitive skills (HOCS) characteristic of Bloom’s Taxonomy (4, 19). We hypothesized that students completing the CRLC, whether as participants in a CURE or ARE, should demonstrate achievement of these learning outcomes as well as express similar gains in self-assessed abilities. Our assessment also enabled us to compare the impact, by path, on student impressions of research and their interest in biology. The results of this study indicate that, while self-reported gains are nearly identical for students in each path, there are marked differences in learning gains ascertained from a rubric-guided evaluation of embedded assignments, suggesting a CURE can reduce the achievement gap between high-performing Path 2 students and their peers in Path 1. If offered during the third or fourth year of college, a CURE provides a pathway for transfer students to enter research. Furthermore, creating a framework that promotes scientific inquiry within authentic, yet diverse, research contexts addresses the capacity issue confronting large, public universities.

TABLE 2.

Learning gains for CRLC students by path.

			Cumulative Meanb
SLO No.	Students completing the CRLC should be able to...	Cronbach’s Alphaa	Path 1	Path 2
1	Demonstrate knowledge of key disciplinary concepts and their relationship to biological systems. (N = 7)	0.83	15.36	14.98
2	Demonstrate knowledge of research project. (N = 1)	ND	ND	ND
3	Develop technical expertise and confidence through hands-on experience. (N = 5)	0.85	12.33	11.88
4	Develop problem-solving skills associated with conducting experiments. (N = 6)	0.81	14.39	13.79
5-1	Address scientific questions using inquiry-related skills. (N = 6)	0.81	14.07	13.98
5-2	Address scientific questions using quantitative and computational skills. (N = 3)	0.84	9.94	9.42
6	Improve presentation skills (oral communication needed for seminar and poster presentations). (N = 5)	0.80	12.83	12.68
7	Improve scientific writing abilities (writing skills needed for research proposals and papers). (N = 4)	0.82	7.56	7.34
8	Effectively work in both individual and collaborative contexts. (N = 3)	0.70	6.59	6.38
9	Value research and its relevance to own life and society. (N = 1)	ND	ND	ND
10	Understand the process of scientific research. (N = 2)	ND	ND	ND

^aThe Cronbach’s Alpha score is reported only for SLOs with three or more survey items.

^bCumulative mean scores for each survey item categorized by SLO on a 3-point Likert scale (1 = none, 2 = some, 3 = great), except for SLO 5-2, in which survey items were on a 6-point Likert scale (1 = no gain, 2 = small gain, 3 = moderate gain, 4 = large gain, 5 = very large gain, and 6 = N/A). N/A responses were removed and calculations performed using a 5-point maximum scale. There were no significant differences in the cumulative means for Path 1 and Path 2 (p > 0.05 for all SLOs).

CRLC = competency-based research laboratory curriculum; SLO = student learning outcome; ND = not determined; N/A = not applicable.

METHODS

Student demographics and study sample

The CRLC was implemented in fall 2010, and data collection started in winter 2011, continuing through fall 2014. Between 2010 and 2014, a total of 1,002 students enrolled and earned a grade in one or both courses of the two-term curriculum. Table 3 provides a summary of demographic characteristics for these students, of whom 860 completed both courses. Also shown is the average cumulative grade point average (GPA) by path.

TABLE 3.

CRLC participant demographics.

Total Participantsa	N = 1,002	Path 1 (CUREs) N = 834		Path 2 (AREs) N = 168
		N	%	N	%
Gender	Female	454	54.4	92	54.8
	Male	380	45.6	76	45.2
URM	URM	91	10.9	19	11.3
	Non-URM	714	85.6	140	83.3
	Unknown	29	3.5	9	5.4
Pell Grant Recipientb	Pell	336	40.3	49	29.2
	No Pell	498	59.7	119	70.8
College Entry Status	Transfer	246	29.5	31	18.5
	Freshman	588	70.5	137	81.5
Year in School	2nd Year	2	0.2	3	1.8
	3rd Year	195	23.4	111	66.1
	4th Year	572	68.6	53	31.5
	5+ Years	65	7.8	1	0.6
		N	4.0 Scale (Letter Grade)	N	4.0 Scale (Letter Grade)
Academic Standingc	Cumulative GPA	831	3.3 (B+)	168	3.7 (A−)

^aIncludes students who earned a grade on one or both courses in a two-term curriculum (excludes UCLA Extension students). Senior-level students who participated in the CRLC during the first year (fall 2010 – fall 2011) were not required to complete both terms. The curriculum requirement for either a CURE or ARE went into effect for all students beginning winter 2012.

^bPell Grant Recipient is a proxy for low socioeconomic status (SES); received Pell Grant for one or more terms while enrolled at UCLA.

^cAcademic standing is the cumulative GPA at the term prior to enrolling in the first CRLC course. The associated letter grade reflects the letter grade assignments made by the UCLA registrar (http://www.registrar.ucla.edu/archive/catalog/2005-07/catalog/catalog05-07acadpol-2.htm). GPA data were missing for three Path 1 students.

CRLC = competency-based research laboratory curriculum; CURE = course-based undergraduate research experience; ARE = apprentice-based research experience; URM = under-represented minority students (American Indian, Native American, Black Non-Hispanic, and Hispanic students); GPA = grade point average.

Assessment data collection and analyses

The study utilized two sources of data: self-report surveys and embedded student assignments. Both data sources were merged with existing institutional data. Qualitative and quantitative approaches were employed, and the use of multiple data sources validated findings (17, 45). UCLA’s Institutional Review Board (IRB) gave approval to work with human subjects on all aspects of the assessment (IRB #10-000904).

Administration of self-report surveys

Two self-report surveys (see Appendices 5 and 6) were administered to all CRLC students beginning winter term 2011. Surveys included a broad collection of open- and closed-ended questions, some developed by the evaluation team and others, when perceived as relevant to the study, borrowed from other instruments (10, 36, 37, 49). Students took the entry survey during week 1 of the first course in the two-term curriculum. They took the exit survey in the second course during the last week of instruction. All surveys were administered electronically during class. Only students who completed Path 1 courses AL and BL or Path 2 courses AR/AS and BR/BS (N = 842), and who completed at least 10% of the surveys between winter term 2011 and fall term 2014 (N = 713), were considered for the study, yielding an 84.7% response rate.

Quantitative analyses of closed-ended questions on self-report surveys

Descriptive analyses of survey responses were conducted to explore students’ initial reported skills (Part V on entry survey, Appendix 5) and their self-assessed gains with respect to these skills upon completion of a two-course sequence (Part IV on exit survey, Appendix 6). At the beginning of this study, a large number of self-reported skills were grouped into broader SLO categories. Confirmatory factor analysis was conducted to statistically test relationships between survey items categorized by SLOs and to highlight latent constructs in the data (see Appendix 7). This two-step process was conducted as described previously (13, 18, 57). Briefly, in the first step of the analysis, the survey items were tested in SLO groups to measure factor loading levels. The factor loading value given to a survey item is an indicator of how well that item connects with the latent construct being measured. In this study, we utilized a minimum standard cut-off for factor loading of 0.5. A minimum of three survey items was required for each construct. SLOs 2, 9, and 10 did not meet this requirement and thus were excluded from subsequent steps of the analysis.

The second step of confirmatory factor analysis tests the reliability of the entire SLO group. The reliability measure used in this study was a test of the Cronbach’s Alpha, in which a minimum standard cut-off of 0.6 was used. As shown in Tables S7-1A and S7-1B (Appendix 7), seven SLOs had a Cronbach’s Alpha score above the cut-off value, with scores ranging between 0.69 and 0.85. Notably, confirmatory factor analysis resulted in the separation of SLO 5 into two distinct SLOs, “inquiry-related skills” (SLO 5-1) and “quantitative and computational skills” (SLO 5-2).

Because we were interested in measuring students’ gains in self-assessed abilities upon their completion of the two-term curriculum, subsequent descriptive analyses focused on exit-survey data. Cumulative means for the seven SLOs that met the conditions for confirmatory factor analysis were calculated by summing the mean score for individual survey items within each SLO grouping. Independent t-tests were used to test for significant differences in cumulative means for each SLO, as reported in Table 2. For all but SLO 5-2, the cumulative mean scores were derived from survey items having a 3-point Likert scale. The highest cumulative mean score possible corresponds to the total number of items (N), grouped in a single SLO, multiplied by the maximum score on the Likert scale (3 pts). By inference, the lowest score possible corresponds to the total number of items in the group (N) multiplied by the minimum score of the Likert scale (1 pt). Survey items categorized into SLO 5-2 were on a 6-point Likert scale, with non-applicable (N/A) responses removed and calculations performed using a 5-point maximum scale, meaning the highest cumulative mean score possible corresponds to the total number of survey items (N = 3) multiplied by 5. Irrespective of the Likert scale employed, the cumulative means for the survey items in each SLO grouping reported in Table 2 fall within the calculated range of maximum and minimum scores.

Qualitative analyses of open-ended questions on self-report survey

Analysis of open-ended data followed previously described procedures (16, 25). The first step in this iterative, multi-step process involved initial review and coding of student responses by a research analyst. In this step, each qualitative item was examined and a corresponding preliminary list of codes, or themes, was developed to capture a sense of meaningful segments of text (i.e., segments of the text were tagged with thematic labels resulting from careful reading and interpretation of student responses). To maintain consistency, the list of codes included examples of text pertaining to each theme. The next step involved recursive examination to see how the data corresponded to (or strayed from) the initial themes, with the aim of revising the themes to increase succinctness and remove redundancies. Once this first stage of coding was complete, frequencies and percentages of student responses according to each theme were calculated, and sample responses were pulled to illustrate how the themes corresponded to the students’ self-reported experiences. Upon review of the results by the entire research team, the themes were further revised, with the collapsing of some themes into broader categories and the omission of categories with low response frequencies (< 5%). The process concluded when the research team reached consensus on the presented themes as accurately and concisely reflecting the sample student responses (see Table 4 for final list of themes).

TABLE 4.

Frequencies and percentages of self-reported responses by Path 1 and Path 2 students: Valuable skills or abilities developed as a result of research participation.

Themea	Path 1			Path 2
	Frequency	Percentb Responses	Percentc Students	Frequency	Percentb Responses	Percentc Students
Research skills, lab techniques, computer skills	124	34.3	53.5	41	19.3	33.3
Independence, teamwork, dedication, perseverance	72	19.9	31.0	29	13.6	23.6
Presentation and communication skills	56	15.5	24.1	45	21.1	36.6
Analysis, critical thinking, thinking like a scientist	42	11.6	18.1	44	20.7	35.8
Writing skills	24	6.6	10.3	24	11.3	19.5
Information literacy	18	5.0	7.8	16	7.5	13.0

^aThemes with less than 5% of responses not reported. Sample responses for each item are provided in Appendix 8.

^bAccounts for students who provided multiple responses, resulting in 362 total responses for Path 1 and 213 total responses for Path 2. Percentage is the proportion of total responses corresponding to a particular theme. Percentages in these columns, when those below 5% threshold are included, sum to 100%.

^cA total of 232 Path 1 students and 123 Path 2 students responded to the prompt. Again, accounting for students who provided multiple responses, the percentage indicates the proportion of students who reported a response corresponding to a particular theme.

Direct evidence of learning from embedded student assignments

The final data source was obtained from a pair of matched course assignments: slides used for oral presentations given by a team of students in Path 1 or by individual students in Path 2 from the first term (course AL or AS) and slides from the same team or individual for each path during the second term (course BL or BS). Team presentations from two of the four CUREs were available for analysis. There were 61 team presentations (corresponding to ~240 students) representing Path 1 and 165 individual presentations for Path 2. A subset of paired presentations (Path 1, N = 36, and Path 2, N = 30) were selected for a rubric-guided evaluation (2) by external content experts, who were individuals with PhDs in biology or a related discipline. The number of presentations sampled should attain an 80% confidence level and reflect a 10% sampling error for the team and/or individual presentations in each path (43). To maintain anonymity, electronic copies of the presentation slides were stripped of identifiers prior to releasing the files to the external content experts.

The Path 1 and Path 2 presentation rubrics used for this analysis are provided in Appendix 9. Each rubric item was classified according to Bloom’s Taxonomy, designated either as a lower-order cognitive skill (LOCS) or a higher-order cognitive skill (HOCS) (4, 19). All rubric items also were grouped by the student learning outcome (SLO) they represented. To ensure dependability and quality of findings (34), the external content evaluators met with two head CRLC instructors, who both had extensive teaching and assessment experience with courses in one or both CRLC paths. Together, these three individuals refined and normalized scoring criteria for the rubrics prior to the formal assessment. Because there were multiple instructors involved in CRLC instruction and the template (e.g., general guidelines, rubrics, or both) provided to students to develop their team or individual presentations varied by instructor, this normalization process ensured that the student performance criteria adequately captured student performance outcomes on this course assignment across instructional preferences. This process also yielded slight variations in the rubrics by path. To measure learning gains, the evaluation scores were compared for the two matched sets of presentations using repeated measures analysis of variance (ANOVA) (55).

RESULTS

Demographic characteristics reveal potential benefits of the research-based curriculum

Distribution of gender and race/ethnicity status were similar by path, with ~54% females and 11% URM students (American Indian, Native American, Black Non-Hispanic, and Hispanic students) enrolled in the CRLC between 2010 and 2014 (Table 3). There were proportionally fewer Pell Grant recipients, a proxy used for low socioeconomic status, in Path 2 compared with Path 1. This finding may suggest that the in-class research experiences (CUREs) available via Path 1 are attractive to students more likely to have jobs or other commitments outside of class time.

Over this four-year timespan, 277 transfer students completed the research-based curriculum, making up ~28% of all CRLC participants. This was surprising given that only ~10% of all STEM (science, technology, engineering and mathematics) majors in the life sciences division at UCLA are transfer students, as compared with freshmen entrants. Upon further examination, the two life science departments participating in the CRLC have collectively experienced a significant increase in the percentage of transfer students graduating in their majors, from 22.1% in 2007/2008 to 30.3% in 2011/2012. The reasons for this shift were not investigated in this study, but this observation alludes to the merits of providing third-year students a pathway for entering research, as it is a suitable timeframe to engage the increasing number of matriculating transfer students nationwide (7).

Self-reported learning gains are similar by path

More than 40 items, reflecting a variety of research-related skills, were individually categorized according to the student learning outcome (SLO) that each addressed, and the validity of these groupings was established using confirmatory factor analysis (Appendix 7). CRLC students were asked on the exit survey to estimate the extent to which they felt their abilities, in relation to these skills, changed as a result of participating in their 20-week research experiences. Students scored their perceived changes in skill level using either a three-point or six-point Likert scale. Results for the exit survey, distinguishing student responses by path, are reported in Table 2. In all seven cases, there were no significant differences (p > 0.05) in cumulative means for Path 1 and Path 2. This result indicates that, irrespective of path, all CRLC students express similar gains in self-assessed abilities.

In addition to these quantitative measures, open-ended questions were included on the surveys to better understand students’ experiences within the bifurcated curriculum. One question on the exit survey asked students to reflect on their experiences in the CRLC and indicate whether or not they met or exceeded expectations. The vast majority of students in both research paths (188/227, or 82.7%, of Path 1 respondents and 102/114, or 89.5%, of Path 2 respondents) indicated that their research experience met or exceeded their expectations. All CRLC students also expressed significant increases (p < 0.05) in their level of interest in biology, as demonstrated by a comparison of student responses to a parallel question prompt on the entry and exit surveys (Table 5). Taken together, survey results provide support for our hypothesis, indicating that CRLC participants, whether engaging in CUREs or AREs, are reporting comparable gains in research-related abilities and complete the curriculum with positive impressions of their research experiences and an overall increased interest in biology. Furthermore, irrespective of path, students seemed to recognize their achievement of the SLOs emphasized by the curriculum (Table 2). These findings indicate that these two student cohorts, despite being immersed in different research modalities, underwent parallel research experiences that led to equivalent affective gains. Encouraged by this finding, we next explored the impact of each path in this bifurcated curriculum on cognitive gains.

TABLE 5.

Disciplinary engagement: What is your current level of interest in biology?

	Entry	Exit	Differencea
Path 1	3.25	3.75	0.50
Path 2	3.5	4.24	0.74

^aWithin-group differences were significant within 95% confidence interval (p < 0.05).

Direct evidence for closing the achievement gap

To directly assess student learning, we examined a set of archived student assignments for a subset of Path 1 and Path 2 students. Thirty-six sets of presentations from two of the five Path 1 courses (Virology AL/BL and Microbiology AL/BL) were selected for analysis. The rubrics included a total of 29 items evaluated on the AL presentation and 31 on the BL presentation. Thirty sets of presentations from all Path 2 AR/AS and BR/BS courses were selected for assessment using rubrics that contained a total of 22 items evaluated on the AS presentation and 25 on the BS presentation. The mean lower-order cognitive skill (LOCS) and higher-order cognitive skill (HOCS) scores for this rubric-guided evaluation of embedded CRLC assignments were plotted for each path at the two time points (Figure 2).

FIGURE 2.

Direct evidence of learning from a rubric-guided evaluation of embedded student assignments in Path 1 (CURE) and Path 2 (ARE). Rubrics were developed using a 3-point performance scale (1 = needs work, 2 = satisfactory, 3 = excellent) and employed to assess student oral presentation slides for Path 1 and Path 2 from the first term (T1) and second term (T2) of the curriculum (for rubrics, see Appendix 9). Rubric items were categorized as (A) LOCS or (B) HOCS (4, 19). For Path 1, there were 14 LOCS rubric items at T1 and another 14 items at T2. For Path 2 T1 and T2, there were 8 and 10 LOCS rubric items, respectively. With respect to HOCS rubric items, for Path 1 there were 15 and 17 at T1 and T2, respectively. For Path 2, there were 14 HOCS rubric items at T1 and another 14 items at T2. Data points on graphs represent the mean score at each time point for each Path. At T1, the mean scores for Path 2 (red lines) were significantly higher († p < 0.05) than Path 1 (blue lines) for both LOCS (A) and HOCS (B). The mean scores for Path 1 (blue lines) at T2 were significantly higher (* p < 0.05) than the mean scores at T1 for both HOCS and LOCS items, while Path 2 students (red lines) did not demonstrate measureable gains between these two time points. LOCS = lower-order cognitive skills; HOCS = higher-order cognitive skills; CURE = course-based undergraduate research experience; ARE = apprentice-based research experience.

Analyses using repeated measures ANOVA indicate that, at the first time point (T1), the LOCS and HOCS mean scores for Path 2 were significantly higher (p < 0.05) than Path 1 mean scores (Figures 2A and 2B). This observation may be attributed to the selection criteria used to channel traditionally high-performing students into Path 2, suggestive of an average cumulative grade point average (GPA) that is slightly higher than the typical Path 1 student (Table 3). Over the 20-week time period, however, the Path 1 students appear to catch up with their Path 2 peers. As shown in Figure 2, Path 1 students demonstrated significant learning gains (p < 0.05) over time at both the lower (panel A) and higher (panel B) levels of Bloom’s Taxonomy, whereas Path 2 students did not exhibit statistically significant learning gains. Comparing either the HOCS or LOCS means at the second time point (T2), there was no significant difference between Path 1 and Path 2 students—the CUREs engaging Path 1 students appear to have closed the achievement gap that previously existed between them and their Path 2 peers. Interestingly, while the data do not fully support our hypothesis with regard to the equivalent impact of each path of this bifurcated curriculum on cognitive gains, we instead observed an unanticipated benefit of participating in a CURE. Further study involving a larger sample size might reveal whether these academic benefits extend to groups of students who are traditionally underserved in higher education (12, 41, 44). This finding led us to consider more specifically what skills or abilities were valued by students, as well as what aspects of the curricular experience helped them develop these competencies.

All students benefit from a research-based curriculum

A subset of CRLC students responded to a multicomponent, open-ended question prompt on the exit survey asking students first to describe valuable skills or abilities that they developed as a result of participating in the research program, and next to explain what aspect of the program helped them acquire those skills or abilities. Six major themes emerged following coding of student responses. Table 4 gives the frequencies for these themes and shows their prevalence both in terms of the overall response rate and the percentage of students whose responses fit into a particular theme. Appendix 8 contains a sample of student responses corresponding to these themes.

As shown in Table 4, Path 2 students most frequently mentioned presentation and communication skills, one of the ten aforementioned SLOs (Table 2). High percentages of Path 2 students also mentioned data analysis and critical thinking as important to their ability to “think like a scientist,” an important element that surrounds understanding the process of scientific research (SLO 10). Relative to the prevalence of other themes, research skills, lab techniques, and computer skills, which collectively correspond to SLOs 3 to 6, were cited by students as the third most valuable skillset imparted by the Path 2 curriculum. Interestingly, more than half (53.5%) of Path 1 students report a response corresponding to this same theme. This latter finding is consistent with a study by Rowland and colleagues (46), which indicated that students in a bifurcated biochemistry laboratory experienced similar gains in technical confidence, irrespective of the type of laboratory experience.

The second part of this open-ended question asked students to specify which aspect(s) of their CRLC experience supported their development of a particular skill or ability. Tables S8-1A and S8-1B (Appendix 8) categorize coded student responses to this portion of the question prompt for Path 1 and Path 2, respectively. Not surprisingly, having multiple opportunities to give oral presentations to their instructors, faculty mentors, and peers was among the most frequently cited aspects of the CRLC (12.2% of Path 1 and 11.0% of Path 2 student responses, respectively), leading to the development of presentation and communication skills (SLO 6). The chance to engage in hands-on laboratory work prevailed (22.2% of Path 1 and 9.8% of Path 2 student responses, respectively) as an aspect of their research experience that facilitated development of a variety of research-oriented skills (e.g., SLOs 5, 6, and 10). The relative frequencies of student responses varied across the other themes, with curriculum activities contributing to multiple outcomes related to student learning. However, the alignment achieved between various curricular interventions and the learning outcomes irrespective of path becomes obvious, supporting the effectiveness of using backwards course design during the early stages of CRLC development.

DISCUSSION

Overall, the study findings point to the alignment achieved using backwards course design (1, 62) during the development and implementation of the CRLC. Course activities were intentionally incorporated into both research paths with the aim of improving students’ capabilities in realizing the ten specified SLOs (Table 2). Our findings are consistent with previous studies of undergraduates who complete CUREs (30, 51), with CRLC students reporting gains similar to those associated with student participation in research apprenticeships. As noted by Corwin and colleagues (15), which aspects of CUREs underlie their effectiveness has not been studied systematically. The significance of our work is that, in addition to documenting student outcomes, we also determined which components of the research curriculum were critical to student learning and skills development. For instance, curricular activities common to both paths (e.g., oral presentations) contributed to student learning in analogous ways (e.g., improved communication skills). The contributions of other curricular components were path-specific. For example, challenges encountered by Path 1 research teams led students to acknowledge collaboration, perseverance, and as one student states, “pure grit” as integral to the process of scientific research (SLO 10).

Using a rubric-guided performance evaluation to measure and compare CRLC learning outcomes by path revealed, contrary to our original hypothesis, that there are marked differences in learning gains for CURE versus ARE participants. The methodology we employed to directly measure student learning highlights a novel insight about CURE experiences: that is, an effectively designed CURE appears to reduce the achievement gap between the highest performing students and their peers. The improved competencies of Path 1 students may have served as a confidence-builder, inspiring a subset of students to continue engaging in research after completing the curriculum. Notably, at least 16% of Path 1 study participants who entered the CRLC in their second or third year and completed course BL (N = 181) subsequently enrolled in a credit-bearing research apprenticeship prior to graduating.

There were several limitations to our study that could, in principle, unduly influence the outcomes observed and bias our conclusions. For example, since our goal was to compare two research-based experiences (CUREs versus AREs), no control group was employed in this study. Only two of the four Path 1 CUREs were included in the rubric-guided evaluation of student team presentation slides. Having a more diverse sample of course assignments from all four CUREs would have been ideal but was not possible due to resource limitations and incomplete archiving of student work by all Path 1 instructors. Furthermore, in contrast to the presentations given by individual students in Path 2, those in Path 1 were done as three- to four-student team presentations. Thus, it is possible that the learning gains observed in the evaluation reflect the summative gains of a collaborating group of students. While, in principle, the comparison of gains in LOCS and HOCS by path is confounded by this variable, the results may point to a benefit ascribed to CUREs, which deliberately foster collaborative learning environments, an effective and inclusive pedagogy that has been used in a variety of other classroom contexts (14, 23, 35, 53, 58).

Another confounding factor is that both Path 1 and Path 2 courses were taught by a multitude of instructors during the four-year study. Furthermore, there were more than 80 different faculty members that served as research mentors for Path 2 students in this timeframe. Faculty members differ in their research mentoring strengths and teaching styles; thus, the research experiences and learning environments likely varied by instructor and/or faculty mentor. In all but three offerings, the two-term AL/BL and AS/BS course sequences were taught by the same instructor, thereby minimizing the impact of instructor and mentor variation on the students’ affective and cognitive gains measured in the study. In addition, students maintained the same research mentor for the research acquaintance and the AR/BR terms, thus ensuring consistency in mentoring style for individual Path 2 students. The faculty mentoring agreement submitted by students with their Path 2 application materials details the expectations of a research sponsor (Appendix 3) when agreeing to mentor a Path 2 student. Faculty mentors also are provided assessment guidelines (Appendix 4), designed to serve as discussion points in regular meetings with their mentees. These curricular materials, combined with intentional communications between the AS/BS seminar instructor and AR/BR faculty mentor, are intended to create a uniform research experience for all Path 2 students.

Despite these limitations, the positive assessment outcomes played a critical role in making the case to UCLA’s administrative leadership to invest in student success and continue supporting the research-based curriculum even after extramural funding sources expired. Thus, the CRLC has come to represent a scalable and sustainable curricular framework by which large public research universities can broaden undergraduate participation in scientific research. Since its launch in 2010, the CRLC has successfully trained hundreds of diverse, talented, and ambitious undergraduates for success in the life sciences. The bifurcated curriculum appeals to Life Science majors with different levels of academic preparedness, assorted proficiencies in laboratory skills, and ranging motivations to enter research. As a third- or fourth-year component of the undergraduate curriculum, the CRLC provides a pathway for transfer students to enter into research. This upper-division research curriculum is producing STEM graduates with life-long learning skills reinforced by a teaching approach that emphasizes the development of competencies recognized as central to biology literacy (3).

SUPPLEMENTAL MATERIALS

Appendix 1: Overview of the Competency-Based Research Laboratory Curriculum (CRLC)

Appendix 2: Path 2 project proposal guidelines

Appendix 3: Path 2 faculty mentoring agreement

Appendix 4: Assessment guidelines for Path 2 faculty mentors

Appendix 5: CRLC entry survey

Appendix 6: CRLC exit survey

Appendix 7: Factor analysis of survey items

Appendix 8: Qualitative analysis of open-ended responses

Appendix 9: Presentation rubrics