IMPACT OF OPEN ENROLLMENT IN COURSE-BASED UNDERGRADUATE RESEARCH EXPERIENCES WITH AT-RISK STUDENT POPULATIONS

Abstract

Participation in authentic research activities, particularly mentored undergraduate research experiences, at the University of Texas at El Paso has long been associated with student success measures such as graduation and matriculation to strong graduate programs in STEM. However, these opportunities typically are available to upper division students, despite evidence suggesting that the first (Freshman) year at university is determinant for individuals to complete STEM degrees. To expand the number of research opportunities and to extend them preferentially to new, entering students, we established the Freshman Year Research Intensive Sequence (FYRIS) in 2015, a course sequence consisting of a research foundations course and one or two laboratory courses redesigned by faculty into small, special topic Course-based Undergraduate Research Experiences (CUREs). CUREs provide authentic research experiences with similar early-, middle-, and late-term benefits to those found in traditional mentored experiences. Several of these benefits can be conceptualized as “hubs”, which derive from earlier benefits, while facilitating later positive outcomes. Self-efficacy is one such hub, while retention and persistence in science enrollment represent late-phase positive outcomes. In this report, we examined self-efficacy of FYRIS participants in surveys administered at the start and end of each course to assess their confidence in conducting fundamental and specific research activities in the foundations and research driven courses, respectively. Specific items from a validated survey were used in addition to items developed for each course based on specific learning objectives. Retention was measured across three years of assessment of participants and non-participants, controlling for key scholastic characteristics. Results on retention rates after FYRIS vary depending on whether students fully or partially participated in the course-sequence. Results will be presented for three cohorts of students: 2015–16, 2016–17, and 2017–18.

1. INTRODUCTION

Undergraduate engagement in authentic research activities is a highly impactful educational practice with short and long-term benefits for students, ranging from enhancement of specific laboratory skills to continuing gains in scholarly professional development and perseverance in the educational and career trajectory [1]. In particular, students report improvement in specific disciplinary technical skills [2], as well as better analytic and research design abilities [3]. Increased self-confidence, sense of accomplishment, and clarification of a scientific career path are among the personal and professional benefits realised from research experiences [4].

The majority of undergraduate research experiences occur in the third or fourth year of study, as might be expected from a high-level educational activity, for which students may have already committed to a career track [1]. Evidence of retention of STEM students in their particular major discipline of study, however, suggests that the first year of university is the most determinant for completing the degree in that major [5]. Thus, the effectiveness of an undergraduate research experience may be greater if undertaken at the earliest possible opportunity.

To address this disparity and offer an authentic research experience to incoming students in their critical first year, the Freshman Year Research Intensive Sequence (FYRIS) was established in 2015 at the University of Texas at El Paso (UTEP) with funding from the Howard Hughes Medical Institute (HHMI) and the National Institutes of Health (NIH) Building Infrastructure Leading to Diversity (BUILD) Initiative. FYRIS is a sequence of courses consisting of a Research Foundations Course (RFC) and one or two Course-based Undergraduate Research Experiences (CUREs) [6]. The CUREs are small, special topic, laboratory-based courses that are part of the degree requirements for various Science and Engineering degrees. In these CUREs students have the opportunity to conduct authentic primary research under the guidance of the instructor. Any interested student, regardless of grade point average or major, is welcome to enroll in FYRIS courses as long as they have completed the mathematics requirement needed for the equivalent courses, which is pre-calculus. Thus FYRIS CUREs are more inclusive than traditional one-on-one mentored research experiences, potentially granting more students an early gateway into a research trajectory [6].

Because CUREs are authentic research experiences, they impart many of the same benefits found from generally participating in undergraduate research experiences [7]. In an extensive review and meta-analysis of CURE outcomes, Corwin et al. [7] catalogued and modelled these outcomes to identify how particular CURE related activities lead to different outcomes and relate to each other. They distinguished between multiple early, middle, and late-phase positive outcomes for the students, and mapped the pathways from specific activities to early, immediate outcomes, which then mapped to later outcomes. In three submodels, knowledge and skills, communication, and sense of ownership, respectively, were modelled before all were integrated into a single, comprehensive model. A key feature of the comprehensive model was that it allowed for the identification of “hubs”, where multiple causal pathways from different outcomes converged before influencing later outcomes. For example, real experience in the laboratory led to increased skills and expertise that all boost self-efficacy in the model, which consequently enhanced scientific identity among other outcomes and ultimately persistence in science. Thus, the model and particularly the hubs, provide a good framework for conceptualizing how specific CURE related activities lead to immediately measureable effects, which in turn influence longer-term outcomes regarding undergraduates’ retention and persistence in research based majors and careers.

In the current study, we evaluated the impact of FYRIS at middle- and late-phase outcome stages as represented in Corwin et al.’s model [7]. As a middle-phase outcome, self-efficacy derives from students’ awareness of their own content knowledge, technical skills and analytic skills, and importantly, functions as a vital hub for longer-term outcomes such as increased motivation and resilience. This critical positioning in the model makes self-efficacy an important middle phase outcome, measureable concurrent with the CURE. Accordingly, self-efficacy was assessed here prior to and at the conclusion of each course. Persistence in science was a culminating late-phase outcome for students partaking in CUREs in the integrated model [7]. For a late-phase outcome in this study, we tracked ongoing course enrollment in semesters following participation in FYRIS as an index of persistence.

2. METHODOLOGY

The complete FYRIS intervention consists of a set of three courses: a Research Foundations Course (RFC) and two, sequential freshman-level laboratory courses (freshman CURES, designated as Research Driven Courses or RDCs). These RDCs fulfil degree requirements freshman laboratories, but have been modified by a professor, who is typically the course instructor, to engage students in a research project that aligns with her/his area of research In general, students take the RFC and one RDC during their first semester and the second RDC during their second semester.

2.1. Participants

Only students who belonged to the College of Science and were first-time, traditional (under age 21) freshman were included in the analyses. In total, 1,652 students were included. The majority of students in all cohorts were approximately 18 years of age. Most students were female and Hispanic. The sample closely resembles the demographic of the College of Science, which is 62.6% female and 85.7% Hispanic. See Table 1 for a complete summary of all demographics by cohort.

Table 1.

Demographics.

	Cohorts
	Fall 2015 (N=489)	Fall 2016 (N=522)	Fall 2017 (N=641)
Male	32.6% – 37.2%	34.1% – 39.7%	34.6% – 43.1%
Female	62.8% – 67.4%	60.3% – 65.9%	56.9% – 65.4%
Hispanic	83.8% – 93.3%	88.4% – 92.9%	91.1% – 95.8%
Age
M	18.0 – 18.1	17.9 – 18.0	18.0
SD	0.357 – 0.452	0.321 – 0.439	0.313 – 0.455
Range	16 – 19	16 – 19	16 – 19
HS Percentile
M	68.5 – 80.2	67.4 – 79.3	65.3 – 77.8
SD	18.709 – 24.746	13.730 – 20.771	16.432 – 26.211
Range	3 – 100	5 – 100	0 – 100
Transfer Credit Hours
M	10.8 – 17.2	11.3 – 16.8	12.4 – 19.6
SD	14.482 – 21.925	14.162 – 22.684	15.315 – 23.863
Range	0 – 81	0 – 83	0 – 84

2.2. Retention

To assess the potential effect that FYRIS courses had on participating students, those included in the analyses were any student who participated in a FYRIS course at any time and students who had no participation in FYRIS courses. Data are separated by cohort, defined by student entry year, 2015, 2016, or 2017. Data in the analyses includes gender, ethnicity, high school percentile rank, and the amount of transfer credit hours for each student to date. It should be noted that although all students are first-time entering freshman, many enter with some college credits from advanced placement or early college high school courses.

Enrollment of each student in each fall semester to date was tracked; this was used to determine if the student was retained. Retention occurred if the student had any enrollment (at least 1 credit hour) in the fall semester as well as any enrollment in the subsequent fall semester. For example, if a student within the 2015 cohort completed a class of at least one credit in the fall of 2015 and was also enrolled for a class in fall of 2016, they were retained for 1 year. Cohort 2015 was tracked for 3 years of retention; cohort 2016 was tracked for 2 years and cohort 2017 was tracked for 1 year.

It was predicted that FYRIS participants would show greater retention than those students who did not participate in FYRIS courses. Literature on college retention controls for such variables as gender, ethnicity, high school performance, and transfer credit hours [9], [10], [11]. Thus, we included these variables as covariates and controlled for them in the following analyses to draw inferences on the unique effect of FYRIS courses.

The results in section 3.1 are organized by cohort and retentions years. The FYRIS courses were proposed as a sequence and students are expected to take one Research Foundations Course (RFC) and two Research Driven Courses (RDC) to complete a full sequence. We further analyze any retention differences in students who completed a full sequence of FYRIS courses (1 RFC and 2 RDCs) versus those who completed a single course or any partial combination of FYRIS courses (i.e. 1 RDC and 1 RFC).

2.3. Research self-efficacy

In order to assess the effectiveness of the RFCs and RDCs in meeting their learning objectives, course evaluation surveys were developed in a collaborative effort between the internal program evaluator and the course instructors. This effort led to the development of surveys that assess students’ gains in general research self-efficacy and course-specific research self-efficacy. For the RFCs, the research self-efficacy survey was developed to align with the specific student learning objectives (SLOs) for the course. This survey consists of 17 items rated using a 5-point Likert-type scale ranging from 1 (Not at all confident) to 5 (Extremely confident). Sample items include “Develop alternative hypotheses for a set of results,” “Identify the different basic sections of scientific articles from different types of publications,” and “interpret data.”

The general research self-efficacy survey for the RDCs was developed using items from the Undergraduate Research Student Self-Assessment (URSSA) survey (i.e., ‘Thinking and working like a scientist’ subscale) [8]. This survey has 14 items rated using a 5-point Likert-type scale ranging from 1 (Not at all confident) to 5 (Extremely confident). Sample items include “Formulating a research question that could be answered with data,” “Analyzing data for patterns,” and “Writing documents in discipline-appropriate style and format.” Course-specific self-efficacy survey items were also developed for each RDC. These survey items are unique to each particular course and align to the SLOs of each course. Each course-specific self-efficacy group of items varies in total number of items. The RDCs that are part of the research self-efficacy aspect of the study reported here correspond to two tracks of General Biology Labs I and II and General Chemistry Labs I and II. The research themes of the General Biology RDCs are Brain Connectomics I and II and Drug Development and Bioassay I and II. The research themes for the General Chemistry RDCs are Supramolecular Materials I and II and Circadian Rhythm Genes and Proteins I and II.

Course surveys were distributed at the beginning of each semester and at the conclusion of each semester. Students were asked to indicate how confident they felt, at the moment, about conducting various research-related activities. In order to assess students’ gains in general research self-efficacy and course-specific self-efficacy, means for each group of items were computed and paired-sample t-tests were conducted. Moreover, reliability analyses were also conducted for each group of items. Since the inception of the FYRIS program in Fall 2015, over 100 RDC and RFC sections that have been offered to students.

3. RESULTS

3.1. Retention

3.1.1. Descriptive Statistics

Retention was first analysed using descriptive statistics for each cohort of students. The frequency distributions show that a greater percentage of students who enrolled in a FYRIS course, regardless of entering cohort, was retained, compared to students who did not enrol in a FYRIS course. Moreover, a larger percentage of students who completed the full FYRIS sequence was retained when compared to students who only completed a partial sequence. See Table 2.

Table 2.

Impact of FYRIS Courses on Retention of College of Science First Time Freshman

2015 Cohort	CoS Students First-time < 20 Yrs	No FYRIS	FYRIS	FYRIS Full	FYRIS Partial
N	489	358	131	46	78
1-year retention rate	73.8%	71.8%	79.4%	95.7%	67.9%
2-year retention rate	64.4%	61.7%	71.8%	89.1%	59.0%
3-year retention rate	58.5%	55.0%	67.9%	84.8%	56.4%
2016 Cohort
N	522	332	190	73	88
1-year retention rate	73.2%	65.1%	87.4%	95.9%	79.5%
2-year retention rate	64.4%	58.4%	74.7%	90.4%	61.4%
2017 Cohort
N	641	428	213	68	144
1-year retention rate	74.7%	71.5%	81.2%	94.1%	75.0%

Because of scheduling, academic, major constraints, or personal reasons, not all students were able to complete the full three-course FYRIS Series. To explore potential differential impact of an incomplete FYRIS series, we compared outcomes of students completing either a full or a partial sequence. Completion of a full, three-semester FYRIS sequence was associated with a substantially higher rate of retention (FYRIS Full) when compared with College of Science —first time freshman averages.

3.1.2. Binary Logistic Regression of FYRIS versus No FYRIS

A binary logistic regression was conducted to determine if FYRIS course enrollment predicted greater retention compared to non-FYRIS course enrollment. Results indicate that, after controlling for key covariates, enrollment in FYRIS courses did not predict 1 or 2 year retention for the Fall 2015 Cohort. Our results only showed HS percentile and transferred credit hours as predictors of retention. Ethnicity had no predictive value in our study given that the majority of the population in question is 83.8 to 95.8 Hispanic. Gender did not show any predictive value in our study, which matches the mixed reports in the literature regarding gender. At 3 years, however, FYRIS enrollment predicted retention. See Table 3 for complete results. Although enrollment in FYRIS courses did not have an effect on retention in Years 1 and 2 for the Fall 2015 cohort, for Fall 2016 and 2017 Cohorts enrollment in FYRIS predicted greater retention in the first year compared to students who did not take any FYRIS courses, controlling for key covariates. Significant results were also observed for Year 2 retention for the Fall 2016 Cohort, as FYRIS participation predicted greater retention compared to students who did not enroll in FYRIS. We also found that, as shown in the literature, HS rank percentile significantly predicts retention at the university [9] [10].

Table 3.

Binary Logistic Regression of FYRIS versus No FYRIS Retention

Cohort	Fall 2015 (N=489)		Fall 2016 (N=522)		Fall 2017 (N=641)
Predictors	B	S.E.	B	S.E.	B	S.E.
Year 1 Retention
FYRIS: No=0, Yes=1	0.359	0.254	1.250 *	0.253	0.475 **	0.215
HS Percentile	0.020 *	0.004	0.023*	0.005	0.017*	0.004
Transferred Credit Hours	0.003	0.005	0.017 *	0.005	0.013 *	0.005
Gender: M=0, F=1	0.050	0.228	−0.013	0.218	−0.256	0.201
Constant	−0.444	0.309	−1.165	0.403	−0.239	0.270
Model χ2	26.797	p < .001	62.025	p < .001	44.948	p < .001
Cox & Snell R2	0.053		0.112		0.068
Year 2 Retention
FYRIS: No=0, Yes=1	0.443	0.229	0.682 *	0.209
HS Percentile	0.017 *	0.004	0.023*	0.005
Transferred Credit Hours	0.009	0.005	0.022 *	0.005
Gender: M=0, F=1	−0.035	0.212	−0.123	0.201
Constant	−0.819	0.303	−1.468	0.388
Model χ2	29.263	p < .001	54.764	p < .001
Cox & Snell R2	0.058		0.100
Year 3 Retention
FYRIS: No=0, Yes=1	0.506 **	0.220
HS Percentile	0.018 *	0.004
Transferred Credit Hours	0.002	0.005
Gender: M=0, F=1	0.073	0.206
Constant	−1.105	0.305
Model χ2	29.684	p < .001
Cox & Snell R2	0.059

^*p<.01

^**p<.05

3.1.3. Binary Logistic Regression of FYRIS Full versus Partial

To further examine the retention rates of students who enrolled in FYRIS courses, we selected only students who completed their first FYRIS course within their first year, and compared those who completed the full FYRIS sequence (1 RFC and 2 RDCs) to those who completed a partial sequence (i.e. 2 RDCs and 0 RFC), as described in section 3.1.1. The logistic regression results showed that, after controlling for key covariates, completing the full FYRIS sequence predicted greater retention for the Fall 2015 Cohort after years 1, 2 and 3. The same pattern emerged for the Fall 2016 and Fall 2016 Cohorts. Across all years observed for each cohort, students who completed the full FYRIS sequence had greater retention rates than those who had a partial sequence.

3.2. Research self-efficacy

In general, results indicate that across semesters, students enrolled in the RFCs reported more self-efficacy in completing general research-related activities at the conclusion of the course in comparison to pre-course self-efficacy scores. Paired samples t-tests confirmed that this increase in self-efficacy was statistically significant for all fall semesters (see Table 5).

Table 5.

Research Self-Efficacy Results from the Research Foundations Course

Semester	Pre Mean	SD	Post Mean	SD	t value	p	df
Fall 2015	2.94	0.75	3.99	0.69	−12.95	<.001	122
Fall 2016	3.01	0.71	3.89	0.77	−13.33	<.001	179
Fall 2017	3.02	0.75	4.08	0.63	−17.41	<.001	196
Fall 2018	3.00	0.71	4.01	0.69	−18.18	<.001	260

Overall, for the chemistry RDCs, a statistically significant increase in students’ self-efficacy performing general research-related activities was observed at the post-survey in comparison to the pre-survey (see Table 6). In addition, a statistically significant increase in students’ self-efficacy completing course-specific research tasks at the conclusion of the semester was also observed for chemistry RDCs sections across the semesters (also see Table 6). A similar pattern was observed for biology RDCs. Students enrolled in Biology I and Biology II reported being more confident performing both general and course-specific research tasks at the end of the semester. While statistical significance varies across course sections, students consistently report increases in general and specific research self-efficacy at the end of every semester.

Table 6.

Research Self-Efficacy Results from Chemistry Research Driven Courses

Supramolecular Materials I & II
Semester	Research SE	Pre Mean	SD	Post Mean	SD	t	p	df
Fall 2015 Lab I	General	3.50	0.51	4.16	0.45	−4.17	0.001	17
	Specific	2.81	1.38	4.17	0.52	−4.07	0.001	17
Spring 2016 Lab II	General	4.03	0.47	4.39	0.55	−1.92	0.092	8
	Specific	3.89	0.51	4.35	0.48	−2.35	0.047	8
Fall 2016 Lab I	General	3.30	0.77	4.23	0.48	−5.58	<.001	19
	Specific	1.83	0.68	3.90	0.75	−8.06	<.001	19
Spring 2017 Lab II	General	4.19	0.43	4.49	0.41	−2.68	0.019	13
	Specific	3.61	0.60	4.07	0.41	−3.51	0.004	13
Fall 2017 Lab I	General	3.45	0.47	3.82	0.61	−1.43	0.187	9
	Specific	2.43	0.67	3.53	0.67	−4.15	0.002	9
Spring 2018 Lab II	General	3.65	0.49	3.70	0.49	−0.46	0.661	8
	Specific	3.32	0.70	3.56	0.59	−1.89	0.095	8
Fall 2018 Lab I	General	3.53	0.63	3.73	0.73	−1.01	0.332	13
	Specific	1.85	0.45	3.48	0.96	−6.01	<.001	13
Circadian Rhythm Genes and Proteins I & II
Semester	Research SE	Pre Mean	SD	Post Mean	SD	t	p	df
Fall 2015 Section I	Evaluated using different measures
Spring 2016 Lab II	General	3.42	0.38	3.97	0.53	−3.79	0.002	13
	Specific	2.14	0.48	4.12	0.45	−10.80	<.001	13
Fall 2016 Lab I	General	3.32	0.50	3.95	0.49	−4.35	0.001	11
	Specific	2.45	0.79	4.04	0.71	−6.95	<.001	11
Spring 2017 Lab II	General	4.06	0.47	4.51	0.40	−3.05	0.014	9
	Specific	3.58	0.98	4.44	0.40	−3.60	0.006	9

4. CONCLUSIONS

4.1. Retention

Descriptive statistics show strong association between participation in FYRIS courses and retention. For the three cohorts across multiple years, students who complete the full FYRIS sequence are better retained than those who complete only a partial sequence.

For long-term retention, our binary logistic regression model results indicate that FYRIS is a strong predictor, even after controlling for the key covariates. A similar analysis comparing FYRIS Full to FYRIS Partial revealed that the full sequence was the primary predictor for both short and long-term retention. The program of courses continues to demonstrate positive results for first-time traditional students in the College of Science.

4.2. Research self-efficacy

We have observed a pattern of student’s self-efficacy for general research self-efficacy to be significant in the RFC and the first RDC; thus, we can infer students’ research self-efficacy is being maintained through these two courses. By the time students, complete the second RDC course, statistical increases in self-efficacy are no longer achieved likely due to a ceiling effect on the Likert scale. We infer that student confidence in completing general research tasks is fostered throughout the FYRIS sequence.

Students are also maintaining course-specific self-efficacy throughout the two-course RDC sequence. This effect is likely due to students being continuously required to learn new skills as they progress in these courses.

Table 4.

Binary Logistic Regression of FYRIS Full versus Partial

Cohort	Fall 2015 (n=124)		Fall 2016 (n=161)		Fall 2017 (n=212)
Predictors	B	S.E.	B	S.E.	B	S.E.
Year 1 Retention
FYRIS: Partial=0,Full=1	2.086 *	0.775	1.626 *	0.473	1.615 *	0.557
HS Percentile	0.012	0.001	−0.001	0.015	0.002	0.008
Transferred Credit Hours	0.033	0.022	0.040**	−0.017	0.035**	0.016
Gender: M=0, F=1	0.084	0.496	−0.842	0.444	−0.006	0.373
Constant	−0.434	0.715	0.733	1.105	0.603	0.606
Model χ2	22.186	p < .001	29.901	p < .001	19.933	p < .01
Cox & Snell R2	0.164		0.169		0.09
Year 2 Retention
FYRIS: Partial=0,Full=1	1.558 *	0.542	1.625 **	0.663
HS Percentile	0.010	0.010	0.002	0.018
Transferred Credit Hours	0.029	0.018	0.028	0.021
Gender: M=0, F=1	−0.224	0.455	−1.040	0.602
Constant	−0.463	0.680	1.724	1.370
Model χ2	20.021	p < .001	16.117	p < .01
Cox & Snell R2	0.149		0.095
Year 3 Retention
FYRIS: Partial=0,Full=1	1.375 *	0.481
HS Percentile	0.007	0.009
Transferred Credit Hours	0.004	0.015
Gender: M=0, F=1	−0.216	0.427
Constant	−0.148	0.664
Model χ2	12.385	p < .05
Cox & Snell R2	0.095

^*p<.01

^**p<.05

Table 7.

Research Self-Efficacy Results from Biology Research Driven Courses

Brain Connectomics I & II
Semester	Research SE	Pre Mean	SD	Post Mean	SD	t	p	df
Fall 2015 Lab I	Evaluated using different measures
Spring 2016 Lab II	General	3.35	0.52	4.08	0.50	−2.99	0.017	8
	Specific	1.88	1.09	3.55	0.70	−5.13	0.001	8
Fall 2016 Lab I	General	3.83	0.37	4.24	0.56	−2.60	0.035	7
	Specific	2.11	0.47	4.00	0.51	−11.24	0.000	7
Spring 2017 Lab II	General	3.59	0.38	4.17	0.43	−4.29	0.001	12
	Specific	1.61	0.80	3.42	0.80	−9.18	0.000	12
Fall 2017 Lab I	General	3.34	0.45	3.98	0.53	−3.47	0.007	9
	Specific	1.94	0.74	3.07	0.64	−4.30	0.002	9
Spring 2018 Lab II	General	3.71	0.52	3.96	0.36	−2.35	0.051	7
	Specific	2.41	0.91	3.35	0.52	−3.42	0.011	7
Fall 2018 Lab I	General	2.81	0.53	3.95	0.33	−4.76	0.000	12
	Specific	1.19	0.24	3.19	0.80	−14.66	0.000	12
Drug Development & Bioassay I & II
Semester	Research SE	Pre Mean	SD	Post Mean	SD	t	p	df
Fall 2016 Lab I	General	3.57	0.55	4.29	0.49	−5.76	0.000	22
	Specific	1.81	0.87	4.36	0.67	−13.96	0.000	22
Spring 2017 Lab II	General	3.74	0.53	4.29	0.36	−5.22	0.000	17
	Specific	3.55	0.67	4.06	0.61	−4.80	0.000	17
Fall 2017 Lab I	General	3.38	0.62	4.24	0.38	−4.47	0.001	13
	Specific	2.03	0.77	4.13	0.49	−11.79	0.000	13
Spring 2018 Lab II	General	3.99	0.46	4.31	0.43	−1.84	0.108	7
	Specific	3.91	0.45	4.21	0.32	−2.34	0.052	7
Fall 2018 Lab I	General	3.26	0.57	4.13	0.82	−3.90	0.001	15
	Specific	1.72	0.66	3.90	1.01	−7.48	0.000	18