Course-based Undergraduate Research Experiences (CUREs) in General Education Courses

Abstract

While much of the promotion for undergraduate research (UR) originates from the natural sciences, this high-impact practice should also occur in social science to prepare students for graduate school/ the workforce and should be integrated into lower-division general education courses. Our study examines content and skills gained by students from two course-based undergraduate research experiences (CUREs) in Introduction to Sociology courses. Pre- and post-course survey analyses, post-survey student outcomes of a CURE class compared against students enrolled in three non-CURE Introduction to Sociology classes, and a content analysis of end-of-semester papers indicate student knowledge gain in specific topical areas, methodological skills, and major sociology theoretical perspectives. We conclude that UR enhances research- and sociology-related knowledge.

Introduction

Introductory courses typically fulfill general education curricula, can recruit future majors, and can respond to students’ concerns about employment outcomes. Because these courses serve such a diversity of needs, faculty have long debated what is at the core—both content and skills—of their introductory courses. Introduction to Sociology offers one example (Persell, 2010; Persell et al., 2007; Wagenaar, 2004). Interview and survey data from peer-recognized leaders in the field identified major elements that students should understand after an Introduction to Sociology course (Persell, 2010; Persell et al., 2007) revealing the importance of the scientific nature of sociology and how sociologists acquire knowledge (Persell et al., 2007).

One effective method to expose introductory students to the methodologies of data collection and analysis is through participation in a course-based undergraduate research experience (CURE). This paper details how we integrated an undergraduate research intervention into two Introduction to Sociology courses. We address two research questions. First, through the use of quantitative pre- and post-course surveys, we assess if students enrolled in two Introduction to Sociology CUREs self-report a gain in sociology and research knowledge. Second, through a qualitative content analysis of final papers from one course, we ask if CURE student outcomes reflect a mastery of course content and research skills knowledge.

Review of the Literature

Undergraduate Research.

Instructors across the country heed the call to reform undergraduate education that stresses high-impact practices (HIPs) such as writing intensive and service-learning courses (Kilgo et al., 2015; Kuh et al., 2013) — a topic covered prior in Understanding Interventions (e.g., Goode et al., 2018). These HIPs, as promoted by the Association of American Colleges and Universities, are a set of practices that have uniquely and repeatedly demonstrated efficacy in fostering learning and training undergraduates for 21^st century careers. Students’ personal and professional gains associated with HIPs are due to their unique and shared traits, including collaborating with faculty, applying theories, synthesizing ideas, and receiving valuable feedback (Kuh, 2008).

A main HIP, undergraduate research (UR), is increasingly implemented in classrooms across a diversity of colleges/universities (Johnson & Stage, 2018). Traditionally, UR refers to practicing science through internships, independent study, or research apprenticeships, where an individual student or small teams work directly with a faculty member. In these arrangements, much of the content acquisition and skill-building occurs outside of class. These types of UR experiences offer a small student-to-teacher ratio, intense training, and exposure to various steps in the research process.

The impact of UR on student learning has been assessed using surveys, tests, observations, and/or instructor evaluations of student work (Weston & Laursen, 2015). The main skillsets of interest in UR are science- and research-related, such as oral/written communication and effective collaboration (Alkaher & Dolan, 2014; Senter et al., 2015; Shaffer et al., 2010). Other desirable (though less commonly assessed) outcomes are scientific literacy and evidence-based decision-making (Feinstein et al., 2013). These outcomes are also desirable for students who do not participate in UR, are current or future STEM or social science majors, and/or will attend graduate school.

Many studies show that UR can increase skills with long-term implications such as: increase productivity and information retention, invoke curiosity, reinforce course content, prepare students for graduate school and the workforce, and strengthen critical/analytical/logical thinking and problem-solving skills (Atkinson et al., 2006; Bauer & Bennett, 2003; Corwin et al., 2015; Craney et al., 2011; Hartmann, 1990; Ishiyama, 2002; Russell et al., 2007; Senter et al., 2015). UR advantages were found to be particularly salient for Hispanics compared to other racial/ethnic groups (Russell et al., 2007) and first-generation college students (Ishiyama, 2002), such as the students in this study. However, other measures of success of HIPs, including UR, on student development is currently under debate; in a study of 101 four-year public institutions, Johnson and Stage (2018) found HIPs had an uncertain relationship with timely college completion, particularly at the least selective institutions.

Course-based Undergraduate Research Experiences.

Because traditional forms of UR are such intensively mentored experiences, they are typically only open to a select group of students (based on institution, skill, class level, and/or availability), and/or those who self-select (Auchincloss et al., 2014). Due to its many benefits, undergraduate research should not be limited to seniors or honors students (Rowlett et al., 2012). To expand student access to UR, instructors can embed UR directly into the course curriculum. The widely-used formal name and approach to this intervention is course-based undergraduate research experiences, or CUREs. One formal, albeit broad, definition of a CURE is “a course in which [all enrolled] students are expected to engage in science research with the aim of producing results that are of interest to a science community” (Corwin et al., 2015, p. 3); which can occur throughout the entire course or in a multi-week module.

Auchincloss and colleagues’ (2014) definition of CUREs includes five dimensions—discovery, relevant work, collaboration, iteration, and use of scientific principles—implemented with varying frequency and intensity. Considering the breadth of these dimensions, CUREs can be tailored for use in diverse course styles (Linn et al., 2015). CUREs have been more commonly associated with the disciplines of natural science, engineering, and mathematics. However, this narrow association has been challenged repeatedly (e.g., Kain, 1999). For sociology, CURE integration is most common in advanced courses, such as Methods, capstone, or special topics courses (Bartholomay, 2018; Hauhart & Grahe, 2012; Johnson et al., 2013). However, as long as the course’s research requirement is suitable for students’ academic abilities (Halonen et al., 2003), CUREs can and should be integrated early and throughout students’ curricula, including foundation courses (Markham, 1991).

Similar to the UR research, student outcomes specifically in CUREs have been studied. In fact, previous scholars have assembled a list of CURE-focused assessment instruments that are available for use in a variety of settings (Shortlidge & Brownell, 2016), with some outcomes mirroring those of more general UR (as outlined above) and some outcomes being more specific to CUREs. Corwin and colleagues (2015) offer an overview of CURE student outcomes: probable outcomes include career clarification and increased content knowledge and technical skills; possible outcomes include increased collaboration skills and enhanced science identity; proposed outcomes include increased access to faculty interaction and enhanced understanding of the nature of science. However, studies are still not completely clear on the underlying mechanisms that generate the value of CUREs (Corwin et al., 2015). This question remains unanswered in part because of the sheer complexity of such courses and the challenge in CURE scholarship is to create studies that demonstrate causality between student outcomes and instructional experiences, which is an especially difficult task given that students may not fully achieve particular outcomes until after multiple semesters or numerous courses (Corwin et al., 2015).

CUREs have a variety of outcomes and stakeholders, and CURE scholars therefore call for a combination of assessment techniques (Auchincloss et al., 2014; Brownell & Kloser, 2015; Gurel et al., 2015; Shortlidge & Brownell, 2016). A few studies have collected data from alumni or faculty, or have analyzed syllabi to examine course and/or faculty outcomes (e.g., Bauer & Bennett, 2003; Brownell & Kloser, 2015; Shortlidge et al., 2017). Some studies utilized comparative assessment techniques, such as collecting interviews to compare students’ understandings of the nature of science across three styles of laboratory courses: traditional, inquiry-based, and research-based, where the research-based curriculum fared the best (Russell & Weaver, 2011). In a study comparing outcomes in a single class, Rowland and colleagues (2012) used pre- and post-test surveys to study a large, bifurcated biochemistry course, one portion with an UR project, and one portion without. The two streams, which the authors reported were parallel and equivalent, were created to support the diverse learning needs of the class. The results indicate that students in both streams had similar overall gains in confidence in a variety of skills (e.g., graphing, interpreting, and calculating). In a study comparing a plant biology CURE across institutions (a two-year junior college, a four-year liberal arts college, and a public level research university) and level (introductory and upper-level), Alkaher and Dolan (2014) completed and analyzed pre and post interviews with 34 of the 83 enrolled students (18 science majors and 16 non-majors). Their interviews indicate that the CURE offered science majors and non-majors alike an opportunity to progress toward self-authorship (moving from accepting knowledge to constructing knowledge).

Study Background

The CUREs and their Contexts.

The assessed Introduction to Sociology courses occurred in Spring 2017 and Spring 2018 at the University of Texas at El Paso (UTEP), where over half the students are first-generation college students and over 80% self-identify as Hispanic. The courses were research-driven CUREs; part of both the First Year Research Intensive Sequence (FYRIS) program and Building Infrastructure Leading to Diversity: Southwest Consortium of Health-Oriented Education Leaders and Research Scholars (BUILDing SCHOLARS, henceforth referred to as BUILD), a program financially supported by the National Institutes of Health. FYRIS targets, but is not limited to, all incoming freshman who placed into a pre-calculus or higher math course. Students enrolled in the FYRIS program take a sequence of courses that includes a research foundations course and then research-driven CURE courses, which comprise courses in the natural, social, and health sciences, and aim to teach both research principles and techniques. BUILD is a prestigious and competitively sought-after research-intensive training program that targets incoming freshman, but is also open to sophomores and juniors. Students enrolled in BUILD take multiple CUREs throughout multiple semesters. Although these Introduction to Sociology courses were research-driven courses, there were no pre-requisites. Because not all students who were enrolled in the CUREs were associated with FYRIS or BUILD, students came in with different knowledge and experience bases.

The course design and materials were organized by a sociology professor and sociology postdoctoral fellow, and almost exclusively taught by the postdoctoral fellow. Grading was primarily done by the postdoc, with quiz grading by a sociology Teaching Assistant. The courses met three times per week for 50 minutes; the 2017 course enrolled 24 students across 10 majors and the 2018 course enrolled 21 students across 7 majors.

Research Topic: Birth.

Whereas the courses mirrored typical Introduction to Sociology courses in many ways (e.g. course description, sociology textbook, typical course assessments), they also uniquely focused on a research topic: the medicalization of childbirth. The theme of the medicalization of childbirth was woven throughout the curriculum on Introduction to Sociology topics. It was also introduced in the first-day ice breaker (what students knew about their own births), and was the focus of our second main textbook and class activities (e.g., watching a recorded homebirth to practice field notes). This topic was selected because it reflected an overlap of research interests of both the course professor and postdoc (the authors of this paper) it was flexible enough to explore numerous sociological themes (e.g., the role of social institutions and gender inequality), and enabled a partnership with a local birth-related organization. The authors also felt that the topic would be relevant to students, the majority of whom will become parents during their lifetimes (Geiger et al., 2019), also because it addressed local issues, given the high infant and maternal health disparities along the Mexico–United States border (Walker et al., 2014).

The courses’ research activities were accomplished in mutually beneficial collaboration with an El Paso nonprofit, El Jardín Birth and Family Resource Center (henceforth referred to as El Jardín), whose mission is to “nurture a regional culture through education and advocacy that supports, cares for, and empowers women during the critical transition of birth and parenthood.” One of their recent efforts is a collection of video-recorded interviews with local mothers, maternal health providers, and researchers that traces the history of the region’s birth culture. These interviews were collected prior to the instructors’ collaboration with the local birth resource center. At the time, it was being produced into an educational documentary, Birth in Pieces, designed as a resource for both health professionals and community members.

Student Engagement in Research.

It is often impractical to introduce and practice every scientific method step within a single CURE (Brownell & Kloser, 2015), and the authors and birth nonprofit staff decided together that students enrolled in each CURE should complete a single common course project to expose them to a subset of research practices. The research components support the three characteristics of effective methods assignments—they utilize real-world data, captivate student interests, and deeply engage student skills (Ghoshal, 2018)—and the five dimensions of a CURE (Auchincloss et al., 2014).

Specifically, the 2017 CURE students were collectively provided with four video interviews for analysis: one with a mother and three with maternal health providers. Video participants each signed a consent form acknowledging that their interviews are the property of El Jardín and available for public use. These interviews offered a helpful illustration for course material on human subjects’ protections, including verbal and/or informed consent. Each student was assigned one 10-minute clip from one video to transcribe. All transcribed segments were then combined to create a complete transcription of the videos, to which students had access.

Throughout the semester, the students intellectually engaged with relevant sociological research and concepts through course readings, discussions, and activities. From this knowledge base, students were taught how to code the interviews and each student individually identified three sociological themes, and individually wrote and orally presented a formal analysis of their identified themes. Each of these assessments had drafts due over the course of many weeks that were either peer-reviewed or reviewed by the postdoc and given structured support from the instructors that allowed for some individual student flexibility and creativity, given the value of detailed structure for students participating in research projects (McKinney & Day, 2012). For example, during Week 9, El Jardín staff visited the class, and each subsequent week students focused on the following tasks (one per week): transcription, preliminary coding, refined coding, paper drafts, oral presentations, and final papers. These various assessments accounted for 45% of students’ final grades. The responsibility to code and write gave students freedom to analyze the data. Student-generated data, including the identified sociological themes and the examples most often quoted by students, were shared with the El Jardín staff.

CUREs often look different each semester, as the new course evolves from the previous efforts of the students and instructor(s). The 2018 CURE continued its collaboration with El Jardín. Based on the nonprofit’s timeline for the documentary, their research needs shifted; the staff were eager to identify peer-reviewed information and data to integrate into their film. Their foci included the evidence-based benefits of doulas, outcomes associated with childbirth education, and women’s birth experiences. The co-instructors were in full support of students completing academic literature searches and writing brief synthesis papers for the nonprofit staff because these steps are vital to any research process. Using this assignment meant we could emphasize the value of a literature review—a common writing assignment in college courses (Ridley, 2012)—including contextualizing research questions, understanding the use of key words, learning about a variety of data sources, and critically evaluating previous publications.

We once again scaffolded the CURE-related responsibilities. We started by asking students to pick one research topic of interest and then they attended an interactive presentation with a university reference librarian on performing thorough literature searches (e.g., use of key words and databases). Formal assessments included submitting five separate worksheets (see Appendix A) that summarized a single article and compared it to any previous articles that were read. Near the end of the semester, students wrote up a final paper that integrated their articles and course-assigned sociological readings. Students also video recorded an oral presentation (of the themes found in their final paper); we devoted a class period to a presentation from UTEP’s Associate Director of Academic Technologies who discussed effective video-making and the integration of minimal features, like a title card. These various assessments accounted for 50% of the students’ final grades. The student-completed worksheets and writing were shared with the El Jardín staff.

Notable, these courses fulfill the five CURE dimensions. As a reminder, CUREs are difficult to define, and often their definition is in relation to other UR learning experiences such as cookbook labs or internships; however, Auchincloss and colleagues (2014) conclude that CUREs uniquely integrate five dimensions (with varying frequency and intensity): use of scientific practices, discovery, relevance, collaboration, and iteration.

The use of scientific practice is quite broad, and can include activities such as asking questions, selecting methods, and communicating findings. Our use of science in our 2017 course included analyzing data, developing an argument, and communicating data themes—in written and oral form. Use of science tools in our 2018 course included identifying and evaluating appropriate literature, synthesizing data, and communicating conclusions in written and oral form.

Discovery is defined as “the process by which new knowledge or insights are obtained” (Auchincloss et al., 2014, p. 31). Our 2017 CURE students generated new understandings based on the interviews they coded and analyzed. Auchincloss and colleagues (2014) apply the term discovery quite broadly yet explain that discovery requires evidence-based reasoning and exploration. Fulfilling this application through the process of literature reviews, our 2018 CURE students had to consider the scope, data sources, and author’s framework in order to critically evaluate and synthesize the literature they encountered. Like other scholars, we reject the argument that empirical research is the only real and valid research and believe literature reviews are legitimate scholarly processes (LeCompte et al., 2003).

Relevance is a CURE dimension we proudly executed. Both sets of students engaged with a topic important to their academic, community, and personal lives. The majority of our students will become parents, maternal health disparities are important issues along the Mexico–United States border, and the topic of birth sheds light on numerous sociological themes. Relevance also applies to the impact beyond the classroom and this CURE invited students to deliver materials to the nonprofit. Direct benefits to El Jardín from both semesters included increased exposure of their work, four transcripts of their video recorded interviews, a report of the main themes that students identified in coding transcripts for analysis, literature review worksheets that reference and summarize peer-reviewed data, and final papers that synthesized numerous publications on the topic they were assigned.

The CURE dimension of collaboration stresses the intellectual value and skill-building associated with group work. In the 2017 CURE, students relied on one another to transcribe the other segments of the video and supported one another in identifying codes and peer-editing final papers. In the 2018 CURE, students assisted one another with brainstorming words for effective literature searches, shared best practices in creating videos, and peer-edited final papers.

However, direct collaboration between the students, in both the 2017 and 2018 courses, and nonprofit staff was quite limited. The faculty members worked in active collaboration with the staff of El Jardín, prior, during, and after the course to support their needs, and we invited them to class at the beginning of each semester to introduce themselves. An El Jardín staff member presented to the class, including discussing the goals and timeline for the documentary, which helped students feel like active contributors to the documentary project. The students from each semester, however, did not extensively engage with the nonprofit’s staff or clients, nor did they reflect on their coursework in relation to the non-profit’s goals; thus there was no extensive service-learning component to the course. While service-learning is important among the HIPs (Kuh, 2008) and is often successfully stacked with UR (Banks & Gutiérrez, 2017), we did not conceptualize the project in this way nor did we communicate the CURE components to students as a civic responsibility or a service we were offering to the nonprofit. Lastly, critical reflection is an essential component of service learning, but students did not formally complete such an activity. This CURE could, however, be considered community-based research.

The final CURE dimension, iteration, insists that students are exposed to the process by which science accumulates over time (Auchincloss et al., 2014). The practice of relying on transcripts from their peers, as well as repeatedly reading, coding, and re-coding the transcripts were two ways the 2017 class was exposed to the iterative process of research. The following year, the instructors shared the 2017 efforts with the 2018 students—including the transcripts and main themes identified. Both 2017 and 2018 students knew they were contributing to a project that had already started and would continue after the semester ended.

Methods

Procedures.

We ran several analyses. First, we draw on pre- and post-course survey data for both courses. The 2017 and 2018 survey questions slightly differed based on the different research projects in which the 2017 and 2018 students engaged. The pre-and post- questions for the 2017 survey were the same, as were the pre- and post- questions for the 2018 survey.

To assess our first research question—CURE student confidence in performing research activities and perceived level of knowledge of sociological topics—students enrolled in the CUREs completed voluntary and confidential pre- and post-surveys. These were part of other BUILD evaluation responsibilities that has IRB approval. The pre-questionnaire was completed the first week of the semester and a post-questionnaire the last week of the semester. These surveys—detailed under the “Measures” subheading below—were designed and collected by the BUILD evaluation team; questions were adopted from the Undergraduate Research Student Self-Assessment (URSSA; Weston & Laursen, 2015). Students complete these surveys for all BUILD courses in which they are enrolled. The professor and postdoc, however, included some specialized questions in the surveys to assess the content- and research-specific objectives that the courses were designed to teach as listed in the syllabi. For the 2017 course, we expect to find knowledge gains in qualitative research methods, specialized sociology of birth knowledge, and basic sociological ideas between the CURE pre- and post-course data. For the 2018 course, we expect to find knowledge gains in research methods related to the literature review, specialized sociology of birth knowledge, and basic sociological ideas between the CURE pre- and post-cure data.

In addition, to enable a nonequivalent-comparison-groups design (Sweet & Cardwell, 2016), where one teaching model was introduced to one class, but not to others, we compared the data from the Introduction to Sociology 2017 CURE students with data from other non-CURE Introduction to Sociology students. The authors asked all UTEP instructors teaching non-CURE Introduction to Sociology courses that semester if they would be willing to distribute a survey in their courses. Three out of four instructors accepted to anonymously offer the same post-questionnaire during the last two weeks of class. Importantly, these classes were led by different instructors and used different curricula, lacking our CURE’s focus on the medicalization of childbirth or UR. These various evaluation efforts were approved by the UTEP Institutional Review Board.

While the first research question relies on student knowledge self-reports, our second research question relies on student product assessment (Sweet & Cardwell, 2016) and asks if 2017 CURE student outcomes in course material reflect a mastery of content knowledge and qualitative coding skills. We assess this by analyzing what sociological themes were most common among the student-submitted final papers and did these final papers meet instructor expectations. As each student was required to independently identify and analyze three sociological themes from the four interviews, we expect to find a variety of themes among the submissions.

Sample.

Content-based knowledge questions and basic student demographic information were collected in the surveys for each course (see Table 1). Of note, there were different sample sizes: CURE 2017 (n= 22), CURE 2018 (n=21), Intro Course 1 (n=18), Intro Course 2 (n=36), and Intro Course 3 (n=82). For the purposes of our analyses, we include “ethnicity” rather than “race” data for two reasons. First, 83% of UTEP students earning undergraduate degrees are Hispanic (UTEP Communications, 2020), thus the “Hispanic” versus “Non-Hispanic” classification is very descriptive. Second, there were problems with the BUILD survey construction (we asked if we could change these, but were denied). Students were asked the question “What is your race?” followed by “Are you Hispanic or Latino/Latina?” The race question offered six different response options, American Indian, Asian, Black or African American, Native Hawaiian or other Pacific Islander, White, and Other; there was no option to identify as Hispanic or Latino. The term “other” was selected by 16.7% of students, and when prompted to “please specify,” students frequently wrote-in answers such as “Latino,” “Hispanic,” and “Chicano,” where Chicano was not an answer category option given in the ethnicity question. We believe that the survey had an ordering problem—had the survey asked the Hispanic/Latino question prior to the race question, we would have had more accurate race/ethnicity data. Moreover, the survey design given to us by the BUILD evaluation team only asked the demographic (e.g. race/ethnicity) questions in the pre-course survey. Perhaps the post-course data might have reflected students’ increased awareness of how sociologists and researchers identify “race” and “ethnicity” and perhaps their answers would have differed by semester’s end.

Table 1:

Demographics

	2017 CURE		2018 CURE		Intro Course 1		Intro Course 2		Intro Course 3
	Percent	Frequency	Percent	Frequency	Percent	Frequency	Percent	Frequency	Percent	Frequency
Gender
Male	29.2%	7	23.8%	5	51.4%	5	51.4%	19	45.2%	38
Female	62.5%	15	76.2%	16	43.2%	13	43.2%	16	52.4%	44
Other	0.0%	0	0.0%	0	2.7%	0	2.7%	1	0.0%	0
Ethnicity
Hispanic	79.2%	19	90.5%	19	73.0%	15	73.0%	27	82.1%	69
Non-Hispanic	12.5%	3	9.5%	2	21.6%	3	21.6%	8	10.7%	9
College classification
Freshman	37.5%	9	28.6%	6	37.8%	5	37.8%	14	28.6%	24
Sophomore	20.8%	5	33.3%	7	21.6 %	5	21.6 %	8	44.0%	37
Junior	29.2%	7	23.8%	5	21.6 %	6	21.6 %	8	16.7%	14
Senior	4.2%	1	14.3%	3	13.5%	2	13.5%	5	8.3%	7

Measures.

The pre-designed BUILD survey was three pages long and included three different survey sections that assessed research skills and sociological content separately. The survey begins with standardized questions in a section titled “How confident are you currently to conduct the following general research activities.” Because this survey was designed for all students enrolled in BUILD-funded courses, most of whom are natural science majors, we only found one question from this section particularly relevant to our 2017 social science CURE, which was “analyzing data for patterns.” Responses to this question were on a 5-point scale: not at all (1), slightly (2), somewhat (3), very (4), and extremely (5). Due to small sample sizes, this variable was recoded into three categories: “not at all” remained the same, coded as “1”, “slightly and “somewhat” were combined, coded as “2”, and “very” and “extremely” were combined and coded as “3.”

We analyzed all variables for the 2017 and 2018 CUREs from the next survey section that asked, “How knowledgeable are you currently about the following areas/topics/concepts” because these were questions from our syllabi objectives that were added to the survey before the course began. These questions covered four basic sociology concepts: 1) the main four theoretical perspectives, 2) the social construction of sociological statuses, 3) the ways different types of social institutions shape our lives, and 4) how different social institutions evolve over time. We also added survey questions specific to the CURE research and research topic: 5) how women’s bodies are medicalized, 6) how different forces shape birth decisions, 7) the strengths and weaknesses of qualitative social science data, and 8) the strengths and weaknesses of quantitative social science data. Due to small sample sizes, the 5-point scale was recoded into three categories: “not at all” was coded as “1”, “slightly and “moderately” were combined and coded as “2”, and “very” and “extremely” were combined, coded as “3.”

We analyzed all variables from the third survey section for each course that were added by the course instructors prior to the course that asked, “How confident are you currently to conduct the following activities?” The variables measured were mostly CURE specific. The 2017 survey included: 1) collaborate with community organizations to conduct research, 2) select appropriate research methods to conduct research, 3) transcribe interview data, 4) code interview data, 5) interpret data results, and 6) collaborate with others to meet research goals. Due to small sample sizes, the 5-point scale was recoded into three categories: “not at all” was coded as “1”, “slightly and “moderately” were combined, coded as “2”, and “very” and “extremely” were combined, coded as “3.” The 2018 survey included: 1) collaborate with community organizations to conduct research, 2) distinguish between primary and secondary sources, 3) conduct an academic literature review, 4) understand academic studies, 5) synthesize academic information, 6) write in a scientific manner, 7) distinguish between medical and sociological ideas, and 8) create a video presentation. Due to small sample sizes, the 5-point scale was recoded into three categories: “not at all” was coded as “1”, “slightly and “moderately” were combined, coded as “2”, and “very” and “extremely” were combined, coded as “3.”

For our 2017 CURE student product assessment, we measured outcomes from end-of-semester papers in two ways. The assignment required students to code and independently identify and analyze three sociological themes from four interviews. The paper began as a draft earlier in the semester, where the postdoc provided feedback to students, and students had class time to work on their papers to also receive instructor and peer feedback. Thus, the final paper relied on student knowledge gained during the semester on both general sociological ideas and specialized sociology of birth knowledge, as well as qualitative coding knowledge. We utilized a grading rubric (see Appendix B) for this measurement, which was supplied to students with the paper assignment. This grading rubric was used to assess student’s successful completion of the paper—exhibiting sociology knowledge and qualitative coding skills. This measure goes beyond the frequency and variety of integrated themes and records the number of students who earned scores of “excellent” and “satisfactory.”

The study authors had little training in effectively creating rubrics and acknowledge that the most effective rubrics communicate both criteria and performance expectations (Brookhart & Chen, 2015). Unfortunately, ours lacked detailed performance expectations across the scale from “Excellent” to “Needs Improvement.” However, it was discussed orally with students as a class, where the postdoc communicated that a “Satisfactory” grade reflected identification and presentation of the topics under assessment, while “Excellent” would be earned in response to insightful and/or comprehensive synthesis; this follows the focus of the Association of American Colleges and Universities Value Rubrics (Rhodes, 2010).

Analysis.

All survey analyses were performed in SPSS 24 (SPSS 2016). Numbers and percentages for each course were calculated for student demographics (self-reported gender, ethnicity, and college classification). For comparing the pre- and post-test content knowledge data for the 2017 and 2018 CUREs, differences between scores were assessed to be normal and paired samples two-tailed t-tests were run to assess if students reported more, or less, knowledge by the end of the course. The effect size was calculated as: r = √t² /t²+df (see Tables 2 and 3).

Table 2:

Two-tailed paired sample t-tests comparing pre- and post-knowledge in 2017 CURE students

	Pre-test Scores		Post-test Scores		Paired Samples t Test
	Mean	SE	Mean	SE	t(df)	n
Analyzing data	2.65	.119	2.77	.106	−.808(16)	17
Women medicalized	1.94	.181	2.88	.081	−4.693(16)**^	17
Birth	1.94	.193	2.94	.063	−4.899(15)**^	17
Qualitative methods	2.00	.192	2.77	.106	−3.490(16)**^	17
Quantitative methods	2.00	.192	2.77	.106	−3.490(16)**^	17
Collaborate community	2.41	.150	2.47	.125	−.293(16)	17
Research methods	2.47	.125	2.71	.114	−1.725(16)	17
Transcribe	2.18	.177	2.77	.146	−2.787(16)**^	17
Code	1.88	.190	2.53	.125	−2.864(16)**^	17
Interpret data	2.41	.173	2.65	.120	−1.289(16)	17
Collaborate others	2.47	.151	2.71	.114	−1.289(16)	17
Theory	1.94	.158	2.71	.114	−4.190(16)**^	17
Social Construction	2.24	.161	2.82	.095	−3.050(16)**^	17
Social Institutions	2.06	.160	2.94	.059	−6.061(16)**^	17
Institutions Evolve	2.00	.149	2.82	.095	−5.339(16)**^	17

^*statistically significant at p ≤ 0.05

^**statistically significant at p ≤ 0.01

^{^}large effect above 0.5 level

Table 3:

Two-tailed paired sample t-tests comparing pre- and post-knowledge in 2018 CURE students

	Pre-test Scores		Post-test Scores		Paired Samples t Test
	Mean	SE	Mean	SE	t(df)	n
Create Video	2.08	.149	2.25	.179	−.804(11)	12
Women medicalized	1.75	.179	3.00	.000	−6.966(11)**^	12
Birth	2.08	.149	3.00	.000	−6.167(11)**^	12
Qualitative methods	1.75	.179	2.92	.083	−5.631(11)**^	12
Quantitative methods	1.83	.207	2.92	. 083	−4.733(11)**^	12
Collaborate community	2.08	.193	2.75	.131	−3.546(11)**^	12
Primary v. Secondary Sources	2.42	.149	2.83	.112	−2.803(11)**^	12
Conduct Lit Review	2.25	.179	2.92	. 083	−3.546(11)**^	12
Understand Academic Studies	2.58	.149	2.92	. 083	−2.345(11)*^	12
Synthesize Academic Info	2.33	.142	2.75	.131	−2.803(11)**^	12
Write Scientifically	2.58	.149	2.83	.112	−1.393(11)	12
Distinguish medical v. social	2.17	.167	2.75	.131	−3.023(11)**^	12
Theory	1.5	.151	2.67	.142	−5.631(11)**^	12
Social Construction	2.00	.123	2.92	.083	−6.167(11)**^	12
Social Institutions	2.09	.163	3.00	.000	−5.590(10)**^	11
Institutions Evolve	2.08	.083	2.75	.131	−4.690(11)**^	12

^*statistically significant at p ≤ 0.05

^**statistically significant at p ≤ 0.01

^{^}large effect above 0.5 level

To assess whether other indicators of academic experience might account for some of the variability in our post-course data, we ran log linear analyses (all our variables are discrete). The indicators of academic experience that we had in our 2017 data were “class,” (freshman, sophomore, junior, senior) and “Before this semester, had you participated in mentored research with a professor or through a program?”, “have you engaged in research in the past,” and “are you currently engaging in research.” The 2018 data only had 2 data points: “class,” (freshman, sophomore, junior, senior) and “Before this semester, had you participated in mentored research with a professor or through a program?” We ran these variables with each post-course data variable.

To compare the 2017 CURE to each of the three independent courses, we used a 2X2 contingency table and ran Pearson’s chi-square tests. When cell sizes were less than 5, a Fisher’s exact test was used. Means and standard errors for each variable are reported, as well as significance levels from the Pearson’s chi-square tests. Data are reported in Table 4.

Table 4:

Cross-Tabulation results of Pearson’s Chi-Square Tests Comparing the 2017 CURE to Other 3 Intro Classes in Student Self-reported Knowledge Questions

	CURE (n=19)	Course 1 (n=17)	Course 2 (n=35)	Course 3 (n=76)
	Mean (SE)	Mean (SE)	Mean (SE)	Mean (SE)
Research/ CURE Topical Questions
Analyze data	2.74(.104)	2.39(.118)*	2.51(.091)	2.44(.066)
Women medicalized	2.90(.072)	2.82(.095)	2.22(.106)**	2.13(.069)**
Birth	2.90(.072)	2.72(.109)	2.47(.093)**	2.17(.074)**
Qualitative	2.79(.096)	2.11(.076)**	2.41(.083)**	2.02(.064)**
Quantitative	2.79(.096)	2.12(.090)**	2.39(.082)**	2.02(.075)**
Collaborate	2.42(.116)	2.11(.111)	2.34(.100)	2.21(.068)
Research methods	2.63(.114)	2.17(.121)*	2.42(.094)	2.29(.065)
Transcribe	2.79(.122)	2.06(.127)**	2.40(.095)**	2.12(.067)**
Code	2.52(.118)	1.78(.129)**	2.06(.108)*	1.81(.067)**
Interpret data	2.63(.114)	2.22(.129)*	2.60(.084)	2.30(.073)
Collaborate others	2.74(.104)	2.28(.135)*	2.51(.086)	2.34(.076)
Sociology Questions
Major sociology theories	2.68(.110)	2.33(.114)*	2.20(.078)**	2.08(.062)**
Social Constructions	2.84(.086)	2.67(.114)	2.78(.070)	2.42(.064)**
Social Inst shape lives	2.95(.053)	2.72(.108)	2.83(.063)	2.58(.064)*
Social Inst evolve	2.84(.086)	2.56(.120)	2.50(.093)*	2.38(.062)**

^*significant at p ≤ 0.05

^**significant at p ≤ 0.01

To analyze the 24 end-of-semester papers from the 2017 CURE, we performed a content analysis to record the three sociological themes explicitly mentioned in each paper where we counted the frequency and thematic categories of themes. Based on instructor-provided suggestions, most students created an essay where the three separately identified themes were the body of the paper. These themes were often easy to identify due to student use of headers and/or because students offered a definition of the theme prior to showcasing its presence in the interview data. On average, the papers were 6 pages long. We also performed a content analysis of the graded rubrics by counting the frequency of student submissions that met or exceeded expectations in three categories of the rubric. Success was determined by earning an “excellent” or “satisfactory” in the first three categories, which were the main categories (see Appendix B).

Results

Demographic results are presented for both CURE courses and the three control group courses in Table 1.

Paired t-test results comparing 2017 CURE pre- and post-data reveal negative t values on all variables, which indicate that the first condition (pre-test) had a smaller mean than the second condition (post-test). On average, students reported significant increased knowledge in the following content and skill areas: the medicalization of women’s bodies, how different social forces shape birth decisions, the strengths and weaknesses of qualitative and quantitative data, transcribing and coding interview data, the four major sociological theoretical perspectives, the social construction of social statuses, the ways different social institutions shape our lives, and how different social institutions evolve over time. Students did not report knowledge gain in the following: analyzing data for patterns, collaborating with community organizations to conduct research, interpreting data results, selection of appropriate research methods, and collaborating in a coordinated fashion with others to meet research goals (see Table 2).

Paired t-test results comparing 2018 CURE pre- and post-data reveal negative t values on all variables, which indicate that the first condition (pre-test) had a smaller mean than the second condition (post-test). On average, students reported significant increased knowledge in the following content and skill areas: the medicalization of women’s bodies, how different social forces shape birth decisions, the strengths and weaknesses of qualitative and quantitative data, collaborating with community organizations to conduct research, distinguishing between primary and secondary sources, conducting an academic literature review, understanding academic articles, synthesizing academic information, distinguishing between medical and sociological ideas, the four major sociological theoretical perspectives, the social construction of social statuses, the ways different social institutions shape our lives, and how different social institutions evolve over time. Students did not report knowledge gain in creating a video presentation or writing in a scientific manner (see Table 3).

The log linear analyses of “class,” (freshman, sophomore, junior, senior), “Before this semester, had you participated in mentored research with a professor or through a program,” “have you engaged in research in the past,” and “are you currently engaging in research” with each 2017 course post-data variable and “class,” (freshman, sophomore, junior, senior) and “Before this semester, had you participated in mentored research with a professor or through a program,” with each 2018 course post-data variable revealed no significant values for any variable interactions.

Comparing the post-test results from all four courses, results from the Pearson’s Chi Square and Fisher’s Exact tests show that there were various differences between the 2017 CURE students and the students enrolled in other Introduction to Sociology classes. The CURE students reported significant increased knowledge compared to all three other classes in several areas: the strengths and weaknesses of qualitative and quantitative data, transcribing and coding interview data, and the four major sociological theoretical perspectives. CURE students did not report significant increased knowledge compared to all three other classes in collaborating with community organizations to conduct research. In comparing the CURE to one or two of the other three classes, CURE students showed significant increased knowledge in analyzing data for patterns, how women’s bodies are medicalized, how different forces shape birth decisions, selection of appropriate research methods, interpreting data results, collaborating in a coordinated fashion with others to meet research goals, the social construction of social statuses, the ways different social institutions shape lives, and how different social institutions evolve over time (see Table 4).

Our content analysis of student papers revealed that students collectively identified 32 themes across the four interviews. The most frequently occurring themes were medicalization (12 instances), socialization (8 instances), and standardization (7 instances). From our analysis of graded rubrics, 21 of 24 students received an “excellent” or “satisfactory” on all three main components (number of, appropriate application, and demonstrable understanding of sociological themes).

Discussion

In the last decade, HIPs have been increasingly incorporated into undergraduate courses, and the rapid implementation of CUREs means that the scholarship of teaching and learning cannot keep up; best assessment practices for this teaching intervention have yet to be identified. Currently, CUREs often examine students’ self-reported content knowledge and technical skills, often by comparing pre- and post-surveys (Corwin et al., 2015). Using this method, we found significant gains in student self-assessed knowledge between the pre-and post-course surveys in regards to a variety of specific and general sociology content and research skills.

However, there were no significant self-reported knowledge gains in a few content and skill areas. In the 2017 course, this included: analyzing data for patterns, collaborating with community organizations to conduct research, interpreting data results, selection of appropriate research methods, and collaborating in a coordinated fashion with others to meet research goals. While we were disappointed that students did not report gains in these areas, the lack of significance for analyzing data for patterns could be due to their high pre-test scores in these areas, which may indicate the students’ previous exposure to these skills in other courses or may be related to the type of student who is interested in such a course. Question wording may also be an issue because we discussed “coding” specifically in the course and did not necessarily call this activity “analyzing data for patterns” or “interpreting data results.” The non-significant scores for “selection of appropriate research methods” may be due to the instructors selecting one research method to delve into and pursue, rather than provide an overview of multiple methods.

The lack of significant self-reported knowledge gain in the 2017 course for collaborations may be due to the class having had little collaborative contact with the non-profit and students did not collaborate with one another on certain aspects of research such as writing and presenting the final project. Interestingly, the 2018 course did report a significant knowledge gain in collaborating with community organizations. This might be because they knew that their literature reviews were producing data directly for the documentary; further, the worksheets required five separate submissions to the nonprofit that may have made students feel like they were participating in a more substantial collaboration. In comparison, the 2017 students delivered the video transcriptions collectively, and while the transcripts were helpful to the production team, they were not directly included in the film.

For the 2018 course, students did not report a significant knowledge gain in only two areas: creating a video presentation and writing in a scientific manner. Regarding the former, perhaps this generation of students perceives themselves to be capable with the use of media and technology. For the latter, the course contained a large number of BUILD students, who take workshops on scientific writing, or it might be due to the large amount of health science majors.

Most CURE assessments examine student outcomes in a single course (e.g., Atkinson et al., 2006; Bartholomay, 2018; Cuthbert et al., 2012), but here we compare self-reported CURE student outcomes to three other non-CURE courses (similar to methodologies used by McConnell & Marton, 2011; McKinney & Busher, 2011). We found that the CURE students reported significant increased knowledge over students in all three other classes in the strengths and weaknesses of qualitative and quantitative data and transcribing and coding interview data. These are skills that the CURE offered extensive practice in: students engaged with qualitative data, including transcribing a segment of an interview and identifying sociological themes across several interviews. More surprising is the CURE students’ report of an increased knowledge of the four major sociological perspectives. Considering students’ usual trepidation or disinterest in social theory (Macheski et al., 2008; Pelton, 2013), we hope the gains are because the research curriculum made theory accessible and relevant.

Many CURE assessments do not examine student products, whereas we integrated an analysis of the 2017 CURE end-of-semester student papers that revealed the application of 32 different sociological themes. The most commonly integrated themes—medicalization, socialization, and standardization—and student high scores on the 3 sociology content- and research-skill rubric categories reflect that students chose various directions for their papers where they drew on both more general and specialized sociology content knowledge from throughout the semester. The papers also reflected an ability to code qualitative data—one of our research method goals.

While these particular student outcomes may be perceived as easier to assess, Corwin and colleagues (2015) advise that novice implementers and evaluators of CUREs begin by assessing these short-term outcomes. However, as urged by numerous CURE scholars (Brownell & Kloser, 2015; Shortlidge & Brownell, 2016) we went beyond a single measurement of student enjoyment or interest, instead assessing content- and research-specific objectives, including numerous content areas and skills, and comparing students’ sense of their mastery with those of students across similar classes.

We cannot claim that participation in our CUREs was solely responsible for our students’ increased knowledge and skills; students acquire knowledge from other courses and outside sources. The University of Texas at El Paso is supportive of CUREs, evident by the diversity of workshops available for faculty to develop CUREs and the resources available for students to enroll in numerous CUREs and to present their undergraduate research projects at national conferences. The institutional culture of CUREs, albeit less so in the social sciences, is a variable likely influencing our data. The interplay of this CURE culture and students’ previously acquired knowledge is hard to assess.

Our study had several limitations common in teaching and learning scholarship, and especially in UR assessment: the lack of a randomized control group, small sample sizes, the potential for bias in objectively assessing student papers, and a lack of longitudinal data. It is possible that we could have increased the objectivity of student paper assessment with a more robust rubric. The small sample sizes and limited survey questions restricted more complex multivariate analyses where we might have been able to better analyze the relationships between sociology-content versus research-related content or between methodological and theoretical knowledge.

Although scholarship has not yet established the most effective assessment methods (Brownell & Kloser, 2015), we recommend that future studies address these limitations with inclusion of larger sample sizes and more student outcome measures for both experimental and control groups. Based on the 2017 and 2018 differences in self-reported collaboration skills, we also encourage closer analysis of how students define collaboration and if particular models (e.g., face-to-face collaboration) produce stronger senses of contribution. Long-term outcomes such as student perception of career advising and development might prove particularly valuable. We also recommend assessment of variables such as retention and student engagement, results for which HIPs are known. CUREs are complex, with diverse formats, content, class size, and disciplinary homes, and meaningful assessments require multiple long-term strategies—especially strategies that compare CUREs to other high-impact pedagogical practices such as service-learning.

Nonetheless, our research provides several valuable insights. First, it indicates that diverse curricula can ground introductory courses, where students can handle fairly advanced course themes (e.g. the medicalization of birth) that organize basic disciplinary content. Moreover, our data show that the CURE format—even in a general education course—can successfully expose students simultaneously to basic and specialized disciplinary content and to research processes.

Second, we believe our CURE successes can be translated into most classrooms, with most types of students, because our student population faces unique challenges for academic success. Eighty percent of UTEP’s students identify as Hispanic, and an additional 5% of UTEP students identify as Mexican nationals. Fifty percent are first-generation college students. Because UTEP is committed to providing access to educational opportunity for the people of the USA-Mexico border, many students enter UTEP with English language barriers and diverse levels of preparedness for college. Most students have low socioeconomic status; 46% of UTEP’s financial aid applicants have a family income of $20,000 or less. Many students have other commitments; on average, our students provide 20 hours of dependent care per week (Collins et al., 2017). These statistics mean that most UTEP students carry burdens of disadvantage and discrimination. Even with these social and structural hurdles, our students reported gaining numerous skills from the CURE course, and underrepresented students like ours may glean particular advantages from this participation because UR advantages were found to be particularly salient for first-generation college students (Ishiyama, 2002) and Hispanics compared to other racial/ethnic groups (Russell et al., 2007).

Conclusion

Our CURE students mastered general and specialized sociological content, such as an understanding of the medicalization of childbirth and research skills, which reveals that having specific foci in introductory courses does not come at the expense of learning more general disciplinary knowledge. Moreover, perhaps specific concentrations may aid students in their learning of general disciplinary knowledge.

Our data illustrate the pedagogical power of a CURE in an introductory course. We successfully created freshman-level course curriculum that moves students from consuming science (e.g. listening to lectures) to active learning: critically using and producing science, where students undertake tasks and think about what they are doing and why; a distinction that celebrates students’ knowledge construction and intellectual contributions (Auchincloss et al., 2014). These findings indicate that CUREs may offer important benefits to students and to academic disciplines. Research-related skill development prepares students for success in future college courses, graduate school, the workforce, and/or effective democratic participation (Hartmann, 1990; Korgen & White, 2014; Senter et al., 2015). Further, course curriculum that moves students from consuming to contributing to social science will reinforce the empiricism of the discipline.

CUREs may seem out of reach for some instructors who may be hesitant to ask students with little or no disciplinary training and/or research experience to participate in UR. However, our data show the effectiveness of this pedagogy, even in a freshman-level class of students with varying levels of college preparation. We believe our approach can be adopted in a variety of contexts, and we encourage interested instructors to create curriculum that reflects student abilities (Halonen et al., 2003). Although CUREs may seem a time-consuming intervention, they may be less time-consuming compared to other UR practices where individual students or small teams work with faculty. Thus, we encourage instructors to start small; avoid a complete research project and instead integrate a few scaffolded research steps into the semester’s curriculum. Instructors should consider this HIP for any introductory and topical course because our data indicate that CUREs can enhance research- and content-related skills and knowledge.

Appendix A. Literature Review Worksheet

Appendix B. Introduction to Sociology: Final Paper Grading Rubric

	Excellent 3 points	Satisfactory 2 points	Needs Improvement 1 point	Absent 0 points
Number of sociological themes/ability to recognize multiple sociological issues Theme one:____________ Theme two:___________ Theme three:__________
Appropriate application of sociological themes to examples
Demonstrates understanding of sociological ideas
Number of interviews integrated into paper
Grammar/mechanics
Proper citation style
Paper organization