Modeling Course-Based Undergraduate Research Experiences: An Agenda for Future Research and Evaluation

The authors review relevant literature to determine established and predicted outcomes of course-based undergraduate research experiences (CUREs) and then use this information and social learning theory to model how students may realize desired short-, medium-, and long-term outcomes. This work has implications for future research and evaluation of CUREs.

Abstract

Course-based undergraduate research experiences (CUREs) are being championed as scalable ways of involving undergraduates in science research. Studies of CUREs have shown that participating students achieve many of the same outcomes as students who complete research internships. However, CUREs vary widely in their design and implementation, and aspects of CUREs that are necessary and sufficient to achieve desired student outcomes have not been elucidated. To guide future research aimed at understanding the causal mechanisms underlying CURE efficacy, we used a systems approach to generate pathway models representing hypotheses of how CURE outcomes are achieved. We started by reviewing studies of CUREs and research internships to generate a comprehensive set of outcomes of research experiences, determining the level of evidence supporting each outcome. We then used this body of research and drew from learning theory to hypothesize connections between what students do during CUREs and the outcomes that have the best empirical support. We offer these models as hypotheses for the CURE community to test, revise, elaborate, or refute. We also cite instruments that are ready to use in CURE assessment and note gaps for which instruments need to be developed.

INTRODUCTION

Course-based undergraduate research experiences (CUREs) are gaining attention as an effective way to engage students in doing research early in their college careers. CUREs offer several advantages over traditional lab courses and research internships. They involve many undergraduates in science research at one time, and all students who enroll in a course are able to participate (Corwin Auchincloss et al., 2014). Studies of students who complete CUREs report many of the gains associated with undergraduate participation in research internships (Shaffer et al., 2010; Rowland et al., 2012; Jordan et al., 2014). For example, similar to studies of undergraduate research experiences, studies of CUREs report student gains in research skills, self-efficacy, and intent to persist in science (Lopatto et al., 2008; Shaffer et al., 2010; Harrison et al., 2011; Rowland et al., 2012; Corwin Auchincloss et al., 2014; Jordan et al., 2014).

Most research and evaluation of CUREs have focused on documenting student outcomes. What remains largely unaddressed is which aspects of CUREs lead to desirable student outcomes. In other words, what are the causal mechanisms underlying the efficacy of CUREs? We hypothesize that CUREs allow students to participate in a unique combination of activities that result in progressive achievement of diverse cognitive, psychosocial, and behavioral outcomes. We posit that using experimental concepts to model pathways, or directional relationships between student activities and outcomes, will allow us to identify the critical features that should be included in the design of all CUREs and gain insight into the latitude we have in effectively implementing CUREs.

Logic models are often used to illustrate relationships between program activities and resulting outcomes and define questions for evaluation and research (Westat et al., 2010). However, the simplified lists used in logic models do not capture relationships and feedback loops or the high level of impact certain activities may have. The systems approach to evaluation (Urban and Trochim, 2009; Urban et al., 2014) expands on a traditional logic model by hypothesizing direct and directional relationships among activities and outcomes that evolve over time. Specifically, the systems approach to evaluation does the following:

• Allows for description and visualization of the many diverse connections among activities and outcomes

• Displays multiple pathways students may take to achieve a single outcome

• Accommodates the complexity of education programs by exposing feedback loops and multiple connections among outcomes

• Reveals which outcomes are likely to be most informative by pinpointing where pathways converge

This systems approach to evaluation has been used to design evaluation plans for several existing or planned programs (Urban and Trochim, 2009). We used this approach to generate a broad model of what is known about CURE instruction based on research to date and to identify leverage points at which more information or measures about CUREs are needed. We propose that the resulting model can be used or adapted to generate evaluation plans and an agenda for research on CUREs.

As context for identifying causal mechanisms of CUREs—what makes CUREs work for students—it is useful to consider theoretical perspectives on learning. Social learning theory (Bandura, 1971; Vygotsky, 1978), which has both cognitive and social elements, is particularly useful for considering what makes CUREs beneficial for students. This theory frames learning as a cognitive process, reliant on mental processing and construction, which occurs in a social context through observing and interacting with others. Situated-learning theory is a form of social learning theory that emphasizes the importance of situating learning in an authentic activity, context, and culture (Brown et al., 1989; Lave and Wenger, 1991). For the most part, CUREs are designed to align with the tenets of situated learning. Students do the work that scientists do (e.g., ask questions, design studies, collect and analyze data, build models) in the context of a real scientific problem or question, in which the solution or answer is unknown.

A hallmark of situated learning is “legitimate peripheral participation,” meaning the learner does tasks that experts consider meaningful (“legitimate”) to the work of the discipline. The tasks become increasingly important, moving along a continuum from peripheral to central, as the learner develops expertise (Lave and Wenger, 1991). Students within CUREs are legitimate participants in scientific research, because their actions contribute to achievement of research goals. However, students generally do not perform more central tasks that determine the overall direction and scope of research. For example, in many CUREs, instructors do the central work of posing overarching research questions, which helps steer students in scientifically fruitful directions. Students then do the very real (legitimate) but more peripheral work of collecting and analyzing data to answer those questions. As students develop more expertise, they take on increasingly central roles, for example, by designing or choosing methods and eventually posing research questions themselves.

Working with more expert individuals is another important element of situated learning, because experts model how to do the work, provide feedback about how to improve, and validate the legitimacy of students’ experiences and accomplishments. Situated-learning theory emphasizes the importance of immersion in real culture—in the case of CUREs, the culture of science, which is characterized by scientific ways of thinking, behaving, and working. Through immersion in the culture of science, students not only have opportunities to see science and scientific thinking in action but also to develop in terms of their scientific identity and sense of belonging to the broader scientific community. CUREs are thought to more closely reflect the culture of science through immersion in the process of science when compared with traditional lab-learning experiences (Spell et al., 2014) According to situated-learning theory, we would predict that participating in CUREs would lead to students developing a stronger scientific identity and sense of belonging to the scientific community. We would also expect to observe positive relationships between the degree to which students interact with more expert individuals and the positive outcomes they achieve.

To date, most program evaluation of CUREs has focused on high-stakes outcomes, such as student completion of a science, technology, engineering, and mathematics (STEM) major or matriculation into a science graduate program. Yet the causal elements of these outcomes have not been systematically identified (Corwin Auchincloss et al., 2014). Although it is important to continue to measure these outcomes, this should be paired with efforts to understand how they are achieved. Building and testing models, such as the ones we present here, will help to direct and improve research and evaluation of CUREs as the biology education community shifts focus beyond “What works?” to “How does it work?” Thus, the purpose of this work is to provide the biology education community with a working model that can be used, adapted, and revised to drive future research on CUREs and assist in CURE evaluation efforts. Our model is intended to show how CUREs “work”—in other words, what it is that students do during CUREs that may lead to the outcomes they experience. To build this model, we:

1. reviewed relevant studies to determine what is known about student outcomes from undergraduate research experiences and how they have been measured;

2. evaluated the studies related to each outcome to determine the extent of empirical support for each outcome—in other words, whether the outcome is probable, possible, or proposed;

3. connected learning activities with the outcomes they are likely to influence to produce several small models of how CUREs work (“mini-models”);

4. integrated our mini-models into a single, large model to show student progression from activities to short-, medium-, and long-term outcomes; and

5. identified points at which multiple pathways converge or diverge, called “hubs” (see Urban and Trochim, 2009), which we identify as key points for future research and evaluation of CUREs.

METHODS

Step 1. Review of Relevant Literature to Identify Potential CURE Outcomes

We started by assembling a comprehensive set of papers describing studies of CUREs and undergraduate research internships to identify the range of student outcomes. We included studies of internships, because a number of CUREs were developed as a mechanism for involving students in research when sufficient internships were not available (Dolan et al., 2008), and CURE outcomes have been compared with internship outcomes (Lopatto et al., 2008; Shaffer et al., 2010). We included literature cited in a recent report on CURE assessment (Corwin Auchincloss et al., 2014) and in two comprehensive reviews of undergraduate research (Seymour et al., 2004; Laursen et al., 2010). We also searched Google Scholar using the terms “course based undergraduate research experience,” “undergraduate research experience,” “undergraduate research internship,” and “class undergraduate research“ for additional studies published within the past 5 yr. All studies included in our analysis met three criteria. They 1) were published within the past 25 yr, 2) presented data gathered in undergraduate settings, and 3) actively examined student outcomes of a CURE or a research internship (i.e., they had elements of experimental design and provided more than anecdotal or descriptive evidence of outcomes).

For the purposes of this paper, we defined a CURE as “a course in which students are expected to engage in science research with the aim of producing results that are of interest to a scientific community.” This definition is purposely broad, since the processes of science research conducted in laboratory courses vary widely, and we have not yet reached consensus on the essential elements that constitute a CURE or make CUREs successful. Laboratory learning experiences described in these studies were deemed CUREs if they reported that the students performed externally relevant scientific research at any point during the course and for any duration. We identified 14 studies on CUREs that met these criteria and 25 studies on undergraduate research internships. In general, the duration of research performed in the identified CURE studies spanned the majority of the course, involved students in multiple science practices (e.g., collecting data, analyzing data, reporting on data), and involved a minimum of 15 students.

We then generated a list of potential CURE outcomes based on evidence or hypotheses presented in this literature sample. We tested the comprehensiveness of this literature sample by searching for additional studies using the same terms described above. We searched two digital libraries (ERIC and JSTOR) and a crowd-sourced database (Mendeley) and reviewed the resulting studies for additional outcomes. We found one new study on CUREs, two studies on internships, and no new outcomes. Thus, we deemed our literature sample sufficient for building models of CUREs. When conducting step 2, we included all studies that fit the four criteria noted earlier.

Step 2. Analysis to Determine the Level of Empirical Support

To determine which outcomes had sufficient empirical support to be included in a model, we designated outcomes as “probable,” “possible,” and “proposed,” using a framework similar to that used by Grayson et al. (2001). Probable outcomes were 1) investigated in a minimum of three studies, 2) measured in at least three different student populations (i.e., groups of students), 3) measured in at least three different courses or curricula, and 4) assessed using at least two different methods or instruments. Possible outcomes were 1) investigated in a minimum of two studies, 2) investigated in two different populations, 3) measured in at least one course or curriculum, and 4) assessed using at least one method. Proposed outcomes were investigated only in a single instance or were supported by learning theory but were not present in the literature sample. Purely descriptive accounts and anecdotes were not used to support these designations. We chose to represent outcomes using discrete ordinal designations in order to determine which outcomes had sufficient support to include in our models. We did not identify any outcomes as certain in our analysis, because current research on CUREs is not sufficient for this designation. We used probable or possible outcomes in subsequent modeling steps.

Step 3. Alignment of Activities and Outcomes to Generate Mini-Models

We connected activities to outcomes to generate several mini-models (Figures 1–3). Following on the systems approach to evaluation (Urban and Trochim, 2009), we parsed outcomes into short, medium, and long term. We defined short-term outcomes as those that can be achieved immediately during a CURE and medium-term outcomes as those that result primarily from achievement of short-term outcomes and occur later, at the end of a CURE or after CURE participation. We defined long-term outcomes as those that result from short- and medium-term outcomes and can only be measured after the CURE is completed. These are often highly valued goals of science education and research experiences in particular, such as development of students’ scientific identity, student persistence in science, and increased public science literacy. All modeling was performed using the Netway tool for program evaluation and planning (Cornell Office of Research and Evaluation [CORE], 2009)

Figure 1.

Knowledge and skills mini-model. Arrows represent positive directional relationships between activities and outcomes.

Figure 2.

Communication and collaboration mini-model. Arrows represent positive directional relationships between activities and outcomes.

Figure 3.

Ownership mini-model. Arrows represent positive directional relationships between activities and outcomes.

To support the presence and direction of connections in our models, we used social learning theory (described above), investigations of STEM retention outcome relationships (e.g., Estrada et al., 2011), and hypotheses proposed in the CURE literature. We connected activities to short-term outcomes in mini-models and then connected short-to medium-term outcomes. Collectively, the mini-models include the majority of the probable or possible outcomes of CUREs (Table 1 and Figures 1–3).

Table 1.

Support for CURE outcomes based on a review of relevant CURE literature^a

^aGreen shading indicates probable outcomes, yellow shading indicates possible outcomes, and gray shading indicates proposed outcomes.

Step 4. Integration of Mini-Models into a Comprehensive Model

Using the Netway (CORE, 2009), we combined the mini-models into a single large model that depicts the relationships among all activities and outcomes and expands the model to include long-term outcomes (Figure 4). To accomplish this, we used learning theory to hypothesize additional connections among short- and medium-term outcomes and between medium- and long-term outcomes (Brown et al., 1989; Lave and Wenger, 1991).

Figure 4.

Large CURE model. Arrows represent positive directional relationships between activities and outcomes. Bold black arrows indicate new connections between activities and short-term outcomes in the mini-models. Bold blue arrows indicate new connections between medium- and long-term outcomes.

Step 5. Identification of “Hubs”

Hubs are points, or “nodes,” in a model at which multiple paths converge (Urban and Trochim, 2009). Hubs are important, because many pathways must pass through a hub for longer-term outcomes to be achieved. Thus, students would be expected to achieve at least one hub outcome in order to achieve longer-term outcomes. For the present study, we designated as hubs those outcomes that had a minimum of six relationships, or directional connections, with other outcomes or activities (Figure 4).

RESULTS

Our review and analysis revealed eight probable, seven possible, and four proposed outcomes (Table 1). More than half of the outcomes that may result from CURE participation are only possible or proposed in the CURE literature.

Level of Empirical Support for Outcomes

Probable Outcomes.

Increases in content knowledge, technical skills, analytical skills, scientific self-efficacy, project ownership, and career clarification are all probable outcomes, according to our criteria. In addition, two studies tracked students after their participation in CUREs and found that they pursued science graduate degrees or science-related careers at a higher rate than other science majors (Bascom-Slack et al., 2012; Hanauer et al., 2012). A third study showed that CURE students persisted in science majors at a higher rate than students who completed other lab courses (Jordan et al., 2014). In three additional studies, students reported increased intentions to continue doing research, including pursuing graduate education. This is important, because several researchers have found that students’ educational aspirations are one of the strongest predictors of enrollment in a graduate degree program (Sewell et al., 1969; Heller, 2001; Mullen et al., 2003; Walpole, 2003; Nevill and Chen, 2007; Eagan et al., 2013). We grouped the outcomes of continuing in a science major, entering a graduate program, or pursuing a science career, including students’ intentions to do so and their actual behavior, under the broad heading of “persistence in science.” A number of studies showed evidence of validation of students and their contributions by a broader scientific community, which we call “external validation.” Authorship or acknowledgment in a peer-reviewed journal, presentation at a professional conference, and acceptance of results into a national database all constitute forms of external validation reported in CURE studies.

These results raise several points for consideration. First, probable outcomes with the best support were described in seven unique instances (i.e., different courses or distinct groups of students). Although this met our criteria, future work should focus on investigating the prevalence of these outcomes across a broader set of CUREs that represent the diversity of instruction and students. Second, in a few CURE studies, skill gains were measured through tests, observation, or instructor evaluations of student work. In most, however, students reported gains in their skills. This can be problematic, because novices tend to overestimate their own knowledge and skills, a phenomenon described as the Dunning-Kruger effect (Boud and Falchikov, 1989; Falchikov and Boud, 1989). More direct measurements of technical and analytical skill development in CUREs are needed to corroborate current findings. Third, certain CUREs may be subject to the same recruitment or selection bias as research internships, in that they only involve a small group of high-achieving or research-interested students (Brownell et al., 2013). Moving forward, it will be important to study the impact of CURE instruction using approaches that control for student-level variables and nonrandom assignment (Theobald and Freeman, 2014; Beck and Bliwise, 2014), such as propensity score matching (e.g., Schultz et al., 2011) or regression discontinuity design (e.g., DesJardins et al., 2010, 2014).

Possible Outcomes.

We categorized increased communication skills, collaboration skills, motivation to pursue science learning, and enhanced science identity as possible outcomes of CUREs, because they were documented in at least two unique instances (i.e., different courses and different student groups). We classified descriptions of successful peer–peer instruction (Shaffer et al., 2010) and productive scientific discourse with peers (Alkaher and Dolan, 2014) as “positive interaction with peers” and also categorized this as possible. Two studies reported increases in the following two outcomes: 1) tolerance for obstacles and 2) an increased sense of belonging to a larger community (Shaffer et al., 2010; Jordan et al., 2014). Because these studies examined two distinct CUREs, we categorized these outcomes as possible. However, both outcomes were measured with a single question each, using a single instrument for which validity and reliability information is not available. Moving forward, it will be important to establish the validity and reliability of outcome measures and demonstrate outcomes with multiple kinds of evidence and within many different kinds of CUREs.

Proposed Outcomes.

In our analysis, we found only single instances of increased access to faculty (Alkaher and Dolan, 2014), increased access to mentoring functions (Hanauer et al., 2012), enhanced understanding of the nature of science (Russell and Weaver, 2011), and development of self-authorship (Alkaher and Dolan, 2014). These are notable shortcomings, because the development of meaningful relationships with faculty, access to mentoring, and self-authorship are predictors of persistence in science and career decision making, especially for students who are underrepresented in science (Maton et al., 2000; Packard, 2004; Creamer and Laughlin, 2005; Eagan et al., 2010, 2011). More evidence is needed to elucidate whether these outcomes are replicable in particular CUREs or in CUREs in general.

Mini-Models: Connecting Activities to Outcomes

We used both probable and possible outcomes to construct mini-models of connections between activities and outcomes of CUREs. Although there is theoretical support for many of the relationships, the connections we show indicate that most have not been empirically investigated. We intend this set of models to illustrate how to go about pathway modeling of CURE instruction; they are not the only models possible for CUREs or as a comprehensive set of such models.

We chose to focus on six activities that students typically engage in during CURE instruction: reading and evaluating science literature, selecting or designing methods, collecting novel data, analyzing results, working collaboratively, and presenting results outside class (Hatfull et al., 2006; Drew and Triplett, 2008; Caruso et al., 2009; Shaffer et al., 2010; Siritunga et al., 2011; Brownell et al., 2012; Hanauer et al., 2012; Jordan et al., 2014). We do not expect that all CUREs will include all of these activities or be limited to them. If this model is adapted for use in research and evaluation of CUREs, it should be tailored accordingly.

Knowledge and Skills Mini-Model.

This model depicts how students develop content knowledge and hone their technical and analytical skills when they read and evaluate scientific literature, collect data, and analyze results (Figure 1; Shaffer et al., 2010; Brownell et al., 2012; Rowland et al., 2012; Siritunga et al., 2011; Kloser et al., 2013). Increases in knowledge and skills lead to improved scientific self-efficacy (Thiry and Laursen, 2011; Hanauer et al., 2012) and, ultimately, 
increased motivation to learn more science, which further improves science efficacy (Graham et al., 2013).

Communication and Collaboration Mini-Model.

This model depicts how, when students work collaboratively and communicate about their work outside class (Figure 2; Caruso et al., 2009; Bascom-Slack et al., 2012; Shaffer et al., 2014; Jordan et al., 2014), they improve their communication and collaboration skills (Seymour et al., 2004; Laursen et al., 2010; Jordan et al., 2014; Shaffer et al., 2014). The more students collaborate and communicate, the more they feel they belong to a larger community (Alexander et al., 1998), such as a lab community (Barlow and Villarejo, 2004) or a science learning community (Siritunga et al., 2011; Jordan et al., 2014). Theoretically, students who have opportunities to present to members of a scientific community or who develop professional and personal networks that connect them with a broader community would have increased opportunities for external validation (e.g., publishing; Lave and Wenger, 1991; Wenger, 1998). This external validation further solidifies a student’s role as a community member (Lave and Wenger, 1991). Through feedback, support, and role modeling from more experienced scientists, students can learn to tolerate obstacles and failure and develop the temperament necessary for research work (Thiry and Laursen, 2011; Seymour et al., 2004).

Ownership Mini-Model.

This model depicts how students develop a sense of project ownership when they have agency to design their own studies, choose experimental methods, and collect data of interest to them or their community (Figure 3; Hanauer et al., 2012). A growing sense of ownership increases students’ tolerance for obstacles and perseverance, motivating them to complete their projects even in the face of challenges (Ward et al., 2002; Laursen et al., 2010; Hanauer et al., 2012; Alkaher and Dolan, 2014). This has also been conceptualized as “grit” (Duckworth et al., 2007). When students experience success by overcoming obstacles in their research, they develop a greater sense of their scientific self-efficacy, which increases motivation. These outcomes could operate as a feedback loop. When students experience success in overcoming obstacles, their self-efficacy improves, which increases their motivation, which increases likelihood of success in overcoming new obstacles, and so on (Ward et al., 2002, Thiry et al., 2012, Graham et al., 2013).

An Integrated Large Model: Progress toward Long-Term Outcomes

We combined the three mini-models to construct a large CURE model representing our current understanding of how students could achieve outcomes by participating in CUREs (Figure 4). The mini-models are interrelated, because single activities can lead to multiple outcomes and achieving short- and medium-term outcomes is likely to be important for realizing long-term outcomes. To create the large CURE model, we formed two additional connections between short-term outcomes: 1) students who work collaboratively with peers make improvements in their technical skills by receiving modeling and feedback on how to perform particular tasks (Lave and Wenger, 1991); and 2) students who feel ownership over their projects develop an increased sense of belonging to a science community, since they see the science projects as extensions of themselves (Wiley, 2009; Hanauer et al., 2012).

We also formed five new connections between medium- and long-term outcomes and three new connections among long-term outcomes to form the new pathways depicted in Figure 4. We designated enhanced science identity, career clarification, and persistence in science as long-term outcomes, since they are likely to occur late in CURE participation or after CURE participation. Here, we define “persistence” as students staying in a science track 1 yr after they participate in a CURE. Thus, persistence includes staying in a science major, matriculating into a graduate program, or pursuing a career in science depending on when the CURE falls in a student’s academic trajectory. Many medium-term outcomes have potential to enhance students’ science identity. As students grow in their sense of belonging to a larger scientific or laboratory community and as they receive validation from that community, they are likely to further identify as scientists and decide whether a career in science is of interest (Lave and Wenger, 1991). Tolerance for obstacles is a broadly recognized characteristic of a scientific disposition; self-recognition of this characteristic also enhances science identity (Seymour et al., 2004; Thiry and Laursen, 2011; Thiry et al., 2012). Increased scientific self-efficacy increases motivation and scientific identity, both of which influence persistence in science (Harrison et al., 2011; Estrada et al., 2011; Adedokun et al., 2013).

We expect other factors, such as lower social barriers and increased environmental support (see Lent et al., 1994; Estrada et al., 2011; Graham et al., 2013), to mediate and contribute to identity and persistence in science. Despite support for these hypotheses, they are not modeled, because they have not yet been established as probable or possible CURE outcomes.

Identification of Hubs, Reciprocal Relationships, and Feedback Loops

A distinct advantage of the systems approach to evaluation is the process of identifying hubs, reciprocal relationships, and feedback loops. Our model shows that scientific identity is a hub, because multiple activities and short- and medium-term outcomes converge on scientific identity, and scientific identity is on multiple paths to the highly desired outcome of persistence in science (American Association for the Advancement of Science, 2011; President’s Council of Advisors on Science and Technology, 2012). When evaluation resources are limited, evaluation of hubs should be prioritized. For example, if a CURE instructor or designer wants to know whether his or her CURE is effective, it may be best to prioritize measuring changes in students’ scientific identity over changes in content knowledge. Similarly, when resources for instrument development are limited, developing valid and reliable ways of measuring hub outcomes should be prioritized.

In the large model, we identified what we believe to be reciprocal relationships and feedback loops, which we posit may be useful for setting evaluation priorities. We define a “reciprocal relationship” as one in which outcomes are linked via mutual positive relationships. For example, self-efficacy and motivation are likely to be reciprocally linked. As a student develops greater self-efficacy, he or she is likely to be more motivated, which in turn will further his or her sense of self-efficacy. There is good consensus on how to define self-efficacy, and there are published measures of scientific and research self-efficacy (Lent et al., 1994; Chemers et al., 2010; Estrada et al., 2011). A similar consensus does not exist about what constitutes student motivation in science. Thus, priority should be placed on evaluating only one of two reciprocally related activities or outcomes, with a mind to which may be most easily or accurately measured.

We propose that feedback loops occur when an outcome has an indirect effect on itself via downstream outcomes. Feedback loops are distinct from reciprocal relationships, because feedback loops operate over time and should be empirically identifiable. For example, increased self-confidence should lead to increased motivation, which will help students overcome obstacles. This will further increase students’ confidence (Graham et al., 2013). Thus, if students have the opportunity to encounter and fix problems, we should continue to see incremental increases in self-efficacy with each new obstacle they overcome. This can be empirically tested in a longitudinal study. Longitudinal studies will be useful for evaluating longer pathways to achieving outcomes and for identifying long-term feedback loops. For example, designing a project should lead to an increased sense of ownership, which in turn leads to an enhanced sense of belonging to a larger community, enhanced scientific identity, and, ultimately, completion of a science major and possible pursuit of a graduate degree, which will afford more opportunity for identity development and so on. However, this may take years to assess. When time and resources are limited, priority should be placed on measuring short- and medium-term outcomes. If these are not achieved, then effort should be invested in improving CURE instruction to better achieve short- and medium-term outcomes rather than the labor-intensive, long-term tracking of students.

DISCUSSION

The systems approach to evaluation emphasizes the importance of developing a model of a program before selecting which outcomes to measure. We can use models to identify outcomes of particular interest. Urban and Trochim (2009) proposed three categories of outcomes to prioritize: 1) “hubs,” which are highly connected diagnostic outcomes; 2) “low-hanging fruit,” which are easily measurable outcomes; and 3) “pinnacle” outcomes, which are important for stakeholder interests (e.g., funding agencies, institutional priorities, accreditation) or necessary for continuing the program. We should also consider the availability of existing measures (protocols, rubrics, surveys, etc.) in determining which outcomes to assess and when to assess them. We can designate early, middle, and late evaluation phases that are developed based on collective consideration of this information, the timeline for CURE development, and the timeline for when students are expected to realize particular outcomes.

We offer three phases for evaluation of CUREs to illustrate this process (Figure 5). We discuss which outcomes constitute low-hanging fruit, pinnacle outcomes, and hubs in each phase. We designed phases around when outcomes are likely to be achieved, taking into account the ease of assessment. We recommend that each CURE evaluation be designed in program-specific phases that consider particular CURE activities, stakeholder interests, programmatic goals, and evaluation resources.

Figure 5.

Large CURE model showing phases and hubs. Arrows represent positive directional relationships between activities and outcomes. Blue boxes indicate separate evaluation phases. Red borders surrounding outcomes indicate “hub” outcomes.

Early-Phase Evaluation

When CURE instructors or designers are first implementing and evaluating their CUREs, this is the place to start. This phase focuses on whether activities are resulting in short-term outcomes. Measures exist for many of these outcomes (low-hanging fruit), including assessment of analytical research skills (e.g., developing hypotheses, designing experiments, analyzing data; Feldon et al., 2011; Sirum and Humburg 2011; Brownell et al. 2014; Dasgupta et al., 2014; Deane et al., 2014), project ownership (Hanauer and Dolan, 2014), and the hub of self-efficacy (Chemers et al., 2010; Estrada et al., 2011). In addition, these outcomes should be measurable during or immediately after CURE instruction.

Middle-Phase Evaluation

This phase focuses on assessment of the communication and collaboration elements of CUREs and includes one hub: becoming part of a larger community. This phase will require the development and testing of new instruments, because, to the best of our knowledge, there are no measures of communication or collaboration skills or of students’ sense of belonging to a science community that have been tested with CURE students, although there are tools that could be adapted for these purposes. For example, Sevian and Gonsalves (2008) offer a rubric for measuring the effectiveness of scientific explanations, but it was developed for use with graduate students explaining their work to middle-school students. Similarly, there are general measures of undergraduate students’ sense of belonging or sense of community and the relationship between these and students’ intended or actual persistence in college (Bollen and Hoyle, 1990; Chipuer and Pretty, 1999; Hausmann et al., 2007, 2009). Again, these tools need to be adapted and tested to demonstrate their usefulness for CURE assessment. Some of the outcomes in the middle phase also take longer to achieve. For instance, it may take time to establish a mechanism through which students can present work outside class or it may take multiple iterations of a CURE to yield sufficient data for publication in a science journal.

Late-Phase Evaluation

Late-phase evaluation focuses on medium- and long-term outcomes of CUREs, including scientific identity (Chemers et al., 2010; Estrada et al., 2011), which is both a hub and a low-hanging fruit. As with the middle-phase example, we could not find established, valid, and reliable measures of science student motivation, tolerance of obstacles, or career clarification. There are measures that have some potential to be adapted for CURE assessment. For example, the Science Motivation Questionnaire, developed for use with non science majors, might be adaptable for use with majors, depending on whether and how motivation differs between majors and nonmajors (Glynn et al., 2009). The “grit scale” is a general measure of perseverance and passion for long-term goals that could be adapted to be more science specific (Duckworth et al., 2007). The Undergraduate Research Student Self-Assessment includes a career clarification scale that is well grounded in qualitative research but for which validity and reliability information have not been published (Hunter et al., 2009). Although the “pinnacle” outcome of persistence in science is the long-term outcome featured here, there are other long-term outcomes likely to be of interest and significant value, such as scientific literacy and understanding of the nature of science.

There is a temptation to focus on assessing pinnacle outcomes because of their value to stakeholders and because they are often more compelling than more immediate outcomes. Yet it is important to keep in mind that pinnacle outcomes, such as persistence in science and understanding the nature of science, may take longer to achieve and may only be achieved for a subset of CUREs or by certain students or following multiple CURE experiences. It is also exceedingly difficult to design and conduct a study that shows a causal relationship between an instructional experience and student outcomes, and this difficulty is compounded when the outcomes take an extended time to realize. For example, we included persistence in science as an outcome with sufficient evidence to include in models used to study CUREs moving forward. However, as noted above, the studies we found did not control for the extent to which students may already be planning to persist in science (self-selection). In addition, these studies did not make use of established methods for designing and interpreting quasi-experimental studies in a way that provides definitive evidence that participation in CUREs improves persistence in science. The CURE community should consider a handful of larger-scale collaborations aimed at examining the relationship between CURE activities and desired long-term outcomes rather than expecting individual instructors to collect data that can be used to answer questions about long-term impacts of CURE instruction.

Each of these models, as well as each pair of nodes and the relationship between them, is a testable hypothesis about how CUREs work. For example, we can investigate whether students who develop an improved sense of self-efficacy are more tolerant of obstacles, whether students who develop technical skills also develop an increased sense of self-efficacy, and whether students who collect novel data improve their technical skills. We expect that other aspects of CUREs not addressed here, such as the duration of the experience, will influence these relationships. It is also important to investigate how student differences, such as gender, major, ethnicity, race, first-generation college-bound status, and other factors known to affect realization of science-related outcomes, may influence students’ experiences with CUREs.

The mechanisms by which these outcomes come about and the connections between activities and outcomes and between short-, medium-, and long-term outcomes have not been empirically investigated. In addition, some outcomes demonstrated in studies of research internships are unexplored or only proposed in CUREs. For example, we found that undergraduates who complete research internships make progress in understanding of the nature of science and expand their social and professional networks (e.g., Russell et al., 2007; Adedokun et al., 2012; see Supplemental Table 1), but these outcomes have not been broadly explored in CUREs.

Previous work has proposed a set of features or dimensions that distinguish CUREs from other lab-learning experiences (Corwin Auchincloss et al., 2014), Specifically, CUREs are thought to be distinctive because they: 1) involve students in multiple science practices, 2) provide opportunities for students to make discoveries, 3) involve students in work that has relevance outside the classroom, 4) involve students in collaborative work, and 5) provide opportunities for iteration. The activities highlighted here relate to some but not all of these dimensions. Additional models need to be developed and tested to identify what is both necessary and sufficient in the design and implementation of CUREs to make them effective for students.

Another important avenue for study of what makes CUREs effective relates to faculty experiences and outcomes. For example, research and theories of how faculty members decide to change their teaching to more actively engage students and emphasize understanding of science concepts and practices raises interesting questions of the potential for CUREs to be a “gateway” to instructional change. For example, faculty members may be quicker to adopt evidence-based teaching approaches when they teach a CURE because of the CURE’s potential to connect to their research interests and their identity as researchers (Brownell and Tanner, 2012). In addition, CUREs may resonate with faculty members’ own experiences learning to do science, and thus they may be willing to sacrifice “content coverage” to offer course experiences they see as more authentic (Spell et al., 2014).

CONCLUSION

In summary, we identified studies on CUREs and research internships and considered learning theory that is useful for explaining the mechanisms by which students learn through research experiences. Then, we characterized the outcomes that students have the potential to realize from participating in CUREs. Using a subset of outcomes that had the most empirical support, we used a pathway-modeling process to generate models of the components and contexts that make CUREs effective for students. We offer these models as hypotheses for the CURE community to test, revise, elaborate, or refute. We cite instruments that are ready to use in CURE assessment and note gaps for which instruments need to be developed. We hope that the community effort to assess CUREs using a common set of tools, which began with the efforts of Lopatto and colleagues (Lopatto, 2004, 2007, 2008; Lopatto et al., 2008) will embrace this new set of tools such that we can continue to compare results across CUREs and between CUREs and research internships. We describe our approach so others can use it to build and test their own models of CURE instruction, which is especially important given the diverse long-term outcomes that may result from research experiences. Understanding how CUREs function will help improve existing CUREs, aid in the design of new CUREs, and promote a common understanding of the utility of CUREs as an educational intervention.