Using Student Learning and Development Outcomes to Evaluate a First-Year Undergraduate Group Video Project

Abstract

Students in an interdisciplinary undergraduate introductory course were required to complete a group video project focused on nutrition and healthy eating. A mixed-methods approach to data collection involved observing and rating video footage of group work sessions and individual and focus group interviews. These data were analyzed and used to evaluate the effectiveness of the assignment in light of two student learning outcomes and two student development outcomes at the University of Minnesota. Positive results support the continued inclusion of the project within the course, and recommend the assignment to other programs as a viable means of promoting both content learning and affective behavioral objectives.

INTRODUCTION

This article reports on the efforts of a biology professor at a large research university to address both science content learning objectives and behavioral and affective understandings for first-year students in an interdisciplinary course. The class assignment discussed here aims to provide experiences deep enough to transfer beyond a single class and assist students in developing skill sets useful to future endeavors both inside and outside academia. Helping students develop the capacity to know their own strengths, weaknesses, and motivations is a necessary part of their transformation into adults able to engage in “collaborative social relations with diverse others,” a skill employers now expect higher education to address (Baxter Magolda, 2008, pp. 269–270). Student assignments that promote multiple institutional goals are needed in light of new requirements that coursework address both specific content and developmental outcomes. By exploring the effectiveness of particular approaches to meet changing institutional needs, science instructors will be better prepared to draw upon a variety of strategies in a shifting academic climate.

Science educators have long valued a range of skills beyond a student's content knowledge, and often require demonstration of laboratory procedures, proficiency in writing lab reports, and oral communication skills in presentations. Competencies associated with student development goals, however, represent a new domain for most instructors at the college level. Few science educators work in learning environments in which there has been overt direction to emphasize the need for interpersonal skills to better interact with diverse groups, yet we recognize that ability as valuable—if not essential—for success in professional science careers. Concerned instructors find ways to recognize the symbiotic relationship between these skills by promoting and combining learning and development outcomes in the same assignments. Providing opportunities for students to work in groups on assignments that prompt them to communicate ideas and promote both content and developmental goals is often recommended as a strategy to improve science, technology, engineering, and medical (STEM) education at the college level (Lord, 2001). Particularly in classes attended by nonscience majors, engaging students is important for success. Additionally, failing to create classroom environments in which students are encouraged to ask questions and feel comfortable expressing confusion contributes to many promising students' deciding not to pursue a science major while in college (Seymour and Hewitt, 1997). In their discussion of a program designed to increase scientific literacy in a humanities-focused student body, McPhearson and colleagues (2008) emphasize the effectiveness of inquiry-based, multidisciplinary approaches to teaching science content in a diverse class of incoming freshmen. Reforms designed to improve retention and address the problem of underrepresentation in science majors have also been presumed to benefit all students (Seymour, 2001).

In 1999, the National Research Council (NRC) called for science courses to become more authentic and to engage students in activities practiced by professional scientists. In the institutional context of this study, “authenticity” is interpreted as the need for students to develop skills to effectively collaborate with others in finding solutions to problems and creating a product in a team setting. The NRC specifically recognized the need to transform teaching and learning in institutions of higher education and recommended the exploration of STEM concepts “as practiced by scientists and engineers,” which includes the sharing and discussion of ideas (NRC, 1999). Additionally, the NRC report also called for introductory science courses to be more inclusive and to meet the needs of learners from diverse backgrounds, a goal shared by higher educational institutions concerned with retention and recruitment of diverse student bodies (and related to the establishment of student development and learning goals discussed in this article). Many other national policy documents (see A New Biology for the 21st Century [NRC, 2009]) recommend such behavioral goals, which cannot be readily achieved through the traditional lecture setting, and emphasize the need for alternative pedagogies to promote more collaborative science experiences. Designing a curriculum that optimizes student–student interaction to stimulate creative, critical thinking requires attention to the way in which group activities are structured, and also the type of questions that are explored collectively. Cooperative group learning has historically been recommended as a means to begin addressing developmental and academic outcomes that cannot be targeted through traditional lecture but, as Seymour (2005) notes, “…how to best infuse large university classes with more of the active and interactive learning methods remains a consistent challenge” (p. 1).

Cooperative group learning is an admittedly complex teaching and learning strategy with which many science instructors have limited experience. Grounded in the constructivist theory of learning (see Johnson and Johnson, 1989), this pedagogical approach posits that knowledge is fostered and organized through social interactions, such as those emphasized in cooperative group learning. Johnson and Johnson (1989) report that specific criteria must be met to achieve genuine cooperation among individuals within a group. For example, in order for groups to be truly cooperative, they must have some degree of positive interdependence among group members, and there must also be a strong individual accountability component in assignments. Group assignments that promote positive interdependence and individual accountability are part of creating more dynamic and interactive classrooms and thus initiate important steps toward the goals outlined by NRC and other invested groups. Research on the effectiveness of cooperative group learning is robust in terms of promoting both learning and development goals (Lord, 2001). The effectiveness of group learning has been shown in studies by Knight and Wood (2005), who found that increasing interactive and participatory activities significantly improved student learning; Eisen (1998), who found that group projects help engage students and improve content literacy; and DebBurman (2002), who showed that working in groups promotes developmental skills, such as critical thinking, communication, and social responsibility. In addition to being a heavily researched pedagogical strategy, cooperative learning has a “variety of positive and measurable outcomes on students at a variety of cognitive levels and in a variety of disciplines” (Tanner et al., 2003, p. 2).

In addition to changing pedagogical strategies, organizational documents promote modifying assessment practices. For example, the NRC (2003a) advocates the inclusion of both student learning and development outcomes in evaluation. More specifically, instructors are encouraged to move beyond a reliance on assessment of biological content knowledge through lecture exams, and even procedural skills shown in lab exams. Efforts to include skills and personal characteristics often documented in letters of recommendation, but not frequently assessed in traditional STEM courses (such as oral and written communication skills and the ability to work well with others) require broadening instructors' understanding of assessment and class objectives. Instructor observation can be used to identify interpersonal communication process skills, such as the ability to interact with diverse groups of people. Dutson et al. (1997) also described how this observation simultaneously provides instructors the opportunity to inspect for more traditional learning outcomes, such as students' application of content knowledge. Seymour (2001) includes the creation of assessment instruments to match reformed teaching strategies as a key to innovation in undergraduate science education.

Universities and colleges are now expected to do more than produce content specialists; we are to prepare students to function as productive and well-informed citizens (American College Personnel Association [ACPA] and National Association of Student Personnel Administrators [NASPA], 2004). To promote these goals, many universities now promote both learning outcomes and developmental outcomes. Learning outcomes have an extensive history in academics and typically state what knowledge should be known at the time of graduation (e.g., “ways of knowing” within a specific discipline). Comparatively, developmental outcomes are a new entity, and include such characteristics as the ability to work well in diverse environments and tolerance of ambiguity. NASPA and ACPA promote the “integrated use of all of higher education's resources in the education and preparation of the whole student” and describe “learning” as a combined activity of academic and student development experiences (ACPA and NASPA, 2004). Baxter Magolda's framework of constructive-developmentalism links learning conditions that embrace constructivist pedagogical approaches and understanding of learning with individual development processes (Baxter Magolda, 2000); such an understanding of the ways in which student growth occurs both intra- and interpersonally underlies the simultaneous pursuit of student development and learning outcomes. Furthermore, the interdisciplinary, cooperative approaches show great promise in improving student learning gains (as compared with traditional lecture style) and can serve as a more accessible entry point for nonscience majors to engage with and explore scientific investigation (McPhearson et al., 2008).

Many higher education institutions now apply constructivist theories of knowledge development to first-year student experiences; tracking student satisfaction and success is also an important application of efforts to evaluate courses for their incorporation of particular outcomes. Historically, freshman students at large universities have frequently attended large lecture courses that do not afford much individual attention. Recently, however, researchers such as Kuh (2008) and Tinto (2006–2007), have targeted the freshman experience as critical to establishing and maintaining student satisfaction and related to retention and graduation rates. Based on this research, many institutions have directed efforts toward improving the first-year experience by creating programs such as freshman seminars, extended freshman orientation, and common book experiences. Such ambitious goals of improving both instruction and student experience pose new challenges for instructors. In this paper, we offer our experiences at the University of Minnesota (UMN) and suggestions of how a similar approach could be applied at other institutions.

The following section of this article gives an overview of the history and current implementation of student learning and development outcomes at the heart of the study discussed here; this study aimed to explore the effectiveness of a first-year class project in supporting student progress toward selected objectives.

Student Learning Outcomes and Student Development Outcomes at UMN and the First-Year Experience Initiative

In the College of Education and Human Development (CEHD) at UMN, all incoming students participate in a First-Year Inquiry (FYI) course that promotes “multidisciplinary ways of knowing” (see Supplemental Material). Such courses are considered a “high-impact” activity in Kuh's discussion of initiatives likely to promote undergraduate student success and engagement (Kuh, 2008). The FYI course was developed in 2007 by a team of administrators and instructors within the Department of Postsecondary Teaching and Learning, who incorporated two UMN student learning outcomes (SLOs; Can Communicate Effectively; Have Acquired Skills for Effective Citizenship and Lifelong Learning) and two student development outcomes (SDOs; Responsibility and Accountability; Appreciation of Differences) into the course proposal. These four outcomes address both the individual self-reflection necessary for critical thinking and the interpersonal skills required for effective social relations—a process referred to by Baxter Magolda (2008) and others as “self-authorship.” Additionally, these outcomes were aligned with the writing skills and multicultural focus of the college as a whole.

A set of SLOs and a separate set of SDOs were developed and published at UMN in an organized effort to stimulate the transformation of undergraduate education. Efforts began in May 2003, when the UMN's Provost's Council for Enhancing Student Learning (CESL) adopted a set of statements designed to guide the development of the outcomes. These statements called for the deliberate collection and analysis of evidence to determine student progress toward high standards, with the overall purpose of improving teaching and learning (personal communication, C. Murdoch, September 2010). Following these initial steps, seven SLOs were developed in the 2003–2004 school year by a curriculum-assessment working group of the Provost's CESL. The seven SLOs were subsequently adopted by the university's Faculty Senate as official policy. In 2007, the University Senate Education Policy Committee used a similar process to develop and adopt seven SDOs to complement the SLOs. Together, UMN views the SLOs and SDOs “entwined as critical elements of the student experience” (UMN, 2011a). Figure 1 contextualizes the role of SLOs and SDOs from an institutional, instructor, and student perspective. Table 1 displays a complete list of the SDOs, which “assist students to become lifelong learners and engage as effective citizens when they leave the University” (UMN, 2011a), and the SLOs, which guide “what students know and can do” (UMN, 2011b), and indicates the four outcomes discussed in this article.

Figure 1.

Contextual framework of student learning and development outcome implementation.

Table 1.

UMN student development and learning outcomes

UMN SDOs
Accountability and Responsibilitya
Independence and Interdependence
Goal Orientation
Self-Awareness
Resilience
Appreciation of Differencesa
Tolerance of Ambiguity
UMN SLOs
Can Identify, Define, and Solve Problems
Can Locate and Critically Evaluate Information
Have Mastered a Body of Knowledge and Mode of Inquiry
Understand Diverse Philosophies and Cultures within and across Societies
Can Communicate Effectivelya
Understand the Role of Creativity, Innovation, Discovery, and Expression across Disciplines
Have Acquired Skills for Effective Citizenship and Lifelong Learninga

^aAn outcome targeted in this study.

Currently, university policy directs instructors to incorporate the SLOs and SDOs into their course expectations but allows flexibility in the determination of how to do so. Curriculum committees within UMN use SLOs and SDOs in the process of developing and approving new courses, and also for periodic evaluation of existing courses. Instructors are not currently required to include all of the SLOs and SDOs in their class designs; ongoing evaluative projects, such as this study, will help inform the use of these outcomes in future course syllabi and assessments.

Six different sections of the FYI course have been offered each Fall semester since the course was initially proposed and approved by the college's curriculum committee. Each Fall semester, ∼450 students are divided into six different FYI classes, each of which is team-taught by three instructors (∼75 students per class section). Groups of 75 students meet with all three instructors for 2 h/wk during the semester, and also meet in groups of 25 students with one instructor for 2 h/wk. The six different sections of the FYI course have a wide variety of topics and titles (e.g., Ecological Hotspots, Energy…Illuminated), but all must use a common set of assignments. For example, all sections must meet the requirements for a “writing-intensive” class, and thus must require at least 12 pages of revised text. A significant percentage of the course grade is to be based on students' writing abilities. All FYI students must participate in an end-of-semester group project in which students build on the course concepts to collectively answer the question “How can one person make a difference?” The student projects are publicly presented at the end of Fall semester at a public capstone showcase that features all 450 students showing and viewing projects.

For 3 yr, the Food for Thought and Action FYI course has used a group video assignment to fulfill the requirements of the group capstone project. In 2010, a small grant was secured from UMN's Academy of Distinguished Teachers to conduct an evaluation of the group video project in light of UMN's SLOs and SDOs. Funds were used to hire a graduate student to coordinate the study independent of the day-to-day operation of the course, with results of the evaluation intended to be used in instructor and department decisions to drop or modify the capstone video assignment, or to retain and publicize it as a viable mechanism to promote a practical cooperative learning assignment promoting student behaviors that meet UMN's SLOs and SDOs. In keeping with institutional and departmental goals of promoting SDOs, as well as requirements to assess student progress toward learning outcomes (see Figure 1), FYI instructors endeavor to create class assignments that provide an opportunity for both authentic science experiences and collaboration with others.

The purpose of the research project discussed here was to determine how a cooperative class assignment could be used to ascertain student progress toward both SLOs and SDOs. We conducted a qualitative evaluation in which student behaviors were observed and rated to identify specific instances consistent with the objectives of the selected SLOs and SDOs. The data from this evaluation were used to construct an understanding of how successful the cooperative group assignment was in meeting the instructional goals—both content learning and affective personal development—of the video activity.

METHODS

Course Design: Food for Thought and Action

The three instructors of the Food for Thought and Action class included a biologist, a writing instructor/lawyer, and a social scientist. The curriculum for the course was based on three different texts: Food Inc. (Weber, 2009) and two books from Michael Pollan—In Defense of Food (2008) and Food Rules (2009). Daily lessons in the course were developed by each of the three instructors and ranged from understanding the requirements for the U.S. Department of Agriculture's “Organic” label (led by the writing instructor/lawyer), exploring the sociological dynamics involved in local farmers' markets (led by the social scientist), and examining the biology of atherosclerosis, diabetes, and other diseases related to obesity (led by the biologist). The instructional strategies used in the large class meetings (75 students with all three instructors) were largely influenced by the classroom design—a 90-student active-learning room that featured 10 round tables.¹ The room's structure intentionally makes long lectures impractical but facilitates group interaction. In the Food for Thought and Action class, a typical lesson involves a short presentation by the instructor followed by a student group activity, and then a follow-up large-group discussion and summary. Assessment of student performance in the course involved several writing assignments (including several short papers and a traditional term paper), midterm and final exams, and the final group video project.

Capstone Assignment Design²

The capstone assignment was introduced during the 10th week of the semester, when instructors assigned students to groups. The three professors used different selection methods, but each used an intentional strategy to create groups of four students. One professor focused on how well the students had done on the midterm exam, matching students who performed at different levels of academic proficiency; another created groups with a mixture of previously demonstrated social participation and personalities; and the third professor aimed to balance gender and students from different racial and ethnic backgrounds in each group. The assignment and description of the video project was then explained to the students. (See Supplemental Material for project description and class syllabus.) The stated goal of the assignment was to “create a 30 to 60 second public service announcement (PSA) video focusing on food and targeting a specific audience.” More specifically, videos were to focus on one or two specific “food rules,” as outlined in Michael Pollan's book Food Rules (2009).

The project description provided recommendations for the production process, such as guidelines for managing the initial brainstorming sessions. The students were encouraged to draft a storyboard to assist in developing and connecting ideas when initiating the project. The final product was to be filmed using a digital camera and edited using computer applications, such as iMovie or Movie Maker. Cameras, computers, and video-editing software were made available to students through the university library system, though many groups elected to use their own equipment. Students completed about one-fourth of the project during regular class hours, with the remaining hours scheduled according to group members' availability (and without instructor oversight). Students had 5 wk to complete the project, and all videos were presented to the class at the culmination of the semester.

Participants in the Research Study

All students in the course were invited to voluntarily participate in the research study prior to the beginning of the capstone project. Students indicated their willingness to participate by signing a consent form that had been approved by the university's institutional review board office. After groups were initially formed by instructors, a sample was selected by identifying those groups in which all four members had consented to participate in the research study. A total of six groups were selected: two from each instructor's list. The groups were not notified that they would be recorded on video until the first day of the project, during their initial group meeting. Demographic data for the Food for Thought class showed it to be a relatively representative sample of students enrolled in CEHD, a college that is generally more diverse than the university as a whole (Table 2). Additionally, students in the course represented a cross-section of programs, such as university honors, TRIO (federally funded programs that support college opportunities for students from disadvantaged backgrounds; Council for Opportunity in Education), Access to Success (ATS), and Commanding English (CE; Table 3).

Table 2.

Comparison of race/ethnicity of students enrolled in the Food for Thought and Action FYI course, freshman students in the CEHD, and freshman students at UMN

	FYI class (n = 84)	CEHD freshmen (n = 447)	UMN freshmen (n = 4876)
White	61%	61%	78%
Asian	19%	16%	9%
Black	14%	15%	2%
American Indian	4%	2%	1%
Hispanic	1%	2%	3%
Hawaiian/Pacific Islander	0%	1%	1%
Total	99%a	97%a	94%a

^aNot all students chose to identify ethnicity.

Table 3.

Enrollment of students in FYI Food for Thought and Action class in support services and programs for incoming freshmen within the CEHD

	FYI class (n = 84)	CEHD freshmen (n = 447)	UMN freshmen (n = 4876)
Honorsa	7%	5%	11%
TRiOb	20%	23%	0%
CEc	17%	10%	0%
ATSd	46%	43%	5%

^aHonors: University honors program assists high-achieving students in making the most of their undergraduate education.

^bTRiO: Academic development services for low- to modest-income, first-generation college students, and students with disabilities.

^cCE: Program offered to freshmen for whom English is not their first language. Students in this program work on academic English while taking standard courses of a typical freshman student.

^dATS: Program designed for students whose experiences and high school records indicate strong potential for success, but who do not have high school rank or test scores that meet the typical profile of students admitted to CEHD; College of Food, Agriculture, and Natural Resources; or College of Liberal Arts.

Evaluation Design and Data Collection

The research design utilized in this study includes both open-ended and close-ended approaches to data collection, in keeping with Cresswell's description of concurrent mixed-method strategies (Cresswell, 2009). Using a structured rubric to track frequency data, videographers collected measures of student behaviors, which were analyzed quantitatively, while observation and emergent interview design allowed for qualitative investigation. The data-collection techniques used in this study provide a degree of triangulation aimed at establishing validity of the conclusions drawn from the evaluation. In particular, data were gathered through observing and recording video of group meetings, conducting individual interviews with representative students, and interviewing a focus group composed of students from each of the groups. Qualitative research methods allow for the investigation of a particular phenomenon or people's experience of it within a defined context (Patton, 2001); educational settings and group dynamics are well suited to investigation using this approach. Following Tashakkori and Teddlie's framework for mixed methods (Tashakkori and Teddlie, 2002), this study is primarily qualitative in nature, with research objectives aimed at building understanding of a particular experience supported by the collection of quantitative data (i.e., QUAL-quan). A combination of statistical and text analyses were used to investigate student perceptions and experiences, in keeping with the pragmatic methods advocated by many mixed-methods proponents (Johnson and Onwuegbuzie, 2004).

Evaluation as it is understood in this study follows Patton's (2001) definition of this mode of investigation as “the systematic collection of information about the activities, characteristics, and results of programs to (1) make judgments about the program, (2) improve or further develop program effectiveness, (3) inform decisions about future programming, and (4) increase understanding” (p. 39). Following Merriam (1998), interviews serve an important interpretive function, because they allow individuals to explain internal thought processes and explain personal experiences. Focus groups can allow for more open-ended questioning and for participants to interact with one another and build upon others' answers, and are a way to encourage participants to express their opinions and perceptions of a shared experience (Krueger and Casey, 2009). The evaluation of the group capstone video project was conducted using the four SLOs and SDOs incorporated in the design of the FYI course and used as a guide to measure student progress in keeping with the university's goals.

As described earlier in this paper, the four SLOs and SDOs were identified by a departmental curriculum committee and associated objectives were specified for each outcome. This list of objectives was intended to provide a mechanism through which data generated from observing student behaviors could be analyzed. A program evaluation specialist developed the particular wording of the objectives, and they were previously used on other projects. The four SDOs, as well as the objectives for each outcome are listed in Table 4.

Table 4.

Student development and learning outcomes and associated objectives

SDO 1: Accountability and Responsibility

Group members decide and carry out specified roles.

Group meetings are set for times that meet all group members' schedules.

Group members show up for meetings.

Group members create a feasible timeline that includes who will complete specific tasks.

SDO 2: Appreciation of Differences

All group members invite and listen to one another's ideas and perspectives.

All group members are given time to offer support and explanation for their ideas and/or perspectives.

Final decisions relative to video content are negotiated and agreed upon by all group members.

SLO 1: Can Communicate Effectively

Group members discuss and decide upon the audience for their video.

Group members communicate effectively with one another.

Group members develop together a storyboard that includes how the video message will be conveyed to a selected audience.

Actors use eye contact, confident and audible voice tone, and effective language throughout video presentation.

SLO: 2: Have Acquired Skills for Effective Citizenship and Lifelong Learning

Group members bring at least two different disciplinary perspectives to their chosen topic.

Group members articulate how they envision their video message impacting the intended audience.

Group members demonstrate awareness of which tasks will benefit from several people working together.

Videographers for the project were selected from a group of graduate students from the Counseling and Student Personnel Psychology program within CEHD, who were trained in recording techniques by a university media specialist and instructed in how to record, secure, and share their video data. Each group was recorded by the same individual throughout the project. Recordings of 25–30 min were made of each of the group's meetings. The first recording took place during a regular classroom session, immediately after the students were assigned to groups. The second recording also took place during a regular classroom session, and occurred as groups set up storyboards and developed scripts for their final videos. The third took place outside regular classroom time and documented groups filming and/or editing their final products. All six groups were taped for the first and second sessions, but only four of the six groups were taped during the third session, due to scheduling conflicts.

Scoring Rubric and Assessment

Scoring rubrics were developed for each of the three recording sessions (see Table 5). Each rubric contained the four SLOs and SDOs, with two or three identified objectives for each outcome. Specific objectives used to evaluate each session were defined by the nature of the student tasks involved during each meeting. For example, within the SLO Accountability and Responsibility, one objective was “Group members carry out decided-upon roles and continue to decide what each member is responsible for throughout the remainder of the process.” This objective was germane to the beginning stages of the project and was used to rate sessions 1 and 2, but was not included in the rubric for session 3.

Table 5.

Sample scoring rubric for SDO 1 and one associated objective

Student Accountability and Responsibility:

Group members carry out decided-upon roles and continue to decide what each member is responsible for during the remainder of the project.

Yes/No

Notes:

Two individuals rated each video session. “Rater 1” was the coordinating graduate student, who rated all three sessions for all six groups, and “Rater 2” was the videographer for each specific group. The scoring rubric used a Yes/No (i.e., forced-choice) system to identify whether or not specific objectives were met by the behaviors of the group during the recorded session. For example, regarding the objective “Group meetings are set for times that meet all group members' schedules,” a Yes was marked if the group was observed verbally committing to future meeting times during the video session, and a No was marked if this did not occur. No time-stamp comparisons were used while rating video sessions, and thus it was possible for the two raters to agree that a group did engage in a specific behavior related to an outcome, but identify different times within the session when they considered the criterion to have been met.

Student behaviors from the first group meeting were rated based on nine objectives (within the four SLOs/SDOs) and six groups, and thus a maximum of 54 Yes markings was possible per rater. Rater 1 identified 30 instances related to the objectives, whereas the six videographers identified 34 instances. Rating student behaviors for the six groups during the second group meeting (in which students made storyboards and specific plans for their videos) involved 10 objectives and thus a maximum of 60 possible Yes markings per rater. Rater 1 identified 40 instances related to objectives during this session, whereas the six videographers identified 39 instances. Eight objectives were used in rating the third student session, when students were filming and/or editing their projects. Only four of six groups were taped during this session, and thus the maximum number of possible Yes markings per rater was 32. Both Rater 1 and the four videographers identified 29 such instances while rating the third session. Out of a maximum 146 possibilities, Rater 1 identified 99 instances of student behaviors that were identified as meeting specific objectives within the scoring rubric, and the six videographers identified 102 instances. Based on the differences in rating, interrater reliability between the lead graduate student and the six videographers was determined to be ∼98%.

Student Interviews

Individual and group interviews were conducted after completion of the class video project. Six students were interviewed individually, and six others met collectively for a focus group interview. Each of the six study groups had one participant in the individual interviews and one member in the focus group. Selection of students was based on schedule availability, (i.e., convenience sample). Both individual and group interviews were conducted using an open-ended, six-question protocol; two questions were the same for each group and four were intended to query students about events and opinions particular to their groups' experiences. Individual interviews ranged in length from 10 to 15 min, and the focus group interview lasted 21 min. Both the individual and focus group interviews were recorded and transcribed. Specific student quotes that were germane to the analysis of the four SLOs/SDOs were extracted from the transcripts.

The groups included in this study were representative of students enrolled in the larger class in terms of gender, race, ethnicity, and English language proficiency. As this was an introductory course during the first semester of the students' first year of college, students had similar levels of (un)familiarity with each other and with college-level course expectations. The data show that student groups generally made similar progress toward attainment of desired development and learning outcomes. The following section looks further at individual student experiences and examines nuances between group dynamics and individual reactions.

RESULTS AND DISCUSSION

The aforementioned study design was used to determine whether observed student behaviors indicated that targeted student development and learning outcomes were addressed through participation in this project. Following the evaluative approach described above, six student groups were recorded three times³ each throughout the course of the video project. After the project was completed, six individual students were interviewed about their experiences and a different set of six students participated in a focus group interview. Using the established rubric to assess recordings of group interactions, frequency measures were made to ascertain to what extent groups met the targeted development and learning outcomes. Transcript data from interviews show students developing the ability to work through problems with classmates and discussing the course's impact on how they may participate in cooperative group activities in the future. Students demonstrated the ability to assign individual roles within groups, while also entertaining multiple perspectives within discussions to meet the objectives of the video assignment.

Evidence that student behaviors were consistent with the goals of the selected learning and development outcomes was found in both the quantitative data (the frequency with which group members were observed to have met the objectives for each outcome) and the qualitative data (student responses in interviews and focus groups). This section summarizes analysis of video recordings and of narrative data gathered about each of the four learning and development outcomes, relying heavily on students' own words to express what was learned.

Data Related to SDOs

SDO 1: Accountability and Responsibility.

Coding for SDO 1 (Accountability and Responsibility) was based on objectives that identified behaviors related to assigning and carrying out specific roles, scheduling, developing a timeline, and showing up for meetings. Data for this outcome (Table 6) show 50% (for both Rater 1 and Rater 2) of the objectives met during the first recording, 72% (for both raters) met during the second recording, and 88% (for both raters) met during the third recording.

Table 6.

Scoring data for SDO 1: Accountability and Responsibility

	Taping 1: first group meeting (n = 6 groups)		Taping 2: storyboard/script development (n = 6 groups)		Taping 3: filming/editing (n = 4 groups)
Student objectives	Rater 1	Rater 2	Rater 1	Rater 2	Rater 1	Rater 2
1. Group members decide and carry out specified roles.	33% (2 out of 6)	33% (2 out of 6)	67% (4 out of 6)	67% (4 out of 6)	100% (4 out of 4)	100% (4 out of 4)
2. Group meetings are set for times that meet all group member's schedules.	67% (4 out of 6)	67% (4 out of 6)	83% (5 out of 6)	83% (5 out of 6)	—–	—–
3. Group members create a feasible timeline that includes who does what, as well as when parts of the video are due.	50% (3 out of 6)	50% (3 out of 6)	67% (4 out of 6)	67% (4 out of 6)	—–	—–
4. Group members show up for meetings.	—–	—–	—–	—–	75% (3 out of 4)	75% (3 out of 4)
Percentage of Yes codes	50% (9 out of 18)	50% (9 out of 18)	72% (13 out of 18)	72% (13 out of 18)	88% (7 out of 8)	88% (7 out of 8)

The data suggest that students in some groups initially struggled with scheduling demands and task organization and distribution. Group members found it easier to build consensus, however, as they became more comfortable with one another and more engaged in the project. For example, one student recalled:

It was easier to meet at the end because I think everyone really wanted to get this done, and really wanted to do a good job, but I think it was harder at the beginning because we didn't have any ideas and we did not know what to expect.

The above response also indicates that students were indeed interested in producing a quality final video, not in merely completing the assignment. Some groups immediately prioritized meetings and felt encouraged by one another's commitment. A member of one such group reported that:

It was a very great experience. Everybody was on time and everybody was willing to do something. Everybody did their part on time without missing anything.

In other groups, however, peer accountability took longer to develop. For example:

The hardest part with everything was with our schedules. We had a lot of conflicts, and finally we said, “This is what we are doing and everything else is put aside.” If we wouldn't have done that, I don't think we would have gotten as much done as fast.

Results here are also consistent with Johnson et al. (1991) and others who advocate groups staying together for more than one or two class sessions so that interpersonal dynamics can develop. This development outcome also highlights the way in which group accountability and responsibility was reliant on individual accountability and responsibility, which is a necessary criterion for cooperative group learning (Johnson et al. 1991). Overall, most of the groups evaluated were observed to demonstrate a high degree of interdependence and to coordinate their efforts in a manner in which members held each other accountable for the final project.

SDO 2: Appreciation of Differences

Coding for the SDO 2 (Appreciation of Differences) was based on objectives that identified behaviors related to listening to the ideas of others, supporting and encouraging fellow group members, and coming to an agreement about the quality of the final project. Data for this outcome (Table 7) show 75% and 67% (Rater 1 and Rater 2, respectively) of the objectives met during the first recording, 92% and 83% (Rater 1 and Rater 2, respectively) during the second recording, and 100% for both raters during the third recording.

Table 7.

Scoring data for SDO 2: Appreciation of Differences

	Taping 1: first group meeting (n = 6 groups)		Taping 2: storyboard/script development (n = 6 groups)		Taping 3: filming/editing (n = 4 groups)
Student objectives	Rater 1	Rater 2	Rater 1	Rater 2	Rater 1	Rater 2
1. All group members invite and listen to each other's ideas and perspectives.	67% (4 out of 6)	67% (4 out of 6)	83% (5 out of 6)	83% (5 out of 6)	—–	—–
2. All group members are given time to offer support and explanation for their ideas and/or perspectives.	83% (5 out of 6)	67% (4 out of 6)	100% (6 out of 6)	83% (5 out of 6)	100% (4 out of 4)	100% (4 out of 4)
3. Final decisions relative to video content are negotiated and agreed upon by all group members.	—–	—–	—–	—–	100% (4 out of 4)	100% (4 out of 4)
Percentage of Yes codes	75% (9 out of 12)	67% (8 out of 12)	92% (11 out of 12)	83% (10 out of 12)	100% (8 out of 8)	100% (8 out of 8)

The researchers involved in this project concluded that many of the most powerful student reactions to the video project were related to this development outcome. Students reflected upon the experience as one that introduced them to new ideas, new people, and new ways of behaving in a classroom setting. Although cross-cultural sensitivity and open-mindedness can take more than a semester to develop, such experiences are valuable in helping students confront internalized stereotypes and preconceptions (Hlyva and Schuh, 2004).

Similar to the Accountability and Responsibility SDO, students reported understanding how their individual preferences and experiences related to the overall group interaction:

I like working alone, but this was my first group project in college. I felt like this introduced me to the college world working with three other people who you do not know very well. I was really quiet at first. As we worked through, I was opening up and going outside of my box.

As discussed earlier, a wide range of students from diverse backgrounds were enrolled in the FYI course, and many described the experience as one of learning how to interact with others in a new way. The need to be able to adapt one's expectations to accommodate the ideas of others was frequently expressed; one student stated, for instance:

One thing that I would take away from this project was to keep an open mind when going into a project like this. An open mind with your group members, an open mind with the ideas people have, an open mind with how the filming process is going to go and how the editing is going to go. I guess I really learned what it means to be part of a group and to work cohesively together to create this great big awesome final product that we can be proud of.

Awareness of metacognitive processes was expressed most overtly in student responses to interview questions related to this outcome. As Baxter Magolda (2000) has noted, “the cognitive dimensions of self-authorship are intertwined with the interpersonal dimension” (p. 11), showing this link between this outcome and others investigated in this study. When asked to explain one of the most significant things he learned from the experience of working on the group project, one student described:

Working with somebody and having different opinions. They told you their side of the story and why they think that. [And I thought:] “OK, I did not think about that.”

Student responses also indicated that having a structured project guiding their group interactions made it easier to begin to get to know one another, but that once they felt more comfortable, the project itself became more enjoyable. In interviews with members of at least two groups, the dynamic appeared to switch from a task-oriented focus to one of member interaction over the course of the project.

Data Related to SLOs

SLO 1: Can Communicate Effectively.

Coding for SLO 1 (Can Communicate Effectively) was based on objectives that identified student behaviors related to identification of a target audience for the video, development of a functional storyboard, and actors' abilities to communicate within the final video. Data for this outcome (Table 8) show 50% and 67% (Rater 1 and Rater 2, respectively) of objectives met during the first recording, 75% (for both raters) during the second recording, and 100% (for both raters) for the third recording.

Table 8.

Scoring data for SLO 1: Can Communicate Effectively

	Taping 1: first group meeting (n = 6 groups)		Taping 2: storyboard/script development (n = 6 groups)		Taping 3: filming/editing (n = 4 groups)
Student objectives	Rater 1	Rater 2	Rater 1	Rater 2	Rater 1	Rater 2
1. Group members communicate effectively with each other.	17% (1 out of 6)	33% (2 out of 6)	83% (5 out of 6)	83% (5 out of 6)	100% (4 out of 4)	100% (4 out of 4)
2. Group members discuss and decide the audience for video.	83% (5 out of 6)	100% (6 out of 6)	—–	—–	—–	—–
3. Group members develop a storyboard that includes how their message will be conveyed.	—–	—–	67% (4 out of 6)	67% (4 out of 6)	—–	—–
4. Actors use eye contact, confident and audible voice tone, and effective language throughout video presentation.	—–	—–	—–	—–	100% (4 out of 4)	100% (4 out of 4)
Percentage of Yes codes	50% (6 out of 12)	67% (8 out of 12)	75% (9 out of 12)	75% (9 out of 12)	100% (8 out of 8)	100% (8 out of 8)

To investigate how well students were able to use strategies to encourage effective communication within their groups, representatives were asked to explain how decisions were made and how different ideas were considered. The selection of an audience for the public service component of the final video was an issue of considerable discussion in groups. When asked to explain why her group chose to create a video highlighting dining hall food options, one student said:

We thought that because of the problems with obesity now, it would be good to get our point to college students who can be the ones who change the obesity problem of the future.

The way in which groups decided to present their message also demonstrated an awareness of different communication styles and effectiveness. One group appealed to logic, for example:

We thought that we can't change the older people, so we thought we would target the younger kids. We showed the younger kid going into the store, and then choosing which cereal is right for them, and going back and thinking, “OK, this is what I have.” We gave them reason[s] why this one is good and why this one is bad.

In contrast, another group relied on emotion:

If parents were to watch our video, they would definitely see our message from our skit. When you watch it, it shows “Why would you ever put soda into a sippy cup?” That goes right through their teeth. Their teeth are going to rot. The kid is going to be malnourished. It is going to be terrible. So when you watch it, you think, “Why would a parent ever do that?” I definitely think that we got our point across well.

Communication skills were demonstrated by group members' interactions with one another during classes and meetings, as well as in the way in which the final video transmitted each group's desired message. Despite the short length (30–60 s), the videos were informative and entertaining and designed to reach a variety of audiences. (Example student video projects can be viewed at: http://msjensen.cehd.umn.edu/student-videos/food.asp and in this article's Supplemental Material.)

SLO 2: Have Acquired Skills for Effective Citizenship and Lifelong Learning.

Coding for SLO 2 (Have Acquired Skills for Effective Citizenship and Lifelong Learning) focused on the identification of skills that went beyond the requirements of the course, behaviors related to the potential impact of the video on the target audience, using multiple perspectives to convey a message, and awareness of group cooperation. Data for this outcome (Table 9) show 50% and75% (Rater 1 and Rater 2, respectively) for the objectives met during the first recording, 39% (for both raters) during the second recording, and 75% (for both raters) for the third recording.

Table 9.

Scoring data for SLO 2: Have Acquired Skills for Effective Citizenship and Lifelong Learning

	Taping 1: first group meeting (n = 6 groups)		Taping 2: storyboard/script development (n = 6 groups)		Taping 3: filming/editing (n = 4 groups)
Student objectives	Rater 1	Rater 2	Rater 1	Rater 2	Rater 1	Rater 2
1. Group members articulate how they envision their video message impacting their intended audience.	83% (5 out of 6)	83% (5 out of 6)	50% (3 out of 6)	50% (3 out of 6)	50% (2 out of 4)	50% (2 out of 4)
2. Group members demonstrate awareness of which tasks will benefit from several people working together.	17% (1 out of 6)	67% (4 out of 6)	33% (2 out of 6)	33% (2 out of 6)	100% (4 out of 4)	100% (4 out of 4)
3. Group members bring in at least two different disciplinary perspectives on their chosen topic.	—–	—–	33% (2 out of 6)	33% (2 out of 6)	—–	—–
Percentage of Yes codes	50% (6 out of 12)	75% (9 out of 12)	39% (7 out of 18)	39% (7 out of 18)	75% (6 out of 8)	75% (6 out of 8)

Although broad in possible meaning, demonstration of effective citizenship and lifelong learning skills was defined here as combining information from multiple sources, collaborating on a variety of tasks, and applying knowledge to new contexts. Some students were apprehensive at the beginning of the project, due to the required cooperative nature of the assignment and unfamiliarity with other group members, but gained new appreciation for the learning process throughout the semester. One student described her experience as:

To be honest, at the beginning of the project when we found out who our group members were, I did not really know any of them. I was kind of [wary] as to how it would all pan out. To be honest, I was pretty negative towards who my group members were because I did not know them. For me, personally, I have really high expectations of myself. And when I do projects like this, I really want them to turn out really well because that is who I am. But after talking to my group members, and brainstorming for ideas, and being on the same page, it worked really well. I take back everything that I thought at the beginning of the project, because I think that is what I learned out of working with people that you don't necessarily feel comfortable with, finding a way to make it happen.

The necessity of building interpersonal trust and understanding in order to work together was acknowledged by students as a key part of being able to complete a group task that required combining multiple perspectives and ideas. As recounted by one student:

First of all, I did not know anybody, though I had class with them. To do this project, we had to know each other very well, and do work together. To start out with not knowing your group, and to come out with one final idea—that was a good experience—it was fun.

Students also expressed being intrinsically motivated by the process of working with other students rather than focused on the grade they would receive for the assignment:

The good thing was, when we met up for the video, we tried to have fun. We didn't say, “Oh this is a project, we need to be all serious.” We had fun. We were going to go out there and try our best at it and not think about the grading process. I met a lot of new people. This was one of the first classes where I felt welcomed to the university. I am going to miss it.

The ways in which students were able to create a group space in which multiple perspectives were discussed and considered showed development of collaborative process skills and respect for others. Connections between citizenship and academic achievement were clearly identified by students upon reflection of their experiences.

Study Limitations

This study documented the ways in which students negotiated the inclusion of two SLOs and two SDOs in a single course project. As these outcomes are not department-specific, the rating rubrics used in this evaluation address somewhat broad categories of student behaviors and responses. Despite the difficulty in measuring abstract constructs such as Appreciation of Differences, the high degree of interrater reliability among researcher observations shows attention to consistency in the rating process. It is important to note, however, that the observed behaviors in this study all occurred within intentionally organized small groups of students that were maintained throughout the project. The results are therefore limited to instances that took place in the context of the video project and that were observed to relate to the four outcomes. Although student reflections suggest we can be cautiously optimistic that participants may transfer their learning beyond this experience, no specific claims can be made that these behaviors are transferable to new groups or different courses. Our hope, of course, is that individual gains have been made that can be applied to new group endeavors. This investigation therefore uses qualitative approaches to explore a particular phenomenon in depth, while acknowledging limitations of generalizability to a larger context. All conclusions reached from this evaluation are specific to the goals chosen as indicators of student progress toward the learning and development outcomes included in this particular course capstone project.

Furthermore, the implementation of such assignments requires previous instructor effort to prepare students to complete a final group project. Results of this evaluation are specific to the unique interdisciplinary class design and physical space (active-learning classroom); applying this approach to dissimilar settings likely requires some adaptation. Despite these limitations, the data collected show promising results that are encouraging for instructors and university administrators interested in incorporating multiple objectives into first-year undergraduate courses.

CONCLUSION

Kuh (2008) describes two key goals of collaborative learning as “learning to work and solve problems in the company of others, and sharpening one's own understanding by listening seriously to the insights of others, especially those with different backgrounds and life experiences” (p. 20). Toward the goal of creating a campus culture that promotes student success and engagement, faculty can take advantage of the importance of peer influence by promoting student interaction through collaborative activities (Kuh et al., 2008). Additionally, many national organizations focused on improving higher education highlight the importance of adopting student-centered approaches for many reasons, including the need for effort to adapt to the increasing diversity of student populations (ACPA and NASPA, 2004). Understanding that classroom instruction is linked to student engagement and retention places additional importance on transforming pedagogical practice (Tinto, 2006–2007). Historically, science professors have focused on knowledge gains (learning outcomes) when assessing and evaluating students, assignments, courses, and even programs. Appreciation of differences, accountability and responsibility, and other outcomes relating to student development are fundamentally different goals that cannot be achieved through traditional lecture or individual study. Assessment of such outcomes is also a complex task that requires a shift in focus for instructors.

Creating assessment tools and evaluation structures that establish ways to “hear students' voices” is important in understanding the learning experience beyond academic outcomes (ACPA and NASPA, 2004, p. 33). Students' interpersonal skills were challenged and developed by their participation in this group project, and group members reported recognizing strengths in other classmates. In particular, feedback from English language learner students showed uniquely positive results for this population. Furthermore, research on student retention has found that “student engagement in educationally purposeful activities is positively related to academic outcomes as represented by first-year student grades and by persistence between the first and second year of college” (Kuh et al., 2008, p. 555).

Data from this investigation show that students did indeed demonstrate behaviors consistent with the four SLOs and SDOs through their participation in the capstone video project, and researchers concluded that the assignment is therefore worthy of keeping within the course curriculum. More broadly, this evaluation uncovered a depth and nuance of group dynamics frequently not emphasized in science assignments. Analysis of the interactions among group members involved in the capstone video project showed students engaged in a creative and complex process to combine diverse ideas into a single product.

Without the deliberate implementation of strategies to create a sense of community and belonging, freshman students enrolled in a large class at a large university can easily get a sense of insignificance and may discount their contribution to group endeavors. The efforts described in this study combined a focus on content learning with personal development and are important, because higher education institutions must pay attention to how students' individual growth relates to the larger context of group behaviors. Setting specific expectations that connect individual outcomes to cooperative experiences is challenging, but in keeping with the changing demands on colleges and universities to prepare students civically as well as academically. While results here are indeed positive, it is necessary to note that the project was not the only student-centered assignment of the semester; instructors made deliberate efforts to facilitate cooperative learning across the curriculum. Specifically, while the final group video project was the largest in scale, students were required to work in groups during most class sessions, and instructors coached students on the behaviors required for successful group performance, such as listening to one another and developing roles within groups. Daily group assignments included such activities as reading discussions, interactive review of exam questions, and peer review of student writing samples.

Many research studies, however, have documented the resistance of STEM faculty to give up instructor-centered instructional strategies in favor of a more student-centered approach (Knight and Wood, 2005). Reasons for this resistance are many, but include frustrations with the demands of managing group learning and lack of evidence of its effectiveness. To replicate the positive results shown here, other instructors must be willing to adapt classroom expectations and pedagogical approaches. Cooperative group projects need not take extensive time, elaborate equipment, or even team-taught courses, such as the FYI course used here. It is highly recommended that instructors who are new to cooperative learning begin with short, simple projects, such as cooperative quizzes (Jensen et al., 2002), and then progress to group projects, such as cooperative lab reports and short group presentations from either lecture or lab. Even informal group learning projects, such as think–pair–share, can allow students to begin to foster skills germane to development learning outcomes. Thoughtful consideration of how students are assigned to lab groups in introductory biology classes provides an opportunity for science instructors to take advantage of a preexisting class structure and increase the emphasis on both development and content learning outcomes. Particularly for students enrolled in science classes as nonmajors, authentic learning activities help address institutional challenges of creating “a scientifically literate population” (McPhearson et al., 2008, p. 150) despite great variation in previous exposure to scientific thinking and concepts and limited requirements to explore these topics in college.

Allen and Tanner (2005) also document the resistance of many students to increased responsibility when engaged in active-learning strategies. The varied experiences of first-year undergraduate students in their high school science classes may also impact how prepared students are to work on group projects. Given this resistance and the wide possible variety of group learning experiences, it is again important to emphasize that the capstone assignment was not the only group activity of the course. Much smaller-scale and lower-stakes group work was used throughout the course. Students learned to take responsibility for their own learning, while also taking on the responsibility of participating in groups in a positive manner. An awareness of personal learning and interpersonal communication style prior to beginning group work may aid students in their initial interactions, but scaffolding opportunities in class for students to explore how to work with others in new ways can provide a stepping-stone toward success in higher-stakes experiences. Despite positive results in this study, an important issue not addressed is related to the transferability of the behaviors exhibited, and hopefully “learned,” in this one course during the completion of one large project. Pursuing a research question related to the transferability of behaviors germane to SDOs is a logical next step in this expanding field of research, and consistent with the more heavily investigated area of the transferability of cognitive skills related to content learning objectives. In keeping with constructivist philosophy, however, it is logical to assume that integration of both developmental and content outcomes into multiple parts of the curriculum will allow for repeated opportunities for learning, and, hopefully, retention of this learning.

Changes in what students are expected to be able to do upon graduation, in addition to what they are expected to know are clear. Increasing numbers of graduate and professional schools (including medical schools and programs in other health-related career fields) require more from their applicants than evidence of raw content knowledge—rather, they are looking for multiple skill sets that include the ability to work on diverse teams, demonstrated appreciation of differences, and other characteristics related to development outcomes. The implications for instructors also clearly point to the need for a shift in pedagogical approach. While working in groups, students used skills that are recognized as essential for future productive citizens, scientists, and engineers (see NRC, 1999, 2003b). Individual and focus group interviews allowed students to further reflect on the group video assignment. Many participants showed signs of personal growth relative to these indicators, and also showed genuine enthusiasm for the project.

Undergraduate science professors should respond by modifying individualistic assignments and traditional content-focused exams, and move to the underutilized world of cooperative group learning. It is also through such experiences that students engage in more authentic, project-based science work. Scientists and medical professionals do not work alone—they work in teams and their performance is interdependent. A clear need exists for future efforts to focus more attention on the development of effective ways to evaluate progress toward these outcomes, to define ways to establish students' incoming level of familiarity with cooperative group work, and to build university-wide support for such projects. Through participation in a final group project, students in this study were indeed engaged in tasks that promote progress toward both learning and development outcomes. Other instructors are encouraged to adapt and share similar ideas and approaches toward the goal of improving learning for all students.