Tools for Change: Measuring Student Conceptual Understanding Across Undergraduate Biology Programs Using Bio-MAPS Assessments

Abstract

Assessing learning across a biology major can help departments monitor achievement of broader program-level goals and identify opportunities for curricular improvement. However, biology departments have lacked suitable tools to measure learning at the program scale. To address this need, we developed four freely available assessments—called Biology-Measuring Achievement and Progression in Science or Bio-MAPS—for general biology, molecular biology, ecology/evolution, and physiology programs. When administered at multiple time points in a curriculum, these instruments can provide departments with information on how student conceptual understanding changes across a major and help guide curricular modifications to enhance learning.

INTRODUCTION

Program-level assessments can be important tools to measure student learning during undergraduate majors. Assessing students at multiple time points throughout the curriculum can motivate faculty to discuss and come to agreement on the essential learning outcomes of their program and consider how students will achieve these outcomes regardless of the specific courses they take. By providing information on student understanding at key time points, program-level assessments can also assist departments in determining the cumulative impact of their courses and pinpointing areas for improvement.

At a broader level, program assessment has been propelled by a national interest in ensuring that college participation leads to measurable learning outcomes. Accreditation agencies are increasingly requiring that programs collect and respond to assessment data (1). A survey of chief academic officers revealed that regional and program accreditation represent the primary drivers for program assessment, even outranking departmental and institutional commitment to improvement (2). In light of rising tuition costs, colleges and universities are also facing increasing pressure to document impacts on students, while accounting for incoming attributes and abilities (3, 4). Currently, programs use a variety of approaches to gauge student outcomes and respond to questions regarding the added value of a college degree. These approaches include capstone projects, learning portfolios, and performance assessments. To ensure that the approaches and assessment results have meaning, it is important for departments to involve faculty throughout the assessment process, including in decisions on what to assess, which approaches would provide adequate evidence of student learning, and how student performance aligns with broader disciplinary standards (5).

To conduct program assessment within a discipline, departments must first identify the central goals of their program. Program-level learning goals specify what students should be able to do at the end of a major. Although crafting and reaching consensus on these goals takes time, biology departments can capitalize on a growing pool of available guides. For example, the Vision and Change report articulates core concepts and competencies for undergraduate biology programs (6), and the BioCore Guide (7) and Conceptual Elements Framework (8) further delineate how the core concepts apply within biology. CourseSource also houses extensive lists of learning goals for biology subdisciplines developed by multiple life science professional societies, including the American Society for Microbiology (click on the relevant course at https://www.coursesource.org/courses).

Assessing student achievement requires that departments have access to instruments that align with their learning goals. Since departments typically have limited financial and personnel resources for program assessment, these instruments ideally would be freely available and easily administered and scored. Until this point, biology departments have had few options for valid and reliable tools to measure advancement toward program-level conceptual learning goals. Most tools have measured student understanding of a specific topic, and few instruments gauge student understanding across the breadth of a major. Although general biology assessments have been developed commercially (e.g., the Biology Majors Field Test), these tests are usually administered only at the end of a degree program and have costs that must be covered by either the student or institution.

An illustrative example: Using program assessment to promote transformation in microbiology education

Within the literature, one prime example stands out in which instructors used program assessment as a mechanism to build and refine their undergraduate program. In this case, biologists who taught related microbiology courses at the same institution formed a teaching group to coordinate their curriculum and improve undergraduate learning. Working together, this group identified content linkages and overarching learning goals, chose specific microorganisms to serve as recurring model systems, and implemented active learning methods across their courses (9). Building on their learning goals, they developed a response-validated assessment focused on host–pathogen interactions and administered this assessment within introductory and advanced courses (10). Students demonstrated growth within their introductory course, maintained this performance over time, and showed further improvements within some advanced courses. The authors proposed using the varied student performance in the advanced courses to make data-driven changes to content and instructional practices. Importantly, assessment data enabled this group to refine their expectations for student learning, reflect on how curriculum coverage compared with actual performance, and understand how the curriculum served particular student groups (11).

New tools for program assessment: Bio-MAPS

While this example provides a roadmap for departments to follow, several steps in the process, such as generating an assessment, can be time consuming and require specialized expertise. To help alleviate issues associated with designing program assessments, we developed a suite of freely available instruments called Biology-Measuring Achievement and Progression in Science or Bio-MAPS. These assessments align with the core concepts from Vision and Change (6, 7), with separate instruments for general biology (12), molecular biology (13), physiology (14), and ecology/evolution (15). We followed established methods of assessment development (16), including optimizing response validity through student interviews, ensuring technical accuracy through expert reviews, pilot testing at a variety of institutions across the country, and addressing any potential item biases favoring particular demographic groups. Each question stem presents a scenario, followed by a series of statements that students evaluate as “true/false” or “likely/unlikely to be true” (Fig. 1). These formats allow the rapid collection of fine-grained information on student understanding of many concepts, while retaining the convenience of automatic grading (17, 18).

FIGURE 1.

Example question from the GenBio-MAPS assessment (12). The question stem presents drawings of chromosomes, gene loci, and alleles. Students answer true/false statements related to the question stem. Correct answers are highlighted in green. Note that any number of statements can be true or false for different question stems.

To support faculty in using these assessments for large-scale evaluations, we also developed a web portal to facilitate Bio-MAPS administration (http://cperl.lassp.cornell.edu/biomaps), building on existing portals in physics education research (19). Through this portal, instructors or assessment coordinators fill out a brief survey and then receive a web link that they share with their students. Students complete the assessment online and outside of class, eliminating the need to devote class time to administering the assessment. This approach has been found to produce scores similar to in-class administrations (20). Because the instruments are typically administered within the context of individual courses, faculty can offer a few points of participation credit, which helps motivate student participation. After students complete the assessment and access has closed, the portal returns summary reports to the instructor or assessment coordinator, providing aggregated information on overall student performance and performance for each of the Vision and Change core concepts (Fig. 2).

FIGURE 2.

Example score report from students at one institution who took the GenBio-MAPS assessment just before graduation (advanced time point, n=127 students). Central bars represent median test scores, boxes represent inner quartiles, and whiskers represent minimum/maximum scores. Each dot represents one student’s average percent correct for all true/false items aligned with the indicated Vision and Change core concept.

Bio-MAPS assessments are specifically designed to measure student conceptual understanding at key time points across a major (Fig. 3); they are not intended to be given as pre/post tests around a single course. These time points typically include beginning of introductory biology: the start of the program (e.g., prior to classes or at the beginning of the first course in the major), end of introductory biology: after the completion of an introductory course series, and advanced: at the end of courses typically taken by seniors just prior to graduation. Within our multi-institution data sets, we find overall increases in student performance over time for each Bio-MAPS instrument (12, 14, 15). Students demonstrated large gains for some concepts, while other concepts remained challenging, even for students about to graduate. Importantly, we showed that student performance trajectories differ across institutions, indicating that these assessments can detect program-specific strengths and limitations.

FIGURE 3.

Example assessment administration timeline based on two-semester academic calendar. Prior to adopting an assessment, departments should discuss their program goals and come to agreement on the use of a particular instrument. Bio-MAPS instruments can be administered through the web portal (A) at the beginning of the fall semester to students who are starting the intro series and incoming transfer students, (B) at the end of the fall semester to students who are finishing the intro series and graduating seniors, (C) at the beginning of the spring semester to students who are starting the intro course series and incoming transfer students, and (D) at the end of the spring semester to students who are finishing the intro course series and graduating seniors. After administration, the score reports automatically generated through the web portal can be reviewed. Throughout the academic year, faculty can meet to discuss data-driven adjustments to their program.

Using program assessment data to facilitate change

After collecting and processing assessment data, departments can compare performance on individual items over time to diagnose student understanding of specific concepts. By comparing longitudinal or cross-sectional data across time points (example shown in Fig. 4), departments can identify informative trends in student performance, such as when a particular concept shows high incoming performance, little growth during a time period, or high achievement at the end of advanced courses. In addition, departments can administer a single instrument across all time points, or they can administer different instruments at different time points to gauge mastery of advanced subdisciplinary concepts. For example, in programs where students take an introductory course series and then specialize into subdisciplinary tracks, faculty may choose to administer the general biology instrument at the first two time points and the subdiscipline-focused instruments at the last time point to students in the corresponding tracks. Furthermore, departments can incorporate more focused assessments within specific courses as needed to achieve fine-scale information on student learning and progress (21).

FIGURE 4.

Student performance at different time points on two of the GenBio-MAPS true/false statements shown in Figure 1 (n=137 beginning of intro, 176 end of intro, and 127 advanced students from one institution). Students show increased performance on Statement a, which assesses the concept that genes located close together on a chromosome are linked and will tend to be inherited together more frequently than genes located far apart on chromosomes or on different chromosomes. In contrast, students show little growth on Statement d, which assesses understanding of the differences between sex chromosomes and the location of genes involved in the development of sex organs.

Program-level assessments support a data-driven approach to helping departments advance their educational missions, rather than relying on the perceptions of individual faculty or anecdotal feedback from a small group of students. Capturing data at the program level over multiple time points encourages faculty to engage in a broad dialogue about overall goals and solutions rather than limiting discussions to the single-course level. For concepts showing little improvement over time, faculty can discuss whether and how current courses address these concepts and consider structural, curricular, or pedagogical changes to improve learning. For example, information about student growth on evolution concepts inspired one department to start requiring that all students take an evolution course at the sophomore level (15). For particularly challenging concepts, faculty can consult the literature to find additional studies and classroom lessons to incorporate into individual courses. In cases where students in a particular curricular track (e.g., students with sub-specializations or taking particular courses) demonstrate low performance, it is helpful for departments to keep the conversation at the curriculum level and focus on collective program goals rather than individual courses.

CONCLUSIONS

The promise of program-level assessment lies in its ability to help departments identify central goals, monitor student learning, and chart a course for program improvement. These assessments also enable departments to determine the impact of subsequent interventions, such as changing the course sequence, introducing clicker questions with peer discussion, or developing online courses. When combined with demographic information, these data allow departments to determine whether their programs adequately serve students from particular groups, including community college transfer students, students from historically underrepresented groups, or students participating in summer bridge programs. Access to assessment data may also prompt departments to explore new questions, such as how background preparation influences student performance and what additional resources faculty and students need to achieve success. Program assessment ultimately has the potential to improve student retention by helping departments understand how students with different backgrounds progress through their major, determine how conceptual understanding at each time point relates to persistence, and make targeted changes to help boost student achievement. Bio-MAPS results disaggregated by demographic data can also inspire departmental discussions about the importance of ensuring that curricular decisions promote inclusive excellence and help provide opportunities for all students to maximize their learning regardless of background (22, 23). While the impact of college cannot be fully captured by a single instrument, Bio-MAPS assessments enable departments to measure student learning of core concepts across key time points and make data-driven decisions about their undergraduate curriculum.