Implementing Concept-based Learning in a Large Undergraduate Classroom

Abstract

An experiment explicitly introducing learning strategies to a large, first-year undergraduate cell biology course was undertaken to see whether awareness and use of strategies had a measurable impact on student performance. The construction of concept maps was selected as the strategy to be introduced because of an inherent coherence with a course structured by concepts. Data were collected over three different semesters of an introductory cell biology course, all teaching similar course material with the same professor and all evaluated using similar examinations. The first group, used as a control, did not construct concept maps, the second group constructed individual concept maps, and the third group first constructed individual maps then validated their maps in small teams to provide peer feedback about the individual maps. Assessment of the experiment involved student performance on the final exam, anonymous polls of student perceptions, failure rate, and retention of information at the start of the following year. The main conclusion drawn is that concept maps without feedback have no significant effect on student performance, whereas concept maps with feedback produced a measurable increase in student problem-solving performance and a decrease in failure rates.

INTRODUCTION

Mastery of science requires understanding a wide range of different concepts, most of which are specific to the particular subject. Within a discipline, however, the essential concepts generally seem to have broad consensus. For example, when course outlines for a random selection of first-year cell biology courses are compared, the concepts appearing in the course outlines are remarkably consistent, and correspond well to the basic concepts inventoried as part of a recent hierarchically ordered biology concept list (Khodor et al., 2004). Cell biology concept inventories, following along the lines of concept inventories in other areas such as physics (Hestenes et al., 1992), are important because they focus attention on the concepts that should be included during course development and can eventually be used to develop standardized tests to ensure that subject mastery has been met according to national scales.

The selection of concepts and preparation of concept-based course material represents only part of the teaching and learning equation. In fact, the desired end point of a course, the acquisition of new knowledge by the students, may be facilitated but is not a direct function of how the course material is organized. In contrast, what students learn can be profoundly influenced by two other factors, the use of strategies (or active learning) and a student's motivation to learn the material. The use of learning strategies is a part of the constructivist perspective, where learning is proposed to involve the active construction of knowledge by a student (Allen and Tanner, 2005). There are many types of learning strategies, classified in general terms into cognitive, metacognitive, emotional, and management categories, and all can have some impact on student learning (Tardif, 1997). However, the cognitive concept mapping (CMAP) strategy of Novak (Novak and Gowin, 1984) seems ideally suited to cell biology, because it is a course already organized by concepts. In the preparation of a CMAP, students are asked to select important facts, organize them, and integrate them into a complete picture. The CMAPs are inherently personal representations of what have been learned, can be used for evaluation purposes (McClure et al., 1999), and have been demonstrated to be an effective learning strategy (Horton et al., 1993; Novak, 2003).

The motivation of students is often ignored during course preparation and delivery but can be a potentially powerful lever to influence student learning. A motivated student effectively uses learning strategies and is willing to make an effort to learn even without an initial success (Tardif, 1997). Some motivational forces are internal to the student and can be learned, such as an effective use of cognitive and metacognitive strategies (Viau, 1994). Other motivational forces are external to the student and can include help from the professor or from other students (Viau, 1994). However, among the most important motivational forces are the perceived value of the material (“why do I do it?”), the perceived ability of the student (“can I do it?”), and the perceived control over the process (“does effort produce results?”). A sense of control over the final outcome of their own learning process is important (Tardif, 1997): Students who have the impression that nothing they do will alter the result of the learning process, or who attribute success to good luck and failure to bad luck, or who see the pedagogy and didactic practice of the professor as the sole determinant of success or failure, will make little effort to contribute to their own learning. In the context of a concept-based cell biology course, the perceived value or significance of the material can be targeted by reference to the same material covered in concurrent courses, to a demonstration that the knowledge is essential for professional expertise, or to its relevance with respect to social issues. Perceived ability can be targeted by careful choice of graded problems presented to the students. Here, accurate estimation of the difficulty of the problems is paramount, as problems that are too easy preclude significant learning, whereas problems that are too difficult frustrate rather than stimulate. Perceived control by the students over their academic performance represents the aspect directly targeted by explicit teaching of learning strategies. Ideally, this teaching should be accompanied by a demonstration for the students that learning strategies do indeed work.

In the spirit of scientific teaching (Handelsman et al., 2004), the experiment described here reports an impact assessment of implanting a concept mapping learning strategy into a concept-based introductory cell biology course. The methods used are applicable to large classrooms and do not require teaching assistants.

COURSE DESIGN

Context

BIO1153 is a three-credit-long, first-year undergraduate cell biology class given to ∼ 220 students at the University of Montreal during their first semester. Students enter the 3-yr university biology program after 13 years of pre-university schooling from a number of different schools, although many use the same text (Campbell, 2001) for biology instruction. This introductory cell biology class is a required course, and the content is relevant for roughly half of the other courses (particularly biochemistry, genetics, molecular biology, and physiology) offered in the department. All formal instruction is in French, but translated versions of most standard cell biology textbooks are available. BIO1153 explicitly references the textbook of Alberts et al. and the related problems book (Alberts et al., 2004; Wilson and Hunt, 2004). The course is expected to require roughly 2 h home study for every hour of class time, although this level of effort may not always be possible for all students, as many students work at least part-time in addition to their university studies.

The choice of material included in this first-year course is also influenced by the fact that a second-year cell biology course is available. The second-year course focuses primarily on signaling, cell cycle, and development, so none of these subjects are treated extensively in the first-year course. It is also noteworthy that the same professor is responsible for both first-year and second-year courses, limiting the time that the professor can devote to grading and assisting students.

Course Objectives

BIO1153 has four goals:

1. to expose students to the basic vocabulary and concepts of cell biology, with particular attention to the relationship between structure and function;

2. to develop critical-thinking skills, with emphasis on the correct interpretation of scientific data;

3. to develop a social conscience in issues where cell biology impacts directly on society; and

4. to explicitly expose students to cognitive and metacognitive strategies in learning and develop facility with the concept mapping strategy.

The learning outcomes expected when these teaching goals have been met are:

1. participants will respond adequately to standard multiple-choice questionnaires testing the basic vocabulary and concepts of cell biology;

2. participants can correctly interpret the results of scientific data given in the context of a problem;

3. participants can understand and discuss the scientific basis of current social issues in cell biology; and

4. participants are able to effectively use concept mapping as a learning strategy.

Preparation of Modules

To achieve the course goals, the course material has been constructed as a series of eight modules, each of which deals with a particular subject and contains between five and 14 concepts. Each concept is designed to take up roughly 15 to 25 min of lecture time and has a standard format typically consisting of nine elements:

• Concept. An explicit statement of the concept.

• Background. A reactivation of previous knowledge.

• Idea. A statement of the basic idea or an analogy to illustrate it.

• Example 1. A concrete example of the concept.

• Example 2. A counterexample (if possible).

• Example 3. Another example of the concept.

• Summary. A resume of the material.

• Questions. A series of questions to test comprehension of the concept.

• Problem. A problem with real data to be solved in class.

This approach is derived from a previously described instructional design method (Gagne, 1977). The goal of the presentation format is to facilitate the construction of new knowledge by making explicit the basic concept and the foundations upon which it lies. The alternation between example, counterexample, and example is performed to allow the students to decontextualize then recontextualize the concept. The end-of-concept questions are very general and are provided to allow the students to evaluate their comprehension of the concept. Lastly, the problems are meant to illustrate the concept using real experimental data or issues in cell biology. These problems are read by the students before the lecture and then discussed in class in the light of the lecture material. The problems used were derived from a standard text of cell biology problems (Wilson and Hunt, 2004), although in many cases the precise question asked has been modified to take into account the students' ability and coherence with the concept in which it is embedded.

The conceptual structure of the modules is formulated in a manner that roughly recapitulates the conceptual order found in the text (Alberts et al., 2004). The Massachusetts Institute of Technology hierarchical concept framework (Khodor et al., 2004) has also been partially incorporated where possible to facilitate the preparation of concept maps from the material.

The decision to incorporate real-data problems into the concept presentation framework was based on three factors. First, problem solving provides a break in the standard lecture format by allowing students to discuss the subject with their colleagues in class. Second, the problems provide an alternative illustration of the concept that the students have to come to on their own. Lastly, the solution of problems during the semester not only increases student interest in the course but increases performance (Kitchen et al., 2003). The integration of the problems was expected to motivate the students into a greater intellectual investment when assimilating the course material.

The concepts each require roughly 20 to 25 min of lecture time, after which ∼5 min class time is allotted for either resolution of a problem related to the concept, an in-class demonstration of the concept, or to solicitation from the students thoughts of how the concept may impact society. There is thus a change of pace and style at regular intervals that is expected to maintain interest and motivation. Problems, when included in the concepts, are taken from the accompanying problems book (Wilson and Hunt, 2004). Class demonstrations involve simple “experiments,” such as passing the beam from a laser pointer through electron microscopy grids with different grid spacings to show that diffraction patterns become more spread out as the grid spacing decreases. Solicitation of student opinions can range from their thoughts on Botox to whether insurance companies should have access to their genome sequence. A detailed plan of the course (provided in Supplemental Material 1) shows the selection of concepts, examples and accompanying problems, strategies, and discussion items.

Preparation of Students for Concept Mapping

CMAP is a learning strategy developed by Novak in the 1980s (Novak and Gowin, 1984), and involves selecting key ideas or concepts from the material, organizing the ideas into a hierarchical structure, and then linking the ideas together by verbs or prepositions. This strategy allows a student to place new material into a knowledge structure compatible with previous knowledge in a manner that makes it accessible in other situations. However, the strategy must be taught explicitly to the students before it can be used. The first step in implanting the strategy of concept mapping (Novak and Gowin, 1984) requires showing the students what the strategy entails, how it will work in the context of the course, and why it is important. During the first several minutes of the first lecture, the students are told that:

• they will be required to hand in visual summaries called concept maps (CMAP) for each module;

• each CMAP package will include a team CMAP and an individual CMAP from each member (team maps can be either a new construct or simply an individual CMAP that adequately reflects the material);

• the first lecture will include examples of CMAPs and a series of exercises designed to teach concept map construction; and

• points will be awarded for handing each CMAP package in on time.

The first example of a CMAP shown represents the course material that will be covered, and serves to both introduce the course and to demonstrate the three essential elements absolutely required for efficient utilization of the concept mapping strategy:

• the selection of key words or concepts;

• the arrangement of the concepts hierarchically on a page; and

• the connection of the concepts by lines labeled with verbs or prepositions.

The first lecture provides practice for each of the three essential elements separately and then together. These practice times are given at the end of concepts, in place of the problems that normally end the concept in the other modules. To provide practice of the first element, the students are asked to select five or six key words that best reflect or describe the material covered. After presentation of the concept, they are allowed to discuss their choices with their immediate neighbors for several minutes, after which the obvious choices are presented and discussed. For practice of the second element, a predetermined series of six key words is given at the end of a concept, and the students are asked to arrange then hierarchically on a piece of paper. Again, discussion is encouraged and the obvious organization is presented and discussed. For practice of the third element, a series of six key words already arranged hierarchically with connection lines lacking labels is provided, and the students are asked to label each connection line. Again, after several minutes the obvious choices are presented and discussed. The last exercise, in which the students are asked to perform all three tasks sequentially, is given for the last concept of the lecture. At the end of the lecture, a summary CMAP of the first module is presented and discussed. This example also illustrates a grading scheme for concept maps that gives 1 point for each correctly identified vertical link, 5 points for each valid hierarchical level, and 10 points for each correctly identified horizontal link (Novak and Gowin, 1984). This scheme is important as it underscores the importance of hierarchical concepts and the ability to make links between different areas of the construct.

The homework is scheduled so that students prepare their individual CMAPs from the module before the class, and have a full week to consult with their team and prepare the consensus team CMAP; individual and consensus maps are handed in together at the start of the next lecture.

Preparation of Students for Problem Solving

No training for solving problems is given during the first lecture because of the emphasis on CMAP training. However, during subsequent lectures, strategies appropriate for problem solving are provided during the in-class discussion of the problem. The problems are given in the form of text with a figure and figure legend. Experience has suggested that the most useful strategy in problem solving is given as a series of steps:

1. scanning the text to identify the question to be answered;

2. writing a description of what is understood in the figure;

3. attempting to answer the question using the description of the figure;

4. identification of any elements in the figure that are not clear;

5. scanning the text for explanation of those elements specifically.

This strategy is meant to make the students more comfortable with the problem solving in an exam situation.

Observations During the Experiment

The first lecture session appeared to be effective in terms of student training. However, formation of the teams was problematic, perhaps exacerbated by shyness and by a growing tendency for students to work in addition to their studies (which severely constrains their availability). In the first year of implementation, teams were suggested but not made mandatory. Roughly three-quarters of the students chose to produce individual CMAPs rather than work in teams, and these students were thus unable to obtain feedback on their CMAPs. During this year (2004), no impact of the CMAP was observable on any measurable parameter of student success (see below). Consequently, teamwork was made obligatory during the following year (2005). Teams were formed randomly using an automated function in WebCT, the university platform for e-learning. Facilitating teamwork is time consuming to the instructor. Difficulty in contacting partners, and student withdrawal from the course both cause problems that have to be solved by the instructor. The method chosen in 2006, in which students were asked to make their own teams, thus appears to have the best cost-per-benefit ratio.

An additional feature of WebCT is that students can send messages either to a specific individual or to a group forum. These online communications can be monitored by the course administrator, and provide a valuable window into problems experienced by the students during the course. With respect to the CMAPs, some discussion in the forum centered around whether or not it was worth investing time in preparing the CMAPs, as several students noted grades were assigned for completion of the maps and did not reflect their quality in any way. The counterargument, slightly more prevalent than the first, noted that the CMAPs were personal study aids and would not contribute to success on exams unless they were done well.

A useful mapping program (CMAP tools) compatible with both PC and Mac is distributed free of charge by the Institute for Human and Machine Cognition (http://cmap.ihmc.us). The computer version is superior to pen and paper in that the maps can be readily reworked, and an advanced version of the program can accommodate multiple users working on the same map from different locations. Although none of the student teams chose to use the team map function, more than 95% of the students did use this tool in preparing their individual CMAPs.

CMAP Evaluations

Evaluating an individual CMAP requires between 20 and 30 min. Thus, given the class size and the total number of maps asked of the students (seven over the semester), extensive evaluations of all maps during the semester is not possible. This is the principal reason for peer group evaluation of the maps, as otherwise no feedback about the validity of the map can be provided to the student.

It is, however, possible to evaluate the maps after the course, and although this evaluation neither helps the students nor contributes to their grades, it does provide a measure that can be tested for correlation with other measures of student performance. CMAPs from two of the modules were selected for analysis: CMAP2, which requires extensive hierarchical ordering, and CMAP6, which lends itself particularly well to horizontal links. Before the students constructed CMAP2, they were informed that hierarchical ordering was an especially important element for proper representation of the material. Each individual student CMAP was then scored as to whether or not hierarchical ordering was present; for those in which hierarchical ordering was present, two different patterns were noted, and these two patterns were scored independently. Before the students constructed CMAP6, they were asked to pay special attention to the formation of horizontal links. As for CMAP2, the individual CMAP6 was scored as to the presence of horizontal links, and when present the type of pattern observed. In the year where both individual and consensus CMAPs were requested, both were scored.

In a more detailed analysis of the CMAPs, 30 students were selected from the class of more than 200 on the basis of their final exam performance (the 10 individuals with the highest grades, the 10 with the poorest grades, and 10 individuals with average grades). For each of these 30 students, two maps (CMAP2 and CMAP6) were evaluated quantitatively using the numerical scheme previously described (Novak and Gowin, 1984).

Evaluating Student Performance and Impact of the CMAP Strategy

Student evaluations for grading use both multiple-choice exams (GRE, or graduate record exam style) and open-ended problems (as found in the problems book of Wilson and Hunt, 2004). Multiple-choice questions are computer graded, and the five or six open-ended problems on the midterm and the final exam (20% of the exam), respectively, are evaluated by the instructor. A student's final grade is the sum of the midterm, the final exams, and a total of 20% given for production of CMAPs. Note that grades are awarded for production of the concept maps and are not influenced by the quality.

To assess the impact of implementing the CMAP strategy, both final examination results and student failure rates were taken as measures of student success. The computer-graded multiple-choice questions and hand-graded answers to the problems in the final exam were noted separately for each student. Student failure rate was scored as the number of students whose final grade was below 50%. Note that the CMAP project began in 2004, so student final grades in 2003 reflect only midterm and final exam scores, whereas in 2004 and subsequent years, the final grade also includes the points awarded for CMAP production.

At the end of the school year (last class), students were also asked a series of multiple-choice questions directly soliciting their opinions on the use of the concept mapping strategy and the inclusion of the problems as a learning aid. The replies are anonymous as questions were given to the students in the context of the yearly course evaluations (the professor is absent). These results were compiled by the same university service that processes the other questions on the evaluation.

An additional evaluation involved a short multiple-choice questionnaire given to students enrolled in a second-year cell biology course. The majority of the students in the second-year course took BIO1153 the previous year, and the questionnaire was intended to estimate the retention of material given during the first-year course. The results from the second-year students during academic year 2006 were not included as the first-year class for these students was interrupted by a labor dispute at the university during which three courses (of 12) were not given and the CMAP experiment was abandoned.

Analysis of the Impact of the CMAP Strategy

The collected data, including student grades during the exams (with problems and multiple-choice questions collected separately) and the CMAP scores for the 30 selected students, were entered into an Excel spreadsheet. In the first analyses, the effect size of Horton (Horton et al., 1993) was calculated using the formula:

Effect size = (Mean_experiment − Mean_control)/SD_control

Means and standard deviations were calculated from the data using the preset programs in the spreadsheet. The problems and multiple-choice questions were maintained separately for this analysis as multiple-choice questions were expected to test lower levels of thinking (knowledge and understanding) as defined by Bloom's taxonomy (Anderson and Krathwohl, 2001), whereas the problems were expected to test higher levels of thinking (application and analysis).

In a second series of analyses, a possible correlation between CMAP scores and final grades was tested. For this analysis, the CMAP score for an individual was plotted as function of the exam score, a straight line drawn through the data points by linear regression, and the degree of correlation tested by calculation of the coefficient R².

A last test was performed with the 2006 group to gauge their performance with respect to national norms. To accomplish this, 28 multiple-choice questions covering the course material were selected from old GREs in biochemistry and molecular biology that are supplied to students as a study guide and preparation for taking the exams (www.ets.org/Media/Tests/GRE/pdf/BioChem.pdf). These practice exams are useful because they contain not only answers but also the national average of successful responses for each question. The means and standard deviations of the success rate of the class were calculated and compared with those derived from the national average for all questions combined. In addition, the success rate for each question individually was tested for a possible correlation by drawing a linear regression line through the data points of class score plotted as a function of national average score.

RESULTS

The rationale for this study is displayed as a concept map in Figure 1. The concepts selected as course material represent the bulk of the information that must be assimilated by the students, and this is expected to be more efficient when students are motivated and when they are effectively using learning strategies. Motivation for map construction is affected by allotting points for completed maps and by the constant feedback students obtain about their maps though team consultation.

Figure 1.

Concept map of the project and its theoretical foundation.

During the first 15 min of the first class, students were asked to complete a short questionnaire to survey the general level of background knowledge in cell biology (10 multiple-choice questions on material found in the text used for most pre-university biology courses). Students were asked to select all applicable answers to a specific question, and on average, selected right answers 67% of the time and wrong answers 11% of the time. This score is similar to the average cell biology grade at the end of the semester (72%) so it may reflect the acquisition of basic knowledge capacity of the class. Students were also asked to list any or all learning strategies they typically used. The responses were classified into real learning strategies or not based on a list of strategies (provided in Supplemental Material 2) compiled from a published work (Barbeau et al., 1997). In contrast to their performance on biology context questions, only 30% of the 236 responses represented real strategies (Table 1). Indeed, the majority of the responses erroneously identified simply reading and rereading the course notes as a learning strategy. This level of awareness suggests that explicit instruction in strategic learning could indeed be beneficial. Lastly, students were also asked why they were enrolled in the biology program, with 51% expressing a desire to work in a biology-related field, although a significant percentage (38%) indicated biology was a stepping-stone to a medical school application.

Table 1.

Most students do not use efficient learning strategies

Study methods	Number of responses
	Strategic	Nonstrategic
Read and reread		68
Make summaries	29
Regular revision		17
Do exercises		17
Take notes in class	17
Make schemas	16
Listen attentively in class	15
Cram		7
Memorize		7
Other	5	28
Total	82	144

Responses elicited by an anonymous survey of study strategies used were grouped into similar types and the types classified as either cognitive learning strategies or not based on a list of learning strategies (provided in Supplemental Material 2) compiled from the literature (Barbeau et al., 1997).

The impact of introducing the CMAP strategy into the class might be expected to depend on the degree to which it is used effectively. Indeed, a problem with implementation was detected during the first semester of the CMAP experiment (2004) by simply counting inadequate CMAPs. For example, during construction of a concept map describing different microscopes (CMAP2), the students were expected to organize what they had learned. This module lends itself particularly well to the use of hierarchical structure in the maps. However, the most frequent type of CMAP produced lacked hierarchical order (Table 2). An example of this type of construction (Figure 2A) is akin to a spider, with all the different microscopes emerging like legs from the same body. To a lesser degree, hierarchical order was found among the maps in one of two different patterns (Table 2), one in which the microscopes were ordered by the wavelength of incident radiation (Figure 2B) and the other which differentiated between microscopes in which light passed through or did not (Figure 2C). An example of an instructor CMAP of the material is also presented for comparison purposes (Figure 2D).

Table 2.

Feedback is important for adequate conceptualization

	Number of students
	2004	2006 Individual	2006 Groups
CMAP#2
No hierarchy	170	149	75
Hierarchy (pattern A)	15	34	71
Hierarchy (pattern B)	24	33	65
Hierarchy (other)		2	7
CMAP#6
No horizontal links	165	111	80
Links (pattern A)	19	64	84
Links (pattern B)	16	15	26

Concept maps from module 2 (microscopy) and module 6 (energy metabolism) were grouped into types (examples of each are given in Figures 2 and 3, respectively), and the number of students producing maps of each type tabulated. Data for 2006 are given both for the individual maps (Individual) as well as for the total number of individuals supporting consensus maps (Groups).

Figure 2.

Student concept maps for the microscopy module illustrating common patterns. Pattern types observed for CMAP 2 (microscopy) include (A) all microscopes treated equally (no hierarchical distinctions), (B) distinction between classes of microscope made on the basis of diffraction, and (C) distinction between classes of microscope made by the wavelength of radiation. An instructor CMAP is supplied as an example (D).

A second example that illustrates problems in the effective use of the mapping strategy can be found in the module treating energy conversion in the mitochondria and chloroplasts (CMAP6). There are many similarities between the two organelles, and thus this map was expected to be particularly appropriate for the formation of horizontal links. However, the majority of the maps simply portrayed the two organelles as two separate hierarchies with no connecting links (Table 2). Examples of the patterns found include those with no links (Figure 3A), as well as those with horizontal links formed either on the basis of the organelle's structure (Figure 3B) or by how energy is converted (Figure 3C). An instructor CMAP is again presented as example (Figure 3D).

Figure 3.

Student concept maps for the energy metabolism module illustrating common patterns. Pattern types observed for CMAP6 (energy metabolism in mitochondria and chloroplasts) include (A) mitochondria and chloroplasts treated separately (no horizontal links), (B) horizontal links between mitochondria and chloroplasts made by structural similarities, and (C) horizontal links between mitochondria and chloroplasts made through the form of energy used. An instructor CMAP is supplied as an example (D).

For the repetition of the experiment in 2006, students were shown examples of CMAP2 without appreciable hierarchy, and of CMAP6 without horizontal links present, and requested to pay special attention to these aspects when constructing their own maps. Given this prompting, it was somewhat surprising to recover so many individual maps without the desired characteristics, although there was improvement over the 2004 responses, particularly for CMAP6 (Table 2). One interpretation of this poor performance is that the students were unaware to what extent their individual maps corresponded to an inadequate conceptualization. This view is supported by the observation that when students were asked to decide on a consensus map as a team, the number of individuals represented by the team's choices shows further increase in the number of adequate maps (Table 2). This result suggests that peer-level feedback is an effective method to ensure a higher quality of maps. We conclude that feedback is an important component to successful implantation of the CMAP strategy.

To determine the impact of CMAP construction, the performance of the students on the final exam was also evaluated as a potential indicator of a positive impact. For the exam taken as a whole, there was only a slight improvement during 2006 compared with previous years, as is found for the final exam of a related course taken concurrently by the same students (Figure 4A, white bars). However, when the multiple-choice questions and problems are considered separately, a marked difference becomes apparent. Student success with multiple-choice questions remains generally stable between 2003 and 2006 (Figure 4A, gray bars). However, the success rate with problems increases dramatically during 2006 (Figure 4A, black bars). Because five of the six problems were the same between 2004 and 2006, the increase is unlikely to have resulted from a selection of easier problems. It is also unlikely to result from increased training during the semester, as the basic course outline remained unchanged. It is thus tempting to hypothesize that the construction of more adequate CMAPs resulted in increased problem-solving capability. To test the significance of this increase, the effect size was calculated for 2006 in comparison to both 2003 and 2004 (Horton et al., 1993). The effect size is calculated from the difference between the experimental and the control mean divided by the SD. No difference between the experimental and the control would return a value of zero, while a difference equal to the SD would return a value of one. For comparison purposes, the reported effect size attributable to CMAP was 0.42 ± 0.18 (n = 14) for science in general and 0.66 ± 0.25 (n = 9) for biology in particular. The effect size calculated for the problems during 2006 as the experimental year was 0.72 and 0.73 using 2003 and 2004 as the control year, respectively.

Figure 4.

Assessment of student performance. (A) Grades for multiple-choice questions (gray bars) and problems (black bars) for both cell biology and for a control course (white bars) for the 3 yr of the project. (B) Course failure rates for BIO1153 and a control course for 3 yr (the number of students was 234 in 2003, 224 in 2004, and 206 in 2006). (C) A comparison of grades in the control course with those obtained in BIO1153.

The increase in problem-solving ability and the resulting increase in the final grade led to a significant decrease in the number of students failing the course (<50% final grade; Figure 4B). Indeed, in 2006 only a single student did not pass. Although the final grades include the points given for completion of the CMAPs, these points are unlikely to be the cause of increased final grades, as the same points were allotted for map construction in 2004. It thus seems more reasonable to attribute the decreased failure rate to the observed increase in exam performance resulting from increased problem-solving capability (Figure 4A).

To determine the relationship between CMAP construction and academic performance, a selection of CMAPs were evaluated and the results plotted as a function of the final exam grade. A general correlation was observed for both CMAP2 and CMAP6 (Figure 5A), which becomes even more pronounced when the scores from both CMAPs are added together (Figure 5C). This is presumably attributable to the fact that these two CMAPs require different skills, one for determining hierarchy and the other for identifying horizontal links. This view is supported by the observation that there is little correlation between the scores of the two CMAPs themselves (Figure 5B).

Figure 5.

Relationship between final exam results and CMAP scores. Thirty students were selected from the class to represent high (top 10), medium (around the mean), and low (bottom 10) final exam grades. (A) Concept maps for the microscopy (CMAP2, open circles, example of adequate hierarchy) and energy (CMAP6, closed circles, example of adequate horizontal links) for all students were scored and individual CMAP scores were plotted as a function of final exam grade. (B) CMAP scores plotted as a function of one another. (C) Scores from the two CMAPs were combined and plotted as a function of final exam grade.

To assess the impact of introducing the CMAP strategy on retention of basic cell biology concepts, a short multiple-choice test was given to second-year students during the first 15 min of the class (Table 3). The test given during 2004 serves as a control, as students taking the first-year course in 2003 were not exposed to the CMAP strategy. Students in 2004 used CMAPs inefficiently, as described above, and the standard test results were virtually identical to those from the previous year's students who did not use CMAPs. Students who took the first-year course in 2005 were not included in the analysis (this course was interrupted by a labor dispute at the university making interpretation of the second-year test results in 2006 difficult). However, when the test was administered to second-year students in 2007, where significant impact of the CMAP strategy was observed with problem solving during the first year, a significant increase in test scores was observed (Table 3).

Table 3.

CMAPs aid knowledge retention by second-year students

School year	Test scores Mean ± SD (n)	Effect size
2004	69 ± 7% (102)	—
2005	69 ± 7% (105)	0.0
2007	76 ± 12% (75)	1.0

A 10-question survey of basic concepts covered in the first-year cell biology course (BIO1153) was given to students beginning a second-year cell biology course (students in 2007 took BIO1153 in 2006).

One concern for the interpretation of the experiment is that the general ability of the students in one year might be quite different from that in other years. Thus, improvement in performance might reflect improvement in students entering the course rather than an effect of the intervention. Two observations argue against this possibility. First, 28 questions were selected from past years' general biology and biochemistry GREs. These exams provide sample questions used in previous years for study purposes and list not only the correct answer but the national success rate. The questions selected were directly related to the course material studied during the year. The class average on GRE multiple-choice questions was 61 ± 21%, almost identical with the national North American average of 62 ± 17% for the same questions. BIO1153 students can thus be considered as a representative North American class. Second, the grade for a comparison course shows no change in year 2006 (Figure 4A).

A second concern for the general applicability of CMAP strategies related to the general perception of the strategy by the students. For example, the strategy is likely to be used effectively if the students perceive the strategy as useful and if the amount of work does not act as an impediment to its implementation. These aspects were tested by adding questions relating specifically to CMAPs to the standard course evaluation administered by the university. More than 90% of the students agree that the strategy is appropriate for keeping up to date and for making links between concepts, whereas more than 80% agree that the strategy prepares them for the exam and helps them understand the material (Table 4). Furthermore, more than 90% of students during the 2006 school year considered that the construction of individual maps and the validation of their maps in teams represented an adequate workload (Table 5).

Table 4.

CMAPs are generally well-perceived by students

Statement	Agree totally	Agree	Disagree	Disagree totally
The CMAP strategy helps in keeping up to date with the course	59%	31%	7%	2%
The CMAP strategy helps in making links between concepts	47%	47%	5%	4%
The CMAP strategy helps in preparing for the exam	41%	40%	12%	7%
The CMAP strategy helps in understanding the material	49%	35%	14%	2%

First-year students at the end of the 2006 school year (135 respondents) were asked the degree to which they agreed with a series of statements referring to the CMAP strategy in an anonymous student evaluation.

Table 5.

CMAPs are perceived to represent an adequate workload by students

School year	Number of respondents	Agree totally	Agree	Disagree	Disagree totally
2003	182	53%	38%	7%	0%
2004	166	47%	39%	11%	3%
2006	135	45%	47%	5%	3%

First-year students at the end of the indicated school year were asked the degree to which they agreed with the statement ″the workload of the course is adequate.″

DISCUSSION

One of the most important messages provided by these results is that effective use of concept mapping as a learning strategy must include a mechanism to ensure feedback to the students about the validity of their interpretations. In particular, the lack of any improvement in student exam performance in 2004 may well have been due to this lack of feedback on individual maps. This possibility is supported by the observation that student CMAPs in 2004 were generally characterized by poor organization and poor integration of the material into a coherent whole. For example, CMAP2 generally lacked proper hierarchical organization, while CMAP6 generally lacked horizontal links. These elements are actually more important than the concepts themselves, as indicated by their contribution to the scoring scheme developed for assigning numerical evaluations to different maps (Novak and Gowin, 1984). Furthermore, when the level of hierarchization and horizontal linking generally increased for the class of 2006 (Table 2), student performance on final exam problems did display a noticeable improvement (Figure 4).

The principal factor that was modified between 2004 and 2006 is the requirement for students to discuss their CMAPs with a team and to select (or make) a consensus map. This requirement imposes peer evaluations in the context of small teams to identify inadequate representations. Actually, as the students were requested to form their own teams, teams in this context should most probably be considered as “groups,” rather than teams whose composition is determined on the basis of student abilities (Michaelsen et al., 2004). Although the formation of true teams whose members have complementary skills might well prove to be more effective, it is difficult to see how team members could be assigned given that instructor access to student records is blocked.

The team evaluation of individual CMAPs was introduced to allow for feedback in large classes where instructor time is limited and individual maps cannot be corrected and returned to the students. By way of reference, in this study correcting the selected CMAPs used to derive the data in Figure 5 required ∼30 min each. Clearly, this is not a cost- effective method of student evaluation unless a suitable rubric could be designed and incorporated into an automated system. It must be noted, however, that even when required to produce individual and group CMAPs, the students themselves do not perceive this to be an unduly taxing workload (Table 5). Thus, given the learning gains indicated from increases in final grades (Figure 4) and the increased retention of information by second-year students (Table 3), the feedback-adapted CMAP strategy does appear to be an effective tool for student learning.

In addition to the critical value of peer evaluation, a second important message of this study is that the methods for evaluating the impact of an intervention and those used for assessing student performance must be coherent with the desired learning outcomes. For example, multiple-choice GRE-style exam questions might not be the most appropriate for evaluating the impact of a particular learning strategy. As shown here, the students do no better on multiple-choice questions after applying a CMAP learning strategy than they did in previous years without the strategy (Table 2). However, it cannot be concluded that implementing the strategy had no impact on student learning without a means of ruling out the possibility that the multiple-choice exam format assesses recall of memorized information rather than learning. The basic recall of information is low on Bloom's taxonomy, whereas the analysis of experimental data tested in the problems is a higher-level objective. The concept mapping strategy itself asks students to organize and synthesize, one of the highest levels (Anderson and Krathwohl, 2001). It is interesting to note that whereas performance on multiple-choice exams remains virtually constant, there was a marked increase in problem-solving ability. Problem solving may be a more adequate indicator of learning because the level of skill it assesses is closer to that helped by the CMAPs.

It is also possible that the increased performance in the test given to second-year students in 2007 (Table 3) results from increased retention of information from their CMAP constructions during the previous year. It would be interesting to evaluate transfer of knowledge from one course to another to determine whether this might have improved significantly. Although not directly assessed in the present experiment, it might be possible to envisage cross-course question exchanges. For example, a cell biology exam question could be given in the context of an exam in another course, after reformatting the question to correspond to material given in the other course. This would require collaboration between several instructors but could prove an interesting experiment in its own right.