Does Repetition Matter? Analysis of Biology Majors’ Ability to Comprehend Journal Articles Across a Major

Abstract

The ability to read and critically analyze the primary literature is a core skill necessary for future success in scientific fields. While many studies have described methodologies to teach journal reading, no studies examine how much practice and repetition is required before students learn how to comprehend a journal article. Here we assessed student journal reading and comprehension throughout an undergraduate biology major, analyzing students in six upper-level elective courses, some of which had no journal reading requirements while others had extensive requirements built into the course. We hypothesized that there would be a strong correlation between number of articles read in a semester and student ability to comprehend the articles, as well as their comfort and confidence with journal reading. Surprisingly, we found that the number of articles required for a class did not affect overall student reading comprehension and critical thinking even though students self-assessed that they gained comfort and confidence with articles as the number increased. Instead, we found that sophomore students in their first upper-level biology course showed significant gains in learning when the course activities include journal article readings. After this initial gain, there were no significant learning gains in future years, no matter the number of journals required in the course. Together, the results shown here indicate that it is not necessary to revise an entire curriculum to improve students’ journal reading and critical thinking skills. Instead, early intervention and exposure to critical journal article reading is most important for this skill development.

INTRODUCTION

The publication of Vision and Change has led to curriculum revisions throughout the country and an increased awareness of not just the concepts that we are teaching students, but the skills our students need to be successful in the field. These skills include: applying the process of science, quantitative reasoning, modeling and simulation, understanding the interdisciplinary nature of science, communicating and collaborating with others, and understanding the relationship between science and society (1). One way in which students can increase their acumen in all of these areas concurrently is through consistent reading of primary literature in order to learn about new discoveries in the field, and then assimilating those findings into their own research. This includes critical evaluation of reported findings, analysis of their own and reported data, drawing conclusions, and determining next logical steps, all while being mindful of ethical implications. While reading journal articles cannot ever be a blanket replacement for hands-on learning in a laboratory setting, it is important for faculty to remember that these analytical skills can be learned through many mechanisms, including journal reading (2).

Several years ago, in response to Vision and Change recommendations, our department revised the biology curriculum to emphasize these core concepts and skills, and one aspect of this revision was the incorporation of writing assignments into every course in the major (3). While the introductory courses utilize a learning-to-write methodology, upper-level electives emphasize a writing-to-learn mechanism in which students read and understand the primary literature in order to develop novel hypotheses that they can describe and/or test in their upper-level courses (4–7). Writing-to-learn is another method by which students hone their core skills, including applying the process of science, modeling and simulation, communicating and collaborating, and dissemination. Therefore, it is imperative that students learn to read and analyze primary literature effectively in order to be successful with these writing assignments. Not only must they understand the article, but they must also be able to critically evaluate the data, identify the gaps in knowledge, and determine the next steps to take, demonstrating their own ability to apply the scientific process in writing, if not in the laboratory setting itself. However, the amount of additional journal article reading done outside of the writing assignments is not standardized and varies throughout our upper-level electives. Because of this, understanding when and how students learn to comprehend journal articles is imperative if we are to help them succeed in a writing-to-learn environment.

There have been many articles describing methods to teach undergraduates to read journals as well as the development of scientific literacy courses focused on teaching students how to read the literature (2, 8–15). However, it is still unknown how many articles are “enough” for students to hone these skills and be able to analyze the data and draw conclusions without help from their faculty. While we have previously shown significant learning gains between first-year and upper-level students in terms of their ability to read and comprehend journal articles, it was not clear at what point the upper-level students gained the skills they lacked as first-year students (16). Moreover, it is unknown whether those gains were due to the methods used in a specific course, repetition of reading, or some undetermined factor. Therefore, the goal of this study was to determine whether student gains in confidence in reading primary literature are matched by the ability to critically analyze the literature at a high level through increasing the volume and diversifying the type of article reading experiences. We hypothesized that the ability to critically analyze a paper for key results and propose next steps would correlate with the required number of journal articles assigned for class discussions. The results of this analysis are shown here.

METHODS

Participants and in-class assignment

Students in six upper-level biology electives, taught by the authors, were asked to read a journal article and complete a short questionnaire regarding the paper during the first and last week of the term (16). Because of the nature of our major, upper-level electives are open to students of all levels who have completed the three-course core curriculum. Therefore, the same class may include sophomores taking their first upper-level elective with seniors taking their fifth and final upper-level. The number of sophomores (in their first upper-level course), juniors (students who had previously completed one or two upper-level courses), and seniors (students in their fourth or fifth upper-level course) who participated in each course can be found in Table 1. The chosen courses spanned the three major categories of biology—cellular/molecular (Immunology and Advanced Cell Physiology), organismal (Microbiology and Advanced Human Anatomy), and population/ecosystem (Evolution of Vertebrates and Microbial Ecology). Furthermore, students in each course were required to read and discuss a specific number of faculty-chosen journal articles (ranging from 0 to 10 depending on the course) throughout the semester as part of the course requirements, but outside of the writing assignment (Table 1). Courses were paired in order to mix the number of articles required, as well as the category of the course, as part of the assessment (Table 1). Specifically, an attempt was made to pair courses that used a large number of articles (e.g., Advanced Cell Physiology) with those that assigned few articles for in-class reading/discussion (Microbiology). Additionally, paired classes were in different biological categories (e.g., a cellular course paired with an organismal course rather than two cellular courses paired together) in order to avoid confounding data of similar topics being discussed during lecture in paired courses that could impact understanding and comprehension of the assigned journal articles during the post-test.

TABLE 1.

Summary of courses and articles utilized for the assignment.

Course Name, Number, Level	Sample Size	Number of Sophomores	Number of Juniors	Number of Seniors	Number of articles read/discussed throughout course	Article students in course read for assessment
Advanced Cell Physiology (BIO443; Cellular/Molecular)a	17	6	4	7	10	Martínez-García et al., 2014 (17)
Advanced Human Anatomy (BIO430; Organismal)b	13	4	2	7	9	Shore-Maggio et al., 2015 (18)
Evolution of Vertebrates (BIO355; Population/Ecosystem)c	19	9	5	5	8	Hutter et al., 2009 (19)
Immunology (BIO341; Cellular/Molecular)c	19	4	8	7	4	Daeschler et al., 2006 (20)
Microbial Ecology (BIO451; Population/Ecosystem)b	7	2	3	2	3	Shimokawa et al., 1998 (21)
General Microbiology (BIO321; Organismal)a	17	7	4	6	0	Ridley and Hall, 1992 (22)

^aThese courses exchanged journal articles for the assessment.

^bThese courses exchanged journal articles for the assessment.

^cThese courses exchanged journal articles for the assessment.

Students were given thirty minutes during class to read the assigned article and answer questions asking them to identify the hypothesis, key results, and conclusions as previously described (16). Students were also asked to rank their comfort and/or confidence with various aspects of journal articles as well as any struggles they had reading the papers (16). The articles were chosen by the instructor for their significance to the field, brevity (when possible), and likelihood that students had little previous exposure to the concept. In each case, paired courses would exchange articles so that students would read an article unrelated to their current course of study. For example, Advanced Cell Physiology students read a Microbiology paper while Microbiology students read a paper relating to Advanced Cell Physiology. The chosen articles for each course are listed in Table 1. It is important to note that of the 92 students who took part in the analysis, only 8 students were enrolled in two of the participating courses; the other 84 students were unique to one course. No students were enrolled in three or more participating courses. Furthermore, no students were in both courses that exchanged papers.

Assessment methods

Students’ responses to questions regarding the article content were scored as previously described (16). Comprehension was blindly assessed using a rubric by the faculty member/author from the opposite course who had chosen the article. For example, Microbiology students read a cell biology paper (22) and all assessments were scored by the cell biologist alone to ensure there was no bias in interpretation. While blind scoring by one individual can lead to potential bias, we believed that it was important that the expert in the field score the appropriate assignments rather than correcting for misunderstandings or miscommunication of field-specific jargon by faculty who are not knowledgeable of the field and findings. Initial assessments compared pre-test and post-test scores of all students within a course (Figs. 1, 4, and 7). Upon completion of all scoring, students were then grouped by their class standing at the time of participation (sophomore, junior, and senior) to assess differences in pre-test and post-test scores in courses with a large number of required articles to read and discuss in class (Advanced Cell Physiology, Advanced Human Anatomy, and Evolution of Vertebrates), a small number of required articles (Immunology, Microbiology, and Microbial Ecology), and all courses combined (Figs. 2–3, 5–6, and 8–9). These larger samples sizes allowed the authors to avoid type II statistical errors. All statistical analyses were conducted using SIGMASTAT 12. A paired t-test was used to determine significance for comprehension and comfort/confidence data unless there was not a normal distribution, in which case a Wilcoxon Signed Rank test was performed. Chi-square analysis was used to determine significance for data about student struggles with reading the articles, using the pre-test value as the expected value.

FIGURE 1.

Assessment of student understanding of a scientific paper. Upper-level student responses to three questions pertaining to the article read as part of the assessment in A) Advanced Cell Physiology, B) Advanced Human Anatomy, C) Evolution of Vertebrates, D) Immunology, E) Microbial Ecology, and F) Microbiology. Questions asked included: 1) identify the hypothesis, 2) identify the key findings, and 3) identify the contribution(s) of the paper to the field. Open bars indicate pre-test results while solid bars indicate post-test results. All scores were averaged, with error bars indicating standard error about the mean, and statistical analysis was performed using a paired t-test or Wilcoxon Signed Rank test when there was not a normal distribution. Statistical significance indicated on the graph.

FIGURE 4.

Self-rating of student comfort/confidence when reading journal articles. Students ranked their confidence/comfort with reading scientific papers, understanding graphs/tables/figures, determining a hypothesis, understanding the methods used, and identifying potential problems/pitfalls in the work. All scores were averaged and statistical analysis was performed using a paired t-test or Wilcoxon Signed Rank test when there was not a normal distribution. Statistical significance indicated on the graph. A) Advanced Cell Physiology, B) Advanced Human Anatomy, C) Evolution of Vertebrates, D) Immunology, E) Microbial Ecology, and F) Microbiology.

FIGURE 7.

Self-assessment of student struggles when reading journal articles. After reading and answering questions about the article, students were asked to indicate whether they struggled in any way while reading the article with: understanding the language (words), understanding the visuals (graphs and tables), comprehension of the topic in general, or other. Bar graph indicates the percentage of students who indicated they struggled in any area. Statistical analysis was performed using a Chi-square, with the pre-test scores performing as the expected value. Statistical significance indicated on the graph. Open bars indicate pre-test results while solid bars indicate post-test results in A) Advanced Cell Physiology, B) Advanced Human Anatomy, C) Evolution of Vertebrates, D) Immunology, E) Microbial Ecology, and F) Microbiology.

FIGURE 2.

Assessment of student understanding of a scientific paper based on class rank and number of articles read. A–B) sophomore, C–D) junior, and E–F) senior students within each course were grouped based on class rank at the time of the assessment and whether they were enrolled in a class with a large number of articles required (8 to 10 articles; A, C, E) or a small number of required articles (0 to 4 articles; B, D, F). Pre- and post-test scores were combined and averaged with error bars indicating standard error about the mean. Open bars indicate pre-test results while solid bars indicate post-test results. Statistical analysis was performed using a paired t-test or Wilcoxon Signed Rank test when there was not a normal distribution. Statistical significance indicated on the graph.

FIGURE 3.

Assessment of student understanding of a scientific paper based on class rank. All A) sophomore, B) junior, and C) senior pre- and post-test scores, irrelevant of course, were combined and averaged, with error bars indicating standard error about the mean. Open bars indicate pre-test results while solid bars indicate post-test results. Statistical analysis was performed using a paired t-test or Wilcoxon Signed Rank test when there was not a normal distribution. Statistical significance indicated on the graph.

FIGURE 5.

Assessment of student comfort/confidence when reading journal articles based on class rank and number of articles read. A–B) sophomore, C–D) junior, and E–F) senior students within each course were grouped based on class rank at the time of the assessment and whether they were enrolled in a class with a large number of articles required (8 to 10 articles; A, C, E) or a small number of required articles (0 to 4 articles; B, D, F). Pre- and post-test scores were combined and averaged with error bars indicating standard error about the mean. Open bars indicate pre-test results while solid bars indicate post-test results. Statistical analysis was performed using a paired t-test or Wilcoxon Signed Rank test when there was not a normal distribution. Statistical significance indicated on the graph.

FIGURE 6.

Assessment of student comfort/confidence when reading journal articles based on class rank. All A) sophomore, B) junior, and C) senior pre- and post-test scores, irrelevant of course, were combined and averaged, with error bars indicating standard error about the mean. Open bars indicate pre-test results while solid bars indicate post-test results. Statistical analysis was performed using a paired t-test or Wilcoxon Signed Rank test when there was not a normal distribution. Statistical significance indicated on the graph.

FIGURE 8.

Assessment of student struggles when reading journal articles based on class rank and number of articles read. A–B) sophomore, C–D) junior, and E–F) senior students within each course were grouped based on class rank at the time of the assessment and whether they were enrolled in a class with a large number of articles required (8 to 10 articles; A, C, E) or a small number of required articles (0–4 articles; B, D, F). Pre- and post-test scores were combined and averaged, with error bars indicating standard error about the mean. Open bars indicate pre-test results while solid bars indicate post-test results. Statistical analysis was performed using a Chi-square, with the pre-test scores performing as the expected value. Statistical significance indicated on the graph.

FIGURE 9.

Assessment of student struggles when reading journal articles based on class rank. All A) sophomore, B) junior, and C) senior pre- and post-test scores, irrelevant of course, were combined and averaged, with error bars indicating standard error about the mean. Open bars indicate pre-test results while solid bars indicate post-test results. Statistical analysis was performed using a Chi-square, with the pre-test scores performing as the expected value. Statistical significance indicated on the graph.

Informed consent and institutional review board protocols

Students signed an informed consent form prior to participating in this exercise and all articles and surveys were anonymous. Approval to conduct this study was granted by the Elmhurst College Institutional Review Board, which determined that the protocol fulfilled the necessary requirements for human subject research.

RESULTS

Students in upper-level biology courses show modest gains in journal article comprehension independent of the number of articles required to read in the course

In order to determine whether the number of articles read in a semester, outside of the writing assignment, correlated with improved student comprehension and critical analysis of journal articles, a three-question survey was given to all students in six upper-level electives at the beginning and end of the semester (16; Table 1 and Fig. 1). All surveys were scored against a rubric and assigned a numerical result, as previously described (16). The results of this analysis are shown in Figure 1. While Advanced Cell Physiology, with the most articles read of any course, showed learning gains in both students’ ability to determine key results and total overall score (Fig. 1A), Microbiology, with 0 articles read, also showed significant learning gains (Fig. 1F). Surprisingly, no other course showed significant learning gains in any area, no matter how many articles were read throughout the semester.

Class standing combined with number of journal articles read is the most important factor to observe learning gains

Because of the combination of sophomores, juniors, and seniors in upper-level courses, it was possible, if not probable, that the surprising results shown in Figure 1 were due to seniors who had been reading journal articles for years confounding the data so that few, if any, significant gains were observed. In order to eliminate that possibility, as well as sample size issues in small classes leading to potential statistical errors, all sophomore, junior, or senior pre- and post-test scores in courses that required a large number of articles to be read (Advanced Cell Physiology, 10; Advanced Human Anatomy, 9; and Evolution of Vertebrates, 8) were combined. The same was done for students in courses with minimal required reading (Immunology, 4; Microbial Ecology, 3; Microbiology, 0). The results in Figure 2 clearly show significant learning gains only in sophomores in courses with a large amount of required reading (Fig. 2A). No other class rank showed significant gains. Finally, when pre-test or post-test scores for sophomores, juniors, and seniors from all courses were combined, it was clear that sophomore students showed significant learning gains in ability to identify the hypothesis, determine key results, and total scores (Fig. 3). There were no significant findings among any other student group.

Student self-assessment of journal article reading skills

Students were asked to rank on a scale of 1 to 5 (with 5 being high) their comfort and confidence in reading papers and understanding/identifying key aspects of the articles such as the hypothesis, figures, results, and conclusions, at the beginning and end of the semester (Figs. 4–6). As expected, students who were required to read and discuss more journal articles in their courses tended to feel more comfortable and confident in their skills at the end of the semester compared with the beginning of the semester (Figs. 4 and 5). Furthermore, the courses with the least amount of journal reading (Figs. 4E, 4F, and 5B, 5D, 5F) did not show any significant increases in learning gains and, in some cases, comfort and confidence trended down at the end of the semester, although these findings were not significant. Interestingly, when all classes were combined, students saw significant increases in their comfort/confidence in different areas, no matter their class rank, irrelevant of the number of articles read (Fig. 6).

Finally, students in all six upper-level courses were asked to select any areas in which they struggled when reading the journal articles (Figs. 7–9). Surprisingly, post-test analysis indicated that students claimed to have fewer struggles in their ability to understand the language, visuals, or overall comprehension of the paper compared with pre-test scores regardless of the class they were in, their class rank, or the number of articles read.

DISCUSSION

The ability to read and comprehend a journal article is imperative for students to be able to apply the process of science, identify gaps in knowledge, and develop their own novel hypotheses. Therefore, it is necessary for faculty to understand when and how these skills are developed: is there a “right” number of articles that must be read or a specific method that must be used before we observe learning gains? Our results indicated that students’ comfort and confidence increase significantly as the number of required articles increases, but actual learning comprehension gains were only seen early in the academic career (sophomores) in courses with a large amount of required journal article reading (Figs. 1–6).

When students were asked to read and identify the hypothesis, key findings, and significance to the field of a provided article at the beginning and end of the semester, our results show that, when comparing students of all class ranks at once, student comprehension of journal articles is not based on a specific number of articles required to be read and/or discussed in a semester. While it seemed as though the hypothesis was supported based on the data provided by Advanced Cell Physiology (10 required articles, Fig. 1A), the same results were not consistent across other courses requiring a large number of articles to be discussed in class (Advanced Human Anatomy – 9 articles – Fig. 1B and Evolution of Vertebrates – 8 articles – Fig. 1C). Moreover, the fact that Microbiology, a course that did not require students to read any articles for class discussion throughout the semester, showed gains in the ability to determine key results (Fig. 1F) indicates that students may gain these skills in a variety of ways, not only through required journal readings supplemented with in-class discussion.

Instead, what our study clearly shows is that early intervention with students is key to gaining critical thinking and comprehension skills necessary for a successful scientific career. When students were grouped based on class rank (sophomore, junior, or senior standing), a surprising trend appeared. Significant growth was primarily seen among sophomore students and no other group (Figs. 2 and 3). Specifically, courses like Advanced Cell Physiology, which included a large number of sophomores in combination with a large number of required readings, saw significant learning gains. However, in Advanced Human Anatomy, no learning gains were seen in any group, likely due to a small sample size of sophomores and the fact that the seniors had already learned these skills earlier in their careers and did not gain anything additional from the large number of required readings. Taken together, these data imply that students are most likely to show learning gains in journal reading and critical thinking skills in their first upper-level class and that the number of required readings in a course is only important to sophomores and not older students.

When our departmental curriculum was revised in 2010, we introduced writing assignments into every upper-level elective course (3). Specifically, students in molecular/cellular courses write a grant proposal based on a relevant journal article a student chooses as “preliminary data.” In organismal courses, students write an annotated bibliography requiring critical analysis of approximately 20 journal articles in the field to support a novel hypothesis. Finally, students in population/ecosystem courses write a scientific paper describing the novel experiments they designed and carried out throughout the course. The ability to critically analyze and interpret the literature is a necessary skill to be successful in all of these assignments. Therefore, whether a class has a guided journal reading/discussion assignment as part of the course or not, students are required to comprehend journal articles in order to be successful with the major project associated with any upper-level course. Because of this, we can assume that in order for students to successfully complete an upper-level course, they have had to develop skills in journal reading and analysis. Indeed, we observed this phenomenon with the unexpected gain in determining key results in Microbiology, even though there were no additional journal article readings/discussions required in the course (Fig. 1F). In this case, it appears that the primary writing assignment for the course, an annotated bibliography, may have had an effect on the learning gains. Because of the large number of sophomores in Microbiology, we suspect that the process of students reading and analyzing the articles for the 20 required annotations on their own led to the learning gains observed. Conversely, Advanced Human Anatomy also required students to write an annotated bibliography analyzing 20 sources and had 9 required readings in class. However, our data suggest that because that class was primarily made up of seniors, learning gains were not observed (Fig. 1B). Furthermore, these findings imply that context for the journal article reading experience and the pedagogical method employed to teach journal reading may not matter when measuring learning gains. In this study, all courses and instructors used the same reading method, based on CREATE (11), to discuss articles, but only some of these courses showed learning gains. Taken together, these data indicate that learning gains can occur in any course that has a high number of early career students who are required to read journal articles in a variety of formats, including discussion or embedded writing assignments.

Student perceptions of their comfort/confidence followed our expectations, with more articles read corresponding to more confidence and fewer struggles. However, as we have previously shown (16), student perception of their ability to read journal articles does not necessarily correlate with ability. The same is true in this study, where students in Advanced Human Anatomy reported significant increases in confidence (Fig. 4B) while their actual knowledge and understanding of the paper was not improved and, in fact, did not reach a 50% average (Fig. 1B). Conversely, students in Microbiology admitted that they struggled with articles (Fig. 7F) and did not have significant gains in comfort/confidence (Fig. 4F), as would be expected since no journals were required to be read/discussed in the course. However, that group of students did show a significant gain in determining key results (Fig. 1F). It is important to note that while this was an ungraded assignment in the course, even when students showed learning gains, they still are not reaching what would be considered a “passing” grade (70%) on the assignment. Indeed, students in Advanced Cell Physiology showed the highest post-test scores and still did not reach this mark. While we would hope to see higher post-test scores in all situations, it is important to remember that students only had 30 minutes to read the article and answer questions, which likely impacted scores. Indeed, senior faculty often spend more time than that to critically evaluate new articles in their field so it is unlikely that an undergraduate, even one with a lot of experience reading articles as a senior, will earn perfect scores on an assignment requiring high-level learning competencies. Our results correlate with what is previously known about students’ inherent lack of metacognition (23). Students in all courses, no matter their class rank or number of articles read, stated that they had fewer struggles in the post-test compared with pre-test scores (Figs. 7–9), but their comprehension scores do not reflect this (Fig. 1). However, it should be noted that students in Advanced Cell Physiology correctly assessed their own ability, declaring that they are more confident and have fewer struggles (Fig. 4A and 7A) with journal articles at the end of the course, which coincides with improved scores (Fig. 1A).

Our results show that it is not necessary to completely revise a curriculum in order for students to learn these critical skills necessary for success at the next level. Indeed, it may not even be necessary to add required readings and formal class discussions to a course to achieve these learning outcomes. This is especially good news for departments that may not have “buy-in” from all faculty about potential curricular revision. Instead, the data are clear that one course, taken relatively early in the career of a biology major, that provides the opportunity for students to critically evaluate a large number of articles to hone their skills will be enough to achieve the desired learning outcomes. The data presented here show that once students obtain that skill set, they do not seem to lose it, no matter how many or which type of class they take in the future. Furthermore, because we know that ability to critically analyze papers is a necessary part of good scientific writing, the fact that students can learn this critical analysis skill early in their career means that there is additional time for writing development throughout the rest of the scientific undergraduate career. It also allows students to be better prepared for independent research activities in their junior and senior years, which is extremely valuable for their sense of belonging and retention in the scientific field as well as at their academic institution (24, 25). Therefore, it is imperative that we support and actively pursue students’ scientific literacy early to make it possible for them to reap the benefit of those abilities throughout their education and careers.