Discussion of Annotated Research Articles Results in Increases in Scientific Literacy within a Cell Biology Course

ABSTRACT

As the amount and complexity of scientific knowledge continues to grow, it is essential to educate scientifically literate citizens who can comprehend the process of science and the implications of technological advances. This is especially important when educating science, technology, engineering, and mathematics (STEM) college students, since they may play a central role in the future of scientific research and its communication. A central part of decoding and interpreting scientific information is the ability to analyze scientific research articles. For this reason, many different approaches for reading scientific research articles have been developed and published. Despite the availability of numerous ways of analyzing scientific research articles, biology students can face challenges that may prevent them from fully comprehending the text. We sought to address student challenges with science vocabulary and content knowledge by adding structural supports to in-classroom article discussions through the use of annotated articles from the Science in the Classroom initiative. We describe the pedagogical approach used for discussing scientific research articles within a required biology course. In this context, we found that students’ scientific literacy skills increased at the end of the semester. We also found that, for each article discussed, the majority of students could interpret graphical representations of article results and that they could identify and comprehend components of the experimental design of the study.

INTRODUCTION

In this digital age, individuals are bombarded with information on a daily basis. Due partly to the complexity of scientific topics, what is portrayed by the popular media does not always reflect the reality of technological innovations. This results in a disconnect between the real and perceived societal benefits of scientific research (1, 2). Additionally, popular resistance to science and technology is partly due to the lack of understanding of how science is done (1, 2). For these reasons, many have underscored the importance of developing citizens who (i) can interpret and critique scientific information disseminated by the media and (ii) comprehend the process of science, including how studies are designed (1, 3–6). This is particularly true for college students majoring in science, technology, engineering, and mathematics (STEM), since they may play a central role in the future of scientific research and its communication. Thus, if STEM instructors want to develop scientifically literate citizens, it is important to instill in students skills that include the interpretation of data and how the application of the scientific method reveals new answerable questions.

Some view reading and writing science as central activities associated with scientific literacy necessary for communicating, debating, and understanding science (7). In fact, the decoding and interpreting of scientific texts have been termed foundational literacy, because they are essential for the consumption of scientific information (8). Therefore, it is not surprising that the inclusion of the analysis of scientific research articles is a popular activity in STEM classrooms, with approaches freely available that lead to an increase in science process skills, the understanding of how science is done, and the ability to critically analyze information (9–18). However, previous studies have shown that biology students face challenges when analyzing scientific research articles, such as lack of content knowledge and science vocabulary, and that these issues may prevent them from fully comprehending the text (14, 16, 19). This barrier may prohibit students from using scientific research articles to gain a deeper understanding of how science is done and to develop scientific literacy skills. To address student issues with science vocabulary and content knowledge, we added structural supports to in-classroom article discussions through the use of annotated articles from Science in the Classroom (SitC; https://www.scienceintheclassroom.org/) (20). These annotated articles are significantly more readable than the original versions (21), and first- and second-year students perceived gains in the interpretation of graphical data as a result of reading them (20). We describe the pedagogical approach we used for discussing these scientific research articles in the laboratory component of a required biology course, and we report the effect of the activity on students’ scientific literacy skills. We present the successful implementation of this methodology using specific articles, but we believe this general approach can be adapted to many disciplines and educational levels due to the wealth of articles found in the SitC site.

Intended audience

Although this activity was done in a required, 3000-level biology majors course (Cell Biology), it can be adapted for use in 1000-level and upper-level science majors courses (see “Possible modifications” in the Discussion section).

Learning time

We discussed each article over two 1 h 50 min course sessions within the course laboratory component (Table 1). Prior to each article discussion, students completed graded assignments to promote engagement with the reading.

TABLE 1.

Overview of the timeline for article discussions

Wk	Topic	Article used
1	No lab
2	Article discussion dos and don'ts
3	Holiday, no lab
4	Preassessment (TOSLS); technique assignment due at start of lab
5	Preassessment not related to this study
6	Discussion 1 part 1	24
7	Discussion 1 part 2
8	Discussion 2 part 1	25
9	Discussion 2 part 2
10	Discussion 3 part 1	26
11	Discussion 3 part 2
12	Discussion 4 part 1	27
13	Discussion 4 part 2
14	Postassessment (TOSLS)
15	Postassessment not related to this study

Prerequisite student knowledge

Annotated articles from SitC contain a glossary and supporting materials on the methods and background for the study and are suitable for students with various levels of background knowledge. We suggest that instructors choose shorter articles with less challenging methods for discussion at the beginning of the term, while students become accustomed to analyzing scientific research articles.

Learning objectives

Learning objectives (LOs) for the course are related to aspects of scientific literacy skills as defined by Gormally and colleagues (22). We chose this definition because it centers on aspects of scientific literacy important in biology courses.

1. At the end of the semester, students will:

   1. Demonstrate an increase in scientific literacy skills.
2. After the article discussion sessions, students will:

   1. Identify and comprehend components of the experimental design of a study.
   2. Interpret graphical representations of data.

PROCEDURE

Materials

To participate in the article discussions, students need to access the SitC website, which is freely available with Internet connectivity. Students are also able to download a PDF of the article on the SitC website. We suggest that instructors encourage students to also read the original version of the article.

Student instructions

Prior to beginning article discussions, instructors gave a presentation on “Dos and don’ts” for analyzing scientific research articles (see Fig. S1 in the supplemental material), providing students with a brief overview of how to read articles, instructor expectations during discussions, and important information about navigating the SitC website.

To ensure that students had a theoretical understanding of the methods discussed in the articles, at the beginning of the semester we compiled a list of the techniques used in the studies and assigned groups of students a technique to research and summarize (see student instructions in the supplemental materials). Students turned in the assignment as a Word document through the learning management system (LMS) prior to the start of article discussions.

To ensure that students engaged with the material in the article prior to the discussion, we assigned students questions on the article, as described elsewhere (15). Briefly, the graded assignments consisted of questions at higher levels of Bloom’s taxonomy (23) and focused on either experimental design and result interpretation (first assignment) or critiquing and connecting results to the overall idea of the study and extending the findings of the study (second assignment). To stimulate conversation, we asked students to write three questions they had on the article as part of the first discussion assignment. Samples of prediscussion assignments are provided in the supplemental materials.

Faculty instructions

We discussed four scientific articles from the SitC website during the course of one 15-week semester. Articles were selected to align with lecture material (Table 1). During the course of the study, the entire lab component of the course was dedicated to article discussions, as this provided time to administer pre- and posttest assessments. However, discussion of three articles and omission of the assessments used in this study would allow for the inclusion of additional traditional lab sessions that may be required by course outcomes.

Prior to the first article discussion, instructors collected student submissions for the technique assignment, vetted and edited the information, and compiled it into one document which they placed in the LMS. This allowed students convenient access to information on techniques as they worked through the articles.

We dedicated two sessions to discussing each article, as detailed below. To ensure that all students contributed to the discussion, participation was graded out of a total of 5 to 7 points each period (5 to 7% of the total course grade over all article discussions). Given that high-stakes assessments may demotivate students to participate in article discussions (16), participation points were awarded for speaking and not for the quality of the contribution.

For the first discussion, at the start of class students self-selected into groups of 3 or 4. One group summarized the rationale, background, and hypothesis for the study, and the others were assigned a figure from the article to analyze. Groups worked on their task for 20 to 30 min before presenting the material. Instructors prompted students to interpret data, connect the results to the hypothesis and overall idea of the study, and discuss experimental design when necessary. For the second discussion, student groups first had 10 min to write two conclusions from the study. We opted to have students list two conclusions to lower the chance that groups would present identical conclusions. We discussed one conclusion from each group as a class. Then, groups were asked to provide strengths and weaknesses of the study and to design a follow-up experiment that would make an appropriate next step for the article discussed. Groups had ~30 min to complete these tasks before they presented their follow-up experiment. Instructors then asked groups to vote on the best follow-up experiment and provide a justification based on the following: Is this the true next step for the study? Is the experimental design well explained? Are appropriate controls included? Is the experiment feasible? Does the experiment test the hypothesis or research question? Were results described for both conditions, if the hypothesis is or is not supported? Groups were not allowed to vote for their own experiment.

In this study, the first discussion session for the first article had a different format: we discussed student answers to assignment questions and followed these with an informal conversation about the article. We did this for two reasons: (i) we wanted the first meeting to give students a general idea of how to read and interpret articles in a low-stress environment, since many of the students in this course had not read scientific articles prior to this class, and (ii) the first article (24) was very short, to ease students into the practice of reading scientific research articles. For this reason, there were not enough figures for student groups to each present one.

Suggestions for determining student learning

To determine the changes in students’ scientific literacy skills, participants completed the Test of Scientific Literacy Skills (TOSLS) questionnaire in weeks 4 and 14 of the semester (Table 1). We used this assessment because it centers on aspects of scientific literacy important in biology courses (22). Questionnaires were scored using the answer key provided previously (22); students received a score out of a total of 28 possible points. Since some of the students had high TOSLS pretest scores, in addition to comparing raw pre- and posttest scores, we grouped participants into quartiles based on their initial scores to determine learning gains while addressing the ceiling effects experienced by participants in the top quartile. To do this, we used the Microsoft Excel quartile function [=QUARTILE.INC(array,quart)] to sort the pretest TOSLS scores. This yielded four groups: TOSLS score of 12 to 18 (first quartile), 19 or 20 (second quartile), 21 to 23 (third quartile), and 24 to 27 (fourth quartile). Students who dropped or withdrew from the course were excluded from the analysis. Given that we were interested in examining students’ experimental design and data interpretation skills, we replicated Gormally and colleagues’ (22) principal-component analysis with a varimax rotation to see if some of the questions on the TOSLS could be extracted to measure these separate constructs. We did not identify two factors as hypothesized and confirmed the original study results (22). Thus, the TOSLS score encompassing all questions was used as a pretest-posttest measure of scientific literacy skills.

To assess students’ scientific literacy skills related to a specific article, we gave them a short quiz after the second discussion. Quizzes contained 3 to 4 multiple choice questions and one short-answer question. We analyzed scores for multiple-choice questions that assessed students’ ability to identify the experimental design of the study and their interpretation of graphical representations of data (see quiz questions in the supplemental materials). All statistical analyses were done using SPSS Statistics software (IBM).

Sample data

Sample student quizzes and technique assignments are provided within the supplemental materials.

Safety issues

There are no safety issues.

DISCUSSION

Field testing

The study was conducted at a five campus, 4-year Master’s university in the southeastern United States with a population of approximately 20,000 students during the spring semesters of 2018 and 2019 in a required Cell Biology course. The activities took place within the laboratory component, and sections were capped at 24 students. Approval for this study was granted by the University of North Georgia Institutional Review board (application 2017117).

A Likert-scale survey (see supplemental material) given at the completion of the course revealed that the majority of students felt that the article discussions helped them to better understand concepts in biology, the process of how science is done, and the importance of scientific research (Fig. 1). The majority of students also perceived that the discussions were educational and that the skills they learned would transfer to other classes (Fig. 1). Additionally, while only about half of students felt that the analysis of scientific research articles was enjoyable, the vast majority of students felt that discussions were a valuable use of their time (Fig. 1). Overall, students found the activity beneficial and educational.

FIG 1.

Student perceived gains. Students were surveyed at the completion of the course (N = 77). Bars represent the percentages of students that selected a particular choice. Survey prompts are shown to the left of the bars. Blue, strongly agree; green, agree; yellow, disagree; orange, strongly disagree.

The survey included an area for student comments. The authors M.S.-T. and M.W. coded comments separately and met to build a code list, organize codes into categories, and resolve disagreements. Double coding a selection was avoided whenever possible. M.S.-T. later placed comments related to facets of scientific literacy (as defined in reference 22) into a separate category. Student comments fell across four categories: (i) academic benefit (48.6% of total comments), (ii) procedural (31.9% of total comments), (iii) affective benefit (11.1% of total comments), and (iv) challenges (8.3% of total comments). Sample student quotes for each code are provided in Table 2.

TABLE 2.

Analysis of student comments on article discussions

Category (%) and code	No. of occurrences	% of total	Quote
Academic benefit (48.6)a
Critical thinking	9	25.7	Overall, the article readings, discussions, and critiques vastly developed my critical thinking skills and my critical reading skills.
Facets of scientific literacy	9	25.7	The sessions helped me think more in depth about the research such as how it could be improved and how to take their results and apply them to future experiments.
Learn how to read articles	6	17.1	It wasn’t the most enjoyable thing but it was helpful in understanding how to read research [articles].
Transfer of skills	6	17.1	I thought that it was good to learn how to read and analyze these articles for preparation for future classes.
Conceptual understanding	4	11.4	Having class discussions and breaking down the material in each research article allowed me to have a greater depth of understanding of the material.
Value of science	1	2.9	I've always thought scientific research is extremely important.
Procedural (31.9)
Discussions helpful	8	34.8	Being able to discuss scientific papers with peers was beneficial.
Class organization	5	21.7	The fact that we had to participate as a part of our grade forced some bad contributions to the discussions.
Preference of traditional lab or discussions	4b	17.4	Really enjoyed my time doing this type of lab as opposed to the traditional lab.
Article annotations helpful	4	17.4	The additional online paper with explanation and additional data and figures really helps me understand and visualize experiments.
Alignment with lecture	2	8.7	More discussion of how each article connected to or displayed the lecture material for that week would have helped connect lab and lecture.
Affective benefit (11.1)
Enjoyed discussions	6	75	I really enjoyed the articles and it was interesting to learn. The discussion sessions were challenging, but manageable.
Increased confidence	1	12.5	I really think this will help me in senior seminar. It makes me more confident about that class.
Increased curiosity	1	12.5	The best part of reading the article is how it sparked my curiosity on the subject.
Challenges (8.3)
Difficulty reading articles	5	83.3	Some of the articles were difficult to read and understand completely […]
Lack of knowledge of methods	1	16.7	This class required so much work outside of the lab to understand the methods of the article, a few simple walkthroughs and demonstrations of the methods would help.

^aCodes fell within four broad categories (shaded gray). Numbers in parentheses refer to the percentage for the category out of total comments. Categories and codes are listed in order of prevalence.

^bThree preferred discussions, one preferred traditional lab.

Within the Academic benefit category, students most often mentioned critical thinking and facets of scientific literacy as benefits of reading and discussing scientific research articles (25.7%) (Table 2). Students also thought discussions were helpful in teaching them how to read articles and in increasing conceptual understanding of the article topic (17.1% and 11.4%, respectively). Moreover, students felt that the skills learned would transfer to other courses (17.1%). Students also voiced having affective benefits from the discussions. Specifically, students found the discussions enjoyable (75%). One student commented that the activity increased their curiosity in a topic while another mentioned increased confidence in reading articles. Several student comments in this category focused on the “next experiment” portion of discussions. These comments mirrored the excitement that we observed in students as they worked together to design experiments to test open research questions stemming from the article results, including the following: “I enjoyed the ‘next step’ experiment discussions, that helped me apply critical thinking to the scientific article and i (sic) ended up understanding it better,” and “Coming up with next step was challenging and enjoyable.”

Most comments within the Procedural category referred to the discussions being helpful (34.8%) (Table 2), and 17.4% of students found the annotated versions of articles to be useful. Students also commented on how discussions were organized (21.7%). The first quote here shows a student who felt groups should always be given the same amount of time to complete the next-step experiment: “I feel like we were not always given the same amount of time to create an experiment in discussion 2 of every article. Sometimes we had more time than others.” While this is something worth considering, instructors determined the time needed to complete experiments by monitoring groups’ progress, and this led to the time discrepancy. “Sometimes i [sic] didn’t quite understand other people's figures because of the way they presented it. Students could use the [time] to practice the delivery so we all can understand what they're talking about.” We agree with the second quote: in our experience, students would benefit from practice presenting figures. We explore this further in the “Possible modifications” section.

Some students stated their preference for discussions or more traditional laboratory setups (17.4%) (Table 2) and commented on the alignment of article topic with lecture topics (8.7%). Several students mentioned that reading scientific research articles is difficult (83.3%), and one student specifically mentioned the lack of knowledge of methods as a challenge. This is in line with our previous findings of students’ perceptions of the analysis of scientific research articles (16, 19).

In summary, student comments showed that they felt article discussions were helpful, with ~60% of comments focused on the various academic and affective benefits associated with the activity.

Evidence of student learning

First, we assessed changes in students’ scientific literacy skills (LO A.1). We compared TOSLS scores and found that there was not a significant interaction between instructor and pretest-posttest differences (P = 0.27, F = 1.22). However, there was a significant increase in student scores at the end of the semester (P = 0.001, F = 11.5) (Fig. 2A). We also noted that some students had high TOSLS pretest scores, and so we wondered if the magnitude of the increase was caused by the ceiling effect experienced by a portion of students. To address this, we grouped participants into quartiles based on their pretest scores. Comparison of student learning gains through a 4 (quartile) by 2 (instructor) analysis of variance (ANOVA) detected an interaction between quartile and learning gain (P = 0.002, F = 5.51) and no significant interaction between quartile and instructor (P = 0.679, F = 0.507). Post hoc Tukey comparisons of mean learning gains by quartile revealed that students in the first (bottom) quartile had significantly higher learning gains than students in the other quartiles, while quartiles 2, 3, and 4 were not significantly different from each other (Fig. 2B). Overall, student scores on the TOSLS suggested that students’ scientific literacy skills increased during the semester, especially for individuals who started the term with lower skills.

FIG 2.

(A) Comparison of pre- and posttest student TOSLS scores. Students scored significantly higher on the TOSLS questionnaire at the end of the semester (P = 0.001, F = 11.537). Light gray, pretest scores; dark gray, posttest scores (N = 75). Error bars represent standard error of the means (SEM). (B) Comparison of TOSLS learning gains by quartiles. Students in the lowest quartile (stippled white bar; N = 21) had significantly higher learning gains on the TOSLS compared to students in other quartiles (second quartile [dark gray bar], N = 17; third quartile [light gray bar], N = 23; fourth quartile [white bar], N = 14.) ‡, P = 0.047; ≠, P = 0.005; *, P = 0.001. Error bars represent SEM.

We determined if students met learning objectives B.1 and B.2 in articles 2 to 4 by using multiple choice quiz questions that assessed either experimental design or data interpretation. Article 1 was not included in this data set because the discussion format differed from the approach we tested. For article 2, 83% and 95% of students correctly answered the experimental design and data interpretation questions, respectively (Table 3). On the article 3 quiz, 65% of students correctly answered the experimental design question, while 91% answered the data interpretation question correctly (Table 3). Finally, for article 4, 61% and 92% of students gave the correct answer in the experimental design and data interpretation questions, respectively (Table 3). We performed a 3 (quiz) by 2 (question category) by 2 (instructor) ANOVA on student quiz scores. Our analysis did not detect a significant interaction between question category score and instructor (P = 0.286, F = 1.16). However, we found that mean scores in the two question categories were significantly different, with students performing significantly higher in questions on data interpretation (P = 0.015, F = 6.30) (Fig. 3). One possible reason for this difference in performance is that students spent more time during discussions interpreting and presenting data compared to the time spent considering the experimental design of studies. Our results suggest that the article discussions met learning objectives B.1 and B.2, since the majority of students obtained the correct answer on quiz questions assessing data interpretation and experimental design. These findings confirm previously reported perceived student gains in the interpretation of graphical data when analyzing annotated articles (20). Overall, our data indicate that the pedagogical approach described for analyzing research articles met the established learning goals.

TABLE 3.

Student performance on quiz questions assessing learning objectives B.1 and B.2

LO	Article	% of students who answered question correctly
B.1	2a	83
B.2		95
B.1	3	65
B.2		91
B.1	4	61
B.2		92

^aFor article 2, 83% of students correctly answered each of two questions assessing LO B.1. All other categories were assessed through one question.

FIG 3.

Comparison of student performance in experimental design and data interpretation questions in quizzes 2 to 4. Mean student scores in data interpretation quiz questions (dark gray; N = 194) were significantly higher than mean scores in experimental design questions (light gray; P = 0.015, F = 6.30; N = 194). Error bars represent SEM.

Possible modifications

Different course schedules. Although for this study students discussed four articles over the span of eight class meetings, we have also discussed three articles in the same course context to allow for the inclusion of wet labs. Specifically, we discuss three articles over six sessions and devote eight lab periods to wet labs (we typically do not hold lab meetings during the first week of the course, but this time can alternatively be used to go over the dos and don’ts of article discussions). Instructors may instead choose to focus entirely on article discussions by incorporating up to a total of six articles in a 15-week semester. The shorter version of the activity is also suitable for courses taught in a 10-week quarter.

1000- and upper-level courses. We adapted this model for an introductory biology course for majors by using articles with less complex methods. Additionally, we discussed one article over the span of three lab meetings so we could provide more instruction and guidance to students. The first session focused on the motivation, hypothesis, and experimental setup of the study. The second session centered on data interpretation, while the third meeting focused on study conclusions and strengths and weaknesses of the experimental design. Students completed assignments prior to each discussion that prepared them for the specific focus of the session.

M.S.-T. adapted this activity for an upper-level, advanced cell biology course. Discussions and assignments were revised to account for deeper student knowledge of cell biology techniques and methods and more experience reading and analyzing articles. Table S1 summarizes the key differences between the approach described here and those used in the 1000- and upper-level courses.

Modifications from student feedback. Student feedback prompted us to consider dedicating the first course meeting to practice presentations. We recommend that students complete an assignment where they interpret one figure in an article ahead of the first course meeting. The instructor can form groups in class and give students time to plan their presentation. Student presentations should be followed by instructor and peer feedback.