Towards a Mastery Understanding of Critical Reading in Biology: The Use of Highlighting by Students to Assess Their Value Judgment of the Importance of Primary Literature

Abstract

An analysis of critical reading styles of freshmen and senior biology students was compared to that of biology faculty members through the use of highlighting a primary research article. Sentence-by-sentence comparisons were made within each group and the data were analyzed; the composite picture from each group was then compared to the other groups. There appears to be a close agreement of what is deemed important content as judged by faculty but less agreement by seniors and even less agreement by freshmen regarding the value of each line of the text. The results imply that experts in a field appear able to discriminate what is important and valuable in the primary literature and that the novice appears to develop some degree of scientific literacy during his or her undergraduate career.

INTRODUCTION

The oft-used saw of the scientific researcher that “those three months in the lab saved me from three hours in the library” is an important one to consider as we would be interested in our students developing into competent researchers: we want them to be able to recognize a biological problem; we want them to be able to design, develop, and perform wet lab experiments; we want them to be able to critically analyze their data; and we want them to be able to interpret the results from those experiments in light of the question being addressed and future areas for further research and discovery. Numerous scientific societies expound on the value of undergraduate research opportunities, such as the Council on Undergraduate Research (http://cur.org), the National Research Council, which is a part of the National Academies (http://www.nationalacademies.org), Sigma Xi, the Scientific Research Society (http://www.sigmaxi.org), the American Association for The Advancement of Science (http://www.aaas.org), and numerous discipline-specific biological societies and organizations such as Beta Beta Beta National Honor Society (www.tri-beta.org), American Institute of Biological Sciences (http://www.aibs.org), Federation of American Societies for Experimental Biology (http://www.faseb.org), the American Society for Biochemistry and Molecular Biology (http://www.asbmb.org), the American Society for Microbiology (http://www.asm.org), as well as a host of others.

The search or quest for knowledge must also include familiarizing one’s self with the known. We want the learner to recognize the importance of analyzing the current developments in the field. As our collective knowledge base grows it is imperative that individuals first and foremost perform a thorough review of the literature to allow one to prioritize one’s own research agenda. We perform this function of staying current in our field by reading the relevant primary literature. One of the values of reading the primary literature is that one can critically evaluate the data, compare findings, draw inferences, assess the strengths of conclusions, and rank the importance of each paper in a particular field. Reading a paper also allows the reader to measure his or her own understanding of a field. All of these skills are components of Bloom’s higher-level domains (7). Critical thinking about the primary literature requires critical reading in the discipline; therefore, scientific literacy is an important part of one’s development as a scientist.

Scientific literacy is defined as “the knowledge and understanding of scientific concepts and processes required for personal decision-making, participation in civic and cultural affairs, and economic productivity. It also includes specific types of abilities” (12, p. 96). Some of these abilities include “being able to read with understanding articles about science in the popular press” and “being able to evaluate the quality of scientific information on the basis of its source and the methods used to generate it. Scientific literacy also implies the capacity to pose and evaluate arguments based on evidence and to apply conclusions from such arguments appropriately” (12, p. 96). It has been argued by the National Research Council in their Bio2010 report (11, p.46) that “communicating how scientific discoveries are made is a crucial part of undergraduate education.” In sum the council notes that exposure to the experimental and conceptual basis of key discoveries gives students a deeper understanding of scientific principles. Students need to explore how discoveries are made. It is important for them to see scientific discovery as inspirational. Individuals acquire such ability through understanding and knowledge. The NRC goes on to state ways to communicate scientific discovery to students, including use of exemplars in science, textbook passages, classic papers, problem sets, and interdisciplinary lab exercises. Therefore, the acquisition of information through inquiry plays an important role in this process, and part of this inquiry is for an individual to understand how the scientific process works and how scientists gather information as they study the natural world, which requires reading scientific literature.

The practice of science is performed by scientists and reported to others through the use of primary, preferably peer-reviewed, research articles. The peer review system entails that published research has undergone critical analysis and interpretation by others who are familiar with the particular line of study and there is a validation of the factual integrity of the results as well as the authors’ interpretations of their findings. Reviewers must be knowledgeable in the subject area and able to be unbiased judges. By contrast the undergraduate learning environment typically entails the use of a textbook where science is described by way of facts, with little, if any reference to the original scientists’ discoveries or interpretations of their experiments. Most modern texts contain experiments in small side notes, or featured investigations, treating the scientist like a fish in a bowl with us, the reader and outsider, able to view what they did in a just-so manner. Scientific discovery, when offered in that method, appears to be a short and sweet process with very simple design and outcomes. Textbooks tend to decrease the scientific method that was employed by the scientist to a set of discrete factoids that can be used as fodder on the exam, with questions such as “Watson and Crick discovered the structure of DNA through the use of: a) X-ray crystallography; b) DNA gel electrophoresis; c) DNA fractionation; d) polymerase chain reaction; or e) S1 nuclease digestion.”

The nature of science is one of inquiry, where individuals investigate a problem by observing the available evidence, posing questions regarding the mechanism based on prior knowledge, performing experiments to test hypotheses, predicting consequences in novel situations, and acquiring new knowledge that is then made available to others typically through primary research articles or presentations at conferences. One means to illustrate the scientific method and the way that scientists perform research is to require undergraduate students to read primary research articles. It has often been assumed that this process would build their scientific acumen. This goal is typically used as part of a senior capstone course in the major (13, 3). Davis (3) describes the “ideal” senior capstone experience where five specific learning outcomes were addressed: a comprehensive exam; a 30-minute oral presentation with questions from peers and faculty; a 1–2 year research experience where the student mastered basic scientific research skills; a written thesis in the format of a scientific journal article; and a skills/techniques assessment test. One of the skills buried in the lab skills/techniques was ability to read and explain a selected piece of primary literature in which the student has some previous background. Obringer et al. (13) uses “hot topics” in biology as part of the way to dissect primary research articles for class discussion and also as the starting point for choosing papers that are presented by the students. Others have chosen to incorporate assignments involving primary research into 100-level classes (19).

Problems? Even in the journal Science there is a section called “This Week in Science” that contains a one-paragraph description of certain research articles in the issue. This is followed by “Perspectives,” which contains brief commentaries that summarize/highlight/dilute the research article findings to a level more understandable by scientists in other fields. The “Perspectives” articles paint the bigger picture or relevance and importance of the research or provide a model of the findings of the article. All of this points to a compelling conclusion, which is that proficiency in reading primary research is difficult at best and impossible for most when it is outside one’s area of research. If it is a challenge for individuals with advanced degrees what hope do individuals beginning in the field have of taking away anything of value from reading the primary literature?

Several individuals have written guidelines regarding how to read a scientific research article. Bogucka and Wood (2) produced a detailed set of sessions with active learning exercises embedded into the assignments. Some of their sessions required students to work in groups to read one, and only one, section of a paper and then report back to the members of the class that were assigned to read other sections of the same paper. It was reported that this exercise reinforced students’ critical evaluation skills and their understanding of the value of each part of a paper. As librarians they had noted that research regarding the methods used to enhance critical analysis skills was light, and the strategies recommended by others were varied in their recommendations to the student. They noted that some authors had proposed reading an article straight through, whereas some recommended reading the methods first, and others stated that the best strategy was to skim through the paper, writing a synopsis at the end of each section. Wenk’s and Tronsky’s advice for reading a paper (19) was to tell the student to read the abstract, followed by the intro, and then jump to the discussion—at which point the reader should attempt to determine what question this research addresses. Then the learner should read the methods and finally the results. At this point they stated that it may be worthwhile to re-read the discussion followed by the bibliography/references to try to sort out what work was performed earlier. Purdue University Physics Library contains an online Flash animation showing how to read a paper (http://www.lib.purdue.edu/phys/assets/SciPaperTutorial.swf). In it they state that one should read the paper in the following order: abstract, followed by discussion, then the introduction, and then the results.

The way most students attack a reading assignment is that they start at the beginning and then read straight through to the end of the paper. McMillan (9) provides guidelines to reading a scientific paper that involve following the information in the order presented. It is further suggested by this author that the student ask questions regarding the study as he or she goes along. This linear means of interpretation is one that individuals use for most reading; one never decides to jump into a middle chapter of a book and then come back to other parts later. Pechenik (15) describes this type of reading as “brain-off reading” when it comes to scientific papers. We generally do not decide to go to a movie half-way through and then sit around and watch the beginning afterwards. Society has created all sorts of devices to assist us to remain in a linear processing mode, whether individuals record materials for later viewing or new electronic means to allow one to start at the beginning of a television show or movie no matter when the presentation began.

Certain journals have changed the order of presentation of the parts of a research article, with the methods moved to the end and shorter intro and discussion. For many journals available in electronic form the methods are available only as supplementary information. McMillan (9) suggests spending little time on the methods (as, typically, it is challenging to understand all of the details) and pay particular attention to the discussion.

Parkinson and Adendorff (14) have used popular science texts as a means to enhance scientific literacy, their argument being that the content in textbooks is similar in its registry and technical language to those of research articles, in that in both the writer is removed from the writing and no emotion or subjective opinion is present. Although this may be true there are very clear differences between the content and writing style of texts and of primary literature. There is no glossary, definition of terms, or explanation of the materials in a primary research article. Also, by the time something makes it into a textbook it is typically well-vetted in the research community as an established fact. Authorship of thoughts and ideas are also typically absent in textbooks whereas in research articles great pains are taken to give appropriate reference to the source of the information. Research articles are designed to persuade one to accept the authors’ conclusions; texts are designed to persuade one to accept what is generally accepted knowledge in the field. In contrast to both of these is the popular science article, where the author can be personal and opinionated. Popular science articles make the science and the researcher accessible. As stated by Hyland (6) science journalism works as journalism, not science. There is also an intermediate genre between the primary research article and the popular press known as the adapted primary literature (APL) that has been constructed by Yarden et al. (20) and its educational value in different teacher models assessed (4).

The development of critical thinking skills in the learner is a goal of many instructors when presenting a student with a formidable challenge; measurement of this development is difficult. Stevens et al. (18) have developed a web-based computer interface called IMMEX (Interactive Multi-Media Exercises) that allows students to work through simulations while tracking their progress (http://www.immex.ucla.edu). The investigators state that through the recording of one’s approaches to problem-solving it should be possible to use the outcome in a formative manner. As each individual’s pathway is recorded it is possible to determine the commonalities of participants. In a similar manner this investigation allowed us to compare what each individual found as relevant and important.

Liu et al. (8), argue that students approach their decision-making based on their scientific epistemological views. They found that individuals who have a better understanding of the nature of science and science knowledge have a better ability to reason through scientific issues through multiple dimensions. A similar conclusion can be proposed from this study: experts were much more adept at evaluating the importance of the information provided in the text. Discriminatory behavior regarding choosing an appropriate paper is also a skill that must be developed by the student. Muench (10) teaches a capstone course where she models paper selection during the first half of the semester and then transfers that responsibility to the student through the course. Papers may be valuable for content or process; a particular paper may provide an excellent way of performing a particular technique with robust explanation of the methodology, but it may be lacking in significance. Conversely, a paper may contain a particularly important topic, but the methods may be impossible to understand and the data may be difficult for a novice to evaluate.

If one asks faculty members how they read a paper most would typically respond that they read the title and abstract first to determine if the paper is of interest and value, then jump to the results to consider the findings. Only then do individuals look at the methodology to determine if sound experimental techniques were used to generate the data. If it is directly in one’s area of interest the individual may scan the references to see if his or her own papers are present. If everything up to this point is intriguing then one typically reads the introduction and discussion in an attempt to find other papers related to this topic; however, related research articles are typically available electronically and the list generated by a search for related articles is typically larger than that associated with the paper itself and may even include newer articles that are not described in the text.

So what do undergraduates with little understanding of the vocabulary, the general field, and the area of research in particular, do when confronted with the task/assignment of reading a primary research article? How do they undergo the task of reading an article? More importantly, what do they feel is valuable information in the article and how do they perform diagnostic assessment of their level of understanding of the article? Do they perform the task in the same manner as an expert in the field? If not, what do they do differently and how can we mentor them towards a method or strategy that will allow them to effectively analyze the findings of someone’s research?

This research is worthwhile in several respects. First, it is useful with respect to formative assessment, namely what do first year biology students deem is significant biological information. The results of this study indicate that analysis of highlighting can be valuable as a diagnostic tool to determine what a student believes are the key findings of a researcher and its larger impact on our knowledge of biology. Second, the research presents a mechanism to use highlighting as a summative assessment tool, namely how do seniors change their view as well as their conceptual understanding of the field of biology as a result of completing an undergraduate study of biology. The seniors involved in this study were enrolled in a biology capstone course and this exercise provided one tangible measure in the shift of the competency of the student to discriminate meaningful information.

The student learning objectives of the Biology program include critical analysis of the primary research and the ability to make connections from coursework to develop a more comprehensive understanding of contemporary biology, and as such this technique provides one means to evaluate this objective.

This paper examines the pattern of reading and the priority placed upon the text in a primary research article by introductory biology students, graduating seniors, and biology faculty members through the use of highlighting. This hypothesis that there would be noticeable changes to the highlighting patterns thereby enabling differentiation between the novice and expert reader yielded two directional hypotheses. First, that there will be a high consistency with respect to highlighting patterns of the expert participants, but that this will not hold true for the novice participants (i.e., as expertise increases there will be an increase in the consistency in the pattern of highlighting). Second, as an individual becomes more proficient in a particular field of study there will be an increase in the discriminatory power of that individual with respect to discerning and highlighting important text (i.e., the amount of text that is highlighted should be negatively correlated with the level of expertise of the reader).

METHODS

In order to engage students in active learning and to increase their critical thinking skills, both first-year and senior-level students who participated in the study were required to read an article, highlight any information that they thought was important, and submit their work. The paper that was chosen for this study was by Sorek et al. (17) and addressed the finding that some genes are not passed on to other organisms via horizontal gene transfer as determined by bioinformatics techniques and when one attempts to move the genes experimentally noticeable barriers are detected. They proposed that certain gene products were toxic, and that expression of them may in fact be useful in the future as novel forms of antibiotics. This topic would most likely be an unfamiliar one to all of the students, and as it is primary research the findings would be new to the faculty as well. Prior to the commencement of the study IRB consent was obtained.

A sample of 110 individuals consisting of 85 freshmen students, 20 senior students, and five faculty were provided a copy of the article. The five faculty members consisted of two females and three males, all of whom were employed at small private universities. All faculty participating in this study held PhD degrees and were familiar with the fields of microbiology and bioinformatics covered in the article. They ranged in rank from assistant professor to professor and had between five and twenty years teaching experience at the college level. The only instruction provided to participants was that they were to read the article and highlight information that they believed to be important. The article consisted of 21 paragraphs of varying length and a total of 100 sentences. A coding system was developed whereby each sentence was first identified by paragraph and sentence location (for example, the first sentence of the first paragraph was coded as P1.1, and the third sentence of the first paragraph was identified as P1.3). Descriptive statistics were independently run to determine the minimum, maximum, range, mean, median, and mode for each group. The responses provided by the faculty were then isolated from the data set and re-examined independently. Based on our finding, 80% agreement existed on 75 of the 100 sentences with respect to whether or not a sentence was deemed important, and 54 of the 100 sentences yielded complete agreement among four of the five participating faculty. The data were scrubbed by removing the data obtained from the outlying faculty member, which had resulted in 46 sentences on which complete agreement was not obtained. This was done in order to establish a base line expertise against which to measure student alignment. The data were then recoded using a binary system which assigned a value of 0 to sentences that were not highlighted and a value of 1 to sentences that were highlighted. Responses provided by the freshmen and senior groups were then scrubbed and re-coded by removing sentences that did not coincide with those 54 sentences established as the baseline. Since the data were dichotomous nominal variables, Chi-square for Goodness-of-Fit tests were conducted to determine whether there were significant differences among the three groups (i.e., freshmen vs. senior, faculty vs. freshmen, and faculty vs. senior).

RESULTS

An examination of the hypothesis that as an individual becomes more proficient in a particular field of study there will be an increase in the discriminatory power of that individual with respect to discerning and highlighting important text (i.e., the amount of text that is highlighted should be negatively correlated with the level of expertise of the reader), yielded the following. The descriptive data for the five faculty showed a minimum number of highlighted sentences of 25 and a maximum of 39, yielding a range of 16. The average number of sentences highlighted was between 34 and 35 (X̄ = 34.4), with a median of 35 and a mode of 35 highlighted sentences (see Table 1).

TABLE 1.

Descriptive statistics for faculty.

Mean	34.4
Standard Error	2.481934729
Median	35
Mode	35
Standard Deviation	5.54977477
Sample Variance	30.8
Kurtosis	3.099384382
Skewness	−1.672584549
Range	14
Minimum	25
Maximum	39
Sum	172
Count	5

Further analysis of the data revealed that four of the five faculty highlighted between 30 and 36 sentences with an average of between 32 and 33 (X̄ = 32.75). The median score for the four faculty did not differ much from the mean (Median = 32.5). Therefore, when faculty member three was treated as an outlier and omitted from the data set there was complete faculty agreement on 11 highlighted sentences and 43 unhighlighted sentences for a total of 54% agreement on the 100 sentences and 75% or better agreement on 77 of the 100 sentences.

The descriptive data for the 20 senior students who participated in this study showed a minimum number of highlighted sentences of 27 and a maximum of 94 yielding a range of 67. The average number of sentences highlighted was 57, with a median of 56 and a mode of 60 highlighted sentences (see Table 2).

TABLE 2.

Descriptive statistics for senior students.

Mean	57
Standard Error	3.22327065
Median	56
Mode	60
Standard Deviation	14.41490457
Sample Variance	207.7894737
Kurtosis	1.73025003
Skewness	0.650851728
Range	67
Minimum	27
Maximum	94
Sum	1140
Count	20

Further analysis of these data revealed two outliers (in excess of two standard deviations from the mean) which were removed and the data re-analyzed yielding a minimum number of highlighted sentences of 41 and a maximum of 80 and a range of 39. For the remaining 18 students, the average number of sentences highlighted was between 56 and 57, with a median of 56 and a mode of 60 highlighted sentences (see Table 3).

TABLE 3.

Descriptive statistics for senior students with outliers removed.

Mean	56.61111111
Standard Error	2.340519591
Median	56
Mode	47
Standard Deviation	9.929983644
Sample Variance	98.60457516
Kurtosis	0.542162191
Skewness	0.786684127
Range	39
Minimum	41
Maximum	80
Sum	1019
Count	18

The descriptive data for the 85 freshmen students who participated in this study showed a minimum number of highlighted sentences of 6 and a maximum of 73 yielding a range of 67. The average number of sentences highlighted was 31, with a median of 29 and a mode of 33 highlighted sentences (see Table 4).

TABLE 4.

Descriptive statistics for freshmen students.

Mean	31.08235294
Standard Error	1.703284993
Median	29
Mode	33
Standard Deviation	15.70351171
Sample Variance	246.6002801
Kurtosis	0.019340352
Skewness	0.722474015
Range	67
Minimum	6
Maximum	73
Sum	2642
Count	85

An examination of the hypotheses that there would be noticeable changes to the highlighting patterns thereby enabling differentiation between the novice and expert reader yielded the following. With respect to the first directional hypothesis that there would be a high consistency in the highlighting patterns of the expert participants, but that this would not hold true for the novice participants (i.e., that as expertise increases there will be an increase in the consistency in the pattern of highlighting) the null hypothesis was rejected.

Faculty had 80% agreement or better on 75 of the 100 sentences whereas freshmen had 80% agreement or better on only 26 of the 100 sentences and seniors on only 24 of the 100 sentences. Faculty reached total agreement on 54 of the 100 sentences whereas neither freshmen nor seniors reached total agreement on any of the 100 sentences. These 54 sentences were then examined by group to determine internal consistency. It was found that senior data yielded high internal consistency (n = 20, Crohnbach’s Alpha = 0.903, p < 0.05), as did freshmen data (n = 85, Crohnbach’s Alpha = 0.874, p < 0.05). Of the 54 sentences agreed upon by faculty, 11 were deemed important and highlighted whereas the remaining 43 were left unmarked. A comparison of these 11 highlighted sentences among the three groups, as seen in Figure 1, shows that the senior class more closely resembled the highlighting pattern of the faculty than did the freshmen class on all but one sentence.

FIGURE 1.

Comparative responses by group.

Note: P3.1 refers to paragraph three, sentence one; P5.1 refers to paragraph five, sentence one in Sorek et al. (18). The same nomenclature is used for the rest of the data points.

The 43 sentences that were deemed unimportant and, therefore, not highlighted by faculty were then examined. A comparison of these 43 un-highlighted sentences among the three groups, as seen in Figures 2 and 3, show that the freshmen class more closely resembled the un-highlighted pattern of the faculty than did the senior class in 27 of the 43 sentences.

FIGURE 2.

Comparative nonresponses by group, paragraphs 4–8.

Note: P4.1 refers to paragraph four, sentence one; P4.3 refers to paragraph four sentence three in Sorek et al. (18). The same nomenclature is used for the rest of the data points.

FIGURE 3.

Comparative nonresponses by group, paragraphs 9–20.

Note: P9.1 refers to paragraph nine, sentence one; P10.1 refers to paragraph ten sentence one in Sorek et al. (18). The same nomenclature is used for the rest of the data points.

The multidimensional Chi-square test was conducted to determine whether there was a significant association among the three groups with respect to the 43 sentences that the faculty deemed unimportant. More specifically, the question posited was whether the group to which an individual belonged (i.e., faculty, freshmen, senior) was independent of his or her determination of whether or not a sentence was important. Using the group as the explanatory variable and whether or not the sentence was highlighted as the response variable three 2X2 contingency tables were constructed (faculty vs. freshmen, faculty vs. seniors, seniors vs. freshmen). For the purpose of analysis, the Pearson Chi-square was used except in cases where cells contained expected frequencies of less than five, in which case the Fisher’s Exact test was used (Siegel and Castellan, p. 103–111 (16)). With respect to the first contingency table (faculty vs. freshmen, n = 89), the null hypothesis was retained for all but one of the 43 sentences, suggesting that there is not a significant association (p > 0.05) with respect to the decision of whether or not a sentence was important and the group to which an individual belongs. The results from the second contingency table (faculty vs. seniors, n = 24) suggest a significant association (p < 0.05) between the group and whether or not a sentence was deemed unimportant for only four of the 43 sentences. The null hypothesis was retained for the remaining 27 sentences. An analysis of the third contingency table (seniors vs. freshmen, n = 105), suggested a significant association (p < 0.05) between the group and whether or not a sentence was deemed unimportant for only six of the 43 sentences. The null hypothesis was retained for the remaining 37 sentences.

DISCUSSION

The data seem to suggest that the difference between novice and expert opinion is lessened from freshmen to senior year. It is interesting to note that the internal consistency of responses within student groups was very high, which seems to indicate that there is a general approach to reading primary literature. With respect to faculty highlighting, the outlier may be explained by the individual’s familiarity or lack thereof in the field of bioinformatics. In the case of four of the five participants, there seems to be a consistent level of understanding, with the exception of participant three, whose data were treated as outliers for some of the analyses. It should be noted however, that although this individual’s data were treated as outliers, they still clustered closely with the other faculty, as inclusion would have extended the range by only nine highlighted sentences.

The significant differences between what faculty deemed to be important and what freshmen students, on the whole, deemed to be important may be explained by the greater range of highlighted sentences found in the freshmen data than in the data of either faculty or senior students. This seems to indicate a greater uncertainty about what in a passage would be considered important. Of the three groups, the faculty was most consistent in their highlighting, followed by the senior class, and lastly the freshmen. This study was interesting in that it examined not only whether there is a difference in the amount of highlighting done by each group but also it examined what specifically was thought to be important and conversely not important by each group.

So what exactly was highlighted by the faculty? Sentence 3.1 identifies one of the limitations of our understanding of gene exchange and hence sets up the reason for this study; sentence 5.1 describes how this research sets out to explore what they have termed “unclonable,” which they predict to be untransferable regions of chromosomes. Sentence 9.3 describes their findings regarding the movement of orthologs is gene-specific, some are transferred easily, whereas others cannot be transferred to another host. Sentences 13.1 and 13.2 describe the findings presented in Figure 2 in the paper, namely, that a similar pattern of transferability was noted among closely related organisms. Sentence 12.3 refers to Figure 3 in the paper in that certain expressed gene products were toxic in another host, and hence the gene would in essence be untransferable. Copy number of a gene could also influence level of expression (sentence 18.6, referring to data in Figure 2 in Sorek et al. (17)) and they found that many of the untransferable genes were single copy in their host. Conversely, sentence 13.5 is a prediction by them that certain genes would be nontoxic if their promoters were not active in E. coli. They then went on to test this hypothesis and the results are described in sentence 14.6. In conclusion they posit that genes that were deemed untransferable did not frequently use homologous recombination to effectively become regulated in the new host and hence be non-toxic (sentence 20.4). They also described one of the limitations of their study: they used small pieces of DNA, less than 1.5 kb, to attempt transfer and, hence, larger regions would not be detected via their bioinformatic screen (sentence 21.3). In sum, faculty highlighted their hypotheses, the major points of their results, and the authors’ conclusions and limitations of the study.

What was not deemed important by faculty? Sentences 4.1 and 4.3 are general descriptions of transformation, transduction, and conjugation and horizontal gene transfer. Faculty would be aware of these terms and hence not deem them important, but it can be noted that many students did find them important. It may be that students find definitions important, as it is a habit of many years of vocabulary exams in the K–12 world. Sentences 5.2, 5.4, 6.2, and 6.4 defined the sizes of their working group and dataset. Sentences 7.2, 8.2, 8.4, and 8.6 were descriptors in figure legends. Sentence 9.1 was the following statement: “We used the clone coverage distribution to identify genes unclonable into the E. coli host.” This statement of fact was most likely considered common practice by the faculty but appeared to be a novel concept to many of the students, as evidenced by the large number that highlighted these sentences. Sentences from paragraph 10 are values derived from the authors’ Figure 2. It is interesting to note that faculty did not deem these values important. Paragraph 11 describes the justification of consideration of copy number. Sentence 14.2 describes how they place “unclonable genes” under different regulatory control. Paragraphs 15 and 16 were highlighted by almost no one; these were the legends for Figures 3 and 4. Paragraph 18 was the authors’ discussion regarding the possible reasons for the lack of transfer of certain regions of a chromosome. Sentences 20.2 and 20.3 discussed the fact that the authors used conjugation of plasmids so that homologous recombination did not play a role in lack of toxicity due to the construction of a chimeric gene. This may have been information unfamiliar to students and hence the reason it was highlighted more frequently by them than by the faculty.

Baram-Tsabari and Yarden (1) argue that many misconstrue the definition of scientific literacy, in that we should want future citizens to not only possess scientific knowledge, but also to know how science is done; they see reading the primary literature as the way to meet that goal. Yarden et al. (20) have used primary research articles in the secondary school curriculum. They felt it would develop a number of skills in the students, including appreciation of the language and structure of a scientist’s research findings, an understanding of the rationale of a research plan and of the ways scientists solve problems, and the recognition of the continuity of science. However, it is a stretch for most students to go from reading popular science articles where the reading level is appropriate, the jargon of the field is stripped away, and the author tries to familiarize one with the heart of the paper by converting the data to “big picture” ideas. Hoskins et al. (5) created a program called “C.R.E.A.T.E” (Consider, Read, Elucidate, hypothesize, Analyze and interpret data, and Think of the next Experiment) in an attempt to humanize research and researchers. One of their approaches is to allow the students to contact the author through email to ask why the author addressed that biological phenomenon and the approaches that they took. They posit that this humanization helps to shift students’ attitudes about science, as evidenced in pre- and post-course scores on the Student Assessed Learning Gains (SALG can be found at http://www.salgsite.org). SALG is used by students to assess and report on their own learning, and on the degree to which specific aspects of a course have contributed to that learning.

CONCLUSION

Self-assessment is important, especially as one considers meta-cognitive development. However, the use of other measures, such as the method described in this paper, provides evidence regarding the students’ understanding as well some insight into their thoughts and feelings regarding the ranking of importance of the information presented in a paper. The next step in the analysis presented in this paper will be to ask students why they found particular passages important. This further analysis will allow for a better understanding regarding the value students place on various sections of a paper and perhaps will provide a means for mentoring and instructional intervention. Presenting to the students the value placed on various parts of the paper by experts (in this case, experienced faculty members) after they have read the paper will assist them in gaining a better understanding of how peer review by scientists occurs. Modeling of the experts’ approach to reading, apprising, and evaluating the primary literature is an area ripe for development and use in the classroom.