Evaluating the Accuracy of Health News Publications in a Drug Literature Evaluation Course

Abstract

Objectives

To design an assignment for second-professional year pharmacy students to assess the accuracy and quality of health information published in the news.

Design

Students in a literature evaluation course were assigned a health-related news publication to review and find the original published research article. They then critically evaluated the quality and accuracy of the news publication based on the original research. All students wrote a critique focusing on the quality and accuracy of the news article and potential responses the lay public might have.

Assessment

Eighty-four percent of students agreed the writing assignment reinforced critical literature evaluation skills, while 90% agreed the assignment contributed to completion of course objectives.

Conclusions

A writing assignment requiring comparison of a news publication to the original research reinforces critical literature evaluation and communication skills, as well as stimulates thought about the accuracy, quality, and public responses to health information published in the news.

INTRODUCTION

Healthcare professionals and the lay public are flooded daily with media releases on new medical findings. This information has the potential to be miscommunicated in the media, due to the sheer volume of information and the speed at which this information is released. With an increasing number of patients taking a more active role in managing their health (eg, seeking health information), it is important that patients, consumers, and healthcare professionals are correctly informed about new medical information. Several publications have reported inaccurate and/or overenthusiastic results in media releases of medical findings.^1–3 For example, Moynihan and colleagues reviewed 180 newspaper articles and 27 television reports about 3 commonly used medications (pravastatin, alendronate, and aspirin) for accuracy.¹ Only 60% of the media releases reported the medications' benefits in a quantitative fashion, only 47% reported potential harm to patients, and only 30% mentioned the costs to patients. Financial ties with drug manufacturers were discussed in 39% of the reports.

Since healthcare professionals and the public often receive information about new medical findings simultaneously, these omissions are troubling.⁴ Given that pharmacists are the most accessible healthcare professionals, they are likely to encounter patient questions about the reported medical findings in the media. Therefore, it is essential that they are aware of medical news releases and their potential impact on the public.

Active-learning methods that relate course material to real-life situations can motivate students to learn.^5–6 To stimulate thought and heighten awareness of the presence, accuracy, and quality of health information published in the news media, an assignment was developed for second-professional year pharmacy students in a Drug Information and Literature Evaluation course at the University of Tennessee (UT) College of Pharmacy.

The goal of this assignment was to provide students with experience in qualifying the accuracy of a sample of news publications by comparing the content of a media release to the original research article. This assignment stressed the importance of literature retrieval and evaluation skills, critical analysis of medical research presented to the lay public, timely review of new literature, and formulation of a well-written analysis. Specific objectives for the assignment included: stimulating thought and heightening awareness of the presence, accuracy, and quality of health information published in the news to second-professional year pharmacy students.

Drug Information and Literature Evaluation is a 2-credit hour course that is taught at the UT College of Pharmacy in the first semester of the second-professional year. It focuses on traditional drug information, drug use policy, and literature evaluation. Previous course evaluations have shown that some students do not always understand the relationship between literature evaluation and improving patient outcomes. This is concerning since the drug information course is the primary means for students to meet the Accreditation Council for Pharmacy Education (ACPE) standards and the American Association of Colleges of Pharmacy (AACP) Center for the Advancement of Pharmaceutical Education (CAPE) outcomes related to drug information and literature evaluation.^7–8

DESIGN

Each student (N = 98) was assigned 1 of 6 recently published (ie, within 2 months of the assignment) health-related news articles from either USA Today or The Wall Street Journal that discussed an original research publication. Students were instructed to search for and retrieve the original research article using the secondary databases discussed in class (eg, MEDLINE, EMBASE) and the information given in the news publication. The titles of the news articles and the corresponding original research articles are listed in Table 1.^9–20

Table 1.

Titles of News Publications and Original Research Articles Used in a Drug Literature Evaluation Course

Students were to read and evaluate the original research article using the knowledge and skills gained throughout Drug Information and Literature Evaluation. After the students evaluated the original research article, they evaluated the quality and accuracy of the news publications' representation of the original research. Using a grading rubric provided by the instructors, students constructed a 3- to 5-page written critique of the news publication, discussing the quality, accuracy, and potential responses the lay public might have to the article. Table 2 provides specific tasks that students were to focus on when comparing the news publication to the original research. The main focus areas included the accuracy of the news publications, identifying important missing information in the publication (ie, completeness), identifying any bias, and brainstorming the types of questions that the public might ask a pharmacist after reading the article.

Table 2.

Tasks Required for Evaluating a News Article in a Drug Literature Evaluation Course

The completed assignments were divided among 3 full-time faculty members at UT who taught in the course. Critiques of the news articles were then graded using the grading rubric and the results from these rubrics were tallied. Prior to the announcement of the assignment, the instructors met to assign a maximum amount of points to the items in the grading rubric so that they totaled 100 points.

Students were also asked to participate in several additional survey assessments. Participation was voluntary and students were told that participation would not affect their grade for the course. Assessments were granted exemption from the University of Tennessee's Institutional Review Board (IRB). Preintervention and postintervention self-efficacy survey instruments specific to the assignment were administered before (ie, immediately after a description of the assignment was given) and after the completion of the assignment. The survey instruments evaluated student impressions of their ability and confidence to search for original research, evaluate the accuracy of news publications, identify limitations of news publications, and identify questions that the lay public might have. On the first survey instrument, students were asked to rank their ability using a 4-point Likert scale (not at all able, slightly able, moderately able, completely able) with a fifth option of “not sure” on the items mentioned above. On the second survey instrument, students were asked to rank their confidence on the same items using a 4-point Likert scale (not at all confident, slightly confident, moderately confident, completely confident) with a fifth option of “not sure.” A final survey instrument administered at the end of the course collected student impressions of the assignment. Specifically, students were asked to rank their level of agreement to the statements that this assignment helped reinforce their critical literature evaluation skills and contributed to their completion of the course learning objectives. A 4-point Likert scale was used (strongly disagree, disagree, agree, strongly agree) with a fifth option of “not sure.”

ASSESSMENT

Ninety-eight students completed the assignment. The majority of students met the objectives of the assignment and scored an average of 91 ± 8 (range: 67-100). The majority of students (67%) felt that the news publications were not biased. However, 76% felt that important data were excluded. Students' specific recommendations for inclusion of additional information are summarized in Table 3. Compared to the original research, 83% felt that the news publication made inaccurate conclusions, while 55% felt that conclusions were incomplete. Table 4 summarizes and groups the most common questions that the students thought the lay public might ask their pharmacist after reading the news publication. In general, the student comments and recommendations were in agreement with the instructors' assessments of the media articles, as demonstrated by an average score greater than 90% awarded by the instructors.

Table 3.

Student Recommendations for Inclusion of Additional Information in News Article Evaluated as Part of a Drug Literature Evaluation Course (N = 98)

Table 4.

Potential Questions Identified by Students That the Public Might Have After Reading Health-related News Articles Evaluated by the Students as Part of a Drug Literature Evaluation Course (n = 98)

Student self-rated ability and confidence appeared to improve in all areas following the assignment. Table 5 summarizes these findings. The sample size of students completing the preintervention and postintervention survey instruments differed, which likely skews the accuracy of these results. Specifically, 75 students completed the preintervention survey instrument and 61 students completed the postintervention survey instrument. The discrepancy in sample size between the total number of students completing the assignment (N = 98), the preintervention survey instrument (n = 75), and the postintervention survey instrument (n = 61) likely occurred because completion of the survey instruments was voluntary. Students who did not complete the preintervention survey instrument were asked not to participate in the post intervention survey. However, there was no mechanism to control for this and some students could not remember whether they had completed the preintervention survey instrument. For this reason, statistical analyses were not conducted, as there was no way to pair data. Theoretically, a few students may have compelted the postintervention survey instrument who did not complete the preintervention survey instrument.

Table 5.

Students' Responses on a Self-efficacy Survey Instrument Administered Before and After Completing an Assignment in a Drug Literature Evaluation Course

Regarding student impressions of the assignment, 84% of students strongly agreed or agreed that the assignment reinforced critical literature evaluation skills and 90% strongly agreed or agreed that it contributed to the completion of course objectives.

DISCUSSION

The main intent of this assignment was to expose students to medical news publications and the impact that they have on the lay public. Students could relate to this aim, as they had experienced similar scenarios as interns working in retail pharmacies or other job settings involving patient care. However, a majority admitted that they had no prior experience in evaluating news publications or original research and were unsure how to even find original research at the beginning of the course. Class discussion following completion of the assignment revealed that students were surprised at the inaccuracy or incompleteness of the news publications. Overall, they thought that it was a valid exercise to think ahead about what types of questions their patients might ask if they read the news publications. Additionally, they appreciated the practice of finding and evaluating the original research, so that they could confidently and correctly answer their patients' questions. These thoughts were reflected in the survey results (Table 5), which indicated that the students' ability increased in all areas by 11%-32% and the students' confidence increased in all areas by 4%-28% after completion of the assignment. Students also commented that they felt less intimidated by writing an evaluation of a news article first, before being graded on an evaluation of an original research article (ie, journal club).

Another intent of this assignment was to have students practice their professional writing skills. Students were expected to turn in organized reports, using correct grammar, professional language, with referencing using the Uniform Requirements for Manuscripts Submitted to Biomedical Journals.²¹ The strenuous nature of the PharmD curriculum and increasing class size make it difficult to incorporate formal writing assignments into courses. However, it is imperative that students are able to accurately and efficiently express their thoughts, especially in a course that asks them to critically evaluate medical literature. Some colleges of pharmacy have taken major initiatives to emphasize and improve writing skills throughout the curriculum. One example is the Writing Across the Curriculum program created by the Albany College of Pharmacy, where writing assignments have been incorporated into most courses and serve to help students learn course objectives.²²

The main limitation with this assignment relates to time. In the Drug Information and Literature Evaluation course, students are expected to read multiple original research articles throughout the semester. As they are inexperienced in reading and evaluating original research, this can be time consuming. To minimize the amount of reading they must do, the original research article used in this assignment was also used in a journal club that was presented 1 week later in the course. However, this made it essential that appropriate media articles were chosen. These media articles must have corresponding primary research articles that were well designed and used concepts that were discussed in class. Another limitation included choosing appropriate articles based on the students' knowledge base. Since the drug information course is taught in the first semester of the second-professional year and students do not begin therapeutics sequences until the spring semester of the second-professional year, original research about complex therapeutic models could not be chosen. It was difficult to find 6 appropriate original research articles and corresponding news publications that met the students' therapeutic knowledge level.

From an instructor perspective, these assignments were time consuming to grade. As our class size increases from 100 to 200 students, writing assignments, such as this, may place especially large constraints on involved faculty members. To facilitate the grading, a grading rubric was devised for the instructors to quickly and objectively determine a student's grade, as well as guide the students in writing their critique. All of the instructors met together to assign maximum points to each item in the grading rubric before the assignment was introduced in the course. The goal of the grading rubric was to reduce any intergrader discrepancies. In further efforts to minimize intergrader discrepancies, each instructor completed grading on all assignments pertaining to the same lay publication and original research articles. If any questions arose regarding grading, the instructors discussed how to address the issue as a group.

CONCLUSION

This writing assignment requiring comparison of a news publication to the original research on which it was based resulted in the reinforcement of critical literature evaluation skills and reinforcement of written communication skills. It also stimulated thought about health information reported by the media and potential public responses to this information. This assignment has now been incorporated into the Drug Information and Literature Evaluation course.