How Do US Journalists Cover Treatments, Tests, Products, and Procedures? An Evaluation of 500 Stories

Abstract

HealthNewsReview.org evaluates US health news coverage of claims made about medical interventions. Gary Schwitzer reports on the project's findings after evaluation of 500 health news stories.

News stories about new treatments, tests, products, and procedures appear daily. Such reporting should ideally be accurate, balanced, and complete so that health care consumers are properly informed and ready to participate in decision making about their health care. If reporting is inaccurate, imbalanced, or incomplete, consumers may have unrealistic expectations and demand of their physicians care that would be of little value or even harmful.

Is the news media doing a good job of reporting on new treatments, tests, products, and procedures? Ray Moynihan and colleagues analyzed how often news stories quantified the costs, benefits, and harms of the interventions being discussed, and how often they reported potential conflicts of interest in story sources [1]. Of the 207 newspaper and television stories that they studied, 83 did not report the benefits of medications quantitatively, and of the 124 stories that did quantify the benefits of medications, only 18 presented both relative and absolute benefits. Of all the stories, 53% had no information about potential harms of the treatment, and 70% made no mention of treatment costs. Of 170 stories that cited an expert or a scientific study, 85 (50%) cited at least one with a financial tie to the manufacturer of the drug, a tie that was disclosed in only 33 of the 85 stories.

Moynihan and colleagues' work was one of the inspirations for the creation of the Australian Media Doctor Web site (http://www.mediadoctor.org.au/) in 2004. That project monitors the health news coverage of 13 Australian news organizations. The project concluded, after its first six months experience, that “Australian lay news reporting of medical advances…is poor” [2].

Summary Points.

• The daily delivery of news stories about new treatments, tests, products, and procedures may have a profound—and perhaps harmful—impact on health care consumers.

• A US Web site project, HealthNewsReview.org (http://HealthNewsReview.org/), modeled after similar efforts in Australia and Canada, evaluates and grades health news coverage, notifying journalists of their grades.

• After almost two years and 500 stories, the project has found that journalists usually fail to discuss costs, the quality of the evidence, the existence of alternative options, and the absolute magnitude of potential benefits and harms.

• Reporters and writers have been receptive to the feedback; editors and managers must be reached if change is to occur.

• Time (to research stories), space (in publications and broadcasts), and training of journalists can provide solutions to many of the journalistic shortcomings identified by the project.

In Canada, Alan Cassels and colleagues documented similar journalistic shortcomings [3]. Cassels heads a team that launched a Canadian Media Doctor Web site (http://www.mediadoctor.ca/) in 2005. That project evaluates health news coverage by 12 Canadian news organizations.

These efforts helped inspire a project to evaluate United States health news coverage of claims made about treatments, tests, products, and procedures. HealthNewsReview.org started publishing evaluations of health news stories in April 2006. The sole support for the project is a grant from the 501c3 nonprofit Foundation for Informed Medical Decision Making (http://www.fimdm.org/about.php), founded in 1989 by Dartmouth's Dr. Jack Wennberg and colleagues, with a mission of “assuring that people understand their choices and have the information they need to make sound decisions affecting their health and well being.”

This article reports on the project's findings after its first 22 months and after evaluation of 500 health news stories.

The HealthNewsReview.org Project

HealthNewsReview.org monitors news coverage by the top 50 most widely circulated newspapers in the US; the most widely used wire service, the Associated Press; and the three leading newsweekly magazines—TIME, Newsweek, and U.S. News & World Report. Each weekday we watch the morning and evening newscasts of the three most watched television networks—ABC, CBS, and NBC.

In order to be eligible for review, a story must include a claim of efficacy or safety in a health care product or procedure (drug, device, diagnostic or screening test, surgical procedure, dietary recommendation, vitamin, supplement).

The rating instrument used (http://www.healthnewsreview.org/ratings_info.php) includes ten criteria used by the Australian and Canadian Media Doctor sites. All of the criteria are addressed in the Association of Health Care Journalists' Statement of Principles [4]. As seen in the methods of evaluation listed in Box 1, each story is evaluated on whether it has met the ten criteria.

Box 1. HealthNewsReview.org Ratings Criteria and Explanation.

1. Adequately discusses costs.

We believe that, in an era when health care spending represents 16% of the US gross domestic product, a story is incomplete if it does not address the costs of an approach, and a comparison with existing alternatives. We also think journalists should explore whether insurers are likely to pay for it.

2. Quantifies benefits.

Stories should give some sense of the size of the potential benefits of the approach being discussed. Stories (and studies, for that matter) should also explain the benefits in absolute, not just relative, terms.

3. Adequately explains and quantifies potential harms.

Stories should give a complete picture of potential harms of an approach, and quantify those potential harms in absolute terms.

4. Compares the new idea with existing alternatives.

We expect that a story would put the new approach being discussed into the context of existing alternatives, with some discussion of the possible advantages or disadvantages of the new approach compared with existing alternatives.

5. Seeks out independent sources and discloses potential conflicts of interest.

We expect, just as the Association of Health Care Journalists does, that journalists should “recognize that most stories involve a degree of nuance and complexity that no single source could provide. To reflect only one perspective of only one source is not wise; [journalists should] be vigilant in selecting sources, asking about, weighing and disclosing relevant financial, advocacy, personal or other interests of those [they] interview as a routine part of story research and interviews”[4].

6. Avoids disease mongering.

This criterion is an attempt to help journalists avoid promulgating the medicalization of normal states of or variations in health (e.g., baldness, menstruation, short stature, etc.). We also try to educate journalists about surrogate endpoints and about how risk factors are not diseases. With this criterion, we also remind them not to exaggerate the prevalence or incidence of a condition.

7. Reviews the study methodology or the quality of the evidence.

The story should reflect an understanding that not all studies are equal. If a story does not point out some of the limitations of an observational study and does not caution about interpreting uncontrolled data, for example, we will judge it unsatisfactory.

8. Establishes the true novelty of the idea

Many “new” products or procedures are not really novel. The product reported may be the sixth new member of a well-established class of drugs. It may be a device that has only been judged to be substantially equivalent to other devices already on the market. Journalists should accurately reflect the novelty (or lack thereof) of “new” products or procedures.

9. Establishes the availability of the product or procedure.

Many stories report on products or procedures that are still in clinical trials. We expect journalists to explain whether something is only available via limited access in clinical trials, whether something is FDA approved, whether insurability limits availability, etc. Many news stories seem to treat FDA approval of an investigational drug as a fait accompli, making predictions about how the drug “could be” or “should be” approved and on the market within a given time frame. Such stories would be rated as unsatisfactory.

10. Appears not to rely solely or largely on a news release.

We expect, just as the Association of Health Care Journalists does, that journalists should “Preserve journalistic independence by avoiding the use of video news releases or the use of quotes from printed news releases; label and credit the source whenever a portion of a video or printed news release is used” [4]. We expect a journalist to use a news release for background information only, and to then seek independent experts to comment on a development.

For each criterion, the story is given a rating of “satisfactory,” “unsatisfactory,” or “not applicable.” Three reviewers—with backgrounds in medicine, health services research, public health, or journalism—analyze each article. (All reviewers are listed online at: http://www.healthnewsreview.org/people.php.) As the publisher of the project, I am always the third reviewer of each story, and I mediate any disagreements between the first two reviewers, gaining consensus before publishing the final evaluation of the health story.

Inter-reviewer reliability was tested using a random sample of 30 stories. Two reviewers coded each story. The average percent agreement between the two reviewers across the ten ratings criteria was 74%.

Results and Impact of the Project to Date

In our evaluation of 500 US health news stories over 22 months, between 62%–77% of stories failed to adequately address costs, harms, benefits, the quality of the evidence, and the existence of other options when covering health care products and procedures (Table 1). This high rate of inadequate reporting raises important questions about the quality of the information US consumers receive from the news media on these health news topics.

Table 1. Percentage of Satisfactory Reviews for 10 Criteria for 500 Stories.

At a time when US health care spending now represents 16% of the gross domestic product [5], only 23% of the health news stories that we analyzed covered the costs of medical treatments adequately. In an era when most news organizations reported on postmarketing problems found with rofecoxib (Vioxx), coronary stents, or hormone replacement therapy, many of those same news organizations still fail to adequately quantify the harms and benefits of the products they report on today. Only 28% of the stories we evaluated adequately covered benefits, and only 33% adequately covered harms.

Many newsrooms across the US have eliminated health and medical reporting positions because of a drop in profits [6]. Those cutbacks may be the reason why only 35% of stories we evaluated were rated satisfactory on the criterion of whether the reporter had discussed the study methodology and the quality of the evidence, issues that only a trained health journalist could be expected to understand.

Only 38% of stories were rated satisfactory for putting the intervention under discussion into the context of existing alternative options. The Statement of Principles of the Association of Health Care Journalists [4] urges journalists to disclose relevant conflicts of interests in their sources “as a routine part” of their work, and to “avoid single-source stories.” Our analysis shows that journalists are following this professional guideline only about half the time. Examples of problems found in stories scoring poorly are provided in Box 2. Of the first 500 stories reviewed, 41 (8%) received our highest scores. They appear online at http://www.healthnewsreview.org/fiveStars.php.

Box 2. Examples of Problems with Stories Scoring Poorly on HealthNewsReview.org.

1. Drug Doubles Endurance, Study Says. The New York Times. November 16, 2006.

Grade: 2 criteria satisfactory, 8 unsatisfactory.

(Story and review can be accessed at http://www.healthnewsreview.org/review/review.php?rid=657.) Excerpt of our summary: “[This story reports] in a manner that is not critical about the information being presented nor mindful about balance. There is an important difference between the results from a few research studies in animals and demonstration of efficacy in people. The story would have been strengthened by providing more sources with healthy skepticism to balance the overwhelming enthusiasm from other sources, several of whom had ties to the drug companies promoting the substance.”

2. Study Shows Proposed Merck Drug for Diabetes Lowers Blood Sugar. The Wall Street Journal. September 14, 2006.

Grade: 2 criteria satisfactory, 7 unsatisfactory, 1 not applicable.

(Story and review can be accessed at http://www.healthnewsreview.org/review/review.php?rid=550.) Excerpt of our summary: “Actual rates of low blood sugar episodes with the new drug were not reported, yet were noted to be a serious side effect of older drugs. The article also reports that there appear to be no serious side effects of the drug, which may be true for the moment, but caution should have been given to these findings, given that the only study was performed by the manufacturer and the study was short-term.”

3. Mystery Writer Recovers Voice Box with Botox. USA Today. January 14, 2007.

Grade: 2 criteria satisfactory, 7 unsatisfactory, 1 not applicable.

(Story and review can be accessed at http://www.healthnewsreview.org/review/review.php?rid=746.) Excerpt of our summary: “This story vividly describes what it must be like to have this difficult condition, but does little to provide consumers with important information, such as the evidence to support the use of Botox, how much the injections cost and any potential harms of the treatment.”

4. To Build a Knee. U.S. News & World Report. July 23, 2007.

Grade: 2 criteria satisfactory, 7 unsatisfactory, 1 not applicable.

(Story and review can be accessed at http://www.healthnewsreview.org/review/review.php?rid=976) Excerpt of our summary: “Unfortunately, in the absence of a balanced view of surgery's pros and cons and an independent voice (the sole source is an enthusiastic surgeon), the story feels more like a puff piece for a top-rated hospital than a balanced look at a medical problem.”

5. New Prostate Cancer Test Shows Greater Accuracy. ABC World News. April 26, 2007.

Grade: 2 criteria satisfactory, 7 unsatisfactory, 1 not applicable.

(Story and review can be accessed at http://www.healthnewsreview.org/review/review.php?rid=883.) Excerpt of our summary: “At least the story could have included one line that screening is controversial regardless of method chosen, because it isn't yet clear if treatment saves lives. It did not disclose what was abundantly clear even in a Johns Hopkins news release: the principal investigator receives a share of the royalties received on sales of the test. He is also a paid consultant to the manufacturer of the test. There were no quotes from anyone expressing skepticism about the development.”

6. Fighting Addiction with a Pill. CBS Evening News. November 29, 2007.

Grade: 1 criterion satisfactory, 9 unsatisfactory.

Story and review can be accessed at http://www.healthnewsreview.org/review/review.php?rid=1103.) Excerpt of our summary: “To illustrate the drug's effectiveness it draws heavily on a single anecdote, which tells the story of a patient who appears to have an unusually successful outcome. It does not cite published research or interview independent sources to verify the drug's safety and efficacy.”

7. Mini-Medical Miracles: Getting Rid of Wrinkles. NBC Today Show. February 1, 2008.

Grade: 0 criteria satisfactory, 9 unsatisfactory, 1 not applicable.

(Story and review can be accessed at http://www.healthnewsreview.org/review/review.php?rid=1156.) Excerpt of our summary: “This story was a classic case of disease-mongering. [It] was part of a four-part series on ‘mini-medical miracles.’ The anchor said: ‘This week we've checked out potential breakthroughs in treatments for baldness, insomnia and dandruff. Today we end with wrinkles. Could having a new laser treatment in your forties or fifties prevent you from ever needing a facelift?’ No one needs a facelift. It is not a matter of need. Similarly, baldness, insomnia and dandruff are not diseases that require treatment—much less miracles or breakthroughs.”

Next Steps

We are working to communicate these findings to news organizations and to editorial decision makers, and to help them see that they often deliver an imbalanced picture of health care interventions. We believe this imbalance may have a profound impact on the decision making of American consumers who rely on these news stories.

When we evaluate a health news story, we e-mail the evaluation to the journalist who wrote that story, and their responses have been overwhelmingly positive. “It's quite sobering to read the reviews,” wrote one journalist. “I imagine you've heard all the laments from reporters, but the lack of both space and research time is enormously frustrating (and will probably drive me out of journalism in the end).” Time and space can provide answers to many of the shortcomings we identified. Reporters who are given more time to research a topic and more space or time in their publications or broadcasts would be better able to address the criteria that we—and their own industry principles—state are important.

Some news organizations choose to offer “health headlines,” “medical minutes,” or “science briefs”—presumably to show the breadth of their coverage. But our analysis suggests that so much is left out in these short items that it is difficult to convey any of the nuance that is so important in almost all such stories. We take the position that if a news organization cannot give sufficient space or time to a story to cover the necessary issues, then it would be better if the story were not published or broadcast at all. Incomplete stories that lack context can cause harm. People may be misled, become anxious, or make ill-informed decisions based on such stories.

Training opportunities have been slashed in many newsrooms [7]. A survey of Midwestern newspapers showed that nearly all reporters covering health news had received no training in this subject matter or in interpreting health statistics; of those, most said that such training would be helpful [8]. Many different forms of health care journalism training opportunities are now available. The National Institutes of Health, the Centers for Disease Control and Prevention, the Knight Foundation, the Association of Health Care Journalists, and several universities now offer specialized training programs of varying lengths, degrees of complexity, and formats.

We observed what we considered to be incomplete coverage in some of the nation's top news organizations, and gave some of our top scores to stories in what might be called secondtier newspapers. These observations suggest that attaining excellence in health news stories is not just an issue of resources.

While our project is in no way a complete analysis of the work done over 22 months by the news organizations we examined, it nonetheless represents the largest dynamic database of content analysis of US health news coverage ever assembled, and it continues to grow daily. It provides the clearest picture yet available of how major newspapers, magazines, and television networks cover treatments, tests, products, and procedures. And this is the first project that gives journalists regular feedback on how they cover health news stories.

We think that our project's findings should be a cause for reflection by all parties involved in the dissemination of news and information to journalists covering health and medicine—including medical journals, government agencies, industry, academic medical centers, and individual clinicians and researchers. Some have described sensationalism in medical science stories as the byproduct of a complicit collaboration between journalists and scientists [9], wherein journalists find it easier to get attention for their stories and scientists see “the practical value of media attention to a successful scientific career” [9]. Rather than allowing the interests of the two professions to influence each another in such an unhealthy manner, we suggest that each should examine its core ethical principles.

One study, by Steve Woloshin and Lisa Schwartz, showed that news releases from medical journals “do not routinely highlight study limitations or the role of industry funding. Data are often presented using formats that may exaggerate the perceived importance of findings” [10]. Vikki Entwistle showed how “the way information flows from medical journals to newspapers influences the balance of medical topics reported, the quality of the research reported (and its appropriateness for public attention), and the quality of news reporting” [11].

We hope that our evaluation of health news will lead news organizations—and all who engage in the dissemination of health news and information—to reevaluate their practices to better serve a more informed health care consumer population.