On the Prevention and Treatment of Exaggeration

At the 1995 American Heart Association meeting, researchers presented an abstract raising questions about the safety of short-acting calcium channel blockers for the treatment of hypertension. The abstract described a population-based case-control study that found that short-acting calcium channel blockers were associated with an increased risk of heart attack in patients with hypertension.¹ The presentation generated extensive media coverage. Unfortunately, the coverage was fraught with exaggeration: sensationalized headlines (e.g., “Drug for blood pressure linked to heart attacks; Researchers fear 6 million imperiled”), overstatements about the size of the effect (e.g., focus on a 60% relative increase in risk rather than what the principal investigator called a “miniscule” absolute risk increase from 1 in 1,000 to 1.6 in 1,000²) and overgeneralization (e.g., implicating all rather than short-acting calcium channel blockers).

In this issue of the Journal, Brunt et al. describe some of the consequences of the exaggerated media reporting.³ Understandably, the news stories generated anxiety and confusion, and led some patients to inappropriately stop antihypertensive medications. Brunt measured 1 aspect of these reactions—changes in the proportion of calcium channel blocker prescriptions filled, recorded in a national pharmaceutical database. While the magnitude of the effect reported by Brunt is small—a short-lived 10% decrease in calcium channel blocker prescriptions—the paper is an important contribution because it is one of only a handful of published studies⁴ actually documenting what many people believe to be true: the news media can affect health behaviors.

Also in this issue, Stamm et al. offer advice on how to improve researchers' interaction with the media.⁵ They provide simple, practical tips on drafting press releases, presenting data, and talking to reporters. Some of the advice—in particular, on the need to specify the base rate when presenting relative differences, and the need to clearly specify the study population—might have helped mitigate some of the problems Brunt describes. But no matter how closely researchers follow this advice, difficulties with the quality of media reporting on medical research will persist. In our minds, 2 fundamental problems stand out: the premature dissemination of research and the exaggeration of medical research findings.

The premature dissemination of medical research in the media, that is, prior to peer review, is an important problem.⁶ This coverage is frequent and often prominent. In our study of 5 major scientific meetings in 1998, roughly 30 abstracts at each meeting received major media coverage; one quarter of the news stories appeared on the newspapers' front page.⁷ Recently, we were surprised to see work from our own research group—a decision analytic model about the possible benefits of obesity surgery presented at a surgical meeting—referenced in a Sunday New York Times staff editorial (without mention of its preliminary nature).⁸ Because the research presented at scientific meetings is often work in progress, it is not surprising that many papers do not survive formal peer review. In our study of scientific meetings, we found that twenty-five percent of the abstracts receiving media coverage remained unpublished more than 3 years after the meeting.⁷ Other abstracts may reach publication, but only after changing substantially by the time they appear in a medical journal.

Exaggeration about medical research findings by the media is also an important problem. For example, a study of news stories about the benefits and harms of 3 popular medications found that reports were often inadequate or incomplete: less than two thirds of stories quantified treatment benefit, and only about half mentioned potential harms.⁹ Furthermore, when stories did quantify benefit, they did so by reporting a relative risk reduction without noting the underlying event rate, a format known to exaggerate the perceived effect of an intervention. Similar findings were reported in a study of the media coverage of mammography, tamoxifen, and breast cancer.¹⁰

Researchers presenting at scientific meetings can do a lot to prevent exaggeration; they can be careful to clearly and repeatedly note the preliminary nature of their work (when that is the case), the need to interpret results with caution and put them into context, and the importance of waiting for their work to undergo scientific peer review. Alternatively, researchers can ask themselves why their meeting presentations need to be promoted at all. Unless there is a compelling reason that the public needs to hear about the results immediately, researchers should consider refusing to participate in the promotion of their early work to the media (e.g., press releases from meeting organizers or their academic institutions), and wait until the time of final publication. Since exaggeration is a multifactorial problem, it also needs to be addressed at other levels. Media outlets could start by raising their threshold for reporting on meeting abstracts, and meeting organizers might reconsider their promotional efforts.

But let's be serious. Powerful and reinforcing self-interests make it unlikely that the foregoing recommendations will be followed. Media outlets are in intense competition with each other for a limited audience; meeting organizers need to attract scientists, advertisers, and sponsors; researchers need to show results to advance their careers; and academic institutions need publicity to raise funds. In each case, self-interests are served by being associated with work that is perceived to be new, big, and important.

For these reasons, we will need to focus on the treatment side of the exaggeration problem. We believe that generalist clinicians can play an important role in mitigating the harms of exaggeration by encouraging patients to develop a healthy skepticism for health news (particularly for stories about preliminary work). Although we recognize that adding to the ever-growing list of clinician responsibilities may be unwelcome, we think physicians can do much to inoculate their patients (as well as medical students and trainees) against exaggeration by teaching them, when opportunities arise, to ask the following questions:

Does this news apply to me? News is about hooking the general reader, but the hook may entail unwarranted extrapolation—from rats to humans, or from very select groups of patients to the entire population.

Is this news too good (or too bad) to be true? Dramatic effects make news, but dramatic effects may not fairly represent the truth. Sometimes the drama instead reflects a misleading use of numbers, invalid comparison groups, or a leap of faith from intermediate outcomes.

Has this news stood the test of time? What is new makes news, but what is new may not turn out to be correct. Research presented at scientific meetings is typically work in progress. Don't be surprised if early reports about breakthroughs fail to pan out, and expect a fair amount of change (and many things fading away) while researchers figure out what is really going on.

Exaggeration serves many interests, but it does not serve the public's interest. And in the end, it is self-defeating, because it undermines the credibility of medical science. After a while people may not believe anything we have to say.