The Quantity and Quality of Scientific Graphs in Pharmaceutical Advertisements

Abstract

We characterized the quantity and quality of graphs in all pharmaceutical advertisements, in the 10 U.S. medical journals. Four hundred eighty-four unique advertisements (of 3,185 total advertisements) contained 836 glossy and 455 small-print pages. Forty-nine percent of glossy page area was nonscientific figures/images, 0.4% tables, and 1.6% scientific graphs (74 graphs in 64 advertisements). All 74 graphs were univariate displays, 4% were distributions, and 4% contained confidence intervals for summary measures. Extraneous decoration (66%) and redundancy (46%) were common. Fifty-eight percent of graphs presented an outcome relevant to the drug's indication. Numeric distortion, specifically prohibited by FDA regulations, occurred in 36% of graphs.

Pharmaceutical manufacturers are in the business of developing and selling new products. Pharmaceutical advertising directed toward health professionals could aspire to the high-level discourse typified by scientific publications or the low-information content, mass-media material designed to achieve name recognition and create “the psychic desire to consume.”¹ The best scientific manuscripts use data graphs to effectively and efficiently portray details and complex relationships. If pharmaceutical advertisements were attempting to convey scientific information to professionals, we would expect them to contain similar, high-quality graphs.

We undertook this descriptive study to characterize the quantity and quality of graphs in pharmaceutical advertisements. Our goals were to examine whether advertisement graphs fully exploited the power of graphs to convey information, and whether graphs misrepresented or distorted trends in the data.

METHODS

We performed a retrospective review of all pharmaceutical advertisements in the 1999 issues of 10 leading-circulation American journals (Table 1). This convenience sample of journals was chosen to encompass general medicine as well as a range of specialties. We chose journals on the basis of their circulation and of their reputation according to local experts in each field. We included 1 large-circulation, non-peer-reviewed journal that is widely distributed to training physicians to capture advertisements that may be distinct from those in other journals. We excluded advertisements for over-the-counter medications, medical devices, and diagnostic test equipment. For each journal issue, we counted the advertisements per issue and digitized all unique advertisements onto a CD-ROM. Our goals were to characterize the visual content of each advertisement and systematically evaluate every scientific graph. We did not perform a detailed evaluation of the FDA-mandated small-print pages.

Table 1.

Number of Issues and Quantity of Advertisements in the 1999 Journals Examined

Journal	1999 Issues, N	Total Ads, N	Ads/Issue, Median (Range)
American Journal of Psychiatry	12	180	15 (10 to 17)
Annals of Emergency Medicine	12	131	11 (7 to 14)
Annals of Internal Medicine	24	308	13 (8 to 17)
Annals of Surgery	12	22	2 (1 to 3)
Hospital Practice	13	356	28 (21 to 37)
Journal of the American Medical Association	48	398	8 (5 to 12)
Neurology	18	332	19 (10 to 26)
New England Journal of Medicine	52	1,017	18 (8 to 34)
Obstetrics and Gynecology	12	251	23 (9 to 28)
Pediatrics	12	190	16 (11 to 21)
Total	215	3,185	14 (1 to 37)

We classified the visual content of each advertisement by noting the number of glossy pages, small-print pages, and type and number of figures on the glossy pages. We characterized figures as (1) pictures—photographs, cartoons, diagrams, drawings; (2) scientific tables—research data in tabular format; (3) scientific graphs—research data presented in any standard graph format; and (4) pseudographs—arrows and diagrams labeled with numbers (e.g., “percent reduction”), but without axes or other standard graphing constructs that would permit meaningful interpretation of the dimensions. We measured the area of each figure and page to the nearest mm² to calculate the percent of all glossy pages devoted to figures.

We conceptualized the components of the scientific graph evaluation as a set of distinct constructs: graph format, comprehensiveness and coherence; visual quality; efficiency of design; and relation of the graph to the remainder of the advertisement. To score the graphs, we used a 34-item data collection instrument modified from our previous work.^2,3 All elements were categorical (e.g., type of graph), or dichotomous (present/absent). Definitions are presented in Appendix A. Two of 3 trained raters not blinded to the study's purpose independently coded each graph. We computed interrater reliability, and the authors adjudicated any discrepancies by consensus.

The intent of this investigation and analysis was descriptive. Results are presented as point estimates. We designed the study to include at least 75 graphs, a number which generates 95% binomial confidence limits for dichotomous variables that are within 12% of the observed value. We used a customized template with data checking in Access (Microsoft) for database entry, and STATA 6.0 for all statistical analyses (STATA Corp., College Station, Tex).

RESULTS

We found 3,185 advertisements in the 10 journals (Table 1). The 484 unique advertisements were comprised of 1,295 pages, 841 glossy and 454 small-print. The average number of pages/advertisement was 2.66 (1.66 glossy pages and 1 small-print page). The content of the glossy pages (by area) was 46.5% pictures, 0.4% tables, 1.6% graphs, and 0.5% pseudographs. The remaining 51% was text or blank.

We found 85 scientific graphs in 63 distinct advertisements. Seventy-four of these graphs were unique. (Eleven graphs appeared in different advertisements for the same product.) These 63 advertisements had a median of 1 graph per advertisement (range 1 to 4; interquartile range, 1 to 2). The following statistics are based on the 74 unique graphs unless otherwise stated. Interrater agreement was 85%.

All graphs were univariate. None contained features typifying graphical excellence (Table 2; see Appendix A for definitions).^4–8 Thirty-six percent of graphs contained sufficient information to interpret the results. Ninety-one percent of graphs defined all abbreviations and symbols, 55% had a title, 78% had clear quantitative labels for the x and y dimensions, 53% had a figure legend, and 69% depicted the sample size.

Table 2.

Characteristics of the 74 Unique Scientific Graphs^*

Characteristic	%
Simple univariate display	96
Pie chart	7
Bar or point graph without CI	85
Bar or point graph with CI	4
Univariate distribution	4
One-way plot	0
Histogram	3
Box-and-whisker plot	0
Survival curve	1
Bivariate display	0
Features of excellence displayed†	0
Internal graph errors	8
External error—discrepancy with other text	1
Visually clear	95
Nonstandard graphing conventions without explanation	12
Numeric distortion	36
Redundancy within the graph	46
Chartjunk	66
Data Density Index—median cm2 (IQR)	0.22 (0.11 to 0.43)

^*Percentages refer to the frequency a characteristic was noted.

^†Features of graphical excellence include those techniques that allow a second dimension or detail of the data to be displayed, including depiction of paired data, symbolic dimensionality, and small multiples display.⁴–⁷

CI, confidence interval; IQR, interquartile range.

The graphs were generally visually clear and upheld standard graphing conventions (Table 2). However, numeric distortion was found in 36% of advertisement graphs. Of the 27 graphs with numeric distortion, 18 had 1, 8 had 2, and 1 had 3 design features that produced visual overestimation or underestimation of the metric graphed. The most common features were: improperly scaled or improperly split axes (16%); 3-dimensional objects that needlessly compared volume instead of location, length, or area (20%); and improper baselines (12%).

Graphs failed to efficiently utilize space, with 66% containing “chartjunk” (extra grid lines, 36%; meaningless background shadings, 35%; color schemes that highlight 1 drug or outcome above others, 21%). The data density index was 0.22 for the advertisement graphs, ¹/₅^th to ¹/₂₅^th the density found in scientific manuscript graphs.^2,3,5

Fifty-eight percent of the graphs depicted the most salient outcome of the drug's purported use. The outcomes graphed (more than 1 may apply) included: intermediate outcome (54%), clinically important outcome by accepted paradigm (82%), cost (3%), or side effect (5%). In the 40 advertisements that graphed an intermediate outcome, only 6 contained the FDA-mandated statement that improvements in intermediate outcomes may not have clinical importance.⁹ When graphs depicted only a subset of the data presented in the advertisement's text (27 graphs), the subset graphed was biased in support of the drug's effect in 70% of cases and biased against it in 0%. (The other 30% were neutral or the effect could not be discerned from the information provided.)

DISCUSSION

In response to criticisms of product advertisements and promotional activities, a pharmaceutical representative stated “there is a need for responsible dissemination of information about drugs to physicians…(and) the availability of a drug is of little value unless the prescriber of the product is aware of its existence and has the scientific and medical information to use it effectively.”¹⁰ Data tables and scientific graphs can concisely and effectively communicate information; however, few pharmaceutical advertisements depicted data in this manner. The glossy pages were mainly text or other images. Other researchers suggest images and symbols (pseudographs) are used to “circumvent logical argument when trying to persuade people (the “targets” of the advertisement) to make choices that are not strictly rational.”¹¹

Pharmaceutical advertisements uncommonly contained scientific graphs (13% of advertisements). The few graphs found were basic univariate displays. Although some might believe a simple graph is easier to understand, there is considerable theoretical and empiric evidence against this assertion.^12,13 Complexity is not inversely related to comprehensibility. A “simple” map that is all 1 color, names every tenth street, employs the same font throughout, has no symbols for landmarks, and no legend is not better than a “complex” map that identifies all streets, varies line width to differentiate major highways from minor roads, applies colors to differentiate structures, includes symbols to identify landmarks, and includes a detailed figure legend.

Pharmaceutical advertisement graphs often did not contain sufficient information to interpret the data presented. Gutknecht reported similar findings in the text of pharmaceutical advertisements, noting P values were commonly presented but confidence intervals, pertinent power calculations, or other information necessary to interpret the results were often missing.¹⁴

We found one third of advertisement graphs contained design features that distorted the data depicted. The FDA states that it is a violation to use “tables or graphs to distort or misrepresent the relationships, trends, differences, or changes among the variables or products studied….”⁹ Other investigations of pharmaceutical advertisements have also described misleading claims information.^15,16

While the FDA has specific pharmaceutical marketing regulations,⁹ they acknowledge they cannot do the job alone.¹⁷ Fifty-seven percent of medical editors agreed that journals have a responsibility to ensure truthfulness in pharmaceutical advertisements, and 40% favored subjecting advertisements to rigorous peer review.¹⁸ However, a poll of peer-review researchers indicated that journal editorial staff seldom review advertisements to detect bias in reporting.¹⁹

A limitation of this evaluation is that the aesthetics of graph design are subjective. Assuming no contention about our choice of criteria, there is a potential bias, because the raters were not blinded to the study hypothesis. We attempted to minimize that bias by developing an explicit set of objective criteria for evaluating important aspects of graph design.^7,12,13 Alternatively, some may argue with our criteria. For example, some may disagree with the aesthetic principle that redundancy within a graph is never desirable, especially when the graph is part of an advertisement. We did not review the small-print pages that the industry believes to be important to conveying their educational message.²⁰ Finally, we chose a broad convenience sample of journals, but cannot be sure the frequency and quality of graphs in these advertisements are representative of all pharmaceutical advertisements.

Pharmaceutical advertisers seldom use graphs. Those presented are basic univariate displays with superfluous adornment. Readers should be aware that graphical displays in pharmaceutical advertisements often fail to convey the complexity of data, and may distort findings.

APPENDIX A

Definition of Characteristics

Features of graphical excellence4–7 (Features that facilitate the depiction of detailed or complex data relationships)

a. Paired data — Data are paired when 2 or more measurements are made in the same subject — either one parameter measured in the same subject at multiple times, or more than one variable measured simultaneously. A graph was given credit for pairing when it portrayed the inherent linkage of these measurements or variables.

b. Symbolic dimensionality — The use of varying symbols (e.g., letters, numbers, shapes) to depict an additional characteristic of the population beyond the comparison that the basic graph is intended to make.

c. Small multiples — The presentation of an array of small graphs, each of the same type, designed to convey information about each individual cell but, more importantly, the relationship of each cell to the entire array. Small multiples can be used when symbolic dimensionality is impractical.

Internal graph errors — contradictions within the graph. For example, axis values that do not correspond to the actual distance along the axis.

Visual clarity — graphical depiction that does not obfuscate the interpretation of data points, confidence intervals, axes or labels.

Nonstandard graphing conventions without explanation — elements that have widely accepted definitions (e.g., the horizontal lines of a box-and-whisker plot) are given alternate meanings without any indication in the graph legend that this has occurred.

External graph error — any discrepancy between the data presented in the text or tables of the advertisement and the values depicted in the graphic.

Redundancy within the graph — repetition of information within the graphic and its legend, e.g., the clear depiction of a parameter's value by an appropriately demarcated axis and the labeling of the parameter's value with its numerical value as well.

Redundancy of the graphic with text/table — declared when the graphic presented no unique information beyond that contained in text, tables, or other graphics.

“Chartjunk”— extraneous decoration that creates “visual noise” and distracts from the informative elements of the graph.8

Data density index — a statistic that measures the amount of information conveyed per square centimeter of graph.2,5 The numerator is the number of pieces of information illustrated in the graphic. The denominator is the number of square centimeters that the graph occupies including axes and axis labels, but not titles or legends.