Evaluation of Selective Outcome Reporting Bias in Efficacy Endpoints in Print and Television Advertisements for Oncology Drugs

Abstract

Importance

Selective outcome reporting bias in oncology drug advertisements may encourage misconceptions about a drug’s efficacy profile.

Objective

We sought to determine the rates of selective outcome reporting in published cancer clinical trials and in television and print advertisements for anticancer medications. We also quantified the number of advertisements that did not include or cite any studies with mature overall survival (OS) data (i.e., data with all required patient events for final analysis).

Design/Setting/Participants

We conducted a cross-sectional investigation of advertisements uploaded to the AdPharm Database (repository of pharmaceutical advertisements); the clinical trials supporting the ads; and the trial registrations associated with the trials. Data were extracted by two investigators who were blinded to each other’s data.

Main outcome measures

The first co-primary objective was to investigate selective outcome reporting between trial registrations and published trials. The second co-primary objective was to investigate selective outcome reporting between the same published trials and drug advertisements.

Results

We included 74 advertisements and 48 clinical trials. Print ads were the most common (n = 66), and most print advertisements were targeted to health care providers (n = 55, 83.3%). Overall, 41/48 (85.4%) trials were registered prior to study enrollment, and 41/48 (85.4%) did not deviate from the registered primary endpoints. Across all advertisements (n = 74), statistically significant endpoints were more often reported (unadjusted risk ratio [uRR] 1.26; 95% confidence interval [CI] (1.14–1.40)) and 22/55 (40.0%) advertisements cited trials with immature overall survival data (i.e., data without the required number of events for final analysis).

Conclusions

In our sample, statistically significant endpoints were more commonly reported than nonsignificant endpoints. Immature endpoints (those analyzed before the required number of accrued patient events) were often reported. By reporting only significant endpoints and those that are immature, advertisers may encourage misconceptions about a drug’s efficacy profile.

INTRODUCTION

Industry-sponsored television and print advertisements targeted to consumers and health care providers (HCPs) compose a multibillion-dollar industry in the USA.¹ Consequently, the benefits and harms of these advertisements have been strongly debated, with much of the discussion focusing on consumers.^{2, 3} Advocates of direct-to-consumer advertisements argue that they function as public service announcements that empower patients with information, lead to doctor-patient conversations, and facilitate the initiation of treatment.^4–6 Opponents argue that direct-to-consumer advertisements may mislead patients,^{7, 8} exaggerate potential drug benefits,^{9, 10} omit quality of life,¹¹ and increase health care spending.^{4, 5} In cancer medicine, drug advertisements have been the subject of particularly intense debate,^11–13 especially given the often high toxicity¹⁴ and cost¹⁵ associated with new cancer medications. The controversial nature of oncology drug advertisements, paired with their prevalence in the lives of HCPs and consumers, raises the critical question of whether the clinical data in oncology drug advertisements are transparent, straightforward, and unbiased.

One threat to the accurate presentation of clinical data is selective outcome reporting bias, which occurs when published study endpoints do not match those prespecified in a trial registry or protocol.¹⁶ Trial endpoints may be added, removed, or reordered for several reasons. Some of these reasons, such as poor study accrual,¹⁷ are ethical and understandable. However, in other cases, selectively reporting endpoints can be dangerous and may affect perceptions of drug efficacy through the omission or demotion of statistically nonsignificant results. A recent analysis of hematology clinical trials found that endpoints were often selectively reported to highlight statistically significant results,¹⁸ and a Cochrane systematic review found that selective outcome reporting bias in clinical trials affected the conclusions of a “substantial proportion of Cochrane reviews.”¹⁹ To avoid misleading readers, authors of medical research studies should accurately report data for all endpoints prespecified in their protocol, regardless of statistical significance.

While much is known about the selective reporting of trial endpoints between protocols and published reports, little, if anything, is known about the selective reporting of trial endpoints between published reports and drug advertisements. Because advertisements represent a snapshot of a drug’s evidence profile, they may be slanted toward selective reporting of endpoints previously analyzed in published trials. The primary objective of the current study was to investigate the rates of selective outcome reporting bias of efficacy endpoints at two junctures: in published cancer clinical trials and in television and print advertisements for anticancer medications. The rationale for this investigation was that selective outcome reporting bias has been shown to be a consistent issue in the biomedical literature,^18–21 and print or television advertisements may unintentionally inflate perceptions of the benefits of oncology drugs.

METHODS

Consistent with a recent investigation of health care advertisements,²² we used the AdPharm database to identify oncology drug advertisements uploaded within an 18-month span between March 1, 2017, and September 1, 2018. AdPharm is an online database that is updated daily with advertisements for health care or pharmaceutical products. Each entry in AdPharm contains basic information about the advertisement, including the target audience or country of origin. AdPharm does not track or list the number of viewers of an advertisement. Advertisements were included in the study if they were for an anticancer drug and if they included quantitative data, were in English, and were marketed to consumers or HCPs.

After screening all advertisements, CW and GA extracted data in a duplicate and masked fashion. The following items were extracted from print and television advertisements: market audience, air or print date, efficacy endpoints, data for efficacy endpoints, design features of the clinical trial that generated the data, any citation for a published trial, and, in the case of a consumer-directed advertisement, any mention of speaking with an HCP.

To compare advertisement endpoints with journal-published endpoints, we used the citations in the advertisements or a PubMed search to identify a matching trial. We used keywords and Boolean operators to search for and identify matching trials, if no citation was included. Trials were matched on the basis of intervention, co-intervention, control, sample size, and cancer type. After identifying matched trials, we extracted the efficacy endpoints reported, data for those endpoints, and the date of article publication. Our analysis of selective reporting bias between published articles and advertisements consisted of determining which endpoints were included in the published paper and which were included in the advertisements. When an endpoint was excluded from the advertisement, we then determined whether or not that endpoint was statistically significant using the published statistics (e.g., confidence intervals or alpha level). Similarly, we investigated selective outcome reporting between the retrieved published papers and their trial registrations. We chose to use trial registrations, rather than protocols, because trial registrations are time-stamped and show a history of changes, which supports an accurate analysis of any endpoint changes or updates.

This is a novel study of selective outcome reporting in drug advertisements. As such, there is no effect size on which to base a power calculation. Therefore, we provide a range of included studies required for sufficient power using standard effect size measurements (Cohen’s d = 0.2, 0.5, 0.8). These effect size measurements were converted to odds ratios for our power calculation, based on the paper by Chen et al.²³ We prespecified a type I error rate of 0.5 and type II error rate of 0.2. The range of included advertisements required ranged from 485 (odds ratio = 1.68, Cohen’s d = 0.2) to 89 (odds ratio = 3.47, Cohen’s d = 0.5) to 45 (odds ratio = 6.71, Cohen’s d = 0.8). We used gpower 3.1 for all power calculations.

We used Stata 15.1 for all analyses except E-values, for which we relied on the formula described by VanderWeele and Ding.²⁴ E-values were used to assess the degree of unmeasured confounding in our analyses. For the two primary endpoints of selective outcome reporting bias of efficacy endpoints in published papers and in advertisements, we calculated unadjusted risk ratios (uRR) and 95% confidence intervals (CIs) to compare the rates of advertising significant and nonsignificant endpoints. We analyzed all advertisements together, as well as consumer- and physician-directed advertisements separately. In all analyses of selective outcome reporting bias, we excluded endpoints from single-arm trials, immature overall survival (OS) data, and endpoints that could not be located in the published paper. We define “immature” data as data that have not accrued the prespecified number of patient events to achieve study power.

RESULTS

We identified 490 advertisements in total, of which 74 were included initially (Fig. 1). Advertisements were excluded for not describing a drug treatment (n = 249), not including quantitative data (n = 88), and not being in English (n = 79). The vast majority of print advertisements (n = 66) were in clinical magazines and designed to target HCPs (n = 55, 83.3%). Print advertisements pertained to 34 unique drugs designed to treat 21 unique malignancies. The drugs that were the most commonly advertised in print were pembrolizumab (n = 8), palbociclib (n = 6), and ribociclib (n = 5). All television advertisements (n = 8) were directed to consumers and were related to four unique drugs and two unique malignancies. Palbociclib was the most commonly television-advertised drug (n = 3), followed by pembrolizumab (n = 2), nivolumab (n = 2), and abemaciclib (n = 1). The only malignancies represented were non-small-cell lung and breast cancers (both n = 4).

Figure 1.

Flow diagram of included and itemized excluded advertisements.

REGISTRATION TO PUBLICATION

Forty-eight clinical trials were identified that supported the 74 included advertisements. All 48 trials reported a trial registration number. Seven trials were registered after the start of subject enrollment, although one trial began in 1999 before ClinicalTrials.gov registration. Besides the six trials that were registered after they began (excluding the trial that began in 1999), an additional six studies deviated from the registered primary endpoints in ways that may have affected the integrity of the trial. For all six, primary endpoints were added to the registry after the start of the study. In one study, an endpoint was demoted from primary to secondary in the published report. With regard to registered secondary endpoints, 16 trials deviated from the registry, with 13 adding secondary endpoints during the trial period. One study promoted a registered secondary endpoint to a primary endpoint in the publication, one removed a secondary endpoint from its registry, and one did not list or report a registered secondary endpoint in the paper. Overall, 41/48 (85.4%) trials were registered prior to study enrollment and 41/48 (85.4%) did not deviate from the registered primary endpoints.

PUBLICATION TO ADVERTISEMENT

After excluding advertisements supported by single-arm trials (n = 8), we next compared the efficacy endpoints cited in the 66 remaining advertisements to the 40 remaining clinical trials supporting them. Of the 539 endpoints eligible for inclusion in advertisements, we excluded 175 endpoints for being from single-arm trials (n = 100), for including immature time to event data (n = 51), or for not including a statistical analysis in the published paper (n = 24). Five trials were cited for advertisements directed to consumers and physicians.

Across all included advertisements (n = 66), statistically significant endpoints were more likely to be reported than nonsignificant endpoints (uRR 1.26; 95% CI 1.14—1.40). Primary endpoints were reported 97.8% (92/94) of the time. Secondary endpoints were reported much less frequently (66/270, 24.4%). Overall, half (33/66, 50.0%) of advertisements included data for immature endpoints.

Among advertisements directed to HCPs (n = 47), if an endpoint was statistically significant, it was more likely to be reported in the advertisement (uRR 1.36; 95% CI 1.20–1.54). For consumer-directed advertisements, there was no significant difference (uRR 1.01; 95% CI, 0.85–1.21) (Table 1).

Table 1.

Selective Outcome Reporting of Endpoints Between Advertisements and Trials

Overall analysis (no. of endpoints)			No. (%)	Statistical analysis	E-value
Physician advertisements (n = 55)	Significant endpoints (n = 207)	Reported	102 (37.9)	uRR 1.36 (95% CI 1.20–1.54)	uRR, 2.06; Lower limit CI, 1.69
		Not reported	105 (39.0)
	Nonsignificant endpoints (n = 62)	Reported	10 (3.7)
		Not reported	52 (19.3)
Consumer advertisements (n = 19)	Significant endpoints (n = 80)	Reported	39 (41.1)	uRR 1.01 (95% CI 0.85–1.21)	uRR, 1.11; Lower limit CI, 1.0
		Not reported	41 (43.2)
	Nonsignificant endpoints (n = 15)	Reported	7 (7.4)
		Not reported	8 (8.4)
Total advertisements (n = 74)	Significant endpoints (n = 287)	Reported	141 (40.8)	uRR 1.26 (95% CI (1.14-1.40)	uRR, 1.83; Lower limit CI, 1.54
		Not reported	146 (42.2)
	Nonsignificant endpoints (n = 77)	Reported	17 (4.9)
		Not reported	60 (17.3)

DISCUSSION

This study is a novel investigation of selective outcome reporting in drug advertisements marketed to consumers and health care providers. We found that statistically significant endpoints were more likely to be reported than nonsignficant endpoints. This finding was mostly driven by physician-directed advertisements, which were more prevalent and where the difference was also significant. Because previous studies investigating selective outcome reporting in drug advertisements do not exist, it is not possible to compare our results within the context of previous literature. In this study, we also evaluated selective outcome reporting between trial registrations and the published trial reports, which is the conventional manner for the investigation of selective outcome reporting ^{16, 25, 26}. There is ample evidence that industry-funded studies are more likely to report more favorable results in published papers^27–29. Our results indicate that the degree of selective outcome reporting was higher between published trial reports and advertisements than between the trial registrations and their publications. These findings raise important questions about perceptions of drug efficacy. Moreover, many included endpoints were surrogate endpoints, which may or may not correlate with improved survival in cancer patients³⁰ and are more likely to be statistically significant³¹. Some cancer trialists have argued that OS should be routinely collected and reported, owing to the importance that patients with cancer place on decreased mortality³².

Our study found that advertisements were often aired or printed before final OS data were available, which may introduce uncertainty and may raise the risk of reporting false-positive results to the public ³³. Previous studies have found that only negligible correlations exist between surrogate outcomes and OS for many types of cancer ³⁰. Furthermore, the results from surrogate outcomes—published as interim analyses before OS data are mature—often do not result in improvements in OS ³¹. Thus, we believe that the surrogate outcomes reported in media advertisements have the potential to overstate the efficacy benefit that will eventually be found when OS data become available.

To our knowledge, the Food and Drug Administration (FDA) does not offer guidance on reporting surrogate endpoints and OS in oncology drug advertisements. Existing draft guidance for advertising efficacy endpoints focuses on the reporting of absolute or relative statistics.³⁴ This gap in FDA guidance may be relevant to patients if advertisements only report surrogate endpoints. A recent review found that there are no high-quality data supporting the idea that patients understand surrogate endpoints and their shortcomings.³⁵ The lack of guidance and patient misunderstanding may multiply issues with oncology drug advertisements. Namely, we have shown that nonsignificant endpoints and immature OS data are often excluded from oncology drug advertisements, resulting in a higher degree of significant surrogate endpoints, which patients may not fully understand.

One must weigh the benefits and harms of oncology drug advertisements as seen in this study. The advertisements that we assessed often excluded nonsignificant endpoints, yet drug advertisement proponents argue that advertisements, any selective reporting aside, initiate a patient-physician conversation.^4–6 Because of the paucity of research into the effects of selective outcome reporting in drug advertisements, we cannot address how the omission of nonsignificant endpoints affects patients’ perceptions of drug efficacy. It is even more difficult to assess the effect of these advertisements on physicians, who in theory should be well versed in clinical endpoints and should have read the clinical trials associated with advertised drugs. However, we believe our study raises questions that could be answered using robust methodologies, following the example of other forms of bias ³⁶.

Our study is limited because we were not able to assign an appropriate weight to each advertisement based on audience size. The television advertisements, which were all marketed to consumers, likely had a larger audience than the print ads, which were mostly marketed to HCPs and published in clinical journals. So, while most advertisements included in this study were marketed to HCPs, these advertisements were likely seen by fewer people. Moreover, it is difficult to determine whether selective outcome reporting in patient-directed advertisements (where it exists) has the same effect as in physician-directed advertisements. We may reasonably assume a higher degree of health literacy among physicians; therefore, selective reporting of endpoints in advertisements directed to physicians may not carry similar weight as for advertisements directed to patients. Last, the computed E-values for this study show that unobserved confounding may affect our results (i.e., that some factor other than the significance of endpoints may drive reporting). However, even if other factors contribute to the reporting of endpoints in advertisements, this finding does not change the fact that we identified a possibly biased drug efficacy portfolio in advertisements.

In conclusion, we found that oncology drug advertisements are more likely to include statistically significant endpoints than nonsignificant endpoints. This effect was most pronounced in advertisements marketed to HCPs. All advertisements relied mostly on surrogate endpoints and frequently omitted nonsignificant OS data. Immature OS data did not create a barrier to advertising a drug as effective to consumers and HCPs. We recommend that advertisements not be aired or printed without clear descriptions of patient-important endpoints, such as OS. Furthermore, we recommend that the FDA critically review advertisements in the preapproval stage to ensure that patients and physicians are not misled (even unintentionally) regarding drug efficacy. We advocate for improved patient education of surrogate endpoints because available studies have shown that patients may conflate surrogate endpoints with clinically meaningful outcomes.³⁵ At minimum, since few, if any, oncology drugs aim to improve quality of life alone, we recommend a clear, prominent declaration of whether or not the drug has shown OS improvements. Future studies should be conducted to confirm our results, using a larger cohort of advertisements.

Contributions

CW and MV conceptualized and designed the study. CW and GA extracted and analyzed all data. CW, GA, and MV wrote and approved the final version of the manuscript.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.