The availability of references and the sponsorship of original research cited in pharmaceutical advertisements

Abstract

Background

The primary goal of pharmaceutical advertisements is to convince physicians to prescribe the manufacturer's product. We sought to determine what materials are cited in support of claims in pharmaceutical ads and medical research articles, and whether health care professionals seeking to verify the claims could obtain these references.

Methods

We reviewed 438 unique ads from the 1999 issues of 10 American medical journals, and a random sample of 400 references in medical research articles selected from the same journals. We classified references as journal article, data on file, meeting abstract or presentation, book or monograph, marketing report, prescribing information, government document or Internet site. We attempted to confirm or obtain each reference through library and Internet searches or by direct request from the manufacturer. The main outcome we sought to determine was the availability of the reference to a clinician. We also ascertained the source of funding for original research cited in the ads and the research articles.

Results

In the 438 ads with medical claims, 126 contained no references and 312 contained 721 unique references. Of these ad references, 55% (396/721) cited journal articles and 19% (135/721) cited data on file. In contrast, in the sample of research article references, 88% (351/400) cited journal articles and 8% (33/400) cited books. Overall, 84% of the citations from the ads were available: 98% of journal articles, 86% of books, 71% of meeting abstracts or presentations and 20% of data-on-file references. In all, 99% of the sample of research article references were available. We determined that 58% of the original research cited in the pharmaceutical ads was sponsored by or had an author affiliated with the product's manufacturer, as compared with 8% of the articles cited in the research articles.

Interpretation

Many pharmaceutical ads contain no references for medical claims. Although references to journal articles were usually obtainable, other published sources were not as easily acquired. The majority of unpublished data-on-file references were not available, and the majority of original research cited to substantiate claims in the pharmaceutical ads was funded by or had authors affiliated with the product's manufacturer.

The primary goal of pharmaceutical advertisements is to convince clinicians to prescribe their product. These ads often cite external documents in support of their claims, but studies have shown that these claims may be misleading, distort the reporting of scientific data or fail to provide enough information to accurately interpret the data they present.^1,2,3,4,5,6 The transparency of pharmaceutical ads is important for 2 reasons. First, there is evidence that physician prescribing is influenced by pharmaceutical ads.^7,8,9 Second, the pharmaceutical industry views ads as one way in which they can educate physicians.^10,11 Given the potential for misrepresentation, health care professionals should be able to examine the cited references to determine whether the manufacturer's claims are justified.

In 2002, pharmaceutical companies submitted over 34 000 ads to the US Federal Drug Administration (FDA) (Thomas Abrams, Director, Division of Drug Marketing, Advertising, and Communications of the FDA, Rockville, Md.: personal communication, 2003). Most ads made several claims about the drug's efficacy or effectiveness. The sheer volume makes it difficult for the FDA to adequately check the validity of many of the ads.¹² The absence of tight government oversight makes it particularly important that individual clinicians and groups (e.g., hospital pharmacy and therapeutics committees) can obtain cited documents to determine whether a claim is adequately substantiated. We performed this study to describe the kinds of documents cited in support of claims made in pharmaceutical ads, and to assess the availability of these documents to health care practitioners. We also examined whether original research cited by the pharmaceutical ads was sponsored or performed by the product's manufacturer. To provide a standard for comparison, we repeated these investigations for randomly selected references in research articles published in the same journals in which we studied the ads.

Methods

We retrospectively examined all references in the 438 unique pharmaceutical ads contained in the 1999 issues of 10 medical journals (American Journal of Psychiatry, Annals of Emergency Medicine, Annals of Internal Medicine, Annals of Surgery, Hospital Practice, Journal of the American Medical Association, New England Journal of Medicine, Neurology, Obstetrics and Gynecology and Pediatrics). This convenience sample of journals was chosen to reflect a range of general and specialty care and, therefore, ads for a range of pharmaceutical products. A database of the ads was initially created to examine the quantity and quality of graphic data displays in pharmaceutical ads.⁵All of the journals are peer-reviewed except Hospital Practice, a journal that is free to house staff in the United States.

Two of 3 trained reviewers independently reviewed each ad and identified the number of unique references cited. We defined a reference as material cited in support of the information in the ad. The reviewers classified each reference as: journal article, generic data on file (a reference to an unspecified, unpublished company document), specific data on file (e.g., “Drug Company packet WP 1234”), meeting abstract or presentation, book or monograph, marketing report (material cited in support of claims such as “prescribed over 2 million times in 1999”), prescribing information (e.g., Physicians' Desk Reference [PDR] or package insert), government document (e.g., the US Centers for Disease Control and Prevention Morbidity and Mortality Weekly Report [MMWR]) or Internet site. The PDR is an annual publication that provides information on drug usage, pharmacology and pharmacokinetics, FDA-approved indications, warnings and drug interactions for over 4000 drugs.¹³

For the purpose of comparison, we randomly selected 1 issue of each of the study journals from 1999, randomly reduced the references to a maximum of 40 per research article, then randomly selected a total of 40 references from each journal. These 400 references were categorized in the same way as the references in the pharmaceutical ads. In addition, we examined a random sample of 100 of the journal references citing original research and all of the pharmaceutical ad references to original research to determine whether the research was sponsored by or had authors affiliated with a pharmaceutical company.

We considered a reference “available” if we could obtain a copy of the cited material. All journal citations found in the National Library of Medicine's MEDLINE/PubMed were considered available (www.ncbi.nlm.nih.gov/entrez/query.fcgi). For other journal citations, abstracts, meeting presentations and books, we attempted to locate and obtain the document through searches of other databases (e.g., WorldCat) and in the California Digital Library (a colibrary of the University of California, www.cdlib.org), which includes a national network of libraries that permit interlibrary loans. Searches were conducted using single and multiple terms (e.g., individual author names, journal issue, title and page numbers) to ensure that typographical errors in the citation did not interfere with verification of the reference. We kept a record of cases in which errors were found.

We used an Internet search engine (www.google.com) to confirm the existence of documents not available through our library search. This included searching titles, authors and, for abstracts and presentations, the meeting's host organization. If no link to meeting proceedings could be found, we asked the host organization to provide a copy of the document.

We determined whether the journal articles cited were from journals in the National Library of Medicine's (NLM's) list of journals indexed for MEDLINE,¹⁴ whether the article appeared in a journal supplement, and the type of article (original research, review article, letter or editorial) cited. When we located a cited journal article that presented original research, we determined the source of funding and whether any authors were affiliated with the manufacturer of the product being studied.

We checked government documents by searching the Web site of the appropriate agency (e.g., www.cdc.gov, www.NIH.gov), and verified Internet references by accessing the listed URLs.

We verified the existence of marketing reports (e.g., Source Prescription Audit, IMS National Prescription Audit, Medi-Span) but did not attempt to obtain these documents, since we did not believe they would be of interest to clinicians.

We wrote directly to the manufacturer to request copies of data-on-file references. For specific data-on-file references, we requested each cited document by number. For generic data-on-file references our request included details about the citation and the journal (name, volume and date) in which the ad appeared. For manufacturers with 1 or 2 references to generic data on file across all of their ads in the database, we requested all material. For manufacturers with more than 2 references to generic data on file, we requested 2 randomly selected documents. All letters included email and US postal addresses to which the documents could be sent. A separate letter was sent for each reference. We used a variety of aliases and addresses so that no company received more than 1 letter request from a specific individual or address. If we did not receive a response within 6–8 weeks, a repeat request was mailed. We attributed responses postmarked within 5 days of our second mailing to the initial mailing. We classified each response as “decline” (“information proprietary” or “not available”), “unpublished study data” or “journal reprint.” We did not categorize package inserts or referrals to the company's Web page as available, since they were not the data-on-file references we requested.

We did not set out to test specific hypotheses, but rather to describe the availability of the references cited in pharmaceutical ads and the sponsorship of original research cited in the references.

Results

Of the 438 ads in the database 126 (29%) offered no references in support of their claims. In the remaining 312 ads, there were a total of 1072 references, with 721 citing a unique source document. The 312 ads listed a median of 2 references (range 1–25). The 10 randomly selected journal issues (1 issue of each journal) contained 184 research articles with a total of 5233 references, of which we randomly selected 40 references per journal, as described in Methods.

The most commonly cited sources in the pharmaceutical ads were journal articles (55%) and data on file (19%). In the sample of research article references, the most commonly cited sources were journal articles (88%) and books (8%). Among the confirmed references (i.e., those to journal articles, books, abstracts and government documents), we found errors (e.g., wrong author, title, journal, page number, year) in 6% (30/476) of the drug ad references and 4% (14/397) of the research article references.

A total of 84% (494/590) of the citations in the pharmaceutical ads and 99% (397/400) of the citations in the research papers were available (Table 1). A majority (98% [390/396]) of the pharmaceutical ad references to journal articles were available; 90% (352/390) were from journals in the NLM's list of journals indexed for MEDLINE, and 12% of the available references were published in journal supplements. Of the research article references, 99% (348/351) of the journal articles were available; 99% (345/348) were from journals in the list of journals indexed for MEDLINE, and 4% (14/351) of the available references were published in supplements. The majority of the journal references in both samples were to original research or review articles (Appendix 1).

Table 1

Of the 294 pharmaceutical ad references citing original research, 58% indicated that the research was sponsored by or had authors affiliated with the product's manufacturer, 19% stated funding by a government or charitable organization, and the remainder (23%) had no statement regarding funding or specifically reported no funding. In contrast, in the sample of 100 original research papers cited in the journal articles, 8% were sponsored by or had authors affiliated with the product's manufacturer, 44% were funded by a government or charitable organization, and 48% had either no statement regarding funding or specifically reported no funding.

Approximately half (54% [167/312]) of the pharmaceutical ads had at least 1 data-on-file reference, for a total of 135 unique references. Forty-two (52%) of the 80 companies in the drug ad database cited to data-on-file references in at least 1 ad. The pharmaceutical companies replied to 37 (42%) of our 88 requests for data-on-file, but 19 (51%) of the responses were letters stating that the data on file would not be provided because it was proprietary (17) or because of company policy (2) (Table 2). Thus, only 20% (18/88) of our requests produced a document. In 3 of these cases, the company indicated that additional material had been withheld because of its proprietary nature. None of our letters was returned as undeliverable.

Table 2

We obtained 71% of the meeting abstracts and presentations from the pharmaceutical ad references, including 17 of the 19 references to published abstracts of meetings and 10 of the 19 references to meetings. Six of the 11 unavailable references were from confirmed meetings but the program details were unavailable. Of the remaining 5, in 2 instances we confirmed the meeting and the author or presentation but could not obtain the abstract, in 2 cases we could not confirm the meeting or abstract, and in 1 case we found meeting proceedings that did not include the cited reference. Five (14%) of the 36 books and monographs cited in the ads were unavailable. In contrast 100% of the abstracts and books cited in the sample of 400 research articles were available (Table 1).

All 28 unique references to government documents in the pharmaceutical ads were confirmed. We checked the 3 Internet references cited in the pharmaceutical ads in November 2002 and found that the sites were operational (e.g., www.NIH.gov), but the specific pages cited were unavailable.

Interpretation

One-third of the pharmaceutical ads did not provide any references in support of their claims. Although the references to articles in journals indexed for MEDLINE were easily obtainable, other published sources were not as easily acquired, and references to unpublished material were seldom available. We confirmed 98% of the references to journal articles, as compared with 83% reported in a study of drug promotional brochures.³ This difference may be due in part to our use of multiple library sources and the interlibrary borrowing privileges of a large university.

In contrast with the research articles, the pharmaceutical ads cited a large number of unpublished documents. Nineteen percent of the ad references were to data on file, which is consistent with Mindell and Kemp's finding of 20% in ads in BMJ.¹⁵ Despite follow-up mailings, only 20% of our requests for data-on-file references yielded information from the pharmaceutical companies. This is consistent with the results of Hafeez and Mirza, who found a 26% response rate for mailed requests for product information.¹⁶ Although companies in the United States must comply with FDA requests for proprietary documents,¹⁷ the companies have no legal obligation to make these documents available to clinicians (Thomas Abrams, Director, Division of Drug Marketing, Advertising, and Communications of the FDA, Rockville, Md.: personal communication, 2003). In Canada, the Pharmaceutical Advertising Advisory Board's (PAAB) Code of Advertising Acceptance requires that data-on-file references be available to the PAAB's commissioner and a summary copy of these data be provided to health care professionals who request it.¹⁸ However, it is not known how often physicians seek this information. The World Health Organization (WHO) has stated that “Scientific data in the public domain should be made available to prescribers and any other person entitled to receive it, on request, as appropriate to their requirements.”¹⁹ Our findings demonstrate that the WHO policy is not being followed.

We also found that 58% of the original research cited in pharmaceutical ads was sponsored by or had authors affiliated with that same pharmaceutical company, as compared with 8% of the references to original research in the research articles. Although it is not surprising that a manufacturer would cite research that it had sponsored, previous research indicates that conclusions of trials sponsored by pharmaceutical companies may be misleading or incompletely reported.^{20,21,22,23,24,25,26} Recent calls for improved transparency of pharmaceutical industry-sponsored research may improve the quality of trial reporting and publication.²⁷

There are a number of limitations to our findings. We examined only the availability of references cited in support of claims. We did not attempt to verify whether each claim made in the ad was supported by a specific citation. In addition, we did not examine the cited documents to determine whether they adequately substantiated the claims made in the ads. Previous research indicates that journal articles cited to substantiate statements in pharmaceutical ads failed to do so in 19%–44% of claims.^1,2,3,6,15 We did not assess the quality of the journal articles, because others have already demonstrated methodological flaws in articles cited in pharmaceutical ads;^15,28 however, we did find that 12% of the articles in our sample were published in supplements, which may be of lesser quality than articles published in the parent journal.^29,30 Our access to the extensive libraries and interlibrary loan requests may not be representative of a typical clinician's access, and our repeated efforts to obtain documents may be greater than those of a practising clinician. Finally, some unconfirmed references may have been due to errors in the citation listing in the ad. We tried to circumvent these problems by using multiple terms when searching.

In contrast to the citations in published research articles, clinicians will be unable to obtain the evidence offered in support of a substantial number of claims in pharmaceutical ads and will therefore be unable to assess the validity of these claims. Improved accessibility to and monitoring of the validity of references in pharmaceutical ads is required to allow clinicians to practise evidence-based prescribing and help policy-makers adequately assess the utility of the pharmaceutical products in question.

Appendix 1.