Analysis of reference sources used in drug-related Wikipedia articles

Abstract

Objective

References from drug-related Wikipedia articles and a drug information database were compared.

Methods

Drugs in Food and Drug Administration (FDA) MedWatch alerts from January–July 2013 were searched in Wikipedia and Lexicomp to compare reference types and to assess the time for drug safety information to be incorporated into Wikipedia articles.

Results

Wikipedia most commonly cited peer-reviewed journal articles (49.2%) and news articles (12.0%). MedWatch citations were incorporated into Wikipedia on average in 5.9 days.

Conclusions

Wikipedia cited various sources but may not be a reliable, up-to-date resource for drug safety information.

INTRODUCTION

Wikipedia is an open-access, online encyclopedia that is collaboratively written and edited by its users [1]. Wikipedia contains information on a variety of topics, including health-related topics. Pharmacists, pharmacy students, and other drug-information seekers are likely to encounter Wikipedia pages when they perform a search online. One study found that Wikipedia entries appeared among the first 10 results in more than 70% of the search engines and keywords evaluated [2]. In a survey of practicing pharmacists, 28% admitted to using Wikipedia for drug information purposes, but only 28% were aware of who was responsible for editing and managing the entries [3].

Several studies have attempted to examine the accuracy and completeness of health and drug information in Wikipedia articles [4–8]. In general, health-related articles in Wikipedia have been found to be fairly accurate, but the information is often incomplete [4–8]. Wikipedia content guidelines state that the content in articles should be based on reliable, published sources [9]. The guidelines recommend using review articles, monographs, or textbooks over primary research articles and advise editors against interpreting the content of the primary source [9].

There is little information on the quality of the reference sources used for health-related Wikipedia articles. In 2011, Haigh analyzed the quality of 2,598 referencing sources used in 50 Wikipedia pages about health-related topics and found that 56% of the references cited were from clearly identifiable, reputable resources, which was defined as peer-reviewed journals, World Health Organization (WHO) studies, and Cochrane Collaboration reviews [10]. However, the review did not report what other types of sources were used for the remaining 44% of sources, and the author noted that future research should use at least 2 independent reviewers when conducting this type of analysis to minimize the chance of bias [10].

In this study, the authors characterized the quality and quantity of references used to construct drug-related Wikipedia articles and compared them to a well-known popular drug information database, Lexicomp. A secondary aim was to determine the amount of time it takes for drug information to be incorporated into Wikipedia articles.

METHODS

To select Wikipedia articles for evaluation, the authors chose a convenience sample of drugs for which a recent safety alert was published on the Food and Drug Administration (FDA) MedWatch website <http://www.fda.gov/Safety/MedWatch/>. A time period of seven months (January to July 2013) immediately prior to the start date of the study (August 21, 2013) was used. All medications included in the alerts mentioned on the FDA’s MedWatch website that reported a specific Drug Safety Communication resulting in a clinically significant label change or a new warning or recommendation to consumers or health care providers were included in the analysis. Medications with recall alerts for product integrity were excluded from analysis. One FDA alert was issued for “incretin mimetics” as a class. In this case, each drug listed specifically in the alert was evaluated separately (i.e., exenatide, liraglutide, sitagliptin, saxagliptin, alogliptin, and linagliptin).

Lexicomp was selected as the comparator database because previously published literature has suggested that it is a popular database and is one of the most preferred resources used by pharmacists and pharmacy students [11–13]. A product of Wolters Kluwer Health, Lexicomp is a pharmacy drug compendium that provides comprehensive drug, disease, and clinical reports. Numerous platforms are available that may be used by pharmacists, physicians, dentists, nurse practitioners, physician assistants, and nurses. Lexicomp can be purchased by organizations (hospitals institutions, academic institutions, community chains) but can also be purchased by individuals to use at home or on their mobile devices. Subscription prices for use on mobile devices range from $175 to $600 per year.

On August 21, 2013, Wikipedia articles for all of the drugs included in the FDA MedWatch alerts (including reference lists) were copied in their entirety and saved as a Microsoft Word document to ensure that all articles were being evaluated at the same point in time. Corresponding drug monographs from Lexicomp were also copied into a Microsoft Word document for analysis on the same date. Reference lists for each Wikipedia article and each monograph in Lexicomp were analyzed in the following manner:

1. For each article, the number of references included was recorded.

2. The following information was also recorded for each individual article or monograph: the age of the reference, presence or accessibility of a hyperlink to the original document, and reference category.

3. For each Wikipedia article, the time to incorporation of the MedWatch information as well as the time to include a reference to the MedWatch website was calculated.

The analysis was conducted by a team of two independent reviewers. After completing the analyses independently, the reviewers met and came to a consensus regarding any discrepancies found. Further details on how each section of the analyses was conducted are included below.

Reference age

To determine reference age, the publication year of each citation was subtracted from 2013. The average age of all references for all of the drug articles in Wikipedia and monographs in Lexicomp was then computed, by adding up all the ages of individual references and dividing this total by the number of references.

Accessibility

Each reference was evaluated for the presence of a hyperlink to the original source. When a hyperlink was included, the reference source was categorized as “accessible” if it included a working link to the source, and “not accessible” if the link was not working or led to an unrelated website. If no link was included with the source reference, it was placed into a separate category, “no link present.”

Reference category

References were characterized as belonging to one of the following categories: journal article, meta-analysis or systematic review, professional evidence-based guideline, book, news article, government website, academic or institutional website, professional organization website, commercial website, prescribing information, meeting abstract or presentation, patent, personal page or blog, or “other.” Additionally, any sources published in the medical literature (i.e., journal articles, systematic meta-analyses, and guidelines) were analyzed to determine whether or not the source was classified as a “Core Clinical Journal,” as defined by the US National Library of Medicine [14].

Time to page update

To obtain the time-to-page update for Wikipedia articles, the “view history” function was used. This function is a feature of Wikipedia that allows users to view any older version of the Wikipedia page, showing exactly what words were specifically added or changed at the time of each revision. The date of the first revision to contain a link to the MedWatch alert was counted as the date that the information was referenced. The date of the first revision to contain information from the MedWatch alert was counted as the date that the article was updated. The number of days between the publication of the MedWatch alert and the date that the information was referenced was calculated and recorded as the time to citation of MedWatch. The number of days between the publication of the MedWatch Alert and the date of reference update was calculated and recorded as the time-to-reference update.

Statistical analysis

To compare continuous variables between the databases, an unpaired t-test was used. Fisher’s exact or chi-squared tests were used, as appropriate for nominal variables. GraphPad® 2014 was used for all analyses.

RESULTS

Number and age of sources

Twenty-one drugs were included in the analysis: acetaminophen, ado-trastuzumab emtansine, alogliptin, azithromycin, cinacalcet, codeine, exenatide, ezogabine, hydroxyethyl starch, ketoconazole, linagliptin, liraglutide, magnesium sulfate, mefloquine, olanzapine, olmesartan, saxagliptin, sitagliptin, tolvaptan, valproic acid, and zolpidem. The total number of references for all drugs analyzed was 955: 601 references from the Wikipedia articles and 354 references from the Lexicomp monographs. On average, Wikipedia articles contained more reference sources than Lexicomp monographs (28.5 versus 16.9, respectively), but this difference was not statistically significant (P = 0.077). The average number of references was calculated by adding up the reference totals for each article or monograph examined and dividing this total number of references by the number of articles or monographs examined. The average age of the references used in Wikipedia articles did not differ significantly from that of Lexicomp (6.1 years versus 7.4 years, respectively, P = 0.336).

Time to reference update

MedWatch alerts were included in the reference lists of 5 of the 21 Wikipedia articles, and information from the alerts was present in 9 of the 21 Wikipedia articles. Other sources of MedWatch alert information included news articles (n = 2) and commercial websites (n = 1). One article had originally cited the MedWatch alert and included information from it in the article, but the citation was later removed (n = 1). Lexicomp monographs cited the MedWatch alerts and contained information from the MedWatch alerts in each instance (n = 21/21). Four of the 12 Wikipedia articles that were missing MedWatch alert information were missing critical information about new contraindications, precautions, or side effects that were being added to the FDA-approved drug labeling. The remaining articles were missing new early warning information about potential side effects (7 of 12) or information about the potential for “look-alike, sound-alike” name confusion (1 of 12).

When MedWatch alert citations were present in the reference lists of Wikipedia articles (n = 5), the average time between the publication of the MedWatch Alert and incorporation of the MedWatch citation into the Wikipedia reference list was 10 days, the time to update ranged from 0 days (ezogabine and tolvaptan) to 42 days (azithromycin). When MedWatch alert information was present in Wikipedia articles (n = 9), the average time between the publication of the MedWatch alert and incorporation of information from the alert into the Wikipedia article was 5.9 days (range 0–42 days). Comparable time-to-update information for Lexicomp could not be obtained.

Source accessibility

Most of the sources included in the reference lists of the drugs evaluated in Wikipedia (84.6%) and Lexicomp (84.7%) contained accessible hyperlinks to the original source. Wikipedia had a higher percentage of inaccessible or broken links than Lexicomp (6.7% vs. 0.3%, respectively).

Source types

A complete breakdown of source types for the evaluated drugs as cited in Wikipedia articles and Lexicomp monographs can be seen in Table 1. Overall, the 2 most common source types cited in Wikipedia were journal articles (50.1%) and commercial (.com) websites (11.1%). The most common source types used in Lexicomp monographs were journal articles (65.5%) and evidence-based guidelines (28.8%). Overall, Lexicomp was significantly more likely to cite journal articles (65.5% versus 50.1%, P = 0.0395) and evidence-based guidelines than Wikipedia (28.8% versus 0.8%, P = 0.0001). Lexicomp monographs also cited more Core Clinical Journal references than Wikipedia (38.8% versus 20.4%, P = 0.0053), although this difference was not statistically significant. Wikipedia articles cited significantly more news articles (10.5% versus 0, P = 0.0007), commercial websites (11% versus 0, P = 0.0007), and government websites (9.2 % versus 0.8%, P = 0.0185) than Lexicomp.

Table 1.

Source types by drug information resource

DISCUSSION

Overall, Wikipedia articles for the evaluated drugs tended to cite a wide variety of sources, with journal articles, commercial websites, news articles, and government websites being the most common sources cited. Wikipedia articles rarely cited evidence-based guidelines, which are generally considered to be important sources of drug information for clinical practitioners. Wikipedia’s heavier reliance on information from news articles and commercial websites compared to Lexicomp may be useful to providers who are interested in seeing the types of information that their patients are likely to see. News articles may better reflect what the public is reading. Of note, while Lexicomp cited more Core Clinical Journal references, the value of Core Clinical Journals has not been established and their use does not indicate more reliability.

When evaluating these two information resources, it is important to note the differences in the ways that they can be accessed and edited. Wikipedia is open to the public and may be edited at any time by anyone. In some cases, this can lead to increased accuracy, because special interest groups have the ability to search for and quickly edit out any factual errors. However, this can also result in inaccuracies, which may or may not be corrected in a timely manner. Lexicomp, on the other hand, requires subscribers to pay a yearly access fee, and monographs are written and revised by experts. There are advantages and disadvantages to both systems of information construction.

Source accessibility is important, because health care practitioners and students may use Wikipedia as a “jumping-off” point to look for more traditional references and information resources (for example, locating journal articles on a particular topic). The majority of sources cited in Wikipedia articles contained accessible links; thus, this may be a way for students and health-care professionals who use Wikipedia as a starting point to be directed to appropriate primary resources. However, it should be stressed that caution should be used when using Wikipedia at any point when searching for health or drug information. Information should be verified and the quality of the references sources assessed before using it to make any health care decisions for patients.

In this analysis, the presence of MedWatch information in the Wikipedia article was used to determine how quickly important medication safety information is added to the pages. Unfortunately, detailed time-of-update information was not available for Lexicomp, so no direct comparison could be made between the databases for this issue. When MedWatch information was present in Wikipedia articles, it was added to the Wikipedia page an average of 6 days after publication of the alert. However, the time to update varied widely (from 0 to 42 days), and more than half of the Wikipedia articles did not contain information from the MedWatch alerts at all. Two Wikipedia articles contained critical drug safety information that was incorrect or out of date: the codeine article cited an old MedWatch alert that recommended “cautious use” in tonsillectomy patients (rather than total avoidance), while the valproic acid article made no mention of the fact that the drug is now contraindicated for the treatment of migraines in pregnant women. This seems to indicate that Wikipedia is not reliably updated with recent safety information and thus should not be used as a drug safety information resource. In contrast, all the Lexicomp monographs examined in this study contained information from corresponding MedWatch alerts, as well as links to the alerts themselves. Thus, Lexicomp would be considered a better resource to use for keeping up-to-date on the latest medication safety information published by MedWatch. Alternatively, health care professionals can go directly to the source of information by visiting the free MedWatch website for detailed, up-to-date information on safety alerts. Individuals can also subscribe to email discussion lists, subscribe to really simple syndication (RSS) feeds, or follow the FDA on Twitter.

An important limitation that is inherent to this type of research is that Internet-based information resources are constantly changing and being updated; thus, the types of sources used and the speed or frequency of information update may change as time goes on. Additionally, it is important to keep in mind that source type is only a surrogate marker for article quality: the presence of a source in the citation list does not guarantee that the information from that source has been correctly interpreted and incorporated into the body of the article. This is particularly true for resources like Wikipedia, where anyone (medical professional or not) may contribute to the content of the articles.

In this study, the authors examined the speed with which Wikipedia is updated to include recent medication safety alerts and characterized the types of sources that are used to construct these Wikipedia articles. The results suggest that Wikipedia may not be the most up-to-date source of information for recent FDA MedWatch Safety alerts. Health care professionals should consider using Lexicomp or the FDA MedWatch website to obtain timely updates on drug safety-related issues.

Biography

Laura Koppen, PharmD, Candidate, Lkoppen52@midwestern.edu; Jennifer Phillips, PharmD, BCPS^*, (Corresponding author), jphillips@midwestern.edu, Assistant Professor, Pharmacy Practice; Renee Papageorgiou, PharmD, rpapag@midwestern.edu, Postgraduate Year 1 Pharmacy Resident; College of Pharmacy, Midwestern University Chicago, 555 Thirty-First Street, Downers Grove, IL 60515