Completeness, Accuracy and Presentation of Information on Interactions Between Prescription Drugs and Alternative Medicines: An Internet Review

Abstract

Background

As the use of the Internet continues to increase across all age groups and education levels, with usage in the US around 78%, consumers are increasingly turning to the Internet for health related information.

Objective

To assess the completeness, accuracy, and consumer friendliness of information on the Internet pertaining to drug-Complementary and Alternative Medicine (CAM) interactions with cardiac drugs.

Methods

A review of online information was performed across three search engines and ten drug-CAM pairs.

Results

Overall, the quality of the drug-CAM interaction information available online to consumers is fairly poor. Only one site contained an interaction checker that provided interaction information for all ten pairs, but with an accuracy rate of 50%. Reading levels ranged from 10.5–23.5, with a mean of 16.7. A value greater than 22 indicates a graduate level reading skill.

Conclusion

Web site developers should be cautious in presenting drug-CAM interaction information unless it is comprehensive and regularly maintained. Consumers should also know how to evaluate sites before trusting the content where the consequences are potentially severe.

Introduction

Complementary and Alternative Medicine (CAM) use is on the rise in the US and patients are becoming aware of potential interactions between prescription drugs and CAM through information provided by their physicians, pharmacists, and via the Internet. [1–6] When consumers want more information on healthcare, increasingly they are turning to the Internet. One study showed that 80% of Internet users have used it for healthcare purposes. [7] Similar behavior is expected when consumers are concerned about interactions between their prescription drugs and CAM. Therefore, there is cause for concern as to the quality of information consumers may be receiving via the Internet to make informed decisions.

In this study, we analyzed the quality of drug-CAM interaction information for consumers on Internet sites. CAM includes herbals such as St. John’s Wort, supplements such as Coenzyme Q10, and minerals such as magnesium. Although CAM is a more inclusive term to refer to these substances, other terms are more commonly used, such as “herbs” or “dietary supplements.”

Our research questions were: 1) How complete and accurate is the information found on the Internet? 2) How readable is the information? 3) How is the information presented? 4) Is scientific evidence available? 5) Does the site provide a recommended action?

Background

According to Medline Plus, the first approach to evaluating online health information is to consider the source. Is the site government sponsored, a university, a hospital, or a business? Is the information peer-reviewed? Is the site trying to sell you something? How current is the information?

Another means of instantly assessing the reliability of the information provided on a consumer health site is to verify if the site is certified by Health on the Net (HON). HON is a nonprofit, non-governmental organization accredited to the Economic and Social Council of the United States. The mission of the HON Foundation is to provide multi-stakeholder consensus to protect citizens from misleading health information. ¹

Increasingly, patients are turning to the Internet for health related information. [7] Due to the large volume of information available to healthcare consumers, there is evidence that consumers are becoming more proactively involved in the management of their own health. [8] For these reasons, as well as the fact that consumer content is for the most part unregulated, [9] it is becoming increasingly important that consumers understand how to assess the quality of the information they are receiving from online sources. One study on consumer information for Inflammatory Bowel Disease indicated that 57% of the 76 Web sites evaluated were of fair to poor quality. [10]

Studies addressing the quality of healthcare information online, in particular encouraging the use of CAM, have shown the potential for harm to patients. One study showed that 25% of the sites contained misleading or false information that could lead to direct harm to the consumer if acted upon, while 97% had omitted information. [11] Studies show that 78% of the information patients are receiving comes from commercial organizations, with 69% for the purpose of commerce and 52% had no references. [12] Those sites, although intended for consumers, most often contain language at a minimum 11^th grade reading level, which is considerably higher than the recommended 7^th grade level per the United States Department of Health and Human Services (USDHHS). Another common problem amongst these consumer health sites is invalid and omitted information.

Many studies exist pertaining to Internet use and the quality of online information for health specific topics such as online pharmacies and drug information, diabetes self-care, asthma, general health information, personal health records, HIV/AIDS, nutrition and exercise, clinical trials, and chronic disease treatments and options. However, very few studies exist assessing the quality of online drug-CAM interaction information and none of these studies evaluated sites that are intended for the consumer. One prior study evaluated drug-CAM interaction sites for use by physicians in answering questions asked by their patients. [13] That study took the information at face value, making no attempt to evaluate accuracy. Another study assessing the quality of CAM information for consumers reported similar results to this study, but did not address drug-CAM interactions. [12]

Materials and Methods

Drug-CAM interaction sites

To identify sites for this study, an Internet review was conducted using popular Web search engines and various search terms. To identify relevant sites, we used three search engines and search terms such as ‘drug-herb interactions’ and ‘drug-supplement interactions’. The search strings related to drug-CAM interactions listed at the bottom of the page in a Google search were used as a means of feeding the search term list used in the study.

According to search engine optimization (SEO) sites, which provide information on the usage of the various search engines, Google was the most popular, with Bing and Yahoo contending for second and third place. [14, 15] One of the studies showed Google at 72%, Yahoo at 14%, and Bing at 10%. Therefore, utilizing these three search engines for this study covered 96% of all Web searches. [16] The 2006 iProspect Search Engine User Behavior study showed that 62% of search engine users clicked on only sites found in the first page of results and 28% in the second and third pages. [16] Using that as a criterion for site selection, we remained within the first two pages of search results across all search engines and search terms.

All search terms were entered across the three search engines. The sites listed on the first or second page were analyzed. We only selected sites containing an interaction checker that provided the ability to enter either a drug or a supplement and view its corresponding drug-CAM interactions. Many sites were excluded because they provided a minimal hard-coded list of CAM or drugs and discussed common interactions, with little or no detail for the individual interactions. Searching for your own medication or CAM in those sites was not possible.

Reference standard

To assess the quality of the information presented on the sites identified, we selected a list of four medications commonly prescribed in the practice of cardiology. Those selected were Warfarin, Lipitor, Simvastatin, and Plavix.

As a reference standard, we searched the Natural Medicines Comprehensive Database (NMCD) to identify major and moderate interactions between the selected drugs and commonly used CAM. The major and moderate interactions were inspected and interactions with well-known and commonly used substances were selected. They included St. John’s Wort, Ginkgo Biloba, and grapefruit. Although this study did not cover interactions between drugs and foods, grapefruit was used as it has severe interactions with many cardiac medications and comes in a highly concentrated extract form for use as a dietary supplement.

A total of 10 drug-CAM pairs were entered into the NMCD and we purposefully selected 10 moderate or major interactions based on how common the interacting CAM was in the management of chronic disease. The interaction description (clinician version), consumer description, and severities were obtained.

Assessment criteria

Web sites were assessed with respect to five criteria: completeness and accuracy, readability, quality of presentation, scientific evidence, and recommended action

Completeness was measured by comparing the ten drug-CAM pairs in the reference standard with those available on each Web site. Because the drug-herb pairs were common and interactions were severe or moderate, it would be important that all 10 pairs were present.

Accuracy was measured by agreement between the sites evaluated and our reference standard regarding the severity of drug-CAM interactions. Five of seven sites evaluated provided interaction severity.

The presentation format was analyzed based on existing guidelines for presentation of information to consumers. Particular attention was paid to visual displays, because prior studies have shown that coloration and simple graphics are preferred to relay information to patients over textual descriptions. [17, 18] Coloration and icons are important to draw attention to critical information, with minimal textual descriptions to corroborate what the consumer believes the icon to be reporting.

To assess readability, the interaction descriptions, when found, were cut and pasted into read-able.com to evaluate the Flesch-Kincaid grade level. Although this tool works best with higher word counts, it is still considered the best tool for readability analysis. Its use is so common that it is bundled with the most common word processing software including Microsoft Word and WordPerfect. Our target age for consumer health sites is grade 7, which is the average reading level as identified by the USDHHS. [19] Anything beyond the 9^th grade reading level would be considered difficult per the USDHHS.

Results

Overall, seven Web sites were selected that met our inclusion criteria. Those sites were evaluated for completeness, accuracy, presentation, recommended action, and readability. Scientific evidence was excluded from our results, as this information was not included on any of the sites (see Table 1).

Table 1.

Site Comparison to Reference Standard

Site	Complete ness	Accuracy	Recommended Action
CVS	80%	6/8 (75%)	Y
Dr. Oz	80%	N/A*	N
Medline Plus	50%	3/5 (60%)	Y
Drugs.com	60%	1/6 (17%)	Y
Vitamin Herb University	50%	N/A*	Y
Healthline.com	70%	5/7 (71%)	Y
Reference.medscape.com	100%	5/10 (50%)	N

^*Unable to evaluate due to missing severity

The interaction descriptions varied significantly across sites. Some sites indicated there was an interaction, but provided very little detail. Other sites would go into great detail, including describing the effect on drug metabolism via the particular CYP450 subclass. This level of detail in most cases would provide no value to the consumer and is likely intended for professionals.

The presentation styles also varied widely, with some sites making generous use of coloration and graphics, while others provided only text (see Table 2).

Table 2.

Presentation Style and Update frequency

Site	Search Style	Coloration and Graphics	Interaction Result Display	Last Update	Flesch-Kincaid Reading Level*
CVS	Type drug and supplement name	No	Bullet list with description and severity	9/4/01	N/A
Dr. Oz	Alphabetic list of supplements	No	Bullet list of drugs classes, no severity	Unknown	10.51
Medline Plus	Alphabetic list of supplements	No	List of drug classes and CYP450 substrates categorized by severity	07/14/12	23.55
Drugs.com	Type either drug or supplement, auto-suggest	Both	Detailed description, consumer or professional, severity presented in traffic sign icons	10/15/12	17.73
Vitamin Herb University	Select supplement from dropdown list	Both	“Caution” followed by interactions list by drug class, no severity	Unknown	17.46
Healthline.com	Type with auto-suggest or select from list, drug and supplement	Both	Severity, meter, coloration for specific drug-CAM pair	Unknown	14.12
Reference.medscape.com	Type with auto-suggest	Coloration	Severity colorized, short description for specific drug-CAM pair	Unknown	19.40

^*Scores over 22 should generally be taken to mean graduate level text.

Discussion

This study sought to evaluate drug-CAM interaction Web sites based on completeness, accuracy, readability from the consumer perspective, and presentation. According to our evaluation criteria, the overall quality was fairly low.

None of the sites evaluated scored high on the combined criteria of completeness and accuracy. Only one of the sites had 100% coverage, although that site had a low accuracy rate of 50%. The other sites had lower completeness and accuracy scores. Two sites provided no severity information; therefore accuracy could not be evaluated.

The result of incomplete and inaccurate information on any given drug-CAM pair could be serious. The lack of interaction information could imply to the consumer that no interaction exists. There is also a risk when the severity is inaccurate, since patients may react differently depending on the level of interaction severity.

The best score achieved from a readability perspective indicated a reading level of approximately grade 11. The scores increased substantially from there, with the highest score indicating graduate level text. Therefore, most of the sites evaluated are written in a language that is inadequate for the majority of the population.

The presentation styles of these sites varied greatly, with some sites using no icons or coloration at all. Prior studies in healthcare and other industries provide several criteria for the presentation of information to consumers, such as simplified and uncluttered user interface, the use of coloration and icons, and minimal text. [17, 18]

None of the sites evaluated would pass HON certification, failing on multiple criteria, such as authority and attribution. The sites did not provide information on who researched the interactions or what studies or trials were used to determine the interaction and severity.

Prior studies show that the predictors of content reliability include the display of the HON logo, having an organization (.org) domain, and citing references. The absence of financial interest is also associated with content accuracy. [20, 21]

Scientific evidence was not included in any of the sites other than our reference standard. This may be acceptable for a consumer site, as many consumers would not understand the evidentiary support provided by scientific studies.

Limitations

This study had four main limitations. First, only seven interaction checkers were analyzed. Yet, we comprehensively searched the Web for eligible sites using several search terms and search engines. Several sites were excluded because they did not meet the definition of an interaction checker as defined for this study. To be considered, the consumer must have the ability to enter either the drug or CAM from the 10 drug-CAM pairs used in the study. They often had hard-coded lists of drug-CAM combinations, or simply provided a link to another site.

Second, the use of SEO in considering the sites is challenging, as this is something that changes continuously and affects a sites ranking in the search order. Some sites that appeared on page one early in our research may have moved beyond page two and consequently would not have been found at a later point in time.

Third, only ten drug-CAM combinations were used in the evaluation of the sites and all but one reported a major interaction in our reference standard. It may be helpful to include common CAM with moderate and minor interactions. In addition, the drugs used were not carefully considered to ensure that a comprehensive set of drug classes was included. Nevertheless, the drugs used in the study are frequently used for a set of prevalent chronic conditions. Therefore, drug-CAM interactions sites were expected to provide complete and accurate information on these drugs.

Finally, we utilized the Flesch-Kincaid readability assessment tool to determine the grade level for the interaction description. This tool works best with documents that have a higher number of words than is typical for an interaction description, so the reported reading age may be less accurate than expected. Nevertheless, Flesch-Kincaid is a widely used tool and has been previously applied to assess the readability of consumer health information. [19]

Future Work

A study involving more drug-CAM pairs would be valuable, including a representative sample of commonly used drugs in general.

While there is some guidance available for the presentation of consumer health information on Web sites, further studies could be conducted to determine the optimal presentation of drug-CAM interaction information to consumers to encourage communication with physicians and minimize unintended consequences.

Studies are needed to evaluate the impact of consumer health sites on the attitudes, knowledge, and behavior of healthcare consumers and whether these sites provide the intended value to the consumer.

Finally, a study focusing solely on the usability of these sites would be beneficial. There is no consistency across these sites as far as the entry of a drug name, CAM or both. Many sites use drug classes, which are often meaningless to consumers, or brand names where generics are more common these days due to insurance restrictions.

Conclusion

With the increasing use of complementary and alternative medicine by the US population, the availability of high quality online information on drug-CAM interactions is critical. However, this study shows that the overall quality of Web sites dedicated to providing drug-CAM interaction information is inadequate to the consumer.

Given the results of our study, we provide the following recommendations to Web site developers and consumers; 1) Careful consideration should be given to the ramifications of incomplete, inaccurate, and out of date information; 2) consumers should be educated on how to assess the reliability of health related content; 3) all Web sites containing drug-CAM information should warn consumers that they should consult their physician before starting or stopping any medications, including CAM.