Identifying Unpredicted Drug Benefit through Query of Patient Experiential Knowledge: A Proof of Concept Web‐Based System

Abstract

Information necessary to recognize unexpected drug efficacy is not routinely collected. Once a drug is approved, opportunities for understanding these phenomena are usually lost within clinical care. We propose that patients are willing to provide a wide range of experiential knowledge about the effects of therapies that is seldom solicited. Experience with various drug therapies might be solicited directly from patients in both structured and unstructured formats. Although the signal to noise ratio is expected to be low, these data, if organized in a constructive manner, could provide a useful hypothesis generation resource for areas of further pharmacologic inquiry. A pilot study was conducted for 18 months; 1,065 individuals using the MyHealthAtVanderbilt.com patient portal clicked on a research link to find more information about the study; 375 completed the survey (response rate of 37%). Of those, 218 patients reported that they were currently taking at least one prescription. Statistical methods applied detected known associations between drugs and their intended effects. This validated the type and frequency of effects being reported by patients and provided evidence for the potential for using patient‐supplied information to generate hypotheses related to unexpected positive benefits associated with medications. Improved data filtering and mining methods will be needed to expand this concept. Clin Trans Sci 2010; Volume 3: 98–103

Introduction

Eighty‐two percent of adults in the United States take at least one medication, including prescription and nonprescription drugs, vitamins, and supplements. ¹ Considerable attention is given to understanding and predicting the negative side effects and outcomes of these medications. Given the large number of prescription drug therapies developed and approved for a single indication, it seems biologically improbable that all previously unknown drug‐response events would be negative. Unexpected efficacy could be a more common phenomenon than realized.

Information necessary to recognize and study unexpected drug efficacy is not routinely collected. First, it is difficult to collect observations of unexpected efficacy during the drug development and testing process. FDA regulated clinical trials are designed to establish the safety, efficacy and security of a drug for a specific indication, while assessing rare and common adverse events. However, the range of potential positive physiological and psychological effects is so large it precludes systematic collection of such data in trials and is not required by regulation. Furthermore, the nature of the drug development process aimed at FDA marketing approval likely discourages capture of observations not relevant to the disease or condition for which a drug trial is being conducted. Finally, “Phase 4” surveillance is often focused particularly on safety; such trials are designed in part to identify and quantify rare and unanticipated negative events. Some have even proposed that there are other “anti‐serendipity” factors in play related to drug discovery and pharmacologic exploration. ² Once a drug is approved, opportunities for understanding these phenomena are usually lost within clinical care, as only a subset of information shared by the patient is typically entered into the health record. Providers commonly inquire about side effects patients are experiencing, but the perceived benefits of drugs unrelated to the condition for which the drug was prescribed, even if communicated to providers, are not routinely recorded (and may be attributed to a placebo effect).

These factors notwithstanding, there are a number of examples of new uses and/or indications that have been discovered serendipitously during development or after drugs were introduced into routine clinical practice (sometimes years af er drugs are initially marketed). Among the most widely known are the well‐established cardiovascular benefits of aspirin. Observations in clinical trials shifted the development of sildenafil from coronary artery disease towards erectile dysfunction. Bupropion was developed as an anti‐depressant but then showed success in smoking cessation (now Zyban, GlaxoSmithKline). Botox (onabotulinumtoxinA, Allergan) was being used for eye‐muscle disorders when the cosmetic benefits became apparent. Minoxidil was used to treat high blood pressure before its effects in hair growth were known. ³ There are other examples, sometimes reported as a set of case reports, such as rituximab efficacy in nephritic syndrome. ⁴ In most, if not all of these examples, “chance” was a strong factor in the ultimate outcome. Patients’ experiential knowledge can also lead to novel hypotheses about drug effects. ⁵

Patients are increasingly aware and engaged in health care and in their personal health management. A 2009 Survey of Health Care Consumers ⁶ shows that 57% of health care consumers want a “secure internet site that would enable them to access their medical records, schedule office visits, refill prescriptions and pay medical bills.” Current generation patient portals maintained by hospitals and health systems contain functions to facilitate patient and provider communications and the availability of patient‐specific clinical results, but these portals are generally not used to enable patients to enter their own observations about their health that become part of their formal record. Patient‐oriented websites (e.g., patientslikeme.com) capture patient entered information concerning efficacy of therapy, among other experiences; however, these sites also focus on efficacy based only on the primary indication (i.e., the provider's treatment intention).

In summary, there are very few mechanisms or methods available to engage and collect information from patients related to unexpected benefits of medications. This information is currently not collected systematically in drug development processes, not recorded in the EMR or patient chart, and not captured routinely in patient‐driven portals. We propose that patients have and are willing to provide a wide range of experiential knowledge about the effects of therapies that is seldom solicited. Experience with various drug therapies might be solicited directly from patients in both structured and unstructured formats. Although the signal to noise ratio is expected to be low, we believe these data, if aggregated, organized, and displayed in a constructive manner for informed researchers, could provide a useful hypothesis generation resource for areas of further pharmacologic inquiry.

Methods

Survey overview

Vanderbilt has developed and deployed a state‐of‐the‐art web‐based patient portal (My Health at Vanderbilt, http://myhealthatvanderbilt.com) that enables secure communication between patients and their providers, allows patients to access personal test results from their medical records, and provides a mechanism for patients to learn about their specific health conditions. ⁷ There are currently over 80,000 registrants, and more than 2,500 unique users of the portal daily. This patient portal has created new avenues of patient interaction and data capture. We have focused on collecting information that is not currently recorded in the electronic medical record system; the study described here leverages the existing portal. The remainder of this paper discusses a survey of patients’ experience with medications conducted specifically via the established MyHealthAtVanderbilt patient portal (http://tinyurl.com/2bbws8k). The survey and all methods were reviewed and approved by the Vanderbilt Institutional Review Board.

A pilot web‐based data capture system was developed to enable patients using the MyHealth portal to anonymously report experience with drugs prescribed by Vanderbilt providers. During the patient login authentication process, the MyHealth system sends a request to a custom web service, which queries patient‐specific drug data in our EMR. If the patient is known to have been prescribed drugs from Vanderbilt and they have not yet participated in the drug survey project, the web service sends an invitation link back to the MyHealth system for display on the patient's MyHealth home page. The link presents potential participants with a secure, anonymous survey where they can verify drug use and associated experiences.

Survey respondents first confirm which drugs they are currently taking from an automatically generated list derived from their Vanderbilt electronic medical record. If at least one drug is verified by the user, an exploratory 10‐minute survey (nonvalidated) is conducted. For each current drug, participants are asked a series of more detailed questions about the effects of that drug in five domains: well‐being/mood, how the body feels/physical abilities, daily life, energy level, and mental/emotional state. The intention of collecting data by prompting across various domains is to encourage individuals to report on all unexpected effects, particularly effects that are not related to the condition for which it was prescribed. For each domain, survey participants are asked to quantify their experiences while taking the drug using a visual analog scale (a “slider bar”) included within the survey interface (e.g., “How much [left‐hand scale label =“worse”; right‐hand scale label =“better”], with a hidden range integer mapping range equal to 0–100). Whenever a strong negative or positive unexpected drug effect is signified through slider placement (translating to numeric scores of <33 or >66), participants are asked to provide a narrative description of how the drug affected them in this particular domain.

Coding responses

The verbatim narrative descriptions were evaluated to develop a list of response categories. These categories were organized empirically into a coding framework that was used to manually assign each patient‐provided description. These were also coded as either a primary or a secondary effect. For example if a respondent stated that the drug oxazepam, “relaxes me in stressful situations that used to trigger migraines so I have fewer migraines” then the primary effect would be [better able to handle stress] and the secondary effect would be [fewer migraines]. A structured list of responses was not provided to respondents within the survey since (1) there was no pre‐existing structured listing to base this framework on and (2) the range of answers given to an open‐ended verbatim about how a drug affects a patient was both large and unpredictable. The data captured as a result of the pilot showed that these responses follow a language pattern of primary effects leading to secondary effects, and could therefore be readily grouped according to the nature of the effect being described. All responses related to lifestyle or quality of life improvements resultant to a primary effect (e.g., because of [fewer migraines] I am able to [do more housework]) were removed from analysis. A single individual, blinded to the drug of interest, performed the coding.

Data analysis

The dataset of recorded medication and positive effects can be thought of as a large and sparse contingency table where the M rows correspond to medications and the E columns correspond to positive effects. To examine a medication (m)‐ effect (e) pair, the M by E contingency table can be collapsed into a 2 × 2 table as shown in Table 1 . The (m,e) cell contains the number of participants reporting effect e associated with medication m. The m–e pairs that occur at unusually or disproportionately high rates relative to that which would be expected by chance may warrant further investigation for the possible serendipitous results.

Table 1.

Two by two contingency table for calculations of the RRR_m,e.

	Event e reported	Event e not reported
Medication m reported	n e,m	n m−n e,m	n m
Medication m not reported	n e−n e,m	n−n e−n m+n e,m	n−n m
	n e	n−n e	n

The Relative Reporting Ratio (RRR) ⁸ for the m–e pair is given by RRR _m,e= (n*n _m,e)/(n _m*n _e). This value captures the observed count in the (m,e) cell n _m,e divided by the expected count, e _m,e, under the assumption that reports of medications and effects are independent (e _m,e=n _m*n _e/n). Values greater than one imply higher than expected reporting rates. While the RRR is an intuitively clear measure of the excessive reporting, with sparse contingency tables, and therefore small expected cell counts, the uncertainty associated with it can be enormous. For example, a RRR equal to 10 is, and should be, given much more weight if the n _m,e is 100 (expected cell count is 10) than if it is 1 (expected cell count is 0.1). To acknowledge the small cell counts and therefore the lack of precision associated with estimating RRR_m,e, one can apply the Multi‐item Gamma Poisson Shrinkage ⁹ (MGPS). MGPS has been described and implemented on numerous occasions to mine the Food and Drug Administration's Adverse Event Reporting System ¹⁰ , ¹¹ , ¹² , ¹³ It uses empirical Bayes methodology to shrink estimates of the RRR _m,e towards a value of 1. Smaller expected cell counts result in larger shrinkage factors. Briefly, MGPS assumes the (m,e) cell count, n _m,e, follows a Poisson distribution with mean μ_m,e. The value Λ_m,e., defined as μ_m,e divided by e _m,e is a measure of disproportionate or excessive reporting (e.g., the rate of reporting for the m–e pair divided by the rate under the assumption of independent m–e reporting). The prior distribution for Λ_m,e is assumed to follow mixture of two gamma distributions, and the expected value of the posterior distribution for Λ_m,e is known as the Empirical Bayes Geometric Mean (EBGM _m,e) estimator. It is a shrunken estimate of the RRR_m,e. If the fifth percentile of the posterior distribution of Λ_m,e is greater than one or if the posterior probability that the null hypothesis (i.e., RRR _m,e= 1) is true is less than 0.05, then the m–e pair occurs at a rate that may be considered significantly higher than would be expected under m–e independence. In analyses, we report RRR _m,e, EBGM_m,e and the posterior probability of the null hypothesis. Although not discussed here, other measures such as those derived from Bayesian Confidence Propogation Neural Networks ¹⁴ may also be applied to these data. Analyses were conducted using the R programming language ¹⁵ and EBGM estimates were computed with the PhViD: Pharmacovigilance Signal Detection Methods Extended to the Multiple Comparison Setting package. ¹⁶

Results

A pilot study was conducted for approximately 18 months until November 2009. A total of 1065 individuals using the MyHealthAtVanderbilt.com patient portal clicked on the research survey link to find more information about the study. Of those individuals, 375 completed the survey for a response rate of 37%.

Of those patients completing the survey, 218 patients also reported that they were currently taking at least one prescription drug and completed the survey components asking about their experiences with the drug(s). Among these 218 individuals, there were 73 drugs represented and catalogued for analysis. Due to the small sample size, we report positive effects associated with therapeutic classes (n= 35) as opposed to the individual medications. However, for consistency of terminology we refer to these therapeutic classes as medications. Table 2 provides demographic information for all individuals completing the survey compared to the Vanderbilt Medical Center population.

Table 2.

Demographics.

Demographics	Completed survey and taking at least one drug	Total completed survey	Vanderbilt Medical Center
Age	N= 218	N= 375
18–24	4%	3%	11%
25–34	22%	22%	15%
35–44	22%	22%	16%
45–54	28%	29%	17%
55–64	18%	19%	15%
65+	5%	4%	26%
Gender
Male	26%	25%	45%
Female	74%	75%	55%
Race
Caucasian	76%	76%	81%
African American	6%	6%	14%
Not specified/other	19%	18%	5%

When asked about those drugs reported as being currently taken by respondents, 75% were reported to have only the effects for which the drug was prescribed while 10% were reported as having had no effect. In these cases, survey participants were not asked for answers to the full panel of questions because this study was focused solely on determining the feasibility of acquiring and analyzing unexpected drug effects as perceived by patients. The remainder of drugs, approximately 16%, were reported by patients as having unexpected positive or negative effects. Importantly, 27% of individuals reporting unexpected effects indicated that they had not shared these experiences with their healthcare provider.

Average numeric scores were calculated (See Methods: Survey Overview) across each of 5 domains, representing how the drug made the respondent feel on each domain. Average scores are shown in Figure 1 for those drug classes with more than one individual responding.

Figure 1.

Average scores on a numeric (1–100) scale for drug classes with more than one individual responding, by domain. These data are intended to be illustrative of the types of scores provided by respondents, and the variability across domains for each drug class. These data are qualitative, given extremely low sample sizes, but show that in these cases, drugs might have perceived effects on patients beyond those for which they are prescribed.

We focused analyses for this pilot study on elucidating unexpected self‐reported positive effects. Eighty‐two (82) participants reported 28 unique positive effects and 35 therapeutic classes of medications. In total, 135 medication + associated positive effects were reported. Forty‐five participants reported one medication–effect, 25 reported two, five reported three, two reported four, and three reported five. Table 3 shows the 10 medication–effect pairs with the largest EBGM_m,e estimates (See Methods: Data Analysis). The largest of these values corresponded to the “antihistamines—decreased allergy symptoms” pair with an EBGM equal to 1.72 and a posterior null hypothesis probability equal to 0.03. The RRR (see Data Analysis section) for this pair was 20.25. Notice that due to data sparsity, the EBGM was less than one‐tenth of the value of the RRR. The smallest posterior null hypothesis probability (0.006) corresponded to the “narcotic analgesic—physical pain relief” pair. While the RRR for this pair was only 4.2, due to relatively large expected cell count (1.67) a much smaller shrinkage factor was applied to obtain the EBGM (1.53). As can be seen from Table 3 , posterior null hypothesis probabilities and EBGM estimates do not necessarily rank‐order medication‐effect pairs consistently. They and other indices (e.g., fifth percentile of the posterior Λ_m,e distribution) capture different measures of evidential strength and we recommend their usage in combination. Even though other medication effect pairs also exhibited weak evidence for disproportionate reporting, none of these associations would merit serious consideration. In analyses of adverse event reporting systems, fifth percentiles of the posterior distribution of Λ_m,e falling below two are generally not large enough to warrant further investigation. Our limited findings demonstrate the potential for using free‐form patient‐reported data to elucidate signal concerning unexpected drug effects. While the associations seen here are well known, it was encouraging to find that our methods could find a signal from self‐reported unstructured information even in a small sample.

Table 3.

Medication—effect pairs with the 10 highest EBGM estimates.

Medication	Effect	nm,e	nm	ne	Cm,e	RRRm,e	EBGMm,e	Posterior null hypothesis probability
Antihistamine	Decreased allergy symptoms	3	5	4	0.15	20.25	1.72	0.030
Narcotic analgesic	Physical pain relief	7	9	25	1.67	4.20	1.53	0.006
Hypoglycemic	Helps control blood sugar	2	2	4	0.06	33.75	1.30	0.101
Contraceptive devices	Decreases menstrual period/PMS symptoms	2	3	3	0.07	30.00	1.29	0.103
Antidepressant	Controls emotions/decreases volatility/moodiness	7	29	10	2.15	3.26	1.28	0.017
CNS stimulant	Increases energy level/alertness	6	10	25	1.85	3.24	1.21	0.029
Nonbarbiturate Hypnotic	Increased ability to sleep	2	3	6	0.13	15.00	1.18	0.12
Vitamin supplement	Increases energy level/ alertness	5	8	25	1.48	3.38	1.17	0.045
Antianxiety	Decreases stress/anxiety	2	2	10	0.15	13.50	1.16	0.124
Herbal supplement	Helps control blood sugar	2	5	4	0.15	13.50	1.16	0.124

n _m,e: Number of medication m and positive effect e reports.

n _m: Number of medication m reports.

n _e: Number of effect e reports.

e _m,e: Expected number of medication m and positive effect e reports if reported independently.

RRR_m,e: Relative Reporting Ratio for the medication m and positive effect e pair.

EBGM_m,e: Empirical Bayes Geometric Mean of the posterior distribution of Λm,e.

Discussion

This project provides evidence for the potential for using patient‐supplied information to help elucidate possible unexpected positive benefits associated with medications. Specifically, the pilot data suggest that: (1) respondents acknowledged that this information was collected anonymously for research purposes, and did not expect reported information to become part of their clinical record, (2) respondents are willing and able to provide structured and unstructured feedback on their experiences with prescribed therapies, (3) patient‐provided information reported via this mechanism is likely not collected in the EMR, (4) patients will provide information concerning drug‐related benefits independent of the prescribed reasons for drug administration, (5) self‐reported effects can be organized and analyzed in a way such that signals may inform research study hypothesis generation, and (6) supplemental qualitative information and textual responses provide additional usefulness for assessing of potential signals.

These results are reported to illustrate the analytic framework that would be employed. In this preliminary dataset, all statistical associations were for known effects. This serves as a validation of the type and frequency of effects being reported by patients. As the volume of these types of data grows, in order to observe unexpected relationships it will be necessary to apply methods to systematically remove known associations. The total population of respondents in this pilot study was low, and this predictably resulted in a sparse contingency table. In addition, the ability to assess the credibility of the signal from the information given is limited. However, in addition to data analysis using statistical frequency methods af er structured coding, clinicians were also asked to review free‐text reported experiences and provide commentary concerning a plausible explanation of reported unexpected effects. As shown in Table 4 these “signals” obtained from a manual review of the data by clinicians generated observations that assist in interpretation of the data. These manual reviews might not be feasible with a larger dataset.

Table 4.

Verbatim excerpts from patient respondents and clinical pharmacologic observations.

Example of a positive effect description (excerpted)	Drug	Clinical pharmacologic commentary
The Effexor seems to have an antihistamine effect and decreases my allergy symptoms. Decreases itchy eyes, sneezing, wheezing.	Venlafaxine (Effexor; Wyeth Pharmaceuticals, Collegeville, PA, now Pfizer)	Anticholinergic and antihistaminic effects may occur with venlafaxine (Effexor) and that may account for the reduced itching described by this patient.
I feel much more energetic when taking this medicine. I am not nearly as run down as when I do not take it.	Triamterene	This response could be a real response. However, it could also be that the patient had low potassium and felt somewhat fatigued as a result. In that case administration of a potassium‐sparing drug would have corrected that potassium deficiency and made the patient feel better.
The panic attacks have almost disappeared completely. Anxiety is controllable.	Loratadine	This response could be a real response. However, mast cell activation disorder is more common than formerly believed. In such individuals histamine contributes to the spells these individuals experience that are sometimes referred to as hyperventilation/ panic attacks. Treatment with an antihistamine (loratidine) would possibly mitigate the stimulus for the hyperventilation, and its attendant sense of panic.

A consideration for this model is that patients using the MyHealth patient portal may not be representative of Vanderbilt patient populations as a whole; it is clear they are distinctly younger than the Vanderbilt patient population ( Table 2 ). MyHealth portal users tend to be more engaged and involved in their health care than the general population and also might be more comfortable using information technology. Larger survey populations will be required for the resource to become a hypothesis generation engine. Verbatim responses are highly important to discerning signals identified, and a coding schema does not currently exist for this work that would encompass the full range of expected responses. As this was a pilot study, a single individual performed the coding, thus there was no validation or inter‐rater reliability assessed. Further, Natural Language Processing methods would be needed to manage large amounts of patient narratives that might be generated.

Conclusions

Academic health centers often serve large populations with diverse diseases, and thus have the potential to act as real‐world platforms for harnessing patient knowledge regarding drug effects that are not routinely captured in the course of care delivery. The system and approach presented here is neutral to both disease and drug type and can therefore meet the needs of a diverse group of investigators, potentially focusing on any drug or drug class currently approved for treatment. The necessary next steps for the work described will be to engage other sites for enhanced sample size. Pharmacy/pharmacology and informatics experts will develop tools to assist in the aggregation, organization, synthesis and display of information for investigators seeking to generate new hypotheses using patient‐reported observations. The main objective ultimately will be to provide tools for research teams to rapidly generate new drug‐related hypotheses, and we believe a strong potential exists to accelerate the translation of research leveraging existing drugs on the market and what is already known about their pharmacodynamic and pharmacokinetic properties. Importantly, studies leveraging this resource might be able to capitalize on information already known about drug metabolism and safety. As such, we believe that this project represents a unique, systematic interaction with a community of patients, and therefore a testable mechanism for reducing the translational barrier of feedback from clinical practice back into research using a novel source of information. With considerable methodological development, a future community participation‐based resource could ultimately help researchers identify hypotheses regarding existing drugs that may lead to new indications other than those for which they are initially licensed and/or currently prescribed.