What do patients learn about psychotropic medications on the web? A natural language processing study

Abstract

Background:

Low rates of medication adherence remain a major challenge across psychiatry. In part, this likely reflects patient concerns about safety and adverse effects, accurate or otherwise. We therefore sought to characterize online information about common psychiatric medications in terms of positive and negative sentiment.

Methods:

We applied a natural language processing tool to score the sentiment expressed in web search results for 51 psychotropic medications across 3 drug classes (antidepressants, antipsychotics, and mood stabilizers), as a means of seeing if articles referencing these medications were generally positive or generally negative in tone. We compared between medications of the same class, and across medication classes.

Results:

Across 12,733 web search results, significant within-class differences in positive (antidepressants: F(24,2682)=2.97, p<0.001; antipsychotics: F(16,4029)=3.25, p<0.001; mood stabilizers: F(8,2371)=6.88, p <0.001) and negative sentiment (antidepressants: F(24,6282)=11.17, p<0.001; antipsychotics: F(16, 4029) =12.13, p<0.001; mood stabilizers: F(8, 2371)=13.28, p<0.001) were identified. Among these were significantly greater negative sentiment for the antidepressants sertraline, duloxetine, venlafaxine, and paroxetine, and for the antipsychotics, quetiapine and risperidone. Conversely, lithium preparations and valproate exhibited less negative sentiment than other mood stabilizing medications.

Limitations:

While these results provide a novel means of comparing medications, the present analyses cannot be linked to individual patient consumption of this information, or to its influence on their future clinical interactions.

Conclusions:

Overall, a subset of psychotropic medications were associated with significantly more negative sentiment. Characterizing these differences may allow clinicians to anticipate patient willingness to initiate or continue medications.

Introduction:

The availability of health information online impacts the modern patient-doctor relationship, with evidence that self-education leads some patients to take greater ownership of their care.(Hartzband and Groopman, 2010) In 2012, 72% of Internet users looked for health information online.(Fox and Duggan, 2013; Kurup, 2010) Internet information arises from a wide variety of sources with differing perspectives, areas of interest, degrees of medical knowledge, and extent of lived experience. Understanding this content may allow physicians to better engage patients by anticipating how it may influence their decision-making during, and between, clinical encounters.(Fox, 2011; Fox and Rainie, 2002) The treatment process provides many moments in which patients might search the web for others’ opinions; for example, after receiving a new prescription, a patient might research the medication before filling the prescription or taking the medication. Although online information is important, it is impractical for any provider to survey the depth and breadth of material available for every possible intervention or to know, on average, what a patient might find for a given medication. To fill this gap in knowledge, we sought to summarize the web-based resources patients are likely to encounter when searching the Internet for psychotropic medications.

To develop a summary of Web perspective, we applied sentiment analysis to a corpus of web pages for each of 51 psychotropic medications across 3 drug classes: antidepressants, antipsychotics, and mood stabilizers. Sentiment analysis, or opinion mining, is a form of natural language processing that computes the subjective valence of free text.(Gilman, 1968; Pang et al., 2002; Pang and Lee, 2005) Sentiment analysis has previously been applied to web-based corpora of health relevance to quantify such topics as patient opinion of health systems, frequency of substance use, and perspectives of cancer survivors.(Greaves et al., 2013; Myslín et al., 2013; Portier et al., 2013) This approach has also been applied to documents authored by medical providers as a means of predicting health outcomes.(McCoy Jr et al., 2015a, 2015b, 2016) Here, we aimed to summarize the range of medication-relevant sentiment captured in web pages identified through web searches; the results should provide clinicians with a summary perspective on the sentiment a patient is most likely to encounter when searching the Internet for information about a medication.

Methods:

Medication Corpus Generation and Text Pre-Processing:

Starting from a curated list of modern neuropsychiatric medications developed by consensus of two academic psychopharmacologists (THM, RHP), including both generic and trade names (when relevant), we performed programmatic searches of the Internet via the Microsoft Bing News and Web search API in English. (Of note, these lists included medications often prescribed off-label, including anticonvulsants such as gabapentin that are sometimes prescribed in this context despite a paucity of efficacy data). We then crawled web search results, downloading the resulting web pages in an approach which builds on our prior work developing medical concept relevant web corpora in a fashion which accounts for more modern client side page rendering.(McCoy Jr et al., 2015b; Salunke, 2014) To extract the content from the downloaded web pages while excluding navigational links and headers, we used the Apache Software Foundation Tika implementation of the Boilerpipe algorithm.(Kohlschutter et al., 2010; Mattmann and Zitting, 2011) This approach to web corpus generation from a set of predefined starting terms follows prior work on web-based corpora.(McCoy Jr et al., 2015b)

To ensure that the content of each web page was of sufficient length for meaningful analysis and yet of plausible length for consumer use, we excluded a priori web pages that were shorter than 100 words (n=1,065; 5.8%) and greater than 30,000 words (n=90; 0.5%) based on visual inspection of page length distributions. To allow for unbiased comparisons, web pages which occurred in the search results for multiple medications were excluded from analysis. When applying this exclusion, the trade and generic name for a medication were considered one medication. In other words, a page that occurred in the both the Prozac and the fluoxetine search result set was retained, whereas a page that occurred in both the Prozac/fluoxetine and Abilify/aripiprazole search result set was excluded. To balance sample sizes after exclusion, and because not all terms (trade and generic medication names) resulted in the same number of search results from the Bing search APIs, we determined prior to analysis the smallest number of web results (n=119) for any trade or generic medication name, and only included the first 119 results for each medication.

Measurements:

For each web page, the cleaned text was scored for sentiment using the Python implementation of Valence Aware Dictionary and sentiment Reasoner (VADER).(Hutto and Gilbert, 2014a, 2014b) VADER is a general-purpose sentiment analysis library based on a curate lexicon of sentiments. For example, words like “rape”, “slavery” and “kill” are among the most negative tokens whereas words like “magnificently”, “euphoria”, and “sweetheart” are among the most positive tokens. VADER scoring results in three scores of interest for each web page: positive sentiment, negative sentiment, and compound sentiment. Positive and negative sentiment scores represent the proportion of words within the web page that were either positive or negative. As in our prior work, we examined positive and negative sentiment separately, as pages could express both positive and negative sentiment.(McCoy Jr et al., 2015a, 2016) To assess the overall sentiment of a given web page, we also used the VADER compound sentiment score. The compound sentiment score is calculated by weighting each word in the web page according to five rules that change the intensity of the word (e.g., punctuation, word order, or negation), summing the adjusted values of each word, and then normalizing the scores to be between −1 (most negative) and +1 (most positive). Consistent with prior applications of the VADER model, all web pages with a compound score less than −0.05 were classified as a negative, and all web pages with a compound score greater than 0.05 were classified as a positive.(Hutto and Gilbert, 2014a, 2014b)

Analysis:

Recognizing that sentiment is composed of positive and negative dimensions, and that patients may selectively attend to one specific dimension during their web search (i.e., patients may be specifically concerned about the negative aspects of a medication), we used one-way analyses of covariance to compare the average negative and positive sentiment between medications of the same class. In post hoc comparisons, we compared the average negative sentiment and positive sentiment of a medication to the average sentiment of all other medications of the same class using a two-sample t-test with Bonferroni correction for multiple comparisons. That is, we identified medications with average sentiment differing significantly from others in the same class. Analyses were done at the pharmacologic class level envisioning that patients might most often compare similar treatment alternatives. Classes were defined a priori based on clinical context rather than mechanisms; they included antipsychotics, antidepressants, and mood stabilizers. This classification approach was selected as it is conventional within clinical practice; however, alternative classifications, including those based on underlying neuroscience, are possible (Zohar et al., 2015).

Next, we tested how likely a patient would be to find a net negative, instead of a net positive, web result operationalized as the document compound sentiment score. This analysis imagines the patient clicks on any of the links resulting from a web search and estimates the probability that it will have predominantly negative content. To complete this analysis, we used the VADER compound score and calculated the proportion of net negative web pages for each medication and compared the proportion of negative web pages between medications of the same class using a Chi-Square test. In post-hoc comparisons, we compared the proportion of net negative web pages for a medication to the aggregated proportion of net negative web pages for all other medications of the same class.

Finally, we conducted two broader sets of analyses. We compared positive and negative sentiment scores between classes, rather than individual medications, using one-way ANOVA and Tukey post-hoc comparisons, to see which classes were regarded more positively or negatively than others. Additionally, we compared web pages referencing the trade names for medications (‘Prozac’) to those referencing the generic name for medications (‘fluoxetine’), controlling for differences between pharmacologic classes using one-way ANOVA. This analysis tests for differences in sentiment arising from searches of trade and generic names. All analysis was performed in R version 3.5.(R Core Team, 2018)

Results:

Across all 51 psychotropic medications, sentiment scores for 12,733 web pages were analyzed. All three pharmacologic classes contained at least one medication that significantly differed in negative sentiment as compared to other medications of the same class (antidepressants: F(24,6282) = 11.17, p < 0.001; antipsychotics: F(16, 4029) = 12.13, p < 0.001; mood stabilizers: F(8, 2371) = 13.28, p < 0.001; Figure 1). Among antidepressants, sertraline, duloxetine, venlafaxine, and paroxetine were among those with greater negative sentiment than other medications of the same class (t(259.36) = 7.80, p <0.001; t(254.22) = 5.63, p <0.001; t(257.00) = 4.36, p <0.001; and t(255.06) = 5.51, p < 0.001, respectively). Among antipsychotics, quetiapine and risperidone exhibited greater negative sentiment compared to the other antipsychotic medications (t(262.29) = 6.96, p = <0.001 and t(274.57) = 5.36, p = <0.001, respectively). Conversely, among mood stabilizers, web results for lithium and valproate contained significantly less negative sentiment than other medications in the class (t(493.38) = −6.50, p = <0.001 and t(299.11) = −4.08, p = <0.001, respectively). The full results of within-class differences in negative sentiment are shown in Table 1 and Figure 1.

Figure 1:

Within-class comparisons of negative and positive sentiment

Table 1:

Within class t-tests of negative sentiment with Bonferroni correction

Medication Class	Ingredient	t	p
Antidepressant	Amitriptyline	1.31	0.19
	Amoxapine	−4.77	<0.001	*
	Bupropion	0.57	0.57
	Citalopram	2.47	0.01
	Desipramine	0.65	0.52
	Desvenlafaxine	−1.86	0.06
	Duloxetine	5.63	<0.001	*
	Escitalopram	1.07	0.29
	Fluoxetine	1.06	0.29
	Fluvoxamine	0.52	0.60
	Imipramine	0.09	0.93
	Levomilnacipran	−2.96	0.003
	Mirtazapine	0.64	0.52
	Nortriptyline	0.06	0.95
	Paroxetine	5.51	<0.001	*
	Phenelzine	−1.51	0.13
	Protriptyline	−3.14	0.002	*
	Selegiline	−7.36	<0.001	*
	Sertraline	7.80	<0.001	*
	tranylcypromine	−1.97	0.05
	trazodone	1.11	0.27
	trimipramine	−2.51	0.01
	venlafaxine	4.36	<0.001	*
	vilazodone	−3.20	0.002	*
	vortioxetine	−3.96	<0.001	*
Antipsychotic	aripiprazole	2.74	0.01
	asenapine	−1.61	0.11
	chlorpromazine	−0.63	0.53
	clozapine	−0.22	0.82
	fluphenazine	−2.71	0.01
	haloperidol	1.38	0.17
	iloperidone	−5.59	<0.001	**
	olanzapine	2.47	0.01
	paliperidone	−1.16	0.25
	perphenazine	−0.08	0.93
	pimozide	−6.13	<0.001	**
	quetiapine	6.96	<0.001	**
	risperidone	5.36	<0.001	**
	thioridazine	−0.12	0.90
	thiothixene	−5.38	<0.001	**
	trifluoperazine	0.62	0.54
	ziprasidone	2.88	0.004
Mood Stabilizer	carbamazepine	−1.88	0.06
	divalproex sodium	2.31	0.02
	gabapentin	5.57	<0.001	***
	lamotrigine	3.23	0.001	***
	lithium salts	−6.50	<0.001	***
	oxcarbazepine	−1.65	0.10
	pregabalin	2.36	0.02
	sodium valproate	−4.08	<0.001	***
	topiramate	2.22	0.03

^*Indicates significant p value after Bonferroni correction for multiple comparisons within the antidepressant class p < .0020

^**Indicates significant p value after Bonferroni correction for multiple comparisons within the antipsychotic class p < .0029

^***Indicates significant p value after Bonferroni correction for multiple comparisons within the mood stabilizer class p < .0056

Note: Negative t values indicate a mean negative sentiment for the given medication that is below the combined mean of all other medications of the same class. In other words, medications with a negative t value have less negative sentiment than the mean of all other medications.

All three pharmacologic classes showed significant within-class differences in positive sentiment (antidepressants: F(24,2682)=2.97, p <0.001; antipsychotics: F(16,4029)= 3.25, p <0.001; mood stabilizers: F(8,2371)=6.88, p <0.001; Figure 1). For example, among mood stabilizers, valproate was associated with significantly less positive sentiment than other mood stabilizers (t(291.89) = −5.35, p < 0.001). The full results of within-class difference in positive sentiment are shown in Table 2 and Figure 1.

Table 2:

Within class t-tests of positive sentiment with Bonferroni correction

Medication Class	Ingredient	t	p
Antidepressant	amitriptyline	0.57	0.57
	amoxapine	−3.64	<0.001	*
	bupropion	1.15	0.25
	citalopram	1.73	0.09
	desipramine	−0.89	0.38
	desvenlafaxine	1.93	0.05
	duloxetine	−0.17	0.87
	escitalopram	2.41	0.02
	fluoxetine	1.93	0.05
	fluvoxamine	−2.40	0.02
	imipramine	−1.10	0.27
	levomilnacipran	−0.02	0.99
	mirtazapine	1.61	0.11
	nortriptyline	0.98	0.33
	paroxetine	0.37	0.71
	phenelzine	−1.75	0.08
	protriptyline	−1.08	0.28
	selegiline	1.60	0.11
	sertraline	0.56	0.58
	tranylcypromine	−1.97	0.05
	trazodone	−3.66	<0.001	*
	trimipramine	−1.95	0.05
	venlafaxine	−0.51	0.61
	vilazodone	1.72	0.09
	vortioxetine	1.05	0.29
Antipsychotic	aripiprazole	1.55	0.12
	asenapine	2.55	0.01
	chlorpromazine	−0.55	0.58
	clozapine	−4.52	<0.001	**
	fluphenazine	−1.70	0.09
	haloperidol	−2.79	0.01
	iloperidone	1.34	0.18
	olanzapine	1.79	0.07
	paliperidone	−2.39	0.02
	perphenazine	−1.19	0.24
	pimozide	0.71	0.48
	quetiapine	2.29	0.02
	risperidone	0.88	0.38
	thioridazine	−0.30	0.77
	thiothixene	0.44	0.66
	trifluoperazine	−1.38	0.17
	ziprasidone	2.28	0.02
Mood Stabilizer	carbamazepine	−1.59	0.11
	divalproex sodium	−0.71	0.48
	gabapentin	2.92	0.004	***
	lamotrigine	−1.51	0.13
	lithium salts	1.71	0.09
	oxcarbazepine	−1.45	0.15
	pregabalin	2.83	0.005	***
	sodium valproate	−5.35	<0.001	***
	topiramate	2.24	0.03

^*Indicates significant p value after Bonferroni correction for multiple comparisons within the antidepressant class p < .0020

^**Indicates significant p value after Bonferroni correction for multiple comparisons within the antipsychotic class p < .0029

^***Indicates significant p value after Bonferroni correction for multiple comparisons within the mood stabilizer class p < .0056

Note: Negative t values indicate a mean positive sentiment for the given medication that is below the combined mean of all other medications of the same class.

In an analysis of compound sentiment – that is, whether a web page was predominantly negative or positive (n=12,697 web pages) – 45 medications (88.2%) had more net negative web pages than net positive web pages. All three pharmacologic classes showed significant within-class differences in the proportions of net negative web pages (Table S1, Table S2, Figure 2). For antidepressants, sertraline and duloxetine had a greater proportion of negative pages compared to other antidepressants (X²(1) = 21.16, p < 0.001 and X²(1) = 11.33, p = 0.001, respectively). Among antipsychotics, haloperidol and quetiapine had significantly greater proportions of net negative web pages as compared to other antipsychotics (X²(1) = 7.53, p = 0.01 and X²(1) = 9.22, p = 0.002, respectively). Within mood stabilizers, lithium preparations had a significantly lower proportion of net negative web pages (X²(1) = 32.78, p <0.001), while gabapentin and lamotrigine had significantly greater proportions of net negative web pages (X²(1) = 11.18, p = 0.001 and X²(1) = 5.02, p = 0.03, respectively).

Figure 2:

Within-class comparisons of compound sentiment

Comparing between pharmacologic classes, there were significant differences in positive sentiment (F(2, 12730) = 14.15, p = <0.001). In post-hoc comparisons, antidepressants had significantly higher positive sentiment than antipsychotics (p < 0.001) and mood stabilizers (p < 0.001), but there was not a significant difference in average positive sentiment between mood stabilizers and antipsychotics (p = 0.58; Figure 3 and Table S3). There were also significant differences among pharmacologic classes in negative sentiment scores (F(2, 12730) = 233.8, p <0.001). Antidepressants yielded the most negative sentiment, and in post-hoc comparisons, there were significant differences between all three classes in negative sentiment (Figure 2, Table S4). Finally, across the corpus as a whole, web pages referencing trade names for medications had significantly more positive sentiment (F(1, 12729) = 188.70, p < 0.001) and negative sentiment (F(1, 12729) = 32.34, p < 0.001) than pages referencing generic medications.

Figure 3:

Between class comparisons of negative and positive sentiment

Discussion:

In this study examining 12,733 web pages across 51 neuropsychiatric medications, we identified significant within-class differences in online positive and negative sentiment for antidepressants, antipsychotics, and mood stabilizing medications. Medications also differed within-class with respect to the proportion of net negative pages returned. Taken in combination, this suggests that the valence of information a patient is likely to find online varies within therapeutic class. That is, one antidepressant, antipsychotic, or mood stabilizing medication searched will have more negative or positive results than another therapeutic equivalent member of the same class.

These within therapeutic class differences are worth physician consideration. Prior research suggests that online sentiment in descriptions of a medication can influence a patient’s treatment decision-making. For example, a study of the smoking cessation medication varenicline showed that increased exposure to positive messages about the medication on social networks significantly increased a patient’s odds of switching to that medication and extended the duration of use for that medication.(Cobb et al., 2013) The present study conceptually extends that work by examining therapeutic classes of medications, enabling within- and across-class comparisons. As more patients turn to the Internet for health-related information, understanding these differences will allow physicians to anticipate concerns that may arise from such information as a means of increasing adherence.

While we could not otherwise identify similar prior efforts, an informative comparison may be with other approaches to capturing consumer sentiment related to psychiatric medications. Notably, a 2010 Consumer Reports survey of patient-reported efficacy and tolerability included sertraline, which was associated with higher negative sentiment in the present study, among the most recommended interventions.(Consumer Reports Best Buy Drugs, 2013) However, their reporting took into account cost, and found very similar results across medications overall; non-antidepressant psychotropics were not considered.

We also note some similarities between sentiment scores and prior research assessing medication efficacy and tolerance in clinical cohorts. For example, our results indicate greater negative sentiment surrounding quetiapine and less negative sentiment surrounding lithium preparations. In prior comparisons of four second generation antipsychotic medications, quetiapine had the lowest proportion of individuals continuing treatment after six months, and was less effective than risperidone and olanzapine.(Stroup et al., 2006) Among mood stabilizers, lithium remains an effective first-line treatment for bipolar disorder, particularly as monotherapy.(Connolly and Thase, 2011; Kessing et al., 2018) Although online medication sentiment does not replace the results of clinical research, side effect profiles and known medication efficacy may be captured by the online sentiment reported in this study and could be a subject of future research in novel therapeutic forecasting strategies building on work in that space.(Castro et al., 2014; McCoy Jr et al., 2017; McCoy Jr and Perlis, 2015)

Although associations with clinical phenomena are interesting, the differences in online sentiment identified in this study likely arise from multiple factors that cannot be isolated using the high-throughput methods required for large-scale research. For example, antidepressants on average exhibit the greatest negative sentiment of any medication class. This result parallels research on medication adherence demonstrating that medication-related concerns influence nonadherence, especially in mood disorders.(Lucca et al., 2015) It is also likely that the increased negative sentiment observed with antidepressant medications could be indicative of the valence of relevant diagnostic indications and associated symptoms such as “sad”, “low”, or even the word “depressed” itself. Current methods do not allow for these to be differentiated using existing validated approaches and models like those used in this study.

In this analysis, trade names were associated with more sentiment (both positive and negative) than generic names. Existing literature on medication name usage in human-authored text suggests that trade names are more commonly used than generic names.(Steinman et al., 2007; Summers et al., 2017) This raises the possibility that the trend toward generally more subjective (both greater negative and positive sentiment) trade name documents may be the result of individual authors’ tendency to use the more memorable or evocative name, whereas less subjective standardized medication catalogs and treatment information sites may be more commonly captured in the generic name sample. More recent medications not yet available in generic form may also be referenced by marketing efforts, with greater positive sentiment on industry-supported web pages optimized to appear at the top of Internet searches.

While these results provide a perspective on healthcare consumerism and a novel means of comparing medications, we do note a number of limitations. First, as noted above, sentiment can arise from multiple sources and these sources cannot be separated at scale. Second, while this study focuses on quantifying the online sentiment around different psychiatric medications, we are not able to link this information to individual patient consumption of this information (i.e., how many sites patients access, or the type of sites that are most likely to be accessed). Whereas 72% of American Internet users accessed health related information only 13.6% Japanese patients diagnosed with schizophrenia only 13.6% turned to the Internet for information and the present study does not address characteristics of patients who turn to the Internet for additional information (Fox and Duggan, 2013; Nagai et al., 2017). We included the first 119 search results for each generic and trade medication, but individuals may only access a fraction of these sites, and search strategies may vary widely (e.g., some patients may only access the top web site, or may prefer different search engines). Furthermore, the present data does not address how exposure to web sites impact future clinical interactions or the role of timing. These results reflect a cross-sectional perspective on a large number of medications; however, sentiment scores are likely to change over time, reflecting news about medications, new studies, and even changes in medication marketing. Our results are thus best interpreted as a cross-sectional view of psychopharmacology, subject to change over time. Finally, classification of medications based on indication is not the only way in which these medications might be grouped. The impact of reclassification, for example by underlying neuroscience, is an area in need of further study (Zohar et al., 2015.). Although these are important limitations, the results are nevertheless a guide to the current state of online information and the differences among agents within a therapeutic class. This information can inform patient education and guidance by prescribers.

Taken together, sentiment toward psychotropic medications, as captured in Internet search results, varies significantly between and within classes. Characterizing these differences may allow clinicians to anticipate patient responses to new prescriptions or guide patient expectations around personal research. As patients increasingly seek out health data online, understanding the content they are likely to view can facilitate more informed conversations about medications on their own, and relative to treatment alternatives.

Highlights.

• Differences in online sentiment exist within classes of psychiatric medications

• Psychotropic medications were associated with significantly more negative sentiment

• Characterizing differences in online sentiment may anticipate patient concerns

Abbreviations:

VADER

Valence Aware Dictionary and sEntiment Reasoner