Using Twitter to Measure Public Discussion of Diseases: A Case Study

Abstract

Background

Twitter is increasingly used to estimate disease prevalence, but such measurements can be biased, both due to biased sampling and to the inherent ambiguity of natural language.

Objective

We characterized the extent of these biases and how they vary with disease.

Methods

We correlated self-reported prevalence rates for 22 diseases from Experian's Simmons National Consumer Study (n=12,305) with the number of times these diseases were mentioned on Twitter during the same time period (2012). We also identified and corrected for two types of bias present in Twitter data: (1) demographic variance between US Twitter users and the general US population; and (2) natural language ambiguity, which creates the possibility that mention of a disease name may not actually refer to the disease (e.g., “heart attack” on Twitter often does not refer to myocardial infarction). We measured the correlation between disease prevalence and Twitter disease mentions both with and without bias correction. This allowed us to quantify each disease's overrepresentation or underrepresentation on Twitter, relative to its prevalence.

Results

Our sample included 80,680,449 tweets. Adjusting disease prevalence to correct for Twitter demographics more than doubles the correlation between Twitter disease mentions and disease prevalence in the general population (from 0.113 to 0.258, P < .001). Also, diseases varied widely in how often mentions of their names on Twitter actually referred to the diseases, from 14.9% of instances (for stroke) to 99.9% (for arthritis). Applying ambiguity correction to our Twitter corpus achieves a correlation between disease mentions and prevalence of 0.208 (P < .001). Simultaneously applying correction for both demographics and ambiguity more than triples the baseline correlation to 0.366 (P < .001). Compared to prevalence rates, cancer appeared most overrepresented in Twitter while high cholesterol appeared most underrepresented.

Conclusions

Twitter is a potentially useful tool to measure public interest in and concern about different diseases, but when comparing diseases, improvements can be made by adjusting for population demographics and word ambiguity.

Introduction

Word use patterns in Twitter, Facebook, newsgroups, and Google queries have been used to investigate a wide array of health concerns. Twitter is perhaps the most popular online data source for such studies, due in part to its relative accessibility. It has been used to monitor health issues including influenza [1,2], cholera [3], H1N1 [4,5], swine flu [6], postpartum depression [7], concussion [8], epilepsy [9], migraine [10], cancer screening [11], antibiotic use [12], medical practitioner errors [13], dental pain [14], and attitudes about vaccination [15].

Such research has demonstrated the utility of mining social media for public health applications in spite of potential methodological challenges, including the facts that (a) Twitter users form a biased sample of the population [16-18], and (b) their word usage within tweets can be highly ambiguous. Focusing just on the medical domain, “stroke” has many non-medical uses (e.g. “stroke of genius”, “back stroke”), most mentions of “heart attack” are metaphorical, not literal, (“just had a heart attack and died the power went out while I was in the shower”) and although doctors associate “MI” with myocardial infarction, on Twitter it refers more often to the state of Michigan.

This paper quantifies, and provides a framework for partially correcting, the error arising when using sources such as Twitter as a proxy for measuring disease prevalence. We investigate the relationship between the frequency of disease mentions on Twitter in the US and the prevalence of the same diseases in the US population. Understanding this relationship could be useful for a variety of applications, including health care messaging and disease surveillance. We use Twitter as the venue for measuring discussion largely because it has already received substantial attention as an inexpensive proxy for tracking disease prevalence [19,20].

Our key contribution is to demonstrate that it is possible to better align Twitter disease mention statistics with actual disease prevalence statistics by correcting for ambiguous medical language on Twitter, and by correcting for the difference between the demographics of Twitter users and the general US population. We observe that a slight correlation exists between general population disease prevalence statistics (sourced from existing survey data) and the number of times each disease is mentioned on Twitter (according to our own counts). We find that we can significantly increase this correlation (a) by restricting the disease prevalence population specifically to Twitter users (i.e., by correlating with existing prevalence data focused specifically on that group), and (b) by adjusting our disease mention counts to correct for word sense ambiguity.

Methods

Overview

We first identified a list of diseases; then for each disease we constructed a list of terms that refer to it (i.e., a disease-specific lexicon). We also collected a large number of tweets and compiled them into a tweets corpus. Next we retrieved a random sample of tweets from our corpus that contained any of our disease-terms. We then determined the relative frequencies (percentage) of medical uses of the disease-terms (i.e., valid positives) versus non-medical uses (i.e., false positives due to ambiguity), using human annotation on the random sample. This allowed us to compute corrected counts of the number of tweets in the corpus that mention each disease (we call this a disease's validated tweet count, whereas an uncorrected count is termed a raw tweet count).

We correlated the corrected disease mention frequencies to US disease prevalence statistics from the Simmons National Consumer Study [21]. The resulting correlation serves as a measure of the relationship between the quantity of disease mentions in the corpus, and the quantity of disease cases in the US population (for either the general population or the Twitter-using population). Comparing the correlations with and without correction demonstrates the size of our corrections.

Data collection

Selection of diseases

We used two criteria for selecting diseases for this research: (1) diseases that could be paired with both US population prevalence data and Twitter-use data; and (2) diseases deemed by previous literature to be most impactful for the healthcare community. Each criterion is satisfied by a different dataset.

The first dataset comes from Experian, a global information services company known as a source of credit scores. Experian also conducts consumer surveys on a variety of topics, including health care. For this study, we used data from Experian's Simmons National Consumer Study and focused on survey questions pertaining to general demographics, health status, and social media use.

Results from the various Experian surveys are combined into a study universe database and released both quarterly and annually. Experian conducts post-stratification on its survey data to create demographically representative estimates for its measured variables. We queried this database to obtain a dataset for the year 2012 that cross-tabulates (a) disease prevalence for all available diseases (n=52) with (b) general demographics and with (c) Twitter use. For the estimated English-speaking or Spanish-speaking US adult population (n=230,124,220) we were able to find the estimated number who suffer from a disease (e.g., backache, n=42 million), and the subset of those disease sufferers who use Twitter (in the case of backache, n=2.6 million). This dataset, then, provides us with parallel sets of disease prevalence statistics for the general US population and for US Twitter users.

The second dataset is from a RAND study designed to broadly measure the quality of health care delivery in the US [22]. Through reviews of the literature and of national healthcare data, and through consultation with panels of medical experts, 46 “clinical areas” were identified in this report that represent the leading causes of illness, death, and healthcare utilization in the US.

The list of 24 diseases (Multimedia Appendix 1) used in the present study is composed of the overlap between the diseases represented in the Experian dataset (n=52) and in the RAND study (n=46). This overlap may be explicit (e.g., asthma appears on both lists) or implicit (e.g., two separate Experian entries, stomach ulcers and acid reflux disease/GERD, are both suggested by the single RAND entry peptic ulcer disease and dyspepsia). The focus in this task was not pinpointing exact matches between the two lists, but rather finding areas of general agreement between them, in order to identify high impact diseases from the Experian dataset.

Compilation of disease-terms

For each disease on our list, we constructed a lexicon of disease-terms that are used to refer to that disease. For example, the lexicon for diabetes used in this study contains three disease-terms: “diabetes”, “diabete”, “niddm.” All lexica in this study are derived from terms found in the Consumer Health Vocabulary (CHV) [23], an online open source thesaurus that associates medical concepts (including diseases, medical procedures, drugs, anatomy, etc.) with a mix of colloquial and technical terms. At the time of this study the CHV contains 158,519 entries, covering 57,819 unique (but often closely related) concepts. Each entry collects (along with other data) at least three term elements: (a) a CHV term, (b) a descriptive phrase, and (c) a related term from a medical vocabulary called the Unified Medical Language System (UMLS). A CHV term can have multiple entries in the thesaurus, thereby associating the CHV term with any number of descriptive phrases or UMLS terms. Each CHV term, then, can be seen as a key-value pair, where the CHV term is the key and a network of associated terms (comprised of descriptive phrases and UMLS terms) is the value.

For each of the 24 diseases included in the study, we processed the CHV to retrieve an entire key-value network of associated terms if any one of the terms (in the key or the value) seemed to refer to the target disease. Multiple networks could be (and often were) collected for any disease. Together, these results constituted a disease's list of candidate disease-terms (these were then vetted, in a process described below). A term was judged to be a potential reference to a target disease (thereby triggering the retrieval of all associated terms) primarily if it contained a search string derived from the target disease's name (including both abbreviated and spelled-out forms). For example, “attention deficit” is a search string for ADD/ADHD; “heart disease” is a search string for heart disease; and “GERD” is a search string for acid reflux disease/GERD. Also included among the search strings were some common disease synonyms, such as “zit” for acne and “tumor” for cancer. The number of search strings per disease varied, ranging from one to seven.

Tweet text corpus

The tweets used in our analysis were taken from a random sample of 1% of all available tweets in 2012, as collected through the Twitter “1% random public stream” API [24]. In order to align our data more closely to the American and mostly English-speaking Experian Simmons sample, we filtered our Twitter corpus to keep only English language tweets originating in the US. To filter for English we only considered tweets with at least 50% of their words found in the HunSpell English dictionary [25]. Tweets were then further restricted to the US by finding tweets with “United States” or non-ambiguous US cities in their location field. (City names were taken from [26]). For example, “Chicago” would match the US while “London”, even though there is a London in Texas, would not. This resulted in a corpus of 80,680,449 tweets.

Vetting disease-term candidates

Grammatical

In this research we focused on finding tweets that specifically name our target diseases. Broadening this focus to include related concepts, such as symptoms and treatments, was desirable but was not possible for the scope of this paper. Because of our focus on terms that name diseases (as opposed to terms that describe or suggest them), we dropped all candidate disease-terms that were not nouns (e.g., adjectives such as “depressed” or “arthritic”).

We then manually expanded the list, adding plural forms where grammatically appropriate.

Medical

We mined the CHV using a keyword search strategy inclined toward inclusiveness. For example, a search on “acne” retrieved terms for medical concepts that might be at best tenuously related to acne. One of these concepts was acne rosacea, whose network of associated terms contains the terms “acne rosacea”, “disorders rosacea”, “rosacea”, and “rosacea acne”.

Because the concept acne rosacea incorporates at least one term containing the text string “acne”, its entire network of terms automatically became candidates for the acne lexicon. This inclusiveness raises the question of whether “rosacea”, “acne rosacea”, “disorders rosacea”, etc., denote acne. To solve this problem, a physician on the research team vetted the candidate terms. For each disease, she dropped candidates that did not denote the disease, ensuring that only medically appropriate terms were admitted into any disease lexicon.

Structural

We produced a list of text strings for each disease that we could use to search the Twitter corpus for mentions of that disease. To achieve this goal we took into account two realities. (1) Many CHV term elements use constructions that are uncommon in natural language (e.g., “fever hay” as in nasal allergies/hay fever), “attack heart”, “attacks hearts”, or “attacking heart” as in heart attack, and “pain, back, radiating” as in back pain). (2) During execution of searches within the Twitter corpus, only the shortest element of a search phrase is required; if a compound search phrase contains a shorter search phrase, the longer one is implied by the shorter (e.g., “asthma” retrieves asthma, allergic asthma, pollen asthma, etc.; “diabetes” retrieves diabetes mellitus, insulin dependent diabetes, diabetes screening, etc.).

Because of these two facts, we were able to significantly shorten the candidate disease-term lists that were produced by the semi-automated CHV search procedure. All reverse-order candidates (e.g., “fever hay”) and compound candidates (e.g., “allergic asthma”) were dropped.

After these three vetting procedures, the 24 disease-lexica contained a combined 488 disease-terms (Multimedia Appendix 1).

Manual Tweet appraisal

We determined how often each of the 488 disease-terms referred to its associated disease when included in a tweet. This began with a basic count for each disease-term of the number of tweets in the corpus in which the term appears (prior to any language ambiguity corrections being applied). This is a disease-term's raw tweet count. Note that we allow a single tweet to be counted twice if it contains multiple disease terms (regardless of whether the two terms refer to the same or a different disease). Throughout this study we consider random instances of disease-terms as they appear in tweets, without consideration for other terms that co-occur with them.

We then performed manual appraisal. For each disease-term we randomly selected 30 tweets containing the term from our tweet corpus for manual analysis. This number was chosen to balance research needs and time constraints. Some disease-terms occurred in 30 or fewer tweets in the tweet corpus. When this occurred, all available tweets were retrieved.

Two English-speaking research assistants independently read each tweet and made a simple appraisal, answering, “For each tweet, in your judgment does the disease-term that flagged the tweet's retrieval refer to a medical meaning of that term?” Each tweet required a Yes or No judgment, as shown in Table 1. The two raters each compiled a complete collection of Y/N judgments, held in secret from the other rater. After these tweet-level appraisals were completed, we aggregated the scores at the disease-term level (independently for each rater's collection of judgments). For each rater and each disease-term (n=488), we calculated the percentage of tweets from the sample that were appraised as referring to a medical meaning. The Cohen's kappa for inter-rater reliability was 0.77.

Table 1.

Example of rating whether each tweet does or does not refer to a medical meaning of the selected term. Here the term is “heart attacks”.

Rater 1	Rater 2	Tweet
Y	Y	Visited a man who has had 2 heart attacks who feels privileged to be in circumstances that allow him to share his trust in God. #realdeal
Y	N	Got room for 1 more? RT @pjones59: Sausage balls, heart attacks on a stick, dip, chips, wings and cheese, cream cheese/pickle/ham wraps
N	N	I still can't believe I saw Kris at work the other day. Talk about mini heart attacks. U_U

The disease-term percentages of the two raters were then averaged, resulting in a correction factor (cf) for each disease-term. We multiplied this coefficient by the disease-term's raw tweet count (rcount), to arrive at an estimated disease-term validated tweet count (vcount).

$$vcount\left(d\right)=\sum _{t\in terms\left(d\right)}rcount\left(t\right)\ast c{f}_t$$

Once this estimate was completed for each disease-term in a disease-lexicon, the disease-term estimates were summed, producing our ultimate metric, a validated tweet count for each disease-lexicon. The validated tweet count for a disease-lexicon is the estimated number of tweets in our corpus that are a valid reference to the disease in question, i.e., correcting for the ambiguity error present in the disease-lexicon's raw tweet count.

As an example, manual appraisal for the diabetes disease-lexicon (Figure 1) illustrates the evolution from a raw tweet count of 9,202 to a validated tweet count of 8,896.

Figure 1.

Disease-terms from the diabetes lexicon undergo manual appraisal. Each term receives appraisal on up to 30 instances. The term-level appraisals are then summed to reach the final lexicon-level diabetes validated tweet count (8,896).

Results

Of the 2,824 tweets containing disease-terms that we manually reviewed, the averaged judgments of our two human raters indicate that 2,276.5 (80.61%) actually referred to diseases, with validity rates that were highly variable across different diseases. For example, stroke terms rarely referred to the medical emergency (only 14.89% of the time), while diabetes terms almost always referred to the medical condition (96.67% of the time).

The raw tweet counts and validated tweet counts for the 24 diseases are compared in Table 2, along with a correction factor (an adjustment according to the percentage of evaluated tweets that were judged as valid). Table 2 also includes disease prevalence data (for both the general US population and among US Twitter users), which come directly from Experian's Simmons National Consumer Study. We see high levels of heterogeneity for all five measurements across diseases. This probably reflects the heterogeneity among the diseases themselves: included among them are acute viral infections (e.g., flu), general maladies (e.g., backache, nasal allergies/hay fever), chronic disorders (e.g., arthritis, osteoporosis), test measures (e.g., high cholesterol, hypertension/high blood pressure), medical emergencies (e.g., heart attack, stroke), and psychological disorders (e.g., depression, ADD/ADHD). Some of the diseases are transitory (e.g., urinary tract infection) and others are long-term (e.g., diabetes). Some are causes of mortality (e.g., cancer, congestive heart failure), while others are relatively superficial (e.g., acne). Given such variety, it is no surprise to see a wide range of values for tweet counts, correction factor, and prevalence across the list of diseases.

Table 2.

Raw and validated tweet counts, correction factor, and US and Twitter disease prevalence for each disease.^a,b,c,d

Disease	Raw tweet count	Validated tweet count	CF	Prev., US (millions)	Prev., US Twitter (millions)
acid reflux disease/GERD	743	631	84.98	32.4	2.4
acne	6,936	6,027	86.89	11.2	2.0
ADD/ADHD	2,794	2,660	95.19	4.9	0.9
arthritis	2,524	2,522	99.92	34.4	1.3
asthma	3,952	3,754	95.00	12.4	1.0
backache	3,035	3,028	99.77	42.0	2.6
cancer	110,760	63,647	57.46	5.0	0.46
congestive heart failure (CHF)	928	313	33.76	--	--
heart disease	2,741	2,410	87.91	--	--
CHF/heart disease	3,669	2,723	74.21	5.9	0.46
COPD	226	188	83.37	5.5	0.86
depression	14,294	10,459	73.17	18.7	2.2
diabetes	9,202	8,896	96.67	20.8	1.2
flu	10,139	8,810	86.90	17.2	1.8
genital herpes	76	66	86.84	1.8	0.33
heart attack	15,027	2,311	15.38	--	--
stroke	12,852	1,914	14.89	--	--
heart attack/stroke	27,879	4,225	15.15	3.0	0.11
high cholesterol	225	218	96.67	37.9	1.7
human papilloma virus	636	545	85.73	1.5	0.12
hypertension/high blood pressure	1,630	1,491	91.49	43.5	1.5
migraine headache	5,958	5,615	94.24	16.4	1.8
nasal allergies/hay fever	481	473	98.27	18.2	1.3
osteoporosis	316	306	96.68	6.0	0.13
stomach ulcers	80	73	91.25	3.3	0.030
urinary tract infection	880	479	54.40	10.0	1.0

^aHeader abbreviations: CF represents “correction factor” (i.e., the percentage of tweets that were appraised as valid); Prev., US (millions) represents a disease's prevalence in the US; Prev., US Twitter (millions) represents a disease's prevalence among US Twitter users.

^bThe source for both Prev., US (millions) and Prev., US Twitter (millions) is the Experian Simmons National Consumer Study.

^cIn the Experian dataset, congestive heart failure and heart disease are collapsed into a single data point. We mined Twitter for these diseases separately, and we applied our evaluation method to tweets containing disease-terms for each one separately. However, because Experian was our source for prevalence statistics, we can only report on the prevalence of these two diseases in a combined state.

^dThe same is true for the diseases heart attack and stroke.

We determined the Spearman correlation coefficients (all P values < .001) between both raw and validated tweet counts and disease prevalence among both the general US population and among US Twitter users (Table 3). Correcting just for Twitter use more than doubles the correlation between tweet count and prevalence (from 0.113 to 0.258). Correcting only for word ambiguity has a similar but slightly smaller effect (0.208). Correcting for both more than triples the baseline correlation (0.366).

Table 3.

Spearman correlation coefficients among raw and validated tweet counts and US population and Twitter user disease prevalence (all P values <.001).

	Prevalence	Prevalence
	US pop.	US Twitter users
Raw tweet count	0.113	0.258
Validated tweet count	0.208	0.366

Discussion

Overview

The correlation improvements we found due to ambiguity correction may seem unsurprising. But the improvements due to demographic correction are less straightforward, particularly because no effort was made to restrict our tweet analysis to first-person self-report mentions of diseases. It is easy to assume that there must be a causal connection between disease prevalence and disease mentions. Indeed, we interpret an increased correlation due to demographic correction as supporting this assumption: it means the signal we measure (i.e., disease mentions on Twitter) demonstrates positive correspondence to a plausible source of that signal (i.e., disease sufferers who use Twitter). However, we find that for certain individual diseases, disease prevalence and disease mentions are wildly out of sync. For the time being just what causes someone to tweet (or not tweet) about a disease remains an open question, particularly since many people mentioning the disease are not suffering from it. In any case, methods utilizing social media to estimate disease prevalence do not need to explain a causal connection. They only demand that social media reliably captures the variance of disease prevalence. We have shown that such measurement can be improved by adjusting for demographic differences between disease sufferers and Twitter users.

Bias correction

We found that naively counting mentions of disease-terms in tweets produces results that are biased (in terms of correlation with known disease prevalence statistics) both due to the demographic pattern of Twitter users, and due to the ambiguity of natural language. These biases can be at least partially corrected, resulting in a three-fold increase in the correlation between counts of disease terms in tweets and known prevalence statistics for the 24 diseases we studied.

The observation that the Twitter population is a biased sample of the US is relatively easily corrected using standard stratified sampling methods, given the known demographics of the Twitter population. We identified this using data from an Experian survey, but other studies of Twitter demographics could be used. We demonstrated that the demographic corrections roughly doubled the correlation between disease mentions and disease prevalence.

Types of ambiguity

The intrinsic ambiguity of language requires more work to correct. We observed that language ambiguity varies significantly across diseases. The fraction of mentions of a disease term that actually refer to the disease ranged from highly specific terms like arthritis (99.92%) to less specific terms like stroke (14.89%). This language ambiguity takes two major forms.

The first is lexical ambiguity. Some diseases such as arthritis, diabetes, and high cholesterol are in practice referred to by terms that almost always refer to their associated disease concepts. In our analysis, tweeters rarely used words from the arthritis lexicon to refer to anything other than the disease arthritis. There are, however a number of disease-terms that are often used to refer to concepts that are not diseases (or not the intended diseases). Frequently occurring example words include “cancer” (the astrological sign), “depression,” “stroke” (non-medical usages and also heat stroke), and “flu” (as in stomach flu, versus influenza). Abbreviations are particularly ambiguous. For example, “copd” (i.e., chronic obstructive pulmonary disease) is a popular variant spelling of “copped” (as in the verb “took”); “uti” (urinary tract infection), “hpv” (human papillomavirus), and “zit” (acne) show up in Internet addresses (particularly in short links using URL redirection); and “CHF” (congestive heart failure) is an abbreviation for the Swiss Franc. Or conversely, “Gerd” is a masculine first name that coincides with an abbreviation for the disease gastroesophogeal reflux disease (part of the acid reflux disease/GERD lexicon). Lexical ambiguity also arises from metaphorical and slang usages of disease-terms. “Heart attack” and “heart failure” are used to mean surprise and “ADHD” to mean distracted.

The second type of ambiguity could be considered disease ambiguity. Some of the 24 diseases included in this study are less clearly delineated than others. One aspect of this problem is intensity. Is it medical depression if a Twitter user reports being depressed about her favorite sports team losing a game? What if she ends a seemingly grave tweet with “LOL”? A second aspect of disease ambiguity is specificity or accuracy. Some Twitter users may use the word migraine for other types of headache or say hay fever when actually they're allergic to cats. A third aspect of disease ambiguity is complexity. A prime example is the range of cardiovascular diseases in this study (i.e., congestive heart failure, heart disease, heart attack, high cholesterol, hypertension, possibly stroke), whose interrelations and exact boundaries are difficult or impossible to draw.

Both types of ambiguity can affect a disease's validation coefficient. The first type, lexical ambiguity (e.g., homographs or metaphorical word usage), is likely to affect the “back end” of the methodology, requiring corrections to the observed term counts (i.e., using the method described in this paper). The second type, disease boundary ambiguity, presents problems on the “front end” and it makes tailoring the disease lexica difficult. This type of ambiguity raises the question of whether the potentially hierarchical relationship between congestive heart failure and heart disease, or the potentially causal relationship between high cholesterol and either heart attack or stroke, could or should somehow be encoded in the disease-lexica.

In this research, we treated each disease-lexicon as a stand-alone entity, and the effects of that decision are necessarily written into the results we derived. We can expect that diseases of a more “stand-alone” quality (i.e., those that are relatively self-contained like osteoporosis, rather than part of a complex like heart disease) will naturally be better represented by their respective disease-lexica than are diseases that potentially harbor a complex relationship with other diseases. It is intuitive that mismatch between a disease's representation on Twitter and its representation within its disease-lexicon is essentially what causes the disease's validation coefficient to drop below 100%.

Correlation of validated tweet count with prevalence

Just as validity rates proved highly variable across diseases, the levels of Twitter discussion relative to disease prevalence also varied. Some diseases were discussed at levels outstripping their prevalence in the population, while others received little relative attention. The relationship between validated tweet count and US prevalence across diseases has a correlation of 0.208 (Table 3, P < .001). To provide a more detailed picture of this relationship, we calculated validated tweet count for each disease as a function of prevalence. The formula we used is, for each disease d: projected prevalence = (validated tweet count of d/sum of all validated tweet counts)*sum of all disease prevalences. This can be understood as the prevalence that validated tweet count (inaccurately) projects for each disease.

We compare this projected prevalence to actual prevalence in Figure 2. The sum of prevalence across all diseases (351,939,580) is identical for both the projected and the actual cases, but the distributions are quite different. We see that cancer is a major outlier, “taking up” over 50% of projected prevalence (176,605,210) while it accounts for less than 2% of actual prevalence (5,031,120). Clearly cancer receives far more attention than merely prevalence warrants. Projected prevalence is more than 35 times as great as, or 3,500% of, actual prevalence. Conversely, high cholesterol is on the extreme end of underrepresentation. Projected prevalence (604,898) is only 1.598% of actual prevalence (37,861,070). These figures demonstrate that other unknown factors besides prevalence influence the amount of discussion a disease receives on Twitter.

Figure 2.

Projected prevalence (as a function of validated tweet count) vs. actual US prevalence for 22 diseases, in millions (sorted by projected prevalence). Some diseases are “over-tweeted” (in particular, cancer) while others are “under-tweeted” (e.g., backache and arthritis).

One hypothesis is that Twitter demographics skew discussion levels upward for diseases that are of high concern to the population of users and downward for diseases that are of less concern. Given the generalization that Twitter users tend to be young, this could explain why arthritis seems to be drastically under-tweeted, and why acne and ADD/ADHD are over-tweeted. But demographics alone cannot explain the extraordinary projected prevalence of cancer. Nor are they likely to explain the over-tweeting of flu and diabetes. We assume that demographics do influence these results (see acne and hypertension/high blood pressure), but that multiple other factors also play roles. Likely candidates include the intensity and history of disease awareness and advocacy campaigns (see cancer, diabetes, and human papilloma virus); and disease stigma or body part stigma (see genital herpes and urinary tract infection). Investigation into these and other possible factors is an area for future research.

Limitations

It remains unclear precisely what is the nature of the relationship between disease discussion (on Twitter or even just in general) and disease prevalence. Twitter disease discussion is likely driven by many more factors than disease prevalence. People tweet about diseases for many reasons, and for the purposes of this paper, we do not attempt to disentangle such reasons. We do demonstrate, though, that Twitter disease mentions correlate with disease prevalence, and that this correlation improves after our demographic and word ambiguity corrections have been applied. This lesson can and should be incorporated into other research or tools that would seek to mine the language found on Twitter (or similar venues) for information about broader populations.

Despite our best efforts, the demographics of our Twitter corpus and of the Experian dataset do not entirely match. Most significantly, the Experian dataset includes disease prevalence estimates for both English-speaking and Spanish-speaking US residents whereas our tweet corpus was restricted to English language tweets. This research was conducted in English; future work should extend a similar analysis to other languages.

We did not account in this research for all possible variables that could influence the interplay of disease prevalence and tweets about diseases. Some of these missed variables are disease-centric. For example, some diseases may actually be more “tweet-able” than others due to any number of disease factors, including intensity, duration, stigma, social salience, and so on, or possibly even due to formal considerations (is the disease easy or quick to spell?). A less tweet-able disease might be expected to have fewer associated tweets, outside of any prevalence-based influence on tweet counts.

We only account for mentions of diseases that specifically name a (properly spelled) disease in a tweet. On the one hand, relying on correct noun-form disease names that are sourced from a recognized health vocabulary like the CHV helps push this study toward semi-automation, objectivity, and reproducibility. But on the other hand, this decision leaves an unknown, but possibly large, quantity of disease-relevant tweets unmined, and so unaccounted for in our analysis. We miss mentions that are slang terms (e.g., “diabeetus”) or are misspelled (e.g., “ashtma”, “hi colesterol”, etc.). On a strictly formal level, our current approach is tuned to precision at the expense of recall. Furthermore, people may discuss health concerns on Twitter by mentioning symptoms, sequelae, locations (such as a hospital), drugs, or treatments, etc. Our focus on disease names is unable to capture this broader domain of health-related tweets. Improving recall is left for future work.

Other missed variables are Twitter-centric. It is well documented that Twitter does not reveal what sampling procedures are used in their APIs [27,28]. Therefore it is unclear how representative the tweeters (whose tweets were captured for this study) are of the US population of Twitter users. This is unavoidable, and it is a shortcoming common to all research using Twitter APIs.

We also did not discriminate in this research between tweeters. A Twitter “user” may not be an individual person. Many health-related or even disease-related organizations mention diseases on Twitter. Factors related to such organizations (their quantity, their social media strategies, etc.) may be relevant to counts of disease-naming tweets. Other researchers have addressed the problem of distinguishing tweets authored by health organizations [29], but this study did not make that distinction.

Comparison with Prior Work

In the normal course of life or business individuals and organizations generate vast amounts of text that can be mined. Much of it is shared or published online in one form or another, and this data is attractive to researchers, including those interested in epidemiology and public health.

Several “infodemiology” studies (e.g., many in the long list cited in the introduction) correlate word use in Twitter with prevalence of a disease or medical condition. Beyond Twitter, web search activity has also been used, for example to monitor Lyme disease [30] and dengue [31], as well as risk behaviors associated with dietary habits [32] and with suicide [33]. Blog posts have been used to predict influenza outbreaks [34]. Facebook has been used to predict “gross national happiness,” i.e., well-being across the US [35].

These studies primarily correlate word use in some medium (e.g., Twitter or Google search) over some time period (e.g., day or week) and in some region (e.g., US county or state) with a disease prevalence level. Such correlational approaches rely on certain assumptions about the homogeneity of the populations they study, which often go unstated and so presumably untested and uncorrected. It is not clear that demographic or word ambiguity biases are typically accounted for. We suppose that researchers implicitly assume that these factors will be handled automatically by the statistical regression methods they use. If demographic and ambiguity biases are constant over time and space, this will be true. However, if populations vary in their usage of the target medium (e.g., Twitter) prediction accuracy can vary, and this variation may be significant.

Google Flu Trends is perhaps the “poster child” for the correlational approach to prediction. It is a widely cited online tool that uses statistical correlations between a broad set of Google search terms and historical flu levels to predict regional changes in US flu levels [36]. Google Flu Trends was initially highly accurate. But it has also been used as a case study of how “big data” predictions can go awry when the statistical patterns upon which they are based are descriptively inaccurate (either from the start or due to drift over time) [37,38], with claims that at one point predictions became exaggerated by nearly a factor of two [39].

A limited number of studies have emphasized concerns about validity in social media analyses [40]. There has also been some work on selecting “high quality” disease-related tweets, mostly achieving high specificity at the cost of poor sensitivity. For example, [41] uses regular expressions and machine learning methods to filter out all but first person self-report tweets. We strive for higher coverage, including all “real” mentions of a disease, and then we seek out previously established data (i.e., disease prevalence) against which to validate our findings. In [42] researchers identify known sick persons, then study their Twitter data to characterize a kind identifying fingerprint for Twitter users who have the flu (utilizing their tweets and their Twitter profile metadata). They then use that model to “diagnose” individual Twitter users with influenza, an approach that the authors imply could be directed toward population level disease surveillance.

In the near term, we think the major use of social media for public health may be to understand attitudes toward health, disease, and treatment. Effective public policy depends on subjective inquiries into what people know and care about. Why do they seek or avoid treatment? How do they reveal disease status? What risk behaviors do they shrug off? Predictions about a phenomenon that one can measure, such as disease prevalence, may have limited utility, especially if the measurements are timely and accurate. Although traditional ground truth measurements have been questioned [43], Centers for Disease Control and Prevention flu estimates appear to be better than Google Flu Trends estimates [44]. Nevertheless, online disease detection and prediction is a rapidly growing research area, and as work continues in this field, our collective ability to make these types of estimates will likely increase.

Conclusions

Several types of research using social media to study public health will benefit from corrections for demographic variation and language ambiguity of the type that are outlined in this paper. Social media datasets are biased convenience samples, and word ambiguity is endemic. Nevertheless, social media provide a relatively cheap way to monitor countless domains, including public health and attitudes toward health and health care.

In this study we began with a large, “poor quality,” non-random dataset (i.e., Twitter), and compared it to a small, “high quality,” random (achieved via post-stratification) dataset, (i.e., the Simmons National Consumer Study from Experian). We filtered the Twitter dataset so that its demographics would match the Experian dataset's. Then we performed both naïve and ambiguity-corrected counts of Twitter disease mentions. Finally we compared both those counts to prevalence data found in the Experian survey. We found that the corrected Twitter counts correlated much more strongly than the naïve Twitter counts with the “high quality” Experian data. We think that this demonstrates both the need and the capacity for other studies using non-random convenience samples (e.g., social media data or Google queries) to take demographic and word ambiguity factors explicitly into account, for example by using our method or other related or novel methods.

Abbreviations

ADD/ADHD

attention deficit disorder/attention deficit hyperactivity disorder

API

Application Programming Interface

CHF

congestive heart failure

CHV

Consumer Health Vocabulary

COPD

chronic obstructive pulmonary disease

GERD

gastroesophageal reflux disease

UMLS

Unified Medical Language System

Multimedia Appendix

Multimedia Appendix 1. This study focuses on a list of 24 diseases. Each of these diseases is represented by a disease lexicon composed of one or more disease-terms. There are 24 lexica, comprising 488 total disease terms. The first row in this appendix holds lexica names, subsequent rows hold disease-terms. Each column represents a different disease.