Twitter Response to the United States Preventive Services Task Force Recommendations against Screening with Prostate Specific Antigen

Abstract

Objective

To examine public and media response to the United States Preventive Services Task Force’s (USPSTF) draft (October 2011) and finalized (May 2012) recommendations against prostate-specific antigen (PSA) testing using Twitter, a popular social network with over 200 million active users.

Materials and Methods

We used a mixed methods design to analyze posts on Twitter, called “tweets.” Using the search term “prostate cancer,” we archived tweets in the 24 hour periods following the release of the USPSTF draft and finalized recommendations. We recorded tweet rate per hour and developed a coding system to assess type of user and sentiment expressed in tweets and linked articles.

Results

After the draft and finalized recommendations, 2042 and 5357 tweets focused on the USPSTF report, respectively. Tweet rate nearly doubled within two hours of both announcements. Fewer than 10% of tweets expressed an opinion about screening, and the majority of these were pro-screening during both periods. In contrast, anti-screening articles were tweeted more frequently in both draft and finalized study periods. From the draft to the finalized recommendations, the proportion of anti-screening tweets and anti-screening article links increased (p = 0.03 and p<0.01, respectively).

Conclusions

There was increased Twitter activity surrounding the USPSTF draft and finalized recommendations. The percentage of anti-screening tweets and articles appeared to increase, perhaps due to the interval public comment period. Despite this, most tweets did not express an opinion, suggesting a missed opportunity in this important arena for advocacy.

Introduction

The USPSTF, a panel whose members are appointed by the Department of Health and Human Services, develops evidence-based guidelines for clinical preventive services, including screening for cancer. Despite not having a direct policy mandate, the Affordable Care Act promotes the influence of these guidelines on insurance company reimbursements. [1] In an effort to increase clarity and transparency, the USPSTF instituted a public comment period prior to finalization of initial drafts of its guideline recommendations. [1, 2]

In October 2011, the USPSTF announced its recommendation to change PSA screening for prostate cancer to a grade D, advising against screening for asymptomatic men of all ages. In May 2012, following a public comment period, the USPSTF finalized these guidelines without additional modification. Although the draft and finalized recommendations received widespread attention from the news media, less is known about the response to these guidelines by individuals and groups on social media.

To gauge public and media perception, we analyzed data from Twitter, a popular microblogging service with over 200 million active users who collectively “tweet” 500 million messages per day, [3] or roughly 5,800 tweets per second. The content of tweets has been used to predict stock market fluctuations, [4] anticipate conflicts and social unrest, [5] and document levels of regional and national happiness. [6, 7] There has also been increased use of Twitter in health care research, [8–12] and reports suggest that more individual healthcare decisions are being made online. [13] We hypothesized that there would be a dramatic rise in Twitter traffic associated with release of the USPSTF recommendations on prostate cancer screening, and that user sentiment on Twitter would be highly polarized. We also hypothesized that the recent initiation of a comment period by the USPSTF, if successful, would lead to an increase in anti-screening Twitter activity by the time of the final guidelines.

Materials and Methods

On Twitter, users can post real-time status updates, known as “tweets,” of 140 characters or less that may express sentiment and/or link to articles using hypertext. These articles can vary widely from those in peer-reviewed medical journals and lay press to individual blogs and advertisements. We used the program Tweet Archivist (www.tweetarchivist.com) to record all publicly searchable tweets using the query “prostate cancer,” during two distinct periods: 1) the 24 hour period following the first appearance of an article about the USPSTF’s draft (October 2011) recommendations and 2) this same time period after the finalized (May 2012) recommendations.

Quantitative Analysis

Tweet rate (number of tweets per hour) was recorded for each study period. As a comparison group, we examined the tweet rate during the four hours prior to the study start point. The type of Twitter user was categorized using a coding system based on information from their profile page. Coding by two independent reviewers (VP, TL) demonstrated excellent agreement when classifying user type (κ_Users = 0.84) in a random sample of 300 tweets.

Qualitative Analysis

Sentiment contained within individual tweets and articles was categorized as either anti-screening or pro-screening if it explicitly agreed or disagreed with the USPSTF report, respectively. Similarly, an article link was considered “anti-screening” or “pro-screening” if it: 1) explicitly agreed or disagreed with the report, respectively, or 2) presented a review of the literature indicating a clear position on the screening debate. Neutral tweets were classified as those which did not take a stand one way or the other, and were further subclassified as to whether they were humorous, confused, inciting discussion, enthusiastic, or simply reporting events. The sentiment expressed in an article link was considered independent of its tweet, unless the user specifically indicated agreement or disagreement with the article. If a user re-tweeted another user’s tweet by simply copying the text of another user’s tweet and tweeting it themselves with or without commentary (as opposed to simply using the “retweet” button without commentary), the tweet and link were counted and coded in the same manner as the original, unless the second user indicated disagreement. If a user instead used the “retweet” button, we did not count this as an additional tweet, since the archiving program did not pick these up as multiple tweets. Use of this coding system in a random sample of 300 tweets and articles demonstrated excellent agreement (κ_Tweets = 0.89, κ_Articles = 0.86) between two independent reviewers (VP, TL). The entire set of tweets was then coded, with disputes being arbitrated by the senior author (DVM).

All statistical analyses were performed in SPSS v19.0 (SPSS Inc., Chicago, IL, USA). Differences were deemed to be statistically significant at a two-sided p value of <0.05. This study was deemed exempt from review by the New York University Institutional Review Board.

Results

During the 24 hours after the USPSTF draft recommendation, there were 3027 tweets about prostate cancer containing 47,270 words. Within 24 hours after the finalized USPSTF recommendation, there were 7385 tweets about prostate cancer containing a total of 116,719 words. After the draft and finalized recommendations, 2042 (67%) and 5357 (73%) of tweets pertaining to prostate cancer focused specifically on the USPSTF, respectively. There were 1591 and 3848 unique users tweeting about the draft and finalized USPSTF recommendations, respectively (Table 1). In both study periods, 4–5% of unique users were physicians.

Table 1.

Unique users tweeting about the United States Preventive Services Task Force’s recommendations on prostate cancer screening. Users were classified utilizing a coding system that demonstrated excellent agreement between two independent reviewers (κ = 0.84).

User Type	Description	Draft	Finalized
		Number (%)
TOTAL		1591	3848
Person		542 (34)	1682 (44)
Physician	Medical doctor	80 (5.0)	161 (4.2)
Health Enthusiast/Professional	Non-medical doctor health professional or self-reported health enthusiast	133 (8.4)	272 (7.1)
Reporter	Reporter working for a news agency or radio station	71 (4.5)	158 (4.1)
Blogger	Person with a blog or listed as a blogger	64 (4.0)	192 (5.0)
Political	Politician, employee thereof, or heavily lists a political affiliation	23 (1.4)	72 (1.9)
Affected	Person affected by prostate cancer, either as a patient, relative, or friend	18 (1.1)	25 (0.6)
Other	Person not otherwise included above	153 (9.6)	802 (20)
Group		576 (36)	1177 (31)
Medical Practice	Group practice, hospital, hospital network, or other center that provides medical services	18 (1.1)	30 (0.8)
Health Business	Business providing health services for profit other than inpatient or outpatient care (i.e. insurance, products)	66 (4.1)	155 (4.0)
Health Organization	Groups with a specific interest in health, not included above (i.e. research foundations, medical journals, nonprofit groups)	31 (1.9)	109 (2.8)
Health News Provider	News provider specializing in delivering health-related news and information to the Twitter community, including automatic newsfeeds	123 (7.7)	244 (6.3)
General News Provider	News provider of general news to the Twitter community, including automatic newsfeeds	185 (11.6)	370 (9.6)
Other	Group not otherwise included above	79 (5.0)	272 (7.1)
Indeterminate	No information, or information provided unable to categorize into above categories	547 (34)	986 (26)

The first article links relating to the draft and finalized recommendations were tweeted just after 3 pm U.S. Eastern Standard Time (EST) on October 6, 2011 and May 21, 2012, respectively. Both reports were associated with a rise in prostate cancer-related tweet rate (Figure), which nearly doubled within two hours. These initial spikes tapered, rose again soon after midnight EST, and increased dramatically again around 8 am EST.

Figure.

Tweet rate before and after the United States Preventive Services Task Force’s (USPSTF) draft (a) and finalized (b) recommendations on prostate cancer screening were released. Red = all tweets meeting search term “prostate cancer.” Blue = Tweets about the USPSTF recommendations.* = when first article about USPSTF recommendations appeared on Twitter.

After the draft recommendations, only 9% of tweets expressed a clear opinion about the report, of which 78% were pro-screening and 22% were anti-screening (Table 2). After the finalized recommendations, only 4% of tweets expressed an opinion: 68% of these were pro-screening, while 32% were anti-screening. The proportion of anti-screening tweets increased from the draft to the finalized study periods (p = 0.03), although the users tweeting in each study period were not identical. As shown in Table 2, subclassification of neutral tweets revealed that the majority were simply reporting the story without humor, confusion, or enthusiasm.

Table 2.

Sentiment expressed in tweets and linked articles about the United States Preventive Services Task Force’s (USPSTF) recommendations on prostate cancer screening.

Category	Definition	Example	Draft	Finalized
			Number (%)
TWEETS			2042	5357
No Opinion			1850 (91)	5141 (96)
Reporting	Simply reporting the story	“U.S. Panel Advises Against Routine Prostate Test”	1626 (88)	4783 (93)
Discussion	Stimulating discussion or indicating controversy	“Will you skip the PSA screening for prostate cancer? why or why not?”	113 (6)	107 (2)
Humorous	Infusing humor	“A panel says most men over 50 don’t need a PSA test for prostate cancer, which is OK since most men over 50 are uninsured due to layoffs”	40 (2)	34 (1)
Confused	Unclear understanding or showing exasperation	“We want to stay healthy - but sometimes we get mixed messages”	39 (2)	60 (1)
Enthusiastic	Showing enthusiasm or drawing special attention	“Must Read! Medical group to say men don’t need prostate cancer screenings, source says […]”	32 (2)	157 (3)
Opinion			192 (9)	216 (4)
Anti-Screening	Agree with USPSTF report, against screening	“Three out of four healthy men with an elevated PSA do not have prostate cancer. Flipping a coin is a better screening test.”	43 (22)	70 (32)*
Pro-Screening	Disagree with USPSTF report, for screening	“My dad is here today because a PSA test detected his prostate cancer at 51. Shame on these people.”	149 (78)	146 (68)
ARTICLES			1754	4528
Neutral	Presents both sides of the issue without taking a stance	“[…] medical group of doctors and advisers vote against routine screenings of prostate cancer […] health groups worried that the move will increase deaths in men at risk of the cancer.”	999 (57)	2258 (50)
Anti-Screening	Reports lopsided evidence against screening or explicitly supports USPSTF decision	“Many physicians have understood these results for several years. I have refused prostate cancer screening since turning 50, because I understood two factors – the test has mediocre characteristics and prostate cancer treatment has significant side effects.”	643 (37)	2122 (47)*
Pro-Screening	Reports lopsided evidence for screening or explicitly opposes USPSTF decision	“Without PSA testing, there is no mechanism for early detection of prostate cancer, leaving thousands of men vulnerable and with no option to protect their health”	112 (6.4)	148 (3.3)

^*significant increase (p<0.05) in proportion of anti-screening content between draft and finalized study periods.

We identified 90 unique article links related to the release of the draft recommendations: 56 (62%) were neutral, 17 (19%) against screening, and 17 (19%) in favor of screening. However, some of these articles were tweeted more frequently than others: 1004 (57%) total articles tweeted were neutral, 643 (37%) were against screening, and 112 (6%) were in favor of screening (Table 2). We identified 105 unique articles related to the release of the finalized recommendations: 47 (45%) were neutral, 30 (29%) anti-screening, and 28 (27%) pro-screening. Like the prior study period, some articles were tweeted more frequently than others: 2258 (50%) were neutral, 2122 (47%) were against screening, and 148 (3%) were in favor of screening. The proportion of anti-screening articles tweeted increased from the draft to the finalized study periods (p<0.01).

Discussion

Our study assessed the immediate impact of the USPSTF draft and finalized recommendations for PSA screening on social media activity. This is one of the first studies to use Twitter in analyzing response to health policy and is also the first study we are aware of that measures public or media reaction to this particular set of screening recommendations. Our study revealed an expected uptick in Twitter traffic, a prevalence of opinionated news media reports, and a surprising lack of opinion and advocacy on the part of Twitter users.

As initially predicted, we observed an increase in prostate cancer-related Twitter activity associated with the announcements of both the USPSTF draft and finalized recommendations and the overwhelming majority of these tweets related to the USPSTF recommendations (Figure). The initial spikes in Twitter traffic were followed by returns to baseline, increases around midnight EST, and a more marked rise at 8am EST, when U.S. Twitter users may have become active again.

Our hypothesis that user sentiment would be highly polarized, however, was largely untrue. Only a small fraction of tweets about the USPSTF draft and finalized recommendations expressed an opinion regarding use of the PSA test (9% and 4%, respectively). Although a small fraction of the overall total, a large majority of opinionated tweets were pro-screening during both study periods (78% and 65%, respectively). This dominant view was espoused by many prostate cancer patients and their relatives as well as many politically minded users who framed the report in the context of healthcare “rationing” and used politically-inclined terms such as “death panels” to describe the USPSTF. While some individuals, including prominent health policy researchers, expressed anti-screening sentiment on social media, we did not observe similar coalitions against screening as appeared in favor of screening.

In stark contrast to the low frequency of user sentiment, nearly half of all articles posted in each study period were opinionated. Despite similar numbers of unique anti- and pro-screening articles posted on Twitter, anti-screening articles (mostly from major news networks) were tweeted much more frequently than pro-screening articles (which were mostly from smaller news networks and groups) in both study periods (37% anti-screening vs. 6.4% pro-screening articles and 47% anti-screening vs. 3% pro-screening, respectively). A prior study demonstrated a similar anti-screening slant among news articles written in the six month period following publication of the conflicting large, randomized controlled trials on prostate cancer screening. [14]

The percentage of both anti-screening tweets and anti-screening articles significantly increased between the draft and finalized recommendations (p = 0.03 and p<0.01, respectively). This suggests that the USPSTF’s public comment period may have helped to recruit additional support for its guidelines, at least as seen through short-term Twitter response. Nevertheless, a majority (65%) of opinionated users opposed the finalized recommendations, despite the opportunity to be convinced by discourse or information provided during the comment period.

Overall, our results seem to suggest several important deficiencies in the advocacy and guideline creation processes. First, despite the heavy media frenzy surrounding these recommendations (≥90 unique articles in each 24 hour study period), there was a tremendous lack of opinionated, user-generated content on social networks like Twitter, a website with over 200 million active users. Second, despite the USPSTF’s institution of a public comment period, it appears that, at least among the small portion of Twitter users advocating opinions, that the USPSTF was unsuccessful in tempering public opposition. This suggests that, in addition to increased advocacy by users on Twitter, there is significant progress still to be made by professional guidelines groups in terms of educating and convincing the public. This can be achieved in a number of ways.

The USPSTF may be able reign in support through more effective messaging. While the guideline currently reads “The USPSTF recommends against PSA-based screening for prostate cancer,” linked articles on Twitter contained quotes from members of the USPSTF emphasizing several important nuances, such as shared decision-making and suggestions that the new guidelines may not apply to high-risk patients or those with urinary symptoms. These nuances are clarified further on the USPSTF website, but only a slim minority of users in our sample posted links directly to that website (<2%). Thus, it is unclear to what extent the general public was made aware of these critical nuances, especially given that many users in our study expressed confusion towards the recommendations.

In the past, visibility of new guidelines was more heavily reliant on traditional media outlets such as newspapers and news networks. Social media platforms including Twitter represent another important source of visibility moving forward. Despite the blast of Twitter activity surrounding the USPSTF recommendations, <10% of tweets expressed an opinion on the subject of PSA testing. Given the large volume of stakeholders, laypersons, and news groups present on social networks, this represents a significant missed opportunity for policy advocacy. For example, the results of recent research by the American Urological Association Social Media Work Group demonstrate that a growing number of urologists now have accounts on social networks, including 36% of surveyed urologists in an e-mail poll. [15] Social media networks may therefore provide a key platform for engagement and involvement of stakeholders regarding new prostate cancer guidelines, which could further increase clarity and transparency. This is in line with the USPSTF Transparency and Accountability Act of 2013, which was introduced to the United States House of Representatives, including a series of changes to increase stakeholder involvement. [16] Our results indicate that Twitter users are ready to further expand upon this: >100 users in both study periods attempted to stimulate discussion directly on Twitter.

Of note, another group performed an analysis of Twitter feeds in the United Kingdom concerning reforms of the English National Health Service. [12] They demonstrated similar spikes in Twitter traffic associated with major changes or events relating to health reform and demonstrated polarized sentiment against policy change. Relative strengths of their design include their ability to archive tweets over almost one year and their development of an index useful for classifying individual Twitter user impact. However, their sentiment analysis was limited to only 200 random tweets after each of three time points. Our qualitative sentiment analysis was much more extensive, including >10,000 consecutive tweets as well as a sub-analysis of neutral tweets for finer detail of sentiment. We also examined >5,000 user profiles to classify user type and analyzed linked articles for bias in order to assess media reporting on Twitter.

Our study has a number of strengths. Twitter users represent a diverse global community. [17] Furthermore, Twitter is accessible via smartphone and tablet, increasing the feasibility of participation because of its ease. [18] Such Twitter posts provide real-time data, eliminating recall bias and potentially providing a more candid picture of user sentiment than would retrospective interviews. Results from previous studies using Twitter in health care have consistently arrived at the same conclusions as studies employing other validated metrics. [11, 19, 20] We were also able to categorize user types, finely detail the hourly rise and fall of Twitter activity, and further characterize attitudes beyond pro-screening, anti-screening, and neutral.

Our study has several limitations. It is possible that our results are not generalizable to the broader Twitter population or to the population as a whole. In fact, reactions on Twitter may be more extreme than other measures of public opinion, especially in response to political events. [21] However, these vocal advocates often reflect an important group in terms of policy change. Additionally, there is no way to obtain all tweets on a given subject in a set timeframe; searching Twitter only obtains a random sample of tweets. The fact that it was random, however, avoids concerns about selection bias. Additionally, our study period of 24 hours may not be representative of the full sentiment regarding screening as a whole. Twitter use fluctuates day to day and there may be daily variations in public opinion. In addition, given that Twitter use in different time zones likely peaks at different times of the day and week, our study period may have underrepresented international viewpoints. However, Twitter reports that over 50% of its users are active daily, [22] suggesting that we captured a large proportion of users. Furthermore, it would have been useful to examine a longer duration of tweets prior to the USPSTF guidelines to more precisely estimate the baseline frequency of prostate cancer-related Twitter activity. In this study, however, tweets were recovered in real-time and Twitter provides public access to a finite number of tweets on any given search query. Finally, we did not assess changes in opinion over time among a particular user category, which is an interesting subject for future study.

In conclusion, the present study demonstrates the utility of social media as a source of mixed methods data on the impact of public health decisions. Our findings suggest that a large majority of opinionated users on Twitter outwardly supported PSA screening, both before and after the USPSTF public comment period. Nevertheless, tweets expressing an opinion represented a relatively small fraction of our sample, suggesting a missed opportunity for advocacy in this important public forum. As social media continues to expand globally, policy makers should systematically use tools like Twitter to further expand public visibility and interaction with key stakeholders.