A Randomized Trial of Social Media From Circulation

Abstract

Background—

Medical journals use social media to distribute the findings of published articles. Whether social media exposure to original articles improves article impact metrics is uncertain.

Methods and Results—

Articles were randomized to receive targeted social media exposure from Circulation, including postings on the journal’s Facebook and Twitter feeds. The primary end point was 30-day article page views. We conducted an intention-to-treat analysis comparing article page views by the Wilcoxon Rank sum test between articles randomized to social media as compared with those in the control group, which received no social media from Circulation. Prespecified subgroups included article type (population/clinical/basic), US versus non-US corresponding author, and whether the article received an editorial. Overall, 243 articles were randomized: 121 in the social media arm and 122 in the control arm. There was no difference in median 30-day page views (409 [social media] versus 392 [control], P=0.80). No differences were observed by article type (clinical, population, or basic science; P=0.19), whether an article had an editorial (P=0.87), or whether the corresponding author was from the United States (P=0.73).

Conclusions—

A social media strategy for a cardiovascular journal did not increase the number of times an article was viewed. Further research is necessary to understand and quantify the ways in which social media can increase the impact of published cardiovascular research.

Medical journals use social media, including blogs and commercial platforms such as Facebook and Twitter, to share medical information. Social media attention after the publication of original research has previously been shown to correlate with the subsequent citation rate of the article.^1–4

Methods

Overall Design

At the time of publication, original articles were randomized to receive either targeted social media exposure from Circulation, including postings on the Facebook and Twitter accounts of Circulation, or no social media exposure from Circulation. Randomization occurred in a blocksize of 4 using envelope randomization on a per-article basis. We excluded all articles that received a national American Heart Association press release (n=11) because of their wide media attention and inability to have a true control group, articles that were presented at the American Heart Association Scientific Sessions 2013 that had a simultaneous publication (n=6), and articles with an accompanying Circulation author profile (n=26). Thus, of 286 articles published during the trial period (Sept 10, 2013 through August 12, 2014), 243 (84.9%) were randomized.

Intervention

The intervention consisted of Facebook and Twitter posts through the official Circulation social media accounts. Briefly, a team of 3 associate editors drafted posts in advance of original article publication. Posts appeared online Monday through Thursday of each weekly journal cycle to be contemporaneous with the formal publication of each article. The Circulation Facebook account (https://www.facebook.com/pages/Circulation) began in March 2012, and currently has >28 000 followers. The Circulation Twitter feed (@CircAHA) began at the same time. Posts were written in simple English to convey the main point of each article with a toll-free link to the full-text version of the article. Whenever possible, a key figure was included to convey the main point of the article. The Circulation social media campaign additionally includes image challenges, ECG challenges, postings of American Heart Association (AHA) scientific statements and guidelines, and Circulation author profiles; these features were not part of the randomized trial. The trial began on September 10, 2013, and ended with articles published August 12, 2014. At the time the trial began, the Circulation Facebook site had 16 215 followers and the Twitter feed had 2219 followers. By the time the trial was completed, the Facebook site had 28 177 followers and the Twitter feed had 4759 followers.

Characteristics of the Circulation Facebook followers were abstracted on September 20, 2014, at www.facebook.com/pages/Circulation. Characteristics of the Circulation Twitter followers were obtained using Twitter Analytics (www.analytics.twitter.com) and were accessed on September 20, 2014. Circulation Web site user analytic data were accessed on September 21, 2014.

Primary End Point

The primary end point was unique 30-day article page views (inclusive of all HTML and pdf downloads and abstract views). Data were abstracted using Google analytics through the Circulation account. Article page views are 1 component of altmetrics, a growing field of article-level metrics that have been proposed as an alternative way of quantifying journal impact, which is traditionally expressed as the journal impact factor. Altmetrics track the impact of a journal article beyond citations and additionally allow for the tracking of individual articles published within a given journal.⁵ We selected 30-day page views as a way of expressing short-term impact because of previous research that has focused on its utility. For example, the distribution of tweets after original article publication has been explored over a 30-day window. These findings demonstrate that tweets follow a Pareto distribution, with the majority of tweets occurring on the day the article is published (43.9%) or on the next day (15.9%) with rapid decay thereafter.¹ Subsequent to this, we compared the correlation between 7- and 30-day page view data in a subset of articles from Circulation (n=19) and found it to be high (r=0.96). Finally, online page views have previously been shown to correlate with subsequent article citations.⁶

Statistical Analysis

We conducted an intention-to-treat analysis (all randomized articles) comparing the distribution of 30-day article page views by the Wilcoxon Rank sum test (a nonparametric test, used because of the skewness in the data) between the articles randomized to social media as compared with those in the control group, which received no social media from Circulation. This was the primary analysis. Prespecified subgroups included article type (population/clinical/basic science), US versus non-US corresponding author, and whether the article received an editorial. We also examined whether season of publication was associated with the intervention. The subgroup analysis was carried out to assess consistency across subgroups; assessment of subgroup-by-randomized group interaction on 30-day page views was carried out using rank analysis of covariance with model effects of subgroup, randomized group, and the subgroup-by-randomized group interaction.⁷ Within subgroup comparisons are reported only when indicated via a significant interaction finding. Given the skewed nature of the data, results are reported as both means and medians. The difference between randomized groups and its 2-sided 95% confidence interval are presented overall and within each subgroup, calculated using the Hodges–Lehmann approach.⁸ All statistical testing was conducted using SAS version 9.3; all P values are 2-sided and are considered significant at a 0.05 level of significance.

Assuming a sample size of 119 articles in each group, we had 90% power to detect a difference of 20% and 80% power to detect a difference of 17% at a 2-sided alpha of 0.05 in the primary end point for social media over the control group, assuming a mean 30-day page view of 560 for the control group and a standard deviation of 265 views for both randomized groups.

Results

Study Sample Characteristics

Overall, 243 articles were randomized: 121 in the social media arm, and 122 in the control arm. The characteristics of the articles by randomization status are shown in Table 1. There were no differences between trial arms with respect to article type (clinical, basic science, population science), corresponding author location (United States versus outside of the United States), season of the publication, or whether the article received an editorial (P≥0.44). Among the articles randomized to social media, 28.3% were posted on a Monday, 37.5% on a Tuesday, 29.2% on a Wednesday, and 5.0% on a Thursday on the Circulation Facebook or Twitter feeds.

Table 1.

Study Sample Characteristics by Trial Arm

	Social Media Arm (n=121)	No Social Media (n=122)	P Value
Article type
Clinical	39.7 (48)	42.6 (52)	0.68
Basic	30.6 (37)	32.8 (40)
Population	29.8 (36)	25.0 (30)
Corresponding author
US	47.1 (57)	48.4 (59)	0.90
Non-US	52.9 (64)	51.6 (63)
Season of publication
Fall	25.6 (31)	26.2 (32)	1.0
Winter	28.9 (35)	29.5 (36)
Spring	25.6 (31)	25.4 (31)
Summer	19.8 (24)	18.9 (23)
Editorial
Yes	44.6 (54)	39.3 (48)	0.44
No	55.4 (67)	60.7 (75)
Day
Monday	28.3 (34)	NA	NA
Tuesday	37.5 (45)
Wednesday	29.2 (35)
Thursday	5.0 (6)

Data are shown as percentages with (n) for dichotomous data and medians with 25^th and 75^th percentiles for continuous data. NA indicates not available.

Demographic Characteristics of the Circulation Facebook Page, Twitter Feed, and Circulation Subscribers

Available demographics for the Circulation Facebook and Twitter feed followers are shown in Table I in the online-only Data Supplement, along with demographic information regarding Circulation Web site visitors. Facebook and Twitter followers were predominantly male. The majority of Facebook followers were between 25 to 34 years of age. English was the predominant language spoken among Facebook followers, but accounted for only slightly more than half of the languages spoken among all followers.

Primary Trial Results and Prespecified Subgroup Analyses

Cumulative distribution curves (Figure 1) demonstrate the distribution of articles in the social media as compared to the control arm, as well as by article type. Overall, there was no difference in median 30-day page views among articles assigned to social media (409) as compared with no social media (392, P=0.80; Table 2). In prespecified subgroup analyses, there was no treatment-by-subgroup interaction by article type (P=0.19), geographic location of the corresponding author (P=0.73), season of publication (P=0.61), or whether the article had an editorial (P=0.87). Results are graphically displayed in Figure 2 and Figure I in the online-only Data Supplement.

Figure 1.

Cumulative distribution of page views in the (A) overall sample, (B) basic science articles, (C) clinical articles, and (D) population science articles. SM indicates social media.

Table 2.

Thirty-Day Page Views: Overall and Stratified by Study Sample Characteristics

	Social Media Arm		No Social Media		P Value Assessing Interaction of Treatment-With-Subgroup
	Mean (SD)	Median (25th/75th)	Mean (SD)	Median (25th/75th)
Overall	526.0 (354.3)	409 (297/619)	523.7 (365.4)	392 (290/645)	NA
Article type
Clinical	640.2 (326.7)	553 (401.5/803.5)	696.7 (447.3)	596 (384.5/923)	0.19
Basic	334.7 (103.6)	327 (267/383)	292.9 (99.9)	279 (210.5/343)
Population	570.3 (466.2)	412.5 (276.5/677)	531.7 (254.3)	484 (371/645)
Corresponding author
US	524.2 (406.5)	393 (293/604)	500.4 (278.9)	402 (249/713)	0.73
Non-US	527.6 (303.7)	419 (299.5/713.5)	545.5 (432.2)	385 (314/613)
Season of publication
Fall	537.9 (284.7)	416 (310/684)	608.2 (466.8)	448.5 (340.5/707.5)	0.61
Winter	534.0 (426.6)	400 (253/611)	490.7 (353.5)	383 (285.5/594.5)
Spring	434.9 (270.4)	335 (255/442)	492.0 (309.4)	388 (243/645)
Summer	616.5 (406.0)	450.5 (355.5/803.5)	500.7 (289.4)	387 (250/642)
Editorial
Yes	577.1 (328.6)	470.5 (355/777)	602.3 (415.2)	473.5 (350/722)	0.87
No	484.8 (371.0)	383 (280/532)	472.8 (321.9)	381.5 (250/599)

NA indicates not available.

Figure 2.

Forest plots displaying the results by trial arm in the overall study and by subgroups. SM indicates social media.

Thirty-Day Page Views by Article Characteristics

The overall study sample characteristics and associated 30-day page view data are presented in Table 3. Clinical articles had the highest number of 30-day page views (median 569), followed by population science articles (median 445 page views); basic science articles had the fewest 30-day page views (median 300, P<0.0001). As expected, articles with an accompanying editorial had more 30-day page views (median 470.5) compared with those without an editorial (383, P=0.0009). Page views categorized by geographic location of the corresponding author did not differ as to whether the author was from the United States (median page views 397.5) or not (median page views 406, P=0.45). Thirty-day page views were similar across seasons (P=0.07) and by day of the week (P=0.24).

Table 3.

Overall Study Sample Characteristics

	Mean (SD)	Median (25th/75th)	P Value for Within Group Comparison
Article type
Clinical	669.6 (393.1)	569 (388.5/828)	<0.0001
Basic	313.0 (103.2)	300 (234/379)
Population	552.8 (382.4)	445 (296/645)
Corresponding author
US	512.1 (346.2)	397.5 (283.5/665.5)	0.45
Non-US	536.5 (371.6)	406 (308/627)
Season of publication
Fall	573.6 (386.5)	416 (318/688)	0.07
Winter	512.1 (389.0)	385 (271/610)
Spring	463.5 (289.6)	371.5 (251/594)
Summer	559.8 (354.8)	439 (331/710)
30-day page views	524.9 (359.2)	400 (296/642)	NA
Editorial
Yes	589.0 (370.2)	470.5 (344/743)	0.0009
No	478.5 (344.9)	383 (271/557)
Day
Monday	573.9 (449.3)	410.5 (310/619)	0.24
Tuesday	460.6 (296.2)	358 (271/532)
Wednesday	557.5 (334.1)	434 (297/777)
Thursday	594.0 (272.9)	581.5 (359/684)

Data are shown as percentages with (n) for dichotomous data and means, standard deviations, and medians with 25^th and 75^th percentiles for continuous data. NA indicates not available.

Discussion

Principal Findings

We have conducted a randomized, controlled trial of social media and its effect on a journal altmetric: 30-day page views. We found no effect of a social media intervention on 30-day page views in the journal Circulation. Further, in all prespecified subgroup analyses, we similarly found no effect of a social media intervention.

In the Context of the Current Literature

To the best of our knowledge, this is the first randomized, controlled trial to test whether a social media intervention increases the number of short-term page views of original biomedical research. A body of literature exists studying whether altmetrics, which consist of social media mentions, are correlated with traditional journal impact citations. For example, 1 study examined tweets between 2008 to 2011 based on articles published in the Journal of Medical Internet Research. The authors found a correlation between tweets the first 3 days after publication and ultimate citations that ranged from 0.42 to 0.72, with tweets explaining up to 27% of the variation in citation behavior.¹ Other studies have confirmed these findings.⁴ However, similar to the limitations of observational science, it is likely that tweets or Facebook mentions serve as early indicators of the impact of an article, because articles of greatest impact will likely receive the most social media attention. Thus, these findings cannot be used to infer that the social media attention per se contributes to the ultimate impact of original published research.

Another study (Allen et al⁹) examined the impact of social media and blogging on 16 articles published over a 5-year window in the area of clinical pain sciences. Researchers selected articles based on their potential interest to their specific scientific community and used a crossover design to test whether featuring an article in social media increased its page views up to 5 years after the article was published. The researchers found that articles promoted via social media had 3-fold higher page views (18 versus 6 per/d) the week after the article was featured in social media (P<0.05), but there was no effect on citations 1 year later.⁹ There are several potential reasons for the differences in this previously published study as compared with the present findings. First, this study was small (n=16 articles) and focused on topics thought to be of greatest interest to the study audience. In contrast, we used a randomized design that was adequately powered to detect a modest effect. Second, this study did not focus on the impact of social media with simultaneous publication, when the potential impact is greatest. Finally, this study used a crossover design in which each article served as its own control, whereas our study used a rigorous randomized design to reduce the possibility of unmeasured confounding.

Implications

These findings suggest that a social media intervention in a traditional cardiovascular journal setting may not increase the number of times that an article is accessed and viewed in the first 30-days after publication. However, our social media postings were comprehensive in that they focused on the main message of the article and included a key figure from the article. Thus, it is possible that social media users did not find it necessary to access the full article and therefore experienced increased awareness of the article but not online access of the primary source. This raises the possible concern that social media could reduce the potential reach of original published research as demonstrated by altmetrics. However, it is important to note that articles randomized to social media did not have lower 30-day page views, suggesting that this concern is likely not valid. These findings raise the interesting question of whether we may have reached a nonoverlapping audience as compared with the traditional readership of Circulation. Indeed, the demographics of our social media followers as compared with the Web site readership of Circulation are qualitatively different. Further research should address whether a social media campaign in a journal with a larger social media reach and more intensive campaign would be more successful than one with a more limited following. However, it is important to note that a high-impact journal in which published original research is covered by the media on a routine basis would have substantial social media exposure regardless of a randomized intervention, resulting in concern for the validity of a study in this setting. Our primary end point was a journal altmetric, which is a newer way of quantifying scholarly reach. However, altmetric end points, and in particular 30-day page views, may not be the optimal way of capturing the effect of social media exposure. Further investigation should focus on the best way to quantify the dissemination of medical information through social media. Finally, further research is necessary to determine what is the most effective way of using social media in the setting of traditional subspecialty journals.

Strengths and Limitations

Strengths of our study include a rigorous trial design to test whether social media exposure of original published research increases 30-day page views. We used articles published in the journal Circulation over an 11-month period. All original research with press releases was excluded. Some limitations warrant mention. Our social media intervention was of modest reach, with ≈28 000 Facebook followers and 4800 Twitter followers at the time of study completion. Thus, we cannot generalize these findings to medical journals with larger social media campaigns. It is unclear whether these findings are generalizable to other medical subspecialty journals or general medical journals. Many of the published articles may not have mainstream appeal to typical users of social media. However, the purpose of our trial was to test whether a social media strategy can increase the reach of published articles in Circulation. Articles with press releases were excluded from randomization. It is possible that these would have been of greatest interest to social media users. However, the small number (n=11) of excluded articles is unlikely to have markedly changed the results of our trial. We note that the standard deviation of 30-day page views was large, indicating substantial variability of article reach. The primary end point, 30-day page views, is an altmetric and not a standard journal metric; however, page views have been correlated to journal citations.⁶ Finally, we focused on only 1 specific altmetric: articles viewed, because this has been previously correlated to article citations,⁵ and article access metrics have been found to be most closely related to ultimate citation indices as compared with other journal altmetric indices.¹⁰ Other metrics, including discussions, saves, and recommendations, were not assessed.

Conclusions

These results indicate that a social media strategy for a cardiovascular journal does not increase the number of times an article is viewed. Further research is necessary to understand the ways in which social media can increase the reach of published cardiovascular research.

CLINICAL PERSPECTIVE.

Social media is commonly used by medical journals to distribute the findings of published articles. However, it is uncertain whether the use of social media in this way increases the dissemination of original published articles. To test this question, we performed a randomized trial of original articles published in Circulation. Articles were randomized to receive targeted social media exposure from Circulation, including postings on the journal’s Facebook and Twitter feeds as compared with control articles, which received no social media exposure from Circulation. Overall, 243 articles were randomized: 121 in the social media arm and 122 in the control arm. There was no difference in 30-day page views (409 [social media] versus 392 [control], P=0.80). No differences were observed by article type, whether an article had an editorial, or whether the corresponding author was from the United States. A social media strategy for a cardiovascular journal did not increase the number of times an article was viewed. Further research is necessary to understand the ways in which social media can increase the impact of published cardiovascular research.

It is, however, likely that articles of greatest impact receive the most social media attention. Therefore, it is not possible to determine from this earlier work whether social media attention increases the impact or reach of a journal article, or whether it merely serves as an early marker of importance.

The journal Circulation initiated a social media campaign in 2012 to increase the reach of published content. The social media campaign consists of a blog, a Facebook page, and a Twitter feed. To test the impact of this social media strategy on original published research, we conducted a randomized trial of social media with a focus on short-term impact. We hypothesized that social media exposure to original research would increase the 30-day page views of original research published in Circulation.