INTRODUCTION
Altmetric Attention Score (AAS) is an emerging tool that measures the online presence of published articles. AAS represents a sum of article mentions on social media, e.g., Twitter, and news outlets (Digital Science, London, UK). AAS is increasingly used by journals as an integral article metric. While some studies have assessed the correlation between AAS and the number of citations in several disciplines and specialties,1–4 it remains unclear whether AAS correlates with article citations among high-impact general medicine journals.
METHODS
We compiled full-length original articles published in 2014 at the three highest impact factor journals: New England Journal of Medicine (NEJM), the Lancet, and Journal of the American Medical Association (JAMA) (according to Journal Citation Reports® 2017: category “General Internal Medicine”). For each article, the reported AAS was extracted (i.e., a higher score corresponds to increased online presence, with a score of ≥ 20 generally representing a high-performing article compared to its contemporaries). Citation counts in the subsequent 3 years (i.e., 2015–2017) were retrieved by searching Web-of-Science for the corresponding article. Pairwise correlations were done using Spearman’s rank correlation coefficient for non-parametric variables and Pearson’s correlation coefficient for linear variables. A simple linear regression model was used to determine the relationship between log (AAS) and log (total 3-year citations). To account for the clustering effect between the journals, a linear regression with Huber–White cluster sandwich estimator of variance was performed.
RESULTS
We retrieved 551 full-length original research articles: 215 (39%), 182 (33%), and 154 (28%) from NEJM, JAMA, and the Lancet, respectively. The median AAS was 129 (interquartile range 74, 207); the median number of news outlets and twitter mentions were 8 (4, 15) and 49 (27, 95), respectively. The median total number of citations at 3 years was 96 (50, 190). The correlation coefficient for the AAS and 3-year citations was 0.33 (P < 0.001) (Table 1).
Table 1.
Descriptive Characteristics of the Included Articles and the Studied Correlations Between Altmetric Attention Score and Total Citations
| Variable | N = 551 |
|---|---|
| Journal, No. (%) | |
| NEJM | 215 (39%) |
| JAMA | 182 (33%) |
| The Lancet | 154 (28%) |
| Article type, No. (%) | |
| Clinical trial | 291 (52.8%) |
| NEJM | 125 |
| JAMA | 82 |
| The Lancet | 84 |
| Observational study | 229 (42.6%) |
| NEJM | 89 |
| JAMA | 88 |
| The Lancet | 52 |
| Systematic reviews and meta-analysis | 31 (5.6%) |
| NEJM | 1 |
| JAMA | 12 |
| Lancet | 18 |
| Altmetric score, median (IQR) | 129 (74, 207) |
| NEJM | 134 (75, 209) |
| JAMA | 124 (87, 194) |
| The Lancet | 100 (41, 212) |
| News outlet mentions | 8 (4, 15) |
| Twitter mentions | 49 (27, 95) |
| 3-year total citations, median (IQR) | 96 (50, 190) |
| Citations in 2015 | 31 (14, 62) |
| NEJM | 51 (30, 99) |
| JAMA | 16 (9, 35) |
| The Lancet | 25 (13, 47) |
| Citations in 2016 | 34 (18, 68) |
| NEJM | 58 (32, 102) |
| JAMA | 21 (13, 42) |
| The Lancet | 27 (18, 59) |
| Citations in 2017 | 30 (16, 62) |
| NEJM | 45 (25, 102) |
| JAMA | 19 (12, 36) |
| The Lancet | 26 (16, 52) |
| Spearman correlationa, correlation coefficient | |
| Altmetric score and 3-year total citations | 0.33 |
| Clinical trials | 0.32 |
| Observational studies | 0.31 |
| Systematic reviews and meta-analyses | .009b |
| General medicine articles (n = 151) | 0.32 |
| Subspecialty articles (n = 400) | 0.37 |
| Primary care topics (n = 368) | 0.29 |
| Hospital-based topics (n = 183) | 0.39 |
| Altmetric score and first-year citationsc | 0.34 |
| Altmetric score and second-year citations | 0.32 |
| Altmetric score and third-year citations | 0.32 |
| News outlet mentions and 3-year total citations | 0.29 |
| Twitter mentions and 3-year total citations | 0.16 |
| Altmetric score percentile ranks and 3-year total citationsd | |
| Altmetric score in the top 50% | 0.15e |
| Altmetric score in the top 25% | 0.08f |
| Altmetric score in the top 10% | 0.18g |
| Altmetric score in the top 1% | 0.20h |
IQR, interquartile range; JAMA, Journal of the American Medical Association; NEJM, New England Journal of Medicine
aP values for all correlations is < .001
bP value = 0.96
cCitations in 2015
dExcluding 1 outlier article with an Altmetric score of 2902 and only 17 citations
eP value = 0.01
fP value = 0.38
gP value = 0.20
hP value = 0.75
On simple linear regression, log (AAS) was a positive predictor of log (3-year citations) (P < 0.001), and the model was statistically significant (P < 0.001). However, the model only explained 12% of this relationship (R2 = 0.12) (Fig. 1a). The adjustment for article type improved R2 to 0.16, but did not significantly affect the overall model. By accounting for the clustering effect between journals in the regression model, R2 remained 0.12, but the model did not reach statistical significance (P = 0.09) (Fig. 1b). Accounting for the journal impact factor as a continuous variable showed a 0.03 increase in the log (total Citations) for each 1-point increase in the impact factor (P < 0.001).
Fig. 1.
a Scatter plot examining the relationship between log (Altmetric Attention Score) and log (3-year total citations), with a fitted line representing the regression model and 95% confidence interval. b Scatter plot examining the relationship between log (Altmetric Attention Score) and log (3-year total citations) after adjusting for the clustering by journal, with fitted lines representing the regression models and 95% confidence intervals. Red line: The New England Journal of Medicine (NEJM); green line: Journal of the American Medical Association (JAMA); green line: The Lancet.
DISCUSSION
In this cross-sectional analysis of full-length original research articles published in specific high-impact general medicine journals, we demonstrated that AAS poorly correlated with the number of citations in the subsequent 3 years. The correlation strength did not change on subgroup analyses of individual journal, study type, or the individual score components. The correlation between AAS and the total citations over 3 years was further diminished after adjusting for the clustering effect between journals.
Given the increasing interest in the use of social media to promote articles, it is important to understand the impact of new online attention metrics on more conventional article metrics, such as article citations. One study of 16 selected articles in pain medicine found that measures of social media engagement did not correlate with article citations at 1 year.5 Another study examined the influence of tweets on the number of citations for articles published in the Journal of Medical Internet Research and found a correlation between number of tweets within 3 days of publication and future citations.6 A study comparing the correlation of AAS and citations in the 6 PLOS journals showed no significant correlation for PLOS Medicine.4 Our study extends beyond these findings by including the highest impact general medicine journals, which are expected to have the highest AAS for their articles due to wider audience. However, we could not comment on the effect of article accessibility (i.e., open versus closed access) on this association. It is also important to highlight that these results might be of limited generalizability to other subspecialty journals. For example, a similar analysis found that AAS moderately correlated with citations for cardiology articles,3 which might be related to the variation in social media engagement among the different specialties.
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Thelwall M, Haustein S, Larivière V, Sugimoto CR. Do altmetrics work? Twitter and ten other social web services. PLoS One. 2013;8:e64841. doi: 10.1371/journal.pone.0064841. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Barbic D, Tubman M, Lam H, Barbic S. An Analysis of Altmetrics in Emergency Medicine. Acad Emerg Med. 2016;23:251–68. doi: 10.1111/acem.12898. [DOI] [PubMed] [Google Scholar]
- 3.Barakat AF, Nimri N, Shokr M, et al. Correlation of Altmetric Attention Score With Article Citations in Cardiovascular Research. J Am Coll Cardiol. 2018;72:952–953. doi: 10.1016/j.jacc.2018.05.062. [DOI] [PubMed] [Google Scholar]
- 4.Huang W, Wang P, Wu Q. A correlation comparison between Altmetric Attention Scores and citations for six PLOS journals. PLoS One. 2018;13:e0194962. doi: 10.1371/journal.pone.0194962. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Allen HG, Stanton TR, Di Pietro F, Moseley GL. Social media release increases dissemination of original articles in the clinical pain sciences. PLoS One. 2013;8:e68914. doi: 10.1371/journal.pone.0068914. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Eysenbach G. Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res. 2011;13:e123. doi: 10.2196/jmir.2012. [DOI] [PMC free article] [PubMed] [Google Scholar]