Abstract
Alternative metrics are unique bibliometrics comprising social, news, and other sources of media outside of traditional academic citations. Some have suggested that these metrics can complement traditional metrics of research impact, including public engagement with research. The COVID-19 pandemic provides a unique opportunity to study alternative metrics and the dissemination of scientific research given the heightened academic and public interest. This study analyzed Altmetric Attention Scores for the top 25 publications on COVID-19 and the top 25 non–COVID-19 publications in 2020. There were significantly higher levels of social attention scores across multiple metrics for COVID-19 articles than for non–COVID-19 articles for that year. There was a slightly higher goodness of fit between Altmetric Attention Scores and academic citations for COVID-19 publications than for non–COVID-19 publications, although trendline differences were not significant. These results suggest that researchers should be aware that their studies can become highly visible on publicly available social and news media platforms, especially during events of high interest (such as a global pandemic).
Keywords: Alternative metrics, altmetric, bibliometrics, COVID-19
Alternative metrics, or measurements of an article’s impact outside of traditional academic and journal citations, have recently become an area of interest in bibliometrics. These alternative metrics include article references in social and news media, such as Twitter, Facebook, Wikipedia, YouTube, and Reddit (a social news aggregator).1,2 While not a direct measurement of impact in academic organizations, as measured by traditional citations, these alternative metrics can provide insight into visibility and possible public engagement with research.3 Several organizations, including Plum Analytics, Impact Story, PLoS Impact Explorer, and Altmetric, collect and provide information on alternative metrics for researchers.4 Since these alternative metrics can reflect community engagement with research, some observers have suggested that alternative metrics should be considered alongside traditional measurements for assessing the impact of research.4 However, the correlation between alternative metrics and citations is unclear at present; some studies identify a correlation between alternative metrics and academic citations,5–14 although often weak to moderate, and others find no correlation at all.15–18 Furthermore, public response to COVID-19 information (or misinformation) can influence behaviors that affect virus transmission.19–21 This study used publicly available data on these metrics to examine how the general public and academic community responded to academic publications during the COVID-19 pandemic.
METHODS
Publicly available alternative metric data for the 2020 Altmetric Top 100 articles were obtained from Altmetric (https://www.altmetric.com/), a data science company specializing in alternative metrics. The Altmetric Attention Score (AAS) was used as a composite measure of article attention via alternative metrics. Academic citations were obtained from Scopus (https://www.scopus.com) and Web of Science (https://www.webofknowledge.com), databases that track academic citations. Only citations for the year 2020 were included. Comparison between COVID-19 and non–COVID-19 metrics were performed using two-sample t tests assuming unequal variances. A two-tailed t test was used to compare trendline slopes. Statistical analysis was performed and graphs were produced using Microsoft Excel 2016 for Windows.
RESULTS
Twenty-six COVID-19 articles and 74 non–COVID-19 articles were listed in the 2020 Altmetric Top 100 articles. The 25 most highly visible articles for COVID-19 and non–COVID-19 articles, as measured by AAS, were compared (Table 1). The AAS scores ranged more widely for COVID-19 (607–34,775) than non–COVID-19 (2,101–10,681) articles. COVID-19 articles had consistently higher mean and median alternative and academic metrics across all categories. Statistically significant differences were noted between the COVID-19 and non–COVID-19 publications in all metrics (P < 0.014–0.048) except for Reddit, News, and Video categories. Most metrics had a large standard deviation, indicating the heterogeneity of the data. The correlations between the AAS and the number of Scopus citations are presented in Figure 1. Scopus citations were used for these figures, as this metric solely includes academic journal citations, as opposed to Google Scholar, which includes a wider range of publication types. For academic citation curators, only differences between Google Scholar and Web of Science for non–COVID-19 articles were significant (P < 0.044). All papers were published in academic journals. The most common study types were epidemiological/public health for the COVID-19 papers and ecological for the non–COVID papers (Table 2).
Table 1.
Alternative and academic citation metrics for top 25 COVID and non–COVID-19 articles
| COVID-19 articles |
Non–COVID-19 articles |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Total | Mean | Median | 25th | 75th | Total | Mean | Median | 25th | 75th | |
| Alternative metrics | ||||||||||
| AAS | 222,132 | 8,885 ± 10,664 | 4,488 | 2,496 | 7,292 | 104,180 | 4,167 ± 2,222 | 3,209 | 2,826 | 4,863 |
| News | 13,579 | 543 ± 768 | 254 | 115 | 468 | 6,819 | 273 ± 280 | 166 | 82 | 386 |
| Blog | 1,251 | 50.0 ± 55.7 | 32 | 13 | 70 | 601 | 24.0 ± 18.7 | 22 | 14 | 29 |
| 383,996 | 15,360 ± 26,005 | 4,265 | 1,825 | 9,401 | 110,263 | 4,411 ± 4,066 | 4,001 | 867 | 6,741 | |
| 615 | 24.6 ± 35.9 | 8 | 3 | 26 | 185 | 7.4 ± 8.0 | 4 | 3 | 11 | |
| 266 | 10.6 ± 14.7 | 4 | 1 | 14 | 135 | 5.4 ± 8.9 | 3 | 1 | 6 | |
| Video | 109 | 4.4 ± 6.9 | 1 | 0 | 6 | 46 | 1.8 ± 3.0 | 1 | 0 | 3 |
| Academic | ||||||||||
| Scopus | 6,489 | 260 ± 548 | 42 | 20 | 216 | 120 | 5.0 ± 6.6 | 2 | 0 | 7.5 |
| Web of Science | 5,440 | 227 ± 468 | 26 | 20.5 | 160.8 | 100 | 4.5 ± 5.7 | 2 | 1 | 5 |
| Google Scholar | 14,736 | 589 ± 1162 | 120 | 51 | 376 | 280 | 11.2 ± 14.6 | 6 | 1 | 16 |
AAS indicates Altmetric Attention Score.
Figure 1.
Altmetric Attention Score (AAS) vs. Scopus citations for (a) top 25 non–COVID-19 articles and (b) top 25 COVID-19 articles for 2020.
Table 2.
Categorization of study types
| Study type | COVID | Non-COVID |
|---|---|---|
| Behavior analysis | 0 | 2 |
| Commentary | 5 | 3 |
| Clinical trial/case control | 2 | 0 |
| Computational/bioinformatics | 3 | 7 |
| Design | 0 | 1 |
| Ecological | 0 | 6 |
| Epidemiological/public health | 7 | 0 |
| Essay | 0 | 3 |
| Laboratory science | 4 | 2 |
| Theoretical | 1 | 0 |
| Review | 1 | 0 |
| Survey | 2 | 1 |
DISCUSSION
The COVID-19 pandemic has created important health and economic concerns throughout all levels of society, including academic and government organizations and the public. COVID-19 articles had a slightly higher linear correlation between AAS and Scopus citations (R2 = 0.15; P < 0.054) than non–COVID-19 papers (R2 = 0.04; P < 0.34), but neither trendlines nor the differences between the slopes of the trendlines (P < 0.38) were significant. This supports previous studies that have not found a significant correlation between alternative metrics and academic citations.22,23 The significant difference in AAS between COVID-19 and non–COVID-19 articles suggests that topics of high visibility, such as COVID-19, can generate a more consistent relationship between alternative metrics and academic citations. This may be due to increased public and academic interest in COVID-19 or more aggressive promotion of COVID-19 research by researchers, academic institutions, and news media. However, the limited correlation between AAS and citations suggests a small effect of alternative metrics on citations and may reflect the engagement of two distinct populations, i.e., the general public and academia. We also note that alternative metrics typically provide an immediate gauge of impact whereas academic citations take much longer to accrue.
While alternative metrics do not necessarily indicate public involvement (articles could be shared on these platforms by researchers and academic institutions), it does highlight the potential for increased visibility of these articles. Social and news media can provide much more direct and accessible ways for the public to engage with scientific literature. Researchers should also be aware that social and public health issues can significantly increase visibility on social and news media platforms. Availability to a wide audience with varying levels of scientific literacy may increase the possibility that research will be misinterpreted.
References
- 1.James A, Raux M.. Altmetrics scores: what are they? Anaesth Crit Care Pain Med. 2020;39(3):443–445. doi: 10.1016/j.accpm.2020.03.015. [DOI] [PubMed] [Google Scholar]
- 2.IONOS . Somewhere between news and memes: what is Reddit? https://www.ionos.com/digitalguide/online-marketing/social-media/what-is-reddit/. Published 2019. Accessed June 20, 2021.
- 3.Trueger NS, Thoma B, Hsu CH, Sullivan D, Peters L, Lin M.. The Altmetric score: a new measure for article-level dissemination and impact. Ann Emerg Med. 2015;66(5):549–553. doi: 10.1016/j.annemergmed.2015.04.022. [DOI] [PubMed] [Google Scholar]
- 4.Melero R. Altmetrics—a complement to conventional metrics. Biochem Med (Zagreb). 2015;25(2):152–160. doi: 10.11613/BM.2015.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Costas R, Zahedi Z, Wouters P.. Do “Altmetrics” correlate with citations? Extensive comparison of Altmetric indicators with citations from a multidisciplinary perspective. J Assoc Inf Sci Technol. 2015;66(10):2003–2019. doi: 10.1002/asi.23309. [DOI] [Google Scholar]
- 6.Thelwall M, Haustein S, Larivière V, Sugimoto CR.. Do Altmetrics work? Twitter and ten other social web services. PLoS One. 2013;8(5):e64841. doi: 10.1371/journal.pone.0064841. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Waltman L, Costas R.. F1000 recommendations as a potential new data source for research evaluation: a comparison with citations. J Assoc Inf Sci Technol. 2014;65(3):433–445. doi: 10.1002/asi.23040. [DOI] [Google Scholar]
- 8.Boetto E, Fantini MP, Gangemi A, et al. . Using Altmetrics for detecting impactful research in quasi-zero-day time-windows: the case of COVID-19. Scientometrics. 2021;126(2):1189–1215. doi: 10.1007/s11192-020-03809-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Rosenkrantz AB, Ayoola A, Singh K, Duszak R. Alternative metrics (“Altmetrics”) for assessing article impact in popular general radiology journals. Acad Radiol. 2017;24(7):891–897. doi: 10.1016/j.acra.2016.11.019. [DOI] [PubMed] [Google Scholar]
- 10.Asaad M, Howell SM, Rajesh A, Meaike J, Tran NV.. Altmetrics in plastic surgery journals: does it correlate with citation count? Aesthet Surg J. 2020;40(11):NP628–NP635. doi: 10.1093/asj/sjaa158. [DOI] [PubMed] [Google Scholar]
- 11.Chang J, Desai N, Gosain A.. Correlation between Altmetric score and citations in pediatric surgery core journals. J Surg Res. 2019;243:52–58. doi: 10.1016/j.jss.2019.05.010. [DOI] [PubMed] [Google Scholar]
- 12.Mullins CH, Boyd CJ, Corey BL.. Examining the correlation between Altmetric score and citations in the general surgery literature. J Surg Res. 2020;248:159–164. doi: 10.1016/j.jss.2019.11.008. [DOI] [PubMed] [Google Scholar]
- 13.Nocera AP, Boyd CJ, Boudreau H, Hakim O, Rais-Bahrami S.. Examining the correlation between Altmetric score and citations in the urology literature. Urology. 2019;134:45–50. doi: 10.1016/j.urology.2019.09.014. [DOI] [PubMed] [Google Scholar]
- 14.Dardas LA, Woodward A, Scott J, Xu H, Sawair FA.. Measuring the social impact of nursing research: An insight into Altmetrics. J Adv Nurs. 2019;75(7):1394–1405. doi: 10.1111/jan.13921. [DOI] [PubMed] [Google Scholar]
- 15.Livas C, Delli K.. Looking beyond traditional metrics in orthodontics: an Altmetric study on the most discussed articles on the web. Eur J Orthod. 2018;40(2):193–199. doi: 10.1093/ejo/cjx050. [DOI] [PubMed] [Google Scholar]
- 16.Ruan QZ, Chen AD, Cohen JB, Singhal D, Lin SJ, Lee BT.. Alternative metrics of scholarly output: the relationship among Altmetric score, Mendeley reader score, citations, and downloads in plastic and reconstructive surgery. Plast Reconstr Surg. 2018;141(3):801–809. doi: 10.1097/PRS.0000000000004128. [DOI] [PubMed] [Google Scholar]
- 17.Delli K, Livas C, Spijkervet FKL, Vissink A.. Measuring the social impact of dental research: an insight into the most influential articles on the Web. Oral Dis. 2017;23(8):1155–1161. doi: 10.1111/odi.12714. [DOI] [PubMed] [Google Scholar]
- 18.Peters I, Kraker P, Lex E, Gumpenberger C, Gorraiz J.. Research data explored: an extended analysis of citations and Altmetrics. Scientometrics. 2016;107(2):723–744. doi: 10.1007/s11192-016-1887-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Allington D, Duffy B, Wessely S, Dhavan N, Rubin J.. Health-protective behaviour, social media usage and conspiracy belief during the COVID-19 public health emergency. Psychol Med. 2021;51(10):1763–1769. doi: 10.1017/S003329172000224X. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Romer D, Jamieson KH.. Patterns of media use, strength of belief in COVID-19 conspiracy theories, and the prevention of COVID-19 from March to July 2020 in the United States: survey study. J Med Internet Res. 2021;23(4):e25215. doi: 10.2196/25215. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Clark C, Davila A, Regis M, Kraus S.. Predictors of COVID-19 voluntary compliance behaviors: an international investigation. Glob Transit. 2020;2:76–82. doi: 10.1016/j.glt.2020.06.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Chen WMY, Bukhari M, Cockshull F, Galloway J.. The relationship between citations, downloads and alternative metrics in rheumatology publications: a bibliometric study. Rheumatology (Oxford). 2020;59(2):277–280. doi: 10.1093/rheumatology/kez163. [DOI] [PubMed] [Google Scholar]
- 23.Han SC, Kang HJ, Lee WJ, Chung HS, Lee JH.. A bibliometric analysis using alternative metrics for articles in the Annals of Rehabilitation Medicine. Ann Rehabil Med. 2020;44(2):158–164. doi: 10.5535/arm.2020.44.2.158. [DOI] [PMC free article] [PubMed] [Google Scholar]