Retracted: "Assessing the Dissemination of COVID-19 Articles Across Social Media With Altmetric and PlumX Metrics: Correlational Study"

Abstract

Background

The use of social media assists in the distribution of COVID-19 information to the general public and health professionals. Alternative-level metrics (ie, altmetrics) and PlumX metrics are new bibliometrics that can assess how many times a scientific article has been shared and how much a scientific article has spread within social media platforms.

Objective

Our objective was to characterize and compare the traditional bibliometrics (ie, citation count and impact factors) and new bibliometrics (ie, Altmetric Attention Score [AAS] and PlumX score) of the top 100 COVID-19 articles with the highest AASs.

Methods

The top 100 articles with highest AASs were identified with Altmetric Explorer in May 2020. The AASs, journal names, and the number of mentions in various social media databases of each article were collected. Citation counts and PlumX Field-Weighted Citation Impact scores were collected from the Scopus database. Additionally, AASs, PlumX scores, and citation counts were log-transformed and adjusted by +1 for linear regression, and Spearman correlation coefficients were used to determine correlations.

Results

The median AAS, PlumX score, and citation count were 4922.50, 37.92, and 24.00, respectively. The New England Journal of Medicine published the most articles (18/100, 18%). The highest number of mentions (985,429/1,022,975, 96.3%) were found on Twitter, making it the most frequently used social media platform. A positive correlation was observed between AAS and citation count (r²=0.0973; P=.002), and between PlumX score and citation count (r²=0.8911; P<.001).

Conclusions

Our study demonstrated that citation count weakly correlated with AASs and strongly correlated with PlumX scores, with regard to COVID-19 articles at this point in time. Altmetric and PlumX metrics should be used to complement traditional citation counts when assessing the dissemination and impact of a COVID-19 article.

Introduction

The SARS-CoV-2 virus is the pathogen responsible for the latest global pandemic that has exhausted the global economy and health care system to a degree that has not been seen since the 1918 influenza outbreak. The virus originated from Wuhan, China in December 2019, and as of May 1, 2020, more than 230,000 COVID-19 fatalities have been reported worldwide, which is higher than the number of fatalities reported for both severe acute respiratory syndrome and Middle East respiratory syndrome combined [1,2]. Due to the rapid spread of the virus and the massive number of casualties, there has been a rapid rate of research dissemination across medical journals and social media platforms to provide real-time guidance for understanding the epidemiology, disease characteristics, and clinical management of, and future treatment development for, COVID-19 to all stakeholders who are invested in managing the COVID-19 pandemic [3,4].

In contrast to medical journals, social media can serve as a useful platform for informing the wider general public (ie, both medical professionals and laypeople alike) and disseminating crucial and novel information during this evolving crisis [5,6]. To capture the level of an article’s dissemination across social media (ie, an article’s “online attention”), many metric tools, such as Altmetric and PlumX, have been created [7]. As opposed to the traditional metrics of article dissemination, such as article citation count, metrics that describe article dissemination across social media are known as alternative metrics or “altmetrics.” The Altmetric Attention Score (AAS), which was developed by Altmetric, is a weighted score of the amount of “online attention” a research article has received across social media platforms. This score solely refers to the number of citations, linkouts, and abstract views, and such social media platforms include Twitter, Facebook, Google+, Wikipedia, blogs, and many others [8]. Similarly, PlumX has developed the PlumX Field-Weighted Citation Impact score (ie, PlumX score), which is also a weighted metric score of the level of article dissemination across similar social media platforms, based on the number of citations, linkouts, and abstract views [9].

Given that AASs and PlumX scores use relevant information from social media platforms, including Twitter and Facebook, these scores can be potentially useful adjunctive metric tools for holistically evaluating an article’s impact or effect on a field of research, instead of just evaluating scholarly impact [10]. These impacts or effects include information uptake, information engagement, and the relevance of results. Citation counts and impact factors reflect the number of citations in other articles or journals, whereas altmetrics reflect the instantaneous attention that an article garners among news outlets, blogs, Twitter, Facebook, and other media platforms [4]. Additionally, since paper journals are moving toward internet-based platforms, the development of these new internet-based technologies provides researchers with a new approach to assessing the effect of research [4]. In terms of biomedical research, the use of Twitter to disseminate article information has increased dramatically over the years. This showcases the degree of social media use in the medical community [11]. Previous studies have assessed the utility of new bibliometrics (ie, AASs and PlumX scores) as complements to citation count in various medical fields, but these studies have reported variable results [4,9,10,12,13]. Furthermore, the utility of altmetrics in COVID-19 research has not yet been evaluated.

Given the lack of research, the massive influx of COVID-19 publications since early 2020, and the fact that altmetrics allow for the rapid assessment of an article’s level of dissemination upon publication, altmetrics have the potential to be used complementarily with traditional bibliometrics (eg, article citation), which typically take years to accumulate [8]. The purpose of this study was to determine the utility of adjunctively using AAS and PlumX altmetrics as complements to traditional bibliometrics in the assessment of the 100 most “trending” COVID-19 articles across social media, as determined by Altmetric. Given that both the scientific community and the general public frequently check for new information in journals and social media platforms to gain a better understanding of how to prevent and manage COVID-19 [14], we hypothesized that there might be a significant correlation between AAS and PlumX metrics and citation count. Such a correlation would suggest an alignment between the interests of academic scholars and the general public. The primary objective of this study was to investigate the correlations between both Altmetric and PlumX scores and citation count.

Methods

Altmetric Explorer was used to identify COVID-19 publications from December 2019 to May 2020, by using the PubMed search terms “COVID-19,” “SARS CoV-2,” and “coronavirus.” Articles that were retracted by May 2020 were excluded from analysis. The top 100 articles with the highest AASs were selected from the list of included articles. The number of mentions from the following Altmetric data components were extracted and examined: news mentions, blog mentions, policy mentions, Twitter mentions, Facebook mentions, Wikipedia mentions, Reddit mentions, Mendeley readers, and the number of Dimensions citations [9]. In addition to the extraction of Altmetric data components, we also collected data on the impact factors of the journals that the included articles were published in, article type, and article citation count [8,10]. For traditional citation analysis, article citation counts were found by using the Scopus database. To assess whether the results from the Altmetric analysis could be generalized across multiple social media metric tools, PlumX scores were collected from the Scopus database. AASs, PlumX scores, and citation counts were log-transformed and adjusted by +1 for linear regression, and Spearman correlation coefficients were used to determine correlations. Statistical significance was defined as P<.05.

Results

The majority of articles were published in biomedical journals (Table 1, Textbox 1). Compared to the other journals, the New England Journal of Medicine published the most articles (18/100, 18%). Of the 100 articles, 42 (42%) were original investigations. The article with the highest AAS (AAS=33,828) was a biomechanistic basic science letter that delineated the features of the SARS-CoV-2 genome and provided evidence that SARS-CoV-2 was not constructed in a laboratory (AAS=33,828; citation count=30) [15]. The article with the highest PlumX score was a prospective cohort study that described the clinical characteristics of patients with COVID-19 [16]. The median AAS, PlumX score, and citation count were 4,922.50, 37.92, and 24.00, respectively. All articles were classified as the top 5% in terms of scientific output [10]. This means that in terms of AAS, these articles ranked in the top 5% when compared to more than 15.5 million research publications. In other words, these articles garnered the most attention. A total of 1,022,975 mentions in social media platforms were assessed. The highest number of mentions for the selected articles (985,429/1,022,975, 96.3%) were found on Twitter. Additionally, 99 articles were open access. In total, 9283 Mendeley mentions and 18,011 Dimensions citations were present. A weak positive correlation was observed between AAS and citation count (r²=0.0973; P=.002) (Figure 1). However, a strong positive correlation was observed between PlumX score and citation count (r²=0.8911; P<.001). Stronger positive correlations were observed between the number of Mendeley readers (r²=0.958; P<.001) and citation count, and between the number of Dimensions citations (r²=0.984; P<.001) and citation count.

Table 1.

Characteristics and components of the top 100 COVID-19 articles with the highest Altmetric Attention Scores.

Characteristic		Value
Altmetric Attention Score, median (range)		4922.50 (2841-33828)
PlumX score, median (range)a		37.92 (0-1862.23)
Journal impact factor, median (range)		41.06 (1.29-70.67)
Traditional citation count, median (range)b		24.00 (0-1096)
Number of news mentions, total (range)		32509 (1-2021)
Number of blog mentions, total (range)		2630 (0-131)
Number of policy mentions, total (range)		154 (0-21)
Number of Twitter mentions, total (range)		985429 (1381-84022)
Number of Facebook mentions, total (range)		1138 (0-58)
Number of Wikipedia mentions, total (range)		177 (0-11)
Number of Reddit mentions, total (range)		938 (0-40)
Number of Mendeley Readers, total (range)		9283 (0-2581)
Number of Dimensions citations, total (range)		18011 (0-2233)
Article type, n (%)
	Original investigation	44 (44)
	Correspondence	31 (31)
	Editorial	14 (14)
	Review	4 (4)
	Viewpoint	6 (6)
	Open access	100 (100)
Study design of original investigations (n=44) , n (%)
	Clinical trials	4 (9)
	Prospective/retrospective cohort studies	21 (48)
	Cross sectional	1 (2)
	Case series	7 (16)
	Basic science in vitro/in vivo studies	8 (18)
	Model validation studies	3 (7)

^aIn total, 16 articles had a PlumX score of 0, because they did not have a score on Scopus at the time of this study. Therefore, these articles were not part of the analysis.

^bIn total, 3 articles did not have a citation count on Scopus or PubMed Central at the time of this study. Therefore, these articles were not part of the analysis.

Journals categorized based on journal impact factor. The number of included articles published in each journal are reported.

Journals with an impact factor of >40

• New England Journal of Medicine (articles: n=18)

• The Lancet (articles: n=14)

• Journal of the American Medical Association (articles: n=12)

• Nature (articles: n=2)

• Science (articles: n=6)

Journals with an impact factor of 20-40

• Cell (articles: n=1)

• Nature Biotechnology (articles: n=1)

• Nature Medicine (articles: n=5)

• British Medical Journal (articles: n=2)

• Lancet Infectious Disease (articles: n=3)

• The Lancet Respiratory Medicine (articles: n=2)

Journals with an impact factor of <20

• Annals of Internal Medicine (articles: n=2)

• Cell Research (articles: n=1)

• Morbidity and Mortality Weekly Report (articles: n=8)

• ACS (American Chemical Society) Nano (articles: n=1)

• Proceedings of the National Academy of Sciences of the United States of America (articles: n=2)

• Emerging Infectious Diseases (articles: n=4)

• Biomedicine (articles: n=1)

• Eurosurveillance (articles: n=1)

• Pediatrics (articles: n=1)

• Circulation: Arrhythmia and Electrophysiology (articles: n=1)

• International Journal of Antimicrobial Agents (articles: n=1)

• Cell Discovery (articles: n=1)

• Engineering (articles: n=1)

• Journal of Travel Medicine (articles: n=1)

• Antiviral Research (articles: n=1)

• Journal of Hospital Infection (articles: n=1)

• Journal of Medical Virology (articles: n=2)

• BioScience Trends (articles: n=1)

• The Lancet Child & Adolescent Health (articles: n=1)

• Médecine et Maladies Infectieuses (articles: n=1)

• JAMA (Journal of the American Medical Association) Network Open (articles: n=1)

Figure 1.

Correlations between citation count and AAS and between citation count and PlumX score. AAS: Altmetric Attention Score.

Discussion

In the past several months since COVID-19 has spread globally, many articles pertaining to the pandemic have been disseminated. Traditionally, high-quality and highly regarded articles can be ascertained via citation count; however, this method is not always possible in a rapidly evolving pandemic, such as the COVID-19 pandemic. Our study assessed the utility of adjunctively using AASs and PlumX scores with citation count in the evaluation of the top 100 “trending” COVID-19 articles. According to the results of our study, AAS weakly correlated with citation count (r²=0.0973; P=.002), whereas PlumX score strongly correlated with citation count (r²=0.8911; P<.001). This suggests that there is an alignment in information dissemination between both peers within the scientific community and those in the general public.

Our results are consistent with the findings of some of the previous literature and inconsistent with those of other literature. Previous studies have shown that PlumX and Altmetric scores positively correlate with traditional citation count [12,17-21]. In more current literature, citation count has been shown to weakly positively correlate with PlumX score and AAS. However, PlumX and AAS measures have not consistently positively correlated with impact factor. It has also been shown that compared to all other alternative metric platforms, PlumX is able to capture the widest range of data regarding the dissemination and popularity of a scientific article [7]. Interestingly, current literature has noted that journals with a high Twitter presence also have high PlumX scores and AASs [22]. Our data also reinforces this finding. In our analysis, we observed that Twitter was the most frequently used social media platform, and that the highest number of mentions for the selected articles (985,429/1,022,975, 96.3%) were found on Twitter. These results are also in line with those of previous studies [23]. All of these studies have stated that PlumX score and AAS are most effective when they are used complementarily with traditional bibliometric measures, and that PlumX score and AAS should not necessarily be used on their own to assess an article’s quality [4,7-10].

Although AASs can be used to estimate how widely an article has been disseminated, its correlation with citation count was lower than the correlation between PlumX scores and citation count in our study, making AASs less useful than PlumX scores. There are likely multiple reasons for this. For instance, there are several challenges with regard to the utility of AASs, such as the heterogeneity of the platforms that Altmetric uses to calculate the score, the dynamic nature of AASs, and the fact that many of the platforms we assessed (eg, Twitter, Facebook, etc) were not focused on academia [4]. These challenges allow for the possibility of article misinformation and the review of an article that may be retracted a few days later, which may why explain the weak correlation between AAS and citation count. Based on the platforms that Altmetric uses to calculate its weighted score and those that the scientific community finds interesting, it was not surprising to observe the strong correlations between citation count and Mendeley citations, and between citation count and Dimensions citations. Mendeley and Dimensions are 2 well-known platforms that are used by scientific researchers to source, organize, and cite preexisting literature. These platforms attract a more professional group of users who typically work in the biomedical science field compared to other platforms (ie, Twitter or Facebook), which attract a broader demographic group of the general public. Furthermore, although PlumX scores are also dynamic, these scores account for citation count in their weights, which make PlumX scores longer to report and more stable than other bibliometrics [9].

It is important to note that original articles, particularly observational cohort studies, were disseminated more quickly in social media platforms than in scientific journals. Furthermore, the top-scoring biomechanistic basic science article [15] was also rapidly disseminated in social media platforms. This is most likely because this article addressed the possibility of SARS-CoV-2 being engineered in a laboratory [15]. Additionally, all the articles were open access and freely accessible to the general public. This is likely because there is an alignment between scientists and the general public concerning the need to disseminate new COVID-19–related findings as quickly as possible, given the enormous burden of the ongoing COVID-19 pandemic. Thanks to the removal of barriers to access, the reading of newly published articles and the dissemination of articles to peers has considerably increased.

There are several limitations to our study. For instance, Altmetric and PlumX scores do not necessarily reflect the scientific quality of an article [12,24]. Just because an article generated more attention among the general public and academic community than other articles, it does not mean that more attention correlates with better study designs, results, or evidence quality. Additionally, AASs and PlumX scores are more dynamic than citation count, making the precision, consistency, and reproducibility of altmetric analyses difficult to achieve. However, due to the high volume of social media attention that COVID-19 articles have received, the reproducibility of our data is much better than the reproducibility of data from other medical fields, as the power of our results was much stronger. Despite these limitations, our study shows that altmetrics can be used to complement citation analysis for COVID-19 articles. We highly recommend that medical providers, the scientific community, and the general public use AASs when initially searching for the most pertinent articles that the general public is interested in, given that AASs are immediately calculated. The initial search should be followed by the assessment of the utility of the articles, with respect to PlumX scores or citation count. Given the high correlation between PlumX scores and citation count, we believe that this method will result in great confidence when reviewing a high-quality COVID-19 article and a low risk of article misinformation. We believe that our findings will be helpful in the current COVID-19 pandemic and in future pandemics, as traditional metrics, such as citation count, are not readily available when assessing an article’s quality.

Abbreviations

AAS

Altmetric Attention Score