Examining the correlation between Altmetric Attention Score and citation count in the gynecologic oncology literature: Does it have an impact?

Highlights

• •The AAS is a useful tool to predict future citation counts for oncology publications in gynecologic oncology.
• •A hybrid metric using impact factor and AAS may help to predict academic influence on a broader scale.
• •In addition to impact factor, AAS may be used by journals to enhance interest in subscriptions.
• •AAS may be another metric used to assess the quality of publications.

Abstract

We sought to determine the correlation between Altmetric Attention Score and traditional bibliometrics in the gynecologic oncology literature. We identified the 10 most-cited gynecologic oncology articles from 5 major gynecology journals and 10 major “oncology” journals that publish on gynecologic oncology during 2014, 2016, and 2018. Article citation count and Altmetric Attention Score (AAS), as well as journal impact factor (IF) and date of Twitter account development were recorded. Pearson's correlation coefficient was used to describe the relationship between AAS, tweets, IF, and citation count. While the median citation counts significantly decreased for the top-cited gynecologic oncology articles from 2014 to 2018 (p < 0.001), the corresponding median AAS continuously increased during this period (p = 0.008). For articles published in 2014 and 2018, there was a strong positive relationship between the median citation count and the median AAS (2014: r = 0.92; 2018: r = 0.97), as well as between the IF (r = 0.78 and r = 0.89, respectively); these correlations were moderate to weak in 2016 (r = 0.5 and r = 0.41, respectively). There was a continuously increasing strong positive correlation from 2014 to 2018 between journal IF and median AAS (2014: r = 0.75; 2016: r = 0.82; 2018: r = 0.92). Gynecologic oncology articles published in higher impact journals are associated with increased social media visibility and attention. Our data support the idea that early online attention scores, like the AAS, might be useful for predicting future citation counts for oncology publications in general and gynecologic oncology specifically.

1. Introduction

In oncology and other medical fields, the historical and traditional quantitative metrics for research quality have been the “bibliometrics” of the article, which combines the impact factor of the journal in which the study is published and the subsequent frequency of article citations in the scientific literature (Barbic et al., 2016). The results of seminal studies, presented at national and international meetings, and published in the highest quality journals with the highest impact factors, are expected to be disseminated via television, radio, and print media to the general public. However, as a new era of digital information technology has emerged in recent years, the internet, particularly social media platforms, has been increasingly used as a primary source of news and medical information. Alternative-level metrics (known as “altmetrics”) derived from social media and internet sources have gained in popularity and represent a new quantitative measure of the attention, dissemination, influence, and impact of scientific publications (Nocera et al., 2019). Numerous altmetric platforms have been described and utilized, of which, the Altmetric Attention Score (AAS) is the most established (Rong et al., 2020).

AAS compiles the number of mentions of a publication in the most frequently used social media platforms such as Twitter, Facebook, research websites, news sites, and blogs and generates a weighted score (Supplemental Table 1) (Huang et al., 2018). Thus, the AAS represents a weighted approximation of all the online attention surrounding a particular scientific article. This study sought to analyze the correlation between traditional bibliometrics and the AAS in the gynecologic oncology literature.

2. Methods

We identified the 10 most-cited gynecologic oncology articles from 5 major gynecology journals (Gynecologic Oncology, International Journal of Gynecologic Cancer, American Journal of Obstetrics and Gynecology, Obstetrics and Gynecology, British Journal of Obstetrics and Gynaecology) and 10 major clinical-oriented oncology journals (or journals that frequently publish on cancer topics) that report on gynecologic malignancies (New England Journal of Medicine, Journal of Clinical Oncology, Clinical Cancer Research, Lancet Oncology, JAMA Oncology, Cancer, Cancer Discovery, Journal of the National Cancer Institute, Annals of Surgical Oncology, Annals of Oncology) during the years 2014, 2016, and 2018. These articles were designated “gynecologic oncology articles”. We also identified the 10 most-cited oncology articles in the same major clinical-oriented oncology journals for the same years as a comparison group. These articles were designated “general oncology articles”. If fewer than 10 articles related to gynecologic malignancies were published in one of the oncology journals in any of these years, we only included the existing publications. Guidelines were excluded because of their disproportionally higher likelihood of being cited. As the Twitter account for JAMA Oncology was established in September 2014, data were only available for 2016 and 2018. For each journal, the following were recorded: impact factor in 2014, 2016, and 2018; the total number of tweets; and the 10 articles with the highest number of citations in Scopus by August 2020. Citation count and AAS were recorded for each article. The journal impact factor and date of Twitter account development were recorded for each journal. Existing journal-specific Twitter accounts were available for all journals, except for Gynecologic Oncology. Continuous variables such as citation count and AAS were expressed as the median and interquartile range (IQR). Descriptive statistics were performed, and Pearson's correlation coefficients were used to describe the relationship between AAS, tweets, impact factor, and citation count. Values between 0 and 0.3 indicated no correlation, between 0.3 and 0.5 a weak positive, between 0.5 and 0.7 a moderate positive, and between 0.7 and 1.0 a strong positive linear correlation. Data analysis was performed in R v.3.6.1 (R Foundation for Statistical Computing, Vienna, Austria).

3. Results

In total, 658 published articles were reviewed and included in the analysis, of which, 368 pertained to gynecologic oncology and 290 to general oncology topics (Table 1, Table 2). While the median citation counts significantly decreased for the top-cited gynecologic oncology articles from 2014 to 2018 (p < 0.001), the corresponding median AAS continuously increased during this period (p = 0.008). The highest AAS for any gynecologic oncology article was 1189 (median, 6.0; IQR [2.0–22.0]) and the highest citation count was 778 (median, 30.5; IQR [15.0–63.0]) (Table 1). The New England Journal of Medicine had the highest median article AAS (274.5), the highest median number of citations (236.5), and the highest median impact factor (72.4), while JAMA Oncology had the highest number of journal tweets (35,700).

Table 1.

Number of articles per journal, citation count, Altmetric Attention Score and impact factor for gynecologic oncology articles in 2014, 2016 and 2018.

Gynecologic oncology articles	Publication year			p
	2014	2016	2018
	n = 122	n = 123	n = 123
Articles per journal (n)
NEJM	2	3	7
Clinical Cancer Research	10	7	10
Lancet Oncology	10	9	10
JNCI	10	6	5
JCO	10	10	4
AJOG	10	10	10
Obstetrics and Gynecology	10	10	10
Gynecologic Oncology	10	10	10
IJGC	10	10	10
Annals of Surgical Oncology	10	10	10
Annals of Oncology	10	10	10
Cancer	10	10	9
Cancer Discovery	0	3	1
BJOG	10	10	10
JAMA Oncology	0	5	7
Citation count (median [range])	50.00 [4.00, 778.00]	43.00 [1.00, 742.00]	15.00 [1.00, 363.00]	<0.001
Altmetric Attention Score (median [range])	5.00 [0.00, 341.00]	7.00 [0.00, 1189.00]	8.00 [0.00, 1165.00]	0.008
Impact factor of the journal (median [range])	4.90 [1.95, 54.42]	7.84 [2.37, 72.41]	6.10 [2.19, 79.26]	<0.001

NEJM = New England Journal of Medicine, JNCI = Journal of the National Cancer Institute, JCO = Journal of Clinical Oncology, AJOG = American Journal of Obstetrics and Gynecology, IJGC = International Journal of Gynecologic Cancer, BJOG = British Journal of Obstetrics and Gynaecology.

Table 2.

Number of articles per journal, citation count, Altmetric Attention Score and impact factor for oncology articles in 2014, 2016 and 2018.

Oncology articles	Publication year			p
	2014	2016	2018
	n = 90	n = 100	n = 100
Articles per journal (n)
NEJM	10	10	10
Clinical Cancer Research	10	10	10
Lancet Oncology	10	10	10
JNCI	10	10	10
Journal of Clinical Oncology	10	10	10
Annals of Surgical Oncology	10	10	10
Annals of Oncology	10	10	10
Cancer	10	10	10
Cancer Discovery	10	10	10
JAMA Oncology	0	10	10
Citation count (median [range])	394.50 [95.00, 2269.00]	285.50 [73.00, 3283.00]	133.00 [29.00, 1247.00]	<0.001
Altmetric Attention Score (median [range])	33.00 [0.00, 541.00]	43.00 [0.00, 2708.00]	84.50 [0.00, 5154.00]	0.004
Impact factor of the journal (median [range])	15.16 [3.94, 54.42]	16.19 [4.04, 72.41]	17.53 [3.86, 79.26]	0.015

NEJM = New England Journal of Medicine, JNCI = Journal of the National Cancer Institute, JCO = Journal of Clinical Oncology.

Correlations between AAS, citation count, journal impact factor, and overall journal tweets for the top-cited articles covering gynecologic oncology topics are shown in Fig. 1. For gynecologic oncology articles published in 2014 and 2018, there was a strong positive relationship between the median citation count and the median AAS (2014: r = 0.92; 2018: r = 0.97) as well as between the impact factor (2014: r = 0.78; 2018: r = 0.89); these correlations were moderate to weak in 2016 (r = 0.5 and r = 0.41). There was a strong correlation between the journal's overall number of tweets and the median AAS of gynecologic oncology articles in 2016 (r = 0.76). In all other years, only a moderate to weak correlation was detected (2016: r = 0.56; 2018: r = 0.49).

Fig. 1.

Correlation between (A) Altmetric Attention Score and citation count, (B) journal impact factor and citation count, and (C) Altmetric Attention Score and overall journal tweets for articles covering gynecologic oncology topics.

As noted for gynecologic oncology articles, median citation counts significantly decreased for the top-cited general oncology articles from 2014 to 2018 (p < 0.001), while the corresponding median AAS continuously increased during this time and almost doubled from 2016 to 2018 (p = 0.004) (Table 2). The highest AAS for any general oncology article was 5154 (median, 50.5; IQR [13.0–152.0]), and the highest citation count was 3283 (median, 265; IQR [124.75–485.5]). The New England Journal of Medicine had the highest median article AAS (440), the highest median number of citations (1022), and the highest median journal impact factor (72.4).

For the top-cited general oncology articles, median citation counts were strongly positively correlated with AAS (2014: r = 0.94; 2016: r = 0.83; 2018: r = 0.99) as well as the journal impact factor in each year (2014: r = 0.94; 2016: r = 0.88; 2018: r = 0.96), with nearly perfect positive linear relationships in 2018 (Fig. 2). In addition, a consistently strong positive correlation between journal impact factor and median AAS was detected for these articles in all 3 years (2014: r = 0.98; 2016: r = 0.87; 2018: r = 0.96) (Fig. 3). Finally, a strong correlation was only noted between the overall number of tweets by the journal and median AAS for general oncology articles in 2014 (r = 0.86), while in 2016 (r = 0.52) and 2018 (r = 0.47), only a moderate to weak correlation was noted (Fig. 2).

Fig. 2.

Correlation between (A) Altmetric Attention Score and citation count, (B) journal impact factor and citation count, and (C) Altmetric Attention Score and overall journal tweets for articles covering general oncology topics.

Fig. 3.

Correlation between journal impact factor and median Altmetric Attention Score for articles covering gynecologic oncology and general oncology topics in 2014, 2016 and 2018.

4. Discussion

Our study represents the first known evaluation of the relationship between AAS and traditional bibliometrics in the gynecologic oncology literature. A significant association was identified between the citation count of the top-cited articles on gynecologic malignancies and the corresponding AAS, as well as journal impact factors. In addition, a strong correlation was detected between the AAS of the top-cited articles and the corresponding impact factor of the publishing journal, which became more pronounced in more recent years. Of note, a similar result was found for the top-cited general oncology articles published in the same journals, with strong positive correlations between citation count, AAS, and journal impact factor.

Our results suggest that gynecologic oncology articles published in high-impact journals are associated with increased social media visibility and online attention. Traditional bibliometrics provide feedback on the impact of an article over an extended period of time. For example, citation counts take several years to accumulate, and multiple years' worth of citation counts are factored into a journal's impact factor. In contrast, the AAS highlights the initial attention or early influence of an article through immediate internet activity, which spikes in the first few days and weeks after publication, and declines afterward (Nocera et al., 2019). Our results are consistent with these previous findings that online attention scores might generate scientific impact much faster than traditional bibliometrics. While the median citation counts significantly decreased for the top-cited gynecologic oncology and general oncology articles from 2014 to 2018 in our study, the corresponding median AAS steadily increased during that time.

Previous studies have found that journals with social media accounts such as Twitter had significantly higher AAS than those without (Wang et al., 2017) and that tweets can predict highly cited articles within the first 3 days of article publication (Eysenbach, 2011). Gynecologic Oncology (median impact factor, 4.54; median AAS: 2), which was the only journal in our study without an a Twitter account, had the lowest median AAS of all gynecologic oncology journals with comparable impact factors, such as the American Journal of Obstetrics and Gynecology (median impact factor, 5.57; median AAS: 8), Obstetrics and Gynecology (median impact factor, 4.98; median AAS: 15.5), and the British Journal of Obstetrics and Gynaecology (median impact factor, 5.19; median AAS: 4).

Our findings represent a clear contrast to the results found in most previous studies in other medical fields (Chang et al., 2019, Barbic et al., 2016, Nocera et al., 2019, Rong et al., 2020) on the effect of social media on scientific impact, which have shown only weak correlations between AAS and citation counts. A possible explanation might be that oncologic articles, in general and in gynecologic oncology, in higher impact journals often constitute landmark studies with significant scientific impact, which subsequently have increased Twitter and social media activity.

Our data support the idea that online attention scores, like the AAS, may be useful tools to predict future citation counts for oncology publications in gynecologic oncology specifically and in general oncology publications overall. In the future, a hybrid metric utilizing these two different measures may help to predict academic influence on a broader scale. Additionally, this metric will provide more immediate feedback on scientific effect, allowing scientific oncology journals to assess their academic influence before the years it takes to determine their impact factor. Furthermore, in addition to just impact factor, AAS may be used by journals to garner subscriptions.

At the author level, since other bibliometrics are taken into account when evaluating candidates' publications for promotion, AAS may be another metric used to assess the quality of publications on a candidate's curriculum vitae. Of note, however, there were also article outliers that received considerable online attention but were not highly cited. For example, the publication “Association of Analgesic Use with Risk of Ovarian Cancer in the Nurses' Health Studies” had an AAS of 873 but only 22 citations in Scopus at the time of analysis (Barnard et al., 2018).

Our study has several limitations. Although we included a broad spectrum of journals, including 5 major gynecologic oncology journals and 10 major clinical-oriented oncology journals, our data might not be generalizable to all journals, especially to the ones with lower impact factors. In addition, our analysis focused on the 10 most-cited articles covering gynecologic or general oncologic malignancies and, therefore, might not be consistent with less-cited articles. Finally, there may not have been enough time for 2018 citations to accumulate; however, the strongest correlation between AAS and citation count was seen in recent years, weakening this argument. Future studies with larger sample sizes are warranted to confirm our findings and explore the feasibility of the combined midterm parameter of academic influence of published articles.

Should scientific journals join social media now? Using social media platforms to share relevant information about scholarly publishing is a unique way by which journals can gain massive exposure and feedback from the academic community in a real-time setting, with a possible immediate impact on their scientific reputation. Besides providing information about the latest research findings, journals can use social media to post calls for submissions and announce current interest topics in publishing. Journals can also help patients, caregivers, and other members of the non-academic community by sharing new scientific findings in plain-language, easy-to-comprehend summaries. Social media platforms will undoubtedly gain further influence on academia over the next years. Therefore, the question is not whether journals should use it, but how they will best use it in the future to attract readers.

Funding

This research was supported in part through the NIH/NCI Cancer Center Support Grant P30 CA008748. Dr. Lisa Q. Rong received funding from the Foundation for Anesthesia Education and Research (MRTG-CT-08-15-2018).

CRediT authorship contribution statement

Andrew J. Chi: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editing. Alexandra J. Lopes: Conceptualization, Data curation, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing - review & editing. Lisa Q. Rong: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editing. Mary E. Charlson: Conceptualization, Investigation, Methodology, Resources, Supervision, Validation, Visualization, Writing - review & editing. Ronald D. Alvarez: Conceptualization, Investigation, Methodology, Resources, Supervision, Validation, Visualization, Writing - review & editing. Thomas Boerner: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: