Can altmetrics predict future citation counts in critical care medicine publications?

Abstract

Introduction

Social media is increasingly used in the dissemination of medical research. Traditional measures of the impact of a paper do not account for this. Altmetrics are a measure of the dissemination of a publication via social media websites. The purpose of this study is to ascertain if the altmetric attention score of an article is a reliable measure of the impact it has in the field of critical care medicine. To this end, we investigated if a correlation exists between future citation count and altmetric attention score.

Methods

The top nine journals by impact factor in the field of critical care medicine were identified for 2014 and 2015. The 100 most cited articles from these journals were recorded to form the Scientific Impact Group, i.e. those with the greatest impact on the scientific community. The altmetric attention score was recorded for each article. The top 100 articles by altmetric attention score were also identified to form the Media Impact Group, i.e. those that generated the most online attention. Their citation counts’ were recorded. Statistical analysis was performed on each group to identify a correlation between altmetric attention score and citation count.

Results

There was a moderately positive correlation in the Scientific Impact Group, with a Spearman r score of 0.4336 (P = 0.0001). A weakly positive correlation was found in the Media Impact Group, with a Spearman r score of 0.3033 (P = 0.002).

Conclusions

There is a positive correlation between traditional bibliographic metrics and altmetrics in the field of critical care medicine. Highly cited papers are more likely to generate online attention. However, papers that generate a lot of online attention are less likely to have a high citation count. Therefore, altmetric attention score is not a reliable predictor of future citation count in critical care medicine.

Introduction

Scientific journals and conferences are increasing their social media presence to reach a wider audience.^1–4 Social media is being used in medical education, research, and scientific conferences to enhance the distribution of knowledge and the learning experience.^5–7 Increasingly, both clinicians and the general public are seeking information via online resources. This is limited to not only online scientific journals but also social and conventional media, such as Wikipedia, Facebook, Twitter, and news websites.⁸

There is a concern that media coverage of medical research may set the agenda for researchers.⁹ Material thought to be of public interest, or “clickbait” articles, are more likely to receive attention from mainstream media.¹⁰ The articles covered are often of weaker methodology than those covered by medical journals.¹¹ Alternatively, certain articles may be publicized to generate interest in a topic.¹² As the consumption of material from social media increases, so too does the need for a reliable measure of the impact this material has.

The impact factor of a journal and citation analysis of individual papers are accepted metrics for the “scientific impact” of a publication. Impact factor is a measure of the frequency with which the average article in a journal has been cited in a particular year.¹³ Publications in a journal with a higher impact factor are more highly regarded.^14,15 The more citations a paper receives, the more influential it is thought to be.^16,17 These measures have well-recognized shortcomings.^16–18 Impact factors are open to manipulation¹⁴ and citations naturally take a long time to accrue. Self-citations are not corrected for, allowing authors to potentially inflate their citation counts by referencing their own papers.¹⁹ Individual paper citations correlate poorly with journal impact factors. These traditional bibliographic measures do not consider how a publication disseminates via social media.

Alternative metrics, such as Altmetric.com, offer a measure of the “media impact” of a scientific publication. Altmetrics provides a numerical score derived from an automated algorithm. This allocates a score based on the amount of online attention detected.²⁰ Attention from different sources is weighted to allow for the relative reach of each type of source. For example, a mention on a news media site generates a score of 8, Wikipedia a score of 3, Tweets a score of 1, while Facebook references are scored at 0.25. News outlets are tiered by reach, re-references/retweets from any source are not scored, and bias is accounted for by scoring a mention by an independent researcher higher than a mention by a publisher journal account. This provides an almost real-time measure of the dissemination of a scientific paper.

Recent publications have investigated the link between the scientific and media impact of publications in a number of medical and surgical specialities.^10,21–23 These found, at best, a weak correlation between the social media metrics and bibliometrics of articles in their respective fields. We are unaware of any previous study that has investigated the link between the altmetric attention score (AAS) and traditional bibliometric analysis in critical care medicine. We hypothesize that the AAS of an article in the field of critical care medicine is a good measure of its impact among the scientific community. To study this, we examined whether there is a correlation between the scientific impact (using citation analysis) and media impact (using AAS) of publications in the field of critical care medicine.

Methods

A search of the Journal of Citation reports was performed and filtered, using the search term “critical care medicine.” Only journals published in English were included in this study. In journals that publish critical care medicine content, we identified those with the six highest impact factors. These were the American Journal of Respiratory and Critical Care Medicine, Intensive Care Medicine, Critical Care Medicine, Chest, Critical Care, and Annals of Intensive Care Medicine, respectively, in 2014. In 2015, they were Lancet Respiratory Medicine, American Journal of Respiratory and Critical Care Medicine, Intensive Care Medicine, Critical Care Medicine, Chest, and Critical Care, respectively. The Lancet Respiratory Medicine appeared in the initial search for 2015 but not 2014. It was also included in 2014 due to the high number of Critical Care Medicine articles published in it. Annals of Intensive Care Medicine had the seventh highest impact factor in 2015 and was included to balance the number of journals studied from each year. The New England Journal of Medicine (NEJM) and the Journal of the American Medical Association (JAMA) were also included due to the high number of Critical Care Medicine articles published in those journals. As they are not exclusively publishers of Critical Care Medicine articles, they did not appear in the initial search. This gave us nine journals in total, all with an impact factor greater than 3.

Scientific impact

Using the Scopus database, the top 10 articles by citation count for the years 2014 and 2015 for each of the journals included were identified on 7 June 2017. The citation count and AAS for each article was recorded. The AAS was obtained using the Altmetric bookmarklet for Internet Explorer (available at altmetric.com, Altmetric LLP, London, UK) on the same date.

Media impact

Altmetric support (support@altmetric.com) were contacted on 7 June 2017 and asked to provide the top 100 highest scoring articles from the journals included in the study for each of the years 2014 and 2015. The results were then combined into a single spreadsheet, and the top 100 highest scoring articles from the combined 2014/2015 database were identified. Citation analysis for these was extracted from the Scopus database.

Statistical analysis

Descriptive statistics were calculated for each group. The D’Agostino-Pearson normality test was used on each group to assess for normally distributed data. In both groups, the data were not normally distributed. Spearman’s rank correlation test was used to assess the relationship between citation count for each article and the corresponding AAS.

Results

Scientific impact group

In the “Critical Care Medicine” category in the Journal of Citation Reports (JCR), 27 journals were included for 2014, of which 16 were suitable. Four journals were excluded for being in languages other than English. Seven further journals were excluded as they primarily published on subjects other than critical care medicine (Figure 1). In total, 33 journals were included for 2015, of which 20 were identified as being suitable for this study. Four journals were excluded for being in languages other than English. Nine further journals were excluded as they primarily published on subjects other than critical care medicine (Figure 2). The journals selected from the search of the JCR are listed in Table 1. The median (range) impact factor was 8.556 (3.31–59.56). The 100 most frequently cited articles from the journals included are listed in Supplemental Table 1. The mean (range) number of citations was 114.3 (47–848), with a median of 77 (interquartile range 56–108.8). The top four most cited articles all concerned septic shock.

Figure 1.

Journal selection 2014.

Figure 2.

Journal selection 2015.

Table 1.

Journals included.

Journal	2014 Impact factor	2015 Impact factor
New England Journal of Medicine	55.873	59.558
Journal of the American Medical Association	35.289	37.684
Lancet Respiratory Medicine	9.629	15.328
American Journal of Respiratory and Critical Care Medicine	12.996	13.118
Intensive Care Medicine	7.214	10.125
Critical Care Medicine	6.312	7.442
Chest	7.483	6.136
Critical Care	4.476	4.950
Annals of Intensive Care	3.311	4.529

The AAS of the top 100 cited articles are also available in Supplemental Table 1. Analysis of these revealed a mean AAS (range) of 86.78 (0–530). “Transfusion of plasma, platelets, and red blood cells in a 1:1:1 vs a 1:1:2 ratio and mortality in patients with severe trauma: The PROPPR randomized clinical trial” by Holcomb et al. was the highest scoring article with an AAS of 530.²⁴

The 100 articles in this group showed a moderately positive correlation between the citation count and the AAS of each article (Rs 0.43, 95% CI: 0.25–0.58, P < 0.0001; Figure 3).

Figure 3.

Scatter plot demonstrating the relationship between citation count and altmetric attention score in the scientific impact group.

Media impact group

The top 100 articles from the same journals based on AAS are listed in Supplemental Table 2. Scores ranged from 63 to 530. The mean (±SD) AAS was 136.39 (97.68). The highest scoring article in this group was also the top AAS in the Scientific Impact Group (Holcomb et al.). The top articles covered a variety of topics, with no discernible pattern.

The citation counts of the top 100 articles by AAS are also available in Supplemental Table 2. Analysis revealed a mean (±SD) citation count of 76.93 (114.82), with a range from 2 to 848. The top 100 articles in this group showed a weakly positive correlation between the AAS and citation count of each article (Rs 0.30, 95% CI: 0.11–0.48, P = 0.002; Figure 4).

Figure 4.

Scatter plot demonstrating the relationship between altmetric attention score and citation count in the media impact group.

Discussion

The results of this study show a positive correlation between the citation count of a paper and the AAS in the field of critical care medicine in both the “scientific impact” (i.e., highly cited) and the “media impact” (i.e., high AAS) groups. However, these positive correlations were moderate and weak, respectively. The stronger correlation in the “scientific impact” group suggests that those highly cited papers are more likely to be shared on social media. In contrast, papers with a high AAS were less likely to have high citation counts. A recent study by Huang et al. of six PLoS journals looked for a correlation between AAS and Web of Science citations.²⁵ With respect to the journal PLoS Medicine specifically, they found no correlation between AAS and citations. Although the Huang study focused on a journal (PLoS Medicine) as opposed to a field (critical care medicine), their results support our study by confirming that AAS is not a good predictor of future citation counts. It is clear that AAS must be interpreted with caution.

In recent years, the use of social media has increased. Medical journals now have a social media presence.¹ Online forums such as Facebook and Twitter have become a popular way of publicising research material.² With this increase in use of social media by the scientific community, the moderately positive correlation between citation count and altmetric score in the scientific impact group is unsurprising. It has previously been shown that social media presence is associated with stronger academic bibliometric profiles.²³ It has also been shown that the size of a general medical journal’s following on twitter is strongly linked to the journal’s impact factor.²⁶ It follows that research published by higher profile medical journals with larger social media profiles will be more widely disseminated. Papers that accrue more citations are considered important by peers and are likely to be promoted by medical journals and researchers alike, accruing a higher AAS.

It has been shown that journals with a social media presence receive more citations.^27,28 This may be because researchers are more aware of publications promoted on social media, and therefore more likely to cite them. This could explain the weakly positive correlation we found in the media impact group. Some publications have examined the relationship between mentions on a specific social media outlet, for example Twitter, and citation counts.^22,27 They found that highly tweeted articles were more likely to have a higher citation count than less tweeted articles. Due to the way in which the AAS is calculated, multiple mentions on a single platform are less beneficial to the score as mentions on different social media sites. Research that is heavily promoted on one platform may, therefore, lead to a high citation count due to increased awareness of the publication, while contributing less to the AAS. This could explain why the correlation we found was only weakly positive.

A recent similar study in the field of urology demonstrated a weakly negative correlation between the AAS and citation count. They noted that publications receiving the most online attention were published in the Journal of Sexual Medicine or were related to sexual intercourse.¹⁰ These are issues that are likely to be of interest to the general public, regardless of the quality of the research, and will be promoted by online resources to generate views. Critical care medicine is a field that likely has fewer publications of interest to the general public and thus less social media sharing from public news outlets. This removes a large confounding factor and may be why our results contrasted with O’Connor et al. AAS generated by critical care publications may be due to social media activity by medical professionals only.

In the last number of years, there has been a move toward open access research material to facilitate the dissemination of knowledge.²⁹ This allows anyone, medical professional or not, to access medical research previously limited to journal subscribers. Internet access has also led to an increase in people using online health resources.³⁰ They are naturally more likely to encounter research promoted on social media. Non-medical professionals are less likely to critique research to a high standard.³¹ They are also highly unlikely to cite medical research in a future publication. They may, however, share a paper they have found online on their personal social media accounts. This leads to an increase in AAS, while making no contribution to the citation count of a paper.

There have been many publications in recent years pointing out the inadequacies of the impact factor and citation counts when assessing the quality of research.^18,32,33 There is no guarantee that authors read and critique every paper they cite. Authors may cite papers that fit their message regardless of quality. Despite these inadequacies, they remain an important marker of the quality of a paper. The limitations of the AAS have been discussed in previous publications.^34,35 While our results show that the AAS may have merits, it is important to also consider the shortcomings of this metric. The AAS, like the citation count, is open to manipulation. Akin to self-citation, authors may use multiple social media accounts to “self-share” their publications, opening AAS to artificial inflation.

It is difficult to ascertain which measure is more reflective of relevance to intensivists. Because critical care may be less susceptible to “clickbait” articles than other disciplines, AAS should be less artificially inflated. In the author’s opinion, this may give a more accurate and up to date measure of dissemination and indicate current trends in critical care medicine. If we accept citation counts as the gold standard measure of the quality of a publication, however, AAS is not a strong indicator of this. As mentioned above, citation counts also have significant shortcomings. Ultimately, there is no substitute for carefully reading and appraising a paper to judge its quality, regardless of citation count or AAS.

There are a number of limitations to this study. Papers from both 2014 and 2015 were included in the sample. Papers published in 2014 will naturally have had more time in which to accrue citations. The AAS has a different temporal pattern where the score accrues in real time, and the majority of the score accrues in the early post publication period. In addition, we studied only the most popular scientific and media impact articles; therefore, our results may not be applicable to those less well cited or shared articles. We also included publications from three journals which did not appear in the critical care medicine category of the JCR that we believed would have highly cited articles relevant to the discipline. This introduced a potential selection bias to the study. We chose not to perform a regression analysis of the two groups as we do not consider the AAS and citation count to have a relationship dependent on each other.

A number of studies have previously studied the correlation between AAS and citation count in scientific papers.^36,37 Few have studied this correlation in specific medical fields.^10,21 To our knowledge, this is the first paper to assess this correlation in the field of critical care medicine.

Conclusion

There is a positive correlation between the AAS and citation count of publications in the field of critical care medicine. This correlation is moderately strong in publications with a high “scientific impact” and weakly strong in publications with a high “media impact.” Highly cited articles are more likely to generate a high AAS, but articles with a high AAS are less likely to generate a high citation count. Therefore, AAS is not a reliable predictor of future citation count in critical care medicine.