Article-level metrics: A new approach to quantify reach and impact of published research

Abstract

A spectrum of measuring tools are available to evaluate the impact of published literature and the journals they are published in. Journal Level Metrics (JLM) such as Journal Impact Factor (JIF) or CiteScore assess the reputation of peer-reviewed journals based on citation analysis. Whereas, Article Level Metrics (ALM) quantify the importance, reach and impact of a particular article, and are a new approach to quantifying the reach and impact of published research. Traditionally JLM has served as a proxy for an individual publication's significance, however, the introduction of contemporary and evolution of Alternative metrics measuring digital or societal influence of a particular article has gained popularity in recent times. These metrics help in rapid dissemination of research, development of newer research strategies and individual academic progress. We highlight the characteristics and importance of currently available ALM, and the newer ones influenced by social media, digital media and Open Access publishing models.

Editorial

In academic publishing, publication metrics have been used to gauge the scientific impact and a multi-dimensional influence of research.¹ Journal Level Metrics (JLM) such as Journal Impact Factor (JIF), Eigenfactor, and CiteScore etc. are used to assess the quality of a journal and have been used as a substitute measure for individual articles.^2,3 However, JLM are unable to measure the quality of a published article and are prone to fluctuations based on underlying data source and method of calculation.⁴ Table 1 highlights the commonly used publication metrics in the quantitative evaluation of research outputs.

Table 1.

Commonly used publication metrics in quantitative evaluation of research outputs.

Publication Metrics
Journal Level Metrics (JLM)	Article Level Metrics (ALM)	Author Level Metrics
Journal Impact Factor (JIF) | Impact Factor (IF) | 5-Year IF | IF Best Quartile	Citation Count |Analysis	h-index
CiteScore | CiteScore Tracker| CiteScore Best Quartile |CiteScore Percentile	Usage Metrics	g- index
SCImago Journal Rank (SJR)	Altmetric Attention Score (AAS)	m-index
Source Normalized Impact per Paper (SNIP)	PlumX metrics	i-10 index
h-index of Journals	PLOS Article Level Metrics
Eigenfactor
Immediacy Index
Article Influence Score

(Abbreviations: h-index = Hirsch's index; PLOS= Public Library of Science; IF= Impact Factor).

Article-level metrics (ALM) are quantifiable measures at individual publication level. The ALM allows quantitative analysis of the significance, reach and influence of a particular research or article.⁵ Traditionally publication level metrics are used to quantitatively analyse the merit and impact of individual articles. In print-based publication models, these are represented by “Citation counts or analysis” derived from various data sources and databases.⁶ The digital revolution has allowed electronic dissemination of research in a broader way and consequently it has become increasingly possible to measure impact of published research from newer data sources such as internet, online resources, social media interactions and digital downloads from the journal's website. These newer ALMs are also called Alternative Metrics (AM) or “Altmetrics” (e.g., Altmetric Attention scores (AAS), PlumX Metrics). These incorporate data from online resources and social media and represent a new approach to quantifying the reach and impact of published research.^7,8 The newer ALM assess impact before academic citations accrue, incorporate both academic and social metrics and reflect changing influence of a work overtime. Fig. 1 depicts the commonly used metrics for measuring scholarly impact in research and publication.

Fig. 1.

Commonly used Metrics for measuring the Scholarly impact in Research and Publication.

We highlight the characteristics of various ALMs and factors driving AM that are currently being used to quantify influence of individual research. This Editorial will provide a guide on how published literature is being discussed, shared and read in the era of digital revolution. Thus, informing authors about the various platforms available to showcase their research in the widely connected world.

1. Contemporary and evolving article-level metrics (ALM)

1.1. Citation Metrics

Citation Counts and Citation analysis are traditional ALM to gauge impact of individual articles or publications in a journal.^2,3,6

• •Individual “Citation Counts” are used to quantify and compare the influence of scientific articles. Citation Count represents the sum of citations a publication has received to date. These are sourced from various databases. Citation Count for an individual article can be calculated by searching a title of Digital Object Identifier (DOI) in the widely available databases such as Clarivate Analytics' Web of Science, Scopus, Google Scholar, Dimensions, Crossref2. A digital object identifier (DOI) is a unique, persistent identifier of a particular article book or a chapter and hence it helps in calculation of citations.^9,10
• •Citation analysis

Citation analysis involves further breakdown of the number of times other authors cite an article. Both “Citation Counts”and “Citation Counts” are terms used interchangeably.

Till the development of newer ALM), Citation Count Analysis represented the most commonly used publication metrics at an article level. Apart from Individual article “Citation Counts,” most of the databases allow further Citation analysis using ‘Filters’ such as time period (e.g., 1/5/10 year citations) or by authorship types (e.g., first vs. last author) or by relevance.^11,12

Aggregation of Citation Counts, Field-Weighted Citation Impact (FWCI), Citation Benchmarking and group analysis of individual articles can be helpful in assessing the influence of publications in the academic world, provide evidence towards institutional promotion, and submissions of research grant proposals. It also allows comparison between peer researchers.

1.2. Usage Metrics

In the era of digital revolution, Open Access (OA) publishing models, electronic prints, internet and global accessibility, there is a paradigm shift in how scientific articles are accessed, read and spread.¹³ Consequently. ‘Usage Data’ representing online article views, reads, full text downloads, and circulation has added a new method of evaluating published. Usage indicates active use of the work such as clicks, downloads, views and reads.¹⁴

Usage Metrics is a comparatively recent way of assessing impact of an academic or scientific article.¹ It measures online views (HTML) of articles, number of document downloads (as Pdf), XML downloads, views on journal websites (e.g., top downloads, most viewed), and thus provides a complementary evaluation of published articles. Usage Metrics along with traditional JLM such as JIF, Eigenfactor from the Web of Science JCR and CiteScore, SJP, SNIP indices, Immediacy Index can allow global assessment of an article's impact. Though there is an inherent limitation of assessment of the clinical impact and influence of such downloads or usage of articles, it does go a long way in assessing the scientific reach of an article.

Usage Metrics assessing the HTML views, Pdf and XML downloads can guide publishing groups about the interest of readership. This can influence Editorial Board decisions about priorities in selecting topics for special issues and improve the existing JLM of a particular journal.

1.3. Newer Metrics

The explosion of internet technology and digital revolution allows data capture on published literature from newer sources such as social media platforms ‘mentions’ (Twitter, Facebook), online usage (Clicks, likes), online print mentions (Newspapers, Blogs), downloads or saved content.^15,16 These AM or “Altmetrics” allow assessment of wider societal impact of an individual article. Newer metrics thus can present a richer and more holistic view of how a published or pre-print version of a research article is being discussed, shared or used by the wider world.

The evolution and development of newer indices provide an extension of article assessment and alternative measure of the uptake of each article.¹⁷ Table 2 highlights the commonly used metrics to measure the impact in calculation of AM or “Altmetrics”.

Table 2.

Categories used to measure impact and calculation of Alternative Metrics or “Altmetrics”.

Categories	Examples	Potential Impact
Captures	Bookmarks, earmarks, code forks, favourites, readers	Indicator of future citations
Mentions	Newspaper articles, blog posts, comments, reviews, Wikipedia references	Measurement of people's true engagement with the research
Social Media	Tweets, Facebook likes, shares, comments, Reddit	Societal impact How the research is being promoted and its spread
Usage Metrics	Clicks, downloads, views ((HTML, PDF, XML)	Analyses if anyone is reading the articles or otherwise using the research or downloading or viewing on the Journal website
Citations	Citation indexes, citation databases such as Scopus and Clinical or Policy Citations	Traditional and immediate Societal Impact

1.4. Altmetric Attention Score (AAS)

The Altmetric Attention Score (AAS) is the most widely used AM that provides a score calculated from the categories of Captures, Mentions, Social media mentions, Usage Metrics popularised by various platforms such as Twitter, Facebook and LinkedIn ¹⁸. Fig. 2 presents AAS in the form of a colourful donut. The different colours indicate a different source of online attention received by a particular article ranging from traditional media outlets, social media mentions, online reference managers such as Mendeley. A strong AAS would show a high number in the centre of the donut and a wide range of colours forming the donut.

Fig. 2.

Schematic of an Altmetric Attention Score Donut. The Altmetric Attention Score in the centre of the doughnut is a count of all of the attention a research output has received from various sources.

There are several advantages of AAS usage. It starts tracking online mentions of a particular research as soon as it has been published with increased visibility and feedback. Some recent studies have shown that use of AAS allows assessment of immediate influence of an article and possibly subsequently positively affects traditional bibliometrics including citation rates resulting in greater scholastic impact of individual articles.^19,20 It also provides a real-time, immediate impact of an article amongst both clinicians and the wider public, whilst conventional metric analysis may take years to accrue. The increased researcher profile can be instrumental in gaining popularity, support academic positions. It also appears a higher AAS could be significant in increasing the scholastic impact of individual researchers, their articles, and consequently help improve JIF of their publishing journals.

However, while these positive influences allow increased interaction amongst the academic world, there are also concerns in verifying their ‘authenticity’, since social media postings can be ‘anonymous’. Nevertheless, it has a great value in assessing the holistic evaluation of an article with various journals and publishing groups having embarked on launching their own individual social media handles and also ‘tagging’ existing academic websites to increase dissemination.^21,22

1.5. PlumX Metrics

PlumX metrics offers insights into people's interaction with individual articles or on how the readers use the components of the research article in an online environment such as social media article views, downloads, clicks, usage data and citations.²³ PlumX metrics provides these metrics as a PlumX tool which generates a Plum Print, an infographic. These infographic diagrammatic blobs represent five different usage portals (Fig. 3). The different coloured blobs denote usage, captures, mentions (newspaper and social media separately), and citations in databases.³ The purple blob represents captures, the green blob indicates usage (Clicks, likes), yellow indicates mentions (newspapers, blogs), the blue represents mentions in social media platforms (such Twitter and Facebook), while the red blob indicates citation gathered in databases such as Scopus.²⁴

Fig. 3.

Schematic of PlumX metric infographic. The different coloured blobs denote usage, captures, mentions (newspaper and social media separately), and citations.

PlumX metrics are currently incorporated on display in Elsevier's publishing portals such as Scopus, ScienceDirect, Pure, and SSRN. PlumX metrics also integrates measurements from Mendeley® reference management platform. PlumX metrics allow evaluation of wider societal impact of articles and has become a central part of research evaluation across the world.

1.6. Public Library of Science (PLOS) Article Level Metrics

Public Library of Science (PLOS) is a non-profit, OA publishing model launched in 2001 with an objective to advance research quicker, widen participation and increase collaboration amongst researchers. Following the publication of its first Journal in 2003, PLOS Biology, PLOS launched PLOS ALM to provide impact of articles in the scientific Community and amongst the wider public.²⁵ PLOS ALM measures attention of articles before they start receiving physical citations and assess different indicators such as views, citations, social media mentions, recommendations, shares, usage statistics, bookmarks and recommendations.²⁶ Consequently, the science is made available publicly immediately and provides broader influence of articles than traditional citation evaluations.⁵ Social media interactions on platforms such as Tweets and Facebook Likes and features such as ‘Saved’ on CiteULike, Mendeley provide early activity indicators and extension to traditional citation evaluations to identify important research.

Data inputs from both contemporary and newer sources provide a broader picture of how an individual article is being cited, discussed, shared and influencing research globally. ALMs are valuable extension of traditional citation-based JLM and provide a way to further assess the impact of a particular article. The introduction of newer ALMs such as AAS (due to expansion of digital publishing models, social media metrics, online resources) have become instrumental in the rapid evaluation of published research. Publishers are now integrating their journal websites with social media platforms to garner online attention and promote both journalistic and article. Higher ‘Altmetrics’ may be influential in improving not only individual researcher's reach but also improve the traditional Journal level metrics such as JIF and CiteScore of their publishing journals.

“Truth can be stated in a thousand different ways, yet each one can be true” - Swami Vivekananda

Author's contributions credit roles

KPI-Conceptualization, literature search, manuscript writing and editing. RV-Literature search, manuscript writing, references, editing and supervision. Both have approved the final draft.

Funding statement

The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Conflict of interests

Both authors are the Editorial Board members of the Journal of Orthopaedics

Statement of ethics

None, since it is a not a clinical article

Statement of patient consent

Not applicable.