Measures of Impact for Journals, Articles, and Authors

Journals and authors hope the work they do is important and influential. Over time, a number of measures have been developed to measure author and journal impact. These impact factor instruments are expanding and can be difficult to understand. The varying measures provide different perspectives and have varying strengths and weaknesses. A complete picture of impact for individual researchers and journals requires using multiple measures and does not fully capture all aspects of influence. There are only a few players in the scholarly publishing world that collect data on article citations: Clarivate Analytics, Elsevier, and Google Scholar (Table 1). Measures of influence for authors and journals based on article citations use one of these sources and may vary slightly because of differing journal coverage.

Table 1.

Citation Databases

Organization	Product	Years	Platform	Details
Elsevier	Scopus	1970–present	SCImago	Contains citation information from over 39,000 journals; continually adding older content; covers 240 academic disciplines; requires subscription
Clarivate	Web of Science	1900–present	Journal Citation Report	Contains citation information from over 21,100 journals; covers over 250 academic disciplines; requires subscription
Google	Google Scholar	Not provided	Google Scholar	Freely accessible product of Google; collects citation and reference information using web crawlers that roam through websites containing scholarly information.1

Individual Authors

Researchers make contributions to their fields in many ways: through education, advocacy, mentorship, collaboration, reviewing grants and articles, editorial activities, and leadership. For better or worse, their impact is usually based on the number of research articles they publish and how often those articles are cited. Some activities, such as writing editorials for leading journals, book chapters, or other clinical texts; testifying before Congress; or helping to shape government or health system policy, can be highly influential, but not credited in these measures of influence.

A common problem authors have in determining their impact is duplicate names, either from being inconsistent in the name they use (e.g., Jackson JL vs Jackson J) or name changes. There are several ways to establish a persistent and unique digital identifier. Researchers should take advantage of all.

ORCID (www.orcid.org)

Many funders require an ORCID identifier as part of grant submission. ORCID is free, and all authors can sign up to create a unique identifier. ORCID does not track measures of impact, but cooperates with other sites that do by maintaining a list of publications that authors can review for completeness and accuracy.

ResearcherID (www.researcherid.com)

This site provides a unique identifier and pulls information from Web of Science (Clarivate) to generate an h-index. It has a dashboard that generates a Web of Science author impact plot, provides authors a year-by-year report on impact, and generates a “citation” map that shows the location of citations. ResearcherID is also used by Publons, another Clarivate product, that tracks peer review and editorial activity. Access requires a subscription.

Scopus and Web of Science

Scopus and Web of Science are independent sites that create unique identifiers for authors based on proprietary software. Identifiers are automatically assigned and may result in the creation of more than one identifier, particularly if authors have had multiple affiliations, have a common name, have changed names, or have been inconsistent in their name. Authors can review the identifiers assigned and merge different listings. Access to these databases requires a subscription.

In addition, authors can create a Google Scholar account, which will also track and assess author impact. Google Scholar is free. Authors should regularly review their account to make sure their article list is accurate.

Measures of Impact for Authors

There are a number of different measures of individual author impact; each has strengths and weaknesses (Table 2). All are limited in that they do not account for author effort and order. Most can be skewed by self-citation and favor those who have been publishing longer.²

Table 2.

Author Measures of Influence

Measure	How calculated	Strengths	Weaknesses
H-index	Number of articles (h) that have been cited h times	Easy to calculate Combines quality/quantity	Skewed by self-citation Does not account for author order or effort Biased against early-career authors
G-index	Sum citations of top articles and take the square root and round	Easy to calculate Combines quality/quantity	Skewed by self-citation Does not account for author order or effort Biased against early-career authors Highly influenced by high-impact articles
i-10-index	Number of articles that have been cited at least 10 times	Easy to calculate Combines quality/quantity	Favors productivity over quality Does not account for author order Biased against early-career authors Ten citations are an arbitrary cut-point
iCite	Field and time adjusted and benchmarked against median for NIH–funded publications	Provides a benchmark Not biased against early-career authors	Difficult to calculate Highly influenced by high-impact articles
Altmetrics	Weighted measure based on 15 sources	Accumulates quickly Provides measure of societal/cultural interest Provides a different perspective on article/author influence than citations	May not predict importance Not predictive of citations Evolving measures Reflects “popular” topics Can be gamed by using “popular” terms in title Uncertain how to use measures
PlumX analytics	Provides metrics in 5 categories: citations, usage, captures, mentions and social media.

H-Index, developed by Jorge E. Hirsch in 2005, is defined as the number of published papers that have been cited at least h times.³ An h-index of 40 means th.e author has 40 articles cited at least 40 times. This simple metric is widely used for evaluating an authors’ impact. Citation databases like Web of Science, Scopus (Elsevier), and Google Scholar provide h-index information in their author profiles, though the reported h-index may vary due to citation coverage. The h-index favors authors that publish a continuous stream of papers with persistent, above-average impact. It measures the cumulative impact of an author’s work and combines quantity and quality. However, it does not account for the author effort and order, is biased against early-career researchers with fewer publications, and can be skewed by self-citation.

G-Index, created in 2006 by Leo Egghe, is defined as the largest number such that the top “g” articles received together at least g² citations.⁴ This metric favors highly cited articles; a single highly cited article will increase the g-index considerably, while only increasing the h-index by 1.

i-10-Index, calculated by Google Scholar, is a straightforward metric that shows the number of publications with at least 10 citations.

Measures of Impact for Individual Articles

ICite

This is an NIH dashboard of bibliometrics for articles. iCite has three modules: Influence, Translation, and Open Citations. Influence is based on a relative citation ratio (RCR), comparing article citations to the median for NIH–funded publications, the value of which is set at 1.0. Among NIH–funded studies, the 90^th percentile for RCR is 3.81. Among all studies, the 90^th percentile is 2.24. Individual paper influence is reported and can be used to select manuscripts that best represent one’s work. Translation provides a measure of translation from bench to bedside by breaking down whether most of the author’s publications are molecular/cellular, animal, or human. Citations provide a count of the total citations and give citation statistics (mean, median, SE, maximum) as well as a list of the citing articles for each paper.

Alternative measures of influence

There are measures of influence of individual articles that are not based on citations. They provide a snapshot of article impact in a number of alternate venues, such as public policy documents, news articles, blogs, and social media.

Altmetric

Altmetric tracks more than 15 different sources, including public policy documents, news articles, blog posts, mentions in syllabi, reference managers, and social networks, such as Twitter and Facebook. The results are weighted; some sources, such as news articles, get greater weight. For example, in 2020, the weights of the various sources were news stories: 8, blogs: 5, Q&A forums: 2.5, Twitter: 1, Google: 1, and Facebook: 0.25. Altmetrics can be displayed as a “badge,” a symbol with a number in the middle of a circle with the strands colored to reflect the elements that went into the score. Researchers can sign up to create an altmetric badge for their articles (www.altmetric.com). To create a badge, the article must have a DOI number. Altmetrics for any specific article reflects popular interest in the topic rather than scientific importance. At JGIM, article altmetrics do not correlate with citations. Altmetrics can accumulate quickly; many metrics, such as Twitter and Facebook mentions, tend to occur within days of publication, while citations can take years. Altmetrics can be applied to scholarly products other than research publications, such as curricula and software. However, altmetrics can be gamed; “popular” topics tend to get more play than others. It is still unclear how to use altmetrics; most rank and tenure committees do not include these measures in promotion deliberations.

PlumX Analytics

PlumX gathers metrics into 5 categories: citations, usage, captures, mentions, and social media. Citations include traditional citations as well as ones that may have societal impact, such as policy documents. Usage measures views, downloads, and measures of how often the article is read. Captures indicate that a reader is planning on coming back to the article; it can indicate future citations. Mentions refer to news articles, blog posts, and other public mentions of the paper. PlumX Social Media refers to tweets and Facebook likes and shares, among several sources. It provides a picture of how much public attention articles are getting. PlumX analytics suffer from the same issues as altmetrics and citations. PlumX analytics are embedded in several platforms, including Mendeley, Science Direct, and Scopus and on many open-access journal platforms.

Measures of Impact for Journals

Historically, there were many reasons why certain journals rose to the top: highly respected editors, a long publishing history, and a track record of influential work policy makers and clinicians cared about. In 1975, Thompson Reuters debuted SCI Journal Citation Reports, ranking journals based on article citations.⁵ Subsequently, this has been the primary basis for journal prestige.

Journal evaluation metrics that use citation data favor some disciplines over others. Disciplines vary widely in the amount of research output, the number of citations that are normally included in papers, and the tendency of a discipline to cite recent articles.⁶ For example, Acta Poetica focuses on literary criticism. Its impact factor would be a poor measure of the journal’s influence. In addition, one needs to consider where the evaluation tool is collecting their data. Databases like Web of Science and Scopus may have stronger coverage of some disciplines, impacting the citation metrics that are generated.⁶

Some resources assign journals to subject categories, making it possible to compare journals within their discipline. A good analogy is points scored in sporting events. Seven points in American football is a poor offensive outing, while 7 points in European football is a juggernaut. Comparing journals within the same discipline provides better information about the journal’s relative importance.

Journal Citation Reports

Journal Impact Factor (JIF). This is published annually by Clarivate and uses citation data from Web of Science. It has been the “gold standard” for measuring journal impact since its creation.⁷ Journal editors nervously await release of their impact factor every summer. The JIF is calculated by dividing the total number of citations in the previous 2 years by the number of “source” articles published the following year. JGIM had 2810 citations in 2020 for articles published in 2018 and 2019; 548 of these articles were categorized as source material. Dividing 2810/548 yields our 2020 impact factor of 5.128. Not everything journals publish is considered source material. Clarivate does not provide guidance to journals on how they decide what types of material to count. In general, letters and editorials are not included. JGIM falls in the Medicine, General & Internal and the Health Care Sciences & Services categories, ranking 27^th and 11^th, respectively, in each. Seeking high JIF has led some journals to reduce the number of articles they publish, increase the amount of non-source papers, and focus on work they believe will be highly cited. The JIF is also susceptible to journal self-citation.

Journal Citation Indicator (JCI) is a normalized metric that debuted in 2021; a score of 1.0 means that journal articles were cited on average the same as other journals in that category.⁸ JGIM has a JCI of 1.48 (Table 3), meaning we have a 48% more citation impact than other journals in our category. Based on the JCI, JGIM ranks 23^rd in Medicine, General & Internal and 15^th in Health Care Sciences & Services.

Table 3.

Journal Measures of Impact

Measure	JGIM score	How calculated	Strengths	Weaknesses
Impact factor	5.13	Number of citations in a given year to articles published in the previous 2 years, divided by the number of source articles	Easy to calculate Combines quality/quantity	Can be gamed by journals Not a measure of quality Not all citation types are counted Skewed by journal self-citation Clarivate is vague about criteria for articles to be counted as source articles Not all journals have an impact factor Favors journals that publish systematic reviews
Citation indicator	1.48	Normalizes the impact factor compared to other journals in that category	Gives a context for a specific journal Combines quality/quantity
5-year impact factor	6.07	Average number of citations over 5 years, divided by the number of source articles	Easy to calculate Combines quality/quantity Provides a measure of how long article influence is sustained
Immediacy index	1.86	Number of citations occurring in the same year of publication	Easy to calculate Combines quality/quantity Provides information on how quickly research is incorporated
Eigenfactor	0.029	Number of journal article citations over 5 years, factoring in the impact factor of the citing journal	Freely available Takes into account quality of journal citing article Covers 5 years Excludes journal self-citations	Assigns journals to 1 category. Difficult to interpret. Similar to raw citation counts. 5 years may be too long Favors disciplines with high-impact journals
Normalized eigenfactor score	6.07	Normalizes the eigenfactor score so that the mean is 1.0	Normalized Same as eigenfactor
Influence score	2.58	Calculated by multiplying the eigenfactor by 0.01, dividing by the number of articles in the journal, normalized a mean of 1.0	Provides measure of influence Normalized Same as eigenfactor
CiteScore	4.9	Calculated by dividing the number of citations to documents (articles, reviews, conference papers, book chapters, and data papers) over 4 years by the number of articles published by the journal during the index year	Longer time allows time for citations to occur Sources are transparent Updated monthly
Scimago journal rank	1.75	Citations of articles in 1 year to articles over 3 years, weighted by the prestige of the citing journals	Updated annually	Favors fields with high-impact journals Susceptible to self-citation
Source normalized impact per paper	1.47	Measures actual citations relative to citations expected for the field	Normalized	Favors journals that publish more review articles Not as reliable for journals that publish fewer articles Sensitive to outliers
Scimago h-index	180	Number of cited articles at least h times in past 5 years	Easy to calculate. Combines volume/quality	Includes self-citations Favors established researchers
H-5 index	65	Number of cited articles at least h times in past 5 years	Easy to calculate

^*Source articles: articles that are counted in the denominator

5-Year Impact Factor is the average number of times articles published in the previous 5 years were cited in the indexed year. It gives information on the sustained influence of journal publications. JGIM’s 2020 score was 6.070, meaning that articles published in 2014–2019 were cited an average of 6 times in 2020.

Immediacy Index is the number of citations that occur in the year of publication. Journals with high immediacy index scores are rapidly cited. JGIM has a score of 1.861. This measure has been criticized for penalizing articles published later in the year.

Eigenfactor Score, a metric created in 2007 by Carl Bergstrom and Jevin West of the University of Washington, is based on the number of times articles from a journal over the past 5 years have been cited in the indexed year and gives citations in highly cited journals more weight than lesser cited ones. Self-citations by the journal are excluded. JGIM’s 2020 eigenfactor score was 0.02895. This measure suffers from being difficult to understand.

The Normalized Eigenfactor Score provides a normalized metric of the Eigenfactor Score, setting a score of 1 as the average for all journals. Like the Eigenfactor Score, citations that come from highly cited journals carry more weight than citations from less cited journals and journal self-citations are excluded. JGIM’s score is 6.07, meaning that JGIM was sixfold more influential than the average journal in the Web of Science database.

Article Influence. This measure is calculated by dividing the Eigenfactor Score by the number of a journal’s articles over the first 5 years after publication. It is calculated by multiplying the Eigenfactor Score by 0.01 and dividing by the number of articles in the journal, then normalized as a fraction of all articles in all publications, such that the mean is 1.0. JGIM’s most recent influence score is 2.579. This indicates that JGIM is more than twice as influential as the average journal.

SCOPUS

CiteScore is calculated by dividing the number of citations from documents (articles, reviews, conference papers, book chapters, and data papers) over the previous 4 years by the number of articles indexed in Scopus published by the journal during those years. JGIM’s CiteScore is 4.6. Cite scores are calculated on a monthly basis. Among 122 internal medicine journals, JGIM is ranked 40^th by the CiteScore.

SCImago Journal Rank (SJR) also uses Scopus data and weights citations according to the prestige of the citing journal, taking into account the thematic closeness of the citing and cited journals.⁹ It is calculated based on citations in 1 year to articles published in the previous 3 years. JGIM’s SJR is 1.746, which puts us 13^th on the list of “internal medicine” journals.

SCImago H-Index calculates the number of journal articles (h) that have been cited at least h times. It is the same calculation used to evaluate authors; SCImago calculates the journal h-index using Scopus citation data. JGIM has an h-index of 180, meaning that 180 of our articles have been cited more than 180 times. The h-index measures the productivity and impact of journal publications.

Source Normalized Impact per Paper (SNIP) compares each journal’s citations per article with the citations expected in its field. It allows a comparison of the journal’s impact across fields, because it adjusts for the likelihood of journal articles in that field being cited. JGIM’s SNIP is 1.471 which ranks us as 23^rd among 112 internal medicine journals.

Google Scholar

H5-index. Google Scholar calculates an H5-index for journals, which is the number of articles in the last 5 years with at least h citations. Google Scholar classifies JGIM as a primary care health journal. JGIM has an H5-index of 65, making it the top-ranked journal in this category. Google Scholar does not make available the citation sources; consequently, it is difficult to tell how complete the data is.

Journal Altmetrics

Like individual articles, altmetrics can be generated for journals. They have the same advantages and disadvantages as individual article altmetrics. In 2020, JGIM had 2.5 million downloads, 61 k linkouts, and 33 k social media mentions. Journal editors may have a poor understanding of altmetrics and struggle to know what to do with the data. Altimetrics reflect popular interest. For example, in 2020, the COVID pandemic captured public interest; articles focused on aspects of the pandemic received considerable public attention. For JGIM, the top altimetric article examined the impact of masking on preventing the spread of COVID and had an altmetric score of 4829.

JGIM is interested in these measures to ensure that we (like our authors) are having an impact. However, we are not obsessed on these measures and will continue to put forward what feels most important and relevant for academic general internists.

Declarations

Conflict of Interest

The authors had no conflicts of interest with this article.