Is the impact factor the only game in town?

Abstract

“Not everything that can be counted counts. Not everything that counts can be counted.”

William Bruce Cameron

Journal metrics mania started over 50 years ago with the impact factor that has since become so well entrenched in publishing. Ask anyone where they would like to publish their research and most will reply by saying in a journal with the highest impact factor. While this suggests quality and a degree of vetting by the scientific community, the impact factor has also been used to benchmark and compare journals.

Impact factors are often used as a proxy of a journal 's quality and scientific prestige. However, is medicine dependent on a valuation system that may be grounded in falsity? Much about this measure is imperfect and destructive. Journals can manipulate the impact factor by refusing to publish articles like case reports that are unlikely to be cited or, conversely, by publishing a large proportion of review articles, which tend to attract more citations. Another tactic that may be used is to publish articles that could be highly cited early in the year, thereby leaving more time to collect citations.

Many use the impact factor as an important determinant of grants, awards, promotions and career advancement, and also as a basis for an individual's reputation and professional standing. Nevertheless, you should remember that the impact factor is not a measure of an individual article, let alone an individual scientist. As long as an article has been cited, the citation will contribute to the journal's impact factor. This is regardless of whether the article's premise is true or false, or whether the cited paper was being credited or criticised. Perversely, a weak paper that is being refuted will augment the impact factor, as will a retracted article, because although the article may have been retracted, the citations of this article will still count.

The impact factor has weathered many storms in the past but criticisms against it are increasing, as is interest in displacing it as a single metric used to measure an article's influence. Many would like the scientific community to assess research on its merits rather than on the basis of the journal in which it is published. With the advent of social media, an article can now be commented on in real time with Tweets, bookmarks and blogs. In future, these measures will complement the impact factor but they will probably not become an alternative.

Despite its imperfections, the impact factor has been around for a long time. As yet, although many alternative metrics have since emerged, nothing better is available. Perhaps it is the scientific community's misuse of the impact factor that is the problem and not the impact factor itself?

In this article, Pippa Smart, who is the guest editor for this series, writes about the ways to measure the impact of a journal and published articles.

JYOTI SHAH

Commissioning Editor

Eugene Garfield founded the impact factor (IF) in 1955 as a means to help choose journals. It was supposed to be a measure of journal quality and not of article quality. Garfield admitted the skewed nature of the impact factor as an 80/20 phenomenon. He explained that 20% of articles might account for 80% of citations and that of the 38 million items cited from 1900 to 2005, only 0.5% were cited more than 200 times with half not cited at all.¹ So why is the publishing industry so fixated on a single measure?

How can you judge the quality of a journal?

There are several ways to evaluate the quality of a journal:

1. The most important criterion is the quality of the content that the journal publishes. Do you think the quality meets your expectations?

2. Who is the editor-in-chief and who is on the editorial board? Are they people you know and respect?

3. Are your peers familiar with the journal? Do they read it? Do they publish their own research in it?

4. Is the journal indexed in MEDLINE^®?

5. What is the journal’s IF?

What is the impact factor?

The IF is a calculation of how many citations the articles in a journal receive (on average). It is used as a proxy measure of the quality of a journal. If the IF of a journal is 5, then it is assumed that every article receives about five citations within the first two years after publication. It is also assumed that the quality of what the journal publishes is five times better than a journal with an IF of 1. However, there are, of course, several caveats of which authors and readers should be aware:

1. As the IF is an average, it can easily be skewed by a few highly cited articles.

2. As the IF is calculated over a two-year window, it cannot take account of a later surge in citations to an article.

3. Several communities read but do not write so they may benefit from articles without citing them. A journal with a low IF may therefore still contain highly valuable content that will impact greatly on the practice of readers.

4. The IF varies depending on article type. Review articles tend to be cited more often and so journals that publish review articles tend to have a higher IF.

5. The IF can be discipline dependent with a content bias. Even the best mathematics journals tend to have low IFs as fewer people cite from them compared with medical journals.

6. The IF is calculated from a small, select list of indexed journals and takes no account of citations appearing in non-indexed journals.

Who calculates the impact factor?

The IF is calculated by the scientific division of Thomson Reuters (TR), a private company, as part of its commercial Web of Science™ database. IFs are published every June in its Journal Citation Reports^®. TR rates and maintains the index as part of its range of products, providing bibliometric information to institutions around the world. TR indexes about 12,000 journals as well as a small but growing number of books and proceedings. It is not known how many journals are published but it is reasonable to assume that the Web of Science™ indexes only about 20% of the total, which means that the majority of scholarly journals published around the world are not counted. There is a heavy bias towards Western high quality journals and only 11.6% of the journals included in the Web of Science™ publish in a language that is not English.²

The data used to calculate the IF are not available publically and it is worth mentioning that other groups have not been able to exactly replicate the IF released by TR. The calculation counts the number of citable items that a journal publishes over a two-year period (denominator) and how many citations these articles have received in the following year (year 3) (numerator). For example, if Journal X published 100 articles in 2013 and 2014 (denominator), and all the journals in Web of Science™ cited these articles 50 times during 2015 (numerator), the IF for Journal X would be 50/100 = 0.5. Citable items included in the denominator are those classified as articles, reviews and proceedings papers but exclude letters to the editor. The numerator, however, includes citations to all papers (including editorials, news items and letters to the editor).

Why don’t all journals have an impact factor?

TR originally started with a mission to only index the best journals and imposes strict quality criteria. All journals are evaluated by specialist panels to judge the quality of what they publish. There are also additional criteria including the timeliness of publication and, importantly, whether the other journals included by TR already cite the applying journal. Journals can be dropped from the index if they do not fulfil the evaluation criteria (eg if they start publishing behind schedule). Of note, 66 journals (a record high) were banned from the 2013 IF list owing to excessive self-citations.

Are there any other ways of measuring the impact of a journal and my research?

PubMed

In the area of biomedicine, being indexed in MEDLINE^® is probably even more important for a journal than having an IF. However, PubMed will only index journals that it decides are of good quality.

Other indexes

There are three main indexes that provide citation metrics for journals and articles: TR’s Web of Science™, Elsevier’s Scopus^® and Google Scholar™. They each provide a variety of citation metrics but the content that is indexed in each varies. As a result, they will not deliver identical metrics.

Elsevier provides a useful short guide to the different metrics that are measured by these indexes.³ In order to discover different metrics on journals that interest you (other than the IF), you can also use these (free) sites:

• http://www.scimagojr.com/journalsearch.php

• http://www.journalmetrics.com/

Table 1 gives a summary of various citation metrics. The following measures are of particular use to researchers:

Table 1.

Citation metrics

Metrics	Description
5-year impact factor	Average citations of papers in a year to papers published in the previous five years. From the Web of Science™. Published annually in the Journal Citation Reports® (JCR).
Age weighted citation rate (AWCR)	Measures the average number of citations to an entire body of work, adjusted for the age of each individual paper.
Altmetrics	Tracks online attention received by individual papers including Tweets, blog posts, citations, Wikipedia mentions, Mendeley and other reference listings, Facebook and Google+ posts.
Article Influence® score	Calculated by dividing the Eigenfactor® score by the percentage of all articles recorded in the JCR that were published in the same journal. Article Influence® is similar to the impact factor and SCImago journal rank.
Aggregate impact factor	Calculated in the same way as the impact factor but takes into account the number of citations to all journals in the category and the number of articles from all journals in the category. An aggregate impact factor of 1.0 means that, on average, the articles in the subject category published in the previous one or two years have been cited once.
Cited half-life	This is a measure of the age of articles being cited. It calculates the halfway point (half of the citations to articles published before a date and half after that date) to give a measure of the longevity of what the journal publishes. For example, if in 2015 the cited half-life of a journal was 5.0, then this means that half of all citations to it were to articles published before 2010 and half to articles published after 2010.
Eigenfactor® score	Citations are weighted according to the prestige of the citing journal so citations from top journals mean more than citations from lesser journals. Uses a five-year citation window. Published annually in the JCR. All journal self-citations are excluded.
Egghe’s g-index	Aims to improve on the h-index by giving more weight to highly cited articles. The g-index is the highest number of papers of a researcher that, on average, have received g or more citations.
Google Scholar™ metrics	Lists the top journals by disciplines and subdisciplines using the journals’ 5-year h-index and h-median.
Hirsch’s h-index	The h-index is an article level measure designed to evaluate individual authors. The h-index indicates the number of papers (h) that have been cited at least h times.
Immediacy index	This is calculated in the same way as the impact factor but looks at only one year (ie number of citations to articles published in a year divided by the number of articles published in the same year). It measures how rapidly the journal is cited and therefore whether it is publishing in a rapidly developing area. Published annually in the JCR.
Impact factor	Average citations in one year to articles published in the previous two years. Published annually in the JCR.
Impact per publication (IPP)	Measures the ratio of citations in a year to scholarly papers published in the three previous years divided by the number of scholarly papers published in those same years. Calculated by Scopus®.
PageRank™ algorithm	Google-based evaluation: The ‘top’ articles tend to be those that have been cited/linked to more than others. The calculation used is a commercial secret and so cannot be fully evaluated.
SCImago journal rank (SJR)	Based on weighted citations in a year to papers published in the previous three years. From Scopus® and published in the SCImago journal and country rank reports. Citations are weighted by the prestige of the citing journal, much like the Eigenfactor® score. Journal self-citations are limited to 33%.
Source normalised impact per paper (SNIP)	Measures average citations in a year to papers published in the previous three years. From Scopus® and published twice a year. Citations are weighted by the citation potential of the journal’s subject category, making this metric more comparable across specialties.
Y-factor	Uses Google PageRank™ with the impact factor to measure and distinguish the quality of citations. Aims to improve the impact factor.

• h-index: Several universities use the h-index to rank academics. This takes into account both the number of publications and the number of citations of an academic's published work. It can be derived from the citation data in either Google Scholar™, TR or Elsevier indexes. The h-index is calculated as the number of publications that have a minimum number of citations. For example:

  • If you have published 15 articles and they have all received at least 9 citations, then your h-index is 9.

  • If you have published 24 articles and they have all received at least 33 citations, then your h-index is 24.

• Google Scholar™: Google provides citation information at an article level. Full details can be found on the Google Scholar™ site (http://scholar.google.com/intl/en/scholar/citations.html). Note that all citations reported by Google Scholar™ will (usually) be very much higher than those reported by Elsevier or TR. This is because Google Scholar™ indexes a much larger database of content and its metrics also pick up some duplicated content.

• Altmetrics: Some journals include alternative metrics (altmetrics) to indicate interest in articles. These include citation data (from the IF, Scopus^®, Google Scholar™ etc) but, importantly, also social networking metrics (eg the number of Tweets that mention an article and the number of blogs that discuss it). This can provide useful information about the interest in an article, which may (or may not) turn into citations.

Can I trust citation metrics?

Always be careful of any metrics. They should be used as indicators and never as absolute quality criteria because of the methodology used to calculate them, the error within them and the limited coverage of the indexes. When looking at the IF, you should only compare journals in the same discipline as the citation behaviour differs between different disciplines. A certain IF may be considered very good in some areas whereas in others it would be considered very poor.

Key messages

• IF is a measure of journal impact and prestige, not of individual articles and researchers.

• Many other metrics are now available to measure and compare the influence of a journal within a specific area.

• All journal quality metrics are based on incomplete information and assumptions, and should only be treated as indicators of quality and impact.