Journal impact factors and the influence of age and number of citations

The impact factor (IF) of a scientific journal reflects the average number of citations of its recent articles, and is considered as a measure or ranking of a journal's importance within its discipline. The calculation of IF is based on a simple relationship between how many times journal articles are cited and how many articles are published. The most frequently used IF is based on the number of citations of publications from within the last 2 years. Thus, if a journal has an IF of 5 in 2013, the papers published in 2011 and 2012 were, on average, cited five times in 2013. Links between the success of individual researchers and the IF of the scientific journals chosen as publication outlets are quite direct and significant: (i) many universities have monetary incentives for researchers to publish in scientific journals with high IF; (ii) the IF of published articles plays a major role in the evaluation process for job applications and promotion evaluations for academic research positions; (iii) publishing in scientific journals with high IF may lead to invitations to join, for example, an important research committee or to become a journal editor; and (iv) graduate students may choose supervisors based on their publication record, including data on IF. The importance of IF in the evaluation of academic performance is underscored by the following quote from the Web of Knowledge website (http://wokinfo.com/essays/impact‐factor/): ‘Perhaps the most important and recent use of impact is in the process of academic evaluation. The impact factor can be used to provide a gross approximation of the prestige of journals in which individuals have been published’. We argue, and this is supported by various published comments (i.e. Tse, 2008; Yew, 2010), that the importance of IF has reached a point at which they potentially drive the research agenda by individual researchers. That is, as part of seeking promotion or other forms of recognition, a researcher may conclude that this may be accomplished by publishing an article in a particular journal. Thus, with a focus on IF, the main objective may not necessarily be the research discovery itself, but rather to get it published in a certain journal with a high IF.

Concerns about the Role of IFs

Many researchers have raised strong concerns about the way IFs are calculated and their influence on research agendas (Agrawal, 2005; Hernán, 2008; Reedijk and Moed, 2008; Tse, 2008; Yew, 2010), including: (i) the selection process of which journals to include in databases such as the Institute of Scientific Information (ISI) Web of Knowledge; (ii) subjective (nonlogical) estimates of how many citable articles a journal has each year and therefore the denominator in the IF calculation (Anonymous 2006); (iii) the, at best, loose correlation between a journal's IF and the quality of a given scientific article; and (iv) the fact that a citation can be positive or negative—that is a citation does not necessarily represent positive recognition or scientific merit (van Nierop 2009). Whether or not it is ethically correct, editors and reviewers sometimes attempt to influence the IF by ‘encouraging’ researchers to include more citations to recent journal articles (Agrawal, 2005; C. Nansen, personal observations). Editors of scientific journals have also joined the ranks of those concerned about bias associated with IF. The editors of PLoS Medicine (Anonymous 2006) commented that: ‘science is currently rated by a process that is itself unscientific, subjective, and secretive’ and ‘we feel the time has come for the process of “deciding” a journal's impact factor to be debated openly. Something that affects so many people's careers and the future of departments and institutions cannot be kept a secret any longer’. Similarly, ‘the impact factor is not always a reliable instrument’; thus, in November 2007, the European Association of Science Editors (EASE) issued an official statement recommending that: ‘journal impact factors are used only and cautiously for measuring and comparing the influence of entire journals, but not for the assessment of single papers, and certainly not for the assessment of researchers or research programmes’. With this analysis, we are joining the open debate about IF. Our goal is to provide insight into thus far little‐noticed trends associated with IF of scientific journals across scientific disciplines. We are strong proponents of a quantitative, logical and transparent measure of the ‘value’ of scientific journals, but we do not believe the current IF measurement provides that. Our goal is to shed light on factors which seem to drive IF, although they cannot be considered as indicators of scientific merit. Finally, we propose simple adjustments to the current IF calculation to account for important between‐research category differences.

Research Categories Analysed

All data concerning scientific journals included in this study were obtained from the ISI Web of Knowledge 2010. At the time of data collection, there were approximately 8000 scientific journals worldwide with IFs ranging from below 1 to above 94 and grouped into one or more general research categories with varying degrees of specialization. In this evaluation of IF, we chose to compare journals from seven ISI Web of Knowledge 2010 research categories: pharmacology and pharmacy, medical research and experimentation, food science and technology, organic chemistry, biology, agronomy, and entomology. These categories were selected because they were considered to be similar in level of specialization (except entomology, which is probably more specialized than the other six categories, but is our area of expertise), and have similar ranges with regard to applied and basic research.

General IF Statistics

The numbers of journals in each category varied considerably, and there was a fairly strong correlation (0.72) between the total number of citations in each category in 2010 and the average IF. Citations in the highest ranked scientific journals in pharmacology and pharmacy and medical research and experimentation were about 10‐fold higher than those of the highest ranked journals in agronomy, and they were about twice as high as the highest ranked journals in food science and technology, organic chemistry, biology and entomology. The average IF of the top 20 journals in the pharmacology and pharmacy category was three times higher than that of the highest ranked journal in agronomy. IF is supposed to be used to measure ‘the relative importance of a journal within its field’ (http://en.wikipedia.org/wiki/Impact_factor), but we suspect that many researchers and academic administrators directly compare IF across research categories. Interestingly, there may be very little fundamental difference in research itself. For example, in a recent study in the journal Neurology (IF = 8.31), it was suggested that drinking two cups of chocolate a day can decrease the risk of developing Alzheimer's disease (Sorond et al., 2013). For 30 days, 60 seniors (aged 72.9 ± 5.4 years) were either untreated controls or on a diet of two cups of hot chocolate a day. Compared with controls, seniors following the chocolate diet showed an 8% improvement in blood flow and a 1‐min faster reaction time in cognitive tests. On a very fundamental level, such a study is no different from testing host plant resistance responses or growth responses in crops to fertilizer or plant growth promoters, although that would be unlikely to be published in a journal with an IF above 2. Although the basic experimental designs and analytical techniques may be quite similar, there is clear discrepancy in the ‘importance’, in terms of IF, of a practical/applied research result across disciplines.

Possible Mechanisms Explaining Differences in IF among Research Categories

The calculation of IF is partially based on how many times articles in the journal are cited during a certain time period after the publication year. The most commonly employed IF uses citations within 2 years after publication. Thus, an article cited more than 2 years after its publication does not contribute to the IF of that journal. The Web of Knowledge also produces an IF based on citations during 5 years. Although the need for a cut‐off is understandable, we are unaware of the reasons for the cut‐offs after 5 years or, particularly, 2 years.

The rate at which articles are cited, known as the ‘speed of citation diffusion’, has been considered in studies of IF of statistical journals (Fok and Franses, 2007, van Nierop 2009). Inspired by the citation diffusion hypothesis, we speculated that a combination of two phenomena might explain why some research areas, such as agronomy and entomology, have journals with low IF compared with, for instance, medical and pharmaceutical journals: (i) agronomy and entomology journals include fewer citations than medical and pharmaceutical journals; and (ii) citations in agronomy and entomology journals are older than citations in medical and pharmaceutical journals.

Initial evaluations of averages of all scientific journals in each of the seven research categories revealed that, in 2010, there were about six to seven times more scientific articles published in the areas of organic chemistry and pharmacology than in agronomy and entomology. There was a marked difference in the number of scientific articles per research category, and there was also an eight‐fold difference in the number of citations between the highest (pharmacology and pharmacy) and lowest (entomology) categories. Differences between categories with respect to the numbers of articles or citations are obviously driven by a wide range of factors, including the numbers of researchers and institutions conducting research in those areas, and the magnitude of available funding. As an example, applied field research in entomology or agronomy typically involves multi‐year datasets and publication is delayed accordingly. Seasons with adverse weather patterns or unforeseen logistical problems can prolong the data collection phase. Thus, data collection takes considerably longer than when feeding seniors hot chocolate for 30 days!

Age and Number of Citations Across Research Categories

We measured three variables in four scientific journals in each of the seven selected research categories: (i) average number of citations per article; (ii) average age of citations; and (iii) percentage of citations that would count towards IF (less than 2 years old). The journals were chosen to obtain a range of IFs and geographical distribution. All journals primarily published original research, but a few included review articles. For each journal, we selected the first 10 research/review articles in the last issue of 2010, and recorded the publication year of the 10 first citations and the total number of citations in the article. Thus, with data collected from the first 10 citations in 10 articles in 28 journals, the analysis was based on 2800 data points. The journals covered an IF range from 0.43 to 14.78, and several trends were detected: (i) there was a considerable range in the average age of citations (5–18 years), but all were older than 5 years, which suggests that a 2‐ or 5‐year IF has limited merit; (ii) in the agronomy journals, less than 15% of the citations contributed to the 2‐year IF calculations (because they were more than 2 years old), whereas, in pharmaceutical journals, the average was 28%; (iii) there was a significantly negative correlation between IF and the percentage of citations less than 2 years old (df = 1, 27; adjusted R ² = 0.412; F = 19.881; P < 0.001), suggesting an IF bias towards journals with articles citing recently published work; (iv) there was a positive correlation between the average number of citations and journal IF (df = 1, 27; adjusted R ² = 0.275; F = 11.255; P = 0.002); and (v) there was a strongly negative correlation between IF and the average age of citations (df = 1, 27; adjusted R ² = 0.421; F = 20.657; P < 0.001).

Conclusion

This analysis indicates that IFs across research categories are positively correlated with the amount of published studies (in terms of the number of journals or articles and the number of citations per article). Not surprisingly, the journal IFs were negatively correlated with the average age of citations. This introduces a bias against ‘old’ (more than 2 or 5 years) research, as well as research with comparatively few references. It is always appropriate to include relevant citations, but we are not convinced that simply adding more citations, with an inordinate fondness for recent ones, is linked to the quality of the research article. The existing cut‐offs for citations contributing to the IF calculation is a mystery to us, and we would counter that older references should be given greater weight than recent ones, as citing an old reference suggests that it is relevant even several years after its publication. We acknowledge the need and potential usefulness of a measure such as IF, and support quantitative, logical and transparent ways to measure the quality of scientific journals. However, we argue that the current IF formula needs to be modified in at least three ways. First, because of the considerable variation in the number of researchers, scientific journals and types of research, the authority responsible (whether the Web of Knowledge or another entity) needs to define about 10 distinct and logical research categories, so that IF actually becomes category specific. Editorial bodies of each journal will determine on which of these categories their journal IF should be based, and a journal could potentially have multiple category‐specific IFs. Second, the IF should be calculated on the basis of citations, and articles published should more closely correspond with the observed average citation age, that is, the cut‐off should be later than 5 years. Third, the conventional IF should be divided by the total number of citations within that category, as this will enable better comparison of IFs across research categories with markedly different levels of research productivity.