The impact factor and psychiatry journals: an international perspective

Abstract

The impact factor (IF) is a metric for assessing academic journals. Despite its shortcomings, the IF is the preeminent yardstick by which the quality of scientific journals is determined. In this paper, the key aspects of the IF are covered from an international perspective, with a specific emphasis on psychiatry journals.

The concept of the impact factor (IF) was introduced by Eugene Garfield, an American linguist, in 1955 (Garfield, 1955). It was originally suggested as a measure that could help librarians decide which journals they should subscribe to. Over the years, its importance has grown exponentially, to the point that the IF is today, by far, the foremost criterion by which a scientific journal’s standing is evaluated. Furthermore, the IF can significantly boost the subscription and advertising revenues of a journal.

In addition, for researchers, getting papers published in high-IF journals can considerably enhance career progression and increase the likelihood of research grants being awarded. The maxim ‘publish or perish’ is well known in academic circles, and that applies particularly to publishing in high-IF journals.

Definition of IF

The IF of a journal is calculated each year. The IF of journal J for year Y is the number of citations in year Y of all articles published in journal J in years Y–1 and Y–2 (i.e. the two preceding years), divided by the total number of articles in journal J in years Y–1 and Y–2.

For example, say that journal J published 100 articles in 2010 and 100 articles in 2011, and that these articles were cited 450 times in other articles in various journals (including journal J) in 2012. The IF of journal J for the year 2012 will be 450 divided by 200 (100 + 100), which is 2.25.

The IF is calculated based on the Journal Citation Reports (JCR) published by Clarivate Analytics (integrated with its Web of Science product, previously owned by Thomson Reuters and formerly known as ISI Web of Knowledge).

General medicine journals and psychiatry journals with the highest IFs

The five general medicine journals with the highest IFs in 2014 were:

• New England Journal of Medicine, IF = 55.8

• Lancet, IF = 45.2

• JAMA, IF = 35.3

• Annals of Internal Medicine, IF = 17.8

• BMJ, IF = 17.4.

The top five psychiatry journals in 2014 in terms of IF were:

• Molecular Psychiatry, IF = 14.4

• World Psychiatry, IF = 14.2

• JAMA Psychiatry (previously Archives of General Psychiatry), IF = 13.2

• American Journal of Psychiatry, IF = 12.2

• Biological Psychiatry, IF = 10.2.

The IFs of some other notable international general psychiatry journals (again for the year 2014) are listed below:

• British Journal of Psychiatry, IF = 8.0

• Psychological Medicine, IF = 5.9

• Acta Psychiatrica Scandinavica, IF = 5.6

• Journal of Clinical Psychiatry, IF = 5.5

• European Psychiatry, IF = 3.4

• Australian and New Zealand Journal of Psychiatry, IF = 3.4

• Canadian Journal of Psychiatry, IF = 2.6

• Revista Brasileira de Psiquiatria (official journal of Brazilian Psychiatric Association), IF = 1.8

• African Journal of Psychiatry, IF = 0.7

• South African Journal of Psychiatry, IF = 0.5.

A comparison of medical specialties

Reynolds et al (2012) analysed the IFs of 2287 journals in 31 medical specialties. In 2010, there were 128 psychiatry journals with an IF. The highest IF was 15.47 and the median IF was 2.05. Psychiatry journals ranked 11 out of 31 in terms of median IF, but when the average IF for the 5-year period 2006–10 was calculated, psychiatry journals rose to third position (behind oncology and infectious diseases), with an average 5-year median IF of 2.71. Similarly, psychiatry occupied the third rank (behind general medicine and oncology) when the average 5-year median IF of only the top three journals in each specialty was calculated.

Characteristics of research articles published in the psychiatry journal with the highest IF

As indicated above, Molecular Psychiatry, as of 2014, was the psychiatry journal with the highest IF. In the December 2015 issue, there were 11 original articles. Of these, seven were genetic studies and one a functional magnetic resonance imaging study, one a positron emission tomography study, one a structural diffusion tensor imaging study and one a receptor binding study. Four were animal studies. The disorders studied included Alzheimer’s disease, schizophrenia, post-traumatic stress disorder (PTSD) and drug misuse. For eight of the 11 papers, the corresponding authors (usually a reliable indicator of the main country of the study) were based in the USA; the other three papers had correspondence addresses in Canada, Germany and South Korea.

Advantages of the IF

The IF is a simple, quantitative measure that helps rate the quality of a journal. It is arrived at through a straightforward calculation and provides a means of comparing different journals, thus aiding contributors to make an informed choice as to where to submit their research work. As it is calculated annually, it serves as an ongoing incentive for editors of journals to maintain quality.

Shortcomings of the IF

The IF of a journal is directly related to the number of citations of articles and inversely related to the number of articles published. So, if a journal publishes only a few articles and one of them is cited numerous times, that can artificially inflate that journal’s IF (Brink, 2013).

The IF can potentially be manipulated. For example, a journal might commission literature reviews on topics published earlier in the same journal (van der Wall, 2012), which would result in its articles being cited, thus raising its IF.

The IF has influenced the content of medical journals. Case reports, which used to be one of the most interesting sections of medical journals, are seldom published these days (apart from as occasional letters). The reasoning is that as case reports, by their very nature, deal with rare events they are unlikely to be cited as much as other types of article and so bring down the IF (Rao & Andrade, 2014).

Even though the IF is primarily used to rank journals, its influence has widened and it is now often used as a proxy measure to evaluate the quality of individual researchers. The need to publish in high-IF journals may force researchers to do their work in already ‘highly populated’ areas of science as there is a readily available, large cohort of fellow researchers in the same field who could potentially cite each other’s works (Alberts, 2013). This might explain the plethora of studies in high-IF psychiatry journals over the past decade in the fields of neuroscience (especially genetics and neuroimaging) with little relevance to clinical practice so far.

Limitations on the IF from an international perspective

Journals that are not indexed in the Journal Citation Reports do not have an IF. This places authors from low- and middle-income (LAMI) nations in an unenviable position. On the one hand, even high-quality articles published in non-indexed journals (especially those in languages other than English) are unlikely to be widely recognised. On the other hand, the academic pressure to publish in high-IF journals may force researchers from LAMI nations to afford more importance to issues that are of interest to the international (mainly Western) readership of high-IF journals, at the expense of topics that are more crucial for the local population; this results in the diversion of precious research resources.

It has been claimed that ‘impact factor mania’ persists because it benefits certain scientists and certain journals in certain countries, even though it may not translate into an overall benefit for the scientific enterprise (Casadevall & Fang, 2014). This over-reliance on the IF has also been termed ‘impactitis’ (Van Diest et al, 2001) and ‘journal mania’ (Colquhoun, 2003).

It has been proposed that the IF may be contributing to the ‘Matthew effect’ in the field of research (Lariviere & Gingras, 2010). The Matthew effect (named after a verse in the Gospel of Matthew), first described by the American sociologist Robert Merton and also called the ‘principle of cumulative advantage’, is a phenomenon whereby those who are already in a favourable position (money, power, fame, etc.) increase their advantage over time. Applying this effect to publishing, high-IF journals, the scientists publishing in those journals, and the subjects covered in those journals (genetics, neuroimaging, etc.) consolidate and perpetuate their respective influential positions over time.

The San Francisco Declaration on Research Assessment (DORA)

This declaration (see http://www.ascb.org/dora) was issued in December 2012 by a group of editors and publishers of (mainly biology) journals during the annual meeting of the American Society for Cell Biology in San Francisco (Way & Ahmad, 2013). The thousands of subsequent signatories have included Nobel laureates and directors of leading research institutes. The Declaration made recommendations for funding agencies, institutions, publishers and researchers. One of the main recommendations was to reduce the undue importance currently being given to the IF.

Alternative measures to assess quality of journals and researchers

Various alternative measures to assess quality of journals and researchers have been proposed. These include:

• The 5-year journal impact factor (5-year JIF). This has been made available since 2007. It is calculated by dividing the number of citations in a particular year by the total number of articles published in the previous 5 years.

• The SCImago Journal Rank (SJR). This uses information from the Scopus database maintained by the company Elsevier. The SJR score is reached using an iterative algorithm, and takes into account both the number of citations and the quality of the journals in which these citations appear.

• The Eigenfactor score. Like the IF, this too rates a journal based on citations, but it gives greater weight to citations in higher-ranked journals and also gives a higher scoring for journals publishing a greater number of articles. This scoring system was developed by Jevin West and Carl Bergstrom from the University of Washington.

• The h-index (Hirsch index). This is an indicator of the quality of a researcher rather than of a journal. It is based on the author’s most cited papers and the journals in which those papers have been cited. This index was developed by a physicist Jorge Hirsch from the University of California, San Diego.

• The source-normalised impact per paper (SNIP). The SNIP is defined as the ratio of a journal’s citation count per paper and the citation potential (average length of reference lists) in its subject field. The impact of a single citation is given higher value in subject areas where citations are less likely, and vice versa. SNIP was developed by Henk Moed at the University of Leiden.

• The article influence score (AIS). The AIS is calculated by dividing a journal’s Eigenfactor score by the number of articles published in the journal. It is a measure of the average influence of a journal’s articles during the 5-year period after publication, similar to the 5-year JIF.

Conclusion

Despite many limitations, the IF seems set to continue as the most important measure of a scientific journal’s status for the foreseeable future. It is gratifying to observe that psychiatry ranks high among the different medical specialties in terms of journals with high IFs. At the same time, however, it should be noted that most of the original research articles published in high-IF psychiatry journals seem to focus on basic neurosciences (especially genetics) and to report studies done in laboratories in high-income countries; moreover, a large proportion of these are animal studies, rather than research directly involving patients and carers and studying real-life experiences in hospitals and clinics, or in the communities where patients live.

The need for more sophisticated measures of journal ranking is being increasingly recognised by the scientific community, and one would hope that future quality indicators would also take into account the relevance of published research to patients, clinicians and society at large.