Causes for the Persistence of Impact Factor Mania

ABSTRACT

Numerous essays have addressed the misuse of the journal impact factor for judging the value of science, but the practice continues, primarily as a result of the actions of scientists themselves. This seemingly irrational behavior is referred to as “impact factor mania.” Although the literature on the impact factor is extensive, little has been written on the underlying causes of impact factor mania. In this perspective, we consider the reasons for the persistence of impact factor mania and its pernicious effects on science. We conclude that impact factor mania persists because it confers significant benefits to individual scientists and journals. Impact factor mania is a variation of the economic theory known as the “tragedy of the commons,” in which scientists act rationally in their own self-interests despite the detrimental consequences of their actions on the overall scientific enterprise. Various measures to reduce the influence of the impact factor are considered.

IMPORTANCE

Science and scientists are currently afflicted by an epidemic of mania manifested by associating the value of research with the journal where the work is published rather than the content of the work itself. The mania is causing profound distortions in the way science is done that are deleterious to the overall scientific enterprise. In this essay, we consider the forces responsible for the persistence of the mania and conclude that it is maintained because it disproportionately benefits elements of the scientific enterprise, including certain well-established scientists, journals, and administrative interests. Our essay suggests steps that can be taken to deal with this debilitating and destructive epidemic.

INTRODUCTION

The journal “impact factor” was conceived by Eugene Garfield in 1955 to help librarians identify the most influential journals based on the number of citations, and the first ranking of journals by impact factor was published in 1972 (1). Today the value of a scientific publication is increasingly judged by the impact factor of the journal in which it is published, and this in turn influences the ability of scientists to be appointed, promoted, and successfully funded. It is not immediately obvious why scientists as a group would embrace the impact factor of a journal as an indicator of the importance of a publication and, by extension, the quality of individual scientists and their work. It has been suggested that the seemingly irrational focus directed on the impact factor amounts to “impact factor mania” (2). Is this an accurate diagnosis? “Mania” is defined as “an excessively intense enthusiasm, interest, or desire; a craze” (3). Evidence of an excessively intense enthusiasm and desire is apparent in the fixation of many scientists to publish their work in journals with the highest possible impact factor. Evidence of interest is apparent in a search of the PubMed database using the keywords “impact factor” and “2013,” which reveals dozens of publications in the past year alone. Evidence of a craze is perhaps more elusive, but after considering that the definition of “craze” is “to cause to become mentally deranged or obsessed; make insane,” we note some trends consistent with the definition. Although the term “mentally deranged” is more applicable to individuals than to a field, some manifestations of field behavior are consistent with insanity. Obsession is evident in the behavior of certain scientists to shop for high-impact journals (4). Therefore, we conclude that according to this definition, the life sciences are arguably in a state of mania with regard to the journal impact factor. The analogy of this behavior to disease has been made by others, who have referred to the same phenomenon as “journal mania” (5) and “impactitis” (6).

CAUSES OF IMPACT FACTOR MANIA

We subscribe to an economic view of human behavior in which choices are made in response to incentives. Accordingly, there must be compelling reasons for scientists to engage in the unscientific behavior of linking the quality of science with publication venue. There are aspects of the current scientific enterprise that can at least render the apparently illogical behavior of impact factor mania understandable, if not completely rational.

Hypercompetition for funding and jobs.

In the United States, success rates for grant applications are at historic lows, and the imbalance between applicants and academic positions for scientists has reached a crisis point (7–9). Review panels are routinely asked to discriminate between grant applications that seem equally meritorious, and search or promotion committees likewise must decide which of many highly qualified candidates are most deserving of hire or professional advancement. In this environment, quantitative bibliometric tools seem to offer an objective, measurable way to ascertain researcher performance (10).

Paucity of objective measures of the importance of scientific work.

There are presently more than 25,000 journals and over one million new articles published each year. The reliance of scientists and granting agencies on the impact factor as a measure of scientific quality is rooted in the need for quantitative measures of importance in science (11). In this context, the impact factor of the journal where work is published is often used a surrogate marker of excellence, despite the fact that citation frequency for the journal does not predict the citation frequency for individual papers (12–14). Although we have proposed a scheme to assess the importance of scientific work (11), our approach remains subjective and does not readily lend itself to quantitative analysis. Hence, impact factor mania is driven by the absence of other readily available parameters to judge the importance of scientific articles.

Hyperspecialization of science.

As science has succeeded as an enterprise, it has become ever more specialized (15). The increasing specialization of science has made it increasingly difficult for scientists working in different fields to understand each other’s work. Relying on publication in highly selective journals as a surrogate measure of quality provides a convenient, if intellectually lazy, alternative to attempting to read and understand a paper outside one’s specialty. At least one author has attributed impact factor mania to laziness on the part of senior scientists and bureaucrats (5).

Benefits to selected journals.

The high-impact journals produce a highly desirable commodity when a paper is accepted for publication. These journals create a scarcity of publication slots by rejecting the overwhelming majority of submissions, resulting in a monopoly that restricts access and corners the market for excellent articles. This in turn creates a sense of exclusivity that encourages more submissions, as scientists equate low acceptance rates with greater merit (16). Thus, high-impact journals are perversely rewarded for rejecting manuscripts. A central error made by many scientists is in assuming that exclusivity is an indicator of exceptional quality, an assumption that is not always justified (17). Nevertheless, the impact factor of a journal is a better predictor of subsequent citation frequency than article quality (18). Nature, Science, and Cell collectively account for 24% of the 2,100 most-cited articles across all scientific fields (19).

Benefits to scientists.

Publication in prestigious journals has a disproportionately high payoff that translates into a greater likelihood of academic success. This has been referred to as acceptance to “The Golden Club,” with rewards in the form of jobs, grants, and visibility (20). There is also a tendency for successful publishing in highly selective journals to beget more success, perhaps in part because greater visibility attracts capable and ambitious trainees and also because editors and reviewers may apply more liberal standards in assessing submissions from authors who publish regularly in such venues (21). This helps to perpetuate the cycle. The tendency for the rich to get richer and the poor to get poorer was designated the “Matthew effect” by the sociologist Robert K. Merton (22). The economic implications of the Matthew effect imply that prominent scientists have an interest in the perpetuation of impact factor mania.

National endorsements.

Some nations have developed schemes to rate the productivity of their scientists, depending on the impact factor of the journals in which their papers are published. Brazil has established a “Qualis” scale based on the average impact factor of their publications, which is used to grade students and faculty (23). China has been reported to offer monetary rewards to editors who increase the impact factors of their journals (24), and China, Turkey, and South Korea offer cash bonuses to scientists who publish in journals with high impact factors (25).

Prestige by association.

The cachet of the most highly selective journals is readily transferred to its contents. Although one may resent the disproportionate influence of the most prestigious journals, one cannot deny that many important and high-quality research articles are published there, and articles are judged by the company they keep. Of course, there are also many articles published in less prestigious venues of equal or greater quality, and these may be unfairly neglected.

We conclude that impact factor mania persists because it is useful to certain scientists, certain journals, and the bureaucracy of science, particularly in certain nations. However, benefits to individuals and special interest groups do not necessarily translate into an overall benefit for the scientific enterprise.

PROBLEMS WITH IMPACT FACTOR MANIA

We submit that the current impact factor mania, whether applied to individual researchers, publications, or grants, is seriously misguided and exerts an increasingly detrimental influence on the scientific enterprise. In particular, we would emphasize the following concerns.

Distortions in the scientific enterprise.

The greatest distortion caused by impact factor mania is the conflation of paper quality with publication venue rather than the actual content of the paper. This encourages the branding of science and scientists with journals in which work is published. The likelihood of obtaining funding, academic promotion, selection for awards, and election to honorific societies becomes dependent, at least in part, on publication venue. The distorted value system has become self-sustaining, with the editors of high impact journals commanding far great power and influence than is healthy for the scientific enterprise.

The full impact of a scientific discovery may not be apparent for many years.

The journal impact factor is calculated by dividing the number of total citations over the number of articles published in the previous 2 years. However, the impact of truly important and novel findings often takes more than 2 years to be fully realized (26). Thus, the central premise of measuring the importance of scientific journals or papers on the basis of impact factor is fundamentally flawed. An unanticipated consequence of the way the impact factor is calculated is that the most novel and innovative research may be less attractive to some journals because such work, by its very nature, will have its major influence at a time when it does not contribute to the calculation. Journals that reject manuscripts because they cannot immediately appreciate the importance of the work may be turning down the next penicillin, PCR, or tricarboxylic acid cycle (27). Problems with short-sighted peer review are not limited to journals; an analysis at the National Institute of General Medical Sciences (NIGMS) has concluded that current grant review procedures have an error rate of approximately 30% and result in many meritorious applications going unfunded (28).

Emphasis on high impact means that many meritorious studies will not be funded or published.

Asking scientists to conduct only high-impact research creates a strong bias that discourages high-risk research and reduces the likelihood of unexpected breakthrough discoveries. We suggest an analogy between scientists and foraging predators, which employ random Brownian movement when prey are abundant and use more complex yet still random Lévy flights when searching for sparser prey (29, 30). Scientists in search of new discoveries cannot hope to investigate the full scope of nature if they must limit their searches to areas that a consensus of other scientists judges to be important. As Vannevar Bush observed nearly 70 years ago, “Basic research is performed without thought of practical ends … Many of the most important discoveries have come as a result of experiments undertaken with very different purposes in mind … It is certain that important and highly useful discoveries will result from some fraction of the undertakings in basic science, but the results of any one particular investigation cannot be predicted with accuracy” (31).

Journal impact factor and individual article citation rate are poorly correlated.

The majority of a journal’s impact factor is determined by a minority of its papers that are very highly cited (13). It is well known that the impact factor does not necessarily predict the citation prospects of other papers published in the same journal (12, 14, 18, 32), and the relationship between impact factor and citation rate may be weakening (33). Furthermore, when the impact of science is analyzed by other criteria, there is at best a very weak positive correlation with the impact factor of the journal in which the work was published, whereas journal rank by impact factor correlates inversely with the reliability of research findings (14, 34–36).

Citation rate is an imperfect indicator of scientific quality and importance.

Citation rate is highly dependent on factors such as field size and name recognition (37, 38). Studies suggest that journal impact factor may correlate poorly with the true value of research to a field (39, 40). Review articles and descriptions of new methods tend to be disproportionately cited (41, 42). Moreover, an emphasis on citation rate as a measure of impact perversely discourages research in neglected fields that are deserving of greater study.

Delays in the communication of scientific findings.

The disproportionate rewards associated with publication in high-impact journals create compelling incentives for investigators to have their work published in such journals. Authors typically submit their work to multiple journals in serial fashion hoping for acceptance by a journal with the highest possible impact factor. This effort can consume considerable time and resources since investigators often respond to reviewers by performing additional experiments in an effort to convince journals to accept their papers. The multiple submissions also consume reviewers’ and editors’ time and delay the public disclosure of scientific knowledge. Publication delay slows the pace of science and can directly affect society when the manuscript contains information important for drug and vaccine development, public health, or medical care. For an investigator, the time spent in identifying a high-impact journal can result in a loss in citations (4). These effects translate into major inefficiencies in the dissemination of scientific information.

Creation of perverse incentives.

The pressure for publication in high-impact journals may contribute to reduced reliability of the scientific literature. Articles retracted due to data falsification or fabrication are disproportionately found among high-impact journals (36), and errors also appear to be more frequently encountered (34, 36). The most selective journals demand clean stories and immaculate data, which seldom match the reality of laboratory investigation, where experiments can produce messy results. Hence, some investigators may be tempted to cut corners or manipulate data in an effort to benefit from the disproportionate rewards associated with publishing in prestigious journals. Impact factor mania also creates great pressures on journals to raise their impact factors, which can lead to practices that range from gaming the system to outright editorial misconduct. Some journals have been reported to pressure authors to include more citations to their own journal in order to artificially increase the journal’s impact factor. Recently a scheme was uncovered in which three journals conspired to cite each other’s papers in a mutual effort to increase their impact factors, a conspiracy that was blamed in part by the use of journal impact factor by the Brazilian funding agency to judge the quality of articles (43).

PROPOSALS FOR REFORM

Scientists’ unhealthy obsession with impact and impact factors has been widely criticized (2, 5, 12–14, 16, 26, 41, 44–50). Yet many feel trapped into accepting the current value system when submitting their work for publication or judging the work of others. Two recent efforts to counter impact factor mania are noteworthy.

DORA.

The American Society for Cellular Biology, in concert with journal editors, scientific institutions and prominent scientists, organized the San Francisco Declaration on Research Assessment (DORA) in December 2012 to combat the use of the journal impact factor to assess the work of individual scientists (51). Some prominent scientists in the biomedical research community have begun to speak out in support of DORA. Nobel Prize winner Harold Varmus is advocating the redesign of curriculum vitae to emphasize contributions instead of specific publication (52). Dr. Varmus has decried a “flawed values system” in science and laments the fact that “researchers feel they will win funding only if they publish in top journals” (52).

A boycott of high-impact journals.

Nobelist and eLife Editor-in-Chief Randy Schekman has recently criticized the monopoly of what he calls “luxury journals” in an editorial published in the British newspaper The Guardian. Schekman has vowed not to publish hereafter in Nature, Cell, and Science, stating that the disproportionate rewards associated with publishing in those journals distorts science in a manner akin to the effects of large bonuses on the banking industry (53). Although such efforts are well-intentioned, we are skeptical that boycotting the impact factor or the “luxury journals” will be effective, because the economics of current science dictate that scientists who succeed in publishing in such journals will accrue disproportionate rewards. This will continue to be an irresistible attraction. Even if the journal impact factor were to disappear tomorrow, the prestige associated with certain journals would persist, and authors would continue to try to publish there. Most scientists do not actually know the impact factor of individual journals—only that publication in such journals is highly sought after and respected. For instance, it is not widely known that Science is actually only ranked 20th among all journals in impact factor, lower than journals such as CA—A Cancer Journal for Clinicians and Advanced Physics. Similarly, we fear that boycotts of specific prestigious journals may hurt the trainees of those laboratories by depriving them of high-visibility venues for their work. In lieu of Science, Nature, or Cell, Schekman has recommended that authors submit their best papers to eLife. However, the managing executive editor has described eLife as “a very selective journal,” further noting that “it’s likely that the rejection rate will be quite high” (54). As long as a critical mass of scientists continues to submit their best work to highly selective journals, impact factor mania, or its equivalent, is likely to persist.

WHAT SCIENTISTS CAN DO

Despite widespread recognition that the impact factor is being widely misused, the misuse continues and is likely to continue until the scientific community acts to reduce its value to those who benefit from its use. We suggest some specific measures to ameliorate the damage done by this superficial measure of scientific quality and importance.

Reform review criteria for funding and promotion.

We strongly advocate the assessment of research quality by peers in an individual’s field rather than using simplistic measures based on the quantity of papers and prestige of the publication venue. Academics should resist attempts by institutions to use journal impact factor in promotion or tenure decisions. When evaluating the performance of scientists, a focus should be on contributions rather than publication venue. Academic administrators should be educated that impact factor is an inadequate measure of individual achievement (55) and should be informed of the DORA principles. Individual scientists may sign to support the DORA initiative at http://am.ascb.org/dora/ (56).

Consider the use of diverse metrics.

If metrics are to be used for judging individual scientists and their projects, review committees and administrators should consider a diverse range of parameters. Although we do not suggest that the quality of scientists’ work can be reduced to a single number, we do note that alternative citation metrics, such as the H-index and eigenfactor, are probably better measures of scientific impact than the impact factor. Evaluations that employ diverse metrics are likely to provide more insight than assessments based on single criteria.

Increase interdisciplinary interactions.

The specialization of science has made it increasingly difficult for scientists to understand research outside their own field (15). However, there is no substitute for actually reading and understanding a scientific article. This requires that scientists who are asked to judge the productivity of another scientist working in a different field must acquire a working familiarity with that field. Increasing opportunities for interaction between researchers from different fields in training programs and at seminars and meetings will help to improve the quality of research assessment as well as stimulate transdisciplinary or multidisciplinary research.

Encourage elite journals to become less exclusive.

Letters of rejection often state that “We regret that we receive many more meritorious submissions than we can publish.” Why should not the elite journals expand to accommodate all meritorious articles? Artificial restrictions on journal size serve to perpetuate the current wasteful system that requires authors to cascade serial submissions from one journal to another.

Address current imbalances in research funding and the scientific workforce.

As long as there is an unreasonable level of competition for grants and jobs, methods to compare scientists with one another are going to be utilized. Scientists must work with policymakers to alleviate the current shortages of research funding and job opportunities that have created the current crisis situation.

A return to essential scientific values.

Ultimately, the only cure for impact factor mania must come from scientists themselves. If scientists fail to curb their current impact factor mania, they will pass onto their trainees a distorted value system that rewards the acquisition of publications in exclusive journals rather than the acquisition of knowledge and one that promotes an obsession with individual career success over service to society. Moreover, the current insistence on funding only high-impact projects is skewing the focus of research efforts and increasing the likelihood that important avenues of investigation will be overlooked. Scientists must return to essential scientific values that place an emphasis on research quality and reproducibility, the advancement of knowledge, and service to society over the accumulation of publications in prestigious journals.

CONCLUSIONS

The persistence of impact factor mania that at first seems irrational becomes more understandable in light of the diverse confluence of forces within the scientific enterprise that encourage, promote, and perpetuate it. The major factor underlying impact factor mania is the disproportionate benefit to those few scientists who succeed in placing their work in highly selective journals, and that knowledge forces most, if not all, scientists to participate. In keeping with the current winner-takes-all economics of science (57), impact factor mania benefits a few, creates many losers, and distorts the process of science, yet can be understood as rational behavior by individual scientists because of the large rewards accrued by those who succeed. In 1968, Garrett Hardin authored an essay where he used the phrase “tragedy of the commons” to describe in economic terms a situation in which individuals carry out behavior that is rational and in their self-interest but detrimental to the community (58). In this regard, impact factor mania exemplifies a tragedy of the commons in the midst of the scientific enterprise. Impact factor mania will continue until the scientific community makes a concerted effort to break this destructive behavior.

The views expressed in this article do not necessarily reflect the views of the journal or of ASM.