In science “there is no bad publicity”: Papers criticized in comments have high scientific impact

Abstract

Comments are special types of publications whose aim is to correct or criticize previously published papers. For this reason, comments are believed to make commented papers less worthy or trusty to the eyes of the scientific community, and thus predestined to have low scientific impact. Here, we show that such belief is not supported by empirical evidence. We consider thirteen major publication outlets in science, and perform systematic comparisons between the citations accumulated by commented and non commented articles. We find that (i) commented papers are, on average, much more cited than non commented papers, and (ii) commented papers are more likely to be among the most cited papers of a journal. Since comments are published soon after criticized papers, comments should be viewed as early indicators of the future impact of criticized papers.

In 1543, Nikolaus Copernicus proposed a heliocentric model that will have revolutionized the human view of the universe¹. The paradigm shift proposed by the new ideas of Copernicus caused an unavoidable controversy in the scientific community of his era, still anchored to the geocentrism. The dispute involved some of the brightest brains of the period –including Galilei, Kepler and Netwon– and lasted for more than two centuries before the complete acceptance of the helio-centric model for the the description of the solar system. The controversy behind the Copernican revolution is just one of the most popular examples of scientific controversies that are part of the history of science. Other well known examples are the controversy which followed the publication of the theory of evolution by Darwin², the Bohr-Einstein debate about the fundamentals of quantum mechanics³, and the dispute originated by Wegener with his theory of continental drift⁴. In the course of the history of science, however, not all scientific disputes have been resolved in favor of the original idea that caused the controversy: the observation of N-rays⁵, the theory of cold fusion⁶ and the finding of water memory⁷,⁸ are all examples of theories or experimental results, associated to fervent scientific controversies, that have been at the end rejected or disregarded by the scientific community.

Either resolving in favor or against the scientific findings that originated the disputes, scientific controversies are thought to be necessary for scientific progress⁹,¹⁰. Even if not all the greatest achievements in science have passed through a dispute, as for example the unification of electricity and magnetism by Maxwell, many major steps in science have been controversial. Revolutionary changes are per se controversial because they reverse previous scientific paradigms, and thus necessarily encounter some resistance before getting accepted. Scientific disputes, however, are not only associated to revolutionary discoveries, but they are also part of the process of scientific production: science is, in fact, simultaneously a cooperative and antagonistic enterprise, where scientists both collaborate, with the interchange of information, and compete, through the exchange of criticisms. While the structure of collaboration networks¹¹ and the importance of teams for the creation of scientific knowledge¹²,¹³ have been empirically analyzed, no much is quantitatively known about scientific disputes. Scientific controversies are usually studied in philosophy of science, but only through the analysis of popular case examples and never in quantitative terms⁹,¹⁰.

Here for the first time, we provide a quantitative and large-scale study of scientific controversies. We focus our attention on modern scientific disputes identifiable with the publication of formal comments. We systematically study the difference between the citations accumulated by commented and non commented papers, and show that comments can be statistically interpreted as early signs of the future impact of criticized papers.

Results

Comments are short publications whose purpose is to address core arguments, theories or experimental results of recently published research papers. The name of this type of publications varies from journal to journal. For example, in Nature they are called “Brief Communications Arising”, in Science “Technical Comments”, in Physical Review Letters “Comments”, in New England Journal of Medicine “Letters to the Editor”, etc. When submitted, comments are considered by journal editors in a similar way as normal articles: in order to be published, they need to satisfy strict requirements of broad interest, and have to pass the scrutiny of peer-review. The vast majority of comments represent formal criticisms to the content of commented papers. Irrespectively of the journal of publication, editorial policies are, in this regard, very explicit: just to cite an example, in Physical Review Letters “a comment corrects or criticizes a specific Letter” and “is not meant to be a vehicle for addenda” (forms.aps.org/author/comments-prl.pdf). In addition, a comment is generally followed by a so-called “reply” or “response”, written by the same authors of the criticized paper, with the purpose of defending their own paper from the criticisms of the comment. Comments and replies are published one after the other in the same issue of the journal, and thus provide a fair way to give birth to a scientific dispute, where both parts present and defend their own opinions.

We identified all formal comments published in the last ten to fifty years in thirteen major publication outlets in science, including multidisciplinary and specialist journals (focusing on research topics in environmental sciences, geology, medicine and physics). We automatically associated each comment to the criticized publication (see supplementary information). Finally, we collected from the Web Of Science database (isiknowledge.com) the number of citations accumulated by each publication (as of April 2012). In our citation analysis, we restricted the attention only to papers published before 2008 in order to rank papers on the basis of stable citation distributions¹⁴ and be confident that the vast majority of comments to these papers have been already published (i.e., in more than the 95% of the cases, comments are published less than 4 years after criticized papers, see Figures 3 and S2–10).

Figure 3. Critical comments follow papers after a short time.

(a) Probability density function of the difference in publication dates between comments and commented papers in Physical Review Letters. Comments are, on average, published τ = 13.5 months later than commented papers (standard deviation σ_τ = 11.8). In the inset, we report the cumulative distribution function of the difference in publication dates between comments and commented papers in Physical Review Letters. The 95% of the comments are published 4 – 5 months later than criticized papers, but also in less 30 – 31 months after the publication of commented papers. (b) The typical difference between the publication dates of comments and commented papers in Science is τ = 8.2 (σ_τ = 3.7). In the 95% of the cases, comments in Science are published 3 – 4 months after the publication of the commented papers, and before 14 – 15 months since the criticized papers have been published. (c) In Water Resources Research, the time difference between the publication dates of comments and commented papers has an average value τ = 18.3 (σ_τ = 11.8). In the 95% of the cases, commented papers in Water Resources Research receive a comment after 9 – 10 months but also before 36 – 40 months since their publication. (d) In Geology, papers are commented after an average period τ = 10.1 (σ_τ = 5.4). In the 95% of the cases, comments and commented papers have publication dates that differ more than 4 – 5 months, but also less than 17 – 18 months.

Though commented papers represent a small percentage of the publications of a journal (Table S1), already with a qualitative analysis it is possible to notice that many of the top cited articles of a journal are papers that were criticized in formal comments (Tables S2–4). In Physical Review Letters for example, the most cited paper is “Generalized Gradient Approximation Made Simple”, Phys. Rev. Lett. 77, 3865 (1996), with over 20,000 citations, and this paper was criticized in a formal comment. More in particular, while the percentage of commented papers in Physical Review Letters in our period of observation is just 3%, we find that the 5 most cited commented papers are in the list of the 16 most cited papers of the journal, a proportion 9.4 times larger than what expected by chance. In the other journals, the situation is similar and the 5 most cited commented papers have an absolute rank, based on the raw number of citations they have accumulated, from 2 to 44 times higher than what expected in the case in which being or not being commented would be unrelated to the number of citations accumulated (Table S1). The only less evident case is represented by the publications in Nature, where the rank of the 5 most cited commented papers of the journal is 1.1 times higher than what expected by chance.

From the previous qualitative analysis based on a limited number of case examples, it seems not only that commented papers receive more citations than non commented papers, but also that commented articles are unexpectedly more present in the population of the most cited publications. In order to statistically confirm the validity of these observations, we present here a systematic analysis.

First, we look at the average citation rates of commented and non commented papers. In order to avoid age-dependent biases in the number of citations¹⁵, we compare the average number of citations received by papers published in the same year. The results for the various journals are presented in Figures 1 and S2–10. We confirm indeed that commented papers are, on average, much more cited than non commented papers. However, the observation that a commented paper typically receives more citations than a non commented article is valid not only by looking at average citation numbers, but also by considering other types of measures (e.g, median citation numbers, Figure S11) and performing non parametric statistical tests (e.g., Mann-Whitney-test, Figure S12). In general, while in single years of publication the fact that a commented paper is more cited than a non commented article could be still explained in terms of statistical fluctuations, the persistence over many years of a positive signal allows us to say that commented papers significantly accumulate more citations than non commented articles.

Figure 1. Commented papers accumulate on average more citations.

(a) Average number of citations received by papers published in Physical Review Letters as a function of the year of publication. The average number of citations accumulated by non commented papers is represented by the black line, while the average number of citations received by commented papers is represented by the gray bars. (b), (c) and (d). Same as in panel a but for articles published in Science, Water Resources Research and Geology, respectively.

Second, we investigate the presence of commented papers in the population of the most cited articles. We compare only papers published in the same journal, and assign to each paper a score equal to the fraction of articles published in the same year that received a smaller number of citations. If the score of a paper is equal to one this means that the paper is the most cited publication of the year, while if the paper's score equals zero this means that the paper is among the least cited articles of the year. Since the score is not dependent on the publication year (i.e., the score is a distribution-free indicator), we can directly compare the impact of papers published in different years, and thus increase the overall number of comparisons and obtain a more clear statistical picture. We then perform, using these scores, a ranking between all papers independently of their year of publication, and select the top 5% of publications (or equivalently those with score larger than 0.95) in the ranking. Notice that this is equivalent to the selection of the top 5% of the most cited papers in each year. We finally count the fraction of commented papers in the top 5%, and compare the measured values with those expected by chance in the hypothesis that being in the top 5% and being a commented paper would be two independent events¹⁶. A realization of this statistical ensemble is obtained by randomly mixing the scores only among papers published in the same year. This procedure ensures that the number of commented papers per year is constant, but removes any eventual dependence between citation numbers and the fact that papers have or have not been commented. By generating 10,000 independent realizations, we find that commented papers overpopulate the top 5% of cited papers in each journal (i.e., measured values are larger than expected median values), and more importantly that, in the majority of the journals, these proportions are at least two standard deviations larger than what one would expect by chance (Figure 2).

Figure 2. Commented papers are more likely to be among the most cited publications of a journal.

Fraction of commented papers (black circles) that are part of the top 5% of the most cited papers published in the journals considered in our analysis. This fraction is calculated as C_5%/C, with C_5% number of commented papers within the top 5%, and C total number of commented papers. The quantity C_5%/C thus represents the conditional probability that, given that a paper is commented, this paper is within the top 5%. We compare the measured fraction of commented papers within the top 5% with the distribution of these values calculated in a statistical model where commented and non commented papers are equally likely to be in the top 5%. The horizontal lines inside the boxes denote the median value (50%) of the distribution, boxes delimit the 68.2% confidence intervals (i.e., one standard deviation), and error bars denote the 95.4% confidence intervals (i.e., two standard deviations).

More in detail, by looking at the rank probability density of commented papers (Figure S13), we can observe a general pattern with the following properties: (i) At low rank positions, commented papers are less present than expected. This is not surprising since low rank positions are mainly occupied by uncited papers or papers with few citations, and commented papers are never uncited because they are cited at least by the comment and the eventual reply. (ii) At medium rank positions, the distribution is compatible with the expected uniform distribution. (iii) At high rank positions, commented papers are more present than expected. This means that commented papers over-populate the set of the most cited publications of each journal, and this observation is statistically incompatible with the assumption that commented and non commented papers accumulate citations with the same rates. Notice that this does not mean that a paper, in order to be highly cited, necessarily needs to be a commented paper, but that the conditional probability to be a top cited paper, given that the paper is a commented article, is much higher than the respective unconditional probability. For example in Physical Review D, the unconditional probability to be in the top 5% is 0.05, while the conditional probability for commented papers is three times larger and equals 0.15.

Discussion

Contrarily to the popularity of the wisdom of “any publicity is good publicity”, according to which success might follow from negative criticisms, there are very few empirical validations of this belief¹⁷. In this sense, it is surprising that one example, probably the most clear empirical observation so far, is indeed offered by science. It should be noticed that our analysis does not include all the possible ways of criticizing previously published articles. Our approach is in fact limited only to the case of explicit comments published in the same journal of publication which published the original, criticized, article, and therefore neglects possible cases of “implicit” criticisms arising from the publication of other regular articles. This clearly precludes the application of the same type analysis to disciplines where the publication of formal comments is not practiced.

Also, since comments and commented articles are published at short time distance (Figures 3 and S2–10), it is difficult to make claims about the causality effect of a comment on the number of citations accumulated by a commented paper. The time gap is in fact too short to monitor eventual differences in the trends of accumulation of citations before and after the publication of a comment, and this does not allow us to understand whether highly cited papers attract comments or instead comments generate citations. Our intuitive interpretation is the following. We think that the potentiality of a paper to attract future citations increases the chances that the paper gets commented. Scientists are, in fact, able to early recognize papers that are predestined to be highly influential, and thus these papers are more likely to be deeply examined and eventually commented. We also think that important scientific discoveries often originate controversies, and the publication of a comment is a way of initiating a dispute. This leads to boosts of attention and consequently to higher rates of citation accumulation. Independently of the possible interpretation, our results reveal a coherent pattern for all the scientific disciplines considered here. On the one hand, comments can be viewed as negative labels for criticized papers because comments are published to criticize or correct other papers. On the other hand, comments can be also viewed as positive labels for criticized papers because they statistically represent early indicators of the future impact of criticized articles.

In all journals for which we could monitor the publication of comments over sufficiently long time windows, we realized that, in the latest ten to fifteen years, the proportion of commented articles has significantly decreased (Figures 4 and S2–10). It seems therefore that, in the process of creation of scientific knowledge, scientists are increasingly preferring to avoid scientific “fights”. There could be multiple reasonable explanations for the observation of such decrement in the rate of publication of comments, and, here, we list only few of them. (i) Scientific teams are increasingly dominating the process of production of knowledge, and thus, even if not mutually exclusive, collaboration is overwhelming antagonism. (ii) Writing a comment represents an investment of time and effort as much as a normal article, but has low benefit for the academic curriculum of the authors of the comment and can be potentially dangerous for the creation of scientific “enemies”. (iii) The number of scientific papers published today is too large. Thus, either scientists have not enough time to read carefully all published documents and eventually commenting on them, or the number of central topics in science that deserve the birth of a controversy is not growing at the same rate. (iv) The process of peer-review has become so precise and efficient that the percentage of published papers potentially criticizable has decreased.

Figure 4. The publication rate of comments has drastically decreased in recent years.

Relative number of commented papers in Physical Review Letters (a), Science (b), Water Resources Research (c) and Geology (d), as a function of the year of publication.

Clearly, a simple citation analysis, as the one performed here, cannot provide an exact clue of the reasons for this evident reduction. Our study just represents a starting empirical observation from which additional questions might arise, and further investigations are required in order to understand more deeply such phenomenon. Our analysis, however, provides already a quantitative recommendation to both scientists and journal editors to consider the fundamental importance of an open scientific discussion for the progress of scientific knowledge. If a paper receives a comment, this does not necessarily represent a negative event for the journal which published the paper or for the scientists who wrote the paper, but could be instead an early indication of the importance of the paper itself. As we demonstrated in fact, if citation numbers truly reflect the scientific impact of a publication (although this statement is also under debate¹⁸,¹⁹,²⁰,²¹), then for a paper it is better being commented than not being commented. Indeed, what Oscar Wilde wrote in the Picture of Dorian Gray about gossip seems to be valid also in science: “there is only one thing in the world worse than being talked about, and that is not being talked about.”

Supplementary Material

Supplementary Information

Supplementary Info