A network-based normalized impact measure reveals successful periods of scientific discovery across disciplines

Significance

Distinct citation practices across time and discipline limit our ability to compare different scientific achievements. For example, raw citation counts suggest that advancements in biomedical research have consistently overshadowed the accomplishments from all other disciplines. Here, we introduce a network-based methodology for normalizing citation counts that mitigates the effects of temporal and disciplinary variations in citations. The method allows us to highlight successful periods of scientific discovery across the disciplines and provides insights into the evolution of science.

Abstract

The impact of a scientific publication is often measured by the number of citations it receives from the scientific community. However, citation count is susceptible to well-documented variations in citation practices across time and discipline, limiting our ability to compare different scientific achievements. Previous efforts to account for citation variations often rely on a priori discipline labels of papers, assuming that all papers in a discipline are identical in their subject matter. Here, we propose a network-based methodology to quantify the impact of an article by comparing it with locally comparable research, thereby eliminating the discipline label requirement. We show that the developed measure is not susceptible to discipline bias and follows a universal distribution for all articles published in different years, offering an unbiased indicator for impact across time and discipline. We then use the indicator to identify science-wide high impact research in the past half century and quantify its temporal production dynamics across disciplines, helping us identifying breakthroughs from diverse, smaller disciplines, such as geosciences, radiology, and optics, as opposed to citation-rich biomedical sciences. Our work provides insights into the evolution of science and paves a way for fair comparisons of the impact of diverse contributions across many fields.

Today’s scientific enterprise is characterized by fierce competitions for limited resources such as funding and academic positions. Consequently, various stakeholders in science, from faculty hiring committees to grant panelists, are tasked to evaluate the scientific accomplishment of individuals, projects, and institutions and project potential future impact. Increasingly, these tasks are assisted by numerical measures that rely on citation data to calibrate the abstract notion of scientific impact, including the h-index and number of papers in high Impact Factor journals (1). The usages of these indicators in evaluation scenarios whose outcome affects researchers’ career make it necessary to perform citation analysis in an unbiased manner.

For a scientific publication, the most widely used impact measure is the raw number of citations, $C$, capturing the volume of subsequent works that build upon it. Citation count is used both as an input for more advanced citation-based indicators for impact, and as a criterion for identifying breakthroughs (2). However, $C$ suffers from two well-recognized biases (3–5): i) temporal bias, reflected by the higher rate of citations accumulated by later papers, an inflation process that makes it difficult to compare the impact of papers published decades apart; and ii) field bias, manifested by systematic differences of $C$ across disciplines, creating the impression that papers in highly cited fields, like cell biology, have inherently bigger impact than, for example, mathematics papers, where citations are fewer.

Many techniques have been proposed to mitigate the influence of temporal and field biases on the assessment of scientific impact (see ref. 6 for a review). These methods typically suppress year- and/or field-level variations in citations by normalizing $C^i$ of a paper $i$ of interest with the average ${〈C^n〉}_{n\in {\mathcal{N}}^i}$ of its similar papers ${\mathcal{N}}^i$. A widely adopted definition of ${\mathcal{N}}^i$ is the set of papers within the same research field as $i$, with field either operationalized based on publication venue (7) or constructed algorithmically (8). However, the choice of field classification systems (e.g., Web of Science’s Subject Category, Scopus’s Subject Area, etc.) can affect the conclusions drawn from normalized indicators based on journal-based classifications of fields (6). Similarly, algorithmically constructed fields may lack the transparency and reproducibility often emphasized in policy-making settings. More critically, partitioning papers into disjoint fields is a priori and assumes that all papers in a field are identical in their subject matter, ignoring significant within-field heterogeneities of subdisciplines and the increasing intermixing between disciplines (9). To avoid the complications of defining crisp disciplines for impact normalization, another set of approaches define ${\mathcal{N}}^i$ as the set of papers that, together with additional criteria, share bibliographic references with $i$ (coreferenced papers) (10). However, these methods only have limited power in identifying similar papers, as they depend on $i$’s references—rather than the focal paper $i$ itself.

Here, we propose a paper-level, citation-based indicator $\widehat{C}$, representing our first contribution. Different from existing methods that require predefined discipline labels of papers for identifying ${\mathcal{N}}^i$, we consider ${\mathcal{N}}^i$ as the set of papers with which $i$ is cocited. Our hypothesis is that the frequent coappearance of two papers in the same reference lists captures the scientific community’s assessment of the topical relatedness of the two papers (9, 11–13). We show that $\widehat{C}$ offers a better ability in identifying papers that the scientific community considers important and corrects for both temporal and field biases. Our proposal is not the first one that relies on cocited papers for citation normalization. Relative citation ratio (RCR), another paper-level citation indicator, is a forerunner in this regard (11). However, there are several key differences between our method and RCR. First, the normalizer used by RCR is the average citation rate of the journals where cocited papers were published, whereas we directly normalize by the average yearly citations of papers. Second, RCR further benchmarks normalized citation rate using papers funded by NIH R01 grants, remaining unclear how to generalize the benchmark to the entire scientific literature. Third, while RCR performs normalization only once, our normalization is performed on a yearly basis. As such, RCR is influenced by papers with different ages, and in theory, it could drop when extending the citation window, whereas $\widehat{C}$ accumulates over time and is nondecreasing. Furthermore, our systematic, quantitative comparisons between $\widehat{C}$ and RCR indicate that $\widehat{C}$ can better correct the field bias than RCR does.

As a second contribution, we use $\widehat{C}$ to generate insights into the evolution of science by revealing research fields that continue to produce high impact research over an extended period of time. We achieve this by quantifying the representation of a field in the set of top papers identified by $\widehat{C}$, given the time- and field invariance of $\widehat{C}$. Our results unveil a diverse set of fields that have been important sources for scientific breakthroughs, allowing us to look beyond the highly cited disciplines.

Results

Defining $\widehat{C}$.

To define our normalized indicator $\widehat{C}$, we look at the total volume of citations generated by all citing papers published in a single year $t$. These citations are to be distributed among papers already published. For a particular paper $i$, it receives $c_t^i$ citations (i.e., yearly raw citations). Typically, a large $c_t^i$ is equated with high impact, but to determine the scale for which $c_t^i$ is considered large, we need to compare it with $c_t$ obtained by other papers similar to $i$ in year $t$, denoted as ${\mathcal{N}}_t^i$. Here, we leverage cocitations to identify similar papers (9, 11–13) and define ${\mathcal{N}}_t^i$ as the set of papers that are cocited with $i$ by papers published in year $t$ (Fig. 1). We then define our yearly normalized citations, ${\widehat{c}}_t^i$, as $c_t^i$ normalized by the average yearly citations of the papers in ${\mathcal{N}}_t^i$, formally,

$$\begin{array}{c}\hfill {\widehat{c}}_t^i=\frac{c_t^i}{{〈c_t^n〉}_{n\in {\mathcal{N}}_t^i}}.\end{array}$$ [1]

Fig. 1.

Defining yearly normalized citations, ${\widehat{c}}_t^i$. For a paper $i$, its ${\widehat{c}}_t^i$ is defined as yearly raw citations, $c_t^i$, normalized by the average yearly citations of the papers that are cocited with $i$ by citing papers published in year $t$. In this figure, paper $i$ at $t$ has $c_t^i=3$ and is cocited with papers $a$, $b$, and $d$, which have yearly citations of 1, 2, 2, respectively. Therefore, ${\widehat{c}}_t^i=\frac{3}{〈1,2,2〉}=1.8$. In another year, new citing papers are published, which leads to the change of $c^i$, the list of $i$’s cocited papers, and their yearly citations, resulting in a different ${\widehat{c}}^i$.

By definition, ${\widehat{c}}_t^i=0$ if $c_t^i=0$. The total normalized citations in $T$ years after publication is the sum of the yearly contributions: ${\widehat{C}}_T^i=\sum _{t=0}^T{\widehat{c}}_t^i$.

Our measure, $\widehat{C}$, differs from the many existing measures in two important ways. First, most prevailing normalization methods are designed with a prospective viewpoint that tracks a paper’s citations accumulated over $T$ years and normalized to similar papers, which complicates the assessment of citation dynamics. In contrast, our method is designed from a retrospective view, focusing on how citations generated by citing papers published in a single year are distributed to papers already published before and constructed $\widehat{C}$ from yearly normalized data. Our rationale is that already published papers compete for citations from citing papers and the comparisons of the collected citations need to be made yearly, since citation volumes are increasing over time, driven by the exponential growth of science publishing and the gradually increasing number of references in a paper (3). Second, most previous normalization procedures involve global partitioning of papers, be it journal based or algorithmically derived, and all the papers within one partition share the same ${\mathcal{N}}^i$. This choice assumes implicitly that those papers are similar to each other, which is hardly the case: Condensed matter physics, for example, is an umbrella for superconductivity, topological materials, quantum magnetism, subdisciplines with different community sizes, and distinct citation practices. By contrast, our method identifies papers that are locally similar to paper $i$, and each paper has its own “personalized” ${\mathcal{N}}^i$ that may change over time.

To further illustrate the advantage of ${\widehat{C}}_T^i$, we compare it with two other popular indicators: total raw citations $C_T^i=\sum _{t=0}^T{c}_t^i$ and ${\tilde{C}}_T^i$, which normalizes $C_T^i$ with the average $C_T$ of papers in the same year and field, as determined by journals (7). We calculate the three indicators for two exemplar papers published in Nature in 1985 (Fig. 2). The first paper, $p_1$, is in cell biology reporting measured calcium levels in muscle cells (14). The second, $p_2$, is a geoscience paper that analyzed density contrasts in the Earth’s lower mantle (15). Based on $C_{10}$ [$T$ is set to 10 y to tradeoff between including more papers in our analysis while keeping a relatively long citation window, following previous studies that have also selected similar values (2, 16, 17)], $p_1$ has a much higher impact than $p_2$ (449 vs. 181; Fig. 2A). But $C_{10}$ does not consider field size; most of $p_1$’s citations came from cell biology and other prolific biomedical fields, while $p_2$ was mostly cited by papers in geosciences—much smaller fields (SI Appendix, Fig. S1). The ${\tilde{C}}_{10}$ indicator accounts for this field size effect using journal-based categorization of fields. Therefore, it continues to rank $p_1$ higher (20.5 vs. 8.3) because $p_1$ and $p_2$ were published in the same journal and hence share identical field-specific normalizer. By contrast, the proposed $\widehat{C}$ automatically identifies the research area of a paper through the list of cocited papers and compares its impact with those papers. Indeed, although $p_1$ acquired many citations, so did many of the other papers with which it was cocited, i.e., a large ${〈c_t^n〉}_{n\in {\mathcal{N}}_t}$ (Fig. 2A). This is in contrast to $p_2$, which received many more citations than its cocited papers (Fig. 2A). Therefore, the yearly normalized citations, ${\widehat{c}}_t$, indicate that $p_1$ has a consistently lower yearly disciplinary impact than $p_2$ (Fig. 2B) and thus a lower total impact (14.1 vs. 30.7). In other words, ${\widehat{c}}_t$ measures the relative impact of each paper within the “discipline” it is embedded in, which may be a single discipline, or a mixture of multiple traditionally defined disciplines.

Fig. 2.

Comparing ${\widehat{C}}_{10}$ with $C_{10}$ and ${\tilde{C}}_{10}$, using two papers. The first paper ($p_1$) refers to ref. 14 (blue), and the second ($p_2$) refers to ref. 15 (orange). Both were published in 1985 in Nature. (A) Solid lines represent yearly raw citations, $c_t$, indicating that $p_1$ constantly received more citations than $p_2$ did and thus had a higher impact than $p_2$, based on $C_{10}$. Given that both papers were in the same year and journal group, ${\tilde{C}}_{10}$ also suggests $p_1$ had a higher impact than $p_2$. Dashed lines represent average yearly citations of the papers that are cocited with $p_1$ ($p_2$), indicating that $p_1$’s cocited papers had more citations than $p_2$’s, i.e., a higher ${〈c_t^n〉}_{n\in {\mathcal{N}}_t}$. (B) Our yearly normalized measure, ${\widehat{c}}_t$, which compares $c_t$ with $〈c_t^n〉$, suggests that $p_1$ has a lower impact than $p_2$.

Validating $\widehat{C}$.

We validate ${\widehat{C}}_{10}$ using external evaluations from domain experts on the importance of papers. For example, Physical Review Letters (PRL) released a list of 87 “Milestone Letters” that made significant contributions to the development of physics and can be considered as breakthroughs within their subdisciplines (https://journals.aps.org/prl/50years/milestones). First, we confirm that the selection of these papers was not driven by raw citation count, as they are not the most cited papers in the journal and their ranks based on $C_{10}$ among all PRL papers span several orders of magnitude (Fig. 3C). Still, milestone papers have higher impact than the average PRL papers, as measured by both $C_{10}$ and ${\widehat{C}}_{10}$ (Fig. 3 A and B). More importantly, when we compare the ranks of milestone papers based on $C_{10}$ and ${\widehat{C}}_{10}$, we find that these papers are consistently ranked higher if we use ${\widehat{C}}_{10}$ than if we use $C_{10}$ (Fig. 3C), indicating a better capability of ${\widehat{C}}_{10}$ to recover important papers. Methods like ${\tilde{C}}_{10}$ that use journals as the proxy for fields lose such capability, given that all the milestone papers were published in the same journal. The same effect is observed for milestone papers identified by three other journals (PNAS, PRE, and Human Relations), confirming the ability of ${\widehat{C}}_{10}$ to select papers that the scientific community considers important, independent of their citation counts and publication venue (SI Appendix, Fig. S2).

Fig. 3.

Validating ${\widehat{C}}_{10}$ using “Milestone Letters” published in Physical Review Letters (PRL). (A and B) Histograms of $C_{10}$ and ${\widehat{C}}_{10}$ for milestone papers and all PRL papers. (C) Comparison between $C_{10}$- and ${\widehat{C}}_{10}$-based ranks of milestone papers among all PRL papers. Rank 1 corresponds to the paper with the largest $C_{10}$ (${\widehat{C}}_{10}$). The diagonal dashed line represents equal ranks, and points above the line indicate better ranks based on ${\widehat{C}}_{10}$.

$\widehat{C}$ Corrects Temporal and Field Biases.

One widely known drawback of raw citation count is its dependence on publication time. This can be readily seen from Fig. 4A, where we plot, for each year from 1945 to 2007, the distribution of $C_{10}$ for papers published in that single year. It is apparent that these distributions shift systematically rightward over time, indicating an inflation process of $C_{10}$. For example, the median $C_{10}$ for papers in 1945 is 1, which increased to 9 in 2007; a paper in 1945 with 17 citations was able to become a hit, i.e., a top 5% most cited paper, but achieving the same status required 65 citations for papers in 2007. Such a dependence on time, however, is not presented in ${\widehat{C}}_{10}$. The distributions of ${\widehat{C}}_{10}$ for all the years considered appear to collapse onto a single shape (Fig. 4B; see also SI Appendix, Fig. S3), demonstrating a universality and lending a strong support for the temporal stability of ${\widehat{C}}_{10}$. We further fit a log-normal distribution for each individual year, since it is one of the most popular functional forms used in previous citation distribution analyses. We find that the distributions of ${\widehat{C}}_{10}$ of papers in individual years are compatible with the log-normal form with nearly the same shape parameter $\sigma =1$ to $1.2$ (SI Appendix, Fig. S4). This result is in line with numerous prior studies that identified a similar range of $\sigma$ for papers in journals (18–20), institutions (20), fields (21–23), as well as Mendeley readerships (24) and patent citations (25).

Fig. 4.

${\widehat{C}}_{10}$ corrects the temporal and field bias characterizing $C_{10}$. (A and B) The inflation of $C_{10}$ over time, in contrast to the temporal stability of ${\widehat{C}}_{10}$. We plot, for each year from 1945 to 2007, the survival distribution of (A) $C_{10}$ and (B) ${\widehat{C}}_{10}$ for papers published in that year. The color of a curve encodes publication year. While the distributions of $C_{10}$ shift rightward, the distributions of ${\widehat{C}}_{10}$ are well described by a single shape. (C) Percentages of papers published in each year that appear in the top 5% ranked by $C_{10}$, ${\widehat{C}}_{10}$, or ${\tilde{C}}_{10}$ among all papers in 1945 to 2007. For clarity, the curve for $C_{10}$ normalized by year is not shown, as it is essentially identical to the ${\tilde{C}}_{10}$ curve. The RCR case is not applicable due to the unavailability of RCR metrics for the entire WoS corpus. (D and E) The entropy of (D) years and (E) fields for the top publications ranked among all papers from 1980 to 2007 using $C_{10}$ (green), $C_{10}$ normalized by each year (red), $C_{10}$ normalized by each year and field (purple), RCR (blue), and ${\widehat{C}}_{10}$ (orange). In both cases, the corpus of papers refers to PubMed where RCR is available (26), the horizontal black line indicates the entropy for the entire corpus, and the vertical dashed line marks the top 5%. See also SI Appendix, Fig. S12 where we focus on the absolute entropy difference from the baseline.

The inflation of $C_{10}$ generates temporal bias favoring more recent papers. To demonstrate this bias more quantitatively, we rank all papers published in 1945 to 2007 according to $C_{10}$ and calculate the percentage of papers published in each year that appear in the top 5% of the global ranking. If $C_{10}$-based ranking is fair, it means that each year would contribute 5% of its papers to the top, which is far from the case: There is a systematic increase in the percentage of papers ranked into the top over time (Fig. 4C), suggesting that ranking by $C_{10}$ creates bias favoring more recent papers. However, if we rank papers by ${\widehat{C}}_{10}$, yearly contributions fluctuate around the baseline, without exhibiting the recency bias as $C_{10}$ does.

Another way to capture the recency bias is to explore the diversity of years present in a ranking of all publications. Here, we take the top $p$-percent of publications as ranked by different citation metrics and then measure the diversity of years using the entropy of the normalized year count distribution. If no bias was present in the ranking and all publications had an equal chance of being at the top, then all years would be present proportional to the total number of publications from that year and we would find an entropy of 3.29 bits, while a smaller entropy reflects less diversity and a bias toward specific years, and a larger entropy occurs for a greater diversity reflecting overcompensation. As shown in Fig. 4D, $C_{10}$ is the most biased indicator, selecting more top publications from the same years. On the other hand, $C_{10}$ normalized by the average in each year has a higher entropy than the corpus baseline, indicating an overcompensation with more publications from early years in the overall top ranking than would be expected based on their frequency. Notably, both RCR and $C_{10}$ normalized by each year and field also display this overcompensation and promote more publications from early years. Finally, ${\widehat{C}}_{10}$ shows a compromise between the tendency to over or under compensate and has an entropy of 3.25 bits for the top 5% of publications.

Another potential bias when recognizing top publications using $C_{10}$ comes from the drastically different sizes and citation norms in different disciplines. This can be observed from SI Appendix, Fig. S5A, where we show the distributions of $C_{10}$ for several fields. We observe that, from cell biology to analytical chemistry to mathematics, there is a systematic, one order of magnitude decease of median $C_{10}$. These differences nearly disappear if we use ${\widehat{C}}_{10}$ (SI Appendix, Fig. S5B).

Once again, we can capture the field bias by exploring the diversity of fields present in a ranking of all publications. Here, we take the top $p$-percent of publications as ranked by different citation metrics and then measure the diversity of fields using the entropy of the normalized field count distribution. If no bias was present in the ranking and all publications had an equal chance of being at the top, then all fields would be present proportional to the total number of publications from that field, resulting in an entropy of 4.35 bits. A smaller entropy reflects less diversity and a bias toward specific fields, and a larger entropy occurs for a greater diversity reflecting overcompensation. As shown in Fig. 4E, $C_{10}$ is an extremely biased indicator, selecting more top publications from the same few fields. Surprisingly, $C_{10}$ normalized by the average in each year is even more biased toward specific fields—its application increases the tendency to favor publications from specific fields. Notably, both ${\widehat{C}}_{10}$ and $C_{10}$ normalized by each year and field show significantly reduced bias toward a few fields and are consistently the closest to the random baseline. More specifically, $C_{10}$ normalized by each year and field is slightly less bias than ${\widehat{C}}_{10}$ for the top 6% of publications but tends to overcompensate for smaller fields at the 5% mark, while ${\widehat{C}}_{10}$ consistently has a smaller entropy and is closer to the random baseline for top publication sets of 6% to 10%. Finally, RCR shows a reduced bias toward specific fields compared to $C_{10}$, but is still more biased than ${\widehat{C}}_{10}$.

In summary, based on these diverse perspectives, we conclude that ${\widehat{C}}_{10}$ best corrects both the temporal and field biases characterizing $C_{10}$.

Fields Producing High-Impact Works.

Finally, we leverage the field- and time invariance of $\widehat{C}$ to assess science-wide advancement across discipline and time. It has long been posited that science advances through intermittent revolutionary breakthroughs that have long-lasting impact by triggering new directions of research and giving birth to new disciplines (27). While much quantitative attention has focused on characterizing the emergence of individual fields (28, 29), their growth dynamics (30–35), and the interactions between disciplines (36), little is known about the dynamics of individual breakthroughs within and across fields. This paucity of knowledge prompts us to ask the following: What fields produce high impact research and how does a field’s ability to stay at the forefront of the research enterprise change over time?

Answers to these questions rely on the accurate identification of breakthroughs. A straightforward way adopted in existing practice considers the top x% most cited papers grouped by year and field as breakthroughs (2). This, on one hand, mitigates the effects of temporal and disciplinary variations in citations but assumes that every field in every year generates breakthroughs at the same rate. Such a strong assumption about the pace of science advancement is highly unlikely to hold, considering, for example, the theoretical argument from Kuhn that scientific revolutions happen only sporadically (27). Since ${\widehat{C}}_{10}$ is not susceptible to temporal and disciplinary biases, we can directly compare the impact of papers across discipline and time. Therefore, we select from all the papers in our sample (including all years and disciplines) the top 5% with the highest ${\widehat{C}}_{10}$ and denote the obtained list as $\widehat{H}$. For comparison, we also identify top papers based on $C_{10}$ and ${\tilde{C}}_{10}$ and denote them respectively as $H$ and $\tilde{H}$. Our first observation is that $\widehat{H}$ and $H$ only share 49% of the papers and $\tilde{H}$ and $H$ 59% (SI Appendix, Fig. S6). This suggests that papers with low $C_{10}$ can still rank high based on ${\widehat{C}}_{10}$.

Turning to the question of which fields produce high impact papers, we compute the fraction of papers in $\widehat{H}$ that belong to a given field, finding that top contributors to $\widehat{H}$ include fields in physics, chemistry, and biomedicine (Fig. 5). Particularly interesting are the three of the top four categories, which are multidisciplinary chemistry, multidisciplinary sciences, and multidisciplinary physics. These correspond to multidisciplinary journals like Nature, Science, and PNAS that are perceived commonly by the academic community to publish selectively, as well as physics and chemistry journals like PRL and Journal of the American Chemical Society that publish quality contributions from any topics within the disciplines. SI Appendix, Fig. S7 confirms that these journals indeed published the most papers in $\widehat{H}$, corroborating the ability of ${\widehat{C}}_{10}$ to pick out high-impact papers. Several fields that account for only a small portion in $H$ become sizable contributors to $\widehat{H}$, including mathematics, electrical and electronic engineering, geochemistry and geophysics, organic chemistry, polymer science, etc. Among them, mathematics is noticeable for its steep ascending from a low representation in $H$ (0.19%; ranked 93rd) to a leading position in $\widehat{H}$ (1.6%; ranked 15th). On the other hand, many biomedical fields like cell biology, genetics and heredity, and microbiology that are ranked high when using $C_{10}$ move to lower positions when using ${\widehat{C}}_{10}$.

Fig. 5.

Rank comparison of fields by their shares in top 5% papers. We identify from all the papers in our sample the top 5% based on $C_{10}$, ${\widehat{C}}_{10}$, and ${\tilde{C}}_{10}$ and compute the fraction of these papers that belong to each field. The horizontal bars represent the shares, and the color represents the broad discipline. We only display the top 25 fields and report the fields with the most rank changes in SI Appendix, Table S1. Multidisciplinary Sciences is a hybrid category that includes multidisciplinary journals such as Nature, Science, PNAS, Scientific Reports, etc.

Dynamics of Production of High-Impact Works.

We characterize the temporal evolution of the production of high impact papers across fields, by taking field size into consideration, as larger fields would have more top papers by chance. In doing so, we introduce ${\widehat{r}}_{f,t}$ that measures the representation in $\widehat{H}$ by papers in field $f$ and year $t$. Specifically, it is the fraction of papers in $\widehat{H}$ that belong to field $f$ and year $t$, normalized by the fraction of papers of the same group in all the papers. Thus, ${\widehat{r}}_{f,t}>1$ indicates that the group is overrepresented in $\widehat{H}$ and ${\widehat{r}}_{f,t}<1$ underrepresented, thereby providing a quantification of the field’s ability to produce breakthrough articles relative to other fields at $t$. Similarly, we calculate $r_{f,t}$.

Let us first use Cell Biology as an example (Fig. 6). This field in 2007 accounted for 0.043% and 0.04% in $\widehat{H}$ and in all the papers, respectively, indicating its overrepresentation in $\widehat{H}$ ($\widehat{r}=1.07$). The heatmaps presented in Fig. 6A that encode ${\widehat{r}}_t$ and $r_t$ by color reveal the dynamics of the production of high impact research from Cell Biology over six decades. Based on $r_t$, it has been consistently a source for revolutionary scientific breakthroughs, which partly reflects high average citations of papers in this field. A much richer dynamics is revealed by ${\widehat{r}}_t$: Relative to other fields, Cell Biology lost its ability to produce breakthroughs between 1965 and 1980s and has regained its leading role in the early 1990s. We hypothesize that the rebound may be related to the emergence of genomics, as exemplified by the Human Genome Project initiated in the 1990s (38). To corroborate this, we identify overrepresented title words of Cell Biology papers published in each decade, compared to papers before 1980, finding that the 1990s were characterized by the rise of studies centered around gene expression (Fig. 6B).

Fig. 6.

A case study of Cell Biology. (A) Dynamics of the production of top cited papers from Cell Biology. (B) Overrepresented title words of Cell Biology papers in each decade, relative to papers before 1980 (37).

We expand the analysis to other fields and group them based on their $r_t$ and ${\widehat{r}}_t$ (Fig. 7). The first group corresponds to fields that continue to produce high impact works disproportionately during the studied period (${\widehat{r}}_t>1$, $r_t>1$), as identified by both measures, including Neuroscience, Astronomy and Astrophysics (Fig. 7A). The second group is featured by ${\widehat{r}}_t<1$ and $r_t>1$. Those include many Biomedical Research and Clinical Medicine fields that are ranked high only by $C_{10}$ (Fig. 7B). The third group includes fields whose importance is dismissed by $r_t$ ($r_t<1$) but picked up by ${\widehat{r}}_t$ (${\widehat{r}}_t>1$). Those fields span diverse disciplines (Fig. 7C), including i) within the Earth and Space category, Geochemistry and Geophysics, Meteorology and Atmospheric Sciences, Limnology, and Oceanography; ii) several physics fields, including Fluids and Plasmas Physics, Applied Physics, and Optics; iii) Electrochemistry and Physical Chemistry; iv) Ecology; and v) Surgery, Clinical Neurology, and Radiology from Clinical Medicine. The contrast of the likelihood to produce high impact works for fields in the third category also leads us to ask the following: Are there no periods of time when there are “excitements” going on in those fields, as suggested by $r_t$? We argue that this is not the case. For example, considering the development of medical imaging, there has been tremendous progresses in this area since the 1970s, and advancements like computer-assisted tomography (CT) and MRI have been quickly applied for medical diagnostics and won the 1979 and 2003 Nobel Prize in Physiology or Medicine (39). The fields of the key breakthroughs behind these development, like radiology (“Radiology, Nuclear Medicine & Medical Imaging”), are dismissed by $r_t$ but captured by ${\widehat{r}}_t$.

Fig. 7.

Production dynamics of top papers in selected fields. We group fields based on their ${\widehat{r}}_t$ and $r_t$ values: (A) fields that are overrepresented based on both ${\widehat{r}}_t$ and $r_t$; (B) fields that are underrepresented based on ${\widehat{r}}_t$ and overrepresented based on $r_t$; and (C) fields that are overrepresented based on ${\widehat{r}}_t$ and underrepresented based on $r_t$.

Finally, going beyond case studies, we examine which features of a field explain its $r_t$ (${\widehat{r}}_t$). We hypothesize that field size and number of references by field may correlate with $r_t$ and ${\widehat{r}}_t$, as previous studies have pointed out that the two factors contribute to the temporal and field biases in raw citations (40). We find that field size is less correlated with ${\widehat{r}}_t$ than with $r_t$ (SI Appendix, Fig. S8). More importantly, the average number of references per paper is significantly less correlated with ${\widehat{r}}_t$ than with $r_t$ (the median coefficient of determination $R^2=0.16$ vs. 0.49; SI Appendix, Fig. S9). These results further support the ability of ${\widehat{C}}_{10}$ to correct the systematic biases of $C_{10}$ and ${\widehat{r}}_t$ rather than $r_t$ as a viable indicator for a field’s tendency to produce high impact research.

Discussion

Citation-based impact metrics have been increasingly adopted for academic performance evaluations of diverse types of actors—authors (41), institutions (42), and even nations (43), playing an important role in hiring, funding, and promotion (44). It is therefore essential to carry out citation analysis in an unbiased way. Yet, raw citations are known to be biased by variations in citation patterns across discipline and time, prompting us to propose a properly normalized measure that corrects those biases.

The fact that both RCR and $\widehat{C}$ rely on cocited papers for citation normalization raises the question of the differences between the two indicators. Our systematic comparisons of them indicate that our $\widehat{C}$ can better correct the field bias than RCR does, yet for year bias, all metrics underperform, meaning that they all identify top papers that were more likely to be published recently. Such a undercompensation might indicate that, at least in biomedicine, disruptive science may not be declining as suggested in a previous study (45). Furthermore, $\widehat{C}$ identifies a very different set of highly influential papers from the set by RCR, sharing only 54% of the top 5% papers. The rankings of fields based on their shapes in top papers also reveal that, while fields like General and Internal Medicine, Neuroscience, and Surgery are found to be important by both methods, the importance of multidisciplinary physics and chemistry as well as radiology are not recognized by RCR (SI Appendix, Fig. S11).

Throughout this work, we used citation data to quantify the scientific impact of a scientific article, aiming to unveil how and when raw citation signals impact. We acknowledge that scientific impact is a complex, multidimensional notion, easier to intuit than to quantify, and getting cited may capture only one aspect of impact as codified in the discourse of science. Supplementing citations with contextual information extracted from text analysis may help capture more nuanced notion of impact (46). Similarly, looking at citations from other domains, like patents and policy documents, may also enrich the multifaceted feature (47, 48). Still, due to the accessibility and quantity of article citation data, citations provide valuable insights into the evolution of scientific discovery and citation-based analyses form the cornerstone of the science of science. In implementing our methodology, given the content and structure of current bibliographic databases, it is easier to calculate our metric for individual papers than existing normalization metrics (SI Appendix, Text). Yet, when the number of citations is small, the cocitation neighborhood might also be small and therefore be less reliable for normalization. Mirroring the raw citation count, the cocitation network may also be influenced by self-citations, large authorship teams, or other factors related to the social processes of science that may affect how citations are generated and consequently affect the network (49). For example, publicity, such as comments and promotion in social media, and attention to high Impact Factor journals may induce additional citations. Similarly, social and epistemological considerations may also generate influence citations that are interpreted as expressions of “discursive relation” or “professional relation” to scientific communities. To address these factors, future work can utilize datasets which capture diverse aspects of the social processes, and derive citation networks that better reflect impact. For example, very high profile results are often not cited explicitly, but only mentioned in the text, acquiring hidden citations (50). Our methodology is then ready to be adapted to address the role of these modified graphs. Finally, we focused only on articles, which may not be the main publication medium for some fields in social sciences, humanities, and computer science.

Despite these limitations, our proposal of a citation-based measure that is time invariant and free from discipline bias allows us to compare the impact of papers across years and disciplines. A key contribution of our method is the elimination of the need to assign a publication to a discipline when measuring its impact. Previous research has demonstrated that science is becoming increasingly interdisciplinary (9), complicating the traditional picture of science structured into well-defined research departments and funding programmes. Thus, notions of locally comparable research, such as we introduced here, provide an important step toward studying the interactions between scientific disciplines and the emergence of new research areas. To this end, we demonstrate that contributions to revolutionary breakthroughs in the past half century came from diverse disciplines, such as radiology, applied physics, ecology, and geosciences, as opposed to be dominated by biomedical sciences.

Materials and Methods

We based our analysis on the Web of Science (WoS) database. We only considered “article,” “letter,” and “note” documents indexed there and limited our attention to 26,792,332 papers published between 1945 and 2007 to ensure a 10-y citation window. Citing documents were constrained within the three selected types; therefore, citations from other types of papers such as editorial and review were not included. Research fields of papers were taken as WoS Subject Category (SC), and papers can have more than one SC.

The $\tilde{C}$ measure uses papers published in the same SC and year as reference set and normalizes the number of citations by the average citations of papers in the set (51, 52). Therefore, this indicator relies on external category labels of papers. Applying the procedure in practice also requires one to make the choice on how to deal with papers with multiple categories, as 36.2% of papers have more than one category (SI Appendix, Table S2). There are several ways to handle those papers. A straightforward one is to count them multiple times. This, however, would artificially increase the number of papers drastically, which would lead to the increase in the number of top papers. Here, we first calculate the ${\tilde{C}}_{10}$ value for each category assigned to a paper and then pick the maximum one. We ignore the 56,991 papers without category labels.

We obtain RCR metrics from the NIH Open Citation Collection (Version 40) (26, 53), which focuses on PubMed papers. We retain only research articles published in 1980 to 2007, as a large fraction of pre-1980 papers do not have RCR values (SI Appendix, Fig. S10). We then match the retained papers to WoS using PubMed ID or DOI and drop unmatched papers from our analysis. The final corpus contains 7,710,057 PubMed papers.

A Python implementation of $\widehat{C}$ is provided in the pySciSci package (54).

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Replication data have been deposited in Github (https://github.com/qke/network-normed-c) (55). Some study data available the raw Web of Science data used in this work cannot be shared due to its proprietary nature but is available upon purchase from Clarivate Analytics at https://clarivate.com/contact-us/sales-enquiries/. Other relevant data supporting the replication of this work have been deposited at https://github.com/qke/network-normed-c (55). A Python implementation of the proposed measure is available in the pySciSci package (https://github.com/SciSciCollective/pyscisci).