Abstract
Objective
This study aims to identify determinants of high impact, measured by Impact Factor (IF) and Eigenfactor score, among otolaryngology journals.
Methods
Bibliometric data of “otorhinolaryngology” journals were collected from the Journal Citation Reports (JCR) database. For the years 2009–2020, we collected normalized Eigenfactor score, 5‐year IF, immediacy index, fraction of IF from journal‐self citation, proportion and magnitude of published citable articles, and total citation counts. High‐IF and ‐Eigenfactor journals were considered those within the top‐quartile of that metric each respective year.
Results
High‐IF and ‐Eigenfactor otolaryngology journals displayed higher 5‐year IFs, immediacy indexes, and IF without self‐citation (p < .05 for all years) including total citations counts and citable articles when ranked by Eigenfactor (p < .05 for all years). Otolaryngology IF correlated with 5‐year IF and immediacy index within the same year (p < .05 for all years) and from previous years (p < .05 for all years; p < .05 for 2017‐2018; p > .05 for 2009‐2016). Eigenfactor correlated with 5‐year IF, total citation counts, and citable articles within the same year (p < .05 for all years) and previous years (p < .05 for 2013–2018). Multilinear regression revealed that 5‐year IF (p < .05 for 2009–2018) and immediacy index from the prior 2 years (p < .05 for 2017‐2018; p > .05 for 2009‐2016) predicted 2019 IF. Similarly, 5‐year IF, total citation counts, and citable articles (p < .05 for 2013–2018) predicted 2019 Eigenfactor score.
Conclusion
Sustained publication of impactful articles is the dominant driver of high IF and Eigenfactor score. Eigenfactor score reflects a unique evaluation of otolaryngology journals; ranking otolaryngology journals by their Eigenfactor scores significantly alters journal ranking compared to ranking by IF.
Level of evidence
NA.
Keywords: bibliometric, citable items, determinants, immediacy index, impact factor, journal citation reports, otolaryngology, scientific literature
Short abstract
Bibliometrics, most prominently the Impact Factor, are commonly used not only as a measure of a journal's impact and influence but also as an unintended measure of prestige and quality for a journal, the articles it publishes and its authors. Herein, we study journal characteristics and metrics to identify the factors that most determine Impact Factor and Eigenfactor Score for journals in the field of otolaryngology. We show that these bibliometrics of journal impact are most dominantly determined by the consistent publication of timely, impactful, and well‐cited articles.
1. INTRODUCTION
The impact, prestige, and reputation of scientific journals can be assessed and judged using various bibliometrics, most prominently with the journal impact factor (IF) which was originally designed by Eugene Garfield in 1963 as a metric of journal influence specifically intended for helping librarians decide which journal subscriptions to purchase. The IF reflects an estimate of the arithmetic mean number of citations garnered by a single, published article within that journal during the preceding 2 years (“citation window”). 1 , 2 , 3 Over time, however, the connotation of the IF has evolved to reflect the quality of a scientific journal, and—by extension—the articles it publishes and the authors who publish in it. 4 , 5 , 6 , 7
Interpretation of any bibliometric, such as IF, as a reflection of the quality or scientific rigor of a journal, its articles and its authors has a number of limitations. For instance, one intrinsic limitation of IF is the presence of citation skew, which is due to a small number of highly cited articles that elevate IF. 8 , 9 , 10 Moreover, the IF may be deliberately elevated by promoting self‐citations, publishing highly cited review articles over primary manuscripts, and prioritizing the publication of articles that garner broad public interest as opposed to niche audiences. 11 , 12 , 13 The recently developed Eigenfactor is one proposed alternative to the conventional IF that aims to address the specific limitations of the IF. The Eigenfactor algorithm considers the “weight” of a citation (e.g., which journal the citation is coming from), eliminates the influence of self‐citations, and utilizes data over a 5‐year period. 14 Nevertheless, despite its limitations and the availability of viable metric alternatives, the IF remains an important metric used in assessing a journal, its publications and its authors by academicians, their institutions, and their funding agencies. 2 , 13 , 15 , 16 , 17 , 18 , 19 , 20 As a result of the IF's perceived reflection on a journal's reputation and influence, increasing IF is frequently an objective of journal editors and publishers. 21 , 22 , 23 , 24 , 25 Moreover, the import that IF has to journals’ readerships and authors is reflected, for example, by the prominent placement of the IF on journal websites as well as reference to IF in editorials. 26 , 27
With the significance held by IF in the reputation of journals as well as their respective articles and authors, an understanding of the determinants of IF is clearly of great significance to the readership and editorial offices of the otolaryngology scientific literature. 7 , 28 While previous studies of other medical specialties' journal IFs have revealed informative insights, 4 , 7 , 10 , 18 , 19 , 24 , 29 , 30 , 31 , 32 , 33 , 34 a focused and thorough study of the determinants of IF has not been performed for the otolaryngology scientific literature. In this study, our objective was to identify characteristics prevalent in otolaryngology journals with high IF. We also evaluated the relationship of journals' IFs to their corresponding Eigenfactor score, an emerging alternative to IF. We believe that a better understanding of the determinants of IF and Eigenfactor score in otolaryngology journals would better inform the readership of how to interpret these bibliometrics as well as inform the broader otolaryngology scientific editorial community about how these bibliometrics may be improved.
2. METHODS
2.1. Study design
This study received an exemption for review from the University of Cincinnati College of Medicine Institutional Review Board. Characteristics and performance metrics of “otorhinolaryngology” journals were collected from the Journal Citation Reports (JCR) database on January 21, 2022. All journals listed in the “otorhinolaryngology” category of the 2020 JCR database were included. Exclusion criteria included not accepting both original/primary articles and review articles and the journal having fewer than 10 citations in a year. We specifically chose to study IFs up to the year 2020 (i.e., did not include 2021 IF) to exclude the unique effect of CoVID‐19‐related scientific literature and citations. For all included journals, we collected the following data from the JCR database for the years 2009–2020: IF, 5‐year IF, immediacy index, normalized Eigenfactor, total citations, IF excluding self‐citations (IFsSC), and the percent and number of published items that were considered “citable items.” Using these data, we also calculated the percentage of the IF that came from journal self‐citations (ΔIFsSC) as (IF − IFsSC)/IF.
2.2. Outcome measures
We specifically focused on short‐term and long‐term publication of impactful articles, as well as factors that can be used to artificially elevate IF: the effect of journal self‐citation and the fraction of “citable” articles published (which can affect the denominator in the IF calculation). The IF is calculated by dividing the total number of citations for publications in a given journal during the preceding 2‐year period by the number of total citable items for that journal within that same time period/citation window. 1 , 2 , 3 As an example, the IF of a journal for 2020 would be calculated by the number of citations the journal received in 2018–2019 divided by the number of citable items published in the 2018–2019 citation window. The 5‐year IF is calculated in the same way as the standard IF but instead uses a 5‐year citation window. Compared to the standard IF, the 5‐year IF more greatly reflects long‐term publication of impactful articles. The immediacy index is also calculated in the same manner as the standard IF but instead of the citation window consisting of the prior 2 years, the citation window is specifically for that year. For example, the immediacy index of a journal for 2020 would be calculated by the number of citations the journal received in 2020 divided by the number of citable items published in the 2020 citation window. Compared to the IF, the immediacy index more greatly reflects the publication of articles focusing on immediately applicable (i.e., “trending” or “hot”) topics. IFsSC is calculated the same as the IF but excludes citations from the corresponding journal (i.e., eliminates instances of a journal citing its own articles). Finally, while the citation of every item published in a journal can potentially be included in the numerator of the IF calculation, not every item published in a journal is counted as a citable item but the denominator of the IF calculation consists only of the number of citable items. Therefore, it is advantageous for a journal to have many citations while simultaneously having as few citable items as possible to maximize the IF calculation. While it is not entirely clear how the JCR defines citable vs. non‐citable items, it is believed that citable items include original manuscripts/studies and review articles while non‐citable items typically include works such as editorials, letters, and brief reports. Although the exact distinction between citable vs. non‐citable items is not published by the JCR, the JCR does report the percentage of items published by a journal that it deems to be citable and we directly obtained this information from the JCR (Table S1).
The Eigenfactor is calculated through a complex algorithm but essentially reflects the total number of differentially weighted citations over a five‐year period (excluding self‐citations) relative to the number of published, citable articles. 14 , 35 Normalization of the Eigenfactor involves its rescaling such that a journal with average influence across JCR has a normalized Eigenfactor of 1. This allows journal scores to be compared relative to a score of 1. For instance, a normalized Eigenfactor of 3 would reflect that a journal is 3 times as influential as the average journal in JCR, across all specialties.
2.3. Statistical analysis
All statistical analyses were carried out with GraphPad Prism 9 software package. For each year, journals were stratified into two comparator categories based on whether they were within the top 25th percentile (i.e., top quartile) or bottom 75th percentile by IF or normalized Eigenfactor in that specific year. Between these two categories, we compared 5‐year IF, immediacy index, IF excluding self‐citations (IFsSC), percentage of IF that came from journal self‐citations (ΔIFsSC), percent of published items that were considered "citable", and the total number of citations and citable articles. Comparisons were made with the non‐parametric Mann–Whitney U. Correlations were calculated using Pearson's method. Multiple linear regressions were calculated using least squares assuming a Gaussian distribution. Statistical significance was defined by a p‐value of less than .05.
3. RESULTS
3.1. Otolaryngology journals with the highest impact factors and normalized Eigenfactor scores publish more cited articles in the short‐ and long‐term
Because IF is based on a calculation that depends on citations over a specific two‐year period, we sought to determine whether the IF reflected a journal's short‐term and long‐term publication of impactful articles. As a reflection of a journal's short‐term impact, we chose the metric of immediacy index and as a reflection of long‐term impact, we chose the metric of 5‐year IF (Table S1). We found that across all years (2015–2020), otolaryngology journals in the top‐quartile of IF had higher immediacy indexes and higher 5‐year IFs compared to journals in the bottom 75th percentile (Figure 1A,B). We also found that IF was significantly correlated with 5‐year IF and immediacy index over all years (Table 1). Likewise, otolaryngology journals in the top‐quartile of normalized Eigenfactor scores also displayed higher 5‐year IFs and immediacy index compared to the bottom 75th percentile across all years, with the exception of immediacy index comparisons in 2018 (Figure 1C,D). However, the normalized Eigenfactor was significantly correlated only with 5‐year IF, but not immediacy index, across all years with the exception of 2015 (Table 2). These findings suggest that otolaryngology journals with higher IF and normalized Eigenfactor scores are more likely to publish articles that garner both more immediate and long‐term attention and value from the scientific community.
FIGURE 1.
Plotted 5‐year IF (A, C) and immediacy index (B, D) of otolaryngology journals defined as either top (blue open circles) or bottom (black open circles) quartile at each year interval between 2015and 2020 based on IF (A, B) or normalized Eigenfactor (C, D). ****p < .0001, ***p < .001, **p < .01, *p < .05, ns—p > .05.
TABLE 1.
Correlation analysis between journal impact factor and bibliometrics at each year interval.
| Bibliometric | 2020 | 2019 | 2018 | |||
|---|---|---|---|---|---|---|
| r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | |
| 5‐year IF | 0.8828 (0.7880 to 0.9367) | <.0001 | 0.9483 (0.9030 to 0.9728) | <.0001 | 0.9493 (0.9047 to 0.9733) | <.0001 |
| Immediacy Index | 0.6052 (0.3619 to 0.7713) | <.0001 | 0.5878 (0.3343 to 0.7620) | <.0001 | 0.4274 (0.1293 to 0.6546) | .0067 |
| ΔIF s Self‐Cites (%) | 0.1349 (−0.1887 to 0.4320) | .4130 | 0.1406 (−0.1831 to 0.4367) | .3934 | 0.0384 (−0.2805 to 0.3497) | .8165 |
| Articles in Citable Items (%) | −0.3680 (−0.6096 to ‐0.06382) | .0195 | −0.2179 (−0.4991 to 0.1048) | .1826 | 0.01517 (−0.3018 to 0.3291) | .9270 |
| Total Citations (#) | 0.3319 (0.01833 to 0.5861) | .0390 | 0.3462 (0.03449 to 0.5966) | .0308 | 0.2744 (−0.04506 to 0.5429) | .0910 |
| Total Citable Articles (#) | 0.1476 (−0.1762 to 0.4425) | .3700 | 0.1233 (−0.2000 to 0.4224) | .4546 | 0.09188 (−0.2303 to 0.3959) | .5780 |
| Bibliometric | 2017 | 2016 | 2015 | |||
|---|---|---|---|---|---|---|
| r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | |
| 5‐year IF | 0.9610 (0.9256 to 0.9797) | <.0001 | 0.9597 (0.9240 to 0.9788) | <.0001 | 0.9462 (0.8991 to 0.9716) | <.0001 |
| Immediacy Index | 0.6278 (0.3854 to 0.7891) | <.0001 | 0.6066 (0.3599 to 0.7740) | <.0001 | 0.7274 (0.5346 to 0.8483) | <.0001 |
| ΔIF s Self‐Cites (%) | 0.1830 (−0.1452 to 0.4749) | .2715 | −0.0265 (−0.3392 to 0.2915) | .8728 | −0.04780 (−0.3579 to 0.2718) | .7726 |
| Articles in Citable Items (%) | −0.001285 (−0.3208 to 0.3185) | .9939 | −0.02504 (−0.3379 to 0.2928) | .8797 | −0.4111 (−0.6432‐ ‐0.1099) | .0093 |
| Total Citations (#) | 0.3451 (0.02374 to 0.6018) | .0364 | 0.4364 (0.1402 to 0.6609) | .0055 | 0.4192 (0.1194 to 0.6489) | .0079 |
| Total Citable Articles (#) | 0.1468 (−0.1861 to 0.4494) | .3860 | 0.3478 (0.03629 to 0.5977) | .0300 | 0.2265 (−0.09586 to 0.5059) | .1656 |
Abbreviations: F, impact factor; r, Pearson correlation coefficient; ΔIF s Self‐Cites, fraction of the impact factor that comes from journal self‐citations.
TABLE 2.
Correlation analysis between journal normalized Eigenfactor score and bibliometrics at each year interval.
| Bibliometric | 2020 | 2019 | 2018 | |||
|---|---|---|---|---|---|---|
| r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | |
| 5‐year IF | 0.4025 (0.09962 to 0.6371) | .0111 | 0.4552 (0.1631 to 0.6739) | .0036 | 0.4368 (0.1407 to 0.6612) | .0054 |
| Immediacy Index | 0.1764 (−0.1473 to 0.4660) | .2826 | 0.2637 (−0.05653 to 0.5347) | .1048 | 0.1459 (−0.1778 to 0.4411) | .3755 |
| Articles in Citable Items (#) | 0.8605 (0.7480 to 0.9249) | <.0001 | 0.8575 (0.7430 to 0.9232) | <.0001 | 0.8496 (0.7297 to 0.9188) | <.0001 |
| Total Citations (#) | 0.9445 (0.8959 to 0.9707) | <.0001 | 0.9323 (0.8738 to 0.9642) | <.0001 | 0.9247 (0.8601 to 0.9601) | <.0001 |
| Bibliometric | 2017 | 2016 | 2015 | |||
|---|---|---|---|---|---|---|
| r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | |
| 5‐year IF | 0.4246 (0.1214 to 0.6554) | .0079 | 0.4796 (0.1933 to 0.6906) | .0020 | 0.4789 (0.1924 to 0.6901) | .0020 |
| Immediacy Index | 0.2961 (−0.02604 to 0.5625) | .0711 | 0.3094 (−0.006759 to 0.5694) | .0553 | 0.3508 (0.03963 to 0.5999) | .0286 |
| Articles in Citable Items (#) | 0.8713 (0.7645 to 0.9315) | <.0001 | 0.8741 (0.7714 to 0.9325) | <.0001 | 0.8371 (0.7088 to 0.9118) | <.0001 |
| Total Citations (#) | 0.9355 (0.8786 to 0.9662) | <.0001 | 0.952 (0.9097 to 0.9747) | <.0001 | 0.9555 (0.9162 to 0.9766) | <.0001 |
Abbreviations: IF, impact factor; r, Pearson correlation coefficient.
3.2. Journal self‐citation is not a primary driver of otolaryngology journal impact factor rankings
Because previous studies of the scientific literature within other medical specialties and the basic sciences have shown that journal self‐citations (and possible promotion of self‐citations) could be a mechanism by which journals raise their IF, 30 , 33 we sought to determine whether this was also the case among otolaryngology journals. We therefore explored this by comparing the IF excluding self‐citations (IFsSC) between otolaryngology journals in the top quartile versus bottom 75th percentile (Table S1). After removal of journal self‐citations, the journals in the top quartile by IF still generally had higher IFsSC (Figure 2A). Without consideration of journal self‐citations, of the journals originally in the bottom 75th percentile by IF, only three journals in 2017, four journals in 2018, 2 journals in 2019 and 1 journal in 2020 would have an IF within the range of the top quartile, that is, by IFsSC (Table S1).
FIGURE 2.
Plotted IF excluding self‐citations (IFsSC) (A) and the proportion of IF from self‐citations (i.e., ΔIFsSC) (B) of otolaryngology journals defined as either top (blue open circles) or bottom (black open circles) quartile at each year interval between 2015 and 2020. ****p < .0001.
Therefore, we next sought to determine what fraction of otolaryngology journals' IFs arose from self‐citations (ΔIFsSC) and whether this was different between the journals within the top quartile of IF versus bottom 75th percentile of IF (Table S1). We found no significant difference in the percentage of the IF that came from journal self‐citations for otolaryngology journals in the top quartile by IF vs. the bottom 75th percentile of IF for all years (Figure 2B). We also found no correlation between IF and percent of IF that came from journal self‐citations for all years (Table 1). In fact, otolaryngology journals in the bottom 75th percentile displayed higher variance than the top‐quartile journals with a higher maximum in the percentage of IF that came from journal self‐citations (Figure 2B). Together, these findings suggest that journal self‐citations do not appear to significantly drive IF differences between otolaryngology journals in the top quartile vs. bottom 75th percentile by IF.
3.3. The impact of total citation numbers and citable items in driving otolaryngology journal impact factor and normalized Eigenfactor scores
Dissimilarities among journals in the frequency of publication of citable articles may in part dictate differences seen between IFs by impacting the denominator in the IF calculation. 36 We next explored whether differences existed in the fraction of articles that are considered citable by JCR in otolaryngology journals that were in the top quartile versus bottom 75th percentile of IF from 2015 to 2020. In other words, we sought to determine whether otolaryngology journals in the top quartile of IF were more likely to have fewer of their published articles count as citable items compared to journals in the bottom 75th percentile of IF as one mechanism for their higher IFs (Table S1). However, we found no significant difference in the percentage of published articles considered to be citable items between otolaryngology journals in the top‐quartile versus bottom‐75th percentile for IF in every year from 2015 to 2020 (Figure 3A). We also found no correlation between IF and percentage of articles deemed citable by JCR for all years except 2015 and 2020 where we observed a modest but significant negative correlation (Table 1). Paralleling these results, we found no significant difference in the total number of citable articles between the top‐quartile and bottom‐75th percentile (Figure 3B), nor did we observe a significant correlation between IF and the total number of citable articles for all years except 2016, where a statistically significant correlation was measured (Table 1). Given that the quantity of citable articles did not seem to drive differences between top‐quartile and bottom‐75th percentile journals, we next wondered whether the magnitude of total raw citations would (Table S1). Given that journal IF calculations rely on total raw citation counts, we suspected that a difference in total citation counts would be seen between our groups. Surprisingly, we found no significant difference in total citation counts between top‐quartile and bottom‐75th percentile groups (Figure 3C). However, we found a significant correlation between IF and total citations across all years examined except 2018 (Table 1). These results suggest that publication of fewer “citable” articles is not a major mechanism for achieving high IF by otolaryngology journals. Conversely, acquiring more raw citations is important to drive high journal IF but as a metric alone does not distinguish well between an influential and non‐influential journal.
FIGURE 3.
Plotted percentage of citable items (A), total number of citations (B, D) or citable articles (C, E) of otolaryngology journals defined as either top (blue open circles) or bottom (black open circles) quartile at each year interval between 2015 and 2020 based on IF (A, B, C) or normalized Eigenfactor (D, E). ****p < .0001, ns—p > .05.
Eigenfactor by design excludes the influence of self‐citations and instead includes in its calculation the total number of “weighted” citations relative to total number of citable articles. 14 Given this unique calculation method for journal impact, we were curious about whether a relationship exists between an otolaryngology journal's Eigenfactor score and its total raw citation count as well as citable articles (Table S1). We explored this by first examining whether top‐quartile otolaryngology journals ranked by their normalized Eigenfactor scores contained more total citations and citable articles than their counterparts. We found that in every year from 2015 to 2020 the top‐quartile journals displayed significantly higher numbers of total citations than their bottom‐quartile counterparts (Figure 3D). However, only between the years of 2018 and 2020 did top‐quartile journals contain significantly higher numbers of citable articles (Figure 3E). We then wanted to know whether a high number of citations gained or citable articles published by a journal correlated with a high Eigenfactor score. Indeed, we found that both the total number of citations and citable articles significantly correlated with a high normalized Eigenfactor score across all years examined (Table 2). These findings suggest that otolaryngology journals with higher normalized Eigenfactor scores generally tended to publish more articles that garner more total citations.
3.4. Correlation of past bibliometric scores to 2019 journal impact factor and normalized Eigenfactor scores
The 5‐year IF, immediacy index, and total citations appear to be important determinants of an otolaryngology journal's IF within a given year (Figure 1A,B, Table 1). We next sought to determine whether these bibliometrics had predictive merit in determining a journal's future IF. We evaluated this by determining the correlation between an otolaryngology journal's IF in 2019 and the 5‐year IF, immediacy index, total number of citations, ΔIFsSC, percentage of citable articles, and the total number of citable articles of prior years between 2009 and 2018 (Table S1). We found that across all examined years, both the 5‐year IF and immediacy index were significantly correlated with a high 2019 journal IF (Table 3). Interestingly, the total number of citations was only found to significantly correlate with a high 2019 journal IF in 2018 but not all other years examined (Table 3). In line with our cross‐sectional findings, neither the ΔIFsSC, the percentage of citable articles, nor the total number of citable articles in the past (2009–2018) significantly correlated with future (2019) journal IF (Table 3). This suggests that an otolaryngology journal with a history of publishing articles of immediate interest as well as those with long‐term citation‐worthy content is more likely to have high journal IF in the future.
TABLE 3.
Correlation analysis between 2019 journal impact factor and bibliometrics at each indicated year between and including 2009–2018.
| Bibliometric | 2018 | 2017 | 2016 | 2015 | 2014 | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | |
| 5‐year IF | 0.9053 (0.8256 to 0.9496) | <.0001 | 0.9195 (0.8508 to 0.9573) | <.0001 | 0.8437 (0.7197 to 0.9155) | <.0001 | 0.8404 (0.7143 to 0.9136) | <.0001 | 0.7418 (0.5499 to 0.8593) | <.0001 |
| Immediacy Index | 0.4942 (0.2117 to 0.7005) | .0014 | 0.6771 (0.4561 to 0.8194) | <.0001 | 0.636 (0.4009 to 0.7925) | <.0001 | 0.5954 (0.3446 to 0.7668) | <.0001 | 0.6039 (0.3563 to 0.7723) | <.0001 |
| ΔIF s Self‐Cites (%) | 0.07259 (−0.2486 to 0.3794) | .6605 | 0.2887 (−0.03418 to 0.5569) | .0788 | −0.07121 (−0.3782 to 0.2499) | .6666 | −0.02232 (−0.3355 to 0.2953) | .8927 | −0.08844 (−0.4010 to 0.2425) | .6027 |
| Articles in Citable Items (%) | 0.02585 (−0.2920 to 0.3386) | .8759 | 0.06676 (−0.2584 to 0.3784) | .6905 | 0.07363 (−0.2476 to 0.3803) | .6560 | −0.3133 (−0.5722 to 0.002464) | .0521 | −0.2659 (−0.5431 to 0.06364) | .1117 |
| Total Citations (#) | 0.3193 (0.004196 to 0.5767) | .0476 | 0.3105 (−0.01502 to 0.5765) | .0614 | 0.2828 (−0.03589 to 0.5493) | .0811 | 0.2802 (−0.03870 to 0.5474) | .0840 | 0.2699 (−0.04987 to 0.5395) | .0966 |
| Total Citable Articles (#) | 0.1240 (−0.1993 to 0.4229) | .4521 | 0.1446 (−0.1883 to 0.4476) | .3932 | 0.1614 (−0.1624 to 0.4538) | .3264 | 0.1078 (−0.2150 to 0.4094) | .5135 | 0.1315 (−0.1920 to 0.4292) | .4249 |
| Bibliometric | 2013 | 2012 | 2011 | 2010 | 2009 | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | |
| 5‐year IF | 0.8125 (0.6572 to 0.9016) | <.0001 | 0.6952 (0.4574 to 0.8402) | <.0001 | 0.7123 (0.4735 to 0.8535) | <.0001 | 0.6289 (0.3410 to 0.8090) | .0003 | 0.6359 (0.3374 to 0.8182) | .0004 |
| Immediacy Index | 0.3252 (0.001318 to 0.5873) | .0495 | 0.7139 (0.5034 to 0.8444) | <.0001 | 0.4543 (0.1372 to 0.6869) | .0070 | 0.6186 (0.3442 to 0.7957) | .0002 | 0.5286 (0.2079 to 0.7467) | .0027 |
| ΔIF s Self‐Cites (%) | −0.1376 (−0.4502 to 0.2050) | .4305 | −0.01194 (−0.3438 to 0.3226) | .9457 | −0.07031 (−0.4039 to 0.2797) | .6974 | −0.2355 (−0.5399 to 0.1233) | .1944 | −0.02786 (−0.3968 to 0.3488) | .8881 |
| Articles in Citable Items (%) | −0.3749 (−0.6295 to −0.04761) | .0265 | 0.004484 (−0.3293 to 0.3372) | .9796 | 0.1326 (−0.2208 to 0.4552) | .4621 | 0.02196 (−0.3350 to 0.3734) | .9066 | −0.2147 (−0.5442 to 0.1721) | .2725 |
| Total Citations (#) | 0.2483 (−0.08232 to 0.5297) | .1384 | 0.2799 (−0.07020 to 0.5685) | .1147 | 0.3165 (−0.01868 to 0.5878) | .0639 | 0.2992 (−0.04337 to 0.5788) | .0856 | 0.2476 (−0.1237 to 0.5581) | .1871 |
| Total Citable Articles (#) | 0.1471 (−0.1858 to 0.4497) | .3850 | 0.01355 (−0.3313 to 0.3552) | .9403 | 0.1362 (−0.2065 to 0.4490) | .4354 | 0.1327 (−0.2206 to 0.4553) | .4615 | −0.05785 (−0.4155 to 0.3153) | .7657 |
Abbreviations: IF, impact factor; r, Pearson correlation coefficient.
5‐year IF, immediacy index, and total citation and citable article numbers were also similarly important determinants of an otolaryngology journal's normalized Eigenfactor score (Figures 1C,D, 3D,E and Table 2). Thus, we set out to determine whether these bibliometrics also had predictive value for normalized Eigenfactor scores (Table S1). We found that previous (2009–2018) 5‐year IF significantly correlated with a high 2019 normalized Eigenfactor score across all years (Table 4). Interestingly, this was not the case with the immediacy index with the exception of immediacy index scores in 2015. Notably, both the past total number of citations and citable articles were significantly correlated with a high 2019 normalized Eigenfactor score across all years (Table 4). This suggests that a prolific journal publishing to accumulate total citations, but not necessarily highly cited individual articles, may be predictive of a future high normalized Eigenfactor score.
TABLE 4.
Correlation analysis between 2019 journal normalized Eigenfactor and bibliometrics at each indicated year between and including 2013–2018.
| Bibliometric | 2018 | 2017 | 2016 | |||
|---|---|---|---|---|---|---|
| r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | |
| 5‐year IF | 0.4592 (0.1680 to 0.6767) | .0033 | 0.4653 (0.1664 to 0.6859) | .0037 | 0.5182 (0.2423 to 0.7165) | .0007 |
| Immediacy Index | 0.1504 (−0.1733 to 0.4448) | .3608 | 0.3163 (−0.008574 to 0.5808) | .0565 | 0.3114 (−0.004534 to 0.5709) | .0536 |
| Articles in Citable Items (#) | 0.8580 (0.7438 to 0.9235) | <.0001 | 0.8672 (0.7554 to 0.9299) | <.0001 | 0.8744 (0.7719 to 0.9326) | <.0001 |
| Total Citations (#) | 0.9258 (0.8622 to 0.9607) | <.0001 | 0.9131 (0.8366 to 0.9547) | <.0001 | 0.9019 (0.8196 to 0.9477) | <.0001 |
| Bibliometric | 2015 | 2014 | 2013 | |||
|---|---|---|---|---|---|---|
| r (95%CI) | p‐Value | r (95%CI) | p‐Value | r (95%CI) | p‐Value | |
| 5‐year IF | 0.5157 (0.2391 to 0.7149) | .0008 | 0.5062 (0.2179 to 0.7132) | .0014 | 0.5271 (0.2401 to 0.7293) | .0010 |
| Immediacy Index | 0.3751 (0.06757 to 0.6175) | .0186 | 0.3110 (−0.005002 to 0.5705) | .0540 | 0.2925 (−0.03004 to 0.5598) | .0748 |
| Articles in Citable Items (#) | 0.8483 (0.7275 to 0.9181) | <.0001 | 0.8637 (0.7536 to 0.9267) | <.0001 | 0.8865 (0.7910 to 0.9399) | <.0001 |
| Total Citations (#) | 0.8845 (0.7892 to 0.9382) | <.0001 | 0.8674 (0.7598 to 0.9287) | <.0001 | 0.8483 (0.7230 to 0.9196) | <.0001 |
Abbreviations: IF, impact factor; r, Pearson correlation coefficient.
3.5. Multiple linear regression of past bibliometric scores in relationship to 2019 journal impact factor and normalized Eigenfactor scores
We were curious to determine which bibliometric among those predictive of journals with high IFs and normalized Eigenfactor scores were the most consequential in driving impact factor or Eigenfactor. We approached this question through multiple linear regression analysis of either 2019 journal IF or Eigenfactor score with 5‐year IF, immediacy index, and total number of citations and citable articles between 2009–2018 and 2013–2018, respectively (Table S1). We found that 5‐year IF remained significantly correlated with 2019 journal IF across all years. However, while 2019 journal IF was correlated with immediacy index in the recent past, 2019 journal IF was not correlated with immediacy index prior to 2016 (Table 5). Neither total citation counts, nor total citable articles significantly correlated with 2019 journal IF at any year interval examined (Table 5). 5‐year IF was significantly correlated with 2019 normalized Eigenfactor scores across all years. However, immediacy index was not found to be significantly correlated at any year examined (Table 6). Additionally, the total number of citations and citable articles were both significantly correlated with 2019 normalized Eigenfactor scores across all years (Table 6). However, the strength of the correlation of 5‐year IF and total number of citations to normalized Eigenfactor scores waned by 2016 and 2015, respectively, compared to that seen with total number of citable articles (Table 6). These findings suggest that 5‐year IF and to a lesser extent immediacy index (in the recent past) together contribute to predict a journal's IF. On the other hand, 5‐year IF alongside the total number of citations and citable articles, but not immediacy index, together contribute to predict a journal's normalized Eigenfactor score.
TABLE 5.
Multiple linear regression of 2019 journal impact factor to the indicated bibliometrics at each indicated year 2009–2018.
| Parameter estimates | Variable | Estimate | Standard error | 95% CI | |t| | p Value | Parameter estimates | Variable | Estimate | Standard error | 95% CI | |t| | p Value | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2018 | β0 | Intercept | −0.006617 | 0.167 | −0.3459 to 0.3327 | 0.03963 | .9686 | 2013 | β0 | Intercept | 0.5255 | 0.2075 | 0.1017 to 0.9494 | 2.532 | .0168 |
| β1 | 5‐year IF | 0.8368 | 0.07581 | 0.6828 to 0.9909 | 11.04 | <.0001 | β1 | 5‐year IF | 0.8959 | 0.1412 | 0.6076 to 1.184 | 6.346 | <.0001 | ||
| β2 | Immediacy Index | 0.4447 | 0.1592 | 0.1212 to 0.7681 | 2.794 | .0085 | β2 | Immediacy Index | −0.2723 | 0.3603 | −1.008 to 0.4636 | 0.7556 | .4558 | ||
| β3 | Total Citations | 0.000002751 | 0.00002432 | −4.668 e‐005 to 5.218 e‐005 | 0.1131 | .9106 | β3 | Total Citations | 0.00001314 | 0.00004514 | −7.904 e‐005 to 0.0001053 | 0.2912 | .7729 | ||
| β4 | Total Citable Articles | 0.0002009 | 0.0008819 | −0.001591 to 0.001993 | 0.2278 | .8212 | β4 | Total Citable Articles | −0.0008981 | 0.001088 | −0.003120 to 0.001323 | 0.8257 | .4155 | ||
| 2017 | β0 | Intercept | −0.04563 | 0.1369 | −0.3246 to 0.2333 | 0.3332 | .7411 | 2012 | β0 | Intercept | 0.8712 | 0.2752 | 0.3055 to 1.437 | 3.165 | .0039 |
| β1 | 5‐ year IF | 0.8551 | 0.07562 | 0.7010 to 1.009 | 11.31 | <.0001 | β1 | 5 year IF 2012 | 0.3466 | 0.1953 | −0.05494 to 0.7482 | 1.774 | .0877 | ||
| β2 | Immediacy Index | 0.4706 | 0.1797 | 0.1046 to 0.8365 | 2.619 | .0134 | β2 | Immediacy Index | 1.64 | 0.6462 | 0.3113 to 2.968 | 2.537 | .0175 | ||
| β3 | Total Citations | −0.00002332 | 0.00001986 | −6.378 e‐005 to 1.714 e‐005 | 1.174 | .2490 | β3 | Total Citations | 0.00004031 | 0.00005925 | −8.148 e‐005 to 0.0001621 | 0.6803 | .5023 | ||
| β4 | Total Citable Articles | 0.0008787 | 0.0006741 | −0.0004944 to 0.002252 | 1.303 | .2017 | β4 | Total Citable Articles | −0.001114 | 0.001524 | −0.004247 to 0.002019 | 0.7308 | .4714 | ||
| 2016 | β0 | Intercept | 0.4245 | 0.1827 | 0.05312 to 0.7958 | 2.323 | .0263 | 2011 | β0 | Intercept | 1.007 | 0.309 | 0.3694 to 1.645 | 3.259 | .0033 |
| β1 | 5‐ year IF | 0.7477 | 0.1186 | 0.5066 to 0.9888 | 6.302 | <.0001 | β1 | 5‐ year IF | 0.8818 | 0.2183 | 0.4313 to 1.332 | 4.04 | .0005 | ||
| β2 | Immediacy Index | 0.3676 | 0.2704 | −0.1820 to 0.9171 | 1.359 | .1830 | β2 | Immediacy Index | −1.42 | 0.977 | −3.436 to 0.5968 | 1.453 | .1592 | ||
| β3 | Total Citations | −0.00001302 | 0.0000308 | −7.563 e‐005 to 4.958 e‐005 | 0.4228 | .6751 | β3 | Total Citations | 0.00006567 | 0.00007524 | −8.963 e‐005 to 0.0002210 | 0.8727 | .3915 | ||
| β4 | Total Citable Articles | −0.00003933 | 0.0008229 | −0.001712 to 0.001633 | 0.04779 | .9622 | β4 | Total Citable Articles | −0.002834 | 0.002147 | −0.007265 to 0.001597 | 1.32 | .1992 | ||
| 2015 | β0 | Intercept | 0.3663 | 0.2019 | −0.04400 to 0.7766 | 1.814 | .0785 | 2010 | β0 | Intercept | 1.022 | 0.3334 | 0.3322 to 1.712 | 3.065 | .0055 |
| β1 | 5‐ year IF | 0.8985 | 0.1414 | 0.6111 to 1.186 | 6.353 | <.0001 | β1 | 5‐ year IF | 0.5566 | 0.3402 | −0.1471 to 1.260 | 1.636 | .1154 | ||
| β2 | Immediacy Index | 0.0383 | 0.3668 | −0.7072 to 0.7838 | 0.1044 | .9175 | β2 | Immediacy Index | 0.4288 | 1.352 | −2.368 to 3.226 | 0.3172 | .7540 | ||
| β3 | Total Citations | −0.00002556 | 0.00003651 | −9.975 e‐005 to 4.864 e‐005 | 0.7 | .4887 | β3 | Total Citations | 0.00003281 | 0.00007857 | −0.0001297 to 0.0001953 | 0.4176 | .6801 | ||
| β4 | Total Citable Articles | 0.0001023 | 0.0009687 | −0.001866 to 0.002071 | 0.1056 | .9165 | β4 | Total Citable Articles | −0.001639 | 0.001945 | −0.005662 to 0.002383 | 0.843 | .4079 | ||
| 2014 | β0 | Intercept | 0.6406 | 0.2458 | 0.1399 to 1.141 | 2.606 | .0138 | 2009 | β0 | Intercept | 1.447 | 0.3519 | 0.7124 to 2.181 | 4.11 | .0005 |
| β1 | 5‐ year IF | 0.7269 | 0.1743 | 0.3719 to 1.082 | 4.171 | .0002 | β1 | 5‐ year IF | 0.45 | 0.2115 | 0.008857 to 0.8911 | 2.128 | .0460 | ||
| β2 | Immediacy Index | 0.5412 | 0.4221 | −0.3186 to 1.401 | 1.282 | .2090 | β2 | Immediacy Index | 0.7996 | 0.8544 | −0.9827 to 2.582 | 0.9358 | .3605 | ||
| β3 | Total Citations | −0.00004288 | 0.00004781 | −0.0001403 to 5.450 e‐005 | 0.8969 | .3765 | β3 | Total Citations | 0.00007381 | 0.00006523 | −6.224 e‐005 to 0.0002099 | 1.132 | .2712 | ||
| β4 | Total Citable Articles | 0.0002782 | 0.001189 | −0.002144 to 0.002701 | 0.2339 | .8166 | β4 | Total Citable Articles | −0.003614 | 0.001813 | −0.007396 to 0.0001687 | 1.993 | .0601 |
Abbreviations: CI, Confidence Interval; IF, impact factor.
TABLE 6.
Multiple linear regression of 2019 journal normalized Eigenfactor to indicated bibliometrics at each indicated year 2009–2018.
| Parameter estimates | Variable | Estimate | Standard error | 95% CI | |t| | p Value | |
|---|---|---|---|---|---|---|---|
| 2018 | β0 | Intercept | −0.4204 | 0.09444 | −0.6123 to −0.2285 | 4.452 | <.0001 |
| β1 | 5‐ year IF2 | 0.1988 | 0.0447 | 0.1080 to 0.2897 | 4.447 | <.0001 | |
| β2 | Immediacy Index | 0.1103 | 0.09311 | −0.07895 to 0.2995 | 1.184 | .2445 | |
| β3 | Citable articles # | 0.002328 | 0.0005084 | 0.001295 to 0.003361 | 4.579 | <.0001 | |
| β4 | Total Citations | 0.0000809 | 0.0000143 | 5.184 e−005 to 0.0001100 | 5.658 | <.0001 | |
| 2017 | β0 | Intercept | −0.4337 | 0.08557 | −0.6080 to −0.2594 | 5.069 | <.0001 |
| β1 | 5‐ year IF | 0.2431 | 0.04836 | 0.1445 to 0.3416 | 5.026 | <.0001 | |
| β2 | Immediacy Index | −0.01409 | 0.1157 | −0.2497 to 0.2216 | 0.1218 | .9038 | |
| β3 | Citable articles # | 0.002728 | 0.0004258 | 0.001861 to 0.003596 | 6.408 | <.0001 | |
| β4 | total citations | 0.0000674 | 0.00001281 | 4.131 e−005 to 9.348 e−005 | 5.263 | <.0001 | |
| 2016 | β0 | Intercept | −0.2856 | 0.0832 | −0.4547 to −0.1166 | 3.433 | .0016 |
| β1 | 5‐ year IF | 0.2094 | 0.05475 | 0.09818 to 0.3207 | 3.826 | .0005 | |
| β2 | Immediacy Index | −0.01115 | 0.1237 | −0.2626 to 0.2403 | 0.09008 | .9287 | |
| β3 | Citable articles # | 0.0021 | 0.0003766 | 0.001335 to 0.002866 | 5.576 | <.0001 | |
| β4 | Total Citations | 0.00007243 | 0.0000142 | 4.358 e−005 to 0.0001013 | 5.101 | <.0001 | |
| 2015 | β0 | Intercept | −0.3661 | 0.09958 | −0.5685 to −0.1637 | 3.676 | .0008 |
| β1 | 5‐ year IF | 0.2666 | 0.07024 | 0.1239 to 0.4094 | 3.796 | .0006 | |
| β2 | Immediacy Index | −0.04442 | 0.1843 | −0.4189 to 0.3301 | 0.241 | .8110 | |
| β3 | Citable articles # | 0.002572 | 0.0004805 | 0.001596 to 0.003549 | 5.353 | <.0001 | |
| β4 | Total Citations | 0.00007021 | 0.00001819 | 3.323 e−005 to 0.0001072 | 3.859 | .0005 | |
| 2014 | β0 | Intercept | −0.2872 | 0.1092 | −0.5096 to −0.06482 | 2.631 | .0130 |
| β1 | 5‐ year IF | 0.2141 | 0.07291 | 0.06558 to 0.3626 | 2.936 | .0061 | |
| β2 | Immediacy Index | 0.06841 | 0.1667 | −0.2712 to 0.4081 | 0.4103 | .6843 | |
| β3 | Citable articles # | 0.002954 | 0.0005411 | 0.001852 to 0.004056 | 5.459 | <.0001 | |
| β4 | Total Citations | 0.00006451 | 0.00002191 | 1.987 e−005 to 0.0001091 | 2.944 | .0060 | |
| 2013 | β0 | Intercept | −0.3404 | 0.08651 | −0.5170 to −0.1637 | 3.935 | .0005 |
| β1 | 5‐ year IF | 0.2436 | 0.06883 | 0.1031 to 0.3842 | 3.54 | .0013 | |
| β2 | Immediacy Index | −0.0457 | 0.1954 | −0.4448 to 0.3534 | 0.2338 | .8167 | |
| β3 | Citable articles # | 0.002987 | 0.0004547 | 0.002058 to 0.003916 | 6.569 | .0001 | |
| β4 | Total Citations | 0.00006719 | 0.00001896 | 2.847 e−005 to 0.0001059 | 3.544 | .0013 |
3.6. Differences in otolaryngology journal ranking between journal impact factor and normalized Eigenfactor score
Our results have suggested the existence of relative similarity between the determinants of a journal's IF and its normalized Eigenfactor score. We next investigated whether otolaryngology journals with IFs that placed them in the top‐quartile similarly displayed higher normalized Eigenfactor scores compared to their counterparts. We found that otolaryngology journals in the top‐quartile had significantly higher normalized Eigenfactor scores compared to the bottom 75th‐percentile, with the exception of years 2017 and 2018 (Figure 4A). Notably, we observed substantial spread in the normalized Eigenfactor scores in the top‐quartile journals, such that ranking of otolaryngology journals by normalized Eigenfactor score would significantly change which journals would be considered top‐quartile compared to ranking by IF. This was particularly true with journals ranked within the top half of all otolaryngology journals and was a pattern that persisted between 2015 and 2020 (Figure 4B‐G). This suggests that while journal IF and normalized Eigenfactor scores share bibliometric elements that significantly correlate with respectively higher scores, they are sufficiently different such that evaluating journal influence and prestige using the normalized Eigenfactor score as opposed to the conventional journal IF would significantly alter otolaryngology journal rankings.
FIGURE 4.
Plotted normalized Eigenfactor score (nEigenfactor) (A) of otolaryngology journals defined as either top (blue open circles) or bottom (black open circles) quartile at each year interval between 2015 and 2020 based on IF. Plotted journal IF (black, connected circles) and normalized Eigenfactor scores (blue, connected squares) (B–G) according to journal rank determined by IF score at each year interval between 2015 and 2020. *p < .05, ns—p > .05.
4. DISCUSSION
The IF remains a metric of great significance to readers, authors, and journals alike. Therefore, understanding the determinants of conventional journal IFs in addition to alternative journal metrics like the Eigenfactor score may both inform their interpretation and guide their improvement. In our study, we set out to identify journal characteristics associated with high‐IF and Eigenfactor otolaryngology journals (which we defined as the top quartile of all journals within the field). Specifically, we evaluated the 5‐year IF, immediacy index, IF excluding self‐citations, the proportion of citable items, and the total number of citations and citable articles as predictors of current and future IF and normalized Eigenfactor score. We discovered that high IF and normalized Eigenfactor score otolaryngology journals were more likely to publish articles of immediate and sustained citation‐worthy interest within the research community, as reflected by higher immediacy indexes and 5‐year IFs. Moreover, otolaryngology journals with high normalized Eigenfactor scores were more likely to contain a higher number of overall citations and citable articles. No means of artificially inflating IF, such as higher journal self‐citations or decreased number of citable articles was significantly associated with higher otolaryngology journal IFs. A high 5‐year IF and to a lesser extent immediacy index were strong predictors of future high journal IF. Conversely, the total number of citable articles and to lesser extents 5‐year IF and total citation numbers were strong predictors of future high normalized Eigenfactor scores. Finally, ranking otolaryngology journals by their normalized Eigenfactor score substantially alters journal positions and thus perceived prestige and influence compared to ranking journals by IF.
Studies evaluating determinants of IF have been performed in a number of fields including plastic surgery, urology, nephrology, psychiatric and general medicine. These studies have identified citation skew, 10 , 29 journal self‐citations, 30 , 33 and fraction of citable items vs non‐citable items published 33 as disproportionately affecting IF in multiple medical specialties. In these cases, the IF may no longer reflect organic citations of a journal's articles and instead reflect a product of strategies to artificially inflate it. As a reflection of this possibility an inverse correlation between IF and clinically relevant data have even been observed in some fields’ scientific literature. 31 Given the many possible caveats to interpretation of IF by readers and authors as well as the different possible avenues for increasing IF for editors and publishers, an understanding of what factors are most responsible for driving the IF in their respective fields is an important necessity. To date, however, no study has comprehensively evaluated determinants of IF (or other alternative bibliometrics of impact) in the otolaryngology scientific literature.
When examined in isolation, IF can be a misleading or even an invalid surrogate when evaluating a journal's merits, the quality of an author's work, or the clinical relevance of a study. 7 , 18 , 19 , 24 , 37 However, when placed in the context of other relevant bibliometrics it can more accurately reflect those desired characteristics. 32 In this study, we found that bibliometrics measuring published articles with immediate applicability or sustained interest (i.e., immediacy index and 5‐year IF, respectively) positively correlated with otolaryngology journal IFs. Moreover, bibliometrics measuring contribution of self‐citations or proportion of citable items, which can be manipulated to artificially drive up IF, displayed no correlation or in some instances negative correlation with otolaryngology journal IFs. These findings suggest that IF of otolaryngology journals is more likely to be reflective of the clinical relevance of published articles as well as the quality of the published work. These findings are also encouraging in that within the otolaryngology literature, the bulk of journals’ impact was not inflated by journal self‐citations or variations in proportion of citable items as has been observed in other fields. 10 , 30 , 33 Our study has important implications for how journals can improve their IFs, and how readers and authors should interpret IFs within the otolaryngology scientific literature. We believe that our results reinforce the notion that the path to higher IF for otolaryngology journals lies in the sustained publication of high‐quality and clinically informative literature. 38 We also believe that these results inform and reassure the readership of the otolaryngology scientific literature that IF in our field predominantly reflects publication of timely and influential articles. This finding is all the more reassuring as our previous study has suggested a relative parity in quality publication across otolaryngology journals. 38
The Eigenfactor score has been proposed as an alternative metric to the conventional journal IF that more comprehensively evaluates a journal's influence within a field. It achieves this through an iterative analysis of citation networks that allow for determining the value of any one citation based on the measured influence of the journal it came from. 14 While there is some debate on whether the Eigenfactor succeeds in more comprehensively evaluating journal influence compared to the conventional journal IF, 35 , 39 , 40 , 41 its iterative calculation algorithm, exclusion of self‐citations, and 5‐year evaluation period could certainly provide a unique evaluation of otolaryngology journal influence. In line with this notion, we found that while top‐quartile otolaryngology journals determined by IF and Eigenfactor had higher 5‐year IF scores compared to their counterparts, this was not generally true when evaluating immediacy index in journals ranked by normalized Eigenfactor scores. Moreover, journals displayed no significant correlation between normalized Eigenfactor and immediacy index. While 5‐year IF was significantly correlated with normalized Eigenfactor, this relationship was not as strong as the correlation seen between normalized Eigenfactor scores and total number of citations or total number of citable articles. The unique differences between the conventional journal IF and Eigenfactor were best observed when comparing changes in otolaryngology journal ranking. Indeed, some otolaryngology journals that are ranked below the top‐quartile by IF would find themselves comfortably as one of the top five journals according to their Eigenfactor score. Our findings underscore a potentially significant change in perceived otolaryngology journal prestige and influence should other metrics like Eigenfactor become incorporated into our regular journal evaluations. This finding also highlights the importance of understanding what bibliometrics of journal influence or impact truly represent for readers, reviewers, editors, and publishers who must interpret them.
A high influence metric, whether that is measured by the conventional journal IF or the Eigenfactor, is undoubtedly one objective of otolaryngology journal editors. Identifying the determinants that drive higher influence metrics may help guide editors in effectively improving their journal metrics and inform the readers' interpretation of those metrics. Our study set out to determine which bibliometrics were most influential in driving high journal IF or Eigenfactor scores. We found that a journal's 5‐year IF from 10 years prior remained a strong predictor of its IF in 2019. However, the predictive strength of a journal's immediacy index was limited to the prior two‐years when predicting a journal's IF in 2019. When predicting a journal's future Eigenfactor score, the 5‐year IF and total number of citations and citable articles from 6 years prior were all strongly predictive of 2019 Eigenfactor score. However, the number of citable articles maintained the strongest predictive strength of 2019 Eigenfactor score while the predictive strength of 5‐year IF and total citation numbers waned comparatively beginning in 2016 and 2015, respectively. Our study suggests that to improve journal IF, journals should place emphasis on publishing articles deemed to have long‐term citation‐worthy findings. On the other hand, to improve Eigenfactor scores, journals should place emphasis on increasing the number of articles they publish, particularly if they have the potential to maintain citation‐worthy interest. The unique differences in calculating IF and Eigenfactor as well as in the bibliometrics that predict higher respective scores uncovered by our study imply that they reflect different traits of a premier journal. For instance, a journal that decides to focus on publishing articles that will garner broad attention without regard to the influence of that attention will have that reflected by a high IF. On the other hand, a journal that focuses on publishing articles that will garner attention from more influential groups or attract attention from other disciplines within and outside the sciences will have that reflected by a high Eigenfactor. In other words, we suspect that interpreting both the IF and Eigenfactor simultaneously will provide readers, authors, and editors a better understanding of the influence of a particular journal. Moreover, impactful journals do not necessarily have both high IF and high Eigenfactor score but instead could have high IF but low Eigenfactor and vice versa.
Our results should not be interpreted beyond the limitations of our study. One notable limitation is the cross‐sectional nature of our analysis. We independently evaluated 6–10 1‐year intervals for each bibliometric studied and conducted comparisons and correlations within each year. While this limits the generalizability of our findings to the years studied and the interyear comparisons among those studied, we believe this timeframe is adequate to confidently identify strong correlations between the studied bibliometrics and top quartile otolaryngology journals. Second, our dichotomous categorization of journals into top‐quartile or bottom 75th percentile by IF or normalized Eigenfactor may have also reduced our ability to identify nuanced differences between higher‐ and lower‐IF or ‐normalized Eigenfactor journals. Thirdly, while our study underscores the identified bibliometrics as strongly influencing the IF or Eigenfactor, there may be other metrics not investigated in this study that may also play a role in determining high IF or Eigenfactor scores. Finally, our multilinear regression analysis of historical bibliometric data was limited to the availability of normalized Eigenfactor scores in JCR, as these scores were not available between the years 2009–2012. Nevertheless, we believe that our results at a minimum qualitatively capture the dominant factors driving high‐IF and ‐Eigenfactor among otolaryngology journals.
CONFLICT OF INTEREST STATEMENT
The authors declare that there are no conflicts of interests regarding the publication of this article.
Supporting information
Supplemental Table S1:
Ali A, Phillips KM, Sedaghat AR. Determinants of impact factor and Eigenfactor score in otolaryngology journals. Laryngoscope Investigative Otolaryngology. 2023;8(2):380‐393. doi: 10.1002/lio2.1018
REFERENCES
- 1. Garfield E, Sher IH. New factors in the evaluation of scientific literature through citation indexing. Am Doc. 1963;14:195‐201. [Google Scholar]
- 2. Garfield E. Fortnightly review: how can impact factors be improved? BMJ. 1996;313:411‐413. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295:90‐93. [DOI] [PubMed] [Google Scholar]
- 4. Kurmis AP. Understanding the limitations of the journal impact factor. J Bone Joint Surg Am. 2003;85:2449‐2454. [DOI] [PubMed] [Google Scholar]
- 5. Bornmann L, Marx W, Gasparyan AY, Kitas GD. Diversity, value and limitations of the journal impact factor and alternative metrics. Rheumatol Int. 2012;32:1861‐1867. [DOI] [PubMed] [Google Scholar]
- 6. Hecht F, Hecht BK, Sandberg AA. The journal "impact factor": a misnamed, misleading, misused measure. Cancer Genet Cytogenet. 1998;104:77‐81. [DOI] [PubMed] [Google Scholar]
- 7. Saha S, Saint S, Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc. 2003;91:42‐46. [PMC free article] [PubMed] [Google Scholar]
- 8. Mech E, Ahmed MM, Tamale E, Holek M, Li G, Thabane L. Evaluating journal impact factor: a systematic survey of the pros and cons, and overview of alternative measures. J Venom Anim Toxins Incl Trop Dis. 2020;26:e20190082. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Kiesslich T, Beyreis M, Zimmermann G, Traweger A. Citation inequality and the journal impact factor: median, mean, (does it) matter? Scientometrics. 2021;126:1249‐1269. [Google Scholar]
- 10. Asaad M, Kallarackal AP, Meaike J, Rajesh A, de Azevedo RU, Tran NV. Citation skew in plastic surgery journals: does the journal impact factor predict individual article citation rate? Aesthet Surg J. 2020;40:1136‐1142. [DOI] [PubMed] [Google Scholar]
- 11. Taşkın Z, Doğan G, Kulczycki E, Zuccala AA. Self‐citation patterns of journals indexed in the journal citation reports. J Informet. 2021;15:101221. [Google Scholar]
- 12. Ioannidis JPA, Baas J, Klavans R, Boyack KW. A standardized citation metrics author database annotated for scientific field. PLoS Biol. 2019;17:e3000384. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Seeber M, Cattaneo M, Meoli M, Malighetti P. Self‐citations as strategic response to the use of metrics for career decisions. Res Policy. 2019;48:478‐491. [Google Scholar]
- 14. Bergstrom CT, West JD, Wiseman MA. The Eigenfactor metrics. J Neurosci. 2008;28:11433‐11434. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Adler R, Ewing J, Taylor P. Citation statistics: a report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Stat Sci. 2009;24:1‐14. [Google Scholar]
- 16. Seglen PO. From bad to worse: evaluation by journal impact. Trends Biochem Sci. 1989;14:326‐327. [DOI] [PubMed] [Google Scholar]
- 17. Opthof T, Wilde A. The Hirsch‐index: a simple, new tool for the assessment of scientific output of individual scientists. Neth Hear J. 2009;17:145‐154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Donato H. Understanding journal evaluation and strategies to increase impact. Pulmonology. 2016;22:67‐69. [DOI] [PubMed] [Google Scholar]
- 19. Dempsey JA. Impact factor and its role in academic promotion: a statement adopted by the international respiratory journal editors roundtable. J Appl Physiol. 2009;107:1005. [DOI] [PubMed] [Google Scholar]
- 20. Kaldas M, Michael S, Hanna J, Yousef GM. Journal impact factor: a bumpy ride in an open space. J Investig Med. 2020;68:83‐87. [DOI] [PubMed] [Google Scholar]
- 21. Argüelles JC, Argüelles‐Prieto R. Are the editors responsible for our obsession with the impact factor? MBio. 2017;8(6):e02019‐17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Uzun C. Increasing the impact factor in the ethical way. Balkan Med J. 2017;34:482‐484. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Helgesson G, Radun I, Radun J, Nilsonne G. Editors publishing in their own journals: a systematic review of prevalence and a discussion of normative aspects. Learn Publ. 2022;35:229‐240. [Google Scholar]
- 24. Marcovitch H. Editors, publishers, impact factors, and reprint income. PLoS Med. 2010;7:e1000355. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. KRELL F‐T. Should editors influence journal impact factors? Learn Publ. 2010;23:59‐62. [Google Scholar]
- 26. Stewart MG. Classics from the laryngoscope: celebrating 120 years of impact. Laryngoscope. 2015;125:2237. [DOI] [PubMed] [Google Scholar]
- 27. Piccirillo JF. JAMA Otolaryngology–Head & Neck Surgery—The Year in Review, 2021. JAMA Otolaryngol Head Neck Surg. 2022;148:399‐401.35323890 [Google Scholar]
- 28. Liskiewicz T, Liskiewicz G, Paczesny J. Factors affecting the citations of papers in tribology journals. Scientometrics. 2021;126:3321‐3336. [Google Scholar]
- 29. Hegarty P, Walton Z. The consequences of predicting scientific impact in psychology using journal impact factors. Perspect Psychol Sci. 2012;7:72‐78. [DOI] [PubMed] [Google Scholar]
- 30. Kulkarni AV, Aziz B, Shams I, Busse JW. Author self‐citation in the general medicine literature. PLoS One. 2011;6:e20885. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Lauritsen J, Moller AM. Publications in anesthesia journals: quality and clinical relevance. Anesth Analg. 2004;99:1486‐1491. [DOI] [PubMed] [Google Scholar]
- 32. Lokker C, Haynes RB, Chu R, McKibbon KA, Wilczynski NL, Walter SD. How well are journal and clinical article characteristics associated with the journal impact factor? A retrospective cohort study. J Med Libr Assoc. 2012;100:28‐33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Sewell JM, Adejoro OO, Fleck JR, Wolfson JA, Konety BR. Factors associated with the journal impact factor (JIF) for urology and nephrology journals. Int Braz J Urol. 2015;41:1058‐1066. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Motamed M, Mehta D, Basavaraj S, Fuad F. Self citations and impact factors in otolaryngology journals. Clin Otolaryngol Allied Sci. 2002;27:318‐320. [DOI] [PubMed] [Google Scholar]
- 35. West JD. Eigenfactor: Ranking and mapping scientific knowledge, 2010.
- 36. McVeigh ME, Mann SJ. The journal impact factor denominator: defining citable (counted) items. JAMA. 2009;302:1107‐1109. [DOI] [PubMed] [Google Scholar]
- 37. Didegah F, Thelwall M. Which factors help authors produce the highest impact research? Collaboration, journal and document properties. J Informet. 2013;7:861‐873. [Google Scholar]
- 38. Sedaghat AR. Distribution of article citation frequency, citation skew, and impact factor in otolaryngology journals. Otolaryngol Head Neck Surg. 2022; doi:10.117/1945998221088751. [DOI] [PubMed] [Google Scholar]
- 39. Davis PM. Eigenfactor: does the principle of repeated improvement result in better estimates than raw citation counts? J Am Soc Inf Sci Technol. 2008;59:2186‐2188. [Google Scholar]
- 40. Fersht A. The most influential journals: impact factor and Eigenfactor. Proc Natl Acad Sci U S A. 2009;106:6883‐6884. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Kianifar H, Sadeghi R, Zarifmahmoudi L. Comparison between impact factor, Eigenfactor metrics, and SCimago journal rank indicator of pediatric neurology journals. Acta Inform Med. 2014;22:103‐106. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental Table S1: