Evaluating the dependability of reference-driven citation forecasts amid the COVID-19 pandemic: A bibliometric analysis across diverse journals

Abstract

Background:

The journal impact factor significantly influences research publishing and funding decisions. With the surge in research due to COVID-19, this study investigates whether references remain reliable citation predictors during this period.

Methods:

Four multidisciplinary journals (PLoS One, Medicine [Baltimore], J. Formos. Med. Assoc., and Eur. J. Med. Res.) were analyzed using the Web of Science database for 2020 to 2022 publications. The study employed descriptive, predictive, and diagnostic analytics, with tools such as 4-quadrant radar plots, univariate regressions, and country-based collaborative maps via the follower-leading cluster algorithm.

Results:

Six countries dominated the top 20 affiliations: China, Japan, South Korea, Taiwan, Germany, and Brazil. References remained strong citation indicators during the COVID-19 period, except for Eur. J. Med. Res. due to its smaller sample size (n = 492) than other counterparts (i.e., 41,181, 12,793, and 1464). Three journals showed higher network density coefficients, suggesting a potential foundation for reference-based citation predictions.

Conclusion:

Despite variations among journals, references effectively predict article citations during the COVID-19 era, underlining the importance of network density. Future studies should delve deeper into the correlation between network density and citation prediction.

Key Points.

• Investigates network density’s role in predicting article citations.

• This study addresses variations in citation prediction across journals and confirms that references can predict article citations in the COVID-19 epidemic.

• This paper proposes avenues to report network density coefficients for future research in bibliometrics.

1. Introduction

The unprecedented outbreak of the COVID-19 pandemic precipitated a notable surge in academic investigations. With SARS-CoV-2 wreaking havoc across the globe in the early 2020s, there was an immediate and drastic shift in public health dynamics and socioeconomic frameworks.^[1,2] A consequential ripple effect was observed in the academic domain, with a deluge of research publications emerging merely 3 months after the onset of the outbreak.^[2]

1.1. Acceleration in journal dynamics

The magnitude of COVID-19-centric research has significantly reshaped journal operations. Notably, PubMed indexed an astounding 320,419 articles pertinent to COVID-19 since its emergence in 2020, forming a substantial fraction of its overall publications.^[3] This burgeoning influx had a cascading effect on peer review timelines. Astonishingly, the median duration from manuscript submission to its formal acceptance for COVID-19-related research plummeted from 84 days to a mere 6 days in the initial pandemic months.^[4] The majority of this decrease can be attributed to an acceleration of the review process, as noted by Orbach.^[5]

1.2. Literature review

In 2020, there were more than 100,000 articles concerning the coronavirus pandemic, representing approximately 6% of health and medical papers,^[6] of which those focused on COVID-19 accounted for a significant proportion.^[7] As of the latest query period from the AI technology firm Primer,^[8] 510 papers and preprints related to COVID-19 have emerged in PubMed since December 26, 2022, comprising 5.47% of the 9323 articles documented since October 4, 2022.

There was an observed expedited peer review process for articles addressing COVID-19. Preprints on COVID-19 from Metrix (https://www.medrxiv.org/) transitioned to peer-reviewed journals with a median duration of 72 days, which is notably faster than the review times for other topics.^[7] The agility demonstrated by editors and publishers in fast-tracking their peer-review protocols is commendable, as is the ability of researchers to undertake a higher volume of reviews.

A review of 11 medical journals in the initial half of 2020 reveals an accelerated publication rate for coronavirus-related papers. This increase, however, was counterbalanced by a relative deceleration in the publication of research from other domains.^[9] Notably, during the early months of 2020, Palayew et al^[10] reported a significant reduction in the median submission-to-acceptance time for COVID-19 research, from 84 days to a mere 6 days5. A substantial portion of this reduction can be ascribed to the expedited review process, as highlighted by Horbach.^[11]

1.3. Impact on journal impact factors (JIFs)

The journal impact factor (JIF), an esteemed metric in contemporary research evaluation, has not remained immune to the pandemic’s reverberations.^[12] After the unveiling of the 2021 Impact Factor, premier medical journals witnessed their JIFs skyrocketing, with illustrious names such as the Lancet, NEJM, and the Journal of the American Medical Association clinching the top spots.^[13] This escalation in JIFs, particularly in domains pivotal to COVID-19 management, underscores the pandemic’s overarching influence on scholarly publishing.

1.4. References as citation predictors

An enduring premise in bibliometric studies is the potential of references to predict article citations. Historical data suggest a correlation between expansive citation lists and certain article categories, such as reviews, with elevated impact factors.^[12,13] However, in the unique academic environment sculpted by COVID-19, the present study endeavors to discern whether references retain their predictive prowess for article citations during 2020 to 2022. Our focus is sharpened on 4 eminent multidisciplinary journals. A foundational hypothesis steering this exploration postulates the indicative potential of references vis-à-vis article citations.^[13] The estimated optimal number of references in a sociology article is also approximately 66.^[7]

1.5. Study aims

To enrich the contextual depth of our study, the ensuing segments will encompass an exhaustive literature review, elucidating the historical backdrop and intricate nuances that molded the extant research landscape. A central hypothesis underpinning this study posits that references can indeed be indicative of article citations.

2. Methods

2.1. Data sources

To fulfill the purpose of our study, we selected the 4 prestigious multidisciplinary journals mentioned in Section 1.4. A comprehensive search of the literature using the Web of Science core collection (WoSCC) database was conducted for manuscripts published between 2020 and 2022. Citation counts and reference numbers for each manuscript were retrieved as well.

The search of the literature was performed on a single day to reduce daily updates of the database. Manuscripts were restricted to peer-reviewed original research and review articles. No further exclusion criteria were applied to our search. A total of 55,930 articles were downloaded and analyzed, comprising 41,181, 12,793, 1464, and 492 articles in PLoS One (PLoS), Medicine, J Formos Med Assoc (Formos), and Eur J Med Res (EJMR), respectively; see Data S1, Supplemental Digital Content, http://links.lww.com/MD/L268.

As this study did not involve the examination or treatment of patients or review of patient records, it was exempt from review and approval by our research ethics committee.

2.2. Three major parts involved in this study

2.2.1. Descriptive analytics.

In this section, 4-quadrant radar plots^[14] with the CJAL score^[14] (combined with CAL score^[15] and L-index^[16]) were applied to evaluate the research achievements (RAs) of entities, including counties and institutes in the study journals of PLoS, Medicine, Formos, and EJMR. Bubbles were sized by the CJAL score. Y-indexes^[17] were used to display bubble coordinates in 4-quadrant radar plots.^[14,18,19]

The absolute advantage coefficient (AAC)^[20] is determined by the 3 consecutive numbers of values (e.g., CJAL scores in descending order denoted by A1, A2, and A3 in Eqs. 1–3).^[21]

$${\displaystyle \begin{array}{l}{\displaystyle \mathrm{A}\mathrm{A}\mathrm{C}=\left({\mathrm{R}}_{12}/{\mathrm{R}}_{23}\right)/(1+\left({\mathrm{R}}_{12}/{\mathrm{R}}_{23}\right)),}\end{array}}$$ (1)

$${\displaystyle \begin{array}{l}{\displaystyle {\mathrm{R}}_{12}=\mathrm{A}1/\mathrm{A}2,}\end{array}}$$ (2)

$${\displaystyle \begin{array}{l}{\displaystyle {\mathrm{R}}_{23}=\mathrm{A}2/\mathrm{A}3,}\end{array}}$$ (3)

The AAC ranged from 0 to 1.0, representing the strength of dominance for the top 1 when compared to the next 2 CJAL scores. Through the computation of AAC, the dominance strength in RA denoted by the CJAL score can be measured by the effect size, with criteria of <0.5, between 0.5 and 0.7, and not less than 0.7 as the small, medium, and large effect sizes, respectively.^[21]

Analyzing the 4-quadratn radar plots^[14] allowed us to compare publications across affiliated countries and institutes during the COVID-19 period.

2.2.2. Predictive analytics.

To predict article citations using the reference count, we conducted single univariate regressions for each journal with a Type I error threshold of 0.05. Analyzing the regressions allowed us to determine whether references can serve as predictors of article citations. Next, a prediction of citation counts will be made for each journal when cited reference counts are set at 66.^[7]

2.2.3. Diagnostic analytics.

We utilized country-based collaborative maps and cluster analysis with the follower-leading cluster algorithm^[22–24] to discern network pattern variations among journals. To assess the network pattern, we employed the density coefficient (DC)^[24] in a range between 0 and 1.0 as per Equation 4:

$${\displaystyle \begin{array}{l}{\displaystyle \mathrm{D}\mathrm{C}=connectionsinlinks÷(n-1)÷thenumberofclusters}\end{array}}$$ (4)

Here, n represents the network’s members (e.g., 20 in this study). A higher DC signifies greater author collaboration within the network.

Analyzing the DCs allowed us to scrutinize disparities in patterns of country-based author collaborations across journals, differentiating those denser networks with associations to cited references from those without.

2.3. Creating dashboards on Google Maps

All graphs were drawn by author-made modules in Excel (Microsoft Corp). We created HTML pages used for Google Maps when the opacity was set at zero. The method of how to draw those visualizations is deposited with a PDF file in Data S2, Supplemental Digital Content, http://links.lww.com/MD/L269. The statistical software MedCalc^[25] was used to conduct regression analysis. The JIFs for each journal are shown in Figure 1.

Figure 1.

The JIF trend for each studied journal.

3. Results

3.1. Descriptive analytics

In four-quadrant radar plots (Figs. 2 and 3), we observe that (1) China dominates the 2 journals of Medicine and EJMR with large effects (over 0.70 based on AACs of 0.71 and 0.71), while Taiwan dominates the journal of Formos. Med. Assoc., as shown in Figure 2, with a large effect (>0.70 based on AAC = 0.81); US has a medium effect with AAC = 0.58 between 0.5 and 0.7; and (2) no single research institute has a significant advantage over the next 2 because AAC values are too small to have a significant effect (>0.70[22]), as shown in Figure 3.

Figure 2.

The leading countries/regions that dominate publications for journals between 2000 and 2022 using the radar plot (n = 55,438, based on 1st and corresponding authors only).

Figure 3.

The leading research institutes’ dominant journals between 2000 and 2022 using the radar plot (n = 55,438, based on 1st and corresponding authors only).

3.2. Predictive analytics

The hypothesis that references remain effective indicators of article citations during the COVID-19 period is substantiated, with the exception of the EJMR journal, which does not exhibit statistical significance in regression analysis due to her smaller sample size (n = 492) than other counterparts (i.e., 41,181, 12,793, and 1464), as shown in panel D of Figure 4.

Figure 4.

References can serve as an indicator for predicting article citations in 3 journals.

The predicted citation counts are 3.76, 6.56, 3.18, and 5.10 for EJMR, Formos, Medicine, and PLoS, respectively, when the cited reference counts are set at 66, based on their respective regression equations in Figure 5.

Figure 5.

The estimated citation counts for each journal when the cited reference is 66.

3.3. Diagnostic analytics

Figures 6 to 9 reveal that the journals Medicine, PLoS, and Formos exhibit more substantial DCs in collaborative maps at 0.11, 0.50, and 0.25, respectively, while EJMR trails behind at 0.02. This implies that reference-based article citation prediction may hinge on elevated network DCs, warranting deeper investigation in future studies.

Figure 6.

Top 20 countries shown on a collaborative map for medicine (Baltimore).

Among the top 20 affiliated countries across the 4 journals under study, 6 countries consistently emerge: China, Japan, South Korea, Taiwan, Germany, and Brazil, as shown in Figure 10.

Figure 10.

Venn diagrams reveal that 6 countries consistently appear among the top 20 affiliated countries across these 4 journals.

Figure 7.

Top 20 countries shown on a collaborative map for PLoS One.

Figure 8.

Top 20 countries shown on a collaborative map for J. Formos. Med. Assoc.

Figure 9.

Top 20 countries shown on a collaborative map for the euro. J. Med. Res.

3.4. Online dashboards shown on Google Maps

All the QR codes in Figures 2 and 3 are linked to the dashboards. Readers are suggested to examine the displayed dashboards on Google Maps when the opacity is set at zero.

4. Discussion

4.1. Principal findings

While the hypothesis that references serve as effective indicators of article citations during the COVID-19 period is confirmed, the EJMR journal does not exhibit statistical significance in regression analysis. Notably, Medicine, PLoS, and Formos journals display higher DCs of 0.11, 0.50, and 0.25, respectively, in collaborative maps, while EJMR lags behind with a coefficient of 0.02. This suggests that reference-based article citation prediction may hinge on higher network DCs, warranting further exploration in future research. Among the top 20 affiliated countries across the 4 journals studied, China, Japan, South Korea, Taiwan, Germany, and Brazil consistently emerge.

Accordingly, the hypothesis that references can be indicative of article citations is confirmed.

4.2. Impact of expedited peer reviews on COVID-19 research publications

The surge in COVID-19 publications was significantly influenced by the swift editorial peer-review processes implemented for manuscripts, leading to unprecedented processing and acceptance rates for selected articles.^[2,26,27] For instance, during the initial months of 2020, the time from submission to acceptance for COVID-19 research shrunk dramatically from 84 days to 6 days.^[4] Such reductions, as highlighted by Horbach,^[11] can largely be attributed to the quickened review process.

The scientific community’s robust response to the global health crisis is evident in the soaring number of COVID-19 publications.^[2] With the race for a solution, there has been a rush to publish promising findings.^[28] However, this rush raises questions about the maintenance of scientific standards by both researchers and journals. While many journals initiated rapid peer-review processes to hasten the release of COVID-19 findings, our 4 focus journals (PLoS, Medicine, Formos, and EJMR) did not show this trend in 2020, having fewer publications than in 2021, as per the supplemental data.

Prominent medical journals such as Nature and JAMA have made deliberate efforts to prioritize and expedite COVID-19 publications, often at the expense of non-COVID-19 content.^[29,30] Lancet and Nature, for instance, nearly halved their review durations for COVID-19 articles during the pandemic in comparison to other topics.^[31–33] A significant uptick in citation counts was observed across key journals, including Annals, The BMJ, JAMA, The Lancet, NatMed, and NEJM, when considering their COVID-19 publications.^[2]

Furthermore, the quality of peer-reviewed COVID-19 papers in top-tier medical journals, notably The Lancet, NEJM, and JAMA, came under scrutiny due to high rates of retractions and concerns.^[34,35] The race to alleviate submission backlogs might have influenced these patterns.^[2] For instance, The Lancet, which led with an Impact Factor of 203,^[6] has since acknowledged the potential pitfalls of hasty review processes and emphasized the importance of more deliberative publication practices.^[36] The sustainability of the elevated impact factors driven by COVID-19 publications remains uncertain and warrants observation in upcoming years, especially concerning journals such as The Lancet and EJMR.

4.3. Implications and changes

Potential future shifts in bibliographical studies include:

1. Using the CJAL score^[18] and AAC,^[24,25] an author’s dominance over peers can be assessed. Annual rankings by Scopus and WoS (e.g., the top 2% of authors with the most publications in each discipline^[37,38]) may benefit from the AAC as an indicator to gauge research output dominance.

2. Despite COVID-19’s challenges, cited references continue to predict article citations. Exploring this citation–reference relationship further will benefit bibliometric analyses.

3. Our study introduces a collaborative map that merges counts and connections, enhancing the typical single-data representation in bibliometric networks. This provides a more comprehensive view, as seen in our collaboration analyses involving countries such as China, the US, Taiwan, and Germany.

4.4. Limitations and suggestions

While our study offers significant insights into the citation patterns during the COVID-19 pandemic, several limitations and opportunities for future research should be noted.

1. Scope of journals: Our analysis was restricted to PLoS, Medicine, Formos, and EJMR. Thus, generalizing the findings, especially the observed surge in mean-citation changes for COVID-related articles between 2020 and 2022, to other journals remains a challenge.

2. Accessibility of dashboards: The visualizations utilize Google Maps, which is not freely accessible due to its API constraints. As such, the dashboard’s broader accessibility is limited.

3. RA score assumptions: Our calculation of the RA score presumes equal contributions from the first and corresponding authors. Alterations in author positions, especially if they are not the first or corresponding authors, could impact both the CJAL scores and AAC.

4. Computational constraints: Computing the CJAL score is resource intensive. There is a pressing need for specialized software to facilitate this in future studies. Moreover, given its reliance on JIF and journal rankings from WoS’s Journal Citation Reports, the CJAL score’s applicability is limited to WoS-indexed articles.

5. Refinement of collaborative maps: While R-based collaborative maps are standard, the DC used in our study could benefit from further refinement. Future endeavors should consider comprehensive pair cooccurrences and delve deeper into the intricacies of the DC.

5. Conclusion

Our study underscores the persistent efficacy of references as predictors of article citations during the tumultuous COVID-19 era. However, nuances exist across different journals, highlighting the pivotal role of network density. Future bibliometric studies should not only delve deeper into the relation between heightened network DCs and reference-driven article citations but also integrate these insights to enrich the features of bibliographic research. This exploration, coupled with addressing the aforementioned limitations, can propel the field forward, offering more granular and expansive insights.

Acknowledgments

We thank the Coding Service for the English language review of this manuscript.

Author contributions

Conceptualization: Sam Yu-Chieh Ho.

Data curation: Julie Chi Chow.

Investigation: Willy Chou.