Skip to main content
PMC home page
PLOS ONE logoLink to PLOS ONE
. 2022 Jul 27;17(7):e0271071. doi: 10.1371/journal.pone.0271071

Comparison of citation rates between Covid-19 and non-Covid-19 articles across 24 major scientific journals

Michael D Brandt 1,*, Sherief A Ghozy 2, David F Kallmes 2, Robert J McDonald 2,#, Ramanathan D Kadirvel 2,#
Editor: Ayman Elbehiry3
PMCID: PMC9328554  PMID: 35895698

Abstract

Covid-19 has been front and center in the global landscape since the beginning of 2020. In response, the scientific field has dedicated enormous amounts of resources to researching the virus and its effects. The number of times Covid-19 publications are being cited throughout the literature appears remarkably high but has not been directly compared to non-Covid-19 papers in the same journals over an extended period. In our study, we use Clarivate’s Web of Science—Science Citation Index Expanded™ database to identify Covid-19 papers published in 24 major scientific journals over a period of 24 months from January 1, 2020 to December 31, 2021. We conduct our search using keywords “Covid-19”, “coronavirus”, and “sars-cov-2” to locate publications with these words in the title. We then quantify the number of citations these papers have received and compare rates to non-Covid-19 papers in the same journals over the same timeframe. We find that, across 24 open-access and subscription-based scientific journals, Covid-19 papers published in the past 2 years currently have a median citation rate of 120.79 compared to 21.63 for non-Covid-19 papers. When negative binomial regression is used to minimize the influence of other variables such as article number variation and field of research, Covid-19 papers have still experienced more than 80% increase in citations relative to non-Covid-19 papers. These novel findings demonstrate that Covid-19 papers are being cited at remarkably higher rates than non-Covid-19 articles contained within the same journals. This suggests that journal impact factor, which is a product of the number of citations that recently published articles receive, will likely be drastically influenced by the number of Covid-19 papers that a journal has included within its pages in the previous years.

Introduction

In the early days of January 2020, the World Health Organization (WHO) published a bulletin that brought attention to 44 new cases of pneumonia of unknown etiology that had appeared at the end of December 2019 in Wuhan City, Hubei Province of China [1]. Regional cases increased rapidly over the next weeks, and scientists worldwide began to take notice. By the end of January, more than 50 research papers had been published about the outbreak [2]. In the coming months, while researchers scrambled to learn more about the coronavirus that was now spreading internationally, the WHO again released a bulletin on March 11, 2020, officially classifying Covid-19 as a global pandemic [3]. For months after, cases and fatalities rose at staggering clips as governing bodies worldwide grappled with the best measures to contain the virus. Eventually, strict regulations and vaccine rollouts were implemented with enough success to begin to slow case growth [4]. Amidst all of this, there has been an explosion of peer-reviewed literature about Covid-19 as researchers work to uncover details such as structure, infectivity, spread, effects, prevention, and treatment of this novel virus. According to the Web of Science Science Citation Index Expanded™ (WOS), which includes more than 8,300 journals across 150 scientific disciplines, nearly 200,000 Covid-19 related papers were published or presented in 24 months between January 1, 2020 and December 31, 2021 [5].

Unsurprisingly, several seminal Covid-19 publications have been cited at incredibly high rates as researchers have turned to these papers to help guide their next steps [6–8]. However, this citation trend in Covid-19 literature has not been limited to only a select few articles. As Covid-19 remains in the spotlight, the existing Covid-19 body of literature continues to be heavily leaned upon by scientists hoping to expand the existing knowledge further.

The extent to which Covid-19 papers are being cited in selected top journals relative to non-Covid-19 papers in the same journals has yet to be explored in detail. To address this gap in the literature we use WOS to examine citation rates for Covid-19-related articles published between January 1, 2020 and December 31, 2021 across 24 major scientific journals. We compare these rates with citation rates for non-Covid-19 articles published in the same 24 journals over an identical time frame. We anticipate that these findings could be of value to journals editors and researchers when considering efforts for future publications. In addition, we expect these findings to shed light on the influence that the current influx of Covid-19 literature will have on Journal Impact Factor (JIF) in the years ahead.

Materials and methods

Journal selection

Using information provided in Clarivate’s 2020 Journal Citation Reports® [9] published on June 30, 2021, the top three journals by impact factor were selected from eight scientific categories as defined by WOS. Disciplines were selected based on the likelihood of having high relevancy to Covid-19 and, therefore, a sufficient volume of Covid-19-related papers in that field’s premier journals. A baseline of 15 Covid-19 articles was set as a requirement for a journal to be included. If a top three journal did not meet this baseline, the next highest journal by impact factor from that field was included. The disciplines chosen were respiratory, cardiology, immunology, radiology, microbiology, gastroenterology and hepatology, general and internal medicine, and multidisciplinary sciences. With three journals each from eight fields, a total of 24 journals were selected (Table 1).

Table 1. Summary of 24 selected journals.

Journal Specialty COVID-19 article Impact factor Publisher Publication mode (open-access, subscription-based, or hybrid) Country Number of issues/year
Yes No
Articles number Articles number
Nature  Multidisciplinary Sciences 145 1929  49.962  Nature Research  Hybrid  Germany  51
Nature Communications Multidisciplinary Sciences 428 12449  14.919  Nature Research  Open-access  Germany Continuous publishing
Science  Multidisciplinary Sciences 145 1479  47.728 Amer Assoc Advancement Science  Hybrid  USA  51
JAMA  Medicine, General, and Internal 54 350  56.274  Amer Medical Assoc  Subscription-based; free to public after 6 months  USA  48
Lancet  Medicine, General, and Internal 89 382  79.323  Elsevier Science Inc  Hybrid  USA  52
NEJM  Medicine, General, and Internal 98 579  91.253  Massachusetts Medical Soc  Hybrid  USA  52
Cell Host and Microbe  Microbiology 53 217  21.023  Cell Press  Subscription-based; free to public after 12 months  USA  12
Clinical Infectious Diseases Microbiology 569 1837  9.079  Oxford Univ Press Inc  Hybrid  USA  24
Nature Microbiology  Microbiology 27 256  17.745  Nature Research  Hybrid  Germany  12
European Journal of Nuclear Medicine and Molecular Imaging  Radiology 43 685  9.236  Springer  Hybrid  USA  12
Radiology  Radiology 75 497  11.105  Radiological Soc North America  Hybrid  USA  12
Ultrasound in Obstetrics Gynecology  Radiology 27 355  7.299  Wiley  Hybrid  USA  12
Journal of Am Col Cardiology  Cardiology 39 673  24.093  Elsevier Science Inc  Hybrid USA   50
Circulation  Cardiology 22 593  29.690  Lippincott Williams and Wilkins  Hybrid  USA  50
European Heart Journal  Cardiology 20 524  29.983  Oxford Univ Press  Hybrid  England  24
Gastroenterology  Gastroenterology and Hepatology 27 523  22.682  WB Saunders Co-Elsevier Inc  Hybrid  USA  12
Gut  Gastroenterology and Hepatology 35 498  23.059  BMJ Publishing Group  Hybrid  England  12
Journal of Hepatology  Gastroenterology and Hepatology 17 452  25.083  Elsevier  Hybrid  Netherlands  12
Am J of Respiratory and Crit Care Med  Respiratory 30 371  21.405  Amer Thoracic Soc  Hybrid  USA  24
European Respiratory Journal  Respiratory 40 457  16.671  European Respiratory Soc Journals Ltd  Hybrid  England  12
Lancet Respiratory Medicine  Respiratory 67 118  30.700  Elsevier Sci Ltd  Hybrid  England  12
Immunity  Immunology 48 279  31.745  Cell Press  Hybrid  USA  12
Nature Immunology  Immunology 26 249  25.606  Nature Research  Hybrid  Germany  12
Nature Reviews Immunology  Immunology 23 111  53.106  Nature Research  Subscription-based  Germany  12
Open in a new tab

Data collection and sample randomization

The WOS search criteria were customized to include documents categorized as “Article” published between January 1, 2020 and December 31, 2021. The database tags papers as articles if they meet the following criteria: “Reports of research on original works. Includes research papers, features, brief communications, case reports, technical notes, chronology, and full papers that were published in a journal and/or presented at a symposium or conference” [5]. In addition, we included papers categorized by Web of Science as “Review” to capture meta-analyses and systematic reviews as well. All articles published within the 24-month time frame were selected to produce an average citation rate for the journal itself over that period. Next, articles without the terms “Covid-19”, “coronavirus”, or “SARS-CoV-2” in the title were selected to provide an average citation rate for only non-Covid-19 articles in the journal. Finally, articles containing the keywords “Covid-19”, “coronavirus”, or “SARS-CoV-2” in the title were selected to provide an average citation rate for Covid-19-related articles. In this way, three separate citation averages were gathered for each journal: 1) all articles 2) non-Covid-19 articles 3) only Covid-19 articles.

In many journals, there was discrepancy between the number of Covid-19 and non-Covid-19 articles. To establish a comparison that was more evenly matched in terms of article volume, randomization software on Microsoft Excel was used to select a sample of non-Covid-19 papers from each journal to create a 1:1 comparison of citation rates between non-Covid and Covid papers. A single average citation rate was obtained from this smaller sample as well.

Data analysis

Data were analyzed with R software version 4.1.2 [10] using the packages (Rcmdr) [11] and (glm2) [12]. Significance was considered when the P-value was < 0.05. Median and range were used to represent continuous variables (not normally distributed) while we used frequencies and percentages to represent categorical variables. The skewness and Kurtosis tests were used for testing the normal distribution of continuous variables. We estimated the effect of Covid-19 subject on citation counts using a negative binomial regression model. The negative binomial regression model was selected over a linear regression model because it resulted in a better fit to the data and was more appropriate for count data. The negative binomial regression model is similar to the Poisson regression model (for count data) except that it performs better with data over-dispersion [13, 14]. The model was also used to assess any differences in citation rates attributed to the field category. Finally, the model was adjusted to account for the discrepancy in volume between non-Covid-19 papers and Covid-19 papers.

Results

Comparisons between non-Covid-19 and Covid-19 articles including all fields are shown in Table 2. The median citation rate at two years for Covid-19 articles in the top journals across all eight fields is 120.79 (p = <0.001). For non-Covid-19 articles, the median citation rate is 21.63 (p = < 0.001). This equates to a Covid-19: non-Covid-19 ratio of 5.58 citations per article. When comparing Covid-19 papers and the 1:1 randomized sample of non-Covid-19 papers, the median citation decreases from 21.63 to 20.1 for non-Covid-19 citations and the citation ratio between Covid-19 and non-Covid-19 papers climbs to 6.01 (p = <0.001).

Table 2. Quantity of citations for Covid-19 and non-Covid-19 articles with median and range values across all fields.

Variables COVID-19 article Total P-value
No Yes (N = 28,010)
(n = 25,863) (n = 2,147)
Count % Count % Count %
Field Cardiology 1,790 6.9 81 3.8 1,871 6.7 0.006*
Gastroenterology Hepatology 1,473 5.7 79 3.7 1,552 5.5
Immunology 639 2.5 97 4.5 736 2.6
Medicine, General, and Internal 1,311 5.1 241 11.2 1,552 5.5
Microbiology 2,310 8.9 649 30.2 2,959 10.6
Multidisciplinary Sciences 15,857 61.3 718 33.4 16,575 59.2
Radiology 1,537 5.9 145 6.8 1,682 6.0
Respiratory 946 3.7 137 6.4 1,083 3.9
Variables Median Range Median Range Median Range P-value
Average citations 21.63 6.02–83.63 120.79 28.72–824.44 52.815 6.02–824.44 < 0.001*
Average citations–random sample 20.1 6.99–79.31 120.79 28.72–824.44 53.935 6.99–824.44 < 0.001*
Open in a new tab

*Statistically significant

A negative binomial regression model was used to assess for potential confounding variables (Table 3). Only Covid-19 articles within the Medicine, General, and Internal category were significantly affected by the field itself with a 28% boost in citation rate relative to non-Covid-19 articles (p = 0.029, 95%CI [0.13–2.43]). No other fields accounted for significant difference in citation rates amongst Covid-19 and non-Covid-19 papers published within the same field. When controlling for influence of field categorization and article numbers, Covid-19 papers received 84% more citations than non-Covid-19 papers in the non-randomized sample (p<0.001, 95% CI [1.22–2.45]). When the randomized 1:1 sample was analyzed, Covid-19 papers still received 82% more citations than non-Covid-19 papers (p<0.001, 95% CI [1.20–2.43]).

Table 3. Negative binomial regression output reporting independent variable effects on citation count.

Predictors Non-randomized sample Randomized sample
IRR 95% Confidence Interval P-value IRR 95% Confidence Interval P-value
Lower Upper Lower Upper
Cardiology Reference
Respiratory -0.17 -1.32 0.98 0.771 -0.31 -1.46 0.84 0.592
Radiology -0.76 -1.92 0.40 0.197 -0.82 -1.97 0.34 0.167
Multidisciplinary Sciences 0.53 -0.72 1.77 0.407 0.48 -0.76 1.72 0.449
Microbiology -0.22 -1.38 0.93 0.702 -0.29 -1.45 0.86 0.619
Medicine, General, and Internal 1.28 0.13 2.43 0.029* 1.20 0.05 2.34 0.041
Immunology 0.11 -1.03 1.26 0.846 0.15 -1.00 1.30 0.800
Gastroenterology Hepatology -0.26 -1.41 0.89 0.659 -0.33 -1.48 0.82 0.572
Non-COVID-19 article Reference
COVID-19 article 1.84 1.22 2.45 <0.001* 1.82 1.20 2.43 <0.001*
Open in a new tab

¶Adjusted for articles’ number

*Statistically significant

Discussion

We used WOS to determine the difference in rates at which Covid-19 and non-Covid-19 articles from 24 top medical journals are being cited. Looking at all categories and journals that were included, Covid-19 articles published between January 1, 2020 and December 31, 2021 are being cited at considerably higher rates than the non-Covid-19 papers. Across eight selected fields, the median citation for a Covid-19 paper approaches six times that of a non-Covid-19 paper within the same journals. This holds even when article volume is equated using a 1:1 sample of non-Covid to Covid articles as median citation is six times greater for the Covid-19 articles. Using a negative regression model to analyze the entire data, Covid-19 papers have 84% more citations than non-Covid-19 papers when controlling for field and article number discrepancies. This number dips only slightly to 82% when the smaller, randomized sample is compared. `Amongst all fields, only articles categorized as Medicine, General, or Internal according to WOS see a bump (28%) in citation rates that can be attributed to the field itself. Thus, an article’s focus on Covid-19 seems to be a primary driver of increased citations for these articles compared to those that do not deal with a Covid-19 related topic.

Reasons behind the major increase in citations for Covid-19 articles seem straightforward. Covid-19 has dominated global focus since the onset of 2020, affecting over 200 countries across the world [15]. As such, researchers are eager to add their contributions to what is known and what can be done to combat Covid-19 and, to do this, are citing earlier works to support their approaches. This earnest for more knowledge is further bolstered by increased government funding for Covid-19 research. In the United States, the Coronavirus Aid, Relief, and Economic Security (CARES) Act was signed into law on March 27, 2020 and allocates 940 million dollars to the National Institute of Health (NIH) to be used for funding Covid-19 research [16]. Thus, more funding opportunities are available at the researcher level for research that focuses on Covid-19. Another potential reason for the spike in Covid-19 citations is the exhibited capacity of the virus to mutate rapidly. As new variants appear, researchers are offered the chance to publish on new epidemiological or variant characterization studies.

Previous studies have looked at citation rates for preprints of Covid-19 papers [17] and quality of evidence contained within published Covid-19 articles [18, 19]. Additionally, citation rate for Covid-19 papers has been assessed previously without direct comparison to non-Covid-19 papers in the same journals [20]. To our knowledge, this study is the first to quantify the rate at which a large volume of peer-reviewed Covid-19 articles are being cited, on average, in major medical journals and compare these results to non-Covid-19 articles in the same journals. In doing this, we demonstrate the sharp contrast between citation rates for Covid-19 and non-Covid-19 articles published in 24 months during the rise and height of the global pandemic. While our research reveals the substantial degree to which Covid-19 articles are being cited in top journals relative to non-Covid-19 articles, the effect on journals themselves remains to be seen.

One way that these effects may be observed is through the influence that Covid-19 papers will have on JIF. JIF is a commonly used surrogate to determine journal excellence and is calculated by dividing number of citations in the current year for articles published in the previous two years by total number of articles published in that journal during the previous two years [21]. Therefore, it is a direct measure of how many citations recently published articles in a given journal receive. As we have shown, Covid-19 papers across all selected fields are being cited at a vastly increased rate compared to non-Covid-19 papers within the same journals. We would estimate that as Covid-19 papers continue to flood the literature across all scientific fields, so long as Covid-19 continues to hold the global spotlight, Covid-19 papers will continue to be cited, on average, at much higher levels than non-Covid-19 papers. We anticipate that JIFs will be affected in coming years which could change the landscape for how journal excellence is determined in the future.

Limitations

For the scope of this paper, we used papers categorized as “Article” or “Review” in our WOS search to ensure that we were focusing on original works and evidence-based literature. This means that other publications such as letters, editorials, perspectives, and opinions were not included in our search criteria which leaves a large number of publications out of the study. Inclusion of these would undoubtedly have increased the volume of papers and affected the results, though the direction or magnitude of change is unclear. Another limitation of this study is that open-access versus subscription-based journals were not filtered separately during the WOS search. Thus, data include a combination of open-access and subscription-based publications in the non-Covid-19 articles. This would seemingly be an impactful factor in how often these articles are being cited. However, a recent meta-analysis showed that the advantage of open-access is debatable with many studies showing no difference in citation rates andquality and heterogeneity concerns posing challenges for generalization [22].

Finally, though WOS is a comprehensive and highly regarded database, it is not without shortcomings. Most notably, its categorization of a publication as “Article” seems imperfect. To examine this more closely, we randomly selected three journals included for this paper and manually searched each journal’s website for Covid-19-related articles that the journal itself had categorized as an original piece of research over three months. We compared this to the papers under “Article” that WOS captured over the same 3-month period for that journal. We found that the WOS database included 76% of the Covid-19 papers contained within the journals themselves. Further, 91% of the papers under the “Article” category in the database were classified as original works by the journals themselves. In our sample, the database captured about three-quarters of the papers that should be included but is more precise in labeling only reports of research on original work as “Article”. Though our method of locating Covid-19 and non-Covid-19 papers in the selected journals was more expedient, manual scouring of the journals themselves over the 24 months would have optimized both the sensitivity and specificity of locating and comparing the most comprehensive list of publications possible.

Supporting information

S1 Data

(XLSX)

Click here for additional data file. (11.9KB, xlsx)

Data Availability

All relevant data are within the paper and its Supporting Information files.

Funding Statement

The authors received no specific funding for this work.

References

  • 1.COVID-19 –China. World Health Organization. 2020 Jan 5 [Cited 21 Feb 2022]. Available from: https://www.who.int/emergencies/disease-outbreak-news/item/2020-DON229
  • 2.Stoye E. China coronavirus: how many papers have been published? Nature. 2020. Jan 31. Available from: https://www.nature.com/articles/d41586-020-00253-8 doi: 10.1038/d41586-020-00253-8 [DOI] [PubMed] [Google Scholar]
  • 3.WHO Director-General’s opening remarks at the media briefing on COVID-19. World Health Organization. 2020 Mar 11 [Cited 2022 Feb 24]. Available from: https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19—11-march-2020
  • 4.Talic S, Wid H, Maharaj A, Ademi Z, Xu W, Theodoratou E, et al. Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis. BMJ, 2021;375: e068302. doi: 10.1136/bmj-2021-068302 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Web of Science Science Index Expanded Edition©. Clarivate Analytics. 2021. Available from: https://www.webofscience.com/wos/woscc/basic-search
  • 6.Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet. 2020. Feb 15;395(10223): 497–506. doi: 10.1016/S0140-6736(20)30183-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Zhou F, Yu T, Du R, Fan G, Liu Y, Liu Z, et al. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet. 2020. Mar 28;395(10229): 1054–62. doi: 10.1016/S0140-6736(20)30566-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Wang D, Hu B, Hu C, Zhu F, Liu X, Zhang J, et al. Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus-infected pneumonia in Wuhan. J Am Med Assoc. 2020. Feb 7;323(11): 1061–69. doi: 10.1001/jama.2020.1585 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.2020 Journal Citation Reports®. Clarivate Analytics. 2021 Jun 30. Available from: https://jcr.clarivate.com/jcr/home?Init=Yes&SrcApp=IC2LS
  • 10.R Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2018. [Google Scholar]
  • 11.Fox J, Bouchet-Valat M, Andronic L, Ash M, Boye T, Calza S, et al. Rcmdr: R Commander. 2.6–2 ed; 2020.
  • 12.Marschner IM, Donoghoe MW. glm2: Fitting generalized linear models. 1.2.1 ed; 2018. [Google Scholar]
  • 13.Hilbe JM. Negative binomial regression. 2nd ed. Cambridge University Press; 2011. [Google Scholar]
  • 14.Davis PM, Lewenstein BV, Simon DH, Booth JG, Connolly MJL. Open access publishing, article downloads, and citations: randomised controlled trial. Bmj. 2008;337: a568. doi: 10.1136/bmj.a568 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Sanyaolu A, Okorie C, Hosein Z, Patidar R, Desai P, Prakash S, et al. Global Pandemicity of COVID-19: Situation Report as of June 9, 2020. Infect Dis. 2021. Jan 31;14: e1178633721991260. doi: 10.1177/1178633721991260 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Covid-19 funded research projects. National Institutes of Health [Internet]. 2020. Available from: https://covid19.nih.gov/funding
  • 17.Añazco D, Nicolalde B, Espinosa I, Camacho J, Mushtaq M, Gimenez J, et al. Publication rate and citation counts for preprints released during the COVID-19 pandemic: the good, the bad and the ugly. Peer J. 2021. Mar 3;9: e10927. doi: 10.7717/peerj.10927 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Khatter A, Naughton M, Dambha-Miller H, Redmond P. Is rapid scientific publication also high quality? Bibliometric analysis of highly disseminated COVID-19 research papers. Learned Publishing. 2021. Jun 1;34(4): 568–77. doi: 10.1002/leap.1403 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Zdravkovic M, Berger-Estilita J, Zdravkovic B, Berger D. Scientific quality of COVID-19 and SARS CoV-2 publications in the highest impact medical journals during the early phase of the pandemic: A case control study. PLoS ONE. 2020. Nov 5;15(11): e0241826. doi: 10.1371/journal.pone.0241826 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Martinez-Perez C, Alvarez-Peregrina C, Villa-Collar C, Sánchez-Tena MÁ. Citation Network Analysis of the Novel Coronavirus Disease 2019 (COVID-19). Int J Environ Res Public Health. 2020. Oct 21;17(20): 7690. doi: 10.3390/ijerph17207690 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Cross J. Impact factors–back to basics. Editors’ Bulletin. 2007. Nov 13;1(2): 23–30. [Google Scholar]
  • 22.Langham-Putrow A, Bakker C, Riegelman A. Is the open access citation advantage real? A systematic review of the citation of open access and subscription-based articles. PLoS ONE. 2021;1: e0253129. [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision Letter 0

Ayman Elbehiry

2 May 2022

PONE-D-22-05834Comparison of citation rates between Covid-19 and non-Covid-19 articles across 18 major medical journalsPLOS ONE

Dear Dr. Brandt,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jun 09 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Ayman Elbehiry

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

3. Thank you for stating the following financial disclosure: 

(The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.) 

At this time, please address the following queries:

a) Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution. 

b) State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

c) If any authors received a salary from any of your funders, please state which authors and which funders.

d) If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: - The scientific value of the research is low, but the authors followed a good research methodology, data analysis , as well as the unique research idea that enriched the article.

- Judging in this research depends on the availability of data about scientific journals that have been cited, which supports the results, and therefore the authors must attach the list of scientific journals that were included in the study, (it will be useful if added to the methodology), even if that would benefit the journals but it is not considered advertisement for them, it is their right, and in the same time it is essential document for the publishing this article

Reviewer #2: PLOS ONE

Comparison of citation rates between Covid-19 and non-Covid-19 articles across 18 major medical journals

PONE-D-22-05834

Thank you for asking me to review the above-titled manuscript. The topic is interesting. However, there are significant problems in the manuscript.

Abstract: 1) Line 29- What are these major journals? 2) Line 30 - start starting and end dates; why only 18 months and not up to 31 December 2021? 3) How did you search the WOS? 4) Were these non-covid and covid papers published in the same journal? 5) Were these open-access journals or subscription-based journals? Did you balance between them? 6) What were the JIFs of these journals or the median IQR? 7) Lines 34-35-- look strange because all papers on COVID-19 in subscription-based journals were available free of charge. I cannot see any explanation in your discussion 8) JIF is measured every 2 years, not 18 months. 9) Lines 37-39- on what basis did you come. 10) What was the percentage of Covid versus non-covid in each journal selected.

Introduction - 1) China declared a novel virus causing this disease by December 2019. 2) Lines 56, page 3: Why not up to 31 December 2021? to include vaccination research as well? 3) Give references/citations (lines 57-63). 4) Not clear what is the problem that triggered the study? 5) What is the purpose of the study? We all know that citations are most likely higher for COVID-19 papers, you need to dig deeper into this. 6) What is your research question?

Methods- 1) Line 75- was the number of articles on Covid-19 and non-Covid balanced? How? 2) Line 72-Why only top JIF journals? 3) Why did you not target the top JIF (Q1), the middle JIF journals (Q2-Q3) and those with low JIF (Q4)? 4). Why were Journals on Neurology and those on Gastroenterology and Hepatology not included? 5) What do you mean by Multidisciplinary Science journals? Line 78? 6) How many journals were included for each field? STATE THE JOURNALS INCLUDED IN EACH FIELD. 7) WHAT ARE THESE THREE TOP JOURNALS? Were these on Microbiology and or Virology?

Methods- It is unusual and strange that the authors do not provide any details about their work. Research must be transparent.

Methods- Lines 80-85 We know that this is a limitation in the classification of articles in WoS. Did you identify them in the analysis and show numbers of research, numbers of case report, numbers of brief communication etc, and you should define each category.

Methods- Lines 87 (page 4) and linnes 88-90 (page 5) NOT CLEAR.

Methods 94-96 - The aim should be EQUAL NUMBERS of Covid-19 and Non-Covid papers not 2:1. The number of Non-Covid is usually small, so accept it as the basic number and randomise for an equivalent number of COVID-19 papers. This should be the way.

METHODS- WHY THE AUTHORS DID NOT INCLUDE FUNDING AND COMPARE BETWEEN THE COVID-19 PAPERS and NON-COVID PAPERS?

Methods- What were the countries and institutes that produced these papers in both groups?

Methods: WHY WERE THE ALTMETRIC SCORES NOT COMPARED? As well?

English needs editing. Several statements are not clearly stated. Also, statements like "This was done..."

Table 1- 1) What is the number of journals in each category "Multidisciplinary, Cardiology etc" 2) What is the NAME of the Journal in each category? 3) What is the JIF of each journal? 4) State the number of papers from each journal in Covid-19 and Non-Covid 19.

Table- You need a new table summarising key topics raised in each category and numbers in both groups.

Discussion- 1) Poorly written. 2) Why do lines 141-155 have no citations? 2) Funding should be compared between the two groups in the study; SEE COMMENTS UNDER METHODS? 3) We need to know with journals were open access and which one was subscription-based paper.

Discussion- No explanation was given to the 10 times issue of open access journals. We know that all papers on COVID-19 were available free of charge and this is an important cause for the high citations of the COVID-19 papers (not discussed). BUT it does not explain the authors' claim for 10 times issue (also missing).

Limitations: 1) Articles are not only research and what is stated is not accurate. 2) Are case reports evidence-based? 3) Talking about evidence-based work- The top in this are Systematic Reviews and meta-analyses. MMy question, Why these two groups were excluded? Lines 191.

What are these three journals- state their Names. For all journals, you must state title, company/publisher, JIF, open-access or subscription-based, country, and the number of issues per year, the year it was issued. Show the number of papers for each group This could be in a table.

Conclusion- should be rewritten.

References- Should be improved.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr Mustafa Mohammed Mustafa

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2022 Jul 27;17(7):e0271071. doi: 10.1371/journal.pone.0271071.r002

Author response to Decision Letter 0

13 Jun 2022

Comments also included in separate attached file title "Response to Reviewers"

Response to Reviewer Comments

Reviewer #1: The scientific value of the research is low, but the authors followed a good research methodology, data analysis, as well as the unique research idea that enriched the article.

We appreciate the positive review of our study

1. Judging in this research depends on the availability of data about scientific journals that have been cited, which supports the results, and therefore the authors must attach the list of scientific journals that were included in the study, (it will be useful if added to the methodology), even if that would benefit the journals but it is not considered advertisement for them, it is their right, and in the same time it is essential document for the publishing of this article. Thank you for bringing this to our attention. We created an additional table to list all journals included in the study. We will also include all data gathered from each journal in our data set which will be uploaded as a Supporting Information File.

Reviewer #2: Thank you for asking me to review the above-titled manuscript. The topic is interesting. However, there are significant problems in your manuscript.

We thank you for your interest in the study and willingness to provide helpful feedback.

1. Line 29- What are these major medical journals? Because there are 24 journals, we believe that it would detract from the focus of the abstract to list them individually. They are listed separately in Table 1 in the methodology section.

2. Line 30- Start and end dates; Why only 18 months and not up to 31 December 2021? Start and end dates have now been included. Data has been expanded to include 24 months up to 31 December 2021.

3. How did you search the WOS? The search was conducted by searching for all titles that had “Covid, “sars-cov-2”, or “coronavirus” in a given journal and obtaining a single average citation for these papers. We then repeated the process for all titles that did not have any of the aforementioned keywords and obtained a single citation for these papers as well. The process was repeated individually for each journal. We have included an additional sentence explaining briefly how the search was conducted.

4. Were these non-Covid and Covid papers published in the same journal? We have adjusted the wording to make it clearer that these papers were published within the same journal.

5. Were these open-access journals or subscription-based journals? Did you balance between them? We have added a line stating that both open-access and subscription-based journals were included. The great majority of the included journals employ a hybrid model with options for both open-access and subscription-based publishing. WOS does allow for filtering of open-access articles but fails to filter for articles that were originally published as subscription-based but become free to the public after a specified period of time has elapse, as is the protocol for several of the journals that we searched. Further, a recent meta-analysis showed the advantage of open-access is debatable with many studies showing no difference, with quality and heterogeneity concerns posing challenges for generalization.

Reference:

Langham-Putrow A, Bakker C, Riegelman A (2021). Is the open access citation advantage real? A systematic review of the citation of open access and subscription-based articles. PLOS ONE 1e0253129. https://doi.org/10.1371/journal.pone.0253129

6. What are the JIFs of these journals or the median IQR? JIFs for all journals now included as part of additional table (Table 1).

7. Lines 34-35—look strange because all COVID-19 in subscription-based journals were available free of charge. I cannot see any explanation in your discussion. Wording has been adjusted to clarify this point.

8. JIF is measured every 2 years, not 18 months. Data now reflects 2-year period from 1/1/2020 to 31/12/2021 rather than 18 months as previously.

9. Lines 37-39- on what basis did you come? We are arriving at this suggestion based on our findings that Covid-19 papers are being cited at much higher rates than non-Covid papers within the same journals. Therefore, we expect that journals will see their JIF inflate in the coming years due given that they have been publishing (Covid) articles that are, on average, more highly cited than articles that are unrelated to Covid. We have adjusted the wording to make this point more salient.

10. What was the percentage of Covid versus non-Covid in each journal selected? Total number of papers of each type now included in Table 1.

11. China declared a novel virus causing this disease by December 2019. Wording adjusted to make this point clearer.

12. Lines 56, page 3: Why not up to 31 December 2021 to include vaccination research as well? Data now includes all publications up to this date.

13. Give references/citations (lines 57-63). Citation for these lines has now been added.

14. Not clear what is the problem that triggered the study? What is the purpose of the study? We all know that citations are most likely higher for COVID-19 papers, you need to dig deeper into this. What is your research question? We agree that it is widely known that Covid-19 papers are cited at a higher rate across the literature. However, we note that there is a gap in the literature comparing the rate at which Covid-19 papers in top journals are being cited to the rate at which non-Covid articles in the same journals are being cited. By doing this direct comparison we intend to answer the question “How much different is the rate of citation for Covid articles versus non-Covid articles published within the same journals across 24 of the top journals spanning the scientific field?” Though not the main focus of our paper, we believe that these data will be helpful in the future when determining the utility of JIF to quantify journal excellence and prestige in a post-Covid landscape.

15. Methods-1) Line 75- was the number of articles on Covid-19 and non-Covid balanced? How? Yes, in Tables 2 and 3 we include both absolute numbers and a balanced 1:1 random sample of Covid and non-Covid papers.

16. Line 72- why only top JIF journals? Why did you not target the top JIF (Q1), the middle JIF journals (Q2-Q3) and those with low JIF (Q4)? We targeted only top JIF journals from 8 fields with the belief that these 24 journals and the >20000 articles within would provide adequate sample size to highlight the different rates at which Covid-19 and non-Covid-19 articles from the same journals are being cited.

17. Why were journals on Neurology and those on Gastroenterology and Hepatology not included?

Top journals by JIF in Neurology did not meet inclusion criteria of minimum number of Covid articles per WOS search. Thank you for the suggestion of adding Gastroenterology and Hepatology. Data from the top 3 journals in this category have been added.

18. What do you mean by Multidisciplinary Science journals? This is the categorization that is assigned by WOS to journals that publish broadly across the field of science.

19. How many journals were included for each field? State the journals included in each field. What are these three top journals? Were these on Microbiology and or Virology? It is unusual and strange that the authors do not provide any details about their work. Research must be transparent. Table 1 has been added which lists all journals (3 from each field) and will be included in the methodology section. These journals were not specific to Microbiology or Virology. We have added Microbiology as a separate category and included 3 journals from this category as well.

20. Methods- lines 80-85- We know that this is a limitation in the classification of articles in WOS. Did you identify them in the analysis and show numbers of research, numbers of case report, numbers of brief communication etc, and you should define each category. Unfortunately, WOS classifies each of these article types under a single “Article” categorization. Distinguishing between paper types would require manual individual extraction of each paper which is not feasible given volume of included papers. Further, as our main objective is to compare citation rates between Covid and non-Covid articles within the same journals, we believe that focus on sub-categorization does not advance the paper toward the main objective.

21. Lines 87 (page 4) and lines 88-90 (page 5) not clear. Wording has been adjusted for clarity

22. Methods 94-96- The aim should be equal numbers of Covid-19 and Non-Covid papers not 2:1. The number of Non-Covid is usually small, so accept it as the basic number and randomize for an equivalent number of Covid-19 papers. Thank you for this recommendation. We have updated the date to compare a 1:1 sample rather than 2:1.

23. Why did the authors not include funding and compare between the Covid-19 papers and non-Covid papers? Determining funding would require manual extraction as WOS does not allow for filtering by funding. This is not feasible given number of papers.

24. What were the countries and institutes that produced these papers in both groups? Similarly, this would require manual extraction from each paper which was not feasible given volume.

25. Methods: Why were the altimetric scores not compared as well? Given our specific objective of comparing citation rates in Covid and non-Covid articles within top selected journals based on JIF, we believe that this would not influence the objective.

26. English needs editing. Several statements are not clearly stated. Also, statements like “This was done…” We reworded any statements that appeared unclear and removed “This was done..” where present

27. Table 1 1) What is the number of journals in each category “Multidisciplinary, Cardiology etc”. What is the name of the journal in each category? What is the JIF of each journal? State the number of papers from each journal in Covid-19 and Non-Covid-19. We have created a new table (now Table 1) which provides each of these details.

28. Table- you need a new table summarizing key topics raised in each category and numbers in both groups. Given the volume of papers, it would not be feasible to go through each paper to determine key topics as there is not a set way to do this with the WOS search function.

29. Discussion- Poorly written. Discussion reworked to improve fluidity and clarity

30. Why do lines 141-155 have no citations? These lines are in reference to the data accrued from this study. We have added references to the tables for clarity.

31. We need to know which journals were open access and which one was subscription-based paper. No explanation was given to the 10 times issue of open access journals. We know that all papers on Covid-19 were available free of charge and this is an important cause for the high citations of the Covid-19 papers (not discussed). But it does not explain the authors’ claim for 10 times issue (also missing). Table 1 now includes the mode of publication for each journal. In the updated data (Table 3), we have added “Field” as a categorical variable which should help address the 10 times issue and we also added an additional 6 months’ worth of publications which should both help to make the data more logical. We have added the open-access versus subscription-based issue to our limitations as this could impact citations but feel that the recently published large meta-analysis (reference included in response to comment number 5) provides support that there is not clear-cut evidence to suggest that open-access publications will inherently be cited at a greater rate than subscription-based publications.

32. Limitations- Articles are not only research and what is stated is not accurate. Are case reports evidence-based? Talking about evidence-based work- The top in this are Systematic Reviews and meta-analyses. Why were these two groups excluded? Line 191. WOS defines “Article” as “Reports of research on original works. Includes research papers, features, brief communications, case reports, technical notes, chronology, and full papers that were published in a journal and/or presented at a symposium or conference.” Your point that Systematic Reviews and Meta-Analyses are evidence-based work is well-made and these publications fall under the “Review” category for WOS. For the updated data, we included all papers under the “Review” category as well as the “Article” category. We have included a link to the WOS category distinction for reference.

- https://images.webofknowledge.com/images/help/WOS/hs_document_type.html

33. What are these three journals- state their names. For all journals, you must state title, company/publisher, JIF, open-access or subscription-based, country, and the number of issues per year. Show the number of papers for each group. This could be a table. Thank you for this helpful suggestion. Table 1 has been created to consolidate these pieces of information.

34. Conclusion- Should be rewritten. Conclusion was reworked for clarity and fluency

35. References should be improved. References updated for improved clarity and formatting

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file. (23.9KB, docx)

Decision Letter 1

Ayman Elbehiry

23 Jun 2022

Comparison of citation rates between Covid-19 and non-Covid-19 articles across 24 major scientific journals

PONE-D-22-05834R1

Dear Dr. Michael Brandt,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Ayman Elbehiry

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Acceptance letter

Ayman Elbehiry

18 Jul 2022

PONE-D-22-05834R1

Comparison of citation rates between Covid-19 and non-Covid-19 articles across 24 major scientific journals

Dear Dr. Brandt:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Ayman Elbehiry

Academic Editor

PLOS ONE

Articles from PLoS ONE are provided here courtesy of PLOS

RESOURCES