Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2021 Feb 2;16(2):e0246427. doi: 10.1371/journal.pone.0246427

General medical publications during COVID-19 show increased dissemination despite lower validation

Nan Gai 1, Kazuyoshi Aoyama 1,2, David Faraoni 1, Neil M Goldenberg 1,3, David N Levin 1, Jason T Maynes 1,4, Mark J McVey 1,5, Farrukh Munshey 1, Asad Siddiqui 1, Timothy Switzer 1, Benjamin E Steinberg 1,6,*
Editor: Itamar Ashkenazi7
PMCID: PMC7853485  PMID: 33529266

Abstract

Background

The COVID-19 pandemic has yielded an unprecedented quantity of new publications, contributing to an overwhelming quantity of information and leading to the rapid dissemination of less stringently validated information. Yet, a formal analysis of how the medical literature has changed during the pandemic is lacking. In this analysis, we aimed to quantify how scientific publications changed at the outset of the COVID-19 pandemic.

Methods

We performed a cross-sectional bibliometric study of published studies in four high-impact medical journals to identify differences in the characteristics of COVID-19 related publications compared to non-pandemic studies. Original investigations related to SARS-CoV-2 and COVID-19 published in March and April 2020 were identified and compared to non-COVID-19 research publications over the same two-month period in 2019 and 2020. Extracted data included publication characteristics, study characteristics, author characteristics, and impact metrics. Our primary measure was principal component analysis (PCA) of publication characteristics and impact metrics across groups.

Results

We identified 402 publications that met inclusion criteria: 76 were related to COVID-19; 154 and 172 were non-COVID publications over the same period in 2020 and 2019, respectively. PCA utilizing the collected bibliometric data revealed segregation of the COVID-19 literature subset from both groups of non-COVID literature (2019 and 2020). COVID-19 publications were more likely to describe prospective observational (31.6%) or case series (41.8%) studies without industry funding as compared with non-COVID articles, which were represented primarily by randomized controlled trials (32.5% and 36.6% in the non-COVID literature from 2020 and 2019, respectively).

Conclusions

In this cross-sectional study of publications in four general medical journals, COVID-related articles were significantly different from non-COVID articles based on article characteristics and impact metrics. COVID-related studies were generally shorter articles reporting observational studies with less literature cited and fewer study sites, suggestive of more limited scientific support. They nevertheless had much higher dissemination.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic has given rise to an unprecedented quantity of publications in a short period of time as researchers worldwide attempt to report their experiences to better understand this new disease and identify promising treatments [1]. This has contributed to a COVID-19 “infodemic”–an overwhelming quantity of information, leading to the rapid dissemination of less stringently validated information [2].

Given the devastating severity of COVID-19, there is an understandable urgency to disseminate new findings. However, the rush to publish has potentially led to the compromise of scientific integrity [3]. This has led to advocacy for quality over quantity, cautioning that a crisis is no excuse for lowering scientific standards [3–5]. Yet, the COVID-19 pandemic has magnified traditional problems of “uninformative” clinical trials–those whose results are not useful to patients, clinicians, researchers, or policy makers [6, 7].

While specific concerns about COVID-19-related publications have been expressed [8], a formal analysis of the extent to which the medical literature has shifted during the pandemic is lacking. In this analysis, we aimed to quantify how scientific publications changed at the outset of the COVID-19 pandemic by performing a cross-sectional bibliometric study of published studies in four high-impact medical journals to identify differences in the characteristics of COVID-19 related publications compared to non-pandemic related studies.

Methods

This is a cross-sectional bibliometric study of original COVID-19 related research publications in the four general medical journals with the highest impact factors [9]–The Journal of the American Medical Association (JAMA), New England Journal of Medicine (NEJM), The Lancet, and Nature Medicine. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines [10].

We searched for original investigations related to SARS-CoV-2 and COVID-19 published in March and April 2020 through MEDLINE. MEDLINE alone was used because it contained entries for all publications within our four journals of interest. Accordingly, other databases were not consulted. As comparison groups, we retrieved all non-COVID-19 research publications over the same two-month period in 2019 and 2020. We included original scientific research, and excluded opinion, news, and educational pieces. Two reviewers verified studies for inclusion and two reviewers audited extracted data. Any discrepancies in eligibility assessment and data collection were resolved by consensus. Extracted data included publication characteristics, study characteristics, author characteristics, and impact metrics. Impact metrics (numbers of reads, citations, and tweets) were not normalized to the time since publication.

Categorical data are presented as counts and percentages and continuous data as medians and interquartile ranges (IQRs). Our primary measure was principal component analysis (PCA) of publication characteristics and impact metrics across groups. In our study, we sought to discover any differences in multiple article metrics between the 2020 COVID period and historical controls. Principal component analysis allows for the determination of the largest contributors to the variance in the data across all article metrics, in an unsupervised fashion without biasing data segregation [11]. Using PCA allows us to identify the most important features that capture the maximum information about the dataset, reducing dimensionality without any significant loss of information. Comparisons between groups were conducted using Chi-square or Fisher’s exact tests for proportions and non-parametric Kruskal-Wallis tests with Dunn’s multiple comparison for continuous data. Data for each journal were aggregated for analysis. P values less than 0.05 were considered statistically significant. Analyses were performed using GraphPad PRISM software version 7.0 and RStudio version 1.3.1056.

Results

The initial MEDLINE literature search identified 1,119 total articles for consideration (262 COVID-related). We identified 402 publications that met inclusion criteria: 76 were related to COVID-19; 154 and 172 were non-COVID publications over the same period in 2020 and 2019, respectively (data available in S1 Dataset). Principal component analysis utilizing the collected bibliometric data revealed segregation of the COVID-19 literature subset from both groups of non-COVID literature (2019 and 2020), verifying that the bibliometric characteristics capture a change in publication metrics (Fig 1). The most significant contributions to the PCA came from metrics representing article dissemination (reads, tweets, and citations with 57%, 54%, and 43% each towards the first principal component, PC1). The two non-COVID subsets of data possess a near overlap in the PCA, indicating a strong consistency between the two years analyzed and emphasizing the uniqueness of the COVID-related literature.

Fig 1. Principal component analysis of COVID and non-COVID publication characteristics and impact metrics.

Fig 1

Open in a new tab

Each point in the plot corresponds to a single characteristic provided in Table 1 for COVID (green square) and non-COVID publications from 2019 (purple circle) and 2020 (gray triangle). Principal component 1 (PC1) is shown plotted against (A) PC2 and (B) PC3. PC1, PC2, and PC3 respectively account for 32.4%, 24.8% and 16.4% of the variability. Non-COVID publications from 2019 and 2020 clusters overlap, whereas COVID publications cluster separately. This unbiased analysis suggests COVID-related publications differ from both concurrent and historic non-COVID publications.

To further evaluate how the published COVID-19 research literature differed from non-COVID-19 investigations, we first compared their publication characteristics (Table 1). Publication characteristics segregated by individual journal are provided in the Table in S1 Table. COVID-19 publications were more likely to describe prospective observational (31.6%) or case series (41.8%) studies without industry funding as compared with non-COVID articles, which were represented primarily by randomized controlled trials (32.5% and 36.6% in the non-COVID literature from 2020 and 2019, respectively). Moreover, COVID-related publications had lower word counts with fewer citations of other medical literature. While the number of authors was unchanged, the number of author affiliations was decreased, suggesting a lower level of collaborative or multi-institutional studies. There was no observed difference in the proportion of female first or corresponding authors. For Nature Medicine, the only evaluated journal to report submission dates, COVID-related submissions were published in a much shorter amount of time (35.1 days versus 288.3 and 305.3 days for 2020 and 2019 non-COVID publications, respectively).

Table 1. Publication characteristics and impact.

Non-COVID publications COVID publications P value a
2019 2020 COVID vs non-COVID 2019 COVID vs non-COVID 2020
Articles (n) 172 154 76
Article type, No. (%)
Meta-analysis 6 (3.5) 4 (2.6) 0 (0) <0.0001 <0.0001
Systematic review 4 (2.3) 6 (3.9) 2 (2.6)
Narrative review 17 (9.9) 16 (10.4) 4 (5.3)
RCT 63 (36.6) 50 (32.5) 1 (1.3)
Cohort / prospective 30 (17.4) 29 (18.8) 24 (31.6)
Case-control 3 (1.7) 4 (2.6) 2 (2.6)
Case report or series 14 (8.1) 17 (11.0) 31 (40.8)
Basic biomedical research / preclinical 18 (10.5) 18 (11.7) 5 (6.6)
Other 17 (9.9) 10 (6.5) 7 (9.2)
Study characteristics b, No. (%)
Registered trial 75 (47.5) 54 (35.5) 0 (0) <0.0001 <0.0001
Industry funding 37 (22.2) 48 (31.2) 2 (2.7) <0.0001 <0.0001
Publication characteristics
Author number, median (IQR) 15 (17) 12 (16) 10.5 (12.75) 0.4499 1
Author affiliations, median (IQR) 8 (7) 7 (13) 4 (4) <0.0001 <0.0001
Female corresponding or first author b, No. (%) 59 (36.4) 56 (36.8) 24 (33.8) 0.7671 0.7646
Time to publication, days, mean (SD) c 305.3 (124.2) 288.3 (99.7) 35.1 (4.6) <0.0001 0.0001
Word count, median (IQR) 3816 (2063) 3746 (2061) 914 (2139) <0.0001 <0.0001
References, median (IQR) 33 (19.75) 33 (24.5) 6 (22) <0.0001 <0.0001
Publication impact d, median (IQR)
Reads e 17 648 (21 959) 9 652 (15 110) 224 714 (389 243) <0.0001 <0.0001
Tweets 168.5 (250.5) 81.5 (149.5) 1202 (4014 <0.0001 <0.0001
Times cited 25 (33.75) 2 (4) 50.5 (125.3) 0.1414 <0.0001
Open in a new tab

Abbreviations: COVID, Coronavirus Disease; JAMA, Journal of the American Medical Association; NEJM, New England Journal of Medicine; RCT, Randomized Controlled Trial.

a P values, adjusted for multiple comparison, shown for comparison between COVID and non-COVID publications from the indicated year.

b Articles in which study characteristic was not reported or in which gender of author was unknown were excluded from calculation of the proportion.

c Includes only Nature Medicine publications as submission dates not reported for JAMA, The Lancet, or NEJM.

d Reads, tweets and times cited are reported as absolute numbers and are not normalized to their time since publication.

e Excludes articles published in The Lancet, which does not list article reads as part of their Altmetrics.

The observed differences in publication characteristics presumably represents the initial effort to quickly provide clinicians and policymakers with information in the early phase of the pandemic, regardless of quality. To objectively evaluate the extent to which the COVID-19 literature was disseminated, we analyzed the number of accesses, tweets, and citations within our bibliometric dataset. Publications related to COVID had an order of magnitude greater accesses, tweets, and citations compared with non-COVID publications from the same period in both 2019 and 2020 (Table 1). This absolute difference does not consider the greater time since publication of articles from 2019 and therefore may conservatively underestimate the unparalleled rate at which observational data spread across the international medical community.

Discussion

Using an unbiased approach, our PCA suggests that published pandemic-related studies have different article characteristics and impact metrics compared with non-COVID studies. They generally consist of shorter articles reporting observational studies with less literature cited and fewer study sites, suggestive of more limited scientific support. Yet, pandemic-related research is associated with greater reach in terms of readership, citations, and tweets, which speaks to the strong appetite for pandemic-related findings.

The publication characteristics described in our analysis reflect the urgency with which the medical, scientific, and lay communities sought information as the pandemic evolved. This on-going need, however, should be tempered with scientific and ethical oversight that is at least as rigorous as normal times with a focus on well-designed trials and not rapid dissemination of low-quality data. The potential harms of producing multiple iterations of lower-quality studies have been identified, including wasting of resources, lapses in the ethical standard of scientific reporting, delaying the conduct of higher-level evidence trials, diluting the quality of available evidence, and endangering the ethical responsibility to patients who enroll in trials with the expectation of assisting in medical and scientific advancement [6, 12, 13]. Researchers should endeavour to maintain high-quality research methods by increasing collaboration across multiple centres, helping to overcome limitations that may exist from single-centre efforts [3, 14]. International teams working in concert and not in competition on well-designed studies would greatly improve the capacity to detect clinically meaningful effects to inform the international health system’s efforts against COVID-19. For example, research consortia could establish research priorities and promote the implementation of master protocols with adaptive platforms [15–17]. This type of approach is designed for the perpetual investigation of multiple interventions with timely adaptation, an ideal framework for our evolving COVID-19 health crisis that would facilitate wider collaboration and mitigate against the production of low-quality evidence and poor scientific reporting.

Efforts have also focused on the expanding COVID-19 literature itself using both manual and automated methods. Content experts have been vetting the published literature to provide health care workers and policymakers with curated digital compendiums of high-quality research papers, such as the 2019 Novel Coronavirus Research Compendium [18]. Computational approaches are being used to mine the published COVID-19 literature to answer key questions related to the pandemic [19]. As these resources continue to grow, increasing effort will be required to ensure that the medical, scientific, and lay communities can engage with the resulting data and analyses in a meaningful way.

Our analysis, however, has limitations. We focus on the earliest phase of pandemic in order to capture how the medical community first pivoted to acquire and disseminate COVID-19-related knowledge. This potentially biases our results towards observational studies as there would be limited time to advance and report more rigorous study designs, such as randomized controlled trials. Moreover, to efficiently disseminate medical knowledge, the included journals made pandemic-related content freely available, which may have contributed to the observed increase in impact metrics. Lastly, our bibliometric analysis does not consider the root cause of the disparity between COVID and non-COVID publications. This is likely multifactorial but could, in part, reflect the feasibility of a timely study completion, variable adherence to reporting standards, and a strained peer review system. Ongoing evaluations of the publication process over the entirety of the pandemic will inform how the scientific community can most effectively, safely, and ethically disseminate valuable medical knowledge in a time of acute crisis.

Conclusion

COVID-19 led to a significant change in the characteristics of research studies across high-impact general medical journals. During this pandemic, the rapid and broad dissemination of research findings, regardless of underlying quality, were amplified and potentially contributed to the infodemic of misinformation at a time when best evidence needs to be emphasized. Ultimately, relaxing the rigorous standards for scientific research, although tempting for many altruistic reasons during a pandemic, may not actually achieve the objective of producing a solid evidence-based foundation upon which patients, clinicians, and policymakers can make meaningful decisions. The scientific and medical communities must strongly advocate for the thoughtful selection of high-quality research that will ensure the generation of meaningful knowledge and that participants of scientific trials who volunteer their health experience do not do so in vain.

Supporting information

S1 Dataset. The dataset used for the analyses in this study.

(XLSX)

Click here for additional data file. (62.4KB, xlsx)
S1 Table. Publication characteristics and impact by journal.

(DOCX)

Click here for additional data file. (25.4KB, docx)

Data Availability

All relevant data are within the paper and its Supporting Information files.

Funding Statement

Funding was provided through departmental funds from the Department of Anesthesia and Pain Medicine at the Hospital for Sick Children.

References

  • 1.Balaphas A, Gkoufa K, Daly M-J, de Valence T. Flattening the curve of new publications on COVID-19. J Epidemiol Community Health. 2020. July 3;jech–2020-214617. 10.1136/jech-2020-214617 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Tangcharoensathien V, Calleja N, Nguyen T, Purnat T, D’Agostino M, Garcia-Saiso S, et al. Framework for Managing the COVID-19 Infodemic: Methods and Results of an Online, Crowdsourced WHO Technical Consultation. J Med Internet Res. 2020;22(6):e19659 10.2196/19659 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.London AJ, Kimmelman J. Against pandemic research exceptionalism. Science. 2020. May 1;368(6490):476–7. 10.1126/science.abc1731 [DOI] [PubMed] [Google Scholar]
  • 4.Bauchner H, Fontanarosa PB. Randomized Clinical Trials and COVID-19: Managing Expectations. JAMA. 2020;323(22):2262–3. 10.1001/jama.2020.8115 [DOI] [PubMed] [Google Scholar]
  • 5.McDermott MM, Newman AB. Preserving Clinical Trial Integrity During the Coronavirus Pandemic. JAMA. 2020. June 2;323(21):2135–6. 10.1001/jama.2020.4689 [DOI] [PubMed] [Google Scholar]
  • 6.Zarin DA, Goodman SN, Kimmelman J. Harms From Uninformative Clinical Trials. JAMA. 2019. September 3;322(9):813–4. 10.1001/jama.2019.9892 [DOI] [PubMed] [Google Scholar]
  • 7.Pundi K, Perino AC, Harrington RA, Krumholz HM, Turakhia MP. Characteristics and Strength of Evidence of COVID-19 Studies Registered on ClinicalTrials.gov. JAMA Intern Med. 2020. July 27; 10.1001/jamainternmed.2020.2904 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Salazar JW, McWilliams Jr JM, Wang TY. Setting Expectations for Clinical Research During the COVID-19 Pandemic. JAMA Intern Med. 2020. July 27; 10.1001/jamainternmed.2020.2882 [DOI] [PubMed] [Google Scholar]
  • 9.Clarivate Web of Science. Web of Science Journal Citation Reports [Internet]. 2020 [cited 2020 Aug 4]. Available from: https://clarivate.com/webofsciencegroup/web-of-science-journal-citation-reports-2020-infographic
  • 10.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007. October 20;370(9596):1453–7. 10.1016/S0140-6736(07)61602-X [DOI] [PubMed] [Google Scholar]
  • 11.Jolliffe IT, Cadima J. Principal component analysis: a review and recent developments. Philos Trans R Soc A Math Phys Eng Sci. 2016. April 13;374(2065):20150202 10.1098/rsta.2015.0202 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Bauchner H, Golub RM, Zylke J. Editorial Concern—Possible Reporting of the Same Patients With COVID-19 in Different Reports. JAMA. 2020. April 7;323(13):1256 10.1001/jama.2020.3980 [DOI] [PubMed] [Google Scholar]
  • 13.Califf RM, Hernandez AF, Landray M. Weighing the Benefits and Risks of Proliferating Observational Treatment Assessments: Observational Cacophony, Randomized Harmony. JAMA. 2020. July 31; 10.1001/jama.2020.13319 [DOI] [PubMed] [Google Scholar]
  • 14.Cheng MP, Lee TC, Tan DHS, Murthy S. Generating randomized trial evidence to optimize treatment in the COVID-19 pandemic. Cmaj. 2020;192(15):E405–7. 10.1503/cmaj.200438 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Angus DC, Alexander BM, Berry S, Buxton M, Lewis R, Paoloni M, et al. Adaptive platform trials: definition, design, conduct and reporting considerations. Nat Rev Drug Discov. 2019;18(10):797–807. 10.1038/s41573-019-0034-3 [DOI] [PubMed] [Google Scholar]
  • 16.Dean NE, Gsell P-S, Brookmeyer R, Crawford FW, Donnelly CA, Ellenberg SS, et al. Creating a Framework for Conducting Randomized Clinical Trials during Disease Outbreaks. N Engl J Med. 2020. April 1;382(14):1366–9. 10.1056/NEJMsb1905390 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Woodcock J, LaVange LM. Master Protocols to Study Multiple Therapies, Multiple Diseases, or Both. N Engl J Med. 2017. July 5;377(1):62–70. 10.1056/NEJMra1510062 [DOI] [PubMed] [Google Scholar]
  • 18.The Johns Hopkins University. 2019 Novel Coronavirus Research Compendium (NCRC) [Internet]. 2020. [cited 2020 Aug 25]. Available from: https://ncrc.jhsph.edu
  • 19.Wang LL, Lo K, Chandrasekhar Y, Reas R, Yang J, Burdick D, et al. CORD-19: The COVID-19 Open Research Dataset [Preprint]. arXiv:2004.10706. 2020. April [cited 2020 Aug 25]. 32510522 [Google Scholar]

Decision Letter 0

Itamar Ashkenazi

4 Dec 2020

PONE-D-20-29109

General Medical Publications During COVID-19 Show Increased Dissemination Despite Lower Validation.

PLOS ONE

Dear Dr. Steinberg,

Thank you for submitting your manuscript to PLOS ONE.

Two reviewers submitted their reviews and their recommendations do not exactly match.  I wish to say that the topic is important.  Indeed, the epidemic has opened doors for anyone wishing to publish anything that has to do with COVID-19.  However, the comments by one of the reviewers is critical.  The authors need to justify the methodology used.  I am not a statistician.  I know PCA can be used to identify subjects with similar attributes.  Whether dispersion in PCA proves that COVID-19 publications are of inferior quality is a question readers will ask. Reviewer no. 2 projected this problem - is PCA the right tool to evaluate the study question?

Due to the importance of this subject, after careful consideration, I feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, I invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jan 18 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Itamar Ashkenazi

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Funding Statement Section of your manuscript:

"Funding was provided through departmental funds from the Department of Anesthesia and Pain Medicine at the Hospital for Sick Children."

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

"The authors received no specific funding for this work."

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: During COVID-19, many articles published in various journals. However, I noticed that many authors are not related to medical field!! Many authors want to gain more citations by choosing COVID-19 topic. Also, many journals are publishing COVID-19 related publications without much review (rapid review) so, sometimes quality of the manuscript is poor but dissemination (reach) is greater as non-medical field persons are searching for COVID-19 related articles.

This manuscript has novelty and timely to analyze the COVID-19 and non-COVID-19 related articles characteristics.

Manuscript is well written and it can be published with minor corrections.

Line-2: There should not be period (full stop) in title.

Line-60: MEDLINE is the pioneer database but author should have to describe for choosing this database and excluding other databases.

Line-62: Single reviewer

Reviewer #2: I will focus on methods and reporting

Major

1) Abstract is quite short and poor. no information on methods or results.

2) The methods are unclear, especially the role of PCA and what the outcomes are e.g. impact metrics.

3) Why don't the authors use established quality assessment tools for observational studies and report that instead?

4) I don't follow why the data were aggregated within each journal and each study was not included as a different case.

Minor

1) Rstudio is the shell, reference what's under the hood i.e. R version.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Anjum Sherasiya

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Feb 2;16(2):e0246427. doi: 10.1371/journal.pone.0246427.r002

Author response to Decision Letter 0

28 Dec 2020

RE: PONE-D-20-29109

General Medical Publications During COVID-19 Show Increased Dissemination Despite Lower Validation.

PLOS ONE

Dear Dr. Ashkenazi:

We would like to thank you and the expert reviewers for the thoughtful evaluation of our paper and helpful comments. Having addressed and incorporated the reviewers’ suggestions, we now submit a much improved manuscript for your consideration. In addition, our revised manuscript incorporates the requested changes to conform to your journal’s requirements. Please find attached below detailed point-by-point responses to the reviewers’ concerns.

Thank you again to you and the reviewers.

Sincerely,

Benjamin Steinberg, on behalf of all co-authors

Response to Dr. Ashkenazi, Academic Editor

(1) The authors need to justify the methodology used. I am not a statistician. I know PCA can be used to identify subjects with similar attributes. Whether dispersion in PCA proves that COVID-19 publications are of inferior quality is a question readers will ask. Reviewer no. 2 projected this problem - is PCA the right tool to evaluate the study question?

Thank you for this insightful and generally positive review. We agree that a greater justification of our use of PCA is needed, which we provide in our revised manuscript. This is described in greater detail below in response to Reviewer #2’s specific query.

In brief, we sought to discover any differences in multiple article metrics between the 2020 COVID period and historical controls. Principal component analysis allows for the determination of the largest contributors to the variance in the data across all article metrics, in an unsupervised fashion without biasing data segregation [Jolliffe IT and Cadima J. (2016) Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A. 374, 20150202]. Importantly, in our cross-sectional study of publications in four general medical journals, COVID-related articles were significantly different from non-COVID articles based on article characteristics and impact metrics. While it is difficult to ascertain quality per se, we describe how COVID-related studies were generally shorter articles reporting observational studies with less literature cited and fewer study sites, suggestive of more limited scientific support. We discuss these findings in our Discussion section.

Response to Dr. Sherasiya, Reviewer #1

(1) This manuscript has novelty and timely to analyze the COVID-19 and non-COVID-19 related articles characteristics… Manuscript is well written and it can be published with minor corrections.

Thank you for your kind and thoughtful review. We have updated our manuscript with your suggested corrections as listed below.

(2) Line-2: There should not be period (full stop) in title.

The period in the title has been removed.

(3) Line-60: MEDLINE is the pioneer database but author should have to describe for choosing this database and excluding other databases.

Thank you for bringing up this important point of clarification. As you note, MEDLINE is a pioneering database, commonly used for bibliometric studies. We chose MEDLINE specifically to facilitate our study identification and data extraction because it is easy to use and contained entries for all publications within our four journals of interest. It was not necessary to consult other databases because our MEDLINE search captured all potential studies for inclusion.

A broader search, not limited to specific journals would have necessitated inclusion of other repositories such as Embase, Cochrane Controlled Clinical Trial Register, PubMed, and CINAHL. That, however, was not the case with our investigation.

Our revised manuscript has been updated to include a point of clarification and justification of our choice of MEDLINE over other databases as follows:

“We searched for original investigations related to SARS-CoV-2 and COVID-19 published in March and April 2020 through MEDLINE. MEDLINE alone was used because it contained entries for all publications within our four journals of interest. Accordingly, other databases were not consulted.”

(4) Line-62: Single reviewer

Thank you for alerting us to this error. In fact, two reviewers verified each study for inclusion. We have corrected this error in our revised manuscript:

“Two reviewers verified studies for inclusion and two reviewers audited extracted data.”

Response to Reviewer #2

1) Abstract is quite short and poor. no information on methods or results.

We acknowledge that the Abstract in our initial submission did not effectively convey detail on our methods and results. We have since expanded our Abstract to include a more detailed description of our methods and results in our revised submission. The revisions are as follows:

Background: The COVID-19 pandemic has yielded an unprecedented quantity of new publications, contributing to an overwhelming quantity of information and leading to the rapid dissemination of less stringently validated information. Yet, a formal analysis of how the medical literature has changed during the pandemic is lacking. In this analysis, we aimed to quantify how scientific publications changed at the outset of the COVID-19 pandemic.

Methods: We performed a cross-sectional bibliometric study of published studies in four high-impact medical journals to identify differences in the characteristics of COVID-19 related publications compared to non-pandemic related studies. Original investigations related to SARS-CoV-2 and COVID-19 published in March and April 2020 were identified and compared to non-COVID-19 research publications over the same two-month period in 2019 and 2020. Extracted data included publication characteristics, study characteristics, author characteristics, and impact metrics. Our primary measure was principal component analysis (PCA) of publication characteristics and impact metrics across groups.

Results: We identified 402 publications that met inclusion criteria: 76 were related to COVID-19; 154 and 172 were non-COVID publications over the same period in 2020 and 2019, respectively. PCA utilizing the collected bibliometric data revealed segregation of the COVID-19 literature subset from both groups of non-COVID literature (2019 and 2020). COVID-19 publications were more likely to describe prospective observational (31.6%) or case series (41.8%) studies without industry funding as compared with non-COVID articles, which were represented primarily by randomized controlled trials (32.5% and 36.6% in the non-COVID literature from 2020 and 2019, respectively).

Conclusion: In this cross-sectional study of publications in four general medical journals, COVID-related articles were significantly different from non-COVID articles based on article characteristics and impact metrics. COVID-related studies were generally shorter articles reporting observational studies with less literature cited and fewer study sites, suggestive of more limited scientific support. They nevertheless had much higher dissemination.

2) The methods are unclear, especially the role of PCA and what the outcomes are e.g. impact metrics.

We agree that our initial submission did not adequately describe our study’s full methodology. As described below, our revised manuscript now includes an expanded Methods section that specifically describes the role and rationale of PCA, as well as delineates each of our impact metrics.

In our study, we sought to discover any differences in multiple article metrics between the 2020 COVID period and historical controls. Principal component analysis allowed us to determine the largest contributors to the variance in the data across all article metrics, in an unsupervised fashion without biasing data segregation (Jolliffe, I. T., and Cadima, J. (2016) Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A. 374, 20150202). We found the most important features that capture the maximum information about the dataset, reducing dimensionality without any significant loss of information. Since the principal components are independent, we eliminate correlated features, important in our data set as there was a chance that quality metrics could be tied. Were this the case, validating a lack of correlation would have been challenging and time consuming for this dataset using traditional statistical methods. Additional advantages to PCA are that it reduces our chance of overfitting the data and overemphasizing the change in certain article metrics. In these ways, PCA was well suited for the type of data and analysis we were performing, revealing distinct differences between the article categories without prior assumptions or bias while still considering all collected metrics. We have tried other unsupervised methods (i.e. partial least square discriminant analysis and sparse PLA-DA) and found similar results. Lastly, PCA has been used in other similar studies (e.g. Bollen, J., Van de Sompel, H., Hagberg, A., and Chute, R. (2009) A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE. 4, e6022).

We have also clarified the impact metrics extracted for analysis: number of reads, number of citations, and number of tweets.

The relevant additions to our revised manuscript are as follows:

“Our primary measure was principal component analysis (PCA) of publication characteristics and impact metrics across groups. In our study, we sought to discover any differences in multiple article metrics between the 2020 COVID period and historical controls. Principal component analysis allows for the determination of the largest contributors to the variance in the data across all article metrics, in an unsupervised fashion without biasing data segregation [Jolliffe IT and Cadima J. (2016) Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A. 374, 20150202]. Comparisons between groups were conducted using Chi-square or Fisher’s exact tests for proportions and non-parametric Kruskal-Wallis tests with Dunn’s multiple comparison for continuous data. Data for each journal were aggregated for analysis.”

“Extracted data included publication characteristics, study characteristics, author characteristics, and impact metrics. Impact metrics (numbers of reads, citations, and tweets) were not normalized to the time since publication.”

3) Why don't the authors use established quality assessment tools for observational studies and report that instead?

This is an excellent suggestion that we had discussed when first designing our study. Specifically, we had considered using established checklists to assess for quality of publication. However, we chose not to include these tools as we were not assessing only observational studies. The resulting collection of studies that met our inclusion criteria were heterogeneous in nature (i.e. by study design) without the necessary sample sizes per type of study to perform a comparison within specific study design groups. Moreover, there is no validated method for comparing between different types of quality assessment tools, which would have been required given the different types of study designs within our inclusion criteria.

4) I don't follow why the data were aggregated within each journal and each study was not included as a different case.

Thank you for bringing attention to this important point of clarification. We did consider separating analyses by journal as well. Given that our included journals are all high-impact, general medical journals, we felt that further individual comparisons would not provide meaningful information when our primary objective was to look at overall changes in publication approach among all high-impact, general medical journals between the COVID and non-COVID literature. The main goal was to look at scientific publications in general, and not focus on any specific journal. Additionally, the small numbers of publications per journal adds a further layer of multiple comparisons that our study is not appropriately powered for.

Nevertheless, we agree that our readership may be interested in the data at the individual journal level. As a result, we have included a supplemental table with the data segregated by each journal (see S1 Table). Moreover, our submission provides our entire dataset (available in S1 Dataset). The interested reader is therefore able to undertake a specific analysis of the data segregated by journal at their discretion.

We have clarified this in the Methods and Results sections:

“We identified 402 publications that met inclusion criteria: 76 were related to COVID-19; 154 and 172 were non-COVID publications over the same period in 2020 and 2019, respectively (data available in S1 Dataset).”

“To further evaluate how the published COVID-19 research literature differed from non-COVID-19 investigations, we first compared their publication characteristics (Table 1). Publication characteristics segregated by individual journal are provided in the Table in S1 Table.”

5) Rstudio is the shell, reference what's under the hood i.e. R version.

Thank you for identifying this oversight and potential source of confusion. The R version is included in our revised manuscript:

“Analyses were performed using GraphPad PRISM software version 7.0 and RStudio version 1.3.1056.”

Journal Requirements: When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming.

Our revised manuscript now meets the PLOS ONE style requirements, including those for file naming.

2. Please include your amended [funding] statements within your cover letter; we will change the online submission form on your behalf.

Thank you for identifying this oversight. Funding-related text has been removed from the manuscript.

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly… We will update your Data Availability statement on your behalf to reflect the information you provide.

Thank you for this reminder about data reporting. As outlined in our revised cover letter, there are no restriction on sharing our dataset. Accordingly, we have uploaded our dataset as a supplementary dataset (S1). We have added a reference to this supplementary information in the Results section:

“We identified 402 publications that met inclusion criteria: 76 were related to COVID-19; 154 and 172 were non-COVID publications over the same period in 2020 and 2019, respectively (data available in S1 Dataset).”

Attachment

Submitted filename: Gai et al - Response to reviewers.docx

Click here for additional data file. (25.1KB, docx)

Decision Letter 1

Itamar Ashkenazi

20 Jan 2021

General Medical Publications During COVID-19 Show Increased Dissemination Despite Lower Validation.

PONE-D-20-29109R1

Dear Dr. Steinberg,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Both one of the reviewers and I had some concerns with ACP being used as a major evaluation tool towards your endpoint.  However with the help of the two reviewers I have decided that it is up to the intelligent reader to decide whether ACP improves the analysis or not.  As you mention, the descriptive analysis provide sufficient information.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Itamar Ashkenazi

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: You attended all the comments satisfactorily.

Reviewer #2: I am still not entirely convinced why the authors did not used established quality tools, and analyse by groups. That would make the analysis more valuable.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Anjum Sherasiya

Reviewer #2: No

Acceptance letter

Itamar Ashkenazi

22 Jan 2021

PONE-D-20-29109R1

General Medical Publications During COVID-19 Show Increased Dissemination Despite Lower Validation

Dear Dr. Steinberg:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Itamar Ashkenazi

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Dataset. The dataset used for the analyses in this study.

    (XLSX)

    Click here for additional data file. (62.4KB, xlsx)
    S1 Table. Publication characteristics and impact by journal.

    (DOCX)

    Click here for additional data file. (25.4KB, docx)
    Attachment

    Submitted filename: Gai et al - Response to reviewers.docx

    Click here for additional data file. (25.1KB, docx)

    Data Availability Statement

    All relevant data are within the paper and its Supporting Information files.

    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES