Is the validity, credibility, and reliability of literature indexed in PubMed at risk?

Dear Editor,

PubMed, a public database of primarily biomedical science with over 30 million citations, is maintained by the United States National Library of Medicine (NLM) of the National Center for Biotechnology Information in the National Institutes of Health. Access to content in MEDLINE, a PubMed database, is provided through companies like Elsevier's Embase or EBSCO while free open access content is available through PubMed Central (PMC). Both PubMed and MEDLINE are registered trademarks (i.e., PubMed® and MEDLINE®). PubMed is a very valuable reference tool for academics who seek a reliable source of mainly biomedical information, and is generally considered to be a reputable and trustworthy database of legitimate scholarly literature.¹ However, 92% of PubMed records indexed in MEDLINE in 2008 decreased to 20% in 2017, those values being 89% and 18%, respectively for PMC.² Do these trends suggest anything about the quality of records at PubMed?

Nursing journals indexed in PubMed include citations to papers published in unscholarly venues or potentially predatory publications, but the absolute number is low, relative to Google or Google Scholar.³ However, numbers are not zero, i.e., potentially unscholarly literature is also being indexed in PubMed, even if it may be a small fraction, for now. There is additional evidence of the inclusion of literature in PubMed from potentially predatory or unscholarly sources. Manca et al. claimed that 23.7%, 16.1%, and 24.7% of rehabilitation, neuroscience, and neurology journals blacklisted by Jeffrey Beall were indexed in PubMed,⁴ suggesting that screening procedures at PubMed might not be effective enough to detect or differentiate legitimate scholarly work from potentially unscholarly work. Readers should be aware that blacklists, like those by Beall, might include false-positive entries, i.e., journals that are labeled as “predatory,” but which, in fact, are not, or that scholarly work might also be published in journals that have been classified as such, making the analyses and conclusions drawn by studies such as those by Oermann et al. and Manca et al potentially erroneous.⁵

Truly predatory journals undergo none or weak peer review, publish worthless science, seek primarily monetary rewards, desire to populate reputable databases like PubMed, and overall threaten the integrity of medical science. The greater concern with such unscholarly or predatory journals is if public funding has been used to pay for the publication of papers in such journals through article processing charges for open access.⁶ Even so, the merits and demerits of a paper should be measured individually rather than judged by its publication venue or indexing.

Biomedical researchers and other academics rely on the efficiency of PubMed management to make sound judgments in their selection of journals for inclusion on that database. This is because, based on the listing of a journal on PubMed, researchers are likely more inclined to select such journals for the submission of their work, confident that those journals have been stringently vetted prior to inclusion on PubMed. They are also more likely to cite papers from PubMed-indexed journals in the belief that they are scholarly. Consequently, to retain public trust and confidence, PubMed should provide open and transparent reasons and selection criteria for inclusion and exclusion of all journals and consistently strive to raise the bar for inclusion. In fact, PubMed/PMC representatives gave public assurances of the stringency of selection and inclusion criteria for MEDLINE, PubMed, and/or PMC, claiming that the NLM “is committed to the integrity of its literature databases and continues to develop its selection processes in response to changes in the scholarly publishing environment.”⁷ If that is true, then how is it possible that several journals (e.g., Journal of the Balkan Union of Oncology, Medical Science Monitor, and Archives of Medical Science, and European Review for Medical and Pharmacological Sciences) with extreme cases of manipulated images and, more worrisomely, “stealth” corrections in the first listed journal in which the content of the record (PDF files) has been manipulated to hide the evidence of potentially unethical behavior, have been indexed in PubMed?⁸ Several of those papers appear to have derived from one or more “paper mills,”⁹ which have also been recently associated with several Chinese hospitals.¹⁰

PubMed (MEDLINE/PMC) is likely one of the most important databases related to biomedical research, including COVID-19. Information in papers or journals that are indexed in PubMed needs to be rigorously screened and vetted prior to inclusion, and a mechanism to expunge unscholarly work from that database is required. Comprehensive analyses of the validity and scholarly nature of the literature that is indexed in PubMed are urgently needed. Failure to do so may erode trust in this public database.