On peer review—then, now, and soon to be?

I’m not by nature a revolutionary, but, when I became editor-in-chief of PNAS, I found myself in the metaphorical equivalent of a center mezzanine seat for an ongoing revolution, albeit a slow-moving and (mostly) bloodless one. This revolution is the digital transformation of scientific publishing, which builds on the remarkable capabilities of digital communication, text processing, and Web technology to enrich scholarly research articles with data, metadata, and interactive access efficiently and affordably (1). The revolution made one of several recent incremental leaps forward on August 25, 2022, when the White House Office of Science and Technology Policy, under Acting Director Alondra Nelson, issued a memo titled Ensuring Free Immediate & Equitable Access to Federally Funded Research, now widely known as the “Nelson Memo.”

May R. Berenbaum.

This memorandum exhorts agencies to “update or develop new public access plans for ensuring…that all peer-reviewed scholarly publications⁴ authored or coauthored by individuals or institutions resulting from federally funded research are made freely available and publicly accessible by default in agency-designated repositories without any embargo or delay after publication” by December 31, 2025 (2). Footnote 4 explains that “Such scholarly publications always include peer-reviewed research articles or final manuscripts published in scholarly journals and may include peer-reviewed book chapters, editorials, and peer-reviewed conference proceedings published in other scholarly outlets that result from federally funded research.” The singling-out of peer-reviewed scholarship, in both the text and a footnote, is unsurprising given the fact that peer review has been widely considered to be a form of quality assurance, yet today, it is the subject of considerable debate, given its (also widely recognized) shortcomings.

A Laborious and Difficult Method

Peer review has a long history, dating back to the very early days of scientific publishing, and its roles have been redefined many times, depending on scientific and social priorities. The first periodical focused on science, Philosophical Transactions of the Royal Society, began publishing reports of research studies on a regular basis in 1665, but the Royal Society of London took nearly a century before adopting a system for peer review. In 1752, the Royal Society established a “committee on Papers” to review manuscripts before publication, following the lead of the Royal Society of Edinburgh, which in 1731 established a policy whereby “Memoirs sent by correspondence are distributed according to the subject matter to those members who are most versed in these matters. The report of their identity is not known to the author” (3).

Although the concept of peer review was in place in the early days of scientific publishing, it was slow to catch on. Among other things, as Ernest Hart, the editor of the British Medical Journal, remarked in 1893, “it is a laborious and difficult method, involving heavy daily correspondence and constant vigilance to guard against personal eccentricity or prejudice, or—that bugbear of journalism—unjustifiable censure” (3). In fact, it wasn’t widely adopted until after World War II, when the rapidly growing scientific community was producing so many manuscripts that journal editorial boards couldn’t keep up with submissions and thus brought in colleagues, “peer reviewers,” to assist with identifying those most deserving of publishing. Peer review—ostensibly the assessment of scientific quality by scientists—came to be the ultimate imprimatur of quality, particularly from the perspective of the non-scientist public.

Today, its future status is uncertain. Some are calling for strengthening the process to guard against ways that it has been twisted, e.g., by predatory journals (4). Others are calling for alternative models to reinvent how it’s practiced, most notably by moving the process from before a manuscript is made accessible to readers (pre-publication peer review) to after it is made accessible to readers (post-publication peer review). Finally, a few others are calling for abolishing it altogether (5). Despite the diversity of viewpoints, it’s safe to say that no one is entirely happy with the process as it’s practiced right now.

A Change to the Status Quo

The challenge is to figure out how to fix it. The Nelson memo states unambiguously that “There should be no delay between taxpayers and the returns on their investments in research,” a charge that creates a dilemma for researchers and provides a reminder that, during a revolution, the goals of all revolutionaries may not necessarily align. The word “taxpayer” does not appear in the 2013 Holdren memorandum “Increasing Access to the Results of Federally Funded Scientific Research,” issued by former OSTP Director John Holdren, the first formal direction of any kind to federal agencies to develop plans for increasing public access to funded research (6). The Nelson memorandum, deliberately or not, explicitly broadens the community of readers to include the ultimate funders—the taxpayers who foot the bill for federal research. It’s incumbent, then, upon the scientific community, to consider what “peer review” has meant for decades to the nonscientist taxpaying public and, for that matter, to journalists and policymakers—that is, review by experts prior to what’s considered “publication,” i.e., inclusion in a journal. Even the most critical of peer review recognizes that prepublication peer review has provided at least a minimal “guarantee to interested but non-expert parties. Science journalists, policy-makers, scientists from outside the field the manuscript is aimed at, or interested non-scientists can take the fact that something has passed peer review as a stamp of approval from the field…[and] having a social mechanism to provide this guarantee for outsiders is useful” (5).

Among the many innovations under discussion (and currently being implemented in some places) is postpublication peer review, which takes many forms, but all these forms require publishing manuscripts online by posting as a preprint prior to peer review (7). After posting (i.e., “publication” as newly defined), review can be carried out by individuals invited by journals or by volunteers from the broad reader community, anonymously or otherwise. Proponents of postpublication peer review argue that it can reduce or eliminate some of the most egregious problems of prepublication peer review—i.e., it’s slow, time-consuming, elitist, and ineffective at detecting fraud, among other things. From outside the scientific community, however, a proliferation of potentially widely diverging comments from multiple sources is likely to create confusion about the authoritative nature of the publication. Moreover, postpublication peer review is inherently “a never-ending process” that is “never finished” by definition (8). While the scientific community clearly benefits from postpublication peer review in that it is a more accurate reflection of the process by which science advances, the fact that the process is never finished likely will create problems for students, early career researchers, journalists, and nonscientist readers. Under the “publish-review-curate” model as practiced by several journals, the authoritative nature of a publication can be established with journal curation as part of their reinvented mission, to provide a form of imprimatur, yet it’s unclear how long one must wait between first finding a publication on a preprint server and ultimately seeing any kind of authoritative scientific “stamp of approval.”

Hinchliffe (9) recently reported on research into attitudes toward preprints, noting that, for teaching purposes, students, whose experience with the scholarly literature is limited, have trouble understanding where preprints fit in the “scholarly communications ecosystem,” whereas the version of record is “an easier format for them to navigate.” Similarly, Cataldo et al. (10) evaluated assessments of preprints by 116 students (at high school, community college, undergraduate, and graduate levels) with respect to helpfulness, credibility, and status as a preprint and found that “preprint” was not a familiar term and that preprints were not generally recognized as a form of publication different from journal publication. Although preprint and peer-reviewed status did not influence assessments of helpfulness or citability per se by these students, “peer review” did influence assessments of credibility.

Unfettered Access and an Infodemic

Presumably, the general public, with even less knowledge of (or interest in) the complexities of the scholarly literature than science students, may be even less well equipped to sort through preprints. For much of the recent past, in the scientific community, the prepublication peer-reviewed version is the one that is privileged in editorial style guides, prioritized by granting agencies, and utilized as the focus for recognition and academic advancement. This is not to say, however, that prepublication peer review as practiced today deserves its privileged status as a signifier of credibility.

There is increasing justification for clarifying the process of manuscript review given the escalating interest in scientific publishing on the part of the nonacademic public, as demonstrated by twenty years of Altmetrics scores (which today, like many other journals, PNAS provides for its published research reports). These scores offer quantitative evidence of interest in scientific publishing well beyond the academic community.

Perhaps the most dramatic increase in public interest in the scientific literature in the recent past (or, arguably, ever) arose in concert with the exponential growth in COVID-19 cases in 2020, characterized as an “astounding thirst for knowledge” (11). Scientists responded by sharing their findings about COVID-19 with unprecedented speed, often circumventing the slow prepublication review process by posting preprints. By the end of 2020, the number of COVID-19-related publications exceeded by some calculations 100,000, including 30,000 preprints; almost one-third of readers of open-access articles were nonacademic (12). The response to the global pandemic has been cited as evidence that “expensive, restrictive scientific journals” may “on balance serve no useful purpose” (13).

Accompanying the explosion of literature related to COVID-19 was the (flawed) conviction that, with so much information not just available to them but often explained at length by journals in collections, guides, and informative accompanying articles, a well-informed public should have been equipped to embrace the overwhelming scientific consensus about the disease, its origins, its spread, and its treatments. Instead, particularly in the United States, despite unprecedentedly unfettered access to the scientific literature, the country has become mired in an “infodemic”—an information overload, characterized by overlapping threads of misinformation, conspiracy theories, and widespread rejection of the emerging scientific consensus. Many of these controversies thrive on Twitter and other social media, with just about every aspect of the scientific community “consensus” in dispute. The perception that knowledge deficits alone are to blame for the infodemic ignores the well-established social science finding that public attitudes and behavior are shaped not just by having more information but by cognitive biases, science-society interfaces, and deeply ingrained value systems (14), often leading to a rejection of science whether it’s peer-reviewed or not.

What’s unusual about the COVID-19 pandemic is not so much that there have been conspiracy theories and rejection of conventional science by the nonscientist public (both of which have been traveling companions of the scientific enterprise since its beginnings); rather, what makes this “infodemic” distinctive is that it’s the first during which the public has had direct access to much of the science itself. For decades, the scientific community has relentlessly insisted that “peer” review is the “gold standard,” but the scientific community has historically provided little guidance to the public on differentiating not only between preprints and conventional journal articles but also between peer review as conducted by “reputable” journals and peer review as conducted by the rapidly increasing ranks of predatory journals or as exploited by opportunistic publishers (and, to some extent, researchers) willing to take advantage of its weaknesses (precious little guidance, for that matter, even to early career researchers).

The Certainty of Uncertainty

In other words, we scientists have done a poor job of explaining what we’re doing when we write and review manuscripts, at least in part because much of what we’re doing has been hidden. Yes, peer review has many flaws—as in 1893, it’s “laborious and difficult,” subject to biases, inconsistent, difficult to replicate, and inefficient at blocking publication of dubious science. Notwithstanding, if the scholarly community decides to move to new models of publication that include publication status for preprints and postpublication peer review, it’s critical to provide guidance for the public to use its access to greatest effect (15). Irrespective of its flaws, prepublication peer review has gained credibility as a form of fixation in time. By its very nature, scientific “consensus” is constantly changing, which is intrinsic to the process of hypothesis testing and an attribute that makes science more dependable in the long run. The nonscientific public, however, is generally uncomfortable with uncertainty (16). Ever-changing conclusions by scientists engender perceptions of uncertainties that can be frustrating for members of the public unaware or unappreciative of the fine details of hypothesis testing, and thus, absent context and clarification, risk eroding public trust in research (17). Taking away even flimsy guardrails without replacement won’t have much effect on scientists who are as a community directly involved in changing policies and experimenting with new models, but the public risks being left without the tools needed to sort through the chaos.

In fixing peer review, it’s imperative to recognize its importance (at least for the short term) as a sign or signifier for nonscientists. For the scientific community to help its public readership sort through thousands of scientific papers with thousands of postpublication peer reviews, there must be an effort to involve the public in discussions about what “publications” represent. Cataldo et al. (10) concluded from their study that there is a need for “information literacy instruction” and guidelines for the use of preprints and, in view of student assessments of preprints, “a more collaborative approach is needed to make preprints more recognizable and understood.” At least in the United States, the nonscientist public is now our audience along with our scientific colleagues. We owe it to the public at large, most of whom are not scientists, to consider their needs as the revolution continues.