Vigilante Science

The founders of PubPeer (Brandon Stell, Richard Smith, and George Smith) came out this past month.¹ Some Plant Physiology readers will be familiar with the PubPeer Web site, but for others, this editorial may be the first you will have heard of this by-product of the social media age. Since its launch in October 2012, PubPeer has sought to facilitate community-wide, postpublication critique of scientific articles. The Web site has also attracted much controversy around its policies and, until recently, the secrecy shrouding its founders.

PubPeer operates as a blog on which anyone can post comments, either to a published article or to comments posted by other participants, and authors may respond. It is a bit like an extended journal club; not a bad idea to promote communication among scientists, you might think, so why the controversy?

The problems arising are twofold, and their roots are self-evident on a quick trawl through the PubPeer site. First, most individuals posting on PubPeer—let’s use the euphemism commenters for now—take advantage of the anonymity afforded by the site in full knowledge that their posts will be available to the public at large. Second, the vast majority of comments that are posted focus on image data (gels, blots, and micrographs) that contribute to the development of scientific ideas but are not ideas in themselves. With few exceptions, commenters on PubPeer do no more than flag perceived faults and query the associated content. Of course, such detail generally informs discussion, but no journal club I ever organized or contributed to was so obsessed with the minutiae of data presentation.

From the perspective of an author (and, I suspect, many readers), missing all too often are courtesy and common sense. In any other setting, the majority of comments on PubPeer would be reserved for quiet discussion, perhaps by e-mail or after a seminar when the commenter might draw an author aside for questioning. This discussion would benefit both the commenter and the author. It would avoid any possible awkwardness on either side, and if the discussion were constructive, then both author and commenter would profit from the outcome. The majority of posts on PubPeer are mounted anonymously. So, while there is no danger of public embarrassment for the commenter, likewise there is no opportunity to gain from a personal exchange with the author. What is the rationale? Given that the majority of comments show the most petty kind of scientific criticism, can there be any doubt that the intent often is to pillory, to do so publicly and without accountability?

I would like to think that this was not the original intent when PubPeer was launched in 2012. Its founders present the site as a forum “to accelerate the exchange of ideas and scientific progress.”² True, the site does allow for postpublication discussion, but I suggest all too often these discussions are one-sided from the outset simply because PubPeer commenters and moderators remain hidden from the scientific community at large. No doubt, Stell and his colleagues anticipated the nature of much of the content when they wrote that “questioning the work of other researchers can have negative consequences for a young scientist’s career.”² Anyone with experience of online forums, regardless of the context, will not be surprised that the overwhelming majority of posts on PubPeer are negative and occasionally malicious. The very language in the guidance for would-be commenters on PubPeer is tendentious, if not confrontational: “What none of us can verify is any conclusion regarding precisely how or why an apparent instance of misconduct occurred,” so “there is absolutely no need to spell out any suspicions you may have … [as your] readers will be able to draw their own conclusion without the slightest difficulty.”² Often enough, posts take on a triumphal tone, even when couched in terms of a question, as if the commenters were already reveling in the discomfiture of the authors. Certainly, there can be little room for worthwhile discussion or an “exchange of ideas” when the discussion is devoid of ideas and polarized from the outset.

My overriding concern with PubPeer is the lack of transparency that arises from concealing the identities of both commenters and moderators. I accept that there is a case for anonymity as part of the peer-review process. However, the argument for anonymity in postpublication discussion fallaciously equates such discussion with prepublication peer review. The essence of prepublication review is a mutual agreement, a social contract between author and editor, who are known to one another. The author submitting a manuscript for review agrees to accept the judgment of the editor, and the editor agrees to judge the worthiness of the manuscript for publication. To make this assessment, the editor may obtain independent reviews of the author’s work before passing judgment, but (and here is the fundamental flaw in the argument for “postpublication peer review” as championed by PubPeer) the reviewers and their anonymity are secondary to the contract between author and editor. In short, anonymity makes sense when reviews are offered in confidence to be assessed and moderated by an editor, someone whose identity is known and who takes responsibility for the decision informed by the reviews. Obviously, this same situation does not apply postpublication, not when the commenters enter into a discussion anonymously and the moderators are also unknown.

I would argue, too, that the consequences of anonymity are more far reaching than PubPeer is prepared to concede, even if comments are factual, scientific, and civil. I concur with Hilda Bastian,³ who notes, on the one hand, the lack of reliable evidence to support the benefits of reviewer anonymity and, on the other, the importance of assessing whether commenters are “outside their areas of expertise … [or] have conflicts of interest.” Anonymity can conceal much mischief and do great damage. Even PubPeer acknowledges that “anonymity does allow low quality and bad faith comments to be made with impunity” but argues “that it is a necessary price to pay to encourage frank and worthwhile discussion.”⁴ In my opinion, anonymity is intimidating in itself, regardless of how polite a commenter may be; psychologically, it wrong-foots the author from the start, and the price often is an absence of worthwhile discussion, if there is any discussion at all.

Let’s not mince words. The self-assumed role of PubPeer is about policing, not discussion. In this context, it has seen some success. Over the past three years, the Web site has helped publicize a handful of instances in which scientific misconduct was alleged, notably the widely publicized Voinnet case and the Sarkar case, which led to a defamation suit and continues to rumble through the courts. These and a number of other allegations of misconduct have increasingly gained the media spotlight over the past decade. The attention reflects a corresponding rise in the numbers of articles retracted across the biomedical sciences and, undoubtedly, an increasing awareness of fraud. This attention, however, does not necessarily equate with an epidemic of scientific malpractice, as intimated by PubPeer and often embellished by the media. The Rockefeller University Press maintains the longest record of detailed image data analysis, having forensically examined the images of all articles accepted for publication in the Journal of Cell Biology, the Journal of Experimental Medicine, and the Journal of General Physiology since 2002.⁵ These records show that a surprisingly high percentage of figures in the articles—some 15% to 25% year on year, on average one figure per article—were queried with the authors postreview; however, the percentage of articles for which acceptance was subsequently withdrawn has remained almost unchanged for more than a decade at around 1%. This percentage is largely consistent with three independent and detailed studies of the incidence of misconduct, its reporting, and its costs that go back over two decades (Fanelli, 2009; Fang et al., 2012; Stern et al., 2014; see also Retraction Watch⁶). Each suggests that the rates of scientific misconduct have remained close to 1% to 2% of all articles published.

What of the bulk of comments posted on PubPeer? These relate to small errors and oversights, not the stuff of misconduct nor likely to arouse any but the most obsessive of temperaments. Still, it is not difficult to imagine the frisson of excitement either in posting on PubPeer or trawling through its listings. There is something primal and voyeuristic in picking through someone else’s closet in the hope of uncovering his or her dirty laundry. PubPeer is unquestionably an opportunity to vent spleen at the imperfections of colleagues. For some, it is also an invitation to undermine competitors with innuendo and the implicit threat of uncovering something perfidious. (I shall not be surprised if this editorial triggers a wave of interest on PubPeer in my own publications.) Certainly, there is little incentive here for an author to respond. PubPeer asserts “currently, most authors keep their heads firmly in the sand, maybe because they have no real answer to the criticisms posted.”² I suggest that many authors do not respond simply because they fail to see any real value in engaging in such exchange, not in a venue of this kind.

PubPeer takes an altogether more sinister tone, however, in its self-proclaimed authority to represent the scientific community and give “referees and members of committees for recruitment, promotion or funding … [the community’s] opinions about the quality and reliability of applicants’ research.”² Legitimate authority demands consensual recognition and identity, both currently lacking for PubPeer. As scientists, we recognize the authority that comes with knowledge and expertise. We expect the identities of those who wield authority to be in the public domain. These are the reasons why funding bodies—including the National Science Foundation and the National Institutes of Health in the United States and the research councils in the United Kingdom, for example—publish the names of the scientists who sit on committees to review grant applications. These are also the reasons why journals maintain an editorial board of peers and publish their names and qualifications on the journals’ mastheads. The editors are individuals upon whose expertise and knowledge a journal depends, who moderate the review process, and who uphold the name of the journal. They have a vested interest in ensuring a journal’s quality and content. Far from the claim of PubPeer that “it is not in the interest of a journal to see its publications (and implicitly its editorial policy) criticised,”² scientific journals and their editors take valid criticism very seriously indeed and will not suffer scientific misconduct. Virtually all scientific journals subscribe to the COPE guidelines,⁷ and are sensitive to legitimate criticism.

The editors of Plant Physiology, for one, consider all incoming correspondence. We encourage concerns to be directed to the corresponding authors in the first instance but, when asked, will relay valid questions about an article to the authors, especially if it is important to protect individual identities. The journal has always acted on all allegations of misconduct when properly documented and will continue to do so. We also foster education as it relates to image data and its handling (Blatt and Martin, 2013). However, for the obvious reasons noted above, no reputable journal, including Plant Physiology, engages in scientific hearsay or anonymous allegations, or publishes unmoderated critiques and comments.

I agree with PubPeer on one account: the peer-review system will never catch all errors, innocent or otherwise. The same is sure to prove true of PubPeer itself. Nonetheless, good science does stand the test of time. As a process, scientific research is self correcting: the content of no scientific article is written in stone but is invariably subject to further analysis. In large measure, it is the predictions arising from a study that form the basis of this self correction, and their testing either validates, discounts, or refines each hypothesis and insight that a study offers. So, whatever the shortfalls of the peer-review process, I do not accept the argument that it is failing, that it is a threat to progress, or that, as scientists, we need to “retake control of our profession.”⁴ Indeed, if there is a threat to the scientific process, I would argue that, unchecked, the most serious is the brand of vigilante science currently facilitated by PubPeer.

Social media can, and does, provide opportunities for informal postpublication discussions, and it does not require anonymity to be effective. As an example, the 2010 Science article about a microbe that purportedly used arsenic in lieu of phosphate was widely discussed, and ultimately discounted, by scientists sharing their thoughts (and identities) online.⁸ This discussion took place long before the advent of PubPeer. The National Center for Biotechnology Information launched PubMed Commons in 2013⁹ with this same idea in mind; in the interest of “open public criticism and discussion,” PubMed Commons does not permit anonymous or pseudonymous comments. Plant Physiology, too, entertains constructive dissent between opposing viewpoints, and I point readers to the excellent discussion by Robinson et al. (2015) as a recent example.

I agree that there should be a place for independent, postpublication discussion that is linked to individual articles. Such discussion may include an element of quality control, and it need not be tied to the journal that originally published the article. However, I believe that open discussion requires that contributors identify themselves or that, at the very least, such discussion is moderated and validated by a publicly identified and recognized body of peers who assume responsibility for the discussion to ensure transparency. These are no more than the standards expected of reportage by any respected newspaper. The challenge, therefore, is for PubPeer (1) to identify not just its founders but its moderators who ensure scientific standards, their backgrounds, and expertise; (2) to institute rigorous and transparent moderating procedures; and (3) to provide the facility to mark all relevant comments as closed once a discussion is complete and any issues are resolved. Until then, I urge scientists publishing in Plant Physiology and other reputable scientific journals not to respond to comments or allegations on PubPeer, and I encourage would-be commenters to communicate with the authors directly, via the acknowledged routes provided by these publications, or using sites such as PubMed Commons on which contributors identify themselves.

Finally, you may wonder why, after three years of secrecy, the PubPeer founders chose this moment to identify themselves. The answer is as old as the hills: money. PubPeer is now seeking philanthropic support as a nonprofit organization, a status that has required Stell and his colleagues to identify themselves. It seems that the dollar has the power to persuade where other arguments failed. Stell hopes to improve and expand the Web site, but he admits that some funds could be used for legal assistance.¹ Let’s hope that he and his colleagues put their efforts into providing rigorous scientific oversight and transparency; such efforts would promote PubPeer in the eyes of the scientific community and are sure to reduce their litigation costs in the long run.