Reviewing post-publication peer review

Abstract

Post-publication peer review (PPPR) is transforming how the life sciences community evaluates published manuscripts and data. Unsurprisingly, however, PPPR is experiencing growing pains, and some elements of the process distinct from standard pre-publication review remain controversial. I discuss the rapid evolution of PPPR, its impact, and the challenges associated with it.

The rise of PPPR in the life sciences

PPPR is having a rapidly increasing impact on science. Rigorous post-publication assessment of papers is crucial for the filtering and potential integration of meritorious data into the scientific collective. It is also faster than traditional forms of evaluation. Despite this, adoption of PPPR has been relatively slow in the life sciences. As early as 2007 Todd Gibson suggested that post-publication review could be helpful [1], but it did not really catch on until recently. It now shows every sign of continuing to have a major influence on the life sciences.

This rapid growth in PPPR has been made possible by several key factors. First, although cultural acceptance within the life science community of PPPR had consistently been rather minimal for decades, it has grown substantially in the past few years, largely due to the broader, perhaps generational shift towards the Internet culture. Second, PPPR is also gaining traction because of the wider availability of popular web platforms where the review can readily take place, such as Faculty of 1000 (F1000), ResearchGate, and PubPeer, as well as blogs (Table 1). The US National Institutes of Health (NIH) is even getting into the act. PubMed Commons now allows and even encourages comments on any article in the database. Sometimes PPPR even happens in real time on social media platforms such as Twitter. Websites that are wholly or in part dedicated to PPPR are popular and influential, as evidenced by their relatively high ranking on the web, which is often similar to or higher than that of journal websites (Table 1).

Table 1. Ranking and influence of PPPR sites and blogs.

	Started	MozRanka	Link	Notes
F1000	2002	5.958	http://www.f1000.com	Early adopter, focused on positive reviews
Tree of Life Blog	2005	5.407	http://phylogenomics.blogspot.com/	Jonathan Eisen blog, some PPPR
RRResearch	2006	5.01	http://rrresearch.fieldofscience.com/	Rosie Redfield blog, debunked arsenic life
ResearchGate	2008	6.387	http://www.researchgate.net/	Community focused, non-anonymous
Wiring the Brain	2009	5.008	http://www.wiringthebrain.com/	Kevin Mitchell brain research-focused blog
Knoepfler Blog	2010	5.261	http://www.ipscell.com	Author's blog
PubPeer	2012	4.601	http://www.pubpeer.com	Largely anonymous post-publication review site
PubMed Commons	2013	6.718	http://www.ncbi.nlm.nih.gov/pubmedcommons/	NIH moderated venue for post-publication comments
bioRxiv	2014	5.102	http://biorxiv.org/	Pre-print server that includes PPPR
Trends in Genetics	-	4.52	http://www.cell.com/trends/genetics/home	Example reference site for MozRank

^aThe MozRank tool is an indicator of online authority and popularity in which higher numbers reflect relatively higher predicted impact. MozRank data shown are from February 2015.

Together, these factors have shifted laboratory journal club type discussions of new papers out of the confines of conference rooms into the public domain online where commentary can be rapidly disseminated and discussions with any interested individual can be facilitated. Although a quantitative assessment of the influence of this invigorated post-publication review in the life sciences is currently difficult [2], direct observations ‘in the field’ of the phenomenon suggest a strongly growing influence. For example, numerous article retractions and corrections have been catalyzed by PPPR, attracting the attention of journal editors, and some authors are directly responding to criticisms in the same online platforms in the public domain.

Fast and furious?

In the stem cell field there has been significant debate over so-called ‘ground state pluripotency’ of human cells and the role of the factor MBD3 (methyl-CpG-binding domain protein 3) in cellular reprogramming to make induced pluripo-tent stem cells (IPSC). Surprisingly, much of that debate has played out on PubPeer (dubbed the ‘stem cell shoot out’ https://pubpeer.com/topics/1/2B2B490DD36C55707411830470926D), as well as on bioRxiv, a preprint server for biology, where PPPR is occurring as well. The two main scientists involved in this debate, Jose Silva and Jacob Hanna, are engaged in an almost real-time, public PPPR and scientific interaction (http://biorxiv.org/content/early/2015/01/16/013904) that seems unprecedented in biology. Hanna has even publicly addressed specific criticisms of his papers and as a result submitted corrections to journals (https://pubpeer.com/publications/C278F3DE939616C4ADBDB9C15DB268#fb21519) only weeks or months after the issues were first raised via PPPR, demonstrating the extraordinary speed at which this process can catalyze concrete outcomes.

Another illustrative recent example of problematic issues in science being resolved strikingly fast largely via PPPR also comes from the stem cell arena in the form of the stimulus-triggered acquisition of pluripotency (STAP) cell case. In late January 2014, two papers on so-called STAP cells were published in Nature reporting a seemingly too good to be true method of cellular reprogramming [3,4]. On PubPeer and other sites, including my own blog, the STAP story quickly started to unravel, ultimately leading to the retraction of those papers and correction of the scientific record with an unprecedented rapidity of only a few months (Table 1) [5,6]. If the STAP cell papers had been published 5 or 10 years ago, I believe it would have taken several years for the record to be corrected. In the meantime valuable resources would have been squandered on STAP and trainee careers redirected to work on STAP could have been in serious jeopardy. Fortunately that did not happen, and I believe that PPPR deserves much of the credit.

Certainly, problematic life science and a corrective role for PPPR are not limited to the stem cell field. Another valuable, earlier example is the ‘arsenic life’ story. Scientist Felisa Wolfe-Simon at the US National Aeronautics and Space Administration (NASA) led a team reporting that they had found a microorganism that could live on arsenic instead of phosphorous. The work was eventually published in Science in 2011 [7]. Both in PPPR on her blog (Table 1) and in traditional publication format [8], Rosie Redfield debunked the arsenic life story in a rapid manner that limited the negative fallout from the flawed science. Even so, it is notable that the original arsenic life paper in Science has to date not been retracted or even corrected.

Although a clear majority of respondents to a poll I carried out on attitudes regarding PPPR was generally positive about it, a minority expressed concern over a ‘gotcha’ mentality (http://www.ipscell.com/2015/01/thumbs-up-for-post-pub-review-in-poll-dissenters-fault-gotcha-mentality/). Indeed, the vast majority of PPPR is negative and sometimes intensely so. In part this inclination may not be surprising given that many view it as a corrective mechanism for dealing with hyped science and inadequacies of standard peer review, particularly for high-profile papers that are perceived to have been given a ‘soft’ review. A potential example is the first paper on successful human therapeutic cloning, which was published in Cell after only a 4 day review process; it contained numerous image duplications rapidly identified on PubPeer (http://news.sciencemag.org/people-events/2013/05/cell-investigating-breakthrough-stem-cell-paper) [9].

Challenges for PPPR

A difficult issue frequently raised regarding PPPR that enables the ‘gotcha’ mentality that surfaces at times is the fact that the reviewers who participate are often anonymous. Although anonymity protects reviewers during both pre- and post-publication peer review from potential retaliation from authors, there is also a possible cost associated with anonymity. Some anonymous participants in PPPR feel emboldened to cross the line to engage in non-constructive criticism. In some cases PPPR comments have seemed targeted at specific individuals, and negative comments about researchers have even been sent to institutions – with negative repercussions leading to litigation against PubPeer (http://news.sciencemag.org/scientific-community/2014/10/researcher-files-lawsuit-over-anonymous-pubpeer-comments). It would be beneficial if more post-publication reviews noted the strengths of papers, and this does occur at times on blogs and on sites such as F1000, but realistically PPPR is likely to continue to be negative more often than not. The scientific community needs to consider how this inclination could limit the positive impact of PPPR and brainstorm ways to balance this culture.

These types of issues likely take place in pre-publication review as well, but in principle the fact that editors know the identity of the reviewers is a partial deterrent. In PPPR that safety net is at best incomplete, and often entirely inoperative, because commenter identities can be masked with pseudonyms and blocked IP addresses. Anonymity also can be a roadblock to fruitful give-and-take discussions between different scientists that largely depend on knowing with whom you are engaged. So-called ‘sockpupp-etry’, where commenters are not merely pseudonymous but sometimes actively take on false identities, or even the identities of real people, has also emerged at times in anonymous PPPR and has had a negative impact. Notably, there has recently been some constructive dialogue and brainstorming about ways to manage the potential downsides to anonymity, including better moderation, comment filtering, a set of standards, and a proposed PPPR editorial board (https://pubpeer.com/publications/F2A7891E2259B6AAD71E7F5BDA1849).

An additional concern about PPPR centers around the role that unpublished data could play. Commenters might be reluctant to publicly back-up challenges to published data with unpublished data of their own for fear of being scooped, by others or by themselves. For example, it remains unclear if a journal might consider the posting of such data online to be ‘prior publication’. This very real concern limits the extent of data-based give-and-take during PPPR.

The power of a new paradigm in peer review

Skeptics or outright opponents of PPPR point out that science is already self-correcting, and that scientists can comment on each other's articles via what are supposed to be relatively rapid journal-based mechanisms such as letters or similar formats. However, the reality is that such mechanisms are sometimes slow, and face their own challenges. For example, journals might be reluctant [10] to publish such responses if they challenge research that the journal has published, which might in some cases even lead to retractions, because no journal is likely to want to see increased retraction numbers. In the STAP case, a response article rebutting the original findings was submitted to Nature by Kenneth Lee, but the journal rejected it without clearly articulating why; it was only later published elsewhere [11]. Although there could have been many valid reasons why Nature rejected the Lee piece, this example is indicative of the complex interplay between multiple stakeholders that can in some cases tend to slow down this type of journal-centered post-publication communication, a limitation that is largely avoided in the dynamic interactions that post-publication review so nicely facilitates.

Rapid PPPR is here to stay, and, if anything, it is only likely to grow in influence and speed. A case has been made that, despite the hurdles remaining, PPPR will improve the quality of research and reduce waste in science [12]. I agree with that sentiment. Ultimately the goal is to make science more efficient, accurate, and reproducible. However, that does not mean that the evolution of PPPR will be painless or simple. Instead it is likely to be a fascinating rollercoaster ride with many twists awaiting us along the way. Hang on.