Testing the Rebound Peer Review Concept

Abstract

This invited editorial addresses the rescue of the article by Skrzypek et al. “Interplay between heme oxygenase-1 and miR-378 affects non-small cell lung carcinoma growth, vascularization, and metastasis.” The work was rejected by the standard peer review system and subsequently rescued by the Rebound Peer Review (RPR) mechanism offered by Antioxidants and Redox Signaling (Antioxid Redox Signal 16: 293–296, 2012). The reviewers who openly rescued the article were James F. George, Justin C. Mason, Mahin D. Maines, and Yasufumi Sato. The initial article was a de novo resubmission of a previously rejected article, which was then reviewed by six reviewers. The reviewers raised substantial scientific concerns, including questions pertaining to the specificity of the findings, quality of the presentation, and other technical concerns; the editor returned a decision of reject. The authors voluntarily chose to exercise the option to rescue the article utilizing the RPR system, where the authors found qualified reviewers who were willing to advocate for acceptance with scientific reasoning. The open reviewers felt that the scientific and technical concerns raised by the reviewers were outweighed by the strengths and novelty of the findings to justify acceptance. The RPR, in this case, was a “success” in that it rescued a rejected article. Despite this assessment, we question the necessity of open peer review as a means to overturn a peer review decision, with concerns for the larger-than-usual peer review process, and the voluntary relinquishing of editorial privilege and disclosure of reviewer identity. Antioxid. Redox Signal. 19, 639–643.

In this issue of Antioxidants & Redox Signaling (ARS), the journal editorial board has launched an interesting variation of the conventional peer review process, identified by the term “Rebound Peer Review” (RPR). RPR is designed to “rescue” articles that were rejected by ARS, essentially giving them a second chance for consideration for publication. This mechanism represents a variant of classical peer review in which an article that has been rejected in a single-blinded peer review process is re-communicated by the author to be re-reviewed in an open peer-review process of four peers. If the outcome of the RPR process is acceptance, the article will then be published. The open referees, recruited by the authors, voluntarily accept to be named and for their rescue comments to be published alongside the finished article (9).

A conventional single-blinded peer review (adopted for most journals) usually depends on two to three independent peer reviewers (although some journals such as ARS may employ approximately four to six reviewers during the initial review) that are selected by the editor or the journal staff from a bank of peer experts. The selection of peer reviewers is usually done on the basis of the relevance of their expertise to the content of the article under review. The peers independently provide the editor with recommendations on the suitability of the article for publication, based on their own technical and scientific knowledge. The peer reviewers are aware of the identity of the authors (but not vice-versa), but are not aware of the identities of the other reviewers. The peer reviewers are also not aware of the other peer review commentaries, but may learn them retrospectively after an editorial decision has been reached. In most cases, an editor makes a final decision on the suitability of an article based largely on those recommendations (3–4).

The end result of conventional peer review is that articles of higher quality (as judged by novelty, technical quality, presentation, interpretation, relevance, and scientific importance) will be more likely to be accepted than articles of lower quality (Fig. 1). Conversely, articles of low quality by the same criteria will be on average more likely to be rejected than articles of middle or high quality (4). In the case of middle-quality articles (the bulk of articles, as judged by ambiguous relevance or novelty, requiring additional experimental work, or rewriting for clarity of message), it can be expected that some will ultimately be accepted after author revision; whereas others may be rejected even after substantial revision. The major constructive purpose of peer review is to impose on this pool of articles a set of compelling expert-generated suggestions for improvement. On receiving these comments, the author has the liberty of choosing to revise the article in partial or full compliance to the peer review report, or resubmitting the article elsewhere. The end result is that a portion of middle-quality articles may be transformed by the peer review process into higher-quality articles. The downside of conventional peer review is that it may happen (due to reviewer bias and random error) that some high-quality articles are rejected, and that some lower-quality work is published without proper refinement. Further, the peer review process may tend to favor acceptance of articles that adhere to conventional hypotheses or the viewpoints of leaders in the field, and tend to reject articles which present very new or radical ideas. Published appraisals of the peer review process contend that peer review generally works to improve the quality of submitted articles, though scientific proof is lacking (1,3–6).

FIG. 1.

Hypothetical outcomes of peer review. Bulk article submissions contain articles of variable quality, ranging from excellent (top quality) to poor (low quality). The triaging effect of peer review is to ensure the top-quality articles will be more likely to be published than low-quality articles, and that some moderate-quality articles will be published. The function of peer review is also to convert moderate-quality articles into higher-quality articles through anonymous peer review commentary sent to authors. Additional steps may include editorial triage designed to reduce the volume of articles that enter initial peer review. Rebound peer review, as implemented by Antioxidants & Redox Signaling, provides a mechanism for the recapture of rejected articles of a presumably high quality. To see this illustration in color, the reader is referred to the web version of this article at www.liebertpub.com/ars

Importantly, peer review represents an arbitration proceeding similar to decisions made in other arenas, such as insurance settlements or political action. In such cases, one person or governing body is appointed to make a final decision based on reviewing the recommendations of a committee. It differs, however, from judicial proceedings in that no consensus is sought between referees. A critical feature of the conventional peer review process is that the editor has the privilege of arbitrating the decision, and is not necessarily bound to the recommendation of the reviewers. For example, three reviewers may recommend revision, and despite this outcome, the editor may choose to reject the article, on the basis of priority for publication. Further, some top-tier journals (e.g., Nature Medicine, Journal of Clinical Investigation, etc.) implement an editorial triage process, as a further means of arbitration (2). In this case, the editor or team of editors may implement a triage decision before engaging in peer review. Here, the editorial team may arbitrarily reject an article without seeking reviewer recommendations, on the basis of priority relative to other submissions, quality, or relevance to journal scope. No outside scientific expertise is sought, and the editor need not provide the author with any rationale or scientific feedback beyond general statements about appropriateness for the journal. This process generally reduces the volume of articles bound for peer review, by eliminating those that are obviously of low quality or do not fit the journal scope. Further, this process may tend to favor articles that conform to widely accepted hypotheses or those held by leaders in the field, and may tend to discriminate against radical hypotheses or new paradigms (9). Unfortunately, this process often results in the arbitrary rejection of high-quality work. The journals that employ this mechanism consider these decisions to be final and rarely honor appeals.

The concept of RPR raises an important and compelling point. That is, if in fact the conventional peer review system is imperfect (in that it contains bias, and instances of faulty or unfair decision making) and is arbitrary in nature (3), it ought to have a process which gives authors recourse after rejection. Most journals already have an informal arbitration and appeals processes in place. When an editor receives mixed reviews, conventionally it is the editor's responsibility to render a decision in spite of the differential feedback. If the editor is unable to make a final decision based on three reviewers, the editor may elect to recruit additional reviewers to provide additional comments until a decision can be reached. Such internal arbitration is usually kept confidential.

The RPR process represents a critical variation of peer review in that it is a form of open peer review. The term “peer” means someone of like-standing or achievement in a particular field who could be capable of judging a work for its scientific and technical merit. Reviewers with obvious conflicts of interest are expected to decline peer review invitations, but there is no formal regulation over this process. In addition, an author can propose the names of individuals who could be excluded for review based on conflict of interest. Unfortunately, the definition of peer is not limited to impartial individuals and may also include acquaintances, colleagues, former trainees or associates, friends, and competitors of the authors. Publishing the names of any reviewer may lead others to perceive a conflict of interest, even when no such conflict exists. Most reviewers surveyed prefer confidentiality, as previous surveys of open peer review, including proceedings published in Nature, have concluded that open forms of peer review remain generally unpopular within the scientific community (2). Nevertheless, participation in the RPR process is voluntary, and open peer reviewers accept to be named.

Another matter of concern raised by RPR is the decision to publish commentary alongside the accepted article. Although the peer review correspondence during RPR is kept confidential, the RPR reviewers provide commentary analyzing the peer review process for the article, and justifying the acceptance of the article, which is published with the article. With the vast amount of scientific literature available to read to keep abreast of any given field, we surmise that a few people would have time or interest to read RPR commentary as part of a published work. However, we suggest that the RPR referees could be invited to write more formal editorials.

To judge the necessity of RPR, we should also examine the fate of rejected articles (Fig. 2). On rejection, an author has several choices. The author may ignore the peer review, and send the article unedited somewhere else, as there are many journal choices, in the hope of receiving another panel of reviewers with more addressable comments. The author may also chose to reshape the article, using the rejection peer review as a guide, for an improved submission to another journal. If an article of substantial merit was unfairly rejected from one particular journal, and given also that it would likely be rendered of reasonable quality on revision by the authors, chances are that it will ultimately be accepted elsewhere by another comparable journal.

FIG. 2.

Decision process during peer review. The peer review process reflects an interchange between peer review recommendations and author decisions. The peer review process coupled with editorial decisions not only results in the acceptance or rejection of articles (outcome), but also prompts the author to revise the article and to resubmit it to the same journal for reevaluation. Authors faced with reports calling for revision may elect to comply in whole or in part, or ignore the peer review comments and seek publication elsewhere. Authors faced with a rejected article may elect to resubmit elsewhere, with or without considering peer review feedback arising from the rejection. Authors of rejected articles may also elect to petition the editor in an informal appeal. The Rebound Peer Review, as implemented by Antioxidants & Redox Signaling, provides an alternative mechanism for the appeal of rejected articles based on an open peer review model. To see this illustration in color, the reader is referred to the web version of this article at www.liebertpub.com/ars

ARS represents one of the leading specialty journals on the subject of Redox Biology and Medicine. Most researchers in the field have published or will seek to publish in ARS during the course of their careers. Is overturning a rejection in ARS of critical importance to a career scientist? The answer is yes if one desires to have a publication in a leading specialty journal with 8.546 impact factor points. RPR provides a voluntary option for those authors rejected from ARS, to continue seeking publication in ARS. The field of Redox Biology is an important component of the biomedical research community that is now recognized to touch many diverse disciplines. These disciplines include biochemistry, molecular biology and genetics, cancer and aging research, immunology, neurobiology, endocrinology, rheumatology, cardiology, hepatology, nephrology, pulmonology, and many other medical subspecialties. In fact, important work on redox-related signaling has been published recently in top-tier journals such as Nature (11), Nature Immunology (7), and Cell (8). If an article is finally rejected from ARS, there are other journals that publish redox biology-related content, including journals of approximately comparable impact factor.

In the case of the article by Skrzypek et al. (10), published in this issue of ARS after acceptance through the RPR mechanism, an examination of the original round of peer review comments has revealed that the original reviewers raised substantial concerns related to the overall quality of the article that precluded acceptance. In this article, the effect of heme oxygenase-1 (HO-1) expression on non-small cell lung carcinoma (NSCLC) cell proliferation and migration was assessed in vitro and in vivo, and attributed to down-regulation of microRNA (miR-378) expression. There were major questions pertaining to the specificity of the response to miR-378 versus many other regulated mRNAs that could affect phenotypic responses. Further observations involving HO-1 regulation were not supported by activity assays. One of the reviewers criticized the use of a single cell line NCI-H292 cells as representative of NSCLC. There were questions raised about the significance of the in vivo results that required a relatively high level of HO-1 overexpression. Additional characterization of relevant mechanistic systems, such as Nrf2, was lacking. Some of the reviewers also felt that the article was poorly written and poorly organized, and that there was insufficient description of methods, incomplete discussion of the competing literature, and insufficient information regarding the selection of clinical samples. Nevertheless, the reviewers generally felt that the examination of miR-378 in the antitumor effects of HO-1 was a novel area of investigation.

During the open RPR process initiated by the authors, the open reviewers argued for reconsideration of the article largely based on the perceived novelty and scope of the findings. It should be noted that at least one of the RPR reviewers reiterated the comments of the original peer review, calling for rewriting of the article, and additional technical concerns, including the addition of activity assays and investigation of alternate NSCLC cell lines. Finally, some of the RPR reviewers also felt that the additional experiments requested such as elucidation of the function of additional candidate RNAs in the response, and elucidation of the Nrf2 pathway, were beyond the scope of the current investigation.

Examination of the peer review correspondence for the article of Skrzypek et al. appears to reflect a borderline case, with some reviewers suggesting major alterations to the article. Based on the peer review feedback, the editor chose to initially reject this article. The editor was justified in making this arbitrary decision solely based on perceived priority, even if the weight of the reviewer's recommendation had been in favor of revision. The article was resubmitted on the RPR track and finally accepted after revision. If the RPR track had not been available or elected by the authors, the likely outcome would be that the article would eventually have been published elsewhere in revised form.

The final question that remains is whether the RPR concept was successful in rescuing an article for publication in ARS that was unfairly treated, and the answer in our view is “Yes…and No.” The process has succeeded in publishing an article in ARS from a scientific group that was likely to target ARS as a preferred publishing vehicle. An arbitrary decision by the editor to reject the article was the initial outcome. However, a decision to confidentially re-review the article to mediate an author-initiated dispute by additional blinded reviewers could have achieved the same result as RPR. Given that the article of Skrzypek et al. (10) is of reasonable scientific quality and novelty, chances are, through a series of single-blinded peer reviews with other journals, that this material would eventually have been published elsewhere, if not in ARS.

In our view, the process of RPR is imperfect in several important ways. It has altered the peer review process by publishing the names of those individuals who elected to have the article published in the second round, and who elected to be named. This has raised its own complications, in that those people identified may or may not be recognized by others in the field as not being absolutely impartial, as having a potential or perceived relationship with the authors, a bias or conflict of interest. Second, it has cost additional time from experts who are voluntarily engaging in peer review. On examining the peer review correspondence in this particular case, after passing through what appears to be two conventional peer reviewed versions (each with four to six reviewers, some redundant) and then an additional cycle of RPR, the time commitment of an estimated 8–12 individuals was solicited to achieve a similar outcome that could have been achieved in our opinion by 3–4 reviewers (3 initial, with 1 solicited to remediate a decision). Third, the process in our opinion has weakened the editorial process, by compromising an editor's privilege of final decision and arbitration. However, it should be noted that editors may be willing to relinquish the privilege of final determination, particularly those who advocate an open peer review system for the sake of improving author satisfaction or scientific quality.

We conclude that the decision to implement a formalized appeals process such as RPR to mediate journal rejection and author grievances is timely and well received. However, this process, unlike the current proposed version of RPR, should ideally remain confidential. Time and continued experience will dictate, in the long term, whether RPR, as proposed in some revised form, will be accepted by the scientific community at large as a modification of conventional peer review.