A transparent black box

A transparent black box

Further enhancements to the peer review process and journal policies ensure a transparent, fast and fair editorial process

Further enhancements to the peer review process and journal policies ensure a transparent, fast and fair editorial process

Vital statistics: high speed and fewer laps to the finish line

A unique feature of The EMBO Journal is that we publish data on the efficiency of the editorial process (find the latest analysis of manuscripts submitted in 2009 at http://www.nature.com/emboj/about/process.html). We accepted 12% of submitted manuscripts, 2% more than in the preceding year, with similar submission volumes. The journal is fast: editorial decisions were made, on average, in 2.3 days and post-review decisions in 29 days1. The majority of manuscripts went through only one major round of revision. Notably, we ensure that no new issues are raised that were not brought forward in the initial review (excluding points on data added in revision, of course). A total of 74% manuscripts were rejected before formal peer review, 21% of these with additional advice from the editorial advisory board. We have a relatively high ‘first cut', so that only manuscripts that fit within the journal's scope, that present a sufficiently striking advance and that have a real chance of being published with realistic further revision continue to the more time-intensive peer review process. As a result, 46% of the peer reviewed manuscripts end up being published. Remarkably, 97% of cases where a revision is invited are published. Thus, once a revision is invited, the odds for publication without delay are high. Are we selecting the right papers for publication? Our analysis of manuscripts rejected in 2008 shows that until now 1% appeared in journals with an impact factor two or more notches above this journal. 9% of the rejected manuscripts cite higher than the average EMBO Journal paper.

Scooping protection

Molecular cell biology is a rather competitive discipline. The next big open questions rarely occur to only one researcher. Technical and research advances, as well as funding priorities, encourage multiple laboratories to pursue similar directions. In contrast to other disciplines, experiments are quickly executed and often don′t require unique hardware or expertise. Derivative work will not be accepted in a highly selective journal such as this. However, in our view, work that has been scooped is not necessarily derivative. We certainly don′t want to encourage premature publication to claim first discovery and we appreciate that authors who go the extra mile to cross every t and dot every i, who delve further in mechanism or expand the physiological significance of their findings run the risk of being scooped. We want to encourage broad and deep studies over ‘salami slices'. For this reason, it is editorial policy that a study will not be rejected on account of having been scooped while it is under consideration. Where additional experimentation is required, we will specify when to expect a revision (usually around 3 months). In this period, an author is also immune from rejection due to scooping, as long as related publications are immediately discussed with the editor so as to minimize delays. If there is good reason that a revision will take longer than specified and no related literature has appeared, we will usually be able to extend the deadline.

Peer review process files

The majority of papers published in The EMBO journal over the past 2 years (408 at the last count) have been accompanied by a detailed document edited only for typos, presenting the referee comments, author responses and editorial decision letters of every round of review (Rørth, 2009; Pulverer 2010a, 2010b). In addition, the actual submission, revision and decision times are published; that is dates are not reset upon resubmission following substantial revision. It is time to take stock: have the peer review process files enhanced the editorial process?

The files are popular. The feedback to the policy is overwhelmingly positive, although some rightly note that viewing reports on the rejected manuscripts would be much more telling. Alas, the only way we can address this is to encourage referee report recycling (see below). Several other journals are adopting similar strategies and notably some BioMed Central journals have had a similar system for some time (Linder et al, 2010). Importantly, the files remain strictly optional; that is an author can opt out at any stage (even after having seen the reports)—but only 5.3% do. In a couple of cases, authors were reluctant to publish ‘for referees' data, but we are willing to remove such data from the files if there are good reasons for doing so. Referees are of course informed that their reports may be published upon acceptance, and yet our referee acceptance rate has not changed. A handful of referees have disagreed with the policy, noting that they think a critical report may detract from the paper, or a fear that their identity might be revealed.

Does anyone actually look at peer review process files? We assessed a 3-month period earlier this year and found that around 1/10th of the readership of a paper seem to go on to visit the process file. The most accessed ones tended to be associated with the most visited papers and indeed manuscripts that had received enthusiastic referee reports. We are encouraged by the positive echo and have now adopted the same policy at the other three EMBO scientific publications: EMBO Reports, Molecular Systems Biology and EMBO Molecular Medicine (Lemberger, 2010; Pulverer, 2010c).

Have the files led to tangible changes in the reports we receive? We have not noticed any systematic change, although a few referees have noted that they are taking extra care with writing their reports. Importantly, neither referees nor authors have become more timid in putting forthright arguments to paper. The hope is, of course, that a more open discussion would be as thorough as before, but more civil.

Why invest so much effort into these reports? A key aim is to show that the refereeing process is usually constructive and effective. Of course, a diverse set of opinions follow many papers, but the expert reader will readily discern the divergent expertise of the referees. Peer review is much maligned, and we hope systematically showcasing the reports may quell some of these concerns. Furthermore, these files can serve as educational tools: peer review is very much at the core of the research process and yet it is rarely formally taught. We are thinking hard at present how to provide more tangible credit for the currently anonymous experts behind these comments. For now, the files remain also a thank you for the help of all our referees: much of the input is far too valuable to let it sink into oblivion unnoted—for rejected manuscripts, we hope some reports can be ‘recycled'.

Manuscript transfer and review recycling

Undoubtedly, the bottleneck of the editorial process is peer review. Science output is growing so rapidly that the number of experienced referees cannot keep up. Furthermore, the pressures to publish in highly selective journals are ever more prominent, so that authors try to submit their data sets to the most prestigious journal possible. The result is extremely high rejection rates at top journals and many frustrating ‘cascade submissions'—every new journal will have to find referees. As referees who have seen a manuscript previously are understandably reluctant to re-review (often because they feel it might result in ‘double jeopardy'), this can rapidly deplete the number of qualified experts in increasingly specialized subject areas. It is in nobody's interest to bypass a high-quality peer review process—certainly an author will not want to publish a study with serious deficiencies. In order to address this catch 22, we are in favour of ‘referee report recycling' through author controlled manuscript transfer between journals. Any author can already transfer a manuscript between the four EMBO titles at the click of a mouse button at any stage of the process (after peer review with the referee reports and identities). In case a promising manuscript is rejected by one journal for scope considerations, the editor will recommend a transfer. With your permission, the editors of both journals will be more than happy to discuss a manuscript to ensure a successful outcome is pre-programmed. For example, a highly novel study that is thoroughly experimentally controlled and documented, but that ultimately lacks a sufficiently developed level of molecular mechanistic understanding, may turn out to be more suitable for EMBO Reports (http://www.nature.com/embor/index.html).

Cross-peer review, co-review and ‘confidential comments'

In our opinion, highly selective journals have not outlived their raison d' etre. On the contrary, with exponentially growing and increasingly specialized research, a quality filtering service appears to be more important than ever to stay abreast of scientific progress. Of course, the specialist literature will remain an important component of documenting research advances, and it should certainly be included prominently in research assessment. Nevertheless, no semantic text search engine has as yet obviated the need for platforms that allow browsing beyond one's immediate area of specialization in a trusted and reliable environment. We aim to provide such a platform with The EMBO Journal. A high-quality peer review process is and will remain at the heart of quality journals. However, it is clear that not only is the process near saturation, but it can also go awry.

One obvious concern is that referees have to be close to the research submitted—it is all too easy to get too close and encounter undeclared (and usually unperceived) conflicts of interest. Obviously, referees are as opinionated as authors about certain experimental approaches and lines of evidence and they will naturally tend to let their preferences guide their reports. It is, therefore, essential that there is an additional layer to arbitrate. We have for this very reason a professional group of scientifically qualified editors supported by an editorial advisory board composed of a broad panel of leading researchers. We read referee reports and author responses impartially, as our only aim is to publish the best papers possible. Professional editors spend a lot of time on getting decisions right. However, we are naturally generalists and it would be naive to state that a well-argued report by a trusted referee would not have a significant impact on the editorial decision—and so it should. The key is to disambiguate essential criticisms from further reaching suggestions based on personal preferences.

Based on where papers rejected at this journal are subsequently published, the fact that we accept most papers after one major round of revision and our low appeals rate (6.4%), the editorial process is robust. Nevertheless, we are acutely aware that every mistaken rejection can seriously damage the author's research career. For this reason, we have added an additional layer to the peer review process: we actively encourage referees to comment on each other's reports. Like other journals, we used to send all referee comments to each referee with a summary of the editorial decision. We almost never heard back. Instead, for the majority of manuscripts, we now send the reports to all referees a day before the decision is made and actively encourage feedback. We are also in the process of adapting our manuscript submission system so that a referee sees the other reports already uploaded as soon as they post their own comments. It is essential to emphasize that we do not expect every referee to comment on every other report—this will be the exception, not the norm. The lack of a post-review comment will in no way lessen our appreciation of the primary report filed. We see two major scenarios where post-review feedback is important: if a referee wants to note that one of the other referees has raised erroneous or non-essential issues, or indeed if a bias is perceived, or if a referee has overlooked an essential point raised by another referee and wants to reinforce that point. Importantly, we will not always go with the last word—the additional feedback will just help us think in a more differentiated way about the decision, and, if need be, engage in further consultation. Note that this additional step in no way delays the editorial process: we make the decision with or without additional feedback! Remarkably, we are receiving a significant level of cross-referee commenting now, and a good fraction of these comments have aided the decision.

As part of our transparency initiative, we have dropped the ‘confidential comments' box from our referee reports. We actually rarely encounter comments that are at odds with the main referee report, although referees sometimes repeat their views in more straight language. Nevertheless, the existence of this additional layer of commenting naturally begs the question ‘What went on behind the scenes?'. Confidential comments are clearly appropriate in the rare cases where there are concerns about ethical standards, data integrity, biosecurity or conflicts of an academic or commercial nature, which should be communicated directly to the editor.

One further change to our referee policy is that we acknowledge the relatively common practice of handing the review onto someone else in the laboratory. If carried out correctly, this can in fact be an important part of training, but it should be reserved for experienced postdocs. Thus, The EMBO Journal allows co-refereeing with one other senior member of a referee's laboratory as part of the mentoring process only if the primary referee has independently evaluated the manuscript and agrees with the report filed. For co-review, conflict of interest and confidentiality rules apply to both referees. In order to provide accountability and appropriate credit, we request that the name of the co-referee be documented to the editors.

We hope that these measures collectively serve to enhance the peer review process. Peer review is here to stay and we will continue to optimize its practice, also based on your feedback.