How to Review a Manuscript

Knowledge in our world is expanding at a dizzying pace. In the realms of medicine and biology, new information is emerging so rapidly that it is difficult to stay abreast of one’s own field, much less areas beyond. This is at once exciting, enabling, and frustrating. And at the same time, we are facing a “reproducibility crisis”, where surprising percentages of published information are held to be irreproducible¹. If true, this is a travesty of time, effort, and resources stemming from a combination of incomplete oversight and, on occasion, outright fraud. Undoubtedly the most tragic consequence of failure to purge bad information is placement of humans at risk, as evidenced by the ongoing questioning of the benefits of vaccination spawned by discredited work².

Peer review is the process whereby good science is enhanced and bad science is dismissed. Central to the entire scientific process, peer review is how we “separate the wheat from the chaff” to optimize scientific progress, enhance reproducibility, and to eliminate poor quality science.

Peer review occurs when content experts evaluate the work of others. In the special case of grant proposals, reviewers evaluate the proposed research, its feasibility and novelty, the proposing investigator, and the work infrastructure and environment. And notably the process goes both ways: grant reviewers are themselves grant writers, and their proposals are similarly evaluated by peers. Quite little objective science has been devoted to the process of peer review of grant applications, and yet it has stood the test of time, despite its flaws³. It remains the best approach identified to distribute limited funding resources. (In a prominent example of a different approach, the Howard Hughes Medical Institute funds individuals as opposed to projects.)

Similar to the process with grants, manuscript peer review is conducted by investigators expert in the field and content of the paper. It is far from perfect, but a better strategy has not yet emerged. In broad overview, the reviewer is asked to comment on the novelty of the work, its veracity, and whether the conclusions are impactful. And again, the process is bidirectional: reviewers are themselves investigators and authors. A mentor of mine codified the peer review process in simple terms: “Is it new? Is it true? Does anybody give a #*&%?” That pretty much says it all!

Peer review is a privilege, and an invitation to conduct such a review is a professional honor. Peer reviewers are the arbiters of scientific quality, and they participate directly in the formulation of new knowledge. They have been invited to the process owing to their stature in the field, their knowledge, and their scientific “taste.” Peer review is a critical service to our profession and to science in general. It is a service that is typically unrecognized and uncompensated, one where individuals toil behind the scenes to enhance the work of others to promote progress.

Again, very little science has been directed at guiding the manuscript peer review process. Junior investigators do not receive formal training in its conduct. Rather, it is learned by experience and mentoring: by doing reviews, receiving reviews of one’s own work, and by obtaining feedback from seasoned reviewers. Some guidance from editors has appeared in the literature previously^{4, 5}. Here, I lay out my perspective on this cornerstone of the scientific endeavor. As there are no hard and fast rules on manuscript peer review, the comments provided here should be interpreted as my personal views.

The process

In order to conduct an appropriate peer review, one must be familiar with the field and the technical approaches being employed. Equally important, the reviewer must have scientific “taste”, grasp the principles of good science, recognize the inevitable limitations of any particular method or strategy, and appreciate the appropriate use of critical controls and rigorous statistical analyses. A reviewer must be perennially skeptical, always seeking alternate interpretations – well beyond the most obvious one or the one put forward by the authors – to explain the findings reported in a manuscript. It has been said that the job of a reviewer is to see what the authors have not seen.

Peer review involves assessment of the importance of the research question, the originality of the work, its strengths and weaknesses, data analysis and interpretation, and the presentation of the information. The work should be evaluated comprehensively in terms of its content, methodology, and ethics.

A good review will briefly summarize the study and the authors’ conclusions. It will identify the strengths of the study and, critically, lay out its flaws. Are important controls lacking? Are additional approaches required to evaluate a central question or to render a conclusion definitive? As one example, it is often possible to corroborate conclusions deriving from pharmacological manipulations with genetic approaches. A single drug employed at a single dose leaves open the prominent possibility of off-target effects. In general, intervention with more than one compound should be used (or more than siRNA, for that matter) and ideally incorporating dose-response analyses.

A common and unfortunate shortcut in peer review is to simply ask for more and more experiments, mandating that the investigators pile on additional data and analyses to expand the scope of the paper. In my view, it is the prerogative of the authors to propose the scope of their report. The reviewer may find that the scope is limited, which would warrant a negative review. Better is it to identify holes in the study, determine whether the conclusions are appropriate and supported by compelling evidence, and then assess whether the scope of that scientific “package” warrants publication in the journal in question. Peer reviewers are not asked to rewrite the manuscript, redirect the research, or expand the scope of the paper.

A well conducted review will break a manuscript down into its components, evaluating each part individually.

• Does the Abstract stand alone, able to be understood without reading the full manuscript?

• Does the Introduction lay out the rationale for the study and explain its goals? The Introduction should not provide a lengthy and detailed review of the field.

• Does the Methods section provide sufficient detail that the experiments could be repeated? Are the statistical analyses appropriate? Nowadays, much, even most, of the Methods is relegated to online status, and it is very easy for reviewers to simply skip over that critical component of the paper.

• The Results section is the piece that arguably warrants the greatest amount of attention. The reviewer must determine whether the data are reliable and bolstered by requisite controls; whether appropriate scientific redundancy and rigor are applied throughout. Can some of the text be eliminated or condensed by judicious use of tables or figures? Are statistically significant findings also biologically significant?

• Does the Discussion put the findings in perspective, rather than simply rehashing what was described in the Results? Does it compare the reported findings with prior ones in the literature, highlighting and discussing commonalities and discrepancies? Does it provide an honest assessment of the inevitable limitations of the study?

• Are the figures clear and clean, with appropriate labeling? Do the figure legends stand alone, illuminating and clarifying each figure? Could someone understand the entire study simply by perusing the figures and legends?

• Do tables, if present, provide added value?

• Are the references carefully selected and appropriate? Now, most reviewers will not take the time to double-check most of the references, but a quick, spot check is important, as this will sometimes point to flaws.

The quality of the writing itself is critical, and a good reviewer will provide insights into how it can be improved. It is easy, even commonplace, to make good work look bad owing to poor presentation. Conversely, it is essentially impossible to disguise bad work as good just by dressing up the presentation. In my view, sloppy presentation raises the concern that the conduct of the science itself was sloppy. Also, can the paper be shortened?

Confidential comments provided to the editors should be congruent with those provided to the authors. If a strongly negative message is conveyed to the editor, while a substantially less negative message is delivered to the authors, the authors will be appropriately confused, which may lead to a rebuttal.

Reviewer comments should at all times be constructive. All reviewers are authors, too! Thus, the golden rule pertains: treat the paper you’re reviewing the same way you wish to have your own manuscripts handled. We all hope that the reviewers of our work are fair, knowledgeable, constructive, and prompt.

Bearing in mind the golden rule, a reviewer is an author advocate. At the same time, he/she functions as a journal advocate. A reviewer is asked to provide an informed opinion about a manuscript, thereby providing advice to the editor. However, the decision to publish a manuscript is made solely by the editor based on a wide range of considerations.

A vital responsibility

Any editor greatly appreciates the service provided by peer reviewers. Indeed, the success of any journal – and, indeed, the entire scientific process – hinges critically on an army of reviewers working silently and behind the scenes. And it is pretty easy to distinguish a good review from a bad one, from a reviewer who takes the process seriously from one who does not.

The peer review process places additional and critical expectations on reviewers. At all times, a thorough and honest self-assessment of potential conflict of interest is paramount. Reviewers are expected to maintain strict confidentiality regarding the information they are evaluating. Comments should be constructive regardless of whether one’s recommendation is to accept, revise, or reject the manuscript.

Manuscript peer review is a noble responsibility, one which is pivotal to the entire scientific process. Learning to be a good reviewer is an important skill for all investigators, situating one as a gatekeeper of scientific truth. It also enhances one’s own science, making each of us a better investigator and writer. A process which is inevitably flawed owing to its inherently human nature, peer review is not going away anytime soon. Let’s all dedicate ourselves to making it as good as it can be.