Reviewer’s Checklists for Evaluating Scientific Manuscripts

Introduction

Peer review is a critical step in the process of publication of a scientific manuscript. The aim of reviewing is to evaluate the quality of a manuscript on various aspects and to make a recommendation to the editor for publication in the concerned journal. A good review considerably increases the quality of a manuscript and is beneficial for the reviewer and the author in creating future manuscripts.

While searching for reviewers, the editors tend to pick those who have accepted most invites as well as those who have completed most reviews, since these data are available in the database. It is difficult to predict a high quality reviewer based on the qualifications, training and experience [1]. Being invited as a reviewer is a recognition of expertise in that particular field.

It has been found that spending more than three hours in reviewing a manuscript did not increase the quality of a review but the quality improved with increasing time spent up to 3 h [2]. Younger reviewers up to the age of 60 years, training in statistics or epidemiology and current research investigator were significantly associated with good review quality according to assessments by editors [2].

The role of a reviewer starts when he/she receives an email from the editor inviting to review a manuscript. The information provided to them would be the title of the manuscript and the abstract. The reviewers have to make a decision to review or not based on this information depending on whether it falls within the remits of their area of specialization and expertise. It is advisable to accept or decline the invitation as soon as possible as it will give an opportunity to refer to other reviewers early and saves time for the editors and the authors. Since the publication by Atul Gawande on checklists in surgery and adopted by WHO in preoperative checks in operation theatres, their use has been established and growing in medicine [3, 4]. While there are exhaustive lists for assessing each type of article, we aim to provide useful basic checklists for reviewers while evaluating manuscripts and formulating their reviews.

What Should the Reviewers Equip Themselves with Before Accepting a Manuscript?

1. Scope of the journal

2. Basic working knowledge of the subject being reviewed

3. Good English and grammar

4. Knowledge of what information goes into each section of a manuscript. (Introduction, Materials and methods, Results, Discussion, Conclusions and References)

5. Awareness of author instructions with reference to

   1. Tables (should be less than 7 combined with figures for IJO)
   2. Figures
   3. Word count
   4. Format of references cited in text as well as bibliography

6. Duration available for review

7. Making note of the format of manuscripts that are submitted to the journal helps

8. Making sure the expertise of the reviewer in the database is updated to avoid manuscripts from unrelated sub-specialities.

During Review

Perform a basic scan of the above aspects of the manuscript. Tables 1 and 2 will aid in the basic review process of scientific manuscripts.

Table 1.

Checklist during the initial assessment by the Editor and the Associate Editor (All contents under acceptable and satisfactory are in green colour shading and those under unacceptable and unsatisfactory are in light red colour shading)

Table 2.

Checklist for evaluating manuscripts for the reviewers (All contents under acceptable and satisfactory are in green colour shading and those under unacceptable and unsatisfactory are in light red colour shading)

The colour coding of the tables would give an idea regarding the decision to accept or reject a manuscript. If the manuscript predominantly scores green shades (from Tables 1 and 2) on most issues mentioned, then the manuscript is probably good to be accepted provided major or minor revisions are not required. This objective tool will also help in rejecting a manuscript that is unsuitable when the manuscript scores red on most issues. Though this is likely to help decision-making, not every manuscript can be judged in green and red shades, some are likely to have grey areas, and herein the reviewer will be required to exercise discretion. Similarly, a checklist to review statistics of a paper may be used. How to present statistics in a paper has been given by previous publications [5–7].

The following is a summary of the most important Dos and Don’ts during reviewing.

Dos

• Respond early to invite (remember somebody else is waiting to review and learn if invited)

• Submit review in time

• Be courteous to authors and address politely and constructively while criticizing

• Assess each part of the manuscript and analyze them separately if possible. Identify exact location of the issue being raised.

• Have a plan on how to assess each type of manuscript

• Look up literature/do a PubMed search to see what’s new on the topic

• Editors feel favourably towards a reviewer when decision is submitted before time

• Assessing grammar and spelling is not mandatory but may be a bonus

• Give an honest opinion on the manuscript

• Decline from reviewing when there is a conflict of interest

• Be aware of guidelines on reporting certain types of manuscripts

• Back your decision with points that justify the decision made.

• Not everybody is an expert in statistics. So basic knowledge is adequate to start reviewing.

Don’ts

• Give decision without reviewing.

• Single sentence comments on manuscripts.

• Paste the same message to editor and author.

• Delay submission of decision.

• Discuss manuscript outside since it is a confidential document.

• Letter to Editor on same manuscript after review will not be encouraged by editor as those points should have been dealt during review process.

• Hide the fact if the research or researcher is known as it will not form a blinded review.

• Avoid advising authors on how to improve grammar.

• Do not solicit citation of own articles but may be pointed if work is important.

• Do not force authors to cite particular journal but may be pointed if work is important and left out.

Reviewing a Revised Manuscript

This is usually quick since the manuscript has been reviewed once before and only the responses by the authors need to be checked along with corrections made. Hence, the time given for reviewing a revised manuscript is usually a week by most journals.

1. Read all the inputs from other reviewers and the responses given by the authors to the reviewers.

2. Have the authors addressed all the issues raised by reviewers satisfactorily?

3. If yes, countercheck the changes made by the authors to the manuscript

4. If no, politely give a reminder. The author may have forgotten to resolve some issues.

Editors’ Role in Reviewing

1. The editor picks the reviewers who can do a good review of the manuscript. He/she favours those who respond fast, provides good quality reviews on time and is reliable. The above information on reviewers will be available in the database. Experienced reviewers are usually preferred for obvious reasons. If you are early in your career as reviewer, and provide a bad review, you may not be favourably considered for another review. From the editors’ point of view, it is valuable time wasted by the reviewer. This delays the editing process. So new reviewers should take this learning opportunity well, take time to meticulously analyse and then provide the review. A good start is to practice critiquing manuscripts in journal club meetings before taking up reviewing.

2. Editors review the manuscript they are assigned and are guided by the inputs from reviewers to make a decision on publishing.

3. They peruse the comments from reviewers and combined with their own assessment, make a plan and take a decision on the document regarding its publishability.

4. They act, as a link between the authors and the reviewers in improving the quality of the manuscript since reviewing is a blinded process.

5. Some of the comments by reviewers may be edited or deleted before they are sent to the authors if the editors find the comments inappropriate [8].

Tips for Book Review

Book review should be undertaken by an expert in the particular field e.g. a foot and ankle surgeon writing a review of a book pertaining to upper limb would not be considered appropriate. These are our recommendations for writing a review on a book.

• Book title, edition number, publisher name,

• Author’s affiliations

• About the book

• Who are the expected readers? Undergraduate students/postgraduate students/General orthopaedic surgeon/Specialist orthopaedic surgeon

• Who are the contributing authors? Are they all from same country or from different countries?

• Is the book published by a specialist organization? e.g. Indian Arthroscopy Academy.

• If there are previous editions of the book then how is the present edition different/similar to the prior edition?

• What is the overall objective of the book?

• What are the unique features of the book?

  e.g. HD photographs/CD/Videos/Complexity of cases

• What are the strengths of the book?

  e.g. Easy readability/Online access to website

• What are the shortcomings of the book?

  e.g. Poor quality photographs/not having radiographic images etc

• What is the overall recommendation for the book?

• What would be the ideal location of the book? (e.g. Personal collection of every orthopaedic surgeon/departmental library/office in the operating theatre for quick reference)

Controversy Surrounding Pre-print Servers

Authors and reviewers need to be cautious while submitting to preprint services such as MedRxiv because manuscripts submitted to these preprint repositories are likely to be picked up by the plagiarism detection software used by reputed journals and likely to show higher similarity. In addition, manuscripts once submitted to preprint servers could not be withdrawn. Plagiarised and unscientific manuscripts are reported to be excluded from these preprint services and the biggest drawback as acknowledged by these preprint servers is the lack of peer review. Since these manuscripts are cited on Google scholar and Crossref, there is a potential for dissemination of non-peer reviewed work to enter public domain and erroneously influence decision-making.

Funding

No funds, grants, or other support was received.

Declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Ethical Standard statement and Informed consent

Informed consent and ethical statement not required as patients are not involved in this manuscript.