‘Scholarly peer reviewing’: The art, its joys and woes

Abstract

Research and publications are an important part of academics. Nowadays, there is an increasing trend amongst professionals including anaesthesiologists to submit scientific articles to journals for publication. Most journals are peer-reviewed which means that the articles they publish go through the peer review process. Peer review is carried out for assessing the inadequacies of research and manuscript preparation so that the best papers are published in a journal. Although peer review is a key part of the process for the publishing of medical research, there are some limitations in the system. Keeping this in mind, all aspects of peer reviewing were searched from books and journals for full text from PubMed and Google search. The information so gathered is presented in this article which focuses on the general aspects of the peer review process.

INTRODUCTION

Peer review is the heart of the scientific publication process and represents the critical phase based on which papers are published, academics promoted and Nobel prizes are won.[1] Peer is a person who is equal in ability, standing, rank or value.[2] Peer reviewers are experts who have knowledge, experience and have interest in the manuscript topic.[3] Scientific peer review is defined as the evaluation of research findings for competence, significance and originality by qualified experts.[4] Peer review, also known as ‘refereeing’, is the critical assessment of manuscripts submitted to journals by experts who are not a part of the editorial staff.[5] Peer reviews represent some of the most valuable and interesting reflections on other peoples’ work.[6]

Of late, the trend to submit manuscripts to journals for publication is increasing rapidly amongst medical professionals, including anaesthesiologists. Most scholarly journals are peer-reviewed, which means that they publish articles which go through the peer review process. People have varied notions about the peer review process,[1] and the process has been increasingly misunderstood.[7] Authors may not clearly realise how and why their article was accepted or rejected after peer review. A sound knowledge of the process of peer reviewing would be beneficial to both the reviewers and authors and also would help to improve the peer review system. A literature search was performed on the topic of peer review with a focus on the meaning, types, nature, benefits and limitations of the peer review process. The information was derived from journal articles and the internet via Google Scholar using the words/phrases ‘peer review’, ‘types of peer review’, ‘benefits and drawbacks of peer review’ and ‘the role of peer reviewers’. The literature search was performed between the years 2000 up to date. Few older articles gave insight into some basic aspects of peer reviewing including the history.

HISTORY OF SCHOLARLY PEER REVIEWING

Earlier, editors of scientific journals often made publication decisions without seeking outside input.[8] The process of soliciting peers to evaluate scholarly work prior to publication was initiated by Henry Oldenburg, editor of the first scientific journal, ‘Philosophical Transactions’.[9] The first peer-reviewed publication might have been the ‘Medical Essays and Observations’ published by the Royal Society of Edinburgh in 1731.[4] The preface to the first volume of this publication stated: ‘Memoirs sent by correspondence are distributed according to the subject matter to those members who are most versed in these matters. The report of their identity is not known to the author’.[10] Peer review in the systemised and institutionalised form as we know today has developed largely since the second world war partly as a response to the large increase in scientific research during this period.[11] Currently, peer review is accepted as a key part of the process for publishing of medical research.[12]

Goals of peer review

Peer review is the means through which journal editors can confer authenticity and authority upon scientific and scholarly papers.[13] Peers act as sentinels on the road of scientific discovery and publication.[4] Peer reviewers are mainly expected to provide constructive comments and suggestions (called as the ‘gift from anonymous’) to improve the quality and value of the manuscript.[14] They can also detect errors and fraud in a publication.[1]

THE PEER REVIEW PROCESS

Manuscript review is an intellectual process with objective and subjective elements.[15] On receipt, editors first determine the overall quality and general suitability of the manuscript. The specifications and guidelines related to the technical aspects such as the word count, the sequence of manuscript subtitles and subheadings, language, correctness of citation entries in the article and the corresponding references, overall study design and methodology are assessed in technical review. If judged to be good, the manuscript is sent to peer reviewers for assessment.[13] The editors need to ensure that appropriate reviewers are selected.[16] An electronic mail invitation is sent to the reviewer with a date within which the completed review is requested (e.g., ‘within 30 days’).[17] After receiving the invitation to review, the reviewers first confirm that the scientific content of the manuscript is within their area of expertise. They may check for conflicts of interest, academic or financial, if obvious. They may accept the invitation or decline.[17] They go through the manuscript and submit their remarks. The journal editor thoughtfully considers the remarks of the reviewers and makes a decision about the acceptability or otherwise of the manuscript for publication.[4,13] When a paper is resubmitted after ‘major revisions’, it is sent to one or more of the original reviewers to get their opinion regarding the revision. This is the process of re reviewing.[17] The manuscript may be sent for additional revisions (’re-revision’) as felt necessary any number of times, till the referee and/or the editor is satisfied. Traditionally, comments of two referees is the mandatory norm (discussed below) but the editor may decide on more based on his assessment of the manuscript and the quality of remarks received from the two. At any stage, opinion of additional reviewers may also be solicited by the editor if he/she is still not sure about the academic quality.[18] The reviewers’ opinions are normally respected by the editor.[18] The reviewers merely give a recommendation. Ultimately, it is the editor who takes the decision and determines what gets published.[6,18]

SELECTION AND GRADING OF PEER REVIEWERS

Skill in scientific peer reviewing may be ill-defined.[12] The reviewer selection processes of most journals, and thus, the qualifications of their reviewers, are ill-defined.[19,20] The editor or his team members select the reviewers. A minimum of two professionals are selected on the basis of qualities like familiarity with the topic, diversity, skill with the review process, sensitivity, honesty and punctuality.[11,13] Many journals keep an electronic database of reviewers with their names and area of expertise.[17] Ideally, editors should monitor the performance of peer reviewers, maintain and update their database and cease to use reviewers who consistently produce poor quality, discourteous or late reviews.[16] Sometimes, a diverse group of reviewers is purposefully sought to gather opinions from various angles.[13] A study showed that the most popular reason given by reviewers for doing the work of reviewing was to play their part as members of the academic community, to enhance their reputation, to achieve fame, to increase their chance of being offered a role in the journal's editorial team, reviewer receptions at conferences etc.[11] Having too little experience in review work and not being good in passing critical comments are some of the reasons which people cite when not agreeing to become peer reviewers.[14]

Types of scholarly peer review

There can be peer reviews either pre-publication or post-publication. Pre-publication reviews include varieties like single-blind review, double-blind review and open review.[21]

In a single-blind review, the identity of the reviewer is not revealed to the author but the identity of the author is known to the reviewer.[22] Single-blind review shelters reviewers from their review consequences and helps them to detect various conflicts of interest concerning the authors.[23] In a double-blind review, both the reviewer and the author remain anonymous. The article file itself is blinded at submission; that is, no identity of author or institution is allowed in the file, and the same file is sent for reviews; or such details are removed before sending the file to referees. The reviewer/reviewers’ comments that the author receives are compiled by the editor through the system where the reviewer identity is not available for the author.[21]

In an open review, the reviewer and author are known to each other.[21] The benefits of open peer review include transparency, accountability and giving credit to reviewers.[24] However, young junior reviewers may be intimidated into writing inappropriately favourable reviews of their powerful senior colleagues.[25] Furthermore, reviewers may become less critical, scientific standards may decline and professional relationships may suffer because of the creation of inappropriate dialogues between author and reviewer in this system.[25] Some studies have revealed that reviewer reports operating under an open peer review system were of higher quality than those under a closed system.[26,27] A study found that open reviews were more courteous and took longer to complete than unsigned reviews.[27] Open review though currently adopted by some reputable journals is still described as an experimental system.[28]

Post-publication review

It is a variant of open review, in which all readers are able to review, comment on the paper and rate it on a numerical scale following publication. This can occur with or without the conventional pre-publication review.[11] It is an experiment designed to increase the speed of the review process.[10] In a study, it was found that it served as a useful supplement to formal peer review. Encouragement of instant reactions and discouragement of thoughtful review are some disadvantages of post-publication review.[11] Newer models of peer review have also been described[21] [Table 1].

Table 1.

Newer models of peer review

Reviewer queries

Reviewers generally ask questions based on checklists (guidelines) sometimes provided by editors[11,13,18,29,30] [Table 2].

Table 2.

Questions that reviewers ask about papers

Effect of peer review comments on the author

Editors usually share all revisions with the author. Sometimes, certain reviews that are biased, not respectful or contain comments that focus on the author rather than the manuscript may be discredited by the editor and not sent to the author.[31] Bad comments can affect the author's confidence or make the author feel humiliated and get confused. Good comments may affirm the author's worth and give specific support directed at improving his/her work. Feedback from peers leads to a thoughtful reception of criticism, a search for confirmation by going to other sources, questioning and self-corrections by the authors.[32]

DRAWBACKS OF THE PEER REVIEW PROCESS

The peer review system though very important, has some imperfections and drawbacks.[12] Many professionals, particularly novice scholars do not possess an adequate knowledge of how to effectively critique research.[33] An experiment on peer review with a fictitious manuscript found that peer reviewers failed to detect some manuscript errors.[34] The most widely recognised failure of peer review is its inability to ensure the identification of high-quality work.

In 1796, the journal editor of ‘Philosophical Transactions’ rejected Edward Jenner's report of the first vaccination against smallpox.[35] A study found that peer reviewers often fail to detect important deficiencies in the reporting of the methods and results of randomised trials.[36] The responses received in a survey of a sample of scientists who were authors of highly cited articles indicated that a majority of them had faced the problems of manuscript rejection, scepticism, ignorance and incomprehension by the peer reviewers.[37]

The reviewer may adopt a stringent approach in an attempt to serve as the journal's gatekeeper, and this can lead to harshness in the tone and content of reviewers’ comments.[13]

Different reviewers may offer conflicting reviews because they may expect, notice and value different qualities in an academic submission leading to problems in the editorial decision.[38] A common source of conflicting advice is the length of the manuscript.[18]

There could be nationality, language and speciality related bias.[11] There may be a strong bias against ‘negative studies’.[39] Peer reviewers can suffer from intellectual suppression due to: (1) The Matthew effect (the rich get richer and the poor get poorer): The manuscripts of famous researchers have greater chances of getting published whereas less popular authors’ works may get rejected. (2) Heider's assimilation-contrast theory: We experience concordant affective reactions to the ideas of persons who belong to our in-groups and discordant reactions to those who do not.[40] Single-blind review according to some authors encourages an unconscious bias towards prominent authors or prestigious institutions.[39] Gender bias is a possibility when reviewers know the identity of the author. Many studies provide evidence that double-blind review is more fair to authors from less prestigious institutions and to women authors.[22] Double-blind review decreases the enthusiasm of reviewers, and reduces their timeliness. It places extra burdens on the editorial team, reviewers and authors.[22]

It is difficult to identify and motivate high quality peer reviewers because they are increasingly busy and often find it difficult to free up time to do reviews.[41,42] Many reviewers devote considerable amounts of time and energy, frequently reviewing for multiple journals without incentive.[4,6,43] Many capable intellectuals avoid review work and if they agree to do it, they give it their last priority. Many reviewers do not finish the review process before the set deadline, thus causing a delay in the publication process.[41] Some reviewers take their task too seriously. They become a menace by decimating an article, or becoming quasi-authors themselves.[41]

Journals vary in their peer review standards. Acceptance of an article by a peer reviewed journal does not tell much about the quality or originality of the article.[11] Nowadays, there are many predatory journals that charge publication fees but deliberately omit the peer review process. This amounts to editorial misconduct.[44]

Lack of facilities like non-availability of access to full text articles on PubMed or other sources to the majority of reviewers and to some editors is one of the major limitations of the peer review process.[45]

MEASURES TO IMPROVE THE PEER REVIEW PROCESS

• Blinding can reduce bias in the review process and encourage reviewers to give their honest appraisal.[29] A survey showed that 56% open respondents preferred double-blind review followed by 25% for single-blind, 13% for open and 5% for post-publication review.[11]

• Specialisation and formal training of young reviewers showed improvement in peer review in a randomised trial.[46] Reviewers of research reports should be well versed in the scientific method and statistics.[29] Training, ongoing appraisal and revalidation if provided to individuals who peer review randomised controlled trials can help them to improve.[21]

• Reviewers can be recognised and rewarded by using reviewer centric approaches like Reviewer Index, Reviewer Index Directory and Global Reviewer Index Directory. This can help produce high-quality reviewers.[41] Incentives like free subscription to the journal, acknowledgement in the journal and offering of discounts in author publication charges by the journal can encourage scientists for reviewing.[11]

• A committed peer reviewer system and use of beneficial technology by peer reviewers can overcome aberrant attitude in the authors and prevent scientific fraud.[47]

• Adopting the Broad Daylight Publication Model may lead to better reviews. This model includes openness at three levels-disclosing submissions and reviews, making reviewers accountable for their actions, reviewer rating by readers and opening up the editorial hierarchy for reviewers with good ratings and reputation.[6]

• A key to the success of the peer review process is the journal editor who must be rigorous in selecting and deselecting reviewers, be vigilant about the subjective elements of the process and ensure that it is fair.[1,13]

CONCLUSION

Peer review is the best way to ensure quality control of submitted scientific material. This can be achieved by adopting different types of scholarly peer review. As with any other system, the peer review system has some imperfections. Several strategies are being tried to improve the system. Honest, timely, competent and fair work by peer reviewers combined with competent and sincere editorial supervision can ensure quality assurance of the peer review process and evolve the system into a process that produces a good scientific output.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.