A scoping review protocol on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals

Abstract

Introduction

The primary functions of peer reviewers are poorly defined. Thus far no body of literature has systematically identified the roles and tasks of peer reviewers of biomedical journals. A clear establishment of these can lead to improvements in the peer review process. The purpose of this scoping review is to determine what is known on the roles and tasks of peer reviewers.

Methods

We will use the methodological framework first proposed by Arksey and O’Malley and subsequently adapted by Levac et al and the Joanna Briggs Institute. The scoping review will include all study designs, as well as editorials, commentaries and grey literature. The following eight electronic databases will be searched (from inception to May 2017): Cochrane Library, Cumulative Index to Nursing and Allied Health Literature, Educational Resources Information Center, EMBASE, MEDLINE, PsycINFO, Scopus and Web of Science. Two reviewers will use inclusion and exclusion criteria based on the ‘Population–Concept–Context’ framework to independently screen titles and abstracts of articles considered for inclusion. Full-text screening of relevant eligible articles will also be carried out by two reviewers. The search strategy for grey literature will include searching in websites of existing networks, biomedical journal publishers and organisations that offer resources for peer reviewers. In addition we will review journal guidelines to peer reviewers on how to perform the manuscript review. Journals will be selected using the 2016 journal impact factor. We will identify and assess the top five, middle five and lowest-ranking five journals across all medical specialties.

Ethics and dissemination

This scoping review will undertake a secondary analysis of data already collected and does not require ethical approval. The results will be disseminated through journals and conferences targeting stakeholders involved in peer review in biomedical research.

Strengths and limitations of this study.

• The large volume of literature informing this scoping review ensures that a broad overview of the roles and tasks of peer reviewers in the manuscript peer review process within biomedical journals will be obtained

• Another strength of this study is the inclusion of grey literature, including the review of journal guidelines.

• As this is a scoping review, the quality of the evidence and risk of bias will not be evaluated.

Background

The publication of peer-reviewed articles in scientific journals has long been the cornerstone of science,¹ and the primary means by which new research is documented and the outcomes disseminated.² Manuscripts that are submitted for publication in scientific journals typically undergo a critical appraisal process by researchers from a similar field who are in the wider sense peers and colleagues — known as peer review — as part of a broader editorial process led by journal editors.³ However, the importance of peer reviewing within this process extends beyond purely academic concerns. Academic publishing lies at the interface between biomedical research and practice, having the potential to influence clinical decisions.^{4 5} Clinical decisions should be guided by the best evidence available, yet these can be misleading if they are based on incomplete or inaccurate information. Any process that influences the accuracy, quality, assessment and dissemination of clinical evidence may therefore have a direct impact on patient care.³ The editorial process within biomedical journals can thus be considered to be a ‘gatekeeper’ for scientific publications, consisting of the following steps:

1. Editors consider the overall ‘fit’ of the research article to the journal, as well as suitability and relevance for the journal and its readership.⁶

2. Selection of reviewers by the editors: within the traditional biomedical sphere, peer reviewers are typically invited by journal editors to review manuscripts on the basis of their apparent expertise, which is often gauged in terms of their article output in their respective research area.

3. Editors communicate with both reviewers and authors and coordinate their interaction during peer review.

Editors are responsible for taking an independent decision regarding the fate of the manuscript (ie, whether it is accepted for publication or not).⁷ However, it has been suggested that journal editors are not entirely independent in their assessment of an article’s suitability for publication once it has undergone peer review. Research indicates that editors give considerable weight to reviewers’ recommendations on whether to reject or accept a manuscript.⁸ This may in part be due to the fact that core competencies for scientific editors in biomedical research have not yet been formally established,⁹ and most scientific editors of biomedical journals do not receive formal training.¹⁰ This is also the case for the majority of reviewers. Despite a significant proportion of reviewers perceiving that they need guidance and formal training on how to conduct a peer review,¹¹ most are not trained in how to write a reviewers’ report. Instead, reviewing is often a skill learnt through feedback received on their own submitted manuscripts.¹² Furthermore, since it is rare for reviewers to receive feedback on their own reviewer reports, it is difficult for them to know whether their reviews are of good quality.¹³

Although journals, authors and reviewers widely support peer review as the primary tool for evaluating research outputs in biomedical research,^{11 14 15} there is concurrently a broad consensus across scientific disciplines that the peer review process may be flawed.^{12 16 17} A growing body of literature has identified several potential problems, including misjudgement by editors, and biased, inconsistent or inadequate reviewing by reviewers.¹⁷

Over the years, there have been various attempts to improve the quality of peer reviewer reports in biomedical science. A recent systematic review evaluating the impact of interventions aimed at improving the quality of peer review of randomised controlled trials (RCTs) for biomedical publications concluded that there is a need to clarify the roles and tasks of peer reviewers as a step forward in quality improvement of peer reviewing.¹⁸ Within the biomedical field, the apparent roles and tasks of peer reviewers are closely related to the structural properties of the editorial process itself. For example, some — but not all — journals require peer reviewers to assess novelty and/or clinical relevance of articles in addition to assessing scientific rigour. Journals also differ with regard to their expectations of how a reviewer report should be written. Some journals encourage reviewers to follow a specific structure in their reporting, whereas other journals prefer free text. Journals also differ in their request for peer reviewer recommendations regarding whether an article should be accepted for publication in the journal or not.

These differences may influence quality of peer review reporting, and thus quality of the peer review process across journals. An RCT aimed at determining the effects of training peer reviewers found only a slight positive impact on the quality of peer review. After receiving training, the quality of the peer reviewers’ reports as measured by the ‘Review Quality Instrument’ — which assesses the extent to which a reviewer has commented on five aspects of a manuscript (importance of the research question, originality of the paper, strengths and weaknesses of the method, presentation, interpretation of results) and on two aspects of the review (constructiveness and substantiation of comments) — was deemed to have improved overall. However, peer reviewers in the study failed to detect all major errors that were introduced to the articles under review.¹⁹ At the same time, a major criticism of this study was that reviewers do not necessarily think that their task is to find all major errors in an article.²⁰ This dissonance was also reflected in a recent study that showed that the most important tasks in peer review, as perceived by peer reviewers evaluating RCTs, were not congruent with the tasks most often requested by journal editors in their guidelines to reviewers.²¹

These differences clearly illustrate the need to clarify the roles and tasks of peer reviewers. Thus far, this has only been somewhat explored, to a limited extent, for RCTs²¹ but not for other study designs.

The primary objective of this research is to determine the specific roles and tasks of peer reviewers as depicted in biomedical research. The wider purpose of this research is to inform and facilitate the future development of a set of core tasks that should be carried out by peer reviewers. This will contribute to improvements in the quality of peer reviewer reports, and ultimately of the biomedical scientific literature in general.

Methods

A scoping review was considered to be the most suitable approach to responding to the broad aim of this study. In contrast to systematic literature reviews that aim to answer specific questions, scoping reviews have been described as a process of producing a broad overview of the field.^22–24 In addition to published biomedical journal articles, grey literature will also be searched because it is likely that most of the information being sought (ie, descriptions of the roles of peer reviewers) would be found in calls for reviewers on journal websites, and guidance documents which would not generally be captured in a traditional review of published research. This approach has been previously adopted by authors of a study that aimed to identify competencies of scientific editors of biomedical journals.⁹ We used the Preferred Reporting Items for Systematic Reviews and Meta-analysis for Protocols (PRISMA-P) to draft this protocol.²⁵

This scoping review will use the methodological framework proposed by Arksey and O’Malley,²² as well as the amendments made to this framework by Levac et al²⁶ and by the Joanna Briggs Institute.²⁷ The framework consists of six consecutive stages: (1) identifying the research question, (2) identifying relevant studies, (3) study selection, (4) charting the data, (5) collating, summarising and reporting results, and (6) consultation. Each stage is discussed in further detail below.

Stage 1: identifying the research question

Arksey and O’Malley suggest an iterative process for developing one or more research questions. In the first stage two research questions have been identified based on gaps in the literature:

1. What are the expected roles of peer reviewers in the editorial peer review process in biomedical journals?

2. What tasks are peer reviewers are expected to perform for biomedical journals?

Given that some overlap between the terms ‘roles’ and ‘tasks’ is expected, we defined ‘roles’ as referring to the overarching nature of peer reviewers’ function, whereas ‘tasks’ refer more specifically to actions that fulfil these roles.

These questions might be refined, or new ones added, as the authors gain increasing familiarity with the literature.

Stage 2: identifying relevant studies

A comprehensive search strategy will be developed in order to identify relevant literature, underpinned by key inclusion criteria (see box). These are based on ‘Population–Concept–Context (PCC)’ framework recommended by the Joanna Briggs Institute for scoping reviews,²⁷ which has roots in the PICO (population, intervention, comparator and outcome) framework commonly used to focus clinical questions and develop systematic literature search strategies.²⁸

Box. Inclusion criteria.

P—Population = journal editors, publishers, peer reviewers, (corresponding) authors in biomedical journals and organisations that offer (educational) resources and training to peer reviewers in the biomedical field

C—Concept = articles with specific focus and/or statements mentioning roles, tasks and competencies pertaining to the role of peer reviewers in the journal editorial process

C—Context = the review will include all study designs, as well as book chapters, editorials and commentaries from the biomedical field; there will be no date and language restrictions

Exclusion criteria

Studies referring to peer review that is not related to manuscript peer reviewing in biomedical journals (eg, grant peer review, professional performance review, peer review of teaching and so on) will be excluded.

Search strategy

The electronic literature search strategy will follow the three-step process recommended by the Joanna Briggs Institute.²⁷ The first step consisted of an initial preliminary search of at least two online databases relevant to the topic. This was undertaken for MEDLINE (via Ovid) using the ‘peer review, research’ medical subject headings and ‘peer review’ keyword in the Cochrane Library, resulting in 2,017 studies in the Cochrane Library and 13,717 in MEDLINE. In the second step, we will closely review potentially relevant text words in the titles and abstracts of the most pertinent papers in order to compile a list of terms that can be used to inform our search strategy. Index terms used to describe the articles will also be included. This list will be combined with search strategies from existing scoping and systematic reviews on peer review^{9 21 29} to develop database-specific search strategies.

The Peer Review of Electronic Search Strategies 2015 Guideline statement will be used to guide the electronic literature search strategies.³⁰ These will be further refined in collaboration with a health sciences librarian. Subsequently, the following databases will be searched: Cochrane Library, Cumulative Index to Nursing and Allied Health Literature, Educational Resources Information Center, EMBASE (via Ovid), PsycINFO (via Ovid), MEDLINE (via Ovid), Scopus and Web of Science. The search strategy for MEDLINE can be found in online supplementary appendix 1.

There will be no time or language restrictions. The authors involved in this protocol are in command of the following languages: Catalan, Croatian, English, French, German, Italian, Russian and Spanish. Relevant articles identified in any other language will be translated.

In the third and last step, reference lists of included studies, as well as websites of journals which display a strong interest in peer review, as evidenced by numerous publications on the topic (such as the Journal of the American Medical Association (JAMA), Nature and Science), will be hand-searched using keywords related to peer review, as outlined in the MEDLINE strategy to identify any additional literature.

The search strategy for grey literature will include searching in websites of existing networks (ie, EQUATOR Network, New Frontiers of Peer Review), biomedical journal publishers (ie, BMJ Publishing Group, Elsevier, Springer Nature, Taylor & Frances, Wiley) and organisations that offer resources for reviewers (including educational courses, eg, those provided by Cochrane³¹ and Publons³²). Relevant blogs, newsletters (ie, The METRICS Research Digest³³), surveys and reports of authors/reviewer workshops will also be considered.

In addition we will review journal guidelines to peer reviewers on how to perform the manuscript review.

The guidelines will be searched for statements around the roles and tasks of peer reviewers.

Journals will be selected using 2016 journal impact factor (Thomson Reuters Journal Citation Reports-Science Citation Index Expanded). We will identify and assess the top five, middle five and lowest-ranking five journals across the medical specialties recognised in the Directive 2005/36/EC of the European Parliament and of the Council of 7 September 2005 (on the recognition of professional qualifications).

It is expected that some journals may directly communicate their instructions to peer reviewers via email or through their submission systems, rather than through publicly available instructions. In order to obtain the content of such instructions for examination, we will contact the editor-in-chief and/or managing editor of the identified journals to request details of any ‘direct to reviewer’ guidance.

Stage 3: study selection

Following the execution of the search strategy, the identified records (titles and abstracts) will be collated in a reference manager for de-duplication. The final unique set of records will be imported into a systematic review paper manager that facilitates independent screening and logs disagreements between reviewers.

The study selection process will be implemented over two stages. The first stage will involve the screening of titles and abstracts by two reviewers (KG and DC) to determine each article’s eligibility for full-text screening based on a priori inclusion criteria. The second stage of the selection process will consist of retrieving the full text of all potentially eligible articles, which will also be independently screened. Disagreements between reviewers regarding eligibility will be resolved by a third member of the research team (DH). Data will also be extracted independently by KG and DC.

We expect that some of the grey literature might subsequently be published elsewhere in the indexed literature. This will be accounted for by cross-checking authors’ names across grey literature and index literature results in order to identify potential duplicates.

An adapted version of the PRISMA flow diagram will be used to report final numbers in the resulting study publication once the review is completed. Reasons for exclusion will be recorded at the full-text review stage.

Stage 4: charting the data

A draft charting form (see table 1) has been developed at the protocol stage to aid the collection and sorting of key pieces of information from the selected articles. It will be pilot-tested and refined during the full-text screening to capture detailed information on each study. The information from research-based and non-research-based publications will be collected in separate extraction forms. Additional categories that may emerge during data extraction will be added accordingly.

Table 1.

Draft data charting form

Study characteristics	Extracted data
General information	First author’s last name Journal Publication year Study design Publication type: journal article, editorial, conference abstract, grey literature, report
Definition of peer review	Underlying definition and conceptualisation of the peer review process
General and specific descriptions of expectations and competencies of peer reviewers	Abilities Knowledge Roles Tasks Training Skills

Another form will be developed for the extraction of information from the journal guidelines to peer reviewers. In addition to the general and specific descriptions of expectations and competencies of peer reviewers, variations according to journals and their peer review models (such as single-blind peer review, double-blind peer review, open peer review, postpublication peer review) and whether peer reviewers have to provide specific recommendations (ie, no revision, minor revision, major revision, reject) will be noted.

Stage 5: collating, summarising and reporting the results

In order to create a useful summary of the data, we will combine all expectations and competency-related statements retrieved from all sources.

The general and specific descriptions of expectations and competencies of peer reviewers extracted from the different sources will be combined and de-duplicated, producing a list of unique statements. These will subsequently be organised into emerging categories. While the primary goal is to extract roles and tasks of peer reviewers, additional items related to particular abilities, knowledge, training and skills will also be extracted.

A checklist for reporting scoping reviews—the ‘Preferred Reporting Items for Systematic Reviews and Meta-Analysis: extension for Scoping Reviews (PRISMA-ScR)’—is currently under development.³⁴ If published by the time the scoping review is complete, the PRISMA-ScR will be used.

Stage 6: consultation

This final stage refers to consultation with stakeholders in the field of peer review to inform and validate findings from the scoping review. This has also been shown to be a knowledge translation activity and an important step in scoping reviews.³⁵

Journal editors will be consulted to explore their views and perspectives on the roles and tasks of peer reviewers. Results will be presented in detail in separate research papers.

Dissemination

To the best of our knowledge this scoping review is the first attempt to systematically identify the roles (overarching nature of the work) and tasks (specific actions carried out to fulfil these roles) of peer reviewers involved in the manuscript review process in biomedical journals.

As a standalone research piece, it will primarily be helpful to determine and highlight the different perspectives around the roles and tasks of peer reviewers, and will be relevant to a variety of audiences including editors, peer reviewers and authors. It will also inform the consequent consultation with stakeholders, with the aim of developing a taxonomy of peer reviewers’ roles and tasks leading to the development of a set of core competencies for peer reviewers of biomedical journals. The study findings could further be used by journal editors to review their instructions to peer reviewers and develop/update training courses for peer reviewers.