High quality review of manuscripts by qualified peers is widely regarded as central to the research enterprise. However, peer review is often viewed as a thankless task. In our editorial experiences, we have noted increasing difficulty identifying peer reviewers, with large numbers of invitations declined and many others never acknowledged. In this commentary, we offer some suggestions to increase engagement by researchers in peer review and underscore the importance of broader recognition of this role.
Proposed Peer Review Index
A frequent reason for declining an invitation to serve as a peer reviewer is that the investigator is “too busy.” With increasing physician and investigator burn out, it is important to prioritize responsibilities and identify opportunities to say “no.” Nevertheless, many who decline requests for review are busy authoring manuscripts themselves, and these manuscripts are contributing to the pool of papers requiring peer review. We believe that investigators must recognize the demand for review that their work generates and provide an equal supply of their time to review work of others.
This demand for review that investigators generate can be calculated by a proposed “Peer Review Index.” Because journal editors typically solicit two to three reviews for each manuscript and we estimate that most manuscripts are reviewed by two to three different journals, we estimate that each submitted manuscript undergoes approximately 2.5*2.5, or 6.25 reviews. We propose a Peer Review Index based on the demand for reviews generated by first and senior authors, and thus for each first/senior author paper that an investigator submits in a year (N), that investigator should be expected to review 6.25*N manuscripts divided by 2 (because we include the two most important authors—first and senior authors—in this calculation). Thus, the proposed Peer Review Index can be calculated as:
or simply, N*3.125.
Based on this proposed Peer Review Index, an investigator who submits 3 manuscripts as first or senior author in the past year would be required to review approximately 9 manuscripts. This number would be 15 for an investigator who submits 5 manuscripts; 31 for 10 manuscripts, and 47 for 15 manuscripts. The numbers in the proposed Peer Review Index are estimates; if we assume more conservative estimates of reviewers sending papers to two reviewers and manuscripts being reviewed by an average of 2 journals, an investigator who submits 3 manuscripts as first or senior author would be required to review approximately 6 manuscripts. This number would be 10 for an investigator who submits 5 manuscripts, 20 for 10 manuscripts, and 30 for 15 manuscripts. Busy investigators who generate substantial demand for manuscript reviews should complete a large number or reviews themselves to offset that demand.
Recognition for Review
Historically, peer reviewers have received limited recognition, instead volunteering their time in the hopes that other qualified reviewers will do the same. One form of recognition some journals have found successful is to offer a title of “Associate Editor” to individuals who agree to a complete a certain number of reviews per year. This benefits the journal (which identifies a reliable reviewer) and the individual (who can include this editorial title on their curriculum vitae). Journals can also recognize top reviewers publicly in the journal and/or by providing certificates or letters to department chairs acknowledging completion of a certain number of high-quality reviews.
More formal recognition is now possible with “publons” (https://publons.com/home/).1 Publons allows registered reviewers to easily notify them after completing a review by forwarding the journal’s acknowledgement of the review. Publons verifies the review with the journal and registers each review; similar processes are available for manuscripts handled as an editor. Publons’ processes are fully compliant with all editorial policies of journals around the world. Reviews are then listed by year and journal on the individual’s profile (the default setting, but this setting could be changed to show only the publisher or not show a specific review), and reviewer statistics can be compared with those of other reviewers. This recognition could have substantial impact if universities were to include review counts in promotions criteria and require that these counts be reported on curriculum vitaes. Although this would be one more statistic to track, it highlights a previously under-recognized form of service that is crucial for the success of the research enterprise. Of course, requiring such counts for promotion means that the importance may diminish once an investigator is promoted to full professor. But engagement in peer review could also be considered by committees recognizing senior investigators for rewards, to provide continued incentives to review.
Other Opportunities
Other actions could also address the challenges journal editors have in identifying sufficient numbers of qualified reviewers. First, more journal editors could refrain from soliciting reviews for manuscripts they are unlikely to accept even if all reviewer comments are generally positive. Such careful consideration would limit unnecessary reviews. In addition, editors should avoid reflexively returning manuscript revisions to reviewers, instead thoughtfully considering the need for an external reviewer’s expertise to help them determine if the revisions adequately addressed the concerns raised. Editors can identify reviewers from among authors who have submitted manuscripts to their journals. In addition, editors of journals affiliated with professional organizations can ask members of those organizations to review. Opportunities to review should attend to inclusivity; Williams et al. found that for the Journal of Pediatrics, although women were often corresponding authors of manuscripts, they were offered fewer opportunities to review.2
Second, experienced investigators could expand their efforts to train peer reviewers. Specifically, when invited to review, investigators could agree to review jointly with a trainee. Both the invited investigator and the trainee read the paper and discuss their impressions. Then, the trainee drafts the review and shares it with the invited investigator for comments and edits. Journals can encourage and facilitate such collaborations.
Third, invited reviewers should never decline an invitation for review without providing names of other suggested reviewers (unless the invitation was for a manuscript outside of their area of expertise). Often, invited reviewers have a better understanding of other qualified reviewers than the editor or editorial staff, thus these suggestions are valuable. Moreover, investigators can nominate trainees whom they know to be qualified based on expertise and experience reviewing jointly with them.
Although others have described strategies to foster high-quality peer review,3,4 consensus is lacking on the quantity of reviews that investigators should complete. The Peer Review Index that we propose coupled with increased transparency could address this gap. Investigators report that high quality reviews take on average three to five hours.5,6 The Peer Review Index that we propose surely constitutes less time for review than completing another manuscript—3 reviews would take an investigator about 15 hours—but it may be no less important. The role of peer review is to contribute to the authors and the field so that the highest quality manuscript is ultimately published.7 Additional work is needed to assure feasibility and effectiveness of the Proposed Peer Review Index from the perspectives of authors, reviewers, journals, and academic promotions committees. Nevertheless, we expect that extending formal acknowledgements for reviewers like we “credit” investigators for publications could to help mitigate reviewer shortages and ultimately encourage investigators to continue to review amidst increasingly busy lives.
Table 1.
Opportunities to enhance engagement and efficiency in peer review
| Proposed strategy | Comments |
|---|---|
| Opportunities for reviewers/authors | |
| Authors use proposed Peer Review Index to estimate number of reviews | Provides incentive for authors to review (particularly productive authors and senior authors); allows for recognition of review, especially if required by promotions committees |
| Reviewers train their trainees in conducting high-quality peer reviews by agreeing to review jointly with a trainee | Increases pool of qualified reviewers. Journals can help encourage/facilitate such collaborative reviews |
| Invited reviewers should always provide other suggested reviewers if declining to conduct a review | Increases identification of other qualified reviewers |
| Opportunities for journals | |
| Journal editors avoid soliciting reviews for manuscripts they are unlikely to accept even if all reviewer comments are positive | Lessens total number of reviewers needed |
| Journal editors avoid reflexively returning manuscript revisions to reviewers | Lessens burden on reviewers |
| Journal editors can identify new reviewers from among authors submitting manuscripts to their journals | Increases pool of potential reviewers. |
| Journal editors can acknowledge reviewers who provide multiple high-quality reviews | Increases recognition for high-quality reviewing. |
| Journals affiliated with professional organizations can solicit reviews from members | Increases pool of potential reviewers. |
| Journal editors can create editorial positions that require a minimum number of reviews per year | Increases engagement of highly-qualified reviewers, increases recognition for high-quality reviewing |
| Journals can re-direct manuscripts that are rejected for publication in top tier journals to lower-tier journals within their portfolio without requiring new reviews | Lessens total number of reviewers needed. |
Acknowledgements
Dr. Keating serves as Associate Editor of the Journal of the National Cancer Institute and is a member of the Editorial Board of the Journal of Geriatric Oncology. Dr. Mohile is Editor-in-Chief of the Journal of Geriatric Oncology.
Brief Author Biographies:
Dr. Keating is Professor of Health Care Policy and Medicine at Harvard Medical School and a practicing general internist at Brigham and Women’s Hospital. She serves as Associate Editor of the Journal of the National Cancer Institute and is a member of the Editorial Board of the Journal of Geriatric Oncology.
Dr. Mohile Professor of Medicine and Surgery at the James Wilmot Cancer Institute at the University of Rochester where she is Director of the Geriatric Oncology Clinic. She is Editor-in-Chief of the Journal of Geriatric Oncology.
References
- 1.Curry S Peer review is essential to good science – it’s time to credit expert reviewers. The Guardian 2017. Accessed on 10/29/2018 at https://www.theguardian.com/science/2017/jun/01/peer-review-is-essential-to-good-science-its-time-to-credit-expert-reviewers.
- 2.Williams WA II, Garvey KL, Goodman DM, Lauderdale DS, Ross LF. The role of gender in publication in The Journal of Pediatrics 2015–2016: equal reviews, unequal opportunities. J. Pediatr 2018;200:254–260.e251. [DOI] [PubMed] [Google Scholar]
- 3.Rennie D Let’s make peer review scientific. Nature July 7 2016;535(7610):31–33. [DOI] [PubMed] [Google Scholar]
- 4.Horbach S, Halffman WW. The changing forms and expectations of peer review. Res Integr Peer Rev 2018;3:8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA July 15 1998;280(3):231–233. [DOI] [PubMed] [Google Scholar]
- 6.Mulligan A, Hall L, Raphael E. Peer review in a changing world: An international study measuring the attitudes of researchers. J. Assoc. Inf. Sci. Technol 2013;64(1):132–161. [Google Scholar]
- 7.Fernandez-Llimos F Scholarly publishing depends on peer reviewers. Pharm Pract (Granada) Jan-Mar 2018;16(1):1236. [DOI] [PMC free article] [PubMed] [Google Scholar]