Peer reviewer survey for Baylor University Medical Center Proceedings

Abstract

Editors of medical journals have important responsibilities and depend on peer reviewers to evaluate the quality of submitted manuscripts. However, invitations to undertake peer review are often declined, and in some cases the reviewer fails to provide a review in a reasonable timeframe. Baylor University Medical Center Proceedings surveyed recent reviewers to determine their motivations for undertaking a review and possible benefits associated with reviewing. Sixty-seven reviewers (12.4%) out of 540 responded to the survey request. Reviewer characteristics included long-standing involvement in research (50 reviewers in research ≥11 years) and prior publication (50 reviewers ≥6 articles). Many reviewers thought that reviewing articles represented a responsibility and provided an opportunity for them to contribute to scientific efforts and medical publications and to learn new information. Survey respondents suggested that recognition by the journal was an adequate benefit. This survey, like others, indicates that editors will continue to depend on a core set of reviewers who consider this activity both an opportunity and a responsibility.

Medical journals depend on peer reviewers to provide expert opinions about the content and clarity of submitted manuscripts. These reviewers can also help identify plagiarism and fraudulent studies. This process of peer review is considered by many to be essential to safeguard the integrity of academic work. However, reviewers often decline requests for peer review and/or fail to submit recommendations promptly. Baylor University Medical Center Proceedings recently analyzed reviewer performance for its journal.¹ Over a 45-month period, 5424 invitations met the inclusion criteria for that study. Forty-two percent of the invited reviewers (2278) agreed to review manuscripts; 331 reviewers (14.5%) did not complete their reviews. This result led to a follow-up survey of reviewers to determine factors associated with their motivation to review articles and their perceptions of the potential benefits to undertake a review.

METHODS

This survey was sent to 540 reviewers for Baylor University Medical Center Proceedings who had completed one or more reviews for the journal in the last 3 years. The survey was open from June 21, 2022, through July 18, 2022. All data were anonymized, and the responders were told that it would be used to identify possible improvements in the review process and as part of a study carried out by the editorial board of the journal on peer review trends.

RESULTS

Sixty-seven reviewers (12.4%) responded to this survey. Seventeen reviewers (25.4%) had been actively involved in research for 0 to 10 years, 22 reviewers (32.8%) had been involved in research for 11 to 20 years, and 28 reviewers (41.8%) had been involved in research for >20 years. Seventeen reviewers (25.4%) had published ≤5 articles during their lifetimes, 21 reviewers (31.3%) had published 6 to 20 articles, and 29 reviewers (43.2%) had published ≥21 articles. Forty-one reviewers (61.2%) had reviewed <6 articles per year, 18 reviewers (26.9%) had reviewed 1 article every 1 to 2 months, and 8 reviewers (11.9%) had reviewed 2 to 4 articles per month. Forty-nine reviewers (73.1%) worked for academic organizations.

Survey respondents answered questions regarding their motivation to review articles, the need for prior experience with a particular journal to review articles for that journal, the benefits from reviewing articles, and the potential rewards for reviewing articles (Table 1). In general, the survey respondents thought that reviewing manuscripts represented a responsibility and was part of their roles as researchers. Often, they were interested in the topic. Benefits associated with reviewing included safeguarding academic quality and integrity, supporting other researchers as authors, and contributing to the field and the quality of the manuscripts.

Table 1.

Reviewers’ motivations, influences, perceived benefits, and preferred rewards

Question	Responses (%)
What motivates you to agree to review an article?a
Having a sense of responsibility since others review your submissions	40 (60%)
Viewing it as part of your role as a researcher	38 (57%)
Having interest in the topic of the article	38 (51%)
Being connected with the journal	26 (39%)
Seeing what is new in your field	17 (25%)
Receiving rewards/credit for reviewing	6 (9%)
What previous experience with a journal influences your decision on whether to agree to review?a
Having previous experience reviewing for the journal	24 (36%)
Not applicable; previous experience with the journal is not usually relevant to my decision	20 (30%)
Having previous experience publishing in the journal	18 (27%)
Knowing the editor	4 (6%)
In what ways do you think reviewing an article benefits you?a
Safeguarding academic quality and integrity	43 (64%)
Having an opportunity to support other researchers/authors	33(49%)
Having an opportunity to contribute to the field	32(48%)
Having an opportunity to improve the article	29 (43%)
Being introduced to new ideas or topics	16 (24%)
Gaining updated information on the topic	12 (18%)
Stimulating your thinking about ideas for possible publication or research projects	12 (18%)
What rewards would help you feel most valued as a reviewer?b
Receiving academic credit for review work, e.g., reviews recorded on Publons	32 (48%)
Having your name included in a published list thanking all reviewers (not revealing which articles were reviewed)	30 (45%)
Receiving feedback from the editor on your contribution as a reviewer	29 (43%)
Gaining free access to journal subscription content	17 (25%)
Receiving honoraria payments for reviewing	11 (16%)
Receiving a certificate for reviewing	10 (15%)
Other	9 (13%)

^a 67 respondents answered this question.

^b 59 respondents answered this question.

DISCUSSION

The reviewers for Baylor University Medical Center Proceedings generally worked for academic organizations, had done this work for more than 10 years, often had published more than 20 articles, and had done multiple reviews for this journal and other journals. Many reviewers thought this activity represented both a responsibility and an opportunity. In particular, they thought they could help other authors, help maintain the integrity of research, and possibly improve the quality of the research project under review. Also, they had the opportunity to learn new ideas that might enhance their own research activities. Finally, recognition, academic credit, and feedback from editors were much more highly valued by reviewers than monetary compensation.

Other publishing companies have surveyed reviewers to try to learn the motivation to do reviews and the reasons for declining reviews. IOP Publishing surveyed over 1200 researchers who had either performed a review or had been invited to do a review for an IOP journal between January 2018 and March 2020.² They did not report the number of responses, and the information was largely presented as percentages. The most important motivating factors to participate in a review included interest in the paper, journal reputation, and engagement with the scholarly community. Most reviewers would like to receive recognition from the journal for which they had completed reviews but did not think that cash benefits would provide much motivation for reviewing. Some reviewers would like to receive information about other reviewer comments, feedback on the final decision regarding the paper, and feedback on the quality of the review. The latter would increase work of the editors and seems relatively unlikely.

Inga Vesper published a news item in Nature News in September 2018 about the challenges of peer review.³ This summary reported the results from a global survey of more than 11,000 peer reviewers throughout the world conducted between May and July 2018 and also used information obtained from Publons, the website used to track academic reviews. This report indicated that editors now need to send out an increased number of review invitations to obtain a completed review. For example, in 2017, 40 million reviews were requested throughout the world. The median review time was 16.4 days and the median number of hours spent reviewing was 5. Ten percent of the reviewers were responsible for 50% of the peer reviews; 41% of the survey respondents thought peer reviews were part of their job.

Tite and Schoter analyzed responses from 551 peer reviewers from five journals associated with BMJ.⁴ The most important reasons for accepting a review were “the opportunity to learn something new from the paper,” “the contribution of the paper to the subject area,” and “relevance of the topic to my own work or interest.” The most important reason for declining a review was “conflicts with other workload.” These reviewers did not think that financial rewards would improve reviewer response rates. Rather, they thought motivating incentives would include free access to the journal, annual acknowledgment of the reviewers on the journal website, feedback about the outcome of the manuscript submission and the quality of the reviews, and appointment of the best reviewers to the journal editorial board.

This project involving peer reviewers for Baylor University Medical Center Proceedings has several limitations. The overall response rate from reviewers was relatively low, and these respondents may have different attitudes and preferences than the average reviewer. In addition, the surveys were sent to the reviewers of one journal that publishes articles in the broad field of medicine. Consequently, these results may not be generalizable to other journals, especially subspecialty journals.

Overall, this study and other reports indicate that the number of reviews needed to manage current manuscript submissions has significantly increased over time. Editors have increasing difficulty identifying reviewers who will complete a review in a reasonable timeframe. A small number of reviewers provide most of the reviews for many medical journals. What should journals do? The most productive pool of reviewers will likely include members of the editorial board, authors who have published manuscripts in the journal, and reviewers with a record of completing reviews in a timely manner. There has been an effort to engage early career researchers in peer review.⁵ These include programs specifically designed to prepare graduate students and early career researchers to be peer reviewers, such as the Certified Peer Reviewer Course offered by the Elsevier Researcher Academy⁶ and the Excellence in Peer Review course by Taylor & Francis.⁷ Early career researchers have expressed a variety of concerns about peer review, including difficulty in evaluating statistics, writing the review report, and determining if findings are novel,⁸ most of which could be reasonably addressed through training and mentorship. Journals could also provide feedback to reviewers, including other reviewer comments and the completed PDF of an accepted article. This could help reviewers recognize the importance of their contributions and their involvement in a broader academic community. Finally, published recognition of reviewers by journals and reviewer portfolios (such as those curated by Publons) could satisfy researchers who want acknowledgment for their reviews. These websites can make peer reviews a quantifiable academic activity. Since academic output, often measured by publications and funded grants, provides a significant measurement for academic promotion, academic organizations should also promote review activities as an important responsibility of their faculty and give credit during promotion evaluations.

Disclosure statement

No potential conflict of interest was reported by the author(s).