Abstract
Background
Peer reviewers are usually unpaid and their efforts not formally acknowledged. Some journals have difficulty finding appropriate reviewers able to complete timely reviews, resulting in publication delay.
Objectives and methods
A survey of peer reviewers from five biomedical journals was conducted to determine why reviewers decline to review and their opinions on reviewer incentives. Items were scored on 5‐point Likert scales, with low scores indicating low importance or low agreement.
Results
551/890 (62%) reviewers responded. Factors rated most highly in importance for the decision to accept to review a paper included contribution of the paper to subject area (mean 3.67 (standard deviation (SD) 86)), relevance of topic to own work (mean 3.46 (SD 0.99)) and opportunity to learn something new (mean 3.41 (SD 0.96)). The most highly rated factor important in the decision to decline to review was conflict with other workload (mean 4.06 (SD 1.31)). Most respondents agreed that financial incentives would not be effective when time constraints are prohibitive (mean 3.59 (SD 1.01)). However, reviewers agreed that non‐financial incentives might encourage reviewers to accept requests to review: free subscription to journal content (mean 3.72 (SD 1.04)), annual acknowledgement on the journal's website (mean 3.64 (SD 0.90)), more feedback about the outcome of the submission (mean 3.62 (SD 0.88)) and quality of the review (mean 3.60 (SD 0.89), and appointment of reviewers to the journal's editorial board (mean 3.57 (SD 0.99)).
Conclusion
Reviewers are more likely to accept to review a manuscript when it is relevant to their area of interest. Lack of time is the principal factor in the decision to decline. Reviewing should be formally recognised by academic institutions and journals should acknowledge reviewers' work.
Peer review is an important part of scientific publishing, but has been criticised for being ineffective, slow and expensive.1,2,3 Reviewers are usually unpaid and their efforts not formally acknowledged, but journal editors depend on them to inform publication decisions and to improve the quality of manuscripts. Editors of some journals have frequent difficulty finding appropriate reviewers who are able to complete timely reviews. This can result in publication delay.
Only a few empirical studies to date have attempted to find ways of improving quality of peer reviews.4,5,6 Tested interventions include the provision of written feedback,4 attendance at training workshops5,6 and the use of self‐taught training materials,6 but none resulted in improvements in review quality. Other studies have evaluated methods of improving the timeliness of reviews.7,8 Pitkin and Burmeister7 conducted a trial and found that requesting permission from reviewers before sending manuscripts for review led to a higher number of refusals than when manuscripts were sent before seeking permission. However, those who agreed to review after a request completed their reviews more quickly than those who had not been initially asked. In another study they found that contacting late reviewers resulted in a review within 7 days in about two thirds of cases, whether contact was by telephone, fax or email.8
We conducted a survey of reviewers for five biomedical journals to determine the reasons why they decline to review, and their opinions on reviewer incentives.
Methods
Questionnaire development
The questionnaire content was developed from two sources.
Interviews with reviewers
We conducted 28 semistructured telephone interviews with reviewers from four biomedical journals owned by the BMJ Publishing Group to determine the range of reasons why reviewers decline requests to review and what rewards might encourage reviewers. We used a random sample of reviewers for the journals Archives of Disease in Childhood, BMJ, British Journal of Ophthalmology and Sexually Transmitted Infections, who had declined a request to review more than once during the previous 12 months. A series of open‐ended questions was asked about the most recent occasion respondents accepted or declined to review for the journal during the study period. The questions covered factors important in accepting and declining to review, and opinions and preferences for reviewer incentives. Responses were recorded by the interviewer at the time of interview and later tabulated.
Reviewer feedback
The BMJ Publishing Group's online reviewing system asks reviewers to state a reason each time they decline a request to review. On the basis of recurrent responses, we developed a set of key reasons.
Questionnaire construction
Items from both sources were grouped into four broad questionnaire themes: reasons why reviewers decline to review, reasons why reviewers accept to review, opinions on financial incentives and opinions on non‐financial incentives. All items were constructed as 5‐point Likert scales, with low scores indicating low importance or low agreement. The questionnaire was piloted and refined before use in the survey.
Survey
After receiving approval for the study from the BMJ Ethics Committee, we generated a list of all invited reviewers for the journals Archives of Disease in Childhood, BMJ, Emergency Medicine Journal, Gut and Journal of Epidemiology & Community Health for a 9‐month period. We divided reviewers for each of the journals into strata on the basis of the number of times they had declined or accepted a request to review during this study period. We then randomly selected 200 reviewers from across the strata from the samples of each journal using the random number generator in MS Excel, giving a total sample of 1000. Invitations to take part in the survey were sent by email, including a link to access the online survey. Reminders were sent by email to non‐responders at weeks 2 and 5 after the initial mail.
Results
Response rates
We received 110 automated responses from invalid email addresses. We received a response from 606 of the 890 (68%) remaining valid email addresses. Fifty five respondents completed only the opening questions of the survey and were excluded from the analysis. We analysed results for the five journals individually (Archives of Disease in Childhood, n = 115; BMJ; n = 133; Emergency Medicine Journal, n = 101; Gut, n = 90; Journal of Epidemiology & Community Health, n = 112). However, results across journals were broadly similar, so we present only the overall findings from the five journals combined (n = 551).
Respondent characteristics
Of the 551 respondents, 30% (n = 386) of the reviewers were women, 71% (n = 389) were aged <55 years, 66% (n = 362) were academics and 88% (n = 486) were active writers of research articles.
Reasons for accepting to review
Data are presented as the mean (standard deviation (SD)) of responses on a 5‐point Likert scale.
In all, 429 respondents reported they had accepted a request to review during the study period. Factors rated most highly in importance to the decision to accept a paper to review included contribution of the paper to the subject area (3.67 (0.86)), relevance of the topic to own work/interests (3.46 (0.99)) and opportunity to learn something new from the paper (3.41 (0.96); table 1). A sense of professional duty and the reputation of the journal were also considered important factors. Factors rated as least important were monetary payment (BMJ reviewers only, 1.64 (0.76)), the reputation of the authors of the paper (1.78 (0.95)) and academic reward such as career enhancement (1.81 (0.96)).
Table 1 Reasons for accepting to review (all journals, n = 429).
| How important was each of the following factors in your decision to accept to review?* | Very or extremely important n (%) | Moderately important n (%) | Not at all or slightly important n (%) | Mean (SD) |
|---|---|---|---|---|
| The opportunity to learn something new from the paper | 214 (50) | 146 (34) | 67 (16) | 3.41 (0.96) |
| The contribution of the paper to the subject area | 259 (60) | 136 (32) | 32 (7) | 3.67 (0.86) |
| Relevance of the topic to my own work or interests | 219 (51) | 144 (34) | 64 (15) | 3.46 (0.99) |
| Desire to keep up‐to‐date on current research | 179 (42) | 148 (35) | 99 (23) | 3.24 (1.01) |
| Academic reward (eg, career enhancement) | 15 (35) | 85 (20) | 325 (76) | 1.81 (0.91) |
| Sense of professional duty | 210 (49) | 138 (32) | 77 (18) | 3.38 (0.98) |
| Reputation of the journal | 201 (47) | 155 (36) | 69 (16) | 3.36 (0.99) |
| Reputation of the authors of the paper | 22 (5) | 76 (18) | 321 (75) | 1.78 (0.95) |
| Being able to use the online review system | 117 (27) | 99 (23) | 184 (43) | 2.62 (1.22) |
| Monetary payment (BMJ reviewers only)† | 1 (1) | 12 (13) | 74 (78) | 1.64 (0.76) |
Items are presented in the order they appeared in the questionnaire.
Percentages do not sum to 100% because of missing data.
*Responses on a 5‐point Likert scale: 1, not at all important; 2, slightly important; 3, moderately important; 4, very important; and 5, extremely important.
†Percentages based on number of BMJ respondents who had completed a review in the study period (n = 95).
Reasons for declining to review
During the study period, 258 reviewers reported that they had declined a request to review. The most highly rated factor important in the decision to decline to review was conflict with other workload (4.06 (1.31), table 2). Having too many reviews for other journals (2.54 (1.45)) and a tight deadline for completing the review (2.70 (1.32)) were also important in the decision to decline. Factors rated as least important were having previously reviewed many papers on the same subject (1.26 (0.59)), length of the manuscript (1.29 (0.63)), comments not taken into account in the past reviewing experience (1.34 (0.81)), lack of formal recognition of reviewer contribution (1.38 (0.77)) and having to use the online review system (1.39 (0.92)).
Table 2 Reasons for declining to review (all journals, n = 258).
| How important was each of the following factors in your decision to decline to review?* | Very or extremely important n (%) | Moderately important n (%) | Not at all or slightly important n (%) | Mean (SD) |
|---|---|---|---|---|
| Insufficient interest in the paper | 53 (21) | 54 (21) | 148 (57) | 2.26 (1.26) |
| Having to review too many manuscripts for this journal | 26 (10) | 31 (12) | 194 (75) | 1.76 (1.11) |
| Having to review too many manuscripts for other journals | 76 (29) | 47 (18) | 130 (50) | 2.54 (1.45) |
| Length of the manuscript | 2 (1) | 17 (7) | 231 (90) | 1.29 (0.63) |
| Quality of the manuscript | 21 (8) | 26 (10) | 204 (79) | 1.63 (1.01) |
| Tight deadline for completing the review | 77 (30) | 62 (24) | 113 (44) | 2.70 (1.32) |
| Conflicts with other workload | 197 (76) | 23 (9) | 33 (13) | 4.06 (1.31) |
| Having conflicting interests | 38 (15) | 20 (8) | 190 (74) | 1.79 (1.29) |
| Knowing someone more appropriate to review the manuscript | 35 (14) | 37 (14) | 174 (67) | 1.99 (1.26) |
| Having previously reviewed several papers on the same topic | 3 (1) | 9 (4) | 223 (86) | 1.26 (0.59) |
| Comments not taken into account in the past reviewing experience | 8 (3) | 13 (5) | 210 (81) | 1.34 (0.81) |
| Concern that subsequent requests to review could become burdensome | 13 (5) | 32 (12) | 185 (72) | 1.66 (0.93) |
| Dislike of open peer review process (BMJ reviewers only)† | 2 (3) | 4 (7) | 42 (70) | 1.40 (0.89) |
| Reputation of the journal | 11 (4) | 15 (6) | 202 (78) | 1.40 (0.86) |
| Lack of formal recognition of reviewer contribution | 8 (3) | 14 (5) | 207 (80) | 1.38 (0.77) |
| Having to use the online review system | 12 (5) | 16 (6) | 201 (78) | 1.39 (0.92) |
| Absence from work | 48 (19) | 20 (8) | 157 (61) | 2.02 (1.44) |
| Delay in accessing the manuscript | 24 (9) | 13 (5) | 187 (72) | 1.60 (1.09) |
Items are presented in the order they appeared in the questionnaire.
Percentages do not sum to 100% because of missing data.
*Responses on a 5‐point Likert scale: 1, not at all important; 2, slightly important; 3, moderately important; 4, very important; and 5, extremely important.
†Percentages based on the number of BMJ respondents who had not declined a request to review in the study period (n = 60).
Opinions on financial incentives
The survey yielded more agreement than disagreement that financial incentives will not be effective when time constraints are prohibitive (3.59 (1.01)) and that small financial incentives would not encourage reviewers to accept reviews (3.53 (0.95)), but many reviewers agreed that financial incentives encourage reviewers to accept requests to review (3.39 (0.86), table 3). There was little agreement that financial incentives could improve the quality of reviews (2.42 (0.98)), but responses to the item “financial incentives could compromise the quality of review” were mixed (3.08 (1.00)).
Table 3 Financial incentives to review (all journals, n = 551).
| How much do you agree with the following statements about financial incentives?* | Agree or strongly agree n (%) | Neither agree nor disagree n (%) | Disagree or strongly disagree n (%) | Mean (SD) |
|---|---|---|---|---|
| Financial incentives bias which journals referees review for | 254 (46) | 148 (27) | 137 (25) | 3.25 (1.06) |
| Financial incentives will not be effective when time constraints are prohibitive | 340 (62) | 94 (17) | 102 (19) | 3.59 (1.01) |
| Small financial incentives alone would not encourage reviewers to accept reviews | 331 (60) | 102 (19) | 106 (19) | 3.53 (0.95) |
| Financial incentives imply a contractual obligation that reviewers would be more likely to meet | 297 (54) | 132 (24) | 110 (20) | 3.36 (0.91) |
| Financial incentives encourage reviewers to accept requests to review | 298 (54) | 148 (27) | 91 (17) | 3.39 (0.86) |
| Financial incentives could improve the quality of reviews | 87 (16) | 130 (24) | 320 (58) | 2.42 (0.98) |
| Financial incentives could compromise the quality of reviews | 194 (35) | 175 (32) | 166 (30) | 3.08 (1.00) |
Items are presented in the order they appeared in the questionnaire.
Percentages do not sum to 100% as a result of missing data.
*Responses on a 5‐point Likert scale: 1, strongly disagree; 2, disagree; 3, neither agree nor disagree; 4, agree; and 5, strongly agree.
Opinions on non‐financial incentives
We found highest agreement that the following incentives would encourage reviewers to accept requests to review: free access or subscription to journal content (3.72 (1.04)); annual acknowledgement on the journal's website (3.64 (0.90)); more feedback about the outcome of the submission (3.62 (0.88)) and quality of the review (3.60 (0.89)); and appointment of reviewers to the journal's editorial board (3.57 (0.99), table 4). There was less agreement that publication of the review with the article (2.62 (1.09)), the option of adding the reviewer's name at the end of the published article (2.77 (1.10)) and gifts would act as incentives for reviewers.
Table 4 Incentives to review (all journals, n = 551).
| How much do you agree that the following incentives would encourage reviewers to accept requests to review?* | Agree or strongly agree n (%) | Neither agree nor disagree n (%) | Disagree or strongly disagree n (%) | Mean (SD) |
|---|---|---|---|---|
| Small financial incentives, eg, <£50 | 226 (41) | 123 (22) | 186 (34) | 3.03 (1.02) |
| Consultancy‐equivalent fee for time spent | 317 (58) | 97 (18) | 117 (21) | 3.45 (1.08) |
| Substantial financial incentives only | 171 (31) | 126 (23) | 234 (42) | 2.85 (1.11) |
| Free access or subscription to this journal | 389 (71) | 63 (11) | 86 (16) | 3.72 (1.04) |
| More feedback from the editor about the quality of the review | 337 (61) | 127 (23) | 70 (13) | 3.60 (0.89) |
| More feedback from the editor about the outcome of the submission | 347 (63) | 116 (21) | 71 (13) | 3.62 (0.88) |
| Token gift, eg, compact disc after each review | 172 (31) | 135 (25) | 229 (42) | 2.79 (1.06) |
| Annual gift for the most regular or best reviewers | 157 (28) | 133 (24) | 245 (44) | 2.74 (1.08) |
| Appointment of best reviewers to the journal's editorial board | 338 (61) | 107 (19) | 86 (16) | 3.57 (0.99) |
| Option of adding reviewer's name at the end of the published paper | 156 (28) | 129 (23) | 228 (41) | 2.77 (1.10) |
| Published acknowledgement of reviewer's contribution to the manuscript | 216 (39) | 124 (23) | 166 (30) | 3.06 (1.08) |
| Publication of the review with the article | 128 (23) | 116 (21) | 263 (48) | 2.62 (1.09) |
| Annual acknowledgement of all reviewers on the journal's website | 342 (62) | 103 (19) | 59 (11) | 3.64 (0.90) |
Items are presented in the order they appeared in the questionnaire.
Percentages do not sum to 100% because of missing data.
*Responses on a 5‐point Likert scale: 1, strongly disagree; 2, disagree; 3, neither agree nor disagree; 4, agree; and 5, strongly agree.
Discussion
Reviewers were more likely to accept a request to review a manuscript when the paper was relevant to their area of interest. Lack of time was the principal factor in the decision to decline. Participants regarded reviewing as an opportunity to learn something new in their area of interest and as part of their professional duty. They were not of the opinion that small financial incentives would be effective, but were favourable to the idea of some non‐financial incentives. The most motivating incentives included free access or subscriptions to the journal, annual acknowledgement of reviewers on the journal's website, feedback about the outcome of the manuscript submission and the quality of their reviews, and the appointment of the best reviewers to the journal's editorial board.
Study implications
Peer review is an important and accepted part of scientific publishing and has become an indicator of perceived journal quality. Many reviewers regard journal peer review as part of their professional responsibility. However, academic institutions do little to train reviewers, and expect staff to review in addition to their formal workload. Although journal editors rely on reviewers to improve the quality of manuscripts and to inform decision making regarding the publication, reviewers are usually unpaid and their efforts not formally acknowledged by the journal. Interestingly, 80% of participants said that lack of formal recognition was not an important factor in their decision to decline. Some journals now offer reviewers continuing medical education or continuing professional development credits, but these are relevant only to those in medical posts.
Reviewers in our study indicated that they were not motivated by financial incentives, possibly because they saw reviewing as part of their academic role and were not usually paid for reviewing. Most journals cannot afford to pay reviewers consultancy rates for their time, but there are some inexpensive options that may act as acknowledgement or incentive for reviewers. For example, more journals could post annual acknowledgement lists of their reviewers on their websites, send letters of thanks to their most valued and regular reviewers, appoint their best reviewers to their editorial boards and give more feedback about the editorial outcome of the papers reviewed. However, although reviewers were favourable to the idea of feedback about their reviews, this would be resource intensive for journals.
Peer review has traditionally been a closed “black box” process. Some journals are considering a move to open peer review where the reviewers of published manuscripts are acknowledged in the publication and have the option of their review being made openly available. Reviewers in this study were divided over the idea of a published acknowledgment of the reviewer's contribution and less keen on the publication of the review in the journal. Further research is necessary to gauge reviewers' perceptions and concerns before switching to an open‐review process.
What is already known on this topic
Peer review is central to the publishing process, yet reviewers' efforts are not formally acknowledged.
Some journals have difficulty finding appropriate reviewers who are able to complete timely reviews, thus resulting in publication delay.
Little is known about why reviewers decline or accept invitations to review and which incentives are likely to motivate them to review.
What this study adds
Respondents regarded reviewing as an opportunity to learn something new in their area of interest and as part of their professional duty. Lack of time was the principal factor in the decision to decline.
Reviewers were not of the opinion that small financial incentives would be effective, but were favourable to the idea of some non‐financial incentives.
Policy implications
Reviewing should be formally recognised by academic institutions, and journals should formally, and perhaps publicly, acknowledge the contribution of their reviewers.
With an increasing number of submissions to journals and demand on reviewers' time, it is becoming increasingly important to solicit reviewers appropriately. Journals should remind reviewers to update their personal profiles, detailing their areas of expertise to ensure that they receive appropriate review requests and are not sent more than the agreed quota.
Study limitations
Our study was limited to a sample of reviewers with internet and email access as our survey was administered online. It is possible that reviewers who use electronic review systems have views different from those who use traditional paper‐based reviewing. However, all journals published by the BMJ Publishing Group now require reviewers to use the journals' online reviewing system, so respondents were representative of reviewers for these journals. Many other journals now use electronic tracking systems. It is also possible that reviewers working in specialties not covered by the journals in this study may have additional considerations when choosing whether or not to review for a journal. We deliberately selected journals with international reviewers and from different specialties, as well as a general medical journal, to increase the generalisability of findings across biomedical reviewers. Unfortunately personal characteristics of reviewers such as age, sex and reviewing experience are not recorded routinely on the manuscript tracking databases, so we were unable to compare responders with non‐responders in terms of these factors.
Conclusion
Reviewing should be formally recognised by academic institutions, and journals should formally, and perhaps publicly, acknowledge the contribution of their reviewers.
Acknowledgements
We thank all the reviewers for taking part in the survey.
Footnotes
Funding: This research was funded by the BMJ Publishing Group's research budget. The study sponsor did not play a role in the study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the paper for publication.
Competing interests: LT and SS are employed by the BMJ Publishing Group.
Ethical approval: The BMJ Ethics Committee approved the study.
Contributors: LT and SS designed the study and developed the questionnaire. LT coordinated data collection. Both LT and SS analysed the data, wrote the paper and approved the final version. SS will act as guarantor.
References
- 1.Rennie D. Editorial peer review: its development and rationale. In: Godlee F, Jefferson T, eds. Peer review in health sciences. London: BMJ Books, 1999
- 2.Smith R. Peer review: Reform or revolution? BMJ 1997315759–760. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Kassirer J P, Campion E W. Peer review: crude and understudied, but indispensable. JAMA 199427296–97. [DOI] [PubMed] [Google Scholar]
- 4.Callaham M L, Knopp R K, Gallagher E J. Effect of written feedback by editors on quality of reviews. JAMA 20022872781–2783. [DOI] [PubMed] [Google Scholar]
- 5.Callaham M L, Wears R L, Waeckerle J F. Effect of attendance at a training session on peer reviewer quality and performance. Ann Emerg Med 199832318–322. [DOI] [PubMed] [Google Scholar]
- 6.Schroter S, Black N, Evans S.et al Effects of training on the quality of peer review: a randomised controlled trial. BMJ 2004328673–675. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Pitkin R M, Burmeister L F. Identifying manuscript reviewers: randomized comparison of asking first or just sending. JAMA 20022872795–2796. [DOI] [PubMed] [Google Scholar]
- 8.Pitkin R M, Burmeister L F. Prodding tardy reviewers: a randomized comparison of telephone, fax and email. JAMA 20022872794–2795. [DOI] [PubMed] [Google Scholar]