Abstract
Objectives
To assess whether reports from reviewers recommended by authors show a bias in quality and recommendation for editorial decision, compared with reviewers suggested by other parties, and whether reviewer reports for journals operating on open or single-blind peer review models differ with regard to report quality and reviewer recommendations.
Design
Retrospective analysis of the quality of reviewer reports using an established Review Quality Instrument, and analysis of reviewer recommendations and author satisfaction surveys.
Setting
BioMed Central biology and medical journals. BMC Infectious Diseases and BMC Microbiology are similar in size, rejection rates, impact factors and editorial processes, but the former uses open peer review while the latter uses single-blind peer review. The Journal of Inflammation has operated under both peer review models.
Sample
Two hundred reviewer reports submitted to BMC Infectious Diseases, 200 reviewer reports submitted to BMC Microbiology and 400 reviewer reports submitted to the Journal of Inflammation.
Results
For each journal, author-suggested reviewers provided reports of comparable quality to non-author-suggested reviewers, but were significantly more likely to recommend acceptance, irrespective of the peer review model (p<0.0001 for BMC Infectious Diseases, BMC Microbiology and the Journal of Inflammation). For BMC Infectious Diseases, the overall quality of reviewer reports measured by the Review Quality Instrument was 5% higher than for BMC Microbiology (p=0.042). For the Journal of Inflammation, the quality of reports was the same irrespective of the peer review model used.
Conclusions
Reviewers suggested by authors provide reports of comparable quality to non-author-suggested reviewers, but are significantly more likely to recommend acceptance. Open peer review reports for BMC Infectious Diseases were of higher quality than single-blind reports for BMC Microbiology. There was no difference in quality of peer review in the Journal of Inflammation under open peer review compared with single blind.
Keywords: Peer review, Journalology, Review Quality Instrument
Strengths and limitations of this study.
This is the first study to compare quality of peer review between two journals that are very similar in most aspects but differ in peer review model (BMC Microbiology operates single-blind peer review while BMC Infectious Diseases operates open peer review).
It is the first study to analyse the effect of change in peer review model across a single journal (Journal of Inflammation).
A large number of reviewer reports were analysed (800), resulting in a well-powered study in three different journals publishing research in both biology and medicine.
We observed moderate agreement between raters for peer review report quality, although the raters did not discuss the rating for BMC Infectious Diseases and BMC Microbiology, and did not confer about their ratings for individual reports for any journal.
We had to use a different study design for the Journal of Inflammation section of the study than that used for the BMC Infectious Diseases and BMC Microbiology section, due to insufficient number of manuscripts fulfilling the inclusion criteria.
Introduction
Most scholarly journals operate one of three types of peer review models: single blind, where the reviewers know the identity of the authors but not vice versa; open peer review, where authors and reviewers both know each other's identity, or double-blind peer review where the author and reviewer names are both blinded. Some journals publish reviewer reports (signed or anonymous) and authors’ responses together with accepted articles. Many journals allow, or in some cases require, authors who submit manuscripts to suggest potential reviewers. This information is available to editors who are responsible for selecting appropriate reviewers.
Previous studies have compared quality of reviewer reports under open and single-blind peer review across various journals,1 and analysed open peer review,2 public peer review,3 the proportion of authors who suggested reviewers4 and differences between recommendations by reviewers who were either suggested or excluded by the authors.5 It has been found that the reviewers chosen by editors are statistically-significantly more critical than those suggested by the authors.1 3–9 Intriguingly, the majority of the studies have been conducted on medical journals and there are no studies on biology journals.
The objective of this study was to assess whether the reviewers suggested by the authors are biased in their assessment of a manuscript, and whether the quality of their reports is different from reports prepared by reviewers suggested by editors. A second objective was to compare open peer review with single-blind peer review. This study was the first to analyse a large number of reviewer reports (800 in total) in both medical and biology journals, and compare quality of reviewer reports between open and single-blind peer review models and between author-suggested and non-author-suggested reviewers.
The initial findings from this research were presented in poster format at the 7th Peer Review Congress in Chicago in 2013. The poster has been deposited on F1000 Posters.10
Methods
Analysed journals
We analysed peer review reports for manuscripts submitted in 2010–2011 to BMC Microbiology, which operates single-blind peer review, and BMC Infectious Diseases, which operates open peer review. BMC Microbiology and BMC Infectious Diseases were launched in 2000 as part of the BMC series of journals.11 All policies and processes are the same across the series of journals. BMC Microbiology and BMC Infectious Diseases are similar in terms of size, impact factors and rejection rates, and cover similar subject areas (table 1). The journals are managed by a team of in-house editors who work very closely with their editorial boards. Manuscripts are handled by academic associate editors and section editors, who select and check all invited reviewers. They may decide to use the reviewers suggested by the authors, by other reviewers, or by BioMed Central's PubMed search tool, or select reviewers based on their own knowledge and searches. The reviewers receive the same invitations to review and the same templates to prepare their reports. The only difference is that for BMC Infectious Diseases reviewers must agree to open peer review, including the publication of their signed reports if the manuscript is accepted.
Table 1.
Information about BMC Microbiology and BMC Infectious Diseases in the period covered by this research
| BMC Microbiology | BMC Infectious Diseases | |
|---|---|---|
| Impact factor 2012 | 3.10 | 3.03 |
| Number of articles published in 2012 | 307 | 386 |
| Rejected submissions in 2010 and 2011, % | 52.5 | 55.5 |
| Peer review model | Single blind | Open |
The third analysed journal was the Journal of Inflammation, which was launched in 2004. It originally operated under an open peer review model, but adopted a single-blind peer review policy on 29 January 2010 (table 2). The study period was from 2007–2011.
Table 2.
The Journal of Inflammation in the period covered by this research
| Journal of Inflammation | 2007–2009 (open peer review) | 2010–2011 (single-blind peer review) |
|---|---|---|
| Impact factor | None | 2.017–2.263 |
| Number of articles published | 58 | 150 |
| Rejected submissions, % | 33 | 50 |
Selection of reviewer reports
For each journal, we analysed reviewer reports on manuscripts presenting original research that had a final decision (accept or reject). In each of BMC Microbiology and BMC Infectious Diseases, we identified 100 manuscripts that had two reviewers each, one suggested by the authors and one by another party (BioMed Central's PubMed search tool comparing the abstract of the manuscript to abstracts in PubMed, another reviewer or editor). This was achieved by searching the journal's database for consecutive submissions in the analysed period 2010–2011. The Journal of Inflammation published fewer original research articles and a larger proportion of review articles and case reports that were not suitable for our analysis. In the Journal of Inflammation, we analysed 200 reviewer reports for research manuscripts submitted consecutively counting back from the date of change of peer review model (under open peer review), and 200 counting forward (under single-blind peer review). There was an insufficient number of research article submissions that had two reviewers, one suggested by the authors and one by another party. We analysed 193 manuscripts, in total, that had between one and five reviewers. As a result, for the Journal of Inflammation, we analysed reports spanning a longer period of time (2007–2011), and the number of reports provided by author-suggested reviewers was not equal to that provided by non-author-suggested reviewers (table 3).
Table 3.
Numbers of reports analysed in each journal
| Open peer review |
Single-blind peer review |
|||
|---|---|---|---|---|
|
BMC Microbiology 100 manuscripts analysed |
NA | 100 reports from reviewers nominated by authors | 100 reports from reviewers not nominated by authors | |
|
BMC Infectious Diseases 100 manuscripts analysed |
100 reports from reviewers nominated by authors | 100 reports from reviewers not nominated by authors | NA | |
|
Journal of Inflammation 193 manuscripts analysed |
50 reports from reviewers nominated by authors | 150 reports from reviewers not nominated by authors | 29 reports from reviewers nominated by authors | 171 reports from reviewers not nominated by authors |
NA, not applicable.
Assessing quality of reviewer reports
Each peer review report was rated using an established Review Quality Instrument (RQI)12 (table 4). Each report was rated separately and independently by two senior members of the editorial staff at BioMed Central. The peer review model and whether the reviewer was author suggested was unknown to the raters. However, the raters were not blinded to the reviewers’ identity. When rating the quality of reports for BMC Microbiology and BMC Infectious Diseases, formal discussions were not held between the raters on how to use the RQI. However, before analysing reports for the Journal of Inflammation, the raters agreed on the criteria for each score on the RQI before rating the peer review reports (see Discussion section).
Table 4.
Reproduction of the Review Quality Instrument (RQI).12
| RQI | ||||
|---|---|---|---|---|
| Q1. Did the reviewer discuss the importance of the research question? | ||||
| 1 | 2 | 3 | 4 | 5 |
| Not at all | Discussed extensively | |||
| Q2. Did the reviewer discuss the originality of the paper? | ||||
| 1 | 2 | 3 | 4 | 5 |
| Not at all | Discussed extensively with references | |||
| Q3. Did the reviewer clearly identify the strengths and weaknesses of the method (study design, data collection and data analysis)? | ||||
| 1 | 2 | 3 | 4 | 5 |
| Not at all | Comprehensive | |||
| Q4. Did the reviewer make specific useful comments on the writing, organisation, tables and figures of the manuscript? | ||||
| 1 | 2 | 3 | 4 | 5 |
| Not at all | Extensive | |||
| Q5. Were the reviewer’s comments constructive? | ||||
| 1 | 2 | 3 | 4 | 5 |
| Not at all | Very constructive | |||
| Q6. Did the reviewer supply appropriate evidence using examples from the paper to substantiate their comments? | ||||
| 1 | 2 | 3 | 4 | 5 |
| None substantiated | Some substantiated | All substantiated | ||
| Q7. Did the reviewer comment on the author’s interpretation of the results? | ||||
| 1 | 2 | 3 | 4 | 5 |
| Not at all | Discussed extensively | |||
| Q8. How would you rate the tone of the review? | ||||
| 1 | 2 | 3 | 4 | 5 |
| Abusive | Courteous | |||
Reproduced with permission, License Number: 3617630208550.
Analysis of reviewer recommendations
For the journals studied, peer reviewers could choose one of six recommendations suggested in the reviewer form, or choose not to provide recommendation. To facilitate the analysis, we grouped these recommendations into four categories (table 5).
Table 5.
Grouping of reviewer recommendations for the purpose of our analysis
| Recommendations provided in the reviewer form | Recommendations grouped for our analysis |
|---|---|
| Accept without revision | Accept |
| Accept after discretionary revisions | |
| Accept after minor essential revisions | |
| Unable to decide on acceptance or rejection until the authors have responded to the major compulsory revisions | Revise |
| Reject because too small an advance to publish | Reject |
| Reject because scientifically unsound | |
| No recommendation | No recommendation |
Author surveys
All corresponding authors were asked to complete author surveys after publication of their article (see online supplementary material). The survey consists of 18 questions related to different aspects of the editorial and production processes, of which one is, ‘How helpful were the reviewers’ comments?’. The authors were asked to score their answers to the questions on a scale from 1 (very poor) to 5 (very good). We compared the authors’ ratings for the question on the helpfulness of peer review comments compared to scores for other questions.
Statistical analyses
It was previously shown1 that a total of 110 papers gives 90% power at p<0.05 to detect a difference in RQI of one-half its SD; we therefore considered our sample sizes to be well powered.
Inter-rater agreement was measured by weighted κ with quadratic weights. For each article and reviewer, the mean of the two rater scores was then used in subsequent analyses. Review quality was compared between open and single-blind review models using the unpaired Mann-Whitney U test. Review quality was compared between author suggested and other reviewers, using the paired Mann-Whitney U test in BMC Microbiology and BMC Infectious Diseases (in which each article had one reviewer of each type), and using the unpaired test in the Journal of Inflammation. Bonferroni correction was applied for eight tests in each journal.
To test whether author-suggested reviewers were more positive than other reviewers, we coded reviewer recommendations as reject=0, revise=1, or accept=2, and treated this value as the response in linear regression with reviewer type (author suggested or other) as the predictor. Interaction terms with review model (open or single blind) were included in exploratory analysis but dropped if found to be non-significant.
Association of reviewer recommendations with the review model was tested in logistic regression with review model (open or single blind) as the response, and reviewer recommendation coded as a three-level categorical predictor (accept, revise, reject).
Association of reviewer recommendations with the final decision was tested in logistic regression with the final decision as the response and reviewer recommendation coded as a three-level categorical predictor (accept, revise, reject) with different effects for author-suggested and non-author-suggested reviewers. Interaction terms with review model (open or single blind) were also included in exploratory analysis but dropped if found to be non-significant.
In all regression analyses, each review was treated as one observation and cluster SEs used to allow for multiple reviews of each paper.
Responses to author surveys were tested using the two-sample Mann-Whitney U test.
All analyses were conducted in R.
Results
Analysed journals and reports
Quality of reviewer reports
Rater agreement on quality of reports
We found moderate agreement between the raters for BMC Microbiology and BMC Infectious Diseases, with weighted κ values generally around 0.4 or higher. For the Journal of Inflammation, where the raters agreed on the criteria for each score of the RQI before rating the reports, the agreement was stronger for six of eight questions, but still in a moderate range (see online supplementary table S1). Our agreements were lower than found in previous studies using the RQI,12 13 which might be due to the particular subject matter of the reviewed articles or to bias in the raters’ perception on what they deemed important as a review comment.
Quality of reviewer reports on open and single-blind peer review
For questions 3 (strengths and weaknesses of the methods), 5 (constructiveness) and 6 (supplying appropriate evidence to substantiate comments) of the RQI, there was a significantly higher score for BMC Infectious Diseases (open peer review) compared with BMC Microbiology (single-blind peer review; figure 1). This led to a 5% improvement of the overall score (p=0.042 averaging both raters); however, this was significant for one rater (p=0.02) but not the other (p=0.39), so may not represent a genuine difference.
Figure 1.
Comparison of Review Quality Instrument scores between BMC Infectious Diseases (open peer review) and BMC Microbiology (single-blind peer review). Three questions obtained statistically significantly higher scores in BMC Infectious Diseases: Q3: Did the reviewer clearly identify the strengths and weaknesses of the method (study design, data collection and data analysis)? (p=0.004) Q5: Were the reviewer’s comments constructive? (p=0.0046) Q6: Did the reviewer supply appropriate evidence using examples from the paper to substantiate their comments? (p=0.0015) This led to a 5% improvement of the overall score (p=0.042). Values in bold red denote p<0.05.
For the Journal of Inflammation, however, no significant differences were seen in review quality between open and single-blind models (see online supplementary table S2).
Quality of reports provided by author-suggested and non-author-suggested reviewers
There is nominally significant evidence that non-author-suggested reviewers obtained higher scores on Q6 (how well the reviewer substantiated their comments) and Q7 (comments on the author’s interpretation of results) in BMC Infectious Diseases (open peer review) than the scores obtained by reviewers in BMC Microbiology (single-blind peer review). However, these differences are not significant when adjusted for multiple testing. No other questions showed significant differences between author-suggested and non-author-suggested reviewers (see online supplementary tables S3 and S4).
In the Journal of Inflammation, a nominally significant difference in review quality was seen for Q3 (comments on strengths and weaknesses of the method), but this was not significant after correction for multiple testing. No other significant differences were seen in review quality between author suggested and other reviewers (see online supplementary table S5).
Reviewer recommendations to accept or reject
In their initial report, most of the reviewers provided a recommendation to either accept or reject the manuscript. For BMC Infectious Diseases (open peer review) and BMC Microbiology (single-blind peer review) the numbers of reviewer recommendations of each kind were similar, suggesting a lack of difference between open and single-blind peer review models (figure 2). In both journals, author-suggested reviewers were more positive compared with non-author-suggested reviewers (BMC Infectious Diseases p=1.4×10−6; BMC Microbiology p=6.3×10−8). About two-thirds of author-suggested reviewers recommend acceptance and only 2–5% recommended rejection (see figure 2).
Figure 2.
Percentage of reviewers providing a given recommendation for the manuscripts in BMC Infectious Diseases and BMC Microbiology.
For the Journal of Inflammation, author-suggested reviewers also returned significantly more favourable recommendations than other reviewers (p=3×10−6). Again, there was no significant difference between open and single-blind review models (figure 3).
Figure 3.
Percentage of reviewers providing a given recommendation for the manuscripts in Journal of Inflammation.
We found 13 manuscripts in BMC Infectious Diseases and 17 in BMC Microbiology, where the author-suggested reviewer recommended acceptance or revisions, while the non-author-suggested reviewer recommended rejection and the final outcome was rejection. However, there were only two accepted manuscripts in BMC Infectious Diseases and four accepted in BMC Microbiology where the author-suggested reviewer recommended acceptance, and the non-author-suggested reviewer recommended rejection. Recommendations of both types of reviewers predicted the acceptance or rejection of the manuscript, but the view of the non-author-suggested reviewer better predicted the final outcome (p=7.3×10−19) compared with the author-suggested reviewer (p=1.3×10−5). There were no differences between BMC Infectious Diseases and BMC Microbiology in this respect. However, in the Journal of Inflammation, there was no significant difference between author-suggested and non-author-suggested reviewers in predicting the final decision.
Author surveys
We analysed all author surveys that were available at the end of September 2013: 741 surveys for BMC Microbiology (single-blind peer review), 685 for BMC Infectious Diseases (open peer review) and 49 (open peer review) and 47 (single-blind peer review) for the Journal of Inflammation.
Authors in BMC Infectious Diseases felt that reviewer comments were more helpful than did the authors in BMC Microbiology (p<0.0001, see online supplementary table S6). However, most other questions relating to managing other editorial and production processes on the manuscript tended to be scored more favourably (p<0.05) as well.
For the Journal of Inflammation, we did not find statistically significant differences between surveys returned by authors of manuscripts that underwent open or single-blind peer review, for any question regarding the editorial and production processes.
Discussion
The peer review process in journals has been criticised for being slow, inconsistent and biased.14 However, there is a paucity of studies investigating how peer review actually works, including the quality of reviewer reports, and differences between peer review models and types of reviewers. We were able to compare quality of peer review between BMC Infectious Diseases (open peer review) and BMC Microbiology (single-blind peer review), two journals that differ in peer review model but are otherwise very similar. We also assessed the effect of change in peer review model within a single journal, the Journal of Inflammation, which has changed its peer review model. It gave us unique opportunity to assess the impact of the peer review model on the quality of peer review and recommendations made by the reviewers. In the analysed journals, we were also able to compare the quality of reports and recommendations provided by reviewers suggested by the authors, compared with reviewers suggested by other parties (ie, BioMed Central's PubMed search tool comparing the abstract of the manuscript to abstracts in PubMed, other reviewers and editors). The large sample size of analysed reports ensured statistically robust results.
The main findings are that the quality of peer review reports was slightly higher in BMC Infectious Diseases (open peer review) compared with BMC Microbiology (single-blind peer review), but we did not find a difference for the open versus single-blind review in the Journal of Inflammation. These results suggest that it may be advantageous to use open peer review but they do not undermine the validity of using the single-blind approach.
In none of the three analysed journals did we find a difference in quality of peer reviewer reports written by author-suggested reviewers compared with those written by reviewers suggested by other parties, but in all three journals the reviewers suggested by the authors were much less likely to recommend rejection and more likely to recommend acceptance than reviewers suggested by other parties. Overall reviewer recommendations were similar in BMC Infectious Diseases compared with BMC Microbiology, suggesting no difference between open and single-blind peer review. However, in the Journal of Inflammation, we found that reviewers were more likely to recommend acceptance under open peer review as compared with under single-blind peer review. Author satisfaction was higher for BMC Infectious Diseases compared with BMC Microbiology, including the response to the question regarding the helpfulness of comments from peer reviewers. In the Journal of Inflammation, there was no difference in author satisfaction between the periods of time when the journal operated on open or single-blind peer review.
Quality of reviewer reports
The single-blind peer review model is the most common model of peer review used in journals in the field of biology and medicine. The BMJ was one of the first journals to open up peer review in 1999.15 This decision was based on the results of a randomised controlled trial2 where the quality of reviewer reports was assessed using the same RQI12 that we used in the current study. Similar to our analysis, the BMJ study found no significant differences between the anonymous and non-anonymous reviewers with regard to the quality of their reports.2
Another study of open peer review, using a modified version of the RQI,12 was conducted at the medical journal, the British Journal of Psychiatry.13 Interestingly, that study found that reports under open peer review were of 5.5% higher quality than unsigned reports, which is consistent with the 5% difference we found between scores for reviewer reports in BMC Microbiology (single-blind peer review) and in BMC Infectious Diseases (open peer review). Moreover, the signed reviewer reports scored significantly higher on the same two questions as our open reports for BMC Infectious Diseases (Q3 regarding methodology and Q5 regarding constructiveness of comments). It was also very encouraging to see that, of the 322 reviewers, as many as 245 (76%) agreed to sign their reports for the British Journal of Psychiatry.13
Under open peer review, reviewers are more accountable for their reviews, which may account for the higher scores we observed for Q3 (discussing the strengths and weaknesses of the method), Q5 (the more constructive reports) and Q6 (supplying appropriate evidence to substantiate the comments), which led to a 5% improvement of the overall score (p=0.042).
There is another possible explanation for the higher score that reviewer reports obtained on open peer review for Question 3 (whether the reviewer clearly identified the strengths and weaknesses of the method, study design, data collection and data analysis). Medical research follows several distinct types of defined study designs (see The EQUATOR Network16 for more information). It is therefore possible that, in evaluating medical research as compared with evaluating biology manuscripts, reviewers are more likely to comment on whether the authors used an appropriate study design and appropriate methodology, as there are existing reporting guidelines to follow. We found that key words such as ‘study design’, ‘guideline(s)’, ‘methodology’ and ‘data analysis’ occurred more frequently in reviewer reports for BMC Infectious Diseases compared to reviewer reports for BMC Microbiology, although it did not reach statistical significance (Fisher exact test, p=0.2).
In the current study, we have not found significant differences in the quality of reports provided by author-suggested reviewers compared with non-author-suggested reviewers. This has also been noted in previous studies.1 9
Reviewer recommendations to accept or reject
Author-suggested reviewers tend to provide more favourable recommendations than other types of reviewers. This result is consistent for the three journals we analysed, irrespective of the peer review model. This is also consistent with previous studies on this topic.1 3–9 However, in both BMC Infectious Diseases (open peer review) and BMC Microbiology (single-blind peer review), recommendations made by non-author-suggested reviewers were a better predictor of the final decision to accept or reject the manuscript than recommendations made by author-suggested reviewers. It seems that these reviews carried more weight with journal editors (unconsciously or consciously).
In the Journal of Inflammation, we found that reviewers provided significantly more favourable recommendations under open peer review. This effect was found in some of the previous studies1 13 but not all.2 Also, acceptance rates were significantly higher under open peer review than on single-blind peer review (67% vs 50%, see table 3). However, it is difficult to determine if this effect is due to the peer review model or other factors. The Journal of Inflammation is smaller than BMC Microbiology or BMC Infectious Diseases, and its analysed reports span a longer period of time, from September 2007 (shortly after the journal was launched) to September 2011. The change in peer review model was also accompanied by a change in editorship and coincided with the journal receiving its first impact factor. It is possible that these factors impacted on how the peer reviewers were selected, and how they perceived the journal and made recommendations.
Author satisfaction
Authors on BMC Infectious Diseases (open peer review) gave slightly higher scores for the question on the helpfulness of reviewers’ comments, compared with BMC Microbiology authors (single-blind peer review). However, they also gave higher scores for every other question, so it may be the case that the overall editorial process was perceived to be better at BMC Infectious Diseases, or that it simply reflects different priorities among research fields. The score for the Journal of Inflammation regarding helpfulness of peer reviewers’ comments was higher for open than for single-blind peer review, but did not reach significance.
Limitations of this study
The RQI12 evaluates how detailed and thorough the reviewer reports are, but not whether the criticism is valid or whether the reviewers recognised the flaws in the manuscript. This kind of assessment requires specialist knowledge. A recent study of decision consistency in peer review of Post-doctoral Fellowship applications suggests that using as many as five reviewers per application may be optimal.17
We found only a moderate level of agreement between the raters of the quality of referee reports. Prior to rating the quality of reports for BMC Microbiology (single-blind peer review) and BMC Infectious Diseases (open peer review), formal discussions on how to use the RQI12 were not held. Subsequently, we found that agreement between the two independent raters was not high on individual questions, but there was moderate agreement for the overall mean score. Given this experience, before analysing reports for the Journal of Inflammation, the raters explicitly discussed the rating scale and agreed broadly on how they would score individual questions in general terms. Following this, we found improved rater agreement for all questions (see online supplementary table S1). However, the raters did not confer about the ratings for individual reports for any journal. The raters were not blinded to the identity of the reviewers. However, as the raters were not involved in managing peer review on these journals, the names of the reviewers were not familiar.
We were not able to use exactly the same research design for the Journal of Inflammation as used for BMC Infectious Diseases and BMC Microbiology. The Journal of Inflammation did not have a sufficient number of manuscripts reviewed by two referees, one of whom was suggested by the authors and one not. As a result, we did not have an equal number of reports from author- and non-author-suggested reviewers.
Another limitation of the study is the low number of author surveys available for the Journal of Inflammation. Although we analysed all available author surveys, it is possible that the lack of statistical significance of the results is due to the low number of available surveys rather than the lack of impact of peer review model.
We chose BMC Microbiology and BMC Infectious Diseases because the journals have a similar profile (table 1) and the same editorial processes, differing mainly in the peer review model. However, the journals do not have identical author bases and there remains the possibility that the differences we observed were due to some other factors despite our efforts to closely match the journals. Similarly, our results for the Journal of Inflammation may reflect temporal trends, such as the change of editorship noted above. To reduce the impact of such biases we selected articles sequentially by order of submission, starting from the date of change of review model.
Conclusions
This study is the first to investigate the effect of author-suggested peer reviewers and openness of peer review, on the quality of peer review in the same journal and between very similar journals in the field of biology as well as medicine. We show that, in agreement with previous studies,2 13 the quality of peer review reports in journals with open peer review is comparable with that of journals with single-blind peer review. Furthermore, open peer review improves constructiveness of peer reviewer comments.
Author-suggested reviewers tend to recommend acceptance more often than non-author-suggested reviewers, but the quality of peer review reports is similar regardless of the source of peer reviewer suggestions, which is also in agreement with previous reports.1 9
Acknowledgments
The authors would like to thank Frances Benson, Rachel Craven and Marco Tranchino for their help with identifying the manuscripts to analyse. They also thank Charlene Colegate, Martha Davey, Jennifer Franklin, Sophie Marchant, Yasmin Mulla, Natasha Munzi and Holly Young, who helped collate the data. They would also like to thank Philippa Harris and Matt Cockerill for useful comments and suggestions.
Footnotes
Twitter: Follow Maria Kowalczuk at @Maria_Kowalczuk, Shreeya Nanda at @ShreeyaNanda, Stephanie Harriman at @stephharriman, Jigisha Patel at @JP6050, and Elizabeth Moylan at @latwec
Contributors: MKK coordinated the project, participated in designing the study and rating the reports, and drafted the manuscript. FD participated in designing the study and performed all the statistical analyses. SN, SLH, JP and ECM were involved in designing the study, rating reviewer reports, interpreting the results and drafting the manuscript.
Funding: This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests: MKK, SLH, JP and ECM are employed by BioMed Central and work closely with the editors of the journals that are the object of this study. SN was employed by BioMed Central when the study was conducted. FD is an associate editor for BMC Bioinformatics, a BioMed Central journal that was not an object of this study.
Provenance and peer review: Not commissioned; externally peer reviewed.
Data sharing statement: No additional data are available.
References
- 1.Schroter S, Tite L, Hutchings A et al. . Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 2006;295:314–17. 10.1001/jama.295.3.314 [DOI] [PubMed] [Google Scholar]
- 2.van Rooyen S, Godlee F, Evans S et al. . Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ 1999;318:23–7. 10.1136/bmj.318.7175.23 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Bornmann L, Daniel HD. Do author-suggested reviewers rate submissions more favorably than editor-suggested reviewers? A study on atmospheric chemistry and physics. PLoS ONE 2010;5:e13345 10.1371/journal.pone.0013345 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Hurst J, Howard E, Wedzicha JA. Reviewer selection: author or editor knows best? Thorax 2005;60:799 10.1136/thx.2005.051870 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Moore JL, Neilson EG, Siegel V, Associate Editors at Journal of American Society of Nephrology. Effect of recommendations from reviewers suggested or excluded by authors. J Am Soc Nephrol 2011;22:1598–602. 10.1681/ASN.2011070643 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Rivara FP, Cummings P, Ringold S et al. . A comparison of reviewers selected by editors and reviewers suggested by authors. J Pediatr 2007;151:202–5. 10.1016/j.jpeds.2007.02.008 [DOI] [PubMed] [Google Scholar]
- 7.Earnshaw JJ, Farndon JR, Guillou PJ et al. . A comparison of reports from reviewers chosen by authors or journal editors in the peer review process. Ann R Coll Surg Engl 2000;82(4 Suppl):133–5. [PubMed] [Google Scholar]
- 8.Helton ML, Balistreri WF. Peering into peer-review. J Pediatr 2011;159:150–1. 10.1016/j.jpeds.2011.02.012 [DOI] [PubMed] [Google Scholar]
- 9.Wager E, Parkin EC, Tamber PS. Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study. BMC Med 2006;4:13 10.1186/1741-7015-4-13 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kowalczuk MK, Dudbridge F, Nanda S et al. . A comparison of the quality of reviewer reports from author-suggested reviewers and editor-suggested reviewers in journals operating on open or single-blind peer review models. F1000Posters 2013;4:1252. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.BMC series journals. http://www.biomedcentral.com/authors/bmcseries
- 12.van Rooyen S, Black N, Godlee F. Development of the Review Quality Instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol 1999;52:625–9. 10.1016/S0895-4356(99)00047-5 [DOI] [PubMed] [Google Scholar]
- 13.Walsh E, Rooney M, Appleby L et al. . Open peer review: a randomized controlled trial. Br J Psychiatry 2000;176:47–51. 10.1192/bjp.176.1.47 [DOI] [PubMed] [Google Scholar]
- 14.Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med 2006;99:178–82. 10.1258/jrsm.99.4.178 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Smith R. Opening up BMJ peer review. BMJ 1999;318:4–5. 10.1136/bmj.318.7175.4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.The EUQATOR Network. http://www.equator-network.org/
- 17.Snell RR. Menage a Quoi? Optimal number of peer reviewers. PLoS ONE 2015;10:e0120838 10.1371/journal.pone.0120838 [DOI] [PMC free article] [PubMed] [Google Scholar]