A time for review

“If peer review were a medicine, it would never get a licence…We had great difficulty in finding any real hard evidence of the system’s effectiveness.”

This distressing quote is attributed to Tom Jefferson, from the Cochrane Collaboration’s Methods Group, during an interview with The Guardian in 2003 after the publication of their systematic review of the effect of peer-review in scientific reporting and grant funding. The last decade has seen a lot of debate on the subject.

The need for quality assurance of scientific papers would seem self-evident. Furthermore, several high profile cases of scientific fraud have underscored the need for as robust a system of quality control as possible. Pre-publication peer-review has been the cornerstone of the scientific method for the last 200 years, and is considered a key contributor to the assessment of quality of submissions for most biomedical journals. However, many have recently questioned whether this process is at all additive to post-publication scrutiny by the scientific community through comments, citations and review articles. Calls for an overhaul of the peer-review process in biomedical publishing have become commonplace.¹

Classical peer-review in the biomedical literature has been closed and “single-blind,” with anonymous reviewers privy to author’s names and institutions. Reviewers are tasked to comment not only on the scientifc quality of the manuscript but are also asked to judge the “interest” of the article to the journals readership. Many have suggested that this process is slow, expensive and unreliable; the chances of two reviewers agreeing on an article has been shown to be only slightly better than a coin flip. Furthermore, do peer-reviewers do a good enough job at detecting errors, ethical problems or fraud? Does the process facilitate unreasonable bias and even abuse? Is it worth all the effort?

These are particularly valid questions in today’s changing landscape of publishing scientific work with aggregated databases, e-print servers, open institutional archives and author’s personal webpages increasingly competing with print or electronic versions of traditional peer-reviewed journals. Some advocate for open reviews, publishing the reviewer’s comments and author’s replies and trialing open, public review prior to formal publishing. We believe pre-publication peer-review still contributes to quality control for publicizing research results in journals and at conferences and that it has not neccesarily run its course. We do however have to acknowledge its many shortcomings; particularly the fact that reviewing capacities are the most limited resource in the publication process. At CUAJ, we rely on a closed (“double-blind”) peer review, where neither authors’ nor reviewers’ names are revealed. As peer-review depends crucially on the availability and performance of reviewers, public acknowledgement of all of our reviewers’ work is essential and we have recently initiated an award for those providing the most careful and constructive reviews over the year (Dr. Kevin Zorn from the Centre hospitalier de l’Université de Montréal (CHUM) was last year’s recipient).

Finally, in an effort to improve the discourse and, most importantly, the practice of peer-review, CUAJ is offering a Peer-Review Workshop during the CUA Annual Meeting in Banff. The Workshop is open to all meeting participants, including seasoned reviewers and neophytes alike, and will cover the evidence on peer-review’s pitfalls and provide exercises on what journals and authors need from reviewers. We encourage you to attend and to lend your voice to this ever important part of scientific research and publishing. The event will be accredited with the CUA Office of Education as a Section 1 Group Learning Activity. The Workshop will be held on Sunday morning, June 24. If you have any comments or would like to participate in the Workshop, please contact the CUAJ Editorial Office at journal@cua.org.