Organised crime against the academic peer review system

‘The most dangerous criminal may be the man gifted with reason, but with no morals.’

Dr Martin Luther King Jr., 1947 1

Editorials are generally about what we did right in our journal and we do not often publish about our failures. Yet, in this Editorial we feel we have to convey the full story of how we went entirely off track with the publication of a paper.

In 2015, we published a paper in our journal 2 that we retracted last month. The peer review process of this paper was compromised, but, in addition, our careful editorial process did not function. In this Editorial, we would like to describe this whole case, with the aim of learning from this and of drawing attention to a serious breach of the trust between scientists, which is the basis of our peer review system (see box 1).

Box 1

The BJCP peer review system

All papers that are uploaded to the Scholar One system are first reviewed by a Senior Editor. All BJCP Senior Editors are experienced general clinical pharmacologists. After review, two things can happen. The paper can be rejected without peer review when the Senior Editor decides the paper is outside the scope of the journal, or has limited interest for our readers, or has obvious deficiencies in quality. This is also to prevent our authors from having to wait for a lengthy review process leading up to the predicted negative outcome.

Alternatively, the paper is deemed suitable for peer review and is matched to the more specific expertise of an Executive Editor. This Executive Editor chooses the peer reviewers, refers the paper to them and communicates the outcome of the peer review process to the authors. This can be a rejection but more often one or more revision cycles are initiated. During this process, the Executive Editor monitors the quality of the reviews and of course the author responses. Finally, the Executive Editor makes a recommendation to reject or accept the revised version. Rejections after the revision cycle occur, but generally only when the authors are unable to comply with the requests of the peer reviewers.

The paper then goes back to the Senior Editor for a final check of the review process and the final decision to publish is made. At this point we also look at possible media interest of the paper (in which case we prepare press releases) and the need for editorial comments.

After the final decision to accept, the paper is transferred to the technical editing process for language and scientific editing and typesetting.

If a paper is of good quality but is deemed to be of less interest to BJCP we give authors an opportunity to refer the paper to our sister open access journal Pharmacology Research & Perspectives, but this is obviously their choice. Finally, we allow appeals against editorial decisions and the appeals are discussed monthly in a meeting of the Senior Editors.

The history of this paper

The paper was a meta‐analysis of the effect of recombinant brain natriuretic peptide on in‐hospital mortality compared with dobutamine, submitted in August 2015, seen by a Senior Editor, referred to an Executive Editor and sent for peer review. The authors suggested two peer reviewers who had authored important papers cited in the meta‐analysis. Both peer reviewers were academic cardiologists and experts in heart failure at high level institutions in the United States. Contrary to our normal practice, the Executive Editor decided to accept both suggestions (our policy is to use no more than one) and the two peer reviewers were invited. They accepted this invitation to review one day later. The reviewers’ comments were returned after two and four days, respectively, and are shown in box 2.

Box 2

Reviewers' comments

Reviewer 1

‘As the authors documented that this meta‐analysis based on a systematic review of existing studies evaluated the efficacy of rhBNP and dobutamine treatments in the clinical management of HF patients, no matter from the title singly or from the well‐constructed abstract, the pre‐set idea was well proved, not to mention the wonderful major text. In my opinion, authors provided a clear background introduction (HF, rhBNP and dobutamine) to draw out a legitimate research; and I am especially interested in the illustration of the Discussion, which illuminate the hypothesis from the Introduction entirely. Good job!’

‘Only one concern, in order to perfect this paper, the statistical data “P” was rarely expressed as “0.000”, please revise this minor question in the Results.’

Reviewer 2

‘I am very interested in this article titled “Recombinant human brain natriuretic peptide achieves to reduce the in‐hospital mortality than dobutamine in heart failure patients: a meta‐analysis”, from such concise and clear title readers might also be attracted from this topic, where the authors elaborated the important role of rhBNP in decreasing the in‐hospital mortality than dobutamine in heart failure patients. The work could be of potential interest for the readership of British journal of clinical pharmacology. The manuscript was properly organized, with rich contents and reasonable methods.

1. Introduction: the part about heart failure introduction as well as the treatment choice regarding rhBNP and dobutamine is partinent.

2. Methods and Results sections are adequately described.

3. In discussion, obtained results was discussed elaborated (very few studies were cited, yet not properly discussed) and further confirmed with similar studies.

4. No comments on tables and Graphics.’

‘I do appreciate the authors effort on this paper after perusing the article, hence I recommended a publication of this article in British journal of clinical pharmacology.’

‘Plus, only a few points should be paid attention to:

“Electronic database search and manual review for eligible articles” in Results, the exact numbers of articles searched manually can be provided clearly; besides, the exact numbers of studies excluded for non‐cohort studies, irrelevant to dobutamine or rhBNP, etc, can also be mentioned. Further, “p = 0.000” would be better to substituted with “p < 0.001”.’

Convinced by these comments, our Editors recommended a revision of the English, and after this was returned, the paper was published.

We would probably not have heard from this paper again if we had not received a letter, which we publish integrally in box 3, written by the two co‐authors of this editorial. The paper was discussed in the journal club session at an academic institution, as a part of training of the clinical pharmacology residents for ‘methods in clinical research’. Dr Smita was the faculty moderator for this session, and a mismatch of the numbers in a table and the forest plot raised suspicion.

Box 3

LET‐00084‐16

Garbage in for the quality of data pooled and garbage out for wrong analysis in a published meta‐analysis

With great interest, we read the meta‐analysis published in British Journal of Clinical Pharmacology by Ming et al. [1]. This is the first ever meta‐analysis that demonstrates, Nesiritide (rhBNP) can improve clinical outcome and reduce in hospital mortality compared with Dobutamine in heart failure patient. The previous two meta‐analyses that were published on the same topic did not show any benefit. One of the meta‐analysis pooled data from 3 RCTs and showed increased in the mortality with Nesiritide [2] and other from 7 RCTs showed no significant change in the mortality [3]. We found serious issues with the current meta‐analysis which need attention of the related scientific fraternity.

Serious methodological flaws have been noted in the conduct of the study.

• 1The forest plot in Fig. 3A, represents the primary outcome of the study i.e. "In‐hospital mortality” but has imputed all the numbers which are the “survival data” for the corresponding studies. Therefore, it rather represents the odds of 'survival' than ‘in‐hospital mortality’. We constructed the forest plot for the same outcome using data from the table 1 of the study for “mortality” [1]. Furthermore, the data from the study by Silver et al. has been included four times taking the all‐cause mortality and heart failure mortality as separate entities in the analysis of the forest plot [Fig. 3A of Ming et al.]. However, mortality due to heart failure is a subset of all‐cause mortality and therefore, it is wrong to include both.. This also means, the study population that represents the denominator in the dobutamine group has been spuriously increased due to multiple inclusions. We reconstructed the forest plot including the data from Silver el al. for all‐cause mortality but not heart failure mortality. The surprise is, the pooled odds ratio has changed to become non‐significant [OR=1.52, 95% CI 0.97 to 2.37,(p=0.07)] and therefore, this changes the conclusion derived by the study.

Reconstructed forest plot for In‐hospital mortality (Fig. 3A of Ming et al.)

• We also constructed the forest plot taking the survival data and found a non‐significant benefit for nesiritide similar to that of the in‐hospital‐mortality outcome.

Reconstructed forest plot for Survival (Original Fig. 3A of Ming et al.)

• In the study by Lissovoy et al., the number of deaths are not matching the data of the original article [4] and also and also quoted wrongly in the Table 1 of the article.

Inadequacies regarding the study‐design

1. The pooling of data from cohort studies, randomized open label study, randomized controlled trial and 3 studies not reporting the study design questions the applicability of the result in the clinical scenario due to unacceptably high risk of bias. Out of the total sample included in the study, randomized controlled trials which represent high quality evidence represent only 9.5%(319/3350) of the population with 14.8% (203/1366) in the nesiritide group and 5.8%(116/1984) in the dobutamine group. Therefore, the result of the meta‐analysis conveying mortality benefit is potentially misleading.

2. The duration of assessment for the primary outcome “in‐hospital mortality” is not mentioned in the study. The study has pooled data for less than 6 months mortality and greater than 6 months mortality, which may mean multiple hospitalization episodes and death beyond the index hospitalization with nesiritide use. Nesiritide is a drug which has short term vasodilatory effect and therefore, the in hospital mortality for the same admission rather than 6 months mortality would be a more meaningful end point for determining its efficacy. It is also nor clear from the manuscript whether the in‐hospital mortality is for same admission or different admissions.

Inappropriate subheading:

The publication bias has been described under sensitivity analysis heading.

This analysis claims that, rhBNP can improve the clinical outcomes and dramatically reduces the in‐hospital mortality compared to dobutamine. Firstly, the results are completely misleading due to poor quality of data pooled. Secondly, the results are certainly incorrect due to wrong input of data from the studies included of the analysis. In contrast to the claims made by this article[1], the largest clinical trial evaluating the efficacy of nesiritide has failed to demonstrate any benefit on reducing the mortality and rate of hospital readmission in acute decompensated heart failure[5]. Moreover, the evidenced based recommendation for treatment of acute decompensated heart failure also does not support the use of Nesiritide with Grade 1A level of evidence [6].

This article does seem not match the quality standards of an esteemed journal like BJCP. This is also highlights the poor quality peer review the manuscript had undergone.

REFERENCES:

1. Lv, M.‐Y., Deng, S.‐L., and Long, X.‐F. (2016) rhBNP therapy can improve clinical outcomes and reduce in‐hospital mortality compared with dobutamine in heart failure patients: a meta‐analysis. Br J Clin Pharmacol, 81: 174–185. doi: 10.1111/bcp.12788.

2. Sackner‐Bernstein JD, Kowalski M, Fox M, Aaronson K. Short‐term risk of death after treatment with nesiritide for decompensated heart failure: a pooled analysis of randomized controlled trials. JAMA. 2005 Apr 20; 293(15):1900‐5.

3. Arora RR, Venkatesh PK, Molnar J. Short and long‐term mortality with nesiritide. American heart journal. 2006 Dec; 152(6):1084‐90.

4. de Lissovoy G, Stier DM, Ciesla G, Munger M, Burger AJ. Economic implications of nesiritide versus dobutamine in the treatment of patients with acutely decompensated congestive heart failure. The American journal of cardiology. 2003 Sep 1; 92(5):631‐3.

5. O'Connor CM, Starling RC, Hernandez AF, Armstrong PW, Dickstein K, Hasselblad V, et al. Effect of Nesiritide in Patients with Acute Decompensated Heart Failure. New England Journal of Medicine. 2011; 365(1):32‐43.

6. Uptodate.com. Nesiritide in the treatment of acute decompensated heart failure Available from: http://www.uptodate.com/contents/nesiritide‐in‐the‐treatment‐of‐acute‐decompensated‐heart‐failure. [Accessed on 26 January 2016].

They wrote to us by the end of the same month of publication of the article in question. This was a highly critical letter, which of course raised some eyebrows in the editorial board, until we examined the paper and the claims made in it and realized we had failed rather badly. It appeared that the flaws in the meta‐analysis were even worse than presented in the letter. The number of extracted events per study was a mix of real events and the number of patients without an event. We were not able to locate two Chinese studies used for the meta‐analysis. The scoring system “Critical Appraisal Skills Programme” used to score the quality of the studies has, as far as we know, never been used before and consists of completely irrelevant questions such as “What are the results of this study?", "What are the implications of this study for practice?", and "Do you believe the results"?.

It was particularly worrying that we published a paper that was not only erroneous, but appeared to claim the reverse of what was generally known about rBNP: that it was not effective in heart failure.

Investigation

We first investigated the editorial process and it quickly became clear that, as an editorial board, we had overlooked all danger signs. The authors were very inexperienced in science and none of them had a single publication on record. Of course, all authors have to start their careers as inexperienced, but normally a group of authors has at least one member with more experience. The Senior Editor assumed, rightly, that a meta‐analysis could be of importance and required peer review. The Executive Editor noted the extremely high level of experience of both referees suggested by the authors. It seemed attractive to invite both, and when they agreed and sent back very positive reports, the next step towards publication was, perhaps, inevitable.

The Senior Editor and the Executive Editor missed the totally inadequate quality of the reviews (box 2), the style of English, which was fairly unusual for two highly ranked professors of medicine in Ivy League US universities, and the suspiciously rapid response. Also the fact that they did not have an institutional e‐mail address should have raised suspicion.

We wrote to the referees and one of them denied all knowledge of this report. The second referee could not be traced. We then realised this was a case of peer review fraud 3.

Peer review fraud is accomplished when authors suggest two preferred referees (who generally are selected from the literature and are high profile). Their email address is forged so that the request for review goes to the author. They accept, write a positive review to increase their chances, in this case with initial success, and would have achieved full success if the co‐authors of this Editorial had not spotted it.

Action

As we now had clear proof of fraud, we wrote to the authors, who claimed that they had transferred the submission of the paper (but according to them not the analysis or writing) to a company to which they paid an amount of RMB 3000 (Figure 1). This company has a website in Chinese (http://www.editpub.net/index.php), which contains biographies and email addresses of experts. We checked several and found that often their photographs and emails were fabricated. Readers can check for themselves, but it is hardly conceivable that a Brazilian female researcher has the picture of Henry Kissinger by accident.

Figure 1.

Payment slip for fraudulent submission of an article by Editpub.com

All this was sufficient to retract the paper, and a retraction was published last month 4. Our concern at this point was whether there had been any use of the paper to make decisions about patient care. The paper was not cited at all and downloaded only 113 times. A sample of 14 papers published around the same time were downloaded on average about 3 to 4 times more often, indicating that the paper had not attracted a lot of attention. We therefore concluded that we should retract the paper and thus amend the public record before any serious implications occurred.

Analysis

There is no doubt that our experienced Editors missed several fairly obvious clues that should have set alarm bells ringing. We are embarrassed by this but we also have to realise that the editorial and peer review system is not designed to withstand fraudulent activities by a commercial criminal organisation blatantly abusing the trust that is the basis of our peer review system.

Our Editors (and those of most scientific journals) do not do this work full time and can only have the right background when they are busy clinicians, scientists and educators. When they see eminent reviewers being suggested by fairly inexperienced researchers it is not unreasonable to invite them. None of our Editors was aware that this could be a scam, and we assume that many others involved in peer review also do not know there can be unscrupulous organisations operating.

Whenever things go wrong, there is a strong urge to take action, especially to prevent whatever went wrong happening again. Such steps, however, will generally make life harder for the majority of people being entirely honest, and generally do not prevent further misconduct. One only has to visit an airport to experience this. Making cockpit doors bullet proof did indeed assure that no terrorist has entered the cockpit since 9/11 (but of course they also did not do so before 9/11 when you could still talk to the pilot). It did, however, assure that a mentally ill pilot could kill several hundred passengers.

When we examined our current review process, we concluded that it is sufficiently robust to be able to detect these mishaps and we remain convinced that selecting experienced scientists is the best safeguard for this. We also are convinced that policing is not the job of journal Editors, and even if it was, the job would be impossible. We cannot check the integrity of data collection, we cannot assure that ethic committees have indeed looked at studies, or that the patients really gave informed consent.

We have written guidelines and policies for authors and editors to follow; however, at the end of the day we simply have to take the word of our fellow scientists and assume that the institutions they work for control the quality of their scientific production.

Scientific quality has to be assured at the source, so within each institution, and we realise all too well that this is often not the case. Scientific journals and the current peer review process can do an overall check that results in the rejection of low quality papers with good guidance from the reviewers on how to improve. Unfortunately, we cannot assure the integral quality, but we have no reason to doubt the vast majority of our authors are trustworthy.

Of course, we have taken some measures as a result of the missteps that occurred in the review process. Our system is robust only when all involved do their job and do not rely on the next level of control entirely, and we have reinforced this within the Editorial Board. We shall as a rule not follow the suggestion by authors for two referees and shall send requests to institutional email addresses only. We have also notified the Chinese Ministry of Science and Technology, with a request to investigate this case further.

There is no doubt that this event has made us wiser (and sadder) and inevitably a bit more suspicious. However, we believe that the peer review system, when applied rigorously, is the best we have. Moreover, the timely action of members of the academic community and the use of a paper in a teaching session saved the day, demonstrating the importance of open academic debate after publication, through the time‐tested method of letters to the editor.

So for the foreseeable future, BJCP will not ask for your fingerprints or your ID card or X ray your data, even though we understand very well that there are bandits out there, and this Editorial serves as another warning for our colleagues.

Cohen, A. , Pattanaik, S. , Kumar, P. , Bies, R. R. , de Boer, A. , Ferro, A. , Gilchrist, A. , Isbister, G. K. , Ross, S. , and Webb, A. J. (2016) Organised crime against the academic peer review system. Br J Clin Pharmacol, 81: 1012–1017. doi: 10.1111/bcp.12992.