Improving the reliability of peer-reviewed publications: We are all in it together

Abstract

The current, and welcome, focus on standardization of techniques and transparency of reporting in the biomedical, peer-reviewed literature is commendable. However, that focus has been intermittent as well as lacklustre and so failed to tackle the alarming lack of reliability and reproducibly of biomedical research. Authors have access to numerous recommendations, ranging from simple standards dealing with technical issues to those regulating clinical trials, suggesting that improved reporting guidelines are not the solution. The elemental solution is for editors to require meticulous implementation of their journals’ instructions for authors and reviewers and stipulate that no paper is published without a transparent, complete and accurate materials and methods section.

The treatment of some scientific topics, particularly in biomedical research, is very much like that afforded to the catwalk fashion industry; something becomes hyped, everyone talks about it and eventually the popular press picks up the topic and generally distort its conclusions, only for the band wagon to move on to the next hot topic. Tellingly, this excitement is usually misplaced and serves more to publicize the particular authors, institutions and journals than it does to contribute to any advancement in scientific knowledge or translational benefit. In contrast, vast amounts of scientific data are published without eliciting any interest whatsoever, leaving the authors to cite their own papers in the hope that their work will, one day, become the hyped fashion. Regardless, the results and conclusions from much, if not most, of the publications of biomedical research are questionable: the majority are not reproducible [1], [2], [3] and so do not satisfy one of the fundamental requirements of scientific research. There are a number of reasons why published results cannot be reproduced:

• 1.The original research was carried out incorrectly, for example without sufficient regard for sample selection, template quality or inappropriate data analysis.
• 2.The attempts to replicate results are flawed because the information provided in the publication is not sufficiently detailed and explicit.
• 3.The replicating laboratories do not have sufficient understanding of the uncertainty associated with their experiments. For example, the high precision of methods like digital PCR can generate different results, but a more focused look at reproducibility may show they are all describing different parts of a data distribution, which, once understood, would allow a definition of what can actually be measured.

Any of these explanations is objectionable and results in billions of dollars being wasted every year [4]. This message is, of course, not new [5] and over the last twenty years or so there have been numerous, often high profile, publications lamenting this state of affairs and proposing solutions, most recently summarized in a review article published in this journal [6].

Why is there this apparent indifference to publication quality? Is it because detailed scrutiny of the reliability, standardization, reproducibility and transparency of methods is perceived as comparatively mundane and unexciting? Is the current peer review process inadequate to provide a reliable analysis of all techniques? In theory, there is no disagreement about the importance of the methods section of a scientific manuscript [7] or that it requires a clear, accurate [8] and, crucially, adequate description of how an experiment was carried out. In theory, it is also accepted that the aim of a methods section is to provide the information required to assess the validity of a study and hence be sufficiently detailed so that competent readers with access to the necessary experiment components and data can reproduce the results.

Certainly, despite the wealth of evidence that published methods are wholly deficient, there has never been any determined, consistent and coherent effort to address these issues and deal with their consequences. Therefore a welcome, recent effort involves the publication of a report based on the proceedings of a symposium held earlier this year, aimed at exploring the challenges and chances for improving the reliability and reproducibility and of biomedical research in the UK (http://www.acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research/). However, a close reading of the report suggests that it simply summarizes all of the findings and opinions that are already published and suggests the same solutions that have been ignored until now. These include “top-down measures from journals, funders and research organisations” that aim to improve the quality of training and institute a research culture and career structure that reduces the emphasis on novelty and publication, as well as “bottom-up ones from individual researchers and laboratories” that address issues of poor study design and statistical practices, inadequate reporting of methods, and problems with quality control. What is lacking is a decisive, headline-grabbing call to action.

Some of the suggestions also imply that the authors of this report appear not to be overly familiar with existing, long standing efforts to standardize protocols and improve transparency. For example, in a section with the heading “strategies to improve research practice and the reproducibility of biomedical research” contains the suggestion that establishing standards could address some of the issues associated with reproducibility and points to the Minimum Information About a Microarray Experiment (MIAME) guidelines [9] as the exemplary standard. In fact, there are numerous “Minimum Information” standards projects following on from that paper, most of which have been registered with the Minimum Information for Biological and Biomedical Investigations initiative (http://www.dcc.ac.uk/resources/metadata-standards/mibbi-minimum-information-biological-and-biomedical-investigations), where they are collected and curated and can be accessed through a searchable portal of inter-related data standards, databases, and https://biosharing.org/standards. Complementary information is also available from the US National Library of Medicine website, which lists the organizations that provide advice and guidelines for reporting research methods and findings (https://www.nlm.nih.gov/services/research_report_guide.html). Medical research studies, in particular, are well served with reporting guidelines, for example by the EQUATOR Network, which aims to improve the reliability and value of the medical research literature by promoting transparent and accurate reporting (http://www.equator-network.org). There are reporting guidelines for many different study designs such as CONSORT (www.consort-statement.org) for randomized trials, STARD for studies of diagnostic accuracy (www.stard-statement.org/) and SPIRIT for study protocols (http://www.spirit-statement.org).

If it were simply a matter of developing standards, then the state of the peer-reviewed literature would not be as scandalous as it is. The real problem stems from the lack of application of those standards. This is most easily demonstrated by looking at, arguably, the most widely used molecular techniques, real-time PCR (qPCR) and reverse transcription (RT)-qPCR. These methods have found supporting roles as part of a huge number of publications in every area of the life sciences, clinical diagnostics, biotechnology, forensics and agriculture. qPCR-based assays are usually described as simple, accurate and reliable. This is true, but only if certain technical and analytical criteria are met. It is especially important to emphasize that the accuracy of results is critically dependent on the choice of calibration, whether this be a control sample or a calibration curve. This method is easily abused and one particularly egregious example is provided by its use to detect measles virus in the intestine of autistic children. Numerous, independent replication attempts, including those carried out by the original authors, failed to reproduce the original data and an analysis of the raw data, carried out as part of the US autism omnibus trial in Washington DC, revealed that the conclusions were based on fallacious results obtained by a combination of sample contamination with DNA, incorrect analysis procedures and poor experimental methods [10], [11]. A paper publishing these data remains to be retracted 13 years after publication. While this delay is typical, it is totally unacceptable and results in an underestimation of the role of fraud in the ongoing retraction epidemic [12], [13].

A typical problem associated with qPCR assay variability is illustrated in Fig. 1, which demonstrates that qPCR assays can behave significantly different under different experimental conditions. As the data demonstrate, at the higher target DNA concentration both assays generate reliable data. However, at the lower concentration, results are reliable from only one of the assays (A), with the ΔCq of 5.98 ± 0.21 between two different target DNA concentrations being in line with the expected value for the dilution factor. In contrast, the results of the other assay (B) are much more variable, (ΔCq of 8.29 ± 1.65) and also do not accurately reflect the dilution factor. The report by Dr. Andreas Nitsche in this issue shows that some assays are particularly sensitive to variability in different buffers and even different batches of same buffer. If assay behavior is not thoroughly assessed such that experimental conditions are simulated, prior to carrying out real-life tests, this can lead to false results and confound any potential conclusions.

Fig. 1.

Comparison of a dualplex qPCR assay targeting Candida dublinensis or Candida glabrata. (1) A PCRmax Eco (http://www.pcrmax.com) was used to amplify 100 pg of C. glabrata (A) and C. dublinensis (B) and DNA. Four replicate reactions were run of 5 μl each using Agilents’s Brilliant III qPCR mastermix, with PCR amplicons detected using FAM-(C. glabrata, A, blue)- or HEX-(C. dublinensis, B, pink) labelled hydrolysis probes. The assays are 98% and 97% efficient, respectively. (2) Plots of the two assays recording average Cqs of 25.16 ± 0.19 and 25.44 ± 0.19 for C. glabrata and C. dublinensis, respectively (Mean ± SD). (3) Conditions as described for 1, except that the replicates contained 1.5 pg of each of the fungal DNAs. (4) Plot of the two assays recording average Cqs of 31.14 ± 0.09 and 33.73 ± 1.64 for C. glabrata and C. dublinensis, respectively (Mean ± SD).

The MIQE guidelines, published in 2009 [14], are among the most cited molecular recommendations (nearly 4000 citations vs around 3500 for the MIAME guidelines published in 2001). They describe the minimum information necessary for evaluating qPCR experiments and include a checklist comprising nine sections to help guide the author to the full disclosure of all reagents, assay sequences and analysis methods and so help to minimise this kind of variability or potential inaccuracy. The guidelines suggest appropriate parameters for qPCR assay design and reporting, and have become widely accepted by both the research community and, especially, the companies producing and selling qPCR reagents and instrumentation. Implementation of these guidelines has been demonstrated to result in the publication of more complete and transparent papers, although the majority of qPCR-based papers continue to provide inadequate information on experimental detail [15].

There can be no doubt that there are a vast number of unreliable and incorrect results published that have been generated by qPCR, a relatively simple technique. This begs the obvious question of how reliable the results are that have been obtained using significantly more demanding methods. An example is digital PCR (dPCR), which involves the dilution and partitioning of target molecules into large numbers of separate reaction chambers so that each contains either one or no copies of the sequence of interest [16]. A comparison of the number of partitions in which the target is detected vs those in which it is not, allows quantitative analysis without the need for a calibration curve. Hence data analysis can be not just more precise, but also more straightforward than with qPCR. However, there are additional parameters that any reader of a publication using this technology needs to be aware of, most obviously the mean number of target copies per partition, the number of partitions, individual partition volumes and the total volume of the partitions measured. Hence the necessary requirement for the publication of the digital PCR MIQE guidelines, which address known requirements for dPCR that were identified during the early stage of its development and commercial implementation [17]. Expression microarrays and next generation sequencing incorporate an additional layer of complexity. Whilst the parameters required to ensure reliable qPCR and dPCR results are reasonably few, those required to assess the validity of expression microarrays or RNA sequencing are significantly more complex. There have been several papers investigating the effects of technical and bioinformatics variability of RNA-seq results [18], [19], [20], [21] and standards for RNA sequencing [22], [23] (http://www.modencode.org/publications/docs/index.shtml) as well as Chromatin immunoprecipitation and high-throughput DNA sequencing (ChIP-seq) [24] are being developed, but again there is no decisive push for their universal acceptance.

There is a correlation between the number of retractions and the impact factor of a journal [12]. While this could be due to the greater subsequent scrutiny afforded results published in a high impact factor journal, it surely is worth noting that despite the exclusivity of such publications and the pre-publication scrutiny they endure, such publications occur at all. It is also striking that there is an inverse correlation between MIQE compliance and impact factor [25] demonstrating the lack of adequate reporting of method for, at least, this single technique. The link between adequate reporting of methods and lack of reproducibility has become more widely accepted, and the publishers of Nature have released a coordinated set of editorial statements that admitted that the Nature publishing group had failed to “exert sufficient scrutiny over the results that they publish” and introduced a checklist to encourage the publication of more detailed experimental methods [26], [27], [28], [29], [30], [31], [32], [33], [34], [35]. However, these laudable aims have not been followed up by meaningful action and a recent analysis of Nature-published articles suggests that none of the parameters deemed to be essential for reliable RT-qPCR reporting are actually being reported [6]. We have repeated the analysis with six papers [36], [37], [38], [39], [40], [41] published in 2015 by the Nature group and found that authors and reviewers are still not heeding their own journals’ publication guidelines. The following description of the RNA extraction, reverse transcription and RT-qPCR steps published in Nature genetics is typical: “RNA was extracted with the Nucleospin II kit (Macherey-Nagel) and reverse-transcribed using the High-Capacity cDNA Reverse Transcription Kit (Applied Biosystems). PCRs were performed either using TaqMan assays with qRT-PCR Mastermix Plus without UNG (Eurogentec) or using SYBR green (Applied Biosystems). Oligonucleotides were purchased from MWG Eurofins Genomics (supplementary data). Reactions were run on an ABI/PRISM 7500 instrument and analyzed using the 7500 system SDS software (Applied Biosystems)” [39]. At least the authors published the primer sequences, although access to this information reveals that, once again, they used a single unvalidated reference gene.

What about other high impact factor journals? We have analyzed ten papers that include RT-qPCR-based results published in 2015 in Cell and its associated journals, to see whether the editors of these high impact factor journals pay any more attention to transparency of reporting (Fig. 2). Not one paper reports any experimental detail, all use inappropriate normalization procedures yet all of them include extensive supplementary information. Since there is no limit on the pages published with these online supplements, there is no excuse for omitting these basic criteria. Furthermore, all but one use single, unvalidated reference genes for data normalization, which is similar to what was found in a previous large scale study confined to colorectal cancer biomarker publications [42]. This practice was demonstrated to be unreliable a long time ago for the kind of small fold-changes these papers are reporting and should have been abandoned by now [43], [44]. Hence the validity of the reported data are, at the very least, open to doubt. The results of this analysis reinforce the shocking conclusion that the editors of these high impact factor journals have still not received and acted upon the requirement for reporting of qPCR and RT-qPCR experimental procedures.

Fig. 2.

Analysis of ten papers published in Cell [45], [46], Cell Stem Cell [47], [48], Cancer Cell [49], [50], [51] and Molecular Cell [52], [53], [54] between June and November 2015. Materials and methods and supplementary sections were screened for information on seven key parameters required to assess the technical validity of RT-qPCR assays, detailed on the x-axis. The number of papers reporting any one of these parameters are shown on the left hand y-axis. The right hand y-axis shows the number of supplemental pages published with each of the ten papers. The one paper using two reference genes (RG) uses both one and two RG in different experiments without explicit validation of the stability of these targets, either alone or in combination.

So what is the solution? Obviously, since the problem is multifactorial involving, researchers, grant awarding bodies and journal publishers, no single party is solely responsible, and therefore no single solution will suffice [55]. However, an essential step is that journal editors enforce their own, published standards and insist that reviewers scrutinize carefully the materials and methods sections of submitted manuscripts. Whilst no single reviewer has the expertise to assess the nuances of every single technique used, adherence to guidelines helps with that evaluation and every journal should retain a pool of technical reviewers who would evaluate not just the technical acceptability but, equally important, the transparency of reporting. As the data in Fig. 2 show, most papers provide no information at all about experimental procedures relating to RT-qPCR. Of course, there is also an onus on the readers of a paper to be more critical of published research findings and use their common sense to evaluate the likelihood of a result being real or not. In this age of blogs and public comments, it would be a simple procedure to email the editor handling any particular paper so that any publication could be open to reader comments regarding manuscripts on the website. This would allow the peer reviewed literature to be updated and those remaining that fail to be sufficiently transparent or where results do not support conclusions could be called out as questionable, thus gradually reducing their impact and removing them from the ongoing citations. The good intentions have been published, now it is time for editors to act.

We would offer some practical solutions for discussion:

• 1.
  Journals

  • a.
    Maintain a checklist of reporting requirements for each technique (such as provided by MIQE guidelines) and reject at submission any manuscript without associated standards checklists completed.
  • b.
    Manuscripts should be submitted with data from replicate experiments, even if these are not to be published in the body of the paper so that the reviewer sees a demonstration that the experiment is reproducible in the authors’ hands.
  • c.
    Insist that papers are reviewed by technical experts, such that all techniques reported in the paper are reviewed, using more than the traditional two reviewers if necessary.
  • d.
    Publish reader comments, queries and challenges to manuscripts in an open forum.
  • e.
    Publishers could pay for group of independent monitors that carry out an annual survey of peer reviewed publications, grouped into various specialties, and publish annual league tables of compliance with sector-specific guidelines.
• 2.
  Reviewers

  • a.
    Call to account papers without sufficient detail for an accurate review of the methods and data.
  • b.
    Declare lack of expertise to the editor for any included techniques so that these can receive additional review.
• 3.
  Authors

  • a.
    Prior to experimentation, review relevant standards databases and use the recommendations to guide the study design.
  • b.
    Consult experts in each of the required techniques.
  • c.
    Include as much experimental detail as possible in manuscripts, making use of supplementary information.

Meanwhile, we will just have to accept that many results published over the last 20 years or so will not stand up to detailed scrutiny and that it will take a while to improve the consistency and reliability of the scientific literature archive. At the same time, we offer a plea that these suggestions are accepted by the scientific community and that the concerns expressed in this editorial become redundant in the very near future. To assist this progression, we recommend that authors, reviewers and editors read the published proceedings of the symposium that analyses the challenges associated with tackling the reliability and reproducibility and of biomedical research in the UK (http://www.acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research/).