Reproducibility of biomedical research – The importance of editorial vigilance

Abstract

Many journal editors are a failing to implement their own authors’ instructions, resulting in the publication of many articles that do not meet basic standards of transparency, employ unsuitable data analysis methods and report overly optimistic conclusions. This problem is particularly acute where quantitative measurements are made and results in the publication of papers that lack scientific rigor and contributes to the concerns with regard to the reproducibility of biomedical research. This hampers research areas such as biomarker identification, as reproducing all but the most striking changes is challenging and translation to patient care rare.

Concern at the robustness, reliability and relevance of much molecular biomedical research has been mounting for at least 20 years. Sporadic warnings about the lack of reliable quality control procedures in molecular diagnostics [1], reports of cross-contamination of tissue cultures [2] and evidence for between-study heterogeneity of genetic association studies for disease outcome [3] have been reinforced by numerous investigations that expose a lack of reproducibility of much basic and preclinical research and identify the accompanying huge waste of research funding (reviewed in [4]).

There are many reasons for this serious situation [5], but one of the most critical factors is the lack of transparency in the reporting of experimental detail [6], [7], which makes it difficult for reviewers and readers alike to judge the validly of the data and conclusions presented in many papers. This problem, which is probably more one of omission that deliberate, has led to solutions being proposed on how to deal with this challenge; for example, there are suggestions for improved training of researchers, enhanced emphasis on appropriate study design, easier access to unpublished primary data and open public discussion of published research [8]. Undoubtedly there is a need to encourage changes in investigator attitude, both in terms of validating reagents and understanding that ambiguity is intrinsic to biological systems [9]. Furthermore, there needs to more of an appreciation that choice of methods for sample preparation and measurement can contribute substantially to technical variance and/or bias. This is rarely a consideration when quantifying nucleic acids, yet can be major reason for poor reproducibility.

There have also been moves towards urging the research community to implement guidelines for the appropriate design, execution and reporting of research methods. There are now a number of guidelines, starting with the MIAME guidelines providing minimal information about microarray experiments [10], minimum information specifications for in situ hybridisation and immunohistochemistry experiments (MISFISHIE) [11], through MIQE [6] and digital MIQE [12] guidelines for real time quantitative PCR (qPCR) and digital PCR (dPCR), respectively, to those dealing with standards for reporting in vivo animal research [13] and pathology data gathered from animal tissues [14].

Nevertheless, it is also clear that whilst the existence of guidelines is useful, they are only part of the solution [15]. For one, although adherence places additional burdens on the researcher in terms of additional cost and effort, failure to observe basic standards of transparency has few immediate consequences, especially with regards to publication. For example, whilst there have been discussions on the need to improve the publication validation process [16], [17] and journal editors have acknowledged that they have failed in their responsibility to monitor the quality of technical information provided and promise to address these issues [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27] there is considerable evidence that this talk has not necessarily been followed by much action [4]. This is perhaps not surprising in light of their earlier failure to ensure that authors comply with primary data availability required by their own author instructions [28].

This failure is perhaps best demonstrated using an illustrative example that is representative of the types of technical errors and omissions typically found in the biomedical literature. qPCR is probably the most widely used enabling technology in molecular research [29] and results obtained using reverse transcription (RT) of cellular RNA followed by simultaneous amplification and detection of PCR amplicons has been used in tens of thousands of papers to investigate changes in RNA expression levels [30]. It is also a good example of a technology that has been subjected to extensive scrutiny and found to be seriously deficient in design, reporting and analysis [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41]. A detailed analysis of around 2000 peer-reviewed publications found that the vast majority of papers do not report sufficient technical detail and use inappropriate or misleading data analysis methods [42]. This has been confirmed by follow-up studies [4], [15], [43], [44], [45], [46]. But perhaps a more tangeable metric is that, despite the tens of thousands of research papers that have used RT-qPCR to quantify RNA expression over the last 20 years, the number of clinical tests developed measuring cellular RNAs to guide medical treatment can be counted in the ones.

In its 2013 announcement acknowledging that journals such as Nature contribute to “failures in the reliability and reproducibility of published research”, the editors admit that journals “compound” them by failing to “exert sufficient scrutiny” and not publishing “enough information for other researchers to assess results properly” [18]. Yet they do not appear to be implementing resolutions to rectify failings. A paper published in a top nature journal in the last two months demonstrates a recent example of this failure. The manuscript, which we will not name, lacks the most basic information required to assess the validity of the experimental protocols, contains critical, yet basic errors that should have been detected during the review process, and makes conclusions based on results that are probably flawed.

• 1The qPCR protocol published in the supplementary information is limited to six lines and provides no information about RNA integrity, quality or quantity used or the conditions of the reverse transcription or qPCR efficiency. This is despite a wealth of published evidence that highlights the importance of RNA quality [47], [48], the variability of the RT step [49], [50], [51] and the importance of appropriate PCR conditions [52], [53], [54], [55] for achieving reliable results.
• 2Data analysis was performed using a less accurate approach, as the chosen method [56] requires an invalid assumption of 100% PCR amplification efficiency across all assays and directly compares the difference in Cqs of different primer sets (a number that will vary with threshold settings if, as is often the case, the assays have different shaped amplification plots). That approach was superseded by a method that only compares the same assay between different samples and factors in the PCR efficiency to provide a far more accurate estimation of the differences [57], [58].

The authors use a single, unvalidated reference gene for normalisation, an approach shown to be inappropriate as far back as 2002 [59]. How do the authors know the reference gene is not effected by the study?

The reference gene in question, β-actin, is not an ideal reference gene in the model investigated in this study as it was reported to have “a tendency to be down regulated” in the same types of tumour samples [60].

• 3An analysis of the primer sequences published in the supplementary information using PrimerBlast finds no target templates for the one of the primer pairs. A BLAST search finds no 100% sequence identity of another forward primer with any target, whereas the reverse primer has 100% sequence identity with a different, unrelated human mRNA; the study used a mouse model. Another reverse primer has a single mismatch with all published target sequences. Yet another primer pair targeting a specific transcript splice variant has sufficient sequence identity with all other variants to suggest it is not specific at all and could also amplify the other variants.
• 4While some of these observations may be less critical if the reported expression changes differed by orders of magnitude, conclusions are drawn on the published qPCR based on 3–5-fold differences that are well within the variability range established for the reverse transcription step [51].

Whilst such differences are measureable using RT-qPCR, with the points highlighted above, doubt is introduced to the conclusions associated with this technique and, arguably far worse, with the lack of detail presented in the manuscript, making it almost impossible to reproduce the study to corroborate the findings. Yet it will not take the reader long, should they be so inclined, to find numerous published studies, where these omissions apply, frequently purporting to find differences of less than two-fold, a feat that, without the most rigorous selection of multiple reference genes, is simply impossible to achieve with any degree of accuracy.

These parameters are elemental for qPCR and it is extraordinary that such lacks of adherence have been allowed to slip through the vetting process. It is truly astonishing that this is but one of hundreds, if not thousands of publications utilising qPCR that are published with similar shortcomings. Biomolecular Detection and Quantification (BDQ) was conceived as a journal committed to publishing molecular studies that adhere to best practice in the areas of study design, measurement, data analysis and reporting. As editors, we handle numerous manuscript and a continuous theme is the initial absence of comprehensive information on aspects of the work being reported. This is corrected in papers published by this journal, but clearly is not done so by the editors of many other journals. It is unfortunate that the editors of Nature, and many other journals, continue to publish papers that have not been adequately reviewed and where the information provided does not follow Nature’s own recommendations. Is it a surprise that we continue to face a crisis of reproducibility and that so many papers are published that report results that are meaningless?

Handled by Jim Huggett