The scandal of unpublished research

I was recently asked, as a statistician, to endorse a statement to the effect that all results of scientific research should be published soon after completion of the investigation. My immediate reaction was that I could not do this. In my mind a major issue was the definition of ‘published’, my preferred term being ‘made publicly available’. It is true that researchers owe it to their sponsors and research participants to make results (and even, I would argue, primary data) available for critique and further analysis. It does not follow that authors are automatically entitled to a paper in a peer‐reviewed journal. Sadly, this is consistent with my practice: as a reviewer of quantitative research in health care I frequently recommend rejection of papers. However, the matter deserves careful thought. There are two important and competing principles to be considered. The first is the obligation of the researcher to make results available; the second is the duty of an editorial board to maintain quality of the papers which it publishes.

Medical research has had a chequered history. Years ago, a clinical trial was more likely to be accepted for publication if the result was positive, even if the study was of dubious quality. This led to serious bias in the clinical evidence base. Nowadays many editors claim to judge papers by their quality rather than by their results, although there is still anecdotal evidence of researchers struggling to get negative findings of high‐quality trials published. A more serious issue is the deliberate withholding of results that are not favourable to the sponsor. Consequently, the Faculty of Pharmaceutical Medicine has recently issued a statement that ‘It is unethical to withhold the publication of any results of research on any pharmaceutical product whether the results are positive, negative or inconclusive’ (Bickerstaffe et al. 2006). It is clearly in the interests of transparency and statistical efficiency to make results available to other researchers and practitioners.

However, misrepresentation of results cannot be encouraged. In a seminal editorial, Douglas Altman (1994) described the scandal of poor medical research outlining serious problems that were common in published papers. In an attempt to overcome this, several sets of guidelines have been developed. For reporting clinical trials there is the Consolidated Standards of Reporting Trials (CONSORT) statement, which now has several extensions for different types of trial designs (Schulz et al. 2010). Some journals have adopted CONSORT in the sense that that they will only publish clinical trial papers that conform to the guidelines. The problem of confounding variables in nonexperimental studies makes them difficult to design and report (von Elm & Egger 2004) and Strengthening the Reporting of Observational Studies in Epidemiology has been produced (von Elm et al. 2007). There are also guidelines for meta‐analyses [Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) and Meta‐analysis of Observational Studies in Epidemiology (MOOSE)], for diagnostic tests [Statement for Reporting of Diagnostic Accuracy (STARD)] and for studies of reliability [Guidelines for Reporting Reliability and Agreement Studies (GRRAS)]. A very helpful website EQUATOR (2010), designed to enhance the quality and transparency of health research, provides a compendium of these and other guidelines for researchers in health care.

It would therefore seem that there is no scientific reason to reject a quantitative paper, but the reality is somewhat different. Not all authors follow the guidelines, and sometimes the lack of detail or clarity in a paper makes it difficult for reviewers to assess the objectives of the study, the relevance of the methods and the appropriateness of the conclusions. Rightly or wrongly, reviewers often then infer that these inadequacies in reporting will be associated with inadequacies in design or analysis. Moreover, it is not the task of the reviewer to re‐write a paper: consequently papers get rejected. This is an unsatisfactory situation and can be avoided.

If an inadequately designed study does run then all the deficiencies and likely implications need to be acknowledged in any publication. The work can then be judged on its merit. I would still argue that the results and data should be made publicly available but not necessarily in a peer‐reviewed journal. However, such studies should not be allowed to run. Many study designs should have a clear protocol, which requires scientific and ethical approval. Poor quality studies should be ‘rejected’ at this stage because bad science is unethical. No study is perfect but a well‐designed study, which addresses a relevant research question, merits a well‐written paper following the appropriate guidelines acknowledging its own weaknesses and not overstating the conclusions. A paper like this does merit publication, and we should aim for all research to generate such papers. Only then can the scandals be laid to rest.