Evolution of evidence-based medicine to detect evidence mutations

Introduction

Frustrating to some and exhilarating to others, medical knowledge is in continual evolution. Medicine’s evidence base evolves in many ways, sometimes in ways that can be compared to random mutations (such as adverse effects of a drug leading to investigation of its use where the effect could be beneficial) and other times through selection pressure that allows substandard evidence to flourish in ecological niches (such as a research community accepting existing reports as an established and unquestioned foundation). We describe several ways in which pathways resembling aberrant evolutionary changes distort our evidence base (how has our evidence evolved?), and discuss areas where evidence-based medicine (EBM) processes can adapt (how does EBM need to evolve?).

The need to unearth buried evidence

Evidence published as original research reports in peer-reviewed journals which are indexed in MEDLINE is more likely to be identified, evaluated, synthesised and referenced in our view of medical knowledge than evidence which is not published in this manner.

Evolution itself, described in On the Origin of Species, was rapidly accepted following its publication. However, ‘natural selection’, without a plausible biologic mechanism, was not accepted for another 60 years. Mendel’s work on the heredity of pea plants, published around the same time, could have provided this support. However, the link was not made until Mendel’s work, buried in an obscure journal, was rediscovered decades later.¹

Similar delays occur in our understanding of evidence for guiding clinical practice. Sometimes not all research findings on a topic are published, with the remainder being either not reported, withheld or ‘buried in plain sight’ by reporting in places not traditionally viewed. For example, the publication of the Celecoxib Long-term Arthritis Safety Study reported less gastrointestinal toxicity with celecoxib than with other non-steroidal anti-inflammatory drugs.² As was later revealed, this report was based on interim results; data reported to the Food and Drug Administration (FDA) describing results of the complete study did not support a difference in gastrointestinal toxicity. It wasn’t until this report was unearthed and these results published in two letters to the editor that the medical public was made aware of this lack of difference.^3,4

Similar problems occur with reports of off-label use for medications. While the published evidence of the treatment of molluscum contagiosum with imiquimod reported a partial benefit when used in children,⁵ two unpublished studies found no benefit. Although these unpublished data were publicly available through FDA reports,⁶ we learned about them years later from a ‘perspectives’ article⁷ that typically would not be classified as evidence or identified in usual evidence searches.

Even research buried in pharmaceutical manufacturers’ archives can be accessed, although with substantial work. Unpublished research, made available following persistent public pressure, revealed the ineffectiveness of reboxetine for treating depression and oseltamivir for reducing complications of influenza.^8,9

The evidentiary Piltdown Man: limits to scientific veracity

Invalid evidence (whether fraudulent in origin, or inappropriately analysed or synthesised) can have serious consequences when it contributes to our evidence base.

The ‘Piltdown Man’ was a fossilised skull and jawbone discovered in Piltdown, East Sussex in 1912. At the time it was thought to represent evidence of a previously unknown species of human (Homo piltdownensis). Actually, a human skull and orangutan jawbone had been artificially aged and modified; it took 40 years to be revealed as a forgery.

Some research results are also distorted. Two of the key trials supporting the use of perioperative beta blockers were conducted by a researcher who had five trials (including one of these two trials) discredited in 2011 due to negligence and scientific misconduct.¹⁰ These articles were not retracted and the effect on our evidence base was not reported in a systematic way until a meta-analysis was published 2 years later.¹⁰ Guideline developers are still wrestling with revising current recommendations following these developments.¹¹

At other times disparate ‘species’ of studies are combined, resulting in misleading conclusions. Two systematic reviews reported that dipeptidyl peptidase-4 inhibitors (gliptins) reduce the risk of cardiovascular events in patients with type 2 diabetes compared with placebo.^12,13 However, this conclusion was primarily supported by a single atypical trial in which patients had severe renal failure, and the placebo group only received placebo for 12 weeks followed by glipizide for 42 weeks.¹⁴ Critical appraisal methods for systematic reviews typically include checking for similarity of studies being combined, but may not include evaluation of the relative influence of atypical studies. Two subsequent large trials have since reported that gliptins do not decrease risk for cardiovascular events and may increase risk for hospitalisation for heart failure.^15,16

Aberrant evidence evolution due to inbreeding

Synthesis of evidence with meta-analysis which finds consistency and increases precision of effect estimates is an accepted method for advancing our medical knowledge with confidence in the findings we consider most valid. But the increased confidence is misleading if it is based on systematic replication of a misleading view of the evidence.

Biased medical research can appear strong and consistent when the same flaws are compounded, a concept we refer to as academic inbreeding. Genetic inbreeding occurs when children are born to consanguineous parents. The likelihood of congenital disorders increases because genetic faults are more likely to be combined.

A recent systematic review concluded increased mortality with sodium restriction in patients with heart failure treated with daily diuretics and fluid restriction.¹⁷ Since diuretics, fluid restriction and sodium restriction are common components of heart failure treatment, this analysis could have been considered a major evolutionary jump in our understanding.

However, the review and all six trials included in the analysis had the same or overlapping authors. The review was retracted soon after publication because of concern over reporting of duplicate data and the inability of the authors to provide the original data for verification.¹⁸ Even without duplicate publication amplifying the results, the data represented a ‘congenital defect’ passed on in the literature. Trials included in the review used abnormally high doses of furosemide (up to 1000 mg/day) so it is unlikely the results apply to conventional clinical practice. If you overdose someone on diuretics, salt restriction may kill them, but this does not apply for routine medical care.

Evolving EBM to remain valid

EBM needs to develop new methods as new evidence problems occur. To overcome the problem of buried evidence we must become evidence archaeologists. Following one of the largest evidence digs in recent history, reviewers unearthed 22,000 pages of clinical data that undermined oseltamivir’s role.¹⁹ They noted such discrepancies that they abandoned use of published trial reports and are developing new methods of reliable evidence synthesis to identify and appraise clinical study reports from industry.

To overcome the problem of passing forward invalid evidence interpretations we have to look carefully behind the published prose for evidence of fraud or inappropriate analyses. We need to consider what data are selected and how they are combined in meta-analyses and not simply accept the statistical result as reported. Along with rigorously applying our own critical appraisal for validity and relevance, we need to use other important evidence quality issues that are brought to light in information exchanges, such as letters to the editor and discussion lists.

To overcome academic inbreeding, we have to check when the preponderance of data comes from a single source such as the same institution or overlapping authors, and we need to recognise that when a substantial proportion of studies have the same bias affecting them a meta-analysis showing consistency and increased precision does not make the evidence any more reliable.

We need to continue to evolve our search and critical appraisal methods to keep pace with new evolutionary paths that evidence may take. As the Evidence Based Medicine Renaissance Group promotes a reappraisal and renaissance of EBM (a campaign described as ‘real EBM’), advancing our evidence capture and analysis should be coupled with other strategies to better integrate it with improved contextualisation and shared decision making.²⁰

Declarations

Competing interests

BSA is a full-time employee of EBSCO Publishing, Inc. which publishes DynaMed. ZF and AFS receive consulting fees from EBSCO Publishing, Inc. SAS and MES are principals in Delfini, LLC which provides consultation services for evidence-based clinical improvement training and other services.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. BSA, ZF, and AFS used time under contractual employment or consulting with EBSCO Publishing, Inc. to edit the paper.

Guarantor

BSA

Ethical approval

Not applicable

Contributorship

BSA identified four unique ‘game-changing’ instances in DynaMed editing in 2013 that stimulated attention to areas where evidence analysis required extraordinary evaluation beyond traditional critical appraisal protocols. BSA, AFS, ZF, SAS, and MES engaged in multiple conversations regarding thematic considerations and how this relates to other EBM developments, including dialogues on the evolution of EBM. AFS conceptualised classic history of evolution concepts to provide a thematic approach. All authors reviewed the manuscript for accuracy and readability.

Acknowledgement

None

Provenance

Not commissioned; peer-reviewed by Julie Reed