Truth and Truthiness in Translational Research

As noted previously in my Editor’s Page entitled “The 2017 March for Science,” there is a growing distrust of science in the government as well as in society (1). This distrust has been fueled, at least in part, by reports of nonreproducibility of basic and clinical research studies 2, 3. Indeed, nonreproducibility has been proffered as an explanation for the low rate of translation of scientific breakthroughs into new therapies (3). While reproducibility is certainly a critical issue in all forms of scientific endeavor, I believe that there is an alternative explanation that may explain some of the false starts that plague translational research—translational truthiness.

American television comedian Stephen Colbert coined the term truthiness on October 17, 2005, for the pilot episode of his political satire program The Colbert Report, where he explained that “We're not talking about truth, we're talking about something that seems like truth—the truth we want to exist” (4). In 2006, the Merriam-Webster dictionary announced that truthiness was the word of the year, and proposed 2 different meanings for truthiness: 1) truth that comes from the gut, not books; and 2) the quality of preferring concepts or facts one wishes to be true, rather than concepts of facts known to be true (5). I believe both of these definitions of truthiness have crept into our interpretations of early phase clinical trials.

Early phase Ib and phase II clinical trials are underpowered to study hard clinical endpoints, such as cardiovascular death or hospitalization. Accordingly early phase studies often employ a variety of softer surrogate endpoints, such as changes in biomarkers or changes in left ventricular function, which may or may not translate into clinically meaningful endpoints in subsequent phase III trials. Because there are no formal statistical guidelines for how investigators should handle secondary endpoints in phase Ib and phase II clinical trials, investigators will commonly search for statistical significance in post hoc analysis by combining or splitting treatment groups. This type of approach has been referred to as inflation bias, data dredging, or p-hacking (6). Occasionally, as investigators we euphemistically refer to these types of findings as encouraging or hypothesis-generating.

Truth, Truthiness, and Translational Research

The obvious risk in sifting through secondary endpoints in search of significant p values is that the statistically significant observations may simply represent the play of chance and, therefore, are unlikely to be replicated in phase III clinical trials. Most translational investigators know this instinctively, which begs the question of why we engage in translational truthiness time and time again? There are several obvious explanations, and at least one less obvious explanation. First, many investigators are under pressure to publish their work in high-impact journals in order to achieve professional advancement. Early phase clinical studies that focus on safety or dose-response relationships are inherently less interesting than a study that reports on a provocative secondary endpoint. A second explanation is that many small companies struggle to keep investors invested in their new therapy. Investors are more likely to be excited by encouraging endpoints than a well-done safety study or a dose-finding study. This is just human nature. However, there may also be another less obvious reason, which psychologists refer to as cognitive fluency (7). Simply stated, facts that are associated with effortless and fast cognitive processing are preferred over facts that are processed with greater relative difficulty. Cognitive psychologist Eryn J. Newman at the University of Southern California suggests that truthiness is rooted in cognitive availability, and that when we encounter new information that conforms to our existing beliefs, the more accurate the information seems, the more likely we are to accept the new information as true (8). In the context of the present discussion about truthiness in translational research, if an investigator encounters a statistically significant secondary endpoint that fits into a preconceived notion of a possible mechanism of action of a new drug or device, he/she is more likely to view the information as valid, regardless of whether or not the finding was pre-specified in the data analysis plan. In other words, as investigators we are cognitively biased to accept the truth that we want to exist because we are already familiar with it.

How can we avoid engaging in translational truthiness? Given the scientific need to search for potential efficacy signals in phase Ib and phase II clinical trials in order to properly plan larger phase III trials, it is perhaps unrealistic to require that investigators pre-specify a data analysis plan that accounts for the multiplicity of testing in all of the exploratory analysis. However, it would not be unreasonable to require that investigators pre-specify secondary endpoints in a hierarchical manner that was based on the investigators’ understanding of the mechanism of action of the drug or device. For example, secondary endpoints that best fit the mechanism of action would be ranked first, whereas endpoints that are less likely to reflect the mechanism of action would be ranked lower. When the findings are reported, secondary endpoints that were not clearly associated with a mechanism of action could still be reported, but would be recognized as being less robust scientifically. I believe that as journal editors we also bear some of the responsibility for not promoting truthiness in translational research, and that we should be more circumspect with respect to the reporting of secondary endpoints in phase Ib and phase II clinical studies that may appear exciting (and hence lead to more citations or coverage by the news media), but are unlikely to translate into new therapies because they may represent the statistical play of chance. As always, I would like to hear your thoughts about how best to balance publishing novel findings in this journal while avoiding translational truthiness, either through social media (#JACC:BTS) or by email (JACCBTS@acc.org).