What the replication crisis means for intervention science

Abstract

The provocative paper by Ioannidis (2005) claiming that “most research findings are false” re-ignited longstanding concerns (see Meehl, 1967) that findings in the behavioral sciences are unlikely to be replicated. Then, a landmark paper by Nosek et al. (2015a) substantiated this conjecture, showing that, study reproducibility in psychology hovers at 40%. With the unfortunate failure of clinical trials in brain injury and other neurological disorders, it may be time to reconsider approaches not only in clinical interventions, but also how we establish their efficacy. A scientific community galvanized by a history of failed clinical trials and motivated by this “crisis” may be at critical cross-roads for change engendering a culture of transparent, open science where the primary goal is to test and not support hypotheses about specific interventions. The outcome of this scientific introspection could be a paradigm shift that accelerates our science bringing investigators closer to important advancements in rehabilitation medicine. In this commentary we offer a brief summary of how open science, study pre-registration and reorganization of scientific incentive structure could advance the clinical sciences.

Can the “Crisis” evoke a Paradigm Shift in Intervention Science?

The provocative paper by Ioannidis (2005) claiming that “most research findings are false” re-ignited longstanding concerns (see Meehl, 1967) that findings in the behavioral sciences are unlikely to be replicated. Then, a landmark paper by Nosek et al. (2015a) substantiated this conjecture, showing that reproducibility hovers at 40%. While one might argue about the terminology and causes used to describe the situation facing behavioral scientists (“replication crisis”; cf. Maxwell et al., 2015), there is growing consensus that there is room for improvement in approach and methods used and these concerns have spared few areas of research in the health sciences (Benjamin et al., 2018; Friesike et al., 2015; Goodman et al., 2016; Munafò et al., 2017; Nosek et al., 2015b).

Central to concerns in the replication crisis is a demand for larger data sets for maximizing statistical power while simultaneously questioning the incentive structure for publishing only statistically significant findings (Cohen, 2016) and adhering to the philosophically flawed null-hypothesis significance testing (NHST) (see Henkel, 2017; Schneider, 2015). One might argue that the goals of rehabilitation medicine and, cognitive remediation specifically, sit at an unenviable intersection occupied by studies with small(ish) sample sizes aiming to detect often subtle effects buried in the noise of inter and intra subject variability (Park and Ingles, 2001; Rohling et al., 2009; Sitzer et al., 2006; Wykes et al., 2011). To make matters worse, the goal to demonstrate statistically significant effects in interventions must be achieved while surpassing increasingly stringent statistical thresholds that have been proposed to handle replication problems based in NHST (Benjamin et al., 2018).

With the unfortunate failure of clinical trials in brain injury and other neurological disorders, it may be time to reconsider approaches not only in our interventions, but also how we establish their efficacy. In the case of pharmacologic interventions in acute traumatic brain injury (TBI) in particular, the staggering 100% rate of failure (see Stein, 2015) has left the community to ponder how so many promising interventions could survive early studies, only to falter so impressively during phase III clinical trials (see Menon & Maas, 2014). To understand where things have gone wrong with both behavioral and pharmacologic interventions, one place to look would be the structures in place guiding how we set-out to study the phenomenon in the first place.

The “replication crisis” not only highlights the limitations of traditional statistical approaches and the circumscribed requirements for scientific publication, but it leads to questions about the culture of science. The culture of medical science includes an incentive structure that requires innovative approaches, novel findings, and validation through statistical significance via NHST. If NHST fails, researchers commonly test many post-hoc hypotheses in order to fit the data (i.e., p-hacking; Head et al., 2015). Unfortunately, this culture does not promote efficient science or the open study of clinical research because researchers are not incentivized to publish or share results with the scientific community when interventions fail. It is essential for the scientific community to be aware of both successes and failures of well-designed clinical interventions, making null findings a vital part of the scientific landscape and ultimately expediting research.

There is also an important need to understand how group data from an intervention study can inform us about the efficacy of any intervention in the individual. Drawing from ergodic theory, Molenaar (2004) predicted that cases where statistical estimates based on group data would rarely reflect processes within individuals. In fact, empirical studies show that individual and group estimates do diverge considerably (Fisher et al., 2018; Seghier and Price, 2018). In the worst case, we would need a different mechanistic model for each person to treat that person’s cognitive deficit or disease process. In other words, it is unclear how to understand and treat cognitive dysfunction with group data without knowing how group-level inferences map into individual processes over time. Thus, directly measuring within-subject variability is a central feature to precision medicine to determine which failures to replicate are driven by a lack of person-level analysis. In rehabilitation medicine, reproducibility is at least partially linked to how well group-level data represent individual responses to treatment. Therefore, the person-level is the appropriate level at which interventions should be conceptualized and studied and our group-level claims should be rooted in models of processes that are validated within persons. These processes could be tested for validity with convergent behavioral, neuroimaging, and other biological measures to identify mechanisms of action that could target precision interventions. In principle, starting with the individual as the basic unit of study could reduce the time needed to discover mechanisms of disorders and change in clinical samples.

What to do?

So where do these challenges leave the intervention scientist? While the problems with NHST have been described for decades (see Cohen, 1994; Rozeboom, 1960), the advent of the “crisis” may bring new perspective and a collective imperative to change how we collect, analyze, and disseminate clinical research findings. A general solution to many of the concerns expressed here is to advocate for transparent and open science. One vital step in an open science landscape is study pre-registration, where interventionists using established methods for even small sample or single-subject designs (e.g., multiple baseline, reversal designs) can register the study goals and the results can be accessed by the scientific community regardless of the study outcome. Because all findings are available to the scientific community, open science allows us to eliminate ineffective interventions and aids in identifying interventions that work through replication. Moreover, providing access to the outcomes of all studies can allow investigators to view treatment effects and their variances as a continuum. This could be one way to mitigate the community’s reliance on arbitrary NHST criteria and the tendency to artificially declare some findings significant and others not (Wasserstein et al., 2019). As others have begun to note more urgently, scientists should accept a statistical and epistemic worldview that embraces uncertainty at its foundation (McShane et al., 2019).

Perhaps unsurprisingly – and encouragingly, recent analysis of pre-registered studies revealed a sharp rise in the publication of null findings and replication studies (Warren, 2018). Because of this, pre-registration will also reduce the tendency for interventions to appear successful largely because they have been propped-up by well-meaning, but naturally biased, researchers who have been incentivized to defend interventions as opposed to critically testing their efficacy. Open science and study preregistration may also help to standardize methods, which are currently lacking in some areas of the clinical neurosciences including functional brain imaging (see Esteban et al., 2019; Hallquist and Hillary, 2018). Moreover, data sharing fostered in open science holds additional opportunities to test the reliability of interventions and their generalizability between research labs. Open science initiatives can ultimately lead to data repositories that permit estimates for population, sample level, and person level effect sizes. By extension, data sharing provides estimates of patient response distributions that can be used as “priors” for testing a range of hypotheses (including the null) within a Bayesian framework which is becoming increasingly accessible for statistical analyses (e.g., Bayes factor estimates) (Hoijtink et al., 2019). This effort should consider the extent to which aggregate data represent any process within the individuals within groups to quantify potential process variability, clarify mechanisms, and tailor treatments. Finally, for the goals presented here to be realized it requires a change in scientific culture so that researchers are awarded and promoted based upon their dedication to support open science, data sharing, and study replication, (Gernsbacher, 2018).

One goal of this Special Issue to address current challenges in rehabilitation medicine. It is an important time for clinical interventionists to have this conversation. The concerns outlined above with regard to NHST and scientific incentive structure are certainly not new and are not the sole reason for difficulties advancing rehabilitation medicine. However, a scientific community galvanized by a history of failed clinical trials and motivated by this “crisis” may be at critical cross-roads for change engendering a culture of transparent, open science where the primary goal is to test and not support hypotheses about specific interventions. The outcome of this scientific introspection might be a paradigm shift that accelerates our science bringing investigators closer to important advances in rehabilitation medicine.