Are Preregistration and Registered Reports Vulnerable to Hacking?

To the Editor:

Preregistration of study hypotheses has been proclaimed as the “revolution”¹ to curb publication bias and data dredging such as p-hacking and HARKing, i.e., hypothesizing after results are known. Many publishers encourage preregistration in their submission policies,² and a similar trend is being seen in registered reports, which are preregistration of a study plan, prereview, and acceptance for publication, all before data/results are seen.³ Sharp growth of platforms to facilitate preregistration, e.g., the Open Science Framework, has ensued.

A recent analysis on preregistered studies showed an increase in reports of null findings,⁴ which should be a reassurance against the specter of p-hacking. However, new evidence indicates that preregistration is being circumvented,⁵ with authors often deviating from registered plans and not fully disclosing it.

Substantial variation in results and conclusions from studies even with the same design has been shown in several recent cases. One “crowdsourcing” study from Advances in Methods and Practices in Psychological Science showed notable variability in results drawn from the same research question and data set, allowing liberty in analytic choices.⁶ Further, the British Journal of Anaesthesia reported different conclusions even when fixing the methods and results.⁷ Even when the study design is very strict and not exploratory, the explosion in analytic configurations—the so-called “garden of forking paths” defined by Gelman and Loken—⁸has been inevitable and uncontrollable, especially with big data; one such typifying example is online social media studies where sampling is likely biased and irreproducible.⁹

Shall we be suspicious that preregistration will eventually degenerate into a mere bureaucratic exercise, thus becoming a poor solution to the reproducibility crisis? Registered reports could also suffer from erosion, if the acceptance decisions were driven by substantial confirmation bias or other forms of cognitive bias, e.g., collaboration networks, prestige of researchers, or institutions.

If proper preregistration can be easily sidestepped, we cannot rely on research ethics alone to curb the reproducibility crisis. We need solutions designed to work in the worst (pessimistic) case where a researcher is a priori likely to end up with voluntary or involuntary data dredging—although this is not the place to discuss why, what, or whom to blame. Perhaps keeping up with the enforcement of replication could attain the same effect with or without preregistration or registered reports. We have to cautiously remind readers that reproducibility and publication bias are intertwined but not the same; better transparency—which is one of the intended consequences of preregistration—can be a necessary prerequisite to both, yet not sufficient.

The bottom line is we do not have a formal model of what to expect from preregistration and registered reports yet. We need a solid theory that models the causes of unmet reproducibility, which we could use to ideate, simulate, and test to what extent interventional, counter factual strategies—one being preregistration—would be successful.