On the Use and Misuses of Preregistration: A Reply to Klonsky (2024)

Abstract

In his commentary, Klonsky (2024) outlines several arguments for why preregistration mandates (PRMs) will have a negative impact on the field. Klonsky’s overarching concern is that when preregistration ceases to be a tool for research and becomes an indicator of quality itself (a primary example being preregistration badges), it loses its intended benefits. Separate from his concerns surrounding policies like preregistration badges, Klonsky also critiques the practice of preregistration itself, arguing that it can impede our use of other valuable research tools (e.g., multiverse analyses, exploratory analyses). We provide a response to Klonsky’s concerns about preregistration and related policies. First, we provide conceptual clarification on the purpose of preregistration, which was missing in Klonsky’s commentary. Second, with a clearer conceptual framework, we highlight where some of Klonsky’s concerns are warranted, but also highlight where Klonsky’s concerns, critiques, and proposed alternatives to the use of preregistration fall short. Third, with this conceptual understanding of preregistration, we briefly outline some challenges related to the effective implementation of preregistration in psychological science.

Klonsky (2024) outlines several arguments for why preregistration becoming a norm in psychological science will be detrimental for the field. His primary concern is that preregistration will cease to be a tool for research and will become instead a target itself (a primary example being preregistration badges), at which point, it cannot achieve its intended goal. Klonsky points to psychology’s unfortunate history of turning tools into superficial indicators of robust findings (e.g., p<.05), which in turn has led to negative consequences (e.g., in Klonsky’s view, it is the driving force behind the replication crisis in psychology). Although Klonsky’s overarching criticism is focused on how the mere presence of a preregistration badge or document is being conflated with strong science, he also criticizes preregistration more generally, arguing that it can impede our use of other valuable research tools (e.g., multiverse analyses, exploratory analyses).

Specifically, Klonsky (2024) outlines several concerns with what he calls “preregistration mandates” (PRMs). Klonsky defines PRMs as policies that seek to elevate preregistration to a norm in the field, best exemplified by preregistration badges (p. 3; emphasis ours). But Klonsky also uses PRMs to refer to preregistration becoming a norm in the field in general, which he believes will negatively impact ideal scientific practice.¹ Labeling these distinct implementations of preregistration with the same PRM term blurs legitimate concerns about unintended consequences of preregistration policies (e.g., Campbell’s Law and the inadvertent consequences of preregistration badges) with misguided concerns about the practice of preregistration, itself (e.g., preregistration becoming a norm in the field will impede ideal scientific practice). In what follows, we separate these distinct components of Klonsky’s criticism and discuss them accordingly.

The focus of the present response is threefold. First, we provide conceptual clarification surrounding the goal of preregistration, which was missing from Klonsky’s commentary. Second, by providing a clearer conceptual background, we highlight where some of Klonsky’s concerns about preregistration remain warranted, but also highlight where Klonsky’s criticisms and proposed alternatives to the use of preregistration fall short. Third, with a clearer understanding of preregistration in mind, we briefly outline some challenges and, importantly, opportunities regarding the effective implementation of preregistration in psychological science.

Campbell’s Law and Methodological Reform in Psychology

Klonsky’s primary concern is that preregistration incentives like badges are becoming an example of Campbell’s Law, which states that the more a tool or indicator is used for decision-making, the more likely the tool or indicator will distort the process it was designed to initially monitor (Campbell, 1979). He highlights empirical evidence to suggest that preregistration badges in flagship journals like Psychological Science are not functioning as intended (e.g., published papers with preregistration badges contain non-trivial, undisclosed deviations from the preregistration; Claesen et al., 2021). Overall, Klonsky’s concern about Campbell’s Law and the potential for history to repeat itself with regard to incentivization policies for preregistration (i.e., badges) is well-placed. It is critical that we learn from past missteps and work to ensure that we understand the ways preregistration can and cannot improve our science. Moreover, we share Klonsky’s broader concerns about common research practices that negatively impact the credibility of research findings (e.g., undisclosed use of researcher degrees of freedom; Simmons et al., 2011), but also share his excitement for the tangible methodological reforms occurring in psychological science.

Although Klonsky’s commentary on preregistration and Campbell’s Law raises an important concern about the mere use of a tool becoming a shallow indicator of rigor, many of his other concerns (i.e., those focused on preregistration becoming a norm in the field) reflect misunderstandings of the principles and applications of preregistration. We believe that when used as intended, preregistration as a norm will improve the field. Importantly, as Klonsky notes, the “used as intended” part is the crux. In order for preregistration to be used as intended, prominent misconceptions must be addressed.

Clarifying the Goal of Preregistration

In his commentary, Klonsky does not provide a clear definition of what preregistration is designed to do. Klonsky names a variety of potential benefits related to the use of preregistration including improved replication rates, increased thoughtfulness in study planning, increased statistical power, clearly demarcating confirmatory and exploratory tests, and the potential to constrain biasing effects of researcher degrees of freedom. However, these are better understood as indirect consequences or “positive externalities” of preregistration (Lakens, 2019). There is nothing about preregistration that guarantees these secondary benefits will be realized—one can always preregister an under-powered, poorly designed study with inappropriate analyses, vague hypotheses, poorly chosen measures, and non-specific inferential criteria (Rubin, 2020; Szollosi et al., 2020). Moreover, preregistration will not necessarily stop researchers from inadvertently capitalizing on researcher degrees of freedom (Navarro, 2020). We agree with Klonsky that preregistration is not a sufficient condition for good science. It should not, by itself, be taken as a marker of credibility.

Because preregistration cannot guarantee the benefits listed by Klonsky, it is essential to be clear about what preregistration is intended to do: the goal of preregistration is to increase the transparency of the research process in order to allow evaluation of the severity with which a particular hypothesis or empirical claim has been tested (Lakens, 2019). Preregistration is not designed to increase credibility per se; it is designed to increase transparency which allows evaluations of credibility (i.e., severity). A test is considered severe when it has a high probability of demonstrating a claim is false when it is false (Mayo, 2018; Lakens, 2019). More generally, the principle of severity provides an answer to the question, “when do data provide good evidence in support of a claim or hypothesis?” (Mayo & Spanos, 2011). The short answer is that more severe tests provide better evidence.

The concept of a “severe test” clearly applies to straightforward confirmatory research questions in psychology (e.g., “Does a therapeutic intervention reduce symptoms of depression?”, “Does amygdalar activation predict psychopathy?”) but evaluating the severity of a test is important to various other empirical claims in psychology as well (e.g., Does a newly developed scale demonstrate predictive improvements over an established scale?; Is a measure unidimensional or does it contain subfactors?; Which hierarchical model fits the data best? How strongly does psychopathy in adolescence predict psychopathy in adulthood?). Thus, the goal of preregistration is relevant for a wide array of psychological research questions.

Being able to transparently evaluate the severity of a test is critical, because in many areas of psychology, available theories do not constrain all possible choices (e.g., which measurement instrument to use; how to exclude outliers; which covariates should be included; Lakens, 2019). Others have highlighted that the various crises in psychology (e.g., the replication crisis, the generalizability crisis, the theory crisis) share a core problem: empirical claims are not based on severe tests (Claesen et al., 2022). The field would likely be in a different place if it was aware of how each study was conducted and, thus, the severity of the test it offered. We would not weigh very heavily papers that extensively searched the data for significant results and then wrote the introduction to predict those findings. We would not weigh very heavily papers that reported the single study out of 10 that found significant results. We would not weigh very heavily papers that determined outlier status or covariate inclusion based on what these decisions did to the results. When such decisions remain opaque, severe tests cannot be distinguished from non-severe ones. The field requires transparency if it is to move forward—this is the core of Open Science. Klonsky’s claims about the benefits of preregistration (e.g., restraining researcher degrees of freedom, increasing power) are not unique to his commentary. These secondary benefits of preregistration are oft-cited reasons to preregister a study. There is correlational evidence to suggest that preregistration, particularly when peer reviewed (i.e., by using a Registered Report format), is related to increased rigor of a study (e.g., Soderberg et al., 2021) via these secondary benefits. Nonetheless, the field’s shift towards understanding preregistration as a tool to increase transparency in order to allow the evaluation of test’s severity reflects conceptual advances made in regard to preregistration, and helps connect the use of preregistration to an explicit philosophy of science (Lakens, 2019; Mayo & Spanos, 2011; Meehl, 1990). What is most noteworthy about the conceptual and theoretical development surrounding preregistration is that it explains why researchers want to know which analyses were exploratory versus confirmatory, what statistical power was for a primary test, whether researchers examined a separate outcome after initial null findings, and so on—this information is critical to evaluating the extent to which a claim has been severely tested–underpowered, exploratory studies with numerous undisclosed researcher degrees of freedom provide very weak (i.e., non-severe) tests. Absent the transparency provided by preregistration, these weak tests cannot be distinguished from the stronger tests. With this conceptual clarification in mind, we are better able to evaluate the validity of Klonsky’s concerns with preregistration.

First, Klonsky’s concerns about the presence of preregistration being treated as a sufficient indicator of robustness still deserves attention. We agree that viewing preregistration in this way lends itself to the kind of “checklist science” that Klonsky rightfully critiques. As a field, we must correct course in light of evidence that preregistration is, at times, being used not as a tool, but as a superficial signal of rigor. Although a preregistration badge or the presence of a link to a preregistration provides some indicator of transparency, as noted by Klonsky, it tells us nothing about the quality of the study. We emphasize Klonsky’s call for researchers to resist simplistic thinking that regards the presence or absence of a research tool as a sufficient indicator of either good or bad science. As such, it is worth considering whether preregistration badges serve much purpose at this stage of the reform movement.² The conceptual framework for preregistration offered above also clarifies why the use of preregistration does not itself serve as an indicator of robust science. Preregistration simply allows others to evaluate the extent to which a particular claim has been severely tested; it makes the process more transparent and, thus, open to evaluation. Preregistration does not, in itself, guarantee a severe test. In some cases, a preregistration will reveal that the methodological approach produced a test that lacked severity and was, therefore, unable to provide strong evidence for a hypothesis or empirical claim. In other cases, there may be so many deviations from the preregistration that readers will take the findings with a much larger, and more appropriate, grain of salt than they would have in the absence of such a preregistration (Lakens, 2024).

Importantly, the conceptual clarification offered above also indicates why Klonsky’s separate concern over preregistration becoming a norm is misguided. It is difficult to see how it would be harmful to the field if all studies were made more transparent through preregistration in order to facilitate the evaluation of the severity of the tests they provided. While this is dependent on researchers assembling preregistrations that do, in fact, increase transparency and facilitate this kind of evaluation (and those preregistrations being read and evaluated by peer reviewers and subsequent readers), it is important to separate criticisms of a tool from criticisms of the tool’s misuse. Klonsky’s overarching concern is focused on the misuse of preregistration. Advocates of preregistration will find no reasons to disagree with concerns about its misuse. However, Klonsky’s other concerns focused on the use of preregistration itself reflect a misunderstanding of preregistration’s limits.

Misunderstanding the Limits of Preregistration

Separate from his concerns about the presence of preregistration being uncritically accepted as an indicator of strong science, Klonsky highlights three concerns with preregistration becoming the norm. First, Klonsky argues that preregistration becoming prevalent will, “establish a norm that researchers should choose one or a subset of reasonable analyses and not others.” (p. 14). This is simply not true, but it is a common misperception regarding preregistration. A number of previous articles have highlighted this misperception and discussed how preregistration can accommodate multiverse analyses or the various other robustness checks described by Klonsky (e.g., see Box 2 in Hardwicke & Wagenmakers, 2023; Table 1 in Chambers & Tzavella, 2022; Figure 1 in Simmons et al., 2021). Klonsky does not provide data to support his claim that preregistrations offer “a blanket reward for a selective rather than comprehensive approach to understanding data, even though the latter approach is optimal.” In fact, when used together, robustness checks and preregistration can be a particularly effective way to demonstrate that a claim or hypothesis has been severely tested. The same is true for Klonsky’s emphasis on replication. The impact of replication results can be amplified by being preregistered; one need look no further than Bem (2011) to understand why non-preregistered replications can be far from compelling. Importantly and ironically, all of Klonsky’s methodological alternatives to preregistration (i.e., robustness checks, multiverse analyses, and replications) can most effectively advance the field when they are preregistered and, thus, transparent.

When understood as a tool to increase transparency in order to facilitate the evaluation of severity, there is no reason to fear that preregistration will detrimentally confine researchers to a limited set of analyses. Researchers should select the analysis or set of analyses that will provide the most severe test(s) of their hypotheses. For examples, Watts and colleagues (2023), in discussing “dos and don’ts” of factor analysis, explicitly encourage researchers to examine multiple factor models, rotations, and extraction approaches due to the “value in approaching statistical modeling using varied methods to converge on shared, and thus robust, conclusions and to minimize the likelihood of reporting spurious effects (p. 114). In this case, such explorations can be explicitly built into a detailed preregistration so as to a) increase the severity of the test (of the factor models) and b) provide some protection for authors from reviewers who may object to this comprehensive, thorough analytic approach.

Second, Klonsky argues that preregistration becoming the norm will “devalue analyses conceived after engaging with the data. From the PRM [Preregistration Mandate] perspective, only the former is badge-worthy, and the latter perhaps worthy of suspicion.” (p. 15). This is neither true in principle nor in practice (e.g., see response 3 in Chambers et al., 2014). In principle, preregistration is designed to increase transparency in order to facilitate the evaluation of severity, thus analyses conceived after engaging with the data can be appropriate (whether author or peer reviewer instigated), and these analyses may enhance, decrease, or be unrelated to the severity of a test. The important thing is that preregistration allows us to properly evaluate the analyses conceived after engagement with the data and the context in which they were undertaken.

In practice, there are published guidelines for how to deviate from preregistrations (e.g., Lakens, 2024; Willroth & Atherton, 2024), which is difficult to square with Klonsky’s belief that these analyses are devalued in the context of preregistration. Lakens (2024) specifically outlines how to report deviations with the principle of severity in mind, and Willroth and Atherton (2024) provide a valuable template for researchers to use when reporting deviations from their preregistration. Moreover, Willroth and Atherton (2024) provide empirical data on journal editors’ perceptions of preregistration deviations. They found that, on average, editors reported that deviations would not affect their perception of the manuscript or reported deviations would have a slight positive impact on their perception of the manuscript, which does not support Klonsky’s claim that deviations are worthy of suspicion.

Third, Klonsky argues that preregistration becoming a norm will produce a different kind of “file drawer” problem. Research labs will continue to conduct numerous studies at a time, only some of which are selectively submitted for publication. However, when preregistration is the norm, the results that are selectively submitted for publication can attain badges vouching for their (unwarranted) credibility. Klonsky does not specify how and when preregistration is being carried out in this hypothetical scenario. But assuming that all studies are being preregistered prior to the studies being conducted, we argue that preregistration can help bring about the exact opposite effect that Klonsky suggests (i.e., preregistration can combat publication bias). Specifically, when null findings can be shown to be the product of a severe test (with this evaluation being facilitated by preregistration), these research labs should want to publish all their findings, not just positive ones. Indeed, in a footnote, Klonsky excludes registered reports from his concern about this new kind of publication bias. But Klonsky does not articulate the reason that registered reports would be immune from his concern. We argue it is largely because of their use of preregistration.³ More specifically, by ensuring that any result, null or otherwise, is the product of a severe test, we can be more confident that the result is informative and thus there is good reason for it to be published. Preregistration can help us differentiate null results that are the product of a severe test from null results that are uninformative (e.g., when an underpowered test produces a null result).

Last, Klonsky states that preregistration becoming the norm will lead to various forms of “preregistering after the results are known”, or PARKing (Yamada, 2018). Klonsky notes that in some cases (e.g., lying in a preregistration about previous access to data), PARKing is simply fraud. We agree, and it should be labeled, described, and sanctioned as such. However, Klonsky also provides a different hypothetical example of PARKing he believes would be more common. In his example, a student conducts a partial exploration of a dataset, discovers an interesting relation between variables, and proceeds to stop exploring and draft a preregistration on advice from their advisor. First, without knowing what information is reported in the preregistration, there is nothing inherently problematic about Klonsky’s example. What matters is transparently reporting how their analyses were carried out to facilitate the evaluation of severity. If the student and advisor claim in their preregistration that they had no prior knowledge of the data, that would be fraudulent behavior that should be considered scientific misconduct. In a more scientifically rigorous (and ethical) scenario, the student and mentor would transparently report in the preregistration that their subsequent analyses were based on some initial exploration of relations in the data. We see no issue with their behavior per se, though their approach is likely to have implications for the severity of their subsequent tests, as it should. If the student and mentor transparently report their approach, preregistration can function as intended and judgments about the strength of their findings can be appropriately calibrated. Without the transparency provided by preregistration, the appropriate calibration cannot be applied.

Taken together, we believe that Klonsky’s concerns about preregistration becoming a norm are products of misunderstanding the purpose of preregistration, and a failure to attend to the growing meta-scientific literature surrounding preregistration. Regarding the latter point, Klonsky relies on Pham and Oh (2021), McDermott (2022), or hypothetical scenarios to criticize preregistration with the unfortunate consequence that he repeats common misunderstandings about preregistration (e.g., preregistration devalues exploratory analyses; preregistration suppresses creativity). There is a rapidly expanding meta-science literature, providing conceptual and empirical overviews of the benefits and limitations of preregistration (e.g., see Lakens et al. (2024), for a recent review). Engagement with this literature can help increase the understanding of preregistration and its nuances, hopefully improving its use in the field.

Preregistration’s Place in Methodological Reforms

In the final section of his commentary, Klonsky outlines that the best way to improve our science is sociocultural in nature—we must build a scientific culture that celebrates robust findings. In his opinion, this sociocultural change will serve as the “straitjacket” against practices that reduce the robustness of our science. But this final section of the commentary highlights an important inconsistency in Klonsky’s views. While Klonsky believes that researchers cannot be trusted to use preregistration honestly or to avoid selectively publishing only significant findings, this cynicism is notably absent from his beliefs about the responsiveness of researchers to sociocultural pressures to produce robust findings. Presumably, if researchers are responsive to the kind of sociocultural pressures encouraged by Klonsky, there is no reason to think that researchers cannot use preregistration effectively. We agree with Klonsky that sociocultural change is essential but highlight that preregistration, when understood appropriately, is an essential tool in the push for the kinds of sociocultural change that are necessary to build a more robust psychological science.

Thus, the most immediate challenge regarding preregistration is to clarify its intended purpose, which is to increase transparency in order to allow evaluation of the severity of a test. As this conceptual understanding of preregistration increases, researchers will be able to more effectively implement preregistration in their own work, understand when it may be less appropriate, and also more effectively evaluate preregistrations of other researchers. Importantly, increasing the field’s understanding of preregistration is also the best remedy to researchers mistakenly viewing the mere presence of a preregistration (e.g., a preregistration badge) as an indicator of strong science. An accurate understanding of preregistration will help researchers avoid thinking that a preregistration (or badge) tells us anything about the quality of the research, in the same way that an accurate understanding of p values helps researchers avoid the common inferential pitfalls associated with “p <.05”.

Relatedly, concerns about researchers who only engage in preregistration to reap the benefits of a preregistration badge (e.g., other researchers’ belief that the results are more robust) are also addressed by an improved understanding of what preregistration is designed to do. For example, researchers who engage in preregistration because they know it will give them an advantage in getting their otherwise low-quality results published will be “successful” only if people continue to misunderstand what preregistration can tell us about a study. This kind of bad-faith actor capitalizes on researchers’ lack of understanding about preregistration and its purpose, and they will be “successful” to the extent that researchers continue to misunderstand the purpose of preregistration. Although we believe that greater conceptual understanding of preregistration’s purpose will help mitigate the belief that mere use of preregistration is an indicator of rigor, this is an empirical question ripe for future meta-scientific research. For now, we must continue to work to convey what preregistration is designed to do and push back against misunderstandings when they arise (e.g., if a study is preregistered, we can trust the findings).

Practically, researchers have developed many tools to aid in the effective implementation of preregistration which include guidelines for preregistering secondary data analyses (van den Akker et al., 2021), how preregistration can be used in clinical assessment research (Tackett et al., 2019), using preregistration for qualitative studies (Haven et al., 2020; but also see Rubin, 2023), and deviating from preregistrations (Willroth & Atherton, 2023). Meta-science research has also begun to provide empirical data on the use of these tools. There is emerging evidence that more structured preregistration templates can increase the rigor and transparency of preregistrations (Bakker et al., 2020; van den Akker et al., 2023), and such tools are valuable aids in bringing the practice of preregistration in line with its goal. As others have noted, developing an effective preregistration is difficult to do (Nosek et al., 2019). Creating and evaluating preregistrations will often require domain-expertise, and, like other skills, it will take time to develop (Hardwicke & Wagenmakers, 2023). As advocates of preregistration ourselves, we note that our own preregistrations have grown more detailed and nuanced with practice and suspect that such growth and improvement is common.

Nonetheless, an immediate practical concern is how preregistrations are assessed during peer review.⁴ While it may not always be necessary to evaluate a preregistration (e.g., there may be a fatal study design flaw identifiable in the manuscript alone), there is evidence from some journals that preregistrations are rarely accessed during the review process (e.g., Syed., 2023). However, it may be unreasonable to ask an already overburdened peer review system to take on the additional responsibility of evaluating both the preregistration and primary manuscript. A valuable direction journals can explore is to create paid roles for experts or teams of experts to evaluate preregistrations and provide summaries of relevant discrepancies to editors and/or reviewers. For example, at Psychological Science and Clinical Psychological Science there are specific editorial teams used to evaluate the various open science practices in submitted and accepted manuscripts, albeit in a limited way given the limited resources available.⁵ These kinds of implementation issues are highly relevant as the use of preregistration expands. Journals and publishers, who benefit from their alignment with the Open Science movement and perceptions of greater credibility, must provide the necessary resources to make sure that preregistration is being vetted during the review process and not falling entirely to overworked and unpaid reviewers. While these structural-level policies will take time to implement and streamline, we also strongly encourage researchers who preregister their studies to include information about deviations from their preregistrations using a comprehensive template like the one provided by Willroth and Atherton (2023). Such templates can substantially increase readers’ abilities to evaluate any deviation’s influence on severity and help bring the practice of preregistration in line with its intended goal. We would also note that there are benefits to preregistrations even when not evaluated during the review process in that in an increasingly “online” academic world, post publication review has become more common. For instance, in a hypothetical case, scholars may review a preregistration and publication after the latter has been published and then point out important deviations that went unreported and evaluated. As such, the field may come to see that the initial test was less severe than previously assumed and update their priors accordingly.

We also encourage the increased use of the registered report format, whereby a study preregistration benefits from peer review prior to data collection or data analysis. By explicitly requiring peer review of preregistrations, registered reports help ensure that 1) the preregistration is actually evaluated 2) the preregistration provides sufficient transparency about the planned approach; 3) the approach is maximally severe so that results, null or otherwise, are informative. The registered report format is the most effective way to ensure that preregistration achieves its intended goal. Although it is possible that the stage 1 review process in a registered report can still miss important methodological shortcomings of a planned study or analysis, oversights are far more likely to be detected and adjusted prior to the study being completed compared to when preregistration does not undergo review. Despite the fact that more and more journals are offering the registered report format, the actual number of published registered reports remains low. For example, Montoya and colleagues (2021) found that at the time of their review 278 journals had adopted the registered report format (137 of which were psychology journals). However, most journals (71.58%) had yet to publish a registered report. The field will significantly benefit from more researchers taking advantage of the registered report format.

Ultimately, preregistration is simply one means among many designed to help improve the robustness of psychological science. Like Klonsky (2024), we believe our science will benefit from using all the open science tools at our disposal (e.g., sharing code and data, registered reports, preregistration). Many researchers may be skeptical of preregistration because it implies that researchers are not capable of developing and enacting a severe testing approach absent the use of preregistration. It is true that researchers can develop severe tests without preregistration being involved, and preregistration does not automatically increase the severity of a test (Lakens, 2019). As Vazire (2019) has written, “Transparency doesn’t guarantee credibility; transparency and scrutiny together guarantee that research gets the credibility it deserves.” In other words, although transparency is not sufficient for credibility, it is necessary for it. Preregistration is the single tool designed to increase the transparency of the research process to allow others to evaluate the severity with which a claim has been tested, and thus provides an additional way for others to see that researchers are conducting rigorous work and evaluate it appropriately. Like Klonsky, we believe that, “we must create a culture that reveres and celebrates robust findings as much as it once revered and celebrated flashy but fragile findings.” (p. 23). Preregistration, when understood and used appropriately, is a tool that aligns with Klonsky’s vision for the future and can help support this cultural change.