Now Is the Time to Assess the Effects of Open Science Practices with Randomized Control Trials

Abstract

We issue a call for the design and conduct of experimental trials to test the effects of researchers' adoption of Open Science (OS) research practices. OS emerged to address lapses in the transparency, quality, integrity, and reproducibility of research by proposing that investigators institute practices, such as preregistering study hypotheses, procedures and statistical analyses, prior to launching their research. These practices have been greeted with enthusiasm by some parts of the scientific community, but empirical evidence of their effects relies mainly on observational studies; furthermore, questions remain about the time and effort required by these practices and their ultimate benefit to science. To assess the outcomes of OS research practices, we propose they be viewed as behavioral interventions for scientists and tested in randomized controlled trials (RCTs), to identify potential benefits and (unintended) harms. As this is a call to action rather than an action plan per se, we sketch out four potential trial designs to encourage further deliberation and planning. Experimental tests to document the outcomes of OS practices can provide evidence to optimize how scientists, funders, policymakers, and institutions utilize these strategies to advance scientific practice.

This perspective issues a call for the design and conduct of experimental trials to identify the impact of researchers' adoption of open science practices. The Open Science movement (OS; also referred to as the “Evidentiary Value Movement” [Finkel, Eastwick, & Reis, 2015; Hamlin, 2017)] and “Reproducibility Science” [Munafo et al., 2017]) arose in response to lapses in the quality, reproducibility, integrity and transparency of scientific research (Nosek, Spies, & Motyl, 2012). With greater attention paid to rates of reproducibility (Nosek et al., 2015), came a greater recognition that they were due to, in part, scientific practices such as insufficient statistical power (Ioannidis, 2005), flexible rules for stopping data collection, selectively reported outcomes (Simmons, Nelson & Simonson, 2011), and hypothesizing after results are known (Kerr, 1998).

OS represents an umbrella term for a variety of activities, policies and resources that are directed toward both individual researchers (e.g., preregistration; NIH, 2019; Munafo et al., 2017) and organizations ranging from scientific societies and publishers to funding agencies (e.g., open access journals (Nosek et al., 2015), signed reviews (Lynam, Hyatt, Hopewood, Wright & Miller, 2019), and public data repositories (NIH, 2017; PsychArchives; Committee on Toward an Open Science Enterprise Board on Research Data and Information Policy and Global Affairs, 2015 ). This call for action to test the impact of OS researcher practices is directed toward both OS proponents and skeptics but has implications for consumers of research, editors, reviewers, funders and policy makers.

Some OS research practices correspond to or resemble tasks that for decades have been required of scientists before they launch any study. These involve formulation of a priori hypotheses (including distinguishing between confirmatory and exploratory hypotheses) and the specification of primary outcomes, procedures, sample size, and statistical analysis plans. Some professional societies (e.g., the American Psychological Association, 2002) have requested researchers retain and provide access to study data, upon request, for some period of time. These practices have, however, been voluntary, not well monitored, and reliant upon unstructured training of variable quality. Failing to meet these standards, for the most part, carried few consequences.

OS practices extend and formalize traditional research practices, partly by capitalizing on opportunities afforded by advances in technology. For example, researchers are now encouraged to preregister their prospective project on online platforms or submit a registered report, where hypotheses, experimental design, and analytic plans are specified prior to data collection. They are also encouraged to use repositories that afford access to methods and data. (See Table 1 for fuller description of OS research practices) (Chambers, Feredoes, Muthukumaraswamy & Etchells, 2014; Munafo et al., 2017; Nosek et al., 2019; Wilkinson et al., 2016).

Table 1.

Open Science Practices, Adapted from Chambers et al., 2014; Munafo et al., 2017; Nosek et al., 2019

Practice	Description	Aims
Pre-registration	• Describe research rationale, hypotheses, primary outcome, methods, and data analysis plan • Upload modifications as amendments	• Provides transparency • Creates a research plan so any deviations are apparent • Maintains record of protocol modifications • Specifies a priori vs post-hoc statistical tests • Provides full disclosure of research process • Shares information to support replication
Registered Report	• Submit research rationale, hypotheses, primary outcome, methods, and data analysis plan to a journal for peer review prior to conducting the study. • If deemed valid, paper is accepted on principle, then study is conducted, data analyzed, and reported in a complete manuscript- regardless of results.	• Provides transparency • Supports the development of a robust research and analytic plan • Specifies a priori vs post-hoc statistical tests • Provides full disclosure of research process • Shares information to support replication • Allows for publication of all results, including null results.
Use of online repository	• Select data repository that is identifiable, accessible, and interoperable, and that provides data code. • Study methods and tools	• Supports reproducibility • Allows error detection • Promote accessibility to data and enables data use for other purposes
Data sharing	• Describe how statistics and data figures will be presented, shared, and updated • Share final data, metadata, and data analysis/code	• Allows others to access relevant information • Promotes in-depth evaluation of research publication • Promotes accessibility to data and enables data use for other purposes
Final report/paper open access	• Commit to publishing the final report as open access, or provide open access by posting the report on a preprint server before peer-review and publication	• Provides public access to the research report • Offers complete transparency about all aspects of the research (when combined with other practices)

The Reception to Open Science Research Practices

Several scientific fields (e.g., biology, medicine, psychology, biomedical engineering) have adopted OS practices (see https://www.cos.io/our-services/registered-reports; see also Adolph, Gilmore, Freeman, Sanderson, & Millman, 2012). As of 2017, all researchers receiving funding for clinical trials from the National Institutes of Health (NIH) must preregister their trials on ClinicalTrials.gov and commit to submitting the results after data collection (NIH, 2017; see also SPARC Europe & Digital Curation Centre, 2019). Many journals (Center for Open Science, 2020b; Hardwicke & Ioannidis, 2018; Universities UK, 2017) require authors to preregister research and submit a registered report, commit to sharing data before submission review and/or award electronic badges to articles whose authors agree to provide study data and materials access (Ioannidis, 2016; Kidwell et al., 2016; Nosek et al., 2015). Charitable organizations, such as the Gates Foundation and the Wellcome Trust, require grant recipients to use OS practices and to publish in journals that support them (Bill and Melinda Gates Foundation, 2020; Wellcome Trust, 2018). Recognizing that trainees (and, in turn, early career researchers) would benefit from understanding and using OS practices, many graduate programs include them in their curriculum (e.g., Tackett, Brandes, Dworak, & Shields, 2020).

While most articles and commentaries highlight the potential benefits of OS practices (e.g., Frankenhuis & Nettle, 2018; Carey, 2015), the reception has not been uniformly positive. Apprehensions include fear of a “one-size-fits-all” approach applied across research domains (Finkel et al., 2015), extra resources needed for preregistration, expenses associated with open access and the imposition of requirements that may prove particularly burdensome for early career professionals (Bahlai et al., 2019). Additionally, although the hope is that adoption of OS practices will improve theories, methods, and analyses (Nosek et al., 2019), others observe that there is no inherent reason why strategies such as preregistration would do so (Szollosi et al., 2020).

Observational data associated with OS practices allay some skepticism and resistance (Ioannidis, 2018). Researchers who agreed to share (vs. not share) data produced findings with fewer errors (Wicherts, Bakker, & Molenaar, 2011), thereby suggesting more diligence in scientific practice and reporting. Since 2016, there has been a dramatic increase in the use of project registrations and platforms such as OSF (Nosek, 2021). Furthermore, pre-registration on OSF correlates with more reports of null findings (61%) than before OS emerged (5% to 20%) (Cristea & Ioannidis, 2018); perhaps because of less-flexible analysis and selective-outcome reporting (Fanelli, 2013; Kaplan & Irvin, 2015). Although promising, these findings rely on correlational analyses of archival data and surveys and focus on a limited set of outcomes. The implementation of OS practices would benefit from a broader and more rigorous evidence base that can not only guide how these practices are used but also address any lingering questions regarding their effects.

An Experimental Approach to Assess OS Outcomes

In 2012, Pierre Azoulay (2012) challenged current peer-review models for funding research. In a Nature commentary, “Turn the scientific method on ourselves,” he proposed that funding models should be tested with RCTs to determine whether current review practices support the most promising research. Echoing Azoulay, this call to action proposes an RCT approach to test the outcomes of OS practices. OS practices should be treated as behavioral interventions for scientists that, accordingly, merit experimental testing, when possible (Norris & O'Connor, 2019). Although improving science constitutes the primary aim, Munafò et al. (2017, p. 7) observed, “Some [open practice] solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible.” RCTs can potentially provide evidence regarding the impact of OS practices—specifying which may afford many benefits (e.g., reproducibility), benefits only for specific outcomes, (unintended) harms (e.g., slows scientific progress), both benefits and harms, and which may prove inconsequential. Like any other scientific conjecture, we call for experimental testing—the preferred method to identify cause and effect—to evaluate the outcomes of OS research practices.

Feasibility of such trials is reinforced by the scientific study of peer review using RCTs for over 30-years in biomedicine (Rennie & Flanigan, 2018; see "Scientific Precedents and Alternative Approaches"). Rowhani-Farid, Aldcroft, and Barnett (2020) suggest trials of OS researcher practices are also possible. They randomly assigned authors submitting to BMJ Open to receive an email that promised a badge incentive for agreeing to share data versus an email not mentioning any incentive. Whether authors followed through and signed a data sharing agreement constituted the dependent variable. Coding of study design (e.g., case-control, RCT) and acceptance/rejection allowed for adjustment for potential confounding variables and pilot data enabled sample size calculation to attain appropriate statistical power. Although no differences emerged between the two conditions, this study illustrates the viability of an experimental approach.

Of course, all OS proposals may not prove appropriate or work for all scientific fields, methods and empirical questions (e.g., Banks, Field, Oswald, O’Boyle, Landis, … & Rogelberg, 2019). For example, the study of close relationships often involves intense investigations of small samples over long durations (Finkel et al., 2015) and the adoption of some OS practices may prove challenging (but see Campbell, Loving, Lebell, 2014). Similarly, qualitative studies in the grounded theory tradition rely on exploratory analyses and explicitly reject making and testing a priori hypotheses, an element of OS practice (Kettler, 2019). However, OS does not impose an "all or none" rule and virtually all scientific fields appear capable of adopting at least a subset of the practices. For example, Tackett et al. (2019) describe how even for archival studies started prior to OS and preregistration procedures, study protocols, materials and data may still be shared on online platforms along with preliminary findings and plans for future statistical analysis may be preregistered. In any case, we should learn what these practices afford before drawing boundaries.

Potential Benefits and Harms of OS Practices

Contemplating design of experimental tests of OS research practices encourages all of us to consider the benefits and unintended harms they might produce, and to specify how they should be assessed. The outcomes in this section are meant to be generative rather than prescriptive or exhaustive. Some practices may have relatively specific consequences, whereas others may have wide-ranging effects on the scientific research enterprise.

Potential Benefits.

The proponents of OS strive to improve scientific research by encouraging more rigorous practices, recording and monitoring. One should expect greater use of appropriately powered statistical tests, clearer distinctions between a priori versus exploratory tests, fewer errors found after publication, less evidence of fraudulent practices and redaction of publications, and increased number of independent replications. These practices may improve publication impact, especially for those that require a registered report that elicits critical feedback prior to the implementation of the study (Nosek et al., 2019). Because no single index of impact (Waltman, 2016) is completely unbiased, an index based on multiple indices (e.g., citation counts, special recognition awards) might be used (while recognizing that "true" impact may only be discernable years or decades later). Another potential benefit might be an increasing level of shareable data and materials and more frequent repurposing of data by other researchers; thereby producing a larger return on the original research investment (by the original investigators, funders, and taxpayers).

Potential Harms.

These include additional time and delays (Allen & Mehler, 2019; Bahai et al., 2019) to complete research projects (that result in a publication), curate shared data (Tenopir et al., 2011) and prepare grant submissions. The premise that OS practices will slow research down seems quite plausible, but remains untested and, thus, the scale of this challenge is unknown. Time-related delays could slow scientific progress and reduce publication rates, which might be especially problematic for early career scientists. On the other hand, "slower science" may produce better quality science (see Firth, 2019).

A possible harm might be a shift in the topics studied or methods used. This might occur because expectations regarding data sharing may (unintentionally) discourage researchers from collecting sensitive data in areas, such as clinical medicine (National Library of Medicine, 2019). Although obtaining special data access permissions and engaging with distributed data networks (to protect confidentiality) can meet such challenges, the need for more than the usual effort may deter researchers. The demand for larger samples and greater statistical power also may cause scientists to avoid resource-intensive areas that require large budgets, long-term follow-up, and expect attrition (Finkel et al., 2015; LeBel, Campbell, & Loving, 2017). OS could shift, narrow, or diminish research interests and practice, with hard-to-reach populations studied less, fewer researchers gathering sensitive data, and some fields/topics more or less frequently studied.

Many OS practices aim to facilitate transparency but evaluating the quality of transparently reported research requires someone to do the verification work (Tenny, Costa, Allard, & Vazire, 2021). If extensively adopted, the new practices will create an enormous bank of study data, protocols and materials requiring considerable effort and costs to verify and correct. Obtaining sufficient resources may necessitate redistribution from other parts of the scientific enterprise; this shift may prove beneficial, but the cost-benefit ratio cannot be forecast and merits empirical attention.

Design Strategies for OS Practice RCTs

The design of experimental tests of OS practices will benefit from extensive discussion, deliberation and planning with several segments of the scientific community. Here we seed this discussion with some initial ideas about possible ways to approach trial design. At the outset, we recognize that RCTs are not a panacea and advocating for an experimental approach does not guarantee an RCT is well-designed nor yields valid results. Furthermore, all types of practices seem unlikely to be appropriate for testing with the same experimental design. At minimum, scope of a trial and the length of time (e.g., months or years) required to measure certain outcomes (e.g., shifts in the popularity of research areas) will necessitate different design strategies. Some practices may lend themselves to individual testing, but others, for practicable reasons, probably need "bundling" into one intervention (e.g., signing and adhering to a data-sharing agreement). Below we sketch-out four trial examples.

Trial of a Single Practice.

The aforementioned Rowhani-Farid et al. (2020) trial tested the effects of a badge incentive for data sharing on a proximal outcome (i.e., a signed data-sharing agreement and data posting). The observed null finding stands in contrast to observational data indicating an association between badges and data sharing (Kidwell et al., 2016). Yet, it may also prompt a more focused analysis of what is needed for incentives to be effective in promoting behavior change. For example, the magnitude of the incentive must be deemed sufficient (e.g., a badge plus article processing charges) and research communities might vary in their perceptions of incentives. An expanded design that varies the level of incentive and includes journals that serve different scientific areas might provide a way to map when this approach is effective.

Multi-factor OS RCT.

A more complex trial might manipulate several levels of financial reward for performing specific practices, such as preregistration, specifying a priori predictions or producing a detailed statistical analysis plan. Under the assumption that larger incentives (vs. minimal or none) produce greater uptake, the resulting publication report should be more consistent with the preregistered protocol or contain fewer data errors. (Unlike Rowhani-Farid et al.'s [2020] focus on practice adoption, our hypothetical example pursues whether positive consequences result from adoption.)

A Bundled OS RCT.

A design strategy might implement a suite of OS practices at the outset of a research project. Researchers or research teams, who are about to plan and launch a research project, would be recruited (e.g., via scientific societies, listservs) for a two-condition randomized trial consisting of a “OS practices” (intervention) arm and an “attention/placebo” (control) arm (see Figure 1). Researchers in both conditions would receive a similar number of hours of assistance and contact at the outset of the project, albeit for different purposes. Teams assigned to the “OS” arm would receive help with preregistering their study, selecting a data repository, controlling versions of code in repositories, and understanding open access. Specifically, these teams would be instructed to submit questions and relevant materials to a specially created resource center that is knowledgeable and experienced in supporting OS best practices. Staff in this center would provide feedback regarding the satisfaction of the various materials before preregistration submission. The research teams assigned to the “attention/placebo” arm, meanwhile, would receive the same number of hours of assistance but in the form of copyediting, library searching, and other tasks not relevant to OS practices. By providing the same amount of time for assistance, the second arm provides an appropriate control for expectation and attention. The timeline for collecting outcome data would depend on whether an effect is likely to be manifested in the short-term (e.g., data errors) or only realistically assessed in the long-term (e.g., data use by other researchers; replication failures).

Figure 1.

A Bundled Open Science RCT

A Training OS RCT.

A different strategy might shift from a focus on the design and implementation to a specific study for the development and training of researchers. Students at the outset of their graduate education would be recruited for a "scientific methods" bootcamp. Half would be randomly assigned to professional support and training about OS; the other half would receive career support and training about organizational skills, literature searches, etc. A summer program might be used for delivery of the two "interventions." Following bootcamp, trainees would be awarded a small seed grant to fund their next independent research project. Outcomes such as data sharing, pre-registration, open access, and data errors, associated with the research project, might be assessed in the short-term. More distal outcomes, such as replicability and research area shifts, would also be possible in long-term follow-up. This kind of design has the advantage of capturing researchers very early in their careers and following long-term effects. The training intervention design also affords selection or stratification of students by research area and could assess whether having had a mentor who practices OS (vs. one who doesn't) augments the effects.

Additional Design Considerations

The preceding description of hypothetical trials, with simple RCT designs, is provided to spark a conversation about potential experiments and actions to evaluate OS research practices. More complex trial approaches, such as before-after designs, the multiphase optimization strategy (Collins, 2018) and cluster-randomized trials, might be better suited to address differences and nuances among practices, scientific fields and topics. Follow-up will differ from short-term (e.g., data sharing soon after article acceptance), to longer intervals, depending on the outcome of primary interest. Assignment to "practice interventions" might conceivably be done at the individual investigator/study level, research team level, institutional level (e.g., training, department, center), research area or by scientific discipline. For example, changes in errors, data sharing, transparency and some aspects of publication quantity and impact might be feasible across levels. Detecting shifts in redactions, resources and interest in particular topics and subject populations seem more appropriate, however, for assessment at higher levels of analysis (e.g., scientific discipline), which will also have bearing about the type of statistical approach adopted.

Researchers will need to strategically grapple with possible cross-contamination across arms, sites, teams, and disciplines (but complex, health finance reform RCTs have coped with comparable challenges; Finklestein, 2020). Similarly, controlling or statistically adjusting for research team size and scientific discipline (medical, social, behavioral or finer-grained categories) and pilot work or, at minimum, correlational data will be needed to calculate adequate sample size and statistical power. Relatively large samples probably will be needed to stratify or statistically control for researcher-to-researcher variation, experiment-to-experiment variation, and pre-manipulation knowledge and experience with OS practices.

Finally, investigators need to be mindful of the strength of their manipulations at the outset; ensuring confidence that it is sufficient to engage the outcome of interest. Investigators may need to anticipate that the effectiveness of specific practices may be heterogenous across conditions (Rothman & Sheeran, 2020).

Scientific Precedents and Alternative Approaches

Some level of skepticism about the feasibility of conducting ambitious, moderate-to-large–scale metascience RCTs seems entirely appropriate. We follow the lead of Donald Campbell who faced skepticism in the 1960s about his proposal for a new scientific enterprise with comparable methodological challenges. He called for reforms (e.g., social reforms, laws) to be treated as experiments and subjected them to experimental validation (Campbell, 1969). Although his proposal has not been fully realized, his call contributed to the thriving field of evaluation research. More recently large-scale, national RCTs have been successfully designed and implemented to assess the effects of healthcare finance reforms and regulatory changes (Finkelstein, 2020).

As noted earlier, an RCT approach has precedent in meta-science. Prior to the rise of the OS movement, experiments were conducted to test the effects of scientific reviewing procedures, which are distinct from OS researcher practices but relevant to other facets of this initiative (Rennie & Flanigan, 2018). Two trials (Goldbeck-Wood, 1999; Justice, Cho, Winker, Berlin, & Rennie, 1998), compared masking (vs. unmasking) author identity. In another (van Rooyen, Godlee, Evans, Black, & Smith, 1999), manuscript reviewers were asked if they would reveal their identities to the authors versus remain anonymous. Neither masking author nor reviewer identity influenced review quality (rated by coders), recommendations or review time, but reviewers randomized to be identified were more likely to decline (van Rooyen et al, 1999). Walsh et al. (2000), however, found signed reviews were higher in quality, more likely to recommend publication, but required more time.

A "near experiment" tested the potential benefits and liabilities of team science by comparing the research output of grants requiring interdisciplinary collaboration versus individual investigators over a 10-year comparison period (Hall et al., 2012). A lag in productivity among the transdisciplinary center grants was offset by their overall higher publication rates.

Although “true experiments” (Campbell, 1969) of OS should be most informative, some practices and outcomes will be unamenable, inappropriate, or unfeasible for full experimental tests. However, descriptive methods, archival and secondary data analysis, advanced statistical procedures (e.g., interrupted time-series analysis), and quasi-experiments can assess practice adoption effects in those cases.

Conclusion

OS represents a movement that could improve the entire scientific enterprise. The OS researcher practices, already being implemented by some scientists, represent one key component. Though researchers may not fully understand whether OS, in its fullest and most robust form, confers multiple benefits (or unforeseen harms), this important hypothesis merits experimental testing. The enterprise we propose is not a "one-off" project and does not involve testing a single hypothesis because there are multiple OS practices (see Table 1) and many possible short- and long-term outcomes. However, experimental meta-science can provide data about which practices confer greater benefits and fewer downsides. A larger outcome may be a consensus about practices that might be uniformly implemented, but additionally which practices should be implemented under more limited conditions. In turn, all segments of the scientific community—scientists, funders, policymakers, and institutions—can optimize engagement in and valuing of OS research practices. Taking a longer view, experimental testing of OS practices should be embedded within a wider ranging activity whereby scientific disciplines and institutions engage in an ongoing process of self-evaluation and improvement through experimentation.

Public significance statement: We call for experimental trials to test the effects of Open Science research practices, an initiative that could improve the entire scientific enterprise. Well-designed and implemented randomized controlled trials can assist in specifying the effects of different practices and inform decisions regarding their implementation.