Open science: My insights into data sharing, preregistration, and replication

Abstract

After a decade of implementing open science practices as a principal investigator, mentor, data repository founder, and Editor-in-Chief, I’ve learned that the question isn’t whether researchers should adopt these practices, but how to adapt them meaningfully. This commentary, based on a talk given at the 2024 CSBBCS conference, argues for two key principles: first, open science implementation must be context-dependent rather than one-size-fits-all, and second, practical research realities require flexible approaches to idealized policies. Through personal examples, from my evolution with preregistration from “recipe” to “guide” during COVID-19 research to challenges with Registered Reports using existing datasets, I show how open science practices work best when researchers approach them as evolving tools rather than rigid rules. I also discuss field-specific differences in open science uptake between psychology and education, and the importance of equity considerations in implementation. The commentary concludes with concrete recommendations for researchers and journals, emphasizing that sustainable open science requires meeting researchers where they are while maintaining transparency and scientific rigor.

I was asked to give a talk on themes related to open science at the Canadian Society for Brain, Behaviour, and Cognitive Science (CSBBCS) annual conference in Edmonton, Alberta, held June 2024. This commentary expands on that talk to share what I’ve learned about data sharing, preregistration, and replication across different professional roles. After a decade of implementing open science practices as a principal investigator, mentor, data repository founder, and Editor-in-Chief of Infant and Child Development, I’ve learned that the question isn’t whether researchers should adopt these practices, but how to adapt them meaningfully.

My research focuses on identifying sources of individual differences in children’s reading and math skill development, working at the intersection of developmental science and education with behavioural genetics methods. This commentary argues for two key principles based on my experience. First, open science implementation must be context-dependent rather than one-size-fits-all. Each field and subfield has different research cultures, resulting in very different open science knowledge and needs. Second, practical research realities require flexible approaches to idealized policies. The gap between open science “rules” and research practice often necessitates transparent adaptation rather than rigid adherence. Third, sustainable open science requires community-building and evidence-based adaptation. We need infrastructure, training, and meta-science research to understand what works, when, and for whom.

To start, I want to lay out upfront that I believe open science—the idea that we should make “scientific processes and practices, including research methodology and outcomes, more open and transparent” (https://science.gc.ca/site/science/en/open-science) should be the default of how we conduct research in Canada. We owe the Canadian public this full openness of our findings and materials and transparency of our process, and in this era of growing public distrust of scientific evidence, we need to engender trust with openness and transparency. Therefore, I believe that we need to use open science practices.

This is more of a personal narrative than a summary of open science or introducing new open science ideas, and it’s written purposely to show that my thoughts on open science practices have evolved. I position my commentary from my experience using and advocating for open science practices in my subfield for about ten years now. My goal with this commentary is to convince every reader that they too can and should use open science practices, and to encourage the reader to consider how they can use open science practices in a way that is beneficial and equitable for them and their subfield. To keep this focused, I assume definitional knowledge of open science practices, but if you would like to catch up on basics, I suggest Kalandadze & Hart (2024).

Context-Dependent Implementation of Open Science

It has become obvious, to me at least, when watching the open science movement that it is very hard (impossible?) to make universal open science policies. Each field of research, and even subfield, has a different research culture that results in their open science knowledge and needs being very different. I have often felt uncomfortable with general, wide sweeping, reform decisions. An example of such a universal statement on an open science practice is the position that preregistration cannot be done if it has been possible to access the data before preregistration (e.g., Lakens et al., 2024). While these authors admit they use preregistration as a narrow term and suggest that secondary data analysts should use other similar approaches to prevent systemic bias, they argue these approaches should not be called preregistration.

For a researcher whose research involves short experimental studies using undergraduate subject pool participants in their lab, such a universal policy about when a preregistration “counts” may seem inconsequential and works just fine for them. However, such a decision would mean that almost all my research is not eligible for preregistration, as I collect or use large datasets and publish many papers from the same data, meaning I have access to the data prior to preregistration. Saying I therefore cannot preregister feels exclusionary, and frankly, unnecessarily limiting for my subfield. I would like instead to discuss when a preregistration can be useful for secondary data analysis.

Field-Specific Realities in Uptake and Attitudes

There are meta-science data on differences in behaviors and attitudes between different fields, illustrating how field-specific research cultures shape open science adoption. Using the example of the two fields I work across, where is psychology and education when it comes to data sharing, preregistration and registered reports, and replication? Some data from a Center for Open Science (COS) survey across many fields suggests that 63% of researchers in psychology and 21% of researchers in education shared data in their most recent publication that used data (these data were shared in a talk by Nosek, 2022, at the Unconference for Education Sciences, and can be found https://osf.io/9k6pd/). Although there is moderate to low uptake of data sharing in my field, those same researchers had positive attitudes about data sharing, with only 4% of researchers in psychology being against data sharing, and 5% in education (Nosek, 2022).

Related to preregistration, 42% of researchers in psychology, and 11% of researchers in education had a preregistered study in their most recent scholarly work, although only 6% in psychology and 24% in education were against preregistrations (Nosek, 2022). Related to replications, meta-science work examined the top 100 education journals for replications and found that .27% of all articles contained replications. Of that .27%, less than half were direct replications, with 45.3% publishing conceptual replications and 18.6% approximate replications (Perry, Morris & Lea, 2022). These data all suggest that there is support but not a lot of uptake in data sharing, preregistration and registered reports, and replications across these fields, but also, there are field specific differences in open science behaviors and beliefs, even comparing two closely related social science fields.

Data Sharing Culture Differences.

As someone who uses extant data in her research, often for secondary data analysis, I have always been supportive of data sharing. I was trained in behavioural genetics, which is a field, as best as I can tell, that has a long history of sharing data. It is normal to be asked for data by others in the field, and to ask back (e.g., as a graduate student, I helped analyze and contribute data to this paper, Haworth et al., 2010). I took my training in behavioural genetics which exposed me to the power of shared data into my first faculty position, which was in an applied research center more closely aligned to education. My first funded grant was made possible because of the kind support of two senior colleagues in education who gave me full access to all their extant data (that project, and all the data, are available at Hart, Al Otaiba et al., 2021; see also van Dijk et al., 2025). However, in education, data sharing is not the norm (Logan, Schatschneider & Hart, 2021). For example, in one survey of education researchers, only 41.8% of respondents said they had shared their data in a repository (Logan et al., 2024). These experiences across fields showed me how data sharing behaviors are embedded in research cultures. This led me to seeing a need for a data repository for researchers working in, or close to, education, to provide the infrastructure and guidance for the field to support more data sharing, so that others could access high quality data like I had. Subsequently, with colleagues I built LDbase, a domain specific data repository that stores behavioral data from researchers who study learning and development (Hart et al., 2020, Hart et al., 2024).

It is hopefully obvious now that I am very supportive of data sharing, for myself and for my field(s). However, I have seen repeatedly that context can matter for data sharing and makes universal data sharing requirements difficulty. For example, as an Editor-in-Chief, I recently saw a manuscript that involved data from children who were in the lowest quartile of poverty in their developing nation. The authors were from the same developing nation, at an under-resourced institution. As both an Editor but also involved in supporting and training a field in data sharing, I found myself stuck on the equity and ethical issues with demanding the same level of open science engagement from these authors than I would from funded authors in a Westernized country. I know it takes time and resources to share data. This example illustrates why context matters—not just scientific domain context, but also resource and equity contexts.

In total, across my different roles, I have seen that context can matter for both behaviors and attitudes when it comes to open science practices. And although I’ve seen how top-down policy change can change behavior (e.g., government funders requiring open science practices), I have seen repeatedly that open science cannot be implemented through one-size-fits-all policies. Field-specific research cultures, resource availability, and ethical considerations all shape how open science practices can and should be adopted. Rather than enforcing universal mandates that may inadvertently exclude certain types of research or researchers, the open science movement would be better served by developing flexible frameworks that recognize these contextual differences while still promoting transparency and rigor. This approach acknowledges that a preregistration in developmental psychology using longitudinal data may look different from one in experimental psychology, that data sharing requirements must consider both technical infrastructure and equity issues, and that meaningful progress toward open science requires meeting researchers where they are rather than demanding they conform to practices designed for different research contexts.

Practical Realities vs. Idealized Policies

Beyond accepting that universalities cannot work across all research contexts, I have also accepted that in my field, open science in practice is never as pretty as when it is conceptualized. Mistakes are common when you first start using open science practices, but also are common even when you have experience with them. I have written my share of guides to open science (e.g., van Dijk et al., Schatschneider & Hart, 2021), but in reality, very rarely does following open science work as easily as following a template.

My Evolution with Preregistration: From Recipe to Guide

My first preregistration was started in around 2017, thanks to the COS Preregistration Challenge (https://www.cos.io/initiatives/prereg-more-information). That preregistration ended up with a paper published in an open access journal, with open materials, code, and data (Hart & Ganley, 2019; preregistration, data and code available https://osf.io/fh752/). I was hooked on the idea of preregistration as soon as I wrote up that paper. At this point, all new papers in my lab are preregistered, and that includes secondary data analysis papers, which we use the preregistration template for (Weston et al., 2018). I love preregistration for my lab. It becomes a recipe we lay out before we access our data, allowing for easy analyses when it comes time to work on a manuscript.

At least that’s what I would have told you until about 2022. Prior to that, as a lab we had common statistical methods we use and generally analyze data with properties we can anticipate. For example, I knew measures of reading and math skills in children would result in data that is normally distributed. I selected questionnaires that I knew were reliable and valid from previous data collections. But around 2022, thanks to a new grant, my lab pivoted to focusing on the short- and long-term impacts of the COVID-19 pandemic on children. Suddenly, we were working in a space where we couldn’t predict what we would face when we opened data. There had not yet been time for the field to establish reliable or valid measures of COVID-19 impacts on children. This meant we didn’t know how the measures we selected, or made ourselves, would act.

This data uncertainly meant we found ourselves getting used to preregistration not being a recipe but instead being a guide that worked until it didn’t. This would result in us having to revise our preregistrations during the analytical pipeline, being responsive to things like unexpected distributions that didn’t meet the assumptions of our preregistered analyses. Given this, we have transitioned to versioning preregistrations as the lab norm and continuing to use radically transparent reporting of our analyses in papers, noting when and how changes occurred from preregistration. This showed me that for many working in research areas with measures less rigorously tested, or using new statistical methods for them, it would be so hard to confidently complete an accurate preregistration without deviations in one draft.

Personal Practical Challenges with Registered Reports

My lab has also dipped our toes into Registered Reports. At this point, we have tried out Registered Reports five times, and successfully have published two (Johnson et al., 2024; van Dijk et al., 2025). The unsuccessful attempts were mostly unsuccessful because we were stopped by reviewers who didn’t like our measures. As we are mostly publishing work that is analyzing one of our large datasets that has already been collected, it is not possible to change our measures. In our unsuccessful cases, we flipped to a preregistration and moved on with the full manuscript.

I get the reviewer frustration about being called upon to speak to methods when the only part of the method that can change is the statistical modeling choices. However, I hope that my lab’s experience with Registered Report review is not shared by others, as I am a firm believer in it for all of science. But also, a main feature of Registered Reports is to safeguard against publication bias. This is particularly worrisome, in my opinion, for experimental work that might show nonsignificant p-values. I work in variance explained, which is less about p-values, and more about effect sizes. Maybe there’s less of a concern of publication bias? (see Plomin et al., 2016, who argue the point that the genetic and environmental origins of individual differences in behavior are consistently replicated) I’m not sure we know the answer to this question yet, but it makes me wonder if all areas of research well served by Registered Reports.

Editorial Challenges with Registered Reports

At Infant and Child Development, we have accepted Registered Reports as a manuscript submission format for many years. However, we do not get many submissions in that format, a small handful in two years, outside of special issue that have focused on them. I would like to increase the number of Registered Reports we get but also recognize that we need a pool of trained reviewers on the mechanism. It is not uncommon to get a review for a Stage 1 Registered Report that says the manuscript is not complete as it only has an introduction and method section. I believe this is because our email explaining this is a Registered Report is long forgotten by the time the reviewer goes to do the review, and understandably they look for what they are used to, a full manuscript.

Something else I’ve noticed with Registered Reports is that they always take longer through the publication cycle, and this makes it hard to fit into how we handle most manuscripts. For example, data collection often takes years in developmental science, and the initial handling editor may no longer be working with the journal when Stage 2 is submitted, never mind the initial reviewers have long since moved on. For other issues regarding Registered Reports, see a recent editorial to a special issue I co-Edited with colleagues on Registered Reports (Cook, Therrien, & Hart, 2024).

Not all Replication Can be the Gold-Standard Direct Replication

Replications are also different in my corner of the research world when compared to common exemplars of replication. It’s just not feasible to go start another large national twin project to directly replicate what we’ve done in the first twin project (and likely not useful for our grant dollars). But we do consider how we can conceptually replicate what other researchers have published, when planning our data collections. For example, nested within a large data collection on a different topic was a replication of Maloney et al. (2015), now available (Poisall et al., 2023). Or we consider how we can conceptually replicate other work using the measures we already have. For example, van Bergen et al. (2018) published a twin direction of causation paper using the Netherlands Twin Register, focused on the causal direction of the relation between reading ability and book reading in children. In my lab, we didn’t have the exact same measures, but we had a different twin project with similar measures, so we published a conceptual replication (Erbeli et al., 2020). Subsequently, a longitudinal study (van Bergen, Vasalampi & Torppa, 2021), and a third different twin study (van Bergen et al., 2023) have been published which all involve similar conceptual replications, and all finding similar effects. In my lab we also like to think about meta-analyses as ways we can test for a sort of replication. We add in study-level moderators, like project/sample, to test the stability of effect sizes depending on different characteristics of the original publication (e.g., Daucourt et al., 2020).

These experiences have fundamentally reshaped my understanding of open science from an idealized set of practices to a more nuanced, adaptive approach. While the theoretical frameworks of preregistration, registered reports, and replication remain valuable, their implementation must be flexible enough to accommodate the messy realities of research. Rather than viewing deviations from these practices as failures, I’ve learned to see them as necessary adaptations that, when transparently reported, still advance the core goals of open science: reproducibility, transparency, and scientific rigor. The key insight is not that open science practices are flawed, but that they work best when researchers approach them as evolving tools rather than rigid rules, maintaining their commitment to transparency while adapting methods to fit the practical realities of their research contexts.

A Vision for Evidence-Based Open Science in Canada

I’m excited to be in Canada in what I perceive as an exciting time for open science in Canada, with good researcher support and growing top-down pressure for open science. While Canada has made important strides with policies like CIHR’s open access requirements and the developing Tri-Agency Research Data Management Policy, I see opportunities for more field-specific implementation guidance and support. Recent grant opportunities for meta-science research on Canadian open science issues represent an exciting development. I love this grant funding idea, because in the end, I think we need to approach our use of open science practices like scientists. As I have laid out, I see open science as not a set of universal steps that can be applied to all research. Instead, I think we should get all researchers open science knowledge and skills, get them practicing open science as they can in their context, and then use meta-science to determine the impacts of those open science approaches on subfields. If we try open science practices in a variety of ways, we create the data needed to do meta-science research on open science practices, and then use that data to determine what and when they are needed, and for whom.

In this commentary I used my experience across many roles in using and supporting open science in my work and across my fields to argue for two principles regarding open science. First, open science is fundamentally context-dependent and second, open science needs flexible approaches. I believe that we need infrastructure, training, and meta-science research to understand what open science practice works, when it works, and for whom it works. The question isn’t whether researchers should adopt open science practices, but how to adapt them meaningfully to different research contexts while maintaining equity and scientific rigor.

From the perspective of these principles, and the belief that we owe Canadian research and the Canadian public reproducible and rigorous research, I offer these concrete recommendations for different members of the Canadian research community:

For researchers, should you share your data?

Yes. I think there’s a way any investigator in Canada can share their data. Note that I do not mean you all have to openly share your data to any internet user, but at the least you should openly share metadata and deposit your data in a data repository using FAIR principles (Findable, Accessible, Interoperable, and Reusable; Wilkinson et al., 2016). Domain-specific repositories are best when available, and controlled access options allow you to balance openness with ethical considerations and early-career protection (see Hart et al., 2024, White et al., 2024).

For researchers, should you preregister your studies?

I think every lab can give it a go. You will make mistakes with them, and it’ll be okay. Just transparently note the deviations in your final manuscript. Expect your approach to evolve—preregistrations may start as “recipes” but become “guides” as your research context changes. Version your preregistrations when needed and embrace transparent reporting of analytical decisions.

For researchers, should you try Registered Reports?

If you are creating new data that you will do hypothesis testing on, I think you should give a Registered Report a go. I am less confident Registered Reports are well-suited for other research contexts, particularly secondary data analysis or variance-explained research where publication bias concerns may be less relevant. Give it a go though, at best you have a rigorous publication, at worst, you have a well planned preregistration.

For researchers, should you try replication?

Yes! Give it a go, at the very least, maybe a conceptual replication. Consider how you can replicate other work using measures you already have, nest replication studies within larger data collections, and think about meta-analyses as ways to test the stability of effects across studies.

For journals, how should you implement open science requirements?

Balance scientific rigor with equity considerations. Demanding the same level of open science engagement from authors at under-resourced institutions in developing nations as from well-funded researchers raises important equity issues that require thoughtful solutions. Consider starting with lower-level requirements (like TOP Guidelines Level 1) and building community capacity before advancing to higher levels. Give all authors the chance to use open science from their context and opportunity. Provide resources, training, and clear guidance rather than just mandates. Train reviewers on new formats like Registered Reports to avoid confusion about incomplete manuscripts.

In my opinion, the open science movement will succeed not through rigid adherence to universal rules, but through thoughtful adaptation that maintains scientific rigor while embracing the diversity of research contexts, career stages, and resource levels in our community. The goal is not perfect compliance, but meaningful transparency that serves both science and society.

Public Significance Statement.

Open science practices like sharing research data and pre-planning studies can make research more trustworthy and useful, but researchers often struggle with how to implement these practices in real-world situations. This commentary shows that rather than having rigid, one-size-fits-all rules for open science, we need flexible approaches that work for different types of research while still maintaining transparency and scientific quality. These insights can help Canadian researchers, funding agencies, and journals develop more practical and equitable policies that actually support better science rather than creating barriers for researchers with different resources or research contexts.