Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Apr 19.
Published in final edited form as: Child Dev Perspect. 2020 Feb 24;14(1):9–14. doi: 10.1111/cdep.12360

Advancing scientific integrity, transparency, and openness in child development research: Challenges and possible solutions

Rick O Gilmore 1, Pamela M Cole 1, Suman Verma 2, Marcel AG van Aken 3, Carol M Worthman 4
PMCID: PMC8055047  NIHMSID: NIHMS1615733  PMID: 33880131

Abstract

Based on the recommendations of a Task Force on Scientific Integrity and Openness1 it appointed, the Governing Council of the Society for Research in Child Development (SRCD) adopted a Policy on Scientific Integrity and Openness (SRCD, 2019a) and accompanying Author Guidelines on Scientific Integrity and Openness for Publishing in Child Development (SRCD, 2019b). Here we discuss some of the challenges associated with realizing SRCD’s vision for a science of child development that is open, transparent, robust, impactful, and conducted with the highest standards of integrity. In identifying the challenges—protecting participants and researchers from harm, respecting diversity, and balancing the benefits of change with the costs —we also offer constructive solutions.

Our science aims to deliver robust and actionable knowledge about issues of critical importance to humanity; success demands that child development researchers embody the highest standards of scientific integrity, openness, and transparency. Recent actions by the Society for Research in Child Development (SRCD) Governing Council make clear that SRCD strongly endorses openness and transparency in research, and that SRCD views these practices as essential components of scientific integrity. Nevertheless, widespread agreement on overall goals and values does not guarantee agreement about what specific practices most effectively advance the goals or represent the values. Derived from deliberations of the SRCD Task Force on Scientific Integrity and Openness on which we served, this paper discusses some of the challenges the child development community and its professional societies like SRCD face in seeking a more open, transparent, and robust science. In identifying the challenges—protecting participants and researchers from harm, respecting diversity, and balancing the benefits of change with the costs —we also offer constructive solutions.

Challenges to the Rigor of Child Development Research

The last decade has seen an upsurge in scholarship focused on revealing the ways that many fields of scientific research produce published findings that do not withstand vigorous scrutiny. The social and behavioral sciences have come under especially intense criticism, much of it directed at specific weaknesses: Studies lack statistical power to detect reported effects (Szucs & Ioannidis, 2017), analysis workflows cannot be readily reproduced, statistical errors are common (Nuitjen et al., 2015), independent replication attempts too often fail to support prior claims (Open Science Collaboration, 2015), and failures to produce robust findings may stem from the careless or self-serving use of questionable research practices at multiple points in the research lifecycle (e.g., Munafò et al., 2017). Issues of reproducibility in science have garnered widespread attention in the popular press (Harris, 2017; Lin, 2012; McEntee, 2019; Yong, 2018), and large numbers of scientists report believing that these problems constitute a “crisis” (Baker, 2016). Developmental research shares many of the features identified as weaknesses in other fields (e.g., Davis-Kean & Ellis, 2019). This elevates concerns about the reproducibility, replicability, and robustness of our work and naturally adds urgency to efforts that seek to address them.

Practical Idealism

How should the community of developmental scientists respond to these challenges? In answering the question, the Task Force deemed it essential to articulate a set of guiding principles (SRCD, 2019a) that could guide future action:

  • Child development research is essential for improving the health and well-being of humanity; its practitioners must strive to produce robust and reliable findings and in doing so act with the utmost integrity.

  • Openness and transparency are (or should be) universal values that reflect the scientific ideals of child development researchers.

  • Diversity along multiple dimensions reflects the reality of the human condition. It is a specific strength of scientific research on child development and of SRCD.

  • Scientific societies like SRCD exist to support excellence in developmental research and among developmental researchers.

Our focus then turned to how enact a practical idealism, one that reflects the personal and professional values of child development researchers while acknowledging the real and meaningful barriers to change we face as individuals and as a scientific community. Here, we address several of these challenges, and wherever possible, recommend solutions to mitigate or overcome them.

Protecting Against Harm

Developmental scientists study children and families more frequently than other disciplines in the social, behavioral, and economic sciences. Many studies collect personally identifiable information (names, faces) of children, or study families from vulnerable populations, practices which invoke special ethical responsibilities to protect participants from harm, especially invasions of privacy and violations of confidentiality. This means that special care must be taken to ensure that sharing data, a practice central to open and transparent scientific research, does not increase risks to participants. Fortunately, solutions exist to permit the sharing of data, analysis procedures, and materials in ways that protect participants.

Seeking permission to share.

The most important way to protect participants from harm when sharing data involves securing permission. Participants and parents should be asked for their permission to share data beyond the research team even if the data are non-sensitive and will be anonymized or pseudonymized prior to release. In seeking their permission, participants should be made aware of potential risks and benefits. Asking permission builds upon the principles of informed consent that have governed ethical research with human participants for decades. It may surprise some readers, but even identifiable data like photographs, video, or audio recordings or exact birth dates can be shared with participant (or parent) permission. Of course, asking permission to share is easier when child development researchers adapt template language that others have developed and openly shared (Gilmore et al., 2018) and when the levels of access to be granted use clear and standard conventions. When data are particularly sensitive, researchers should consider seeking permission to archive data for preservation purposes. It is possible that what is challenging to share today might not be so in the not-to-distant future.

Researchers who plan to share data should also seek permission from the research ethics board, IRB, or equivalent entity that supervises their research. Of course, research ethics boards and IRBs vary in their experience with, level of support for, or comfort with data sharing. So, researchers seeking permission to share data should be prepared to consider questions about what data will be shared, via what mechanism, with whom, and for what length of time. Some of these questions have relatively easy answers: Data are best shared via academic or government data repositories that are specialized for curating data in standard ways and preserving them for the long term. Other questions are harder. For example, must minors whose data were shared previously with parental consent be re-contacted and re-consented when they reach majority age? If the discussion concerns data collected from infants or young children, researchers must consider the risk of storing identifiable contact information for extended periods of time in order to permit participants to be contacted and reconsented. As with all questions in research ethics, risks and benefits must be carefully weighed and a balance struck among them.

Choosing what to share, what not to share, and where.

There are long-standing and well-established practices for de-identifying research data: Standard personal identifiers can be removed, individual elements can be substituted with smoothed or imputed values, faces in images or videos can be blurred, voices can be altered, and so forth. These practices often address concerns about data sharing that participants, researchers, and ethics committees raise, and we anticipate new developments in technologies that will help researchers better evaluate the risks of re-identification using various data and more effectively mitigate those risks.

The apparent ease of de-identifying data may make it seem that unrestricted public data sharing is the more open and transparent scientific practice, and thus preferred. Yet we worry that the unrestricted public sharing of data about children and families poses as-yet-unknown future risks of disclosure as the number of such publicly-shared datasets grows. Fortunately, there is a long and successful history of sharing behavioral data with restricted communities of researchers via domain specific data repositories like the Inter-University Consortium for Political and Social Research (ICPSR), TalkBank, and more recently, Databrary. Child development researchers new to data sharing should be reassured that these repositories implement well-established models for safely and securely sharing many of the most sensitive types of data. Restricted sharing via a data repository can strike a balance between a researcher’s desire for scientific openness and the requirement to protect individual participants. Data so shared can be subject to access restrictions, e.g., the requirement for ethics board or IRB review, researcher training, and so forth, while eliminating case-by-case “gate-keeping” by researchers who are intent on sharing only with specific colleagues for particular purposes.

Nevertheless, scientific approaches to the study of child development vary in the level of risk associated with sharing data. Research involving ethnographic methods, small samples, specific locations, certain types of biological data like DNA, or highly sensitive topics (e.g., sexuality, religious or political beliefs) may pose substantial risks of harm to individual participants or vulnerable communities even if data are altered or aggregated or access is restricted. Thus, notwithstanding the virtues of restricted data sharing via repositories, there are some data and materials associated with child development research that we believe cannot be practically or ethically shared, at least now, perhaps ever, certainly not without extreme caution and careful, deliberate consideration. Rather than make the perfect the enemy of the good, we urge our colleagues to focus on realizing the substantial opportunities for expanding the sharing of data that pose lower levels of risk.

Clearly, we have only touched on some of the challenges that expanding the sharing of data about child development poses for our community, and it is equally clear that professional societies like SRCD, working in concert with research ethics experts, have central roles to play in promoting best practices, sharing information, and fostering constructive conversations about the ethics of sharing.

Acknowledging potential risks to researchers.

While upholding ethical obligations to research participants is paramount, we recognize that some researchers view the push for more widespread data and materials sharing as potentially harmful to them individually. Colleagues aware of the highly competitive nature of our work have expressed concerns about being “scooped” by another team, about harm to one’s professional reputation if an error was found in a shared data set or analysis workflow, about the substantial cost in time, labor, and money required to meet data sharing expectations, and about the risk that some work could be perceived as inferior if data or materials were not shared. When we consider the need for research that represents children in all types of communities around the world, the burdens of engaging in open science practices may fall unequally on scholars at under-resourced institutions who lack sufficient institutional support to prepare data and materials for sharing.

These sentiments highlight a conflict between an ideal—child development research should be open, transparent, and widely shared in order to accelerate progress—and the reality—sharing may put an individual researcher at greater risk of professional or reputational harm, especially in an environment where open practices are neither widespread norms in our community nor receive acknowledgment and reward commensurate with their contributions to scholarship.

Of course, there is no single or simple resolution to this conflict. Scientific research carries intrinsic risk; research is competitive; discoveries are more important to scientific progress than the career trajectories of individual discoverers. Some simple and practical solutions can reduce the risk of scooping: Placing a data-sharing embargo on datasets until after a researcher has secured an initial publication from them. Professional societies like SRCD can and should play constructive roles in mitigating other risks—by providing opportunities for professional development training, access to information about best practices, and advocating for more resources from scientific funders, among other activities.

Monitoring a developing landscape.

Attitudes toward open science practices—and the adoption of them—are changing rapidly, particularly in Europe and North America. Leading developmental researchers have become vocal advocates for a variety of open science practices (Adolph, et al., 2012; Davis-Kean & Ellis, 2019; Gelman, 2012) while others express caution (Goldin-Meadow, 2016). Seeking permission from participants to share data is increasingly commonplace and materials-sharing is becoming standard practice. New sharing platforms like Databrary, developed by developmental researchers with these unique issues squarely in mind, have created a policy framework for sharing identifiable data like video and audio recordings (Gilmore et al, 2018). The framework includes template permission-to-share language that researchers can use to record participants’ wishes concerning data sharing. Furthermore, the Databrary framework builds upon established practices of restricted data sharing: Data access is limited to researchers who have formal institutional approval, to data elements that participants have agreed to share, and only for purposes approved by an authorizing institution.

In sum, we see many challenges in the push to accelerate open data and materials sharing while protecting participants and researchers. Solutions exist or are emerging for many of these, and we welcome leadership by SRCD in helping members navigate this rapidly changing landscape.

Respecting Diversity

A second challenge confronting the effort to make child development research more open, transparent, and robust concerns respecting diversity across the many meanings of that term.

One size won’t fit all.

Notwithstanding public challenges to the robustness of psychological research mentioned earlier, there is disagreement about whether psychology and its allied fields are actually ‘in crisis’ (Barrett, 2015; Stroebe & Strack, 2014). The fields represented by SRCD are diverse—developmental science furthers the understanding of many human concerns—and the problems identified by open science advocates may apply more readily to some kinds of work, specific methods, or specific findings than to others. For example, the focus on replicability can seem misplaced or inapplicable to scholarship that studies individuals longitudinally, investigates singular events or conditions (e.g., disasters, policy changes), or works in unique and changing cultures. Mandating the application of specific open science practices to diverse forms of scholarship poses multiple risks. It may devalue past work that did not anticipate the present moment and undermine present work that does not adopt specific recommended practices for legitimate reasons. If a scholarly community lacks widespread consensus about the nature of a problem or its extent, caution seems warranted in mandating specific solutions. Indeed, even scholars known as strong advocates for greater openness and transparency in science have noted that the term “open science” can mean different things to different people (Yarkoni, 2019).

Engaging a global community.

Outside the U.S., the situation looks even more complicated. SRCD has a large, diverse, and international membership, and the society rightly views this as a strength. Nevertheless, researchers from developing countries have been largely absent from discussions about openness, and discussions about open science policies or practices make scant reference to research originating from under-resourced research settings. Moreover, ethical, cultural, and regulatory differences make it an especially complex task to determine what sort of data can be collected from human research participants, by whom, for what purposes, stored where, shared with whom, and for how long. For example, thus far, there is no standard, widely embraced policy framework for permitting researchers to share data about human research participants across international borders. Even among economically developed countries there are significant cultural and regulatory (e.g., the European Union’s General Data Protection Regulation, GDPR) differences about the definition of personal information, the value of personal privacy, and the extent to which entities that collect data about people must protect it.

Making child development research less WEIRD.

If our knowledge base for the science of human behavior is truly WEIRD (Henrich et al., 2010), reflecting the Western, Educated, Industrial, Rich, and Democratic societies where most data are collected, we should strive to make child development research substantially less so. Our colleagues have pointed out that increasing expectations for open data, materials, and analysis workflow sharing could inadvertently adversely and disproportionately affect researchers and institutions from low- and middle-income countries or those who work with populations largely underrepresented in the current literature. To avoid this, we believe that SRCD’s diverse international membership means it has a unique opportunity and special responsibility to provide leadership in collaborative efforts to make child development research substantially more representative of the diversity of the world’s families AND more open, transparent, and rigorous.

Balancing the Benefits of Change with the Costs

We see significant benefits to promoting greater openness and transparency in child development research. At the same time, we recognize that apparently positive changes come with sometimes unforeseen costs that must be acknowledged, reduced wherever possible, and funded where needed.

Bolstering reproducibility requires new skills.

As a practical example, some researchers use data analysis workflows that involve graphical user interfaces (GUIs) consisting of sequences of button presses or mouse clicks or even manual procedures. The extent to which these workflows can be reproduced depends on the analyst and strict adherence to a pre-determined, often written, protocol. The extent to which these workflows can be shared depends on how well post hoc written descriptions match what the analyst actually did. Automated data gathering, cleaning, visualization, and analysis pipelines can be generated in any number of computer packages or languages (SAS, SPSS, R, Python, MATLAB, etc.), and the underlying code easily shared via free open source sites (Open Science Framework, GitHub, GitLab, BitBucket, etc.). But how does the researcher trained to use a paper and pencil procedures, spreadsheets, or the SPSS GUI for data analysis acquire a completely new skill set in order to embrace the move toward greater openness, transparency, and reproducibility? There are many answers, of course; SRCD intends to support in-service training opportunities and information about free or low-cost training offered by others via the society’s web site and at its meetings.

Openness may conflict with intellectual property rights.

Other challenges may be harder to solve. Consider a common approach to the problem of small sample sizes and under-powered studies, namely to use measures that are developed and normed with large samples. Many, perhaps most standardized research instruments, usually questionnaires or computer-based tests, that meet these criteria are subject to copyright or have other restrictions on open sharing of item-level data (the most valuable to others). In this case, open materials sharing could violate intellectual property restrictions. Without open materials sharing, the value of shared data is diminished. This situation poses thorny and unresolved questions related to ownership rights over data and research materials, issues that are related to the debate about open access to scholarly publications that go far beyond the scope of the Task Force or this paper to address. Nevertheless, a more open science of child development faces possibly costly barriers related to unresolved tensions between openness and intellectual property rights that require strong leadership from the community and from SRCD.

There is no free lunch.

Finally, we come to a question colleagues often raise when the question of promoting greater sharing arises. Who’s going to pay? Curating data and materials for storage in a repository takes time and expertise. The development, maintenance, ongoing operation, and enhancement of repositories takes time, expertise, and money. For the most part, repositories in the fields represented by SRCD (COS/OSF, ICSPR, Databrary, TalkBank) offer no-cost data storage options to researchers with few limits on the amount or duration of storage offered to researchers. But the costs of programmers, transcribers, curators, trainers, and administrators required to manage and improve these services are high. Unfortunately, these operational costs of most repositories are funded by time-limited project-specific grants, individual contributions, and institutional subscriptions. Few if any have core support from institutional (e.g., university) hosts such as a university library or stable long-term funding for ongoing operations from a government source like NSF or NIH. Funding agencies invite researchers to add the costs of data curation to their grant proposals, but researchers may be reluctant, absent mandates, to set aside funds for sharing “old” data that could be used for “new” science—data collection and analysis. Moreover, data stored indefinitely over the long term generate ongoing costs to the repository. Researchers in some countries, e.g., The Netherlands, have access to funds set aside specifically for these purposes, but in the U.S., there is no federal funding source dedicated to support the core operations of data repository services in the social, behavioral, and neural sciences.

One of us has suggested that 15% of the U.S. federal research budget could be set aside for open science activities: 5% for data repositories, 5% for curation of existing, high value data sets, and 5% for secondary analysis of existing data. Whether the research community wishes to embrace this sort of proposal is not known, and the scheme would not address the needs of researchers without access to U.S. federal support. SRCD is committed to a science representing knowledge about all children around the world and can contribute to collaborative international efforts to fund data sharing and reuse in the areas of child development research. Clearly, child developmental scientists who aspire to foster greater openness, transparency, and robustness in our work must work collaboratively with our professional societies and funding agencies to ensure that shared data and materials have stable long-term homes that can seed scholarship for decades to come.

Conclusion

In many ways, child development researchers sit in an enviable position: We enjoy the privilege and pleasure of making discoveries about vital, essential questions concerning the development of the world’s children and the health and well-being of their families. That position carries responsibilities, of course, and among them is the requirement to apply critical scrutiny, the “organized skepticism” that sociologist Robert Merton (Merton, 1942) suggested is central to the practice of scientific research. That scrutiny by us and our Task Force colleagues has led to a new policy statement by SRCD and new guidelines to authors that we are proud to have helped shape.

Nevertheless, the work to create a field of child development research that is a model of openness, transparency, and integrity is not finished. Some of the unfinished business involves overcoming the challenges we have discussed in this paper: protecting participants and researchers from harm, respecting diversity, balancing benefits and costs. No doubt readers will be able to identify challenges we have overlooked. We already know that some critics have asked if challenges to the rigor of child developmental research constitute a crisis, shouldn’t the policies and guidelines to authors go farther? In response we say that what seem like small steps to some are substantial ones for other members of our community. We expect to take further steps—together, thoughtfully, and in time. Central to all we that have said is the belief that openness and transparency in research are integral to the principles of research ethics and inclusiveness in knowledge production (Brakewood & Poldrack, 2013; Frank, 2019) that are the hallmarks of our research community.

Acknowledgments

This work was supported in part by the Eunice Kennedy Shriver National Institute of Child Health and Human Development (U01-HD-076595).

Footnotes

1.

Pamela M. Cole, Lisa Gennetian, Rick O. Gilmore, Judith Smetana, Catherine Tamis-LeMonda, Marcel van Aken, Suman Verma, Carol M. Worthman, and Lisa Braverman.

Contributor Information

Suman Verma, Department of Human Development & Family Relations, Panjab University..

Marcel A.G van Aken, Department of Psychology, Utrecht University..

Carol M. Worthman, Department of Anthropology, Emory University

References

  1. Adolph KE, Gilmore RO, Freeman C, Sanderson P, & Millman D (2012). Toward open behavioral science. Psychological Inquiry, 23(3), 244–247. Retrieved October 16, 2012, from http://www.tandfonline.com/doi/abs/10.1080/1047840X.2012.705133. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Baker M (2016). 1,500 scientists lift the lid on reproducibility. Nature News, 533(7604), 452. Retrieved January 19, 2017, from http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970. [DOI] [PubMed] [Google Scholar]
  3. Barrett LF (2015, September 1). Psychology is not in crisis. The New York Times. Retrieved September 19, 2015, from http://www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html. [Google Scholar]
  4. Brakewood B, & Poldrack RA (2013). The ethics of secondary data analysis: considering the application of Belmont principles to the sharing of neuroimaging data. NeuroImage, 82, 671–676. Retrieved from 10.1016/j.neuroimage.2013.02.040. [DOI] [PubMed] [Google Scholar]
  5. Davis-Kean P, & Ellis A (2019, July). An overview of issues in infant and developmental research for the creation of robust and replicable science. Retrieved from psyarxiv.com/m4576. [DOI] [PubMed] [Google Scholar]
  6. Frank MC (2019, July). The ethical duty to practice open science. Unpublished manuscript. [Google Scholar]
  7. Gelman S (2012). Rigor without rigor mortis: The APS board discusses research integrity - Association for Psychological Science. APS Observer, 25(2). Retrieved September 19, 2016, from http://www.psychologicalscience.org/index.php/publications/observer/2012/february-12/scientific-rigor.html#gelman. [Google Scholar]
  8. Gennetian L, Tamis-LeMonda CT, & Frank MC (2019). Advancing scientific integrity, transparency, and openness in child development research: Opportunities. [DOI] [PMC free article] [PubMed]
  9. Gilmore RO, Kennedy JL, & Adolph KE (2018). Practical solutions for sharing data and materials from psychological research. Advances in Methods and Practices in Psychological Science, 1(1), 121–130. SAGE Publications Inc. Retrieved from 10.1177/2515245917746500 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Gilmore RO, Gennetian L, Kalish C, Tamis-Le-Monda C, Worthman CM, & Smetana J (2019, March). Conversation hour on scientific integrity, transparency, & openness. Meeting of the Society for Research in Child Development, Baltimore, MD. Retrieved from https://gilmore-lab.github.io/2019-03-22-SRCD-conversation/ [Google Scholar]
  11. Goldin-Meadow S (2016). Why preregistration makes me nervous. APS Observer, 29(7). Retrieved from https://www.psychologicalscience.org/observer/why-preregistration-makes-me-nervous [Google Scholar]
  12. Harris R (2017). Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions (1st ed.). Basic Books. [Google Scholar]
  13. Henrich J, Heine SJ, & Norenzayan A (2010). The weirdest people in the world? The Behavioral and Brain Sciences, 33(2–3), 61–83; discussion 83–135. Retrieved from 10.1017/S0140525X0999152X [DOI] [PubMed] [Google Scholar]
  14. Lin T (2012, January 16). “Open science” challenges journal tradition with web collaboration. The New York Times. Retrieved August 26, 2019, from https://www.nytimes.com/2012/01/17/science/open-science-challenges-journal-tradition-with-web-collaboration.html [Google Scholar]
  15. McEntee C (2019, July 23). Convergence: A call for more robust global science collaboration - Eos. Eos Retrieved August 26, 2019, from https://eos.org/agu-news/convergence-a-call-for-more-robust-global-science-collaboration [Google Scholar]
  16. Merton Robert K. (1973) [1942], The normative structure of science, in Merton, R.K. (ed.), The Sociology of Science: Theoretical and Empirical Investigations, Chicago: University of Chicago Press [Google Scholar]
  17. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Sert N. P. du, Simonsohn U, et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. Retrieved January 10, 2017, from http://www.nature.com/articles/s41562-016-0021 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. National Institute of Mental Health (2019, June), NOT-MH-19–033: Notice of Data Sharing Policy for the National Institute of Mental Health. Retrieved August 26, 2019, from https://grants.nih.gov/grants/guide/notice-files/NOT-MH-19-033.html [Google Scholar]
  19. National Science Foundation (2019, May 20). Dear Colleague Letter: Effective Practices for Data (nsf19069) | NSF - National Science Foundation. Retrieved August 26, 2019, from https://www.nsf.gov/pubs/2019/nsf19069/nsf19069.jsp [Google Scholar]
  20. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, Buck S, et al. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Nuijten MB, Hartgerink CHJ, Assen M. A. L. M. van, Epskamp S, & Wicherts JM (2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48(4), 1205–1226. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. [DOI] [PubMed] [Google Scholar]
  23. SRCD (2019a), https://www.srcd.org/about-us/policy-scientific-integrity-transparency-and-openness
  24. SRCD (2019b), https://www.srcd.org/author-guidelines-Scientific-Integrity-Openness-ChildDevelopment
  25. Stroebe W, & Strack F (2014). The Alleged Crisis and the Illusion of Exact Replication. Perspectives on Psychological Science, 9(1), 59–71. [DOI] [PubMed] [Google Scholar]
  26. Szucs D, & Ioannidis JPA (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS biology, 15(3), e2000797. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Yarkoni T (2019, July 13). I hate open science. https://www.talyarkoni.org/blog/2019/07/13/i-hate-open-science/ [Google Scholar]
  28. Yong E (2018, November 19). Psychology’s replication crisis is running out of excuses. The Atlantic. Retrieved August 26, 2019, from https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/

RESOURCES