Standards for Objectivity and Reproducibility in High-Impact Developmental Studies–The COVID-19 Pandemic and Beyond

At the time of this writing, more than a year has passed since COVID-19 pandemic shutdowns began, and research on the effect of these shutdowns on child neurodevelopment is beginning to be published. Review of literature from prior epidemics suggests we are likely to be awash in studies reporting that the COVID-19 pandemic has resulted in severe emotional disorders and a greater risk of developmental delays in childhood.¹ How does the expectation that we will observe these effects shape our research? Initial response to an article that was published on a preprint server in August 2021² suggests we will promulgate studies that fulfill the prophecy that the COVID-19 pandemic has harmed this generation of children. The article concluded that “the environmental changes associated [with the] COVID-19 pandemic [are] significantly and negatively affecting infant and child development.”² The study received news coverage despite being published without formal review by experts in the field. Undeniably, preprint servers represent a great asset in the movement toward open science; however, rapid and broad dissemination of results that have yet to be peer-reviewed may change the scientific landscape, and this potentiality requires careful consideration.³

As we strain to surface from the COVID-19 pandemic, it would be advantageous to discern when reading a study whether the observed effects are consequential at an individual level. Parents want to know if their children are going to experience lasting harms as a result of this pandemic, and a P value does not sufficiently address that concern. Instead, the magnitude of observed effects may reveal more about the significance of a finding. This point is illustrated well by a 2018 study⁴ in which researchers tested correlations between screen time and adolescent well-being. They observed a significant negative effect, with screen use explaining 0.4% of the variation in well-being. They went on to compare the magnitude of the screen time effect with other potentially neutral factors. They found that eating potatoes, religiosity, listening to music, and height had similar or greater effects on adolescent well-being than did screen use, and concluded that observed effects of screen time were too small to warrant policy change.⁴ It is also important to appraise absolute risk when evaluating statistical significance of a finding. That is, the probability that an individual child will possess or be impacted by an observed effect is usually fairly small, which is a point that is frequently lost in communication with the public and the media. These points highlight that as COVID-19 research is conducted in children worldwide, evaluating effect sizes, assessing comparators, and appropriately contextualizing findings in terms of clinical significance will be key for lasting societal advancement.

A rich and full understanding of COVID-19 pandemic effects will be possible only if we examine potential for both favorable adaptations and negative consequences. Childhood is a period of tremendous plasticity and adaptability. It is possible that the adversities of childhood can enhance resilience, or can be protective in other ways. It may be that changes experienced by children and families during the COVID-19 pandemic better prepare our society for future environmental demands. For example, it may be that parents and siblings spending more time at home with young infants promotes infant language development, fosters parent perceptions of self-efficacy, or yields other unanticipated benefits. Furthermore, for many developmental measures, there is not a clear good or bad, yet we assign directionality such that traits in the risk group are interpreted as maladaptive. Overall, it is crucial that in our search we remain attentive to potential benefits to children or families that may come to light following the COVID-19 pandemic.

Another way to fortify balanced and reproducible pediatric COVID-19 studies is to prioritize replication and harmonization of common data elements across research sites. To the former, researchers should be encouraged to use independent data sets for discovery and validation, and should use reproducibility analyses to test the impact of analytic choices on outcomes. For example, Floris and colleagues⁵ examined sex differences in neuroimaging data in children who have autism spectrum disorder. They tested both the impact of their processing pipeline and the reproducibility of their effects in an independent sample. In doing so, they reduced researcher degrees of freedom, the fluidity of study design, and analytic flexibility available to investigators. They were able to discern the generalizability of each observed effect and were able to present the robustness of their conclusions to processing decisions, which is important given prior demonstrations that analysis of the same data set by multiple research groups can yield different results.^6,7 Regarding harmonization, a notable number of new collaborations involving perinatal and pediatric samples has arisen over the course of the pandemic. An example of this is the international COVID Generation Research Alliance, launched in April 2020. The COVID Generation Research Alliance is an international network of investigators who have developed testing protocols and materials translated into more than 15 languages to (1) understand pediatric pandemic effects across cultural and geographical contexts, (2) provide a platform for multisite collaboration and data sharing, (3) improve the rigor of the research by providing opportunity for replication and reliability testing, and (4) reach a scale commensurate with what is needed to inform policy decisions and practice.

An expanding area of scientific reporting that may further improve reproducibility is utilization of registered reports. Preparation of a registered report involves declaration of the research question and scientific approach with a registration service, such as the Open Science Framework, or a journal before conducting the study. Use of registered reports discourages using the same data to generate and test multiple hypotheses. Registered reports have the added benefit of supporting communication of both significant and nonsignificant (null) results, as hypotheses are evaluated at the time of reporting. On the whole, registered reports can promote objectivity and replicability, and raise the likelihood that null findings will be reported more frequently.

All said, research in children and families should be held to standards befitting of the present circumstances. Namely, research studies should be planned and conducted in an equitable and unbiased way, and should ideally involve the communities being studied in the design. Registered reports should be integrated as an early stage of scientific planning to increase objectivity and reproducibility. Research results should be reported with effect size and/or be presented with comparators for contextualizing results. Scientists should be aware of how existing stereotypes and expectations may bias how research results are assessed and disseminated, and peer reviewers should be trained to evaluate and call out biased interpretations of data. Further, direct assessment of reproducibility should be a criterion for high-impact research. Additionally, as is fitting with refinement of testing protocols (eg, development of remote testing procedures in infants),⁸ new studies should include discussion of COVID-19–pandemic-related procedural adaptations and potential bearing of these on reported results.

Overall, change ushered in by the COVID-19 pandemic creates opportunity to address preexisting issues in science and improve standards to which our community is accountable. Shifting investigatory priorities and traditional approaches can be formidable, but over the past 18 months we have endured sustained uncertainty, and this may make us more pliable than in the past. If we choose to hold ourselves to a new and higher standard, we stand to make more significant and enduring discoveries about the effects of the COVID-19 pandemic, with benefits for ourselves and for future generations.