Video can make behavioural science more reproducible

Graphical Abstract

We recommend the widespread use of a simple, inexpensive, easy-to-implement, and uniquely powerful tool to improve the transparency and reproducibility of behavioural research — video recordings.

How robust are published findings in the behavioural sciences? Can the results be reproduced based solely on information in the published paper? We suggest that questions about scientific reproducibility¹ — especially in the behavioural sciences — often arise because text-based descriptions and static images in the methods sections of journal articles cannot specify procedures and dynamic stimuli in sufficient detail for the study to be readily and reliably replicated by researchers outside of the original research team.

We applaud recent efforts to bolster the openness, transparency, and reproducibility of behavioural research, including large-scale replications of published findings²; pre-registration of studies; changes in review and publication practices; new incentives to encourage reproducible practices; and more open sharing of data, materials, and analysis code^3–7. Fortunately, new web-based tools (for example, the Open Science Framework, GitHub, and Dataverse) make adopting open scientific practices increasingly easy.

In addition, we argue that behavioural researchers who seek to improve the reproducibility of their findings should incorporate a simple, inexpensive, easy-to-implement, and uniquely powerful tool: video recordings of experimental procedures and computer-based task displays.

Visual media — first film and now video — have been mainstays in developmental psychology for decades⁸. Video uniquely documents the interactions between people and their physical and social environment. It captures when, where, and how people look, gesture, move, communicate, and interact. As a result, video as data constitutes a valuable resource for discovery of patterns of human and animal behaviour in diverse settings. If a picture is worth a thousand words, then surely a video is worth a thousand pictures⁸.

Regardless of whether video serves as the source data for behavioural research, video as documentation makes procedures more tangible and transparent. Space-limited methods sections in published articles too frequently omit critical details about empirical procedures — who did what, when, and how. The difference between text and video in revealing the subtle but critical details of behavioural procedures is analogous to reading a screenplay versus watching the movie. Given the essential importance of specific procedures to elicit various behaviours, reproducibility and rigour in the behavioural sciences requires the faithful capture and open communication of methodological details.

Video exemplars of procedures reveal how people gave consent to participate in research, the instructions they received, the tasks they performed, the off-task or non-task-related behaviours they produced, and how, when, and where the research took place. Video records who the research staff were, what they wore, said, and did, and when and how they did it. Videos of computer-based tasks or live-action displays used in laboratory, online, or neuroimaging research can reveal details about tasks and displays that static images and text-based descriptions too often (and needlessly) obscure.

It is well-known lore among infancy researchers, for example, that many details about the testing environment or computer display affect whether infants will complete (or ‘succeed’ in) a laboratory task. But what those details are or why they matter remains a mystery. What particular procedural details are of scientific importance in other areas of behavioural research should be an empirical question. Like other empirical questions, this one can be best answered with solid evidence — evidence that video readily provides. Indeed, new systems that integrate video and photographs with version-controlled text-based protocols have begun to improve the reproducibility of complex multi-step biomedical lab procedures (http://klavinslab.org/aquarium-about.html). Nevertheless, despite the demonstrated virtues of video as data and as procedural documentation, no field in the behavioural sciences, including our own field of developmental psychology, has yet made commonplace the open sharing of research videos or videos of procedures and displays.

Rapid progress in creating new tools for open science makes us optimistic about the future. One such tool is the Databrary (https://nyu.databrary.org/) digital library⁹. Launched in 2014 with support from the National Science Foundation and the National Institute on Child Health and Human Development, Databrary specializes in the storage, management, sharing, and reuse of research videos, videos of procedures, and computer-based displays. As of mid-2017, more than 815 researchers representing over 320 institutions across the globe have been granted secure access to more than 7,600 hours of recordings of approximately 7,700 infant, child, and adult participants. Metadata that allow site-wide searches and filtering operations accompany video datasets and sample displays. The system stores preservation copies of videos in their original formats alongside a transcoded version suitable for web-based playback. The project’s leaders represent a diverse array of constituencies in open science (https://databrary.org/community/board.html).

Databrary addresses many practical barriers that limit data sharing and tackles the thorny ethical issues specifically related to video. Formal institutional agreements restrict access to those researchers who demonstrate ethics training and pledge to protect data from misuse. Video data with identifiable faces or voices are shared only with explicit permission from participants. These restrictions enable videos of human participants to be shared more widely and securely than before. Of course, in some cases it may not be practical or possible to secure sharing permission from everyone recorded in a video. In other cases, recordings of every session or display sequence add little value but increase cost. Clearly, researchers must retain control over what to share, with whom, and when. Nevertheless, we suspect that large numbers of our colleagues in developmental, cognitive, and social psychology, cognitive neuroscience, and ethology could share exemplar videos of their procedures and computer-based task displays with few ethical or practical difficulties. Making this common practice would have benefits beyond transparency.

Stepping back, we see no reason to believe that behavioural science is inherently less robust or reproducible than other sciences^1,2. Rather, we have simply cast an early and more public spotlight on the challenges of studying behaviour in all its richness and complexity. This public spotlight creates an opportunity to realize meaningful change.

So, in that spirit we invite our colleagues to join us in making the open sharing of video recordings a standard operating practice across behavioural science, whether using Databrary, other repositories (for example, Open Science Framework, Dataverse, the Inter-university Consortium for Political and Social Research (ICPSR), and Dryad), or a video-based journal such as the Journal of Visualized Experiments (JOVE). Exemplar videos of procedures and displays should be viewed as the gold standard of documentation. Indeed, the use of video for documenting procedures could make moot disagreements about whether replications truly reproduced the original experimental conditions^2,5. More broadly, the widespread sharing of video data, videos of computer-based displays, and exemplar videos of procedures will bolster the openness, transparency, and reproducibility of behavioural science. In turn, video can make our findings more accessible and engaging to a public hungry for meaningful and robust insights into the mysteries of human behaviour.