Rethinking the incentive system in science: animal study registries

Abstract

The Animal Study Registry offers scientists a range of benefits by preregistering their studies. Wider adoption could address the reproducibility problem in biomedical research and enhance animal welfare.

The translation of preclinical biomedical findings into clinical applications has been disappointingly slow despite the drastically increasing amount of knowledge. The number of biomedical research articles has doubled within the past 20 years, and PubMed lists around 1.1 million new peer‐reviewed articles every year. In contrast, the US Food and Drug Administration 5‐year annual statistic reports an average of 43 new drug approvals per year. Although the number has been increasing recently, the percentage of novel, breakthrough designations that promise substantial improvement over existing treatments in serious or life‐threatening diseases is decreasing 1. Meta‐research suggests that this lack of progress in drug development is due to the “reproducibility crisis” that has gained wide attention in the scientific community and the media. The fact that many published findings cannot be reproduced by others was particularly observed in clinical research, but the problem also persists in preclinical and fundamental research.

The causes of irreproducibility

This may sound discouraging, but a crisis should be seen as a challenge to overcome gridlocked situations and to break new ground. It is, however, important to understand the roots of the problem first. One major cause of the reproducibility crisis is the reward system in academic research. For decades, the major currency in science has been the number of publications, impact factor, and citations; to advance their career or to successfully raise research funding, scientists are under pressure to publish, especially in high‐impact journals. On the other side, science is always interested in novel and groundbreaking findings. The “publish or perish” mentality along with the desire for new insights has fostered poor research practices and a tendency to publish positive results, a phenomenon called publication bias. Above all, science is obsessed with positive results; yet, “negative” (null‐hypothesis) findings also contribute to knowledge.

The “publish or perish” mentality along with the desire for new insights has fostered poor research practices and a tendency to publish positive results, a phenomenon called publication bias.

On the operational level, meta‐research has identified a number of other reasons that contribute to irreproducible findings. These include flawed experimental design—the lack of blinding and randomization or insufficiently large control groups; poor statistical practices, such as missing sample size calculation, p‐hacking, or no differentiation between planned and unplanned statistical analyses—and insufficient reporting of methods or raw data that make it challenging or impossible for other researchers to reproduce experiments and results.

Several solutions have been proposed to address these problems. Many journals now require or encourage more detailed method descriptions and supplementing material to include primary and raw data so as to enable other scientists to reproduce the results. Another step is open data practices, including the use of open repositories or preprints. Multi‐laboratory studies to increase the external validity or address systematic variation in experimental conditions can also improve the accuracy and robustness of research data. The outcome and informative value of an experiment can be augmented by a rigorous study design that includes blinding and randomization and by thoroughly planning the statistical analysis before the experiment. Preregistration of preclinical studies including the planned statistical analysis before the data are collected has been proposed as another means to increase research quality and transparency 2. This has been mandatory for several years for most clinical trials; the adaptation to non‐clinical research offers new opportunities for transparency and would help to improve animal welfare.

Improving the reproducibility of research and animal welfare

Preregistration of a detailed study protocol allows other researchers and reviewers to follow the progression of the work and distinguish planned statistical analyses from unplanned ones 3. The latter are common practice in discovery‐based science and can spawn new exciting hypotheses, but they are often reported as intended, which can lead to an over‐ or misinterpretation of the results. A preregistration platform asks scientists to provide a detailed statistical analysis plan including the definition of endpoints, inclusion and exclusion criteria, and sample size calculation. They also have to state a hypothesis in case of confirmatory research or alternatively mark the study as exploratory research. Providing this information before the study actually begins excludes the possibility to create hypotheses after results are known (HARKing) and can prevent questionable research practices such as p‐hacking. Moreover, preregistration of a study protocol may encourage the publication of all results—positive and “negative” findings as well as inconclusive results—in a journal, a data repository, or a preprint server, which would help to counteract selective reporting and reduce publication bias.

Preregistration of a detailed study protocol allows other researchers and reviewers to follow the progression of the work and distinguish planned statistical analyses from unplanned ones

Publication of “negative” findings could also help to improve animal welfare contributing to a better compliance with the 3R: reduce, refine, and replace. If the data and results are not made publicly available, the animals used in the experiments are wasted; in addition, it can lead to redundant and unnecessary follow‐up experiments, such as repetition of inconclusive animal experiments that were already performed in other laboratories. Building research projects on incomplete data sources can lead to a distorted view of the current state of research and to the pursuit of ideas that are not worth being investigated. In contrast, a thoroughly planned and well‐conducted experiment does not only increase the quality of the results, but also enhance animal welfare by overall reducing the number of animals being used in the long term.

Animal study registry

Driven by these concerns, the German Centre for the Protection of Laboratory Animals (Bf3R) at the Federal Institute for Risk Assessment (BfR) developed a registry specifically for animal experiments (http://www.animalstudyregistry.org) that is free for researchers from all over the world. The key reason to introduce this platform was to improve the transparency and reproducibility of biomedical research as well as increasing animal welfare by avoiding “unnecessary” and redundant experiments 4. As a federal government institution, the BfR can guarantee continuous funding, which is indispensable for the long‐term success of the preregistration; the most important preregistration platform for clinical research (http://clinicaltrial.gov) is also provided by a governmental organization, the US National Institutes for Health (NIH).

The Animal Study Registry specifically addresses the needs of research involving either in vivo or ex vivo experiments. Specific questions related to the animal model and housing conditions are asked as they are not only important for animal welfare but also relevant to reproduce an experiment. The application of refinement measures such as analgesics, handling, or environmental enrichment is also part of the questionnaire as these measures are often applied but rarely reported in the final publication.

Integrating preregistration in the scientific process

The success of animal registries will crucially depend on the integration of preregistration practices in the scientific process and their acceptance among scientists. One important step towards this will be addressing scientists at an early stage of their career and introducing preregistration and other open‐science practices. To encourage preregistration, researchers have to first appreciate the benefits that are specifically provided by the Animal Study Registry.

The Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines address all relevant issues linked to animal experimentation and are broadly endorsed by scientific journals. However, in addition to asking specific questions on, for instance, housing conditions, species differences, genetic background, or breed, preregistration has an advantage over reporting guidelines: It tackles the problem at an early stage before the experiment is conducted. Important factors influencing the results like blinding, randomization, and exclusion criteria are raised prior to data collection, so mistakes in the experimental design can be corrected in time.

Another benefit is that preregistration allows researchers to claim an idea as their own early on. The Animal Study Registry provides a digital objective identifier (DOI) with the registration of a study. A registered study protocol can be saved as a PDF, which documents the authorship together with the DOI. Preregistration of a study protocol also allows researchers to demonstrate their commitment to transparency and research quality, which could form a new rewarding system in science.

Publication of “negative” findings could also help to improve animal welfare contributing to a better compliance with the 3R: reduce, refine, replace.

To accommodate concerns about theft of ideas, the Animal Study Registry provides an embargo period for up to 5 years, during which the study only appears with a title, the institution where it is predominantly conducted, a short summary, and optionally, the name of the author. For reasons of transparency and animal welfare, an immediate visibility of the whole registration is desirable, but is often not compatible with the competitive nature of present research. The embargo period gives scientists time to collect their data or to publish their results.

Although it takes more time to fill out the form in addition to other required documentation, preregistration can actually save time in the long run. The Animal Study Registry seeks to find the best balance between collecting the most valuable informative to improve reproducibility and minimizing the effort to fill out the online form. In addition, as the Animal Study Registry's questionnaire addresses about 60% of the ARRIVE items, it saves time when writing the final manuscript. Generally, preregistration is an opportunity for researchers to comprehensively communicate their work to a broader audience. The comment function allows the author to add any information to the original fixed study plan: necessary changes, the final publication, or repositories where the original data are deposited. Any comment is marked with a timestamp and will be published together with the registered study protocol.

To encourage preregistration, researchers have to first appreciate the benefits that are specifically provided by the Animal Study Registry.

Once the use of registries has gathered momentum, preregistration can support a global scientific network to initiate collaborations and sharing experience. Individual scientists will also benefit from more valid publications, more traceable research, and fewer unnecessary experiments in the long term.

How to create new incentives to boost preregistration

Even despite these benefits, a change in the rewarding system of science is inevitable. The commitment to better research practices and transparency as well as to generate solid and reproducible findings has to become a value in its own. To effectively increase the quality of research, to accelerate the production of knowledge, and to avoid unnecessary animal experiments, a broad application of study preregistration is essential. While the preregistration of biomedical studies in which animals are used is currently based on a voluntary commitment, robust incentives are needed to sustain and further expand preregistration in the future.

Currently, more than 200 journals offer the possibility of submitting so‐called registered reports, a publication format that includes a peer review of the study plan before starting the experiments. If the review of the protocol is positive, the publication is accepted irrespectively of whether the findings are positive or “negative” or inconclusive. This a priori acceptance takes away the incentive to embellish data and fit them to support a hypothesis. Registered reports have become popular in psychological science 5, but are still rare in biomedical research. Especially for purely exploratory research or sequential experiments, an online registry such as the Animal Study Registry might be more appropriate as it offers more flexibility to, for instance, register small parts of larger research projects. Also, follow‐up studies can be easily registered later without having to refill the form by simply copying the prior study and updating the relevant parts.

Greater demand for preregistration

While a simple plausibility check is performed before a study can be uploaded to the Animal Study Registry, there is no peer review. This was a conscious decision. First, providing a review process at that early stage is not practicable owing to the large variety of submitted topics. Second, scientists might not necessarily want to share their ideas with other researchers in the field. Third, it allows the registration of studies without delay. Nevertheless, we encourage reviewers who are contacted by journals for peer review to assess and valorize the preregistration of submitted manuscripts. The author of a study in Animal Study Registry can download a PDF file of the registration and can submit the registration together with the manuscript. This could be helpful for journal editors and reviewers to adequately evaluate a manuscript. Editors could also take into account whether a study has been preregistered, as it might indicate a higher credibility of the reported results. For traceability, a published manuscript ideally includes a link to the preregistered study protocol to enable readers to compare the initial study plan with the published data. Also, providing preregistration badges as done by Psychological Science, Clinical Psychological Science, and Advances in Methods and Practices in Psychological Science is another possibility to highlight a publication.

Bearing the developments in clinical research in mind, it is conceivable that preregistration of animal studies will become mandatory in the future. Preregistration of clinical trials is already a legal provision in the USA since 2000 with the primary intention to inform the public. The International Committee of Medical Journal Editors has adopted the idea with the aim to reduce selective reporting of research outcomes and to prevent unnecessary duplication of research. A meta‐analysis has shown that this measure can indeed increase the number of reported “negative” results 6. European regulation went one step further requiring since 2014 that sponsors of clinical trials conducted in Europe have to directly report their results to the European Clinical Trials Register within 12 months of completion. Unfortunately, the compliance is very low, especially within university institutions 7.

Bearing the developments in clinical research in mind, it is conceivable that preregistration of animal studies will become mandatory in the future.

The demand for greater transparency extends to other domains such as risk assessment in the food chain. From 2021 onwards, all studies commissioned by the industry, including animal experiments for regulatory testing, have to be preregistered before authorization. However, the legal obligation to preregister animal studies may carry the risk that scientists see it as yet another bureaucratic hurdle without appreciating the benefits. Thus, to further incentivize preregistration, research funders could adopt it as a perquisite for certain grants. The German Federal Ministry of Education and Research (BMBF) already requires preregistration for a grant opening for confirmatory preclinical studies and systematic reviews, as does the research funding program of The German Centre for the Protection of Laboratory Animals. Furthermore, career decisions at research institutions could move away from evaluating candidates only on the basis of their publications and embrace more sustainable criteria such as open‐science and preregistration practices.

Future developments for preregistration

As a rise in registries can be expected in the future, harmonized minimum standards will become necessary. In addition to the Animal Study Registry, there are other preregistration platforms for non‐clinical trials, such as the Open Science Framework registry, preclinicaltrials.eu, and aspredicted.org. The International Clinical Trials Registry Platform (ICTRP) by the World Health Organization (WHO) has developed common standards, which have to be adopted by clinical registries to become part of the WHO Registry Network. Developing similar standards for preclinical registries would be desirable, but may prove more complex as preclinical and basic research consists of a wide variety of research areas with different characteristics. For example, specific questions on animal housing are most likely not applicable to pure in vitro or preclinical human studies. Thus, a narrower specialization of the particular registries could be useful. Nevertheless, the development of common standards covering some basic questions or data protection is crucial to support the enforcement of preregistration in non‐clinical research.

Transparency and research quality should become a value on their own, and preregistration is one way for researchers to prove their commitment to both.

A reliable and easy retrieval of information from preregistered studies will also be crucial to establish preregistration as an integral part in the scientific process. To develop new research ideas, scientists usually search information on big publication databases. It cannot be expected that they browse every registration platform for relevant studies, especially when preregistration is still at its infancy with a small number of entries. As a first step, preregistration platforms could cooperate to assemble all information into one database. A long‐term solution would be inclusion of preregistration in the big publication databases. Furthermore, to effectively fight publication bias, access to all reported results from a journal, a preprint server, or as raw data saved in a data repository is essential. Only if the original registered study protocols can be linked to the final outcome, the instrument of preregistration will become more effective.

As with every new instrument, the success of preregistration platforms such as the Animal Study Registry needs to be evaluated. Based on the experience from clinical trial registries, thorough monitoring of registered studies compared with other published results is important 8. The most effective check could take place during peer review of a submitted manuscript when reviewers can compare the results with the preregistered study plan. One important outcome would be more reporting of “negative” and inconclusive results. Furthermore, the systematic screening of registrations and comparison with published results could also help to evaluate the impact of preregistration on selective reporting and publication bias. Initiatives such as the COMPare project by the Oxford University (http://compare-trials.org) or a similar project at the University of Leuven have already started to analyze discrepancies between preregistered protocols and published articles [preprint: 9].

Transparency and research quality should become a value on their own, and preregistration is one way for researchers to prove their commitment to both. Journals and funders have to reward this engagement to successfully initiate a change in the system. Until the scientific community's general attitude does not significantly change, and appropriate incentives are found to trigger this process, we will still face the present problems of low quality and lack of reproducibility.

Conflict of interest

All authors are employed at the German Federal Institute for Risk Assessment and are part of the German Centre for the Protection of Laboratory Animals (Bf3R) which developed and hosts the Animal Study Registry.

EMBO Reports (2020) 21: e49709