The Experimental Design Assistant

The quality and reliability of much animal research is in question. Unreliable or low-quality research represents an unacceptable waste of animals and research resources. In the US alone, the cost of irreproducible research has been estimated at $28 billion annually, and issues with research design and reporting are estimated to account for half of that waste¹. To address these issues, the National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs) developed the ARRIVE guidelines to improve the reporting of in vivo research^2,3. We now present the Experimental Design Assistant (EDA; https://eda.nc3rs.org.uk), a freely accessible web-based tool, which was launched to help researchers improve the design, conduct, analysis and reporting of animal experiments. The system was developed by progressive interaction between an expert group experienced in providing advice on experimental design to researchers and a software development team. It includes a computer-aided design tool through which the user develops a diagram that embodies the experimental plan. The diagram offers a new standard notation for describing experiments in which methodological details and analysis plans are explicit (Fig. 1). This facilitates communication between collaborators, funding bodies, ethical review committees, journal editors and peer reviewers; it also allows detailed record keeping and may serve as an ex ante registered protocol⁴. The structure of EDA diagrams is based on a series of relationships between the different components of the experiment. This allows the use of computer-based logical reasoning to provide feedback and advice on the experimental plans⁵. The feedback helps researchers improve their experimental design by, for instance, highlighting missing information or problems with internal consistency. It also provides assistance with identifying and characterizing the independent variables and outcome measures to be included in the analysis. Advice about common nuisance variables, which threaten the internal validity of many animal experiments, is also provided, along with practical suggestions to account for such variables in randomization and data analysis. The feedback does not restrict researchers to using a particular design type, but it promotes a better understanding of the implications of common design pitfalls so that researchers can make informed decisions. The feedback also suggests methods of statistical analysis that are appropriate for a given design, along with advice on any data requirements (‘assumptions’) for a given test as well as possible data transformations. Other features of the EDA include support for randomization, blinding and power calculations, and procedures which are still underused in animal research⁶. Based on the diagram, the system generates a randomization sequence for the study, taking into account any blocking factors. The sequence can be sent directly to a third party nominated by the investigator, thus allowing the investigator to remain unaware of the animals’ group allocation until the data have been collected and analyzed. Animal experiments are often too small to yield meaningful results⁷; the EDA’s power calculation tool—along with extensive guidance on how to choose the appropriate calculation and identify the parameters required—will help researchers determine optimal sample sizes for each experiment.In conclusion, the EDA is a new resource to help improve the quality of animal research. It can help researchers design robust and reliable experiments in two ways. First, it ensures that the experimental plans are explicit and transparent, thus allowing detailed scrutiny before and after data are collected. Second, it encourages improvements by providing researchers with critical feedback, targeted information and access to randomization and power analysis tools. We will continue to incorporate user input in planned future developments to ensure that the EDA evolves in line with the needs of the research community.

Figure 1. Example of an EDA diagram.

EDA diagram representing a two-group comparison in which each cage contains mice randomized to either of two treatments. Diagrams are composed of nodes and links to represent an entire experimental plan. The gray nodes contain high-level information about the experiment such as the null and alternative hypotheses, the effect of interest, the experimental unit and the animal characteristics. The blue and purple nodes represent the practical steps carried out in the laboratory such as the allocation to groups, the group sizes and role in the experiment, the treatments and the measurements taken. The green and red nodes represent the analysis, the outcome measures and the independent variables of interest and nuisance variables (e.g., blocking factors). For more details, see https://eda.nc3rs.org.uk.