Statistical messages from ARRIVE 2.0 guidelines

Concerns over the inability to reproduce the results of basic and preclinical studies have emerged in recent years [1]. As basic and preclinical studies provide novel, exciting ideas as well as a foundation on which future studies are performed, the reproducibility of the results of basic and preclinical studies is particularly important. However, the proportion of studies for which important findings cannot be replicated has been reported to range from 75% to 90% [1].

Poor reproducibility in basic and preclinical studies may prevent readers or other researchers from evaluating the effectiveness and safety of new treatments or interventions, eventually disrupting the entire research field. As animal studies are not an end in themselves but rather a step in a process, patients may be subjected to treatments or interventions that are unlikely to be effective or likely to be harmful. To improve the reproducibility in basic and preclinical studies, accurate, appropriate, and transparent reporting is crucial, which would allow readers and other researchers to determine the scientific rigor of findings, and thus assess their validity and reliability.

As pain physicians, anesthesiologists, and researchers, we have performed animal studies to evaluate the effectiveness and safety of new treatments or interventions, elucidate their underlying scientific basis, and further apply the findings to clinical practice.

The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were recently updated as the ARRIVE 2.0 guidelines in order to align with the current best practices [2] and were published together with an ‘Explanation and Elaboration’ document to facilitate their use in practice [3] . The ARRIVE 2.0 guidelines consist of 2 sets (the ‘Essential 10’ and the ‘Recommended set’) including 21 items. The ‘Essential 10’ focuses on the design of animal research, which contributes to bias reduction, and the ‘Recommended set’ consists of items for the authors to disclose in order to allow readers or other researchers to replicate the experiment. In particular, ARRIVE 2.0 places an emphasis on the methodology and statistics used. Among the items in ARRIVE 2.0, items for study design, sample size, randomization, blinding, outcome measures, statistical methods, and results are related to methodological and statistical issues in the ‘Essential 10’. In the ‘Recommended set’, items for interpretation/scientific implications, protocol registration, and data access can contribute to methodological and statistical issues. The latter two items, protocol registration and data access, are newly introduced to ARRIVE 2.0.

I would like to make a short comment on some methodological and statistical issues that are generally overlooked by readers and researchers, as well as newly introduced items, protocol registration and data access.

First, ARRIVE 2.0 guidelines recommend that statistical methods should be described with enough detail to allow a knowledgeable reader with access to the original data to judge the appropriateness of the methods for the study and to verify the reported results, which is consistent with the International Committee of Medical Journal Editors (ICMJE) recommendations [4]. It would help readers, reviewers, and other researchers to assess the appropriateness of the statistical methods used and the validity and reliability of the results. Authors of basic and preclinical studies often feel that the space of the manuscript is limited, which contributes to the insufficient description of statistical methods. However, many journals, including the Korean Journal of Pain, provide online supplement sections; thus, there is little excuse for the author not to report statistical methods with clarity and provide sufficient detail. A researcher may use the appropriate statistical methods; however, if they did not report with sufficient detail, readers, reviewers, and other researchers may doubt the appropriateness, validity, and reliability of the statistical methods used.

Second, the results from exploratory studies, where no specific hypothesis was tested, should not be used to draw confirmatory conclusions. Exploratory data analysis can summarize the characteristics and reveal new findings beyond formal hypothesis testing, thus leading to the formulation of a novel hypothesis. However, a method of hypothesis testing should be used to confirm if the findings are valid or can be explained by random errors. Therefore, if formal hypothesis testing was not performed, the findings from exploratory data analysis should not be regarded as confirmatory.

Third, ARRIVE 2.0 guidelines recommend explaining how the sample size was determined, and providing details of sample size calculation. The “3Rs”, Replace, Reduce, and Refine, proposed by Russell and Burch became a standard in animal study [5]. Of those, the second R, reduction, represents minimizing the number of animals used, without disrupting scientific aims or quality of information. Inappropriate sample size may raise the scientific, economic, and ethical issues. Over-powered studies may make trivial findings statistically significant, waste of limited resources, and result in an unnecessary use of animals. However, under-powered studies may decrease the power of studies, resulting in inconclusive results. Thus, appropriate and detailed sample size calculation or power analysis and are necessary in basic and preclinical studies.

Fourth, ARRIVE 2.0 guidelines recommend the need for protocol registration. The protocol should include the number of animals studied, the intervention to be used, the primary outcome, and the statistical analysis plan, which was decided in advance, before data collection. This would help prevent the selective reporting of statistically significant results, p-hacking, and the use of a statistical method different from the statistical analysis plan [6]. For protocol registration, we can use the registered report format provided by some journals or online resources such as the Open Science Framework (https://osf.io/).

Fifth, ARRIVE 2.0 guidelines recommend the presentation of a data sharing statement, which describes if and where research data are available. Data sharing allows readers and other researchers to replicate and verify data analysis results and explore new topics using shared data.

The Korean Journal of Pain has attempted to improve the quality, transparency, reliability, and reproducibility of studies by applying reporting guidelines such as CONSORT, STROBE, and PRISMA. We also hope to make a meaningful contribution to the research field and consequently patient care through animal studies. However, efforts to improve the quality and transparency of basic and preclinical studies by applying strict methodologies and guidelines seem to be insufficient, and we feel that there is room for improvement. To improve the quality and transparency of The Korean Journal of Pain, it would be important to encourage authors to report basic and preclinical studies according to ARRIVE 2.0 guidelines, and upload an ARRIVE 2.0 checklist when submitting a manuscript to The Korean Journal of Pain.

In conclusion, the update of the ARRIVE 2.0 guidelines is timely, and we should adopt and apply strict methodologies and transparent reporting based on these guidelines.