Reporting of measures against bias in nonclinical published research studies: a journal-based comparison

Abstract

Background

Historically, systematic review studies of nonclinical published research articles around the life sciences have shown that the overall reporting of information on measures against bias is low. Measures such as randomization, blinding and sample size estimation are mentioned in the minority of the studies. The present study aims to provide an overview of the recent reporting standards in a large sample of nonclinical articles with focus on statistical information.

Methods

Journals were randomly selected from Journal Citation Reports (Clarivate). Biomedical research articles published in 2020 from 10 journals were analyzed for their reporting standards using a checklist.

Results

In total 860 articles; 320 articles describing in vivo methods, 187 articles describing in vitro methods and 353 articles including both in vivo and in vitro methods, were included in the study. The reporting rate of “randomization” ranged from 0%-63% between journals for in vivo articles and 0%-4% for in vitro articles. The reporting rate of “blinded conduct of the experiments” ranged from 11%-71% between journals for in vivo articles and 0%-86% for in vitro articles.

Conclusion

The analysis showed that the reporting standards remained low, also when other statistical information is concerned. Additionally, our results suggest that the reporting in articles on in vivo experiments is better compared to articles on in vitro experiments. Furthermore, important differences in reporting standards between journals seem to exist.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41073-025-00176-w.

Background

Issues with low internal validity of nonclinical animal research have received more attention in recent years [1]. Nonclinical (animal research) includes all studies with animals conducted before clinical trials that is early discovery studies and preclinical good laboratory practice studies included. The internal validity of a research study is the extent to which the design, conduct, analysis and reporting of that study eliminates the risk of bias and thus to which the results of the studies can be attributed to an actual effect [2].

These issues are, at least in part, reflected by the difficulty of reproducing published data of nonclinical research, with irreproducibility rates between 65–89% [3–5]. Adherence to more rigorous experimental design and reporting standards is a strategy to improve internal validity and lower the risk of bias.

Multiple studies evaluated the reporting standards across various in vivo research areas and concluded that reporting of randomization, blinding and sample estimation was poor despite the introduction of the ARRIVE guidelines in 2010 [6–10]. It has been argued that complete reporting, including information on statistical analysis, is essential to improve the reproducibility rates [11].

As a response to the disappointing results in the adherence to the first ARRIVE guidelines and the lack of improvement in the quality of reporting in animal research articles the ARRIVE 2.0 guidelines were published in 2020 [7].

The aim of this study was to provide an overview of the reporting rates in different scientific journals across the life sciences using a checklist with the emphasis on statistical information, as this is believed to be more routinely reported in scientific articles. We further took factors to minimize bias into account. Specifically, we reported if articles reported measures against bias (e.g. animals were randomly allocated to experimental groups). In case articles clearly mentioned that measures against bias were not taken (e.g., no randomization was applied) we could not score them positively for reporting measures against bias. Instead, we created the transparency score for each journal, this is the sum of all articles reporting the measure against bias and the articles clearly reporting that a measure against bias was not taken.

Furthermore, we compared the difference in reporting rates between in vivo and in vitro articles as well as differences in reporting rates between journals. Additionally, we compared the reporting rates of randomization, blinding and sample size calculations within in vivo research articles with the results from the study performed by Kilkenny et al. [12]. Therefore, we assessed the reporting of measures against bias and information on statistical methods within in vivo and/or in vitro biomedical research studies published in 2020.

Methods

Journal selection

Scientific journals were selected from “Journal Citation Reports (Clarivate)”. Only journals that fell within impact factor quartile 1, the top 25% journals within a subject category based on impact factor from the Journal Citation Reports (JCR) year 2020 were included, as this study was conducted in February 2021. The journals were further filtered by the following categories to ensure only journals in the life sciences were included: Behavioral sciences; Biochemical research methods; Biochemistry & Molecular biology; Biology; Cell & Tissue engineering; Cell biology; Developmental biology; Genetics & Heredity; Immunology; Microbiology; Multidisciplinary sciences; Neurosciences; Oncology; Pharmacology & Pharmacy; Physiology; Psychology; Biological; Psychology; Experimental; Toxicology; Virology and exclusion criteria: Reviews only; Clinical only; Human psychology only; Veterinary; Zoology; Ecology; Computational; Biotechnology only; Structural biology only; non-English language; Duplications. From this list 10 journals were randomly selected using the RAND() function in Excel to ensure inclusion of journals across the various domains in biomedical research. Journals were anonymized and assigned a number between 1 and 10.

Article selection

For every journal the issues published in 2020 were searched starting from January until a maximum of 100 articles per journal was reached. Sample calculations were neither performed nor necessary, as no hypotheses were being tested. Given the exploratory nature of the study, there was no requirement for power calculations. Therefore, the sample size was based on reasonable time considerations needed to analyze each article. The selection of the articles was based on the title, abstract and methods sections. Only original in vivo and/or in vitro (cell culture) research studies were included. Review articles, short communications, letters to the editor and meta-analyses were excluded from the study.

Assessment of reporting rates

For the analysis of the reporting rates of measures against bias and information on statistical methods of nonclinical research studies a checklist was used. This checklist was based on the following guidelines:

• “Systematic review of guidelines for internal validity in the design, conduct and analysis of preclinical biomedical experiments involving laboratory animals” conducted by Vollert et al. in 2020[1].

• ARRIVE 2.0 guidelines [7].

• The “Checklist for Reporting In-vitro Studies” (CRIS guidelines) [13].

• The reporting standards recommended by Emmerich and Harris “Minimum Information and Quality Standards for Conducting, Reporting, and Organizing In Vitro Research” [14].

• The “Statistical Analyses and Methods in the Published Literature” or The SAMPL Guidelines”[15]

In this study, we analyzed articles using a checklist with an emphasis on statistical information (e.g., the reporting of confidence intervals, sample size calculations, identification and handling of outliers, etc.), as this was expected to be more routinely reported in scientific articles. Each study characteristic was chosen as the minimum information needed by a reader to be able to fully understand the study and to be able to evaluate the internal validity of the study. Separate checklists were used for the different research categories, one for in vitro (cell culture)/ex-vivo research, one for in vivo research and one for articles that included both (supplementary method 1). This ensured data from different research categories were not pooled. Each item was scored 1 (presence of needed information in the text) or 0 (absence of the needed information in the text). Supplemental methods and information from the articles were included in the analysis. However, supplemental figures and tables were excluded from the analysis. The following study characteristics were extracted during the analysis and will be discussed in this article:

• Experimental Design:

• Randomization

  • ◦
    Randomized allocation of the experimental units to different experimental groups: Have experimental units been allocated to treatment groups, in such a way that every experimental group had the same chance of receiving any of the treatments?
  • ◦
    Type of randomization: Which design was used to allocate experimental units to treatment groups such as simple randomization, block randomization or stratified randomization?
  • ◦
    Package/method used to generate the randomization sequence: Which tools such as Excel, random number generator or other software were used for creating the randomization sequence?
• Blinding

  • ◦
    Allocation concealment: Were experimenters unaware of treatment groups during the allocation of experimental units to treatment groups?
  • ◦
    Blinded conduct of experiment: Were experimenters deliberately kept unaware of the treatment assignments for each group?
  • ◦
    Blinded outcome assessment: Were outcome assessors deliberately kept unaware of the assigned treatment of each group?
  • ◦
    Blinded data analysis: Were experimenters responsible for analyzing the data kept unaware of group treatment?
• Statistics

  • ◦
    A priori sample size calculations: Were the number of experimental units used, based on power calculations for a particular experiment? The power of an experiment is the probability of detecting a true event and depends on the sample size, variability, and the acquired confidence level.
  • ◦
    Identification and handling of outliers: Which methods have been used to detect outliers, including data visualization and statistical tests? It also addresses how data points will be treated once identified as outliers.
  • ◦
    Experimental unit: What was the unit of randomization e.g., single animal, group of animals, litter or cage of animals reported?
  • ◦
    Predefined minimum effect size: Was the predefined minimum effect size, measure to express the difference between groups, or the relationship between variables, reported?
  • ◦
    Confidence interval reported: Was the confidence interval, the probability of correctly accepting the null hypothesis, i.e., not rejecting it when it is true, reported?

Data analysis

Data were collected, organized and processed using Microsoft Excel. Articles reporting an item were scored “1” in a separate Excel file; when an item was not explicitly mentioned, it was scored “0”. General information such as the year of publication, research field, journal, citation count, and publisher were also collected. Results were presented and compared using descriptive statistics, specifically mean values and percentages.

Although the nature of this paper is purely exploratory and results should be interpreted as such, the difference in reporting rates between in vivo and vitro (cell culture) articles was statistically analyzed. For the statistical analysis of the differences in standards of reporting a two-tailed, unpaired Mann–Whitney U test was used with a significance level p < 0.05. The analysis of data was performed using GraphPad Prism software (9.5.1).

Results

After filter and exclusion criteria a list of 297 journals remained, from these 10 journals were randomly sampled. Our search resulted in 1542 unique records after journal search, and 860 articles remained after abstract and title screening. Search results are summarized in a PRISMA inspired diagram (Fig. 1). We analyzed 860 articles across various fields in the life sciences published in 10, randomly selected journals, independent from their impact factor. We compared reporting standards in articles describing in vivo (n = 320), in vitro (cell culture) (n = 187) and in vivo/vitro (n = 353) experiments. Furthermore, we analyzed the reporting if measures weren’t applied in the experiments, which we describe as transparency. No statistics were used to calculate differences of reporting standards between journals and only descriptive statistics were used to discuss the results.

Fig. 1.

PRISMA flow diagram

Comparison with previous studies

Kilkenny et al. (2009) [12] and Baker et al. (2014) [16] conducted a survey study to analyse the reporting and experimental design features in animal studies and concluded that the quality of reporting of biomedical research using animals needed to improve. Figure 2 shows the comparison of the 2009 and the 2014 study with the results of the current study for randomization, blinding and sample size calculations reported in animal studies. Overall, levels of reporting these items increased but remained low.

Fig. 2.

Radar plot of randomization, blinding and sample size calculations compared to results of Kilkenny et al. [12] and Baker et al. [16] studies

Randomization

The percentage of reporting randomization in articles describing in vivo experiments between journals (Fig. 3A) varied greatly with a range of 0%−63%, the articles published in journal 4 and 9 reported randomization more often compared to the other journals. However, when transparency was taken into account (Fig. 3C), meaning articles stated that “randomization was not applied” or “randomization was applied,” respectively, the reporting range became 0%−90%, with journal 3 reporting these items the most. The minority of articles that mentioned randomization also reported the type of randomization (0%−71%) and/or the package used to generate the randomization sequence (0%−94%), journal 3 reported this information more often than other journals included in this study (Fig. 3E). It is important to note that sample sizes can vary between journals. This couldn’t be prevented as this variation arises from the publication rates of each journal in the given year, making it impossible to account for these differences when determining sample sizes in advance.

Fig. 3.

Reporting of randomized allocation of experimental units to different experimental groups in articles describing in vivo or in vitro experiments across 10 journals. A-B Percentage of articles reporting randomization of in vivo or vitro articles. C-D Percentage of articles reporting if randomization was or wasn’t performed. Figures E–F Percentage of articles reporting the type of randomization and the method of randomization if randomization was reported. The numbers between brackets after every journal number indicate the number of articles included per journal in the analysis

Articles describing in vitro experiments rarely report randomization of the experimental units (0%−4%), only journal 2,3,5 and 7 included some articles that mentioned the item (Fig. 3B). Including the transparency of not reporting the item (0%−48%), journal 3 reported the most often if the measure wasn’t applied (Fig. 3D) and also mentioned on every occasion the method used to generate the randomization sequence (Fig. 3F).

Next, articles describing both in vivo and in vitro experiments (Supplementary Fig. 1) were analyzed to evaluate if reporting standards for in vivo experiments had an influence on reporting standards for in vitro research. However, the difference in reporting of randomization was still high between in vivo and in vitro experiments (0%−60% and 0%−4%, respectively). As for in vivo articles, articles of journal 4 reported randomization more often compared to the other journals (Supplementary Fig. 1 A), but taking transparency into account (Supplementary Fig. 1C-D), journal 3 reported randomization, applied or not, the most (3%−91% for in vivo and 0%−18% for in vitro). Reporting of the type of randomization (0%−50%) and/or the package used to generate the randomization sequence (3%−100%) was slightly better for in vivo but worse for (0%) for in vitro research compared to articles only describing one out of two types of experiments.

In Supplementary Fig. 2 the reporting of randomization and transparency of randomization between in vivo and in vitro articles were shown. In general, the standard of reporting information on randomization in in vivo articles was somewhat better compared to in vitro articles.

Blinding

The analysis of the reporting of blinding was divided into four categories: allocation concealment, blinded conduct of experiment, blinded assessment of outcome and blinded data analysis (definitions can be found in the material section of this article). The percentage of reporting allocation concealment in articles describing in vivo experiments was almost zero, the other types of blinding were reported more often, with reporting the outcome assessment in a blinded manner being the most frequent, but still not exceeding 50% (Fig. 4A). However, taking transparency of reporting into account for the blinded conduct of the experiment, blinded assessment of outcome and blinded data analysis (Fig. 4C), the reporting range varied from 0%−62%, 11%−71% and 0%−45%, respectively.

Fig. 4.

Reporting of blinding procedures in articles describing in vivo or in vitro experiments. A-B Percentage of articles reporting blinding procedures of in vivo or vitro articles. C-D Percentage of articles reporting blinding procedures and if blinding wasn’t performed. The box and whiskers plots show the minimum and maximum percentage of reporting over all journals analyzed. Every individual datapoint represents a journal

Articles describing in vitro experiments rarely reported allocation concealment, blinded conduct of experiment, blinded assessment of outcome or blinded data analysis. Still, blinded assessment of outcome was mentioned the most, but it was at most 50% (Fig. 4B). Including the transparency of not reporting the items, the percentage of reporting (Fig. 4C-D) did exceed 50% on some occasions. This, however, was only the case for 1 journal and the ranges varied greatly with 0–86% for blinded conduct of experiment, 0–76% for blinded assessment of outcome and 0–36% for blinded data analysis.

Similar to the analysis for reporting standards for randomization, we analyzed articles describing both in vivo and in vitro experiments (Supplementary Fig. 3) to evaluate if reporting standards for in vivo experiments had an influence on reporting standards for in vitro research. However, the difference for reporting allocation concealment, blinded conduct of experiment and blinded assessment of outcome (0–17% and 0–17%, 0–33% and 0–17%, 4–50% and 0–21% respectively) was still high between in vivo and in vitro experiments (Supplementary Fig. 3 A-C). Taking transparency into account (Supplementary Fig. 3B-D), the difference in rates between in vivo and in vitro experiments reporting standards still exists. Blinded data analysis was not included in articles including in vivo and in vitro experiments, as it was often difficult to distinguish if the blinded data analysis was only done for animal experimental data and/or cell culture data, as this item was often discussed in the general data analysis paragraph, rather than the separate method section of the experiments described in the articles.

In Supplementary Fig. 4 and 5, we compared the reporting of blinding and transparency of blinding between in vivo and in vitro articles. In general, the standard of reporting information on blinding in in vivo articles was in some case significantly higher compared to in vitro articles but overall still low.

Information on statistical methods

The reporting standards for statistical information were analyzed with an emphasis on the reporting of a priori sample size calculations, identification and handling of outliers, whether the experimental unit was clearly specified, if an effect size was predetermined, and whether confidence intervals were reported.

A high variability between journals was observed for the reporting of the items listed above in articles describing in vivo experiments (Fig. 5A). Overall, the reporting of the confidence intervals was the lowest (0–8%), followed by the predefined minimum effect size (0–21%), identification of the experimental unit (0–12%) and identification/handling of outliers (0–38%). Journals number 2, 8 and 9 didn’t include any of this information in the articles from our sample. In contrast to in vivo articles, in vitro articles (Fig. 5B) did report the experimental unit more often (0–50%). However, reporting of the predefined minimum effect size (0%), confidence interval (0–11%) and identification/handling of outliers (0–36%) was comparable to in vivo studies. Next, we compared the reporting standards between in vivo and in vitro experiments when discussed in the same article (Fig. 5C), with similar observations as described above: reporting of confidence intervals was low (0–17%), as was the reporting of the predefined minimum effect size (0–9%), a priori sample size estimation for animals (0–33%) and cell cultures (0–33%). As discussed earlier, blinded data analysis was included here, as the distinction between blinded data analysis for in vivo/in vitro was difficult to make due to description of this variable within the paragraph that handled data analysis more general. Reporting of blinded data analysis ranged from 0–21%, which was lower than the reporting of this information in articles that only described in vivo or in vitro experiments (Fig. 4).

Fig. 5.

Reporting of statistical information in articles. A-B The reporting of statistical information in vivo or vitro articles. C Reporting of statistical information in articles describing in vivo and vitro experiments. The numbers between brackets after every journal number indicate the number of articles per journal included in the analysis

To compare how often articles reported that a certain measure wasn’t applied, an overview is provided in supplementary Fig. 6 A-B of the articles reporting if priori sample size calculations and identification/handling of outlier was or wasn’t done (transparency score). Considering transparency, reporting standards increased for a priori sample calculations (0–95%) and outlier identification/handling (0–79%) for in vivo articles and in vitro articles (0–68% and 0–84%, respectively). Journal number 3 reported this information the most frequent for both types of articles. For journals including both types of experiments, the reporting standards increase for a priori sample calculations in vivo (0–91%) and outlier identification/handling (0–97%; supplementary Fig. 6 C). The reporting of a priori sample size calculation (0–33%) remained the same, while a slight increase was noticed for the reporting of blinded data analysis (0–27%). Again, journal number 3 reported this information most frequently.

Discussion

This study has provided an overview of the reporting of the following measures: randomization, blinding and information related to statistical methods, in 860 articles published in 2020. Furthermore, we compared reporting standards between in vivo and in vitro (cell culture) biomedical research articles and conducted a separate analysis for cases where it was explicitly stated that specific measures were or weren’t included in the experimental design (transparency). This way, we attempted to analyze if these items were more frequently reported compared to previous studies and if an improvement in reporting over time could be observed. Rigorous experimental design is what is expected from research, which means ensuring high internal validity by, among others, including measures against bias. The importance of these measures is partly reflected by the difficulty of researchers to reproduce results obtained by other scientists or even their own work [17]. Another important aspect of being able to reproduce other scientists'work is the complete reporting of the methodology used to obtain the findings.

The ARRIVE guidelines were created as a response to the article published by Kilkenny et al. [12], in which the low reporting standards in articles including animals were brought to attention. They found that randomization was reported in 13% and blinding in 14% of the articles. Although the reporting of these items improved in the following years, the overall reporting still remained low as shown by Baker et al. [16] and Leung et al. [18]. Our study confirmed these findings, randomized allocation of animals to experimental groups was reported in 0%−63% (median 33%) of the articles and blinded assessment of outcome in 11%−50% (median 20%). Other items related to blinding were reported less frequently, with allocation concealment ranging from 0 to 2% (median 0%), blinded conduct of the experiment from 0 to 36% (median 4%) and blinded data analysis 0% to 31% (median 2%). Our results indicate that the quality of reporting of nonclinical research remains low. This study is conducted on articles published in 2020, eleven years after the Kilkenny article and 10 years after the publication of the initial ARRIVE guidelines.

Limitations

Our study does have limitations: due to the exploratory nature, no a priori sample sizes were calculated, and differences in sample size between journals do exist. Furthermore, we only included 10 journals from the first quartile and only focused on one publication year (2020). We evaluated reporting of randomization, blinding and information on statistical methods. However, we do not know if these measures were not considered in the experimental design of the published studies or simply not reported. Further research should be performed on a larger sample size with a focus on including more journals and more publication years. Furthermore, a comparative study between what was reported and what was done should be performed.

No conclusions can be drawn about the impact of the ARRIVE 2.0 guidelines, published in 2020 [7], as our sample size of articles was published in the same year.

Conclusion

Our study shows that, overall, the standards of reporting measures against bias improved but remains low and that the standards of reporting in in vivo articles are somewhat better compared to in vitro articles, which could also be observed in articles describing both in vivo and in vitro experiments. Although we did not use statistics to compare journals, important differences in standards of reporting between journals seem to exist. This difference seems to increase when including the transparency score, also reporting when measures were not done (e.g., if randomization was not applied) in the study.

Abbreviations

ARRIVE

Animal Research: Reporting of In Vivo Experiments

CRIS

Checklist for Reporting In-vitro Studies

SAMPL

Statistical Analysis and Methods in the Published Literature

Authors’ contributions

Conceived and designed the study: SS, TL and TS. Performed the review study: SS. Analyzed the data: SS. Wrote the paper: SS, TL and TS.

Funding

Sara Steele is a Baekeland Mandate holder (HBC.2021.0796), awarded by VLAIO, the Flemish Agency for Innovation and Entrepreneurship.

Data availability

The data supporting the findings of this study are available within the article or will be made available upon reasonable request.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interest

The authors have to declare the following potential conflicts of interest: all authors are employees of Johnson & Johnson.