A study to assess the methodological quality of in vivo animal experiments published in Indian journal of pharmacology: A retrospective, cross-sectional, observational study

Abstract

BACKGROUND:

Good reporting of experimental research is a vital part of research process. Although the reporting guidelines such as Animal research: Reporting in vivo experiments (ARRIVE) require the authors to stick to a standard format, they do not guarantee study reports' validity. For assessing the study reports validity, critical appraisal tools are required.

OBJECTIVE:

The evaluation of the methodological quality of in vivo animal studies of Indian Journal of Pharmacology (IJP) published between 2011 and 2017 through critical appraisal was the primary objective. The secondary objective was to evaluate the adherence of selected studies to the ARRIVE guidelines.

MATERIALS AND METHODS:

All in vivo animal studies published as full-text articles in IJP from January 2011 to December 2017 and satisfying the inclusion norms were included. A checklist based on the underlying principles of ARRIVE statement was applied to in vivo animal research (AR) published in IJP. For critical appraisal of reports, risk-of-bias domains were also applied on studies from in vivo AR.

RESULTS:

One hundred and sixty-one studies satisfied the inclusion criteria. Seventy-three articles (45.34%, 95% confidence interval [CI]: 0.37–0.53) discussed about randomization procedure. Eighteen articles (11.18%, 95% CI: 0.07–0.16) gave details about blinding when assessing results. None of the studies provided details of sample size calculation. Eight articles (4.97%, 95% CI: 0.02–0.09) commented on the study limitations.

CONCLUSION:

It was found that adherence to only some criteria of ARRIVE guidelines was subpar. There is a need for optimal reporting of random distribution of animals to experimental groups, concealment of allocation, blinded outcome evaluation, computation of sample size, and attrition of animals for improving the validity of AR.

Introduction

Replication of preclinical studies' results in humans has been disappointing. There may be two likely explanations behind this. To begin with, animals are intricate living creatures. Second, the methodological nature of animal research (AR) may be deficient, causing ambiguous outcomes.[1] There is considerable corroboration that reporting of experimental research is less than optimal.[2] Moreover, the low replication of ARs can be ascribed to insufficient reporting. Thus, optimal reporting of experimental research is an essential part of any research.[3] In view of these confirmations, there have been demands for improved reporting in AR.[4] The Animal research: Reporting in vivo experiments (ARRIVE) statement in 2010 epitomized a key step to ameliorate the reporting standard of AR.[5,6] The ARRIVE guidelines recommend addition of the methodological factors that have been found to be inadequately reported in animal experiments. Application of ARRIVE may result in enhanced reporting quality of AR (though there is a report contrary to that).[7]

Although the reporting guidelines require the authors to stick to a standard format, they do not guarantee the study validity.[5] The poor reporting of animal experiments, due to compromised internal and external validity, has been implicated in the divergences between the results of preclinical and clinical studies.[8] Validity of the report can be evaluated by critical appraisal tools.[9] It is imperative to appraise the internal as well as external validity to judge whether outcomes of the study represent the source and target population. Internal as well as external validity can be assessed through critical appraisal.[10]

Indian Journal of Pharmacology (IJP) is a globally recognized and highly respected journal. Majority of studies published in IJP consist of in vivo AR. It was thus sensible to evaluate in vivo animal studies adherence to ARRIVE guidelines and to evaluate the methodological quality of the in vivo animal studies through critical appraisal.[11,12]

Objective

The evaluation of the methodological quality of in vivo animal studies published in IJP from 2011 to 2017 through critical appraisal was the primary objective. The secondary objective of was to evaluate the adherence of the published in vivo animal studies to the ARRIVE guidelines.

Materials and Methods

Study design

This was a retrospective, cross-sectional, observational study to evaluate the methodological quality of in vivo animal studies through critical appraisal and to examine their adherence to ARRIVE guideline. The ethical committee sanction was not obligatory because this study completely relies on the information accessible in the public domain and is a quality audit not incorporating sensitive information retrieval.

Data extraction

All in vivo animal studies published as full-text articles in IJP from January 2011 to December 2017 and fulfilling the eligibility criteria were eligible for inclusion. Eligible articles were identified by screening of titles, abstracts, keywords, and methodology.

Eligibility criteria

Inclusion criteria

(1) Animal studies with treatment intervention, (2) only original research articles, and (3) studies published between January 2011 and December 2017.

Exclusion criteria

(1) In vitro experiments, (2) studies on animals that were dead before inception of the study, (3) studies where no intervention was performed, (4) studies with no control group, and (5) review articles, commentaries, communications or correspondences, and short communication.

Assessment of adequacy of reporting and critical appraisal of evidences

The assessment of reporting of the in vivo animal experiments was done by guiding principles on the ARRIVE statement. A checklist was prepared dependent on the ARRIVE statement [Tables 1–5].[6,13] Some items on the checklists were added based on a study by Muhlhausler et al., 2013.[5] The checklist was applied to in vivo AR published in IJP. For critical appraisal of reports, risk-of-bias domains were also applied on studies from in vivo AR.[10,14] The in-depth appraisal of AR's methodology, management, and analysis was done to determine its validity.

Table 1.

ARRIVE checklist for title (item number 1), abstract (item number 2), introduction (item number 3), objectives (item number 4), and methods (item number 5 and 6)

Item	Number	Recommendation	Yes (%)	No (%)	95% CI
Title	1	Provide as accurate and concise a description of the content of the article as possible	161 (100)	0
Abstract	2	Provide an accurate summary of the background, research objectives (including details of the species or strain of animal used), key methods, principal findings, and conclusions of the study	161 (100)	0	1 (0.97-1)
Introduction
Background	3	(a) Include sufficient scientific background (including relevant references to previous work) to understand the motivation and context for the study, and explain the experimental approach and rationale	161 (100)	0	1 (0.97-1)
		(b) Explain how and why the animal species and model being used can address the scientific objectives and, where appropriate, the study’s relevance to human biology	33 (20.50)	128 (79.50)	0.20 (0.14-0.27)
Objectives	4	Clearly describe the primary and any secondary objectives of the study, or specific hypotheses being tested	161 (100)	0	1 (0.97-1)
Methods
Ethical statement	5	Indicate the nature of the ethical review permissions, relevant licenses (e.g., animal [scientific procedures] act 1986), and national or institutional guidelines for the care and use of animals, that cover the research	145 (90.06)	16 (9.94)	0.90 (0.84-0.93)
Study design	6	For each experiment, give brief details of the study design, including
		(a) The number of experimental and control groups	161 (100)	0	1 (0.97-1)
		(b) Any steps taken to minimize the effects of subjective bias when allocating animals to treatment (e.g., randomization procedure)	73 (45.34)	88 (54.66)	0.45 (0.37-0.53)
		(c) When assessing results (e.g., if done, describe who was blinded)	18 (11.18)	143 (88.82)	0.11 (0.07-0.16)
		(d) The experimental unit (e.g., a single animal, group, or cage of animals) A timeline diagram or flow chart can be useful to illustrate how complex study designs were carried out	161 (100)	0	1 (0.97-1)

CI=Confidence interval

Table 5.

ARRIVE checklist for results (item number 15, 16, 17), discussion (item number 18, 19 and 20)

Item	Number	Recommendation	Yes (%)	No (%)	95% CI
Results
Numbers analyzed	15	(a) Report the number of animals in each group included in each analysis. Report absolute numbers (e.g., 10/20, not 50%)	161 (100)	0	1 (0.97-1)
		(b) If any animals or data were not included in the analysis, explain why	0	161	-
Outcomes and estimation	16	Report the results for each analysis carried out, with a measure of precision (e.g., SE or CI)	161 (100)	0	1 (0.97-1)
Adverse events	17	(a) Give details of all important adverse events in each experimental group	0	161 (100)	-
		(b) Describe any modifications to the experimental protocols made to reduce adverse events	0	161 (100)	-
Discussion
Interpretation/scientific implications	18	(a) Interpret the results, taking into account the study objectives and hypotheses, current theory, and other relevant studies in the literature	161 (100)	0	1 (0.97-1)
		(b) Comment on the study limitations including any potential sources of bias, any limitations of the animal model, and the imprecision associated with the results	8 (4.97)	153 (95.03)	0.04 (0.02-0.09)
		(c) Describe any implications of your experimental methods or findings for the replacement, refinement, or reduction (the 3 Rs.) of the use of animals in research	0	161 (100)	-
Generalisability/translation	19	Comment on whether, and how, the findings of this study are likely to translate to other species or systems, including any relevance to human biology	0	161 (100)	-
Funding	20	List all funding sources and the role of the funder (s) in the study	55 (34.16)	106 (65.84)	0.34 (0.27-0.41)

SE=Standard error, CI=Confidence interval

Data analysis

An exploratory and descriptive statistical analysis was done. Descriptive statistics was applied to illustrate the reporting of studies, as well as for the key methodological items. The SPSS statistical software package (SPSS for Windows, Version 16.0. Chicago, SPSS Inc.) was utilized for statistical analysis. Criteria's were demarcated as “yes” if they were unambiguously reported or “no” if they were indistinct or not stated at all. The lower and upper limits of the 95% confidence interval (CI) for the proportions was calculated.

Results

Overall, 1185 articles from IJP between January 2011 and December 2017 were screened for suitability. Figure 1 explains the retrieval and selection of articles in the present study. Out of the total 1185 articles, the screening of title and abstracts excluded 637 articles. These 637 articles included Drug watch (190); Letters to editor or correspondences (144); Short communications (118); Education forum and review articles (54); Editorials (46); and others (n = 85), i.e., Abstracts (29), Book reviews (18), Case reports (13), Obituaries (11), Errata (7), Announcements (4), Special article (2), and Journal club (1). Now, 548 articles were left and these articles were further evaluated.

Figure 1.

Flow diagram of retrieval and selection of articles

One hundred and sixty-nine articles were omitted from 548 articles after full-text review. These 169 articles included randomized control trials (30); cross-sectional studies (25); pharmacokinetic/pharmacodynamics studies (22); drug utilization studies (20); questionnaire studies (19); observational studies (16); nonrandomized clinical trials (10); randomized uncontrolled clinical trials (10); systematic review/meta-analysis (6); case–control/cohort studies (5); drug safety studies (4); and pharmacoeconomic studies (2). Now, 379 preclinical articles were left, and these articles were further evaluated. A total of 218 in vitro and other types of preclinical studies were omitted after evaluation of complete text. Only rest of 161 studies satisfied the inclusion prerequisites and thus incorporated in the study [Figure 1].

Out of 161 included articles, 33 articles (20.50%, 95% CI: 0.14–0.27) explained in what way the animal type being utilized can justify the methodical objectives. All the included articles clearly described objectives of the study. One hundred and forty-five articles (90.06%, 95% CI: 0.84–0.93) mentioned about the ethical permissions. Seventy-three articles (45.34%, 95% CI: 0.37–0.53) discussed about randomization procedure. Eighteen articles (11.18%, 95% CI: 0.07–0.16) described any type of blinding [Table 1]. One hundred and fifty-three articles (95.03%, 95% CI: 0.90–0.97) provided specifics of the animals used [Table 2]. None of the studies provided specifics about computation of sample size [Table 3]. All the articles provided details of statistical methods used [Table 4]. Eight articles (4.97%, 95% CI: 0.02–0.09) commented on the study limitations. Fifty-five articles (34.16%, 95% CI: 0.27–0.41) listed all funding sources [Table 5].

Table 2.

ARRIVE checklist for methods (item number 7 and 8)

Item	Number	Recommendation	Yes (%)	No (%)	95% CI
Methods
Experimental procedures	7	For each experiment and each experimental group, including controls, provide precise details of all procedures carried out. For example
		(a) How (e.g., drug formulation and dose, site and route of administration, anesthesia, and analgesia used [including monitoring], surgical procedure, method of euthanasia). Provide details of any specialist equipment used, including supplier (s)	161 (100)	0	1 (0.97-1)
		(b) When (e.g., time of day)	12 (7.45)	149 (92.55)	0.07 (0.04-0.12)
		(c) Where (e.g., home cage, laboratory, and water maze)	23 (14.29)	138 (85.71)	0.14 (0.09-0.20)
		(d) Why (e.g., rationale for choice of specific anesthetic, route of administration, and drug dose used)	0	161	-
Experimental animals	8	(a) Provide details of the animals used, including species, strain, sex, developmental stage (e.g., mean or median age plus age range), and weight (e.g., mean or median weight plus weight range)	153 (95.03)	8 (4.97)	0.95 (0.90-0.97)
		(b) Provide further relevant information such as the source of animals, international strain nomenclature, genetic modification status (e.g., knock-out or transgenic), genotype, health/immune status, drug or test naıve, previous procedures.	158 (98.14)	3 (1.86)	0.98 (0.94-0.99)

CI=Confidence interval

Table 3.

ARRIVE checklist for methods (item number 9, 10, and 11)

Item	Number	Recommendation	Yes (%)	No (%)	95% CI
Housing and husbandry	9	Provide details of
		(a) Housing (e.g., type of facility, e.g., SPF; type of cage or housing; bedding material; number of cage companions; tank shape and material, etc., for fish)	146 (90.68)	15 (9.32)	0.90 (0.85-0.94)
		(b) Husbandry conditions (e.g., breeding program, light/dark cycle, temperature, quality of water, etc., for fish, type of food, access to food and water, environmental enrichment)	153 (95.03)	8 (4.97)	0.95 (0.90-0.97)
		(c) Welfare-related assessments and interventions that were carried out before, during, or after the experiment	0	161	-
Sample size	10	(a) Specify the total number of animals used in each experiment and the number of animals in each experimental group	161 (100)	0	1 (0.97-1)
		(b) Explain how the number of animals was decided. Provide details of any sample size calculation used	0	161	-
		(c) Indicate the number of independent replications of each experiment, if relevant	0	161	-
Allocating animals to experimental groups	11	(a) Give full details of how animals were allocated to experimental groups, including randomization or matching if done	73 (45.34)	88 (54.66)	0.45 (0.37-0.53)
		Allocation concealment mechanism (provide details of whether the persons generating the randomization schedule were blinded to treatment)*	0	161	-
		(b) Describe the order in which the animals in the different experimental groups were treated and assessed	161 (100)	0	1 (0.97-1)

*Additional checklist item added based on study by Muhlhausler BS, 2013. CI=Confidence interval, SPF=Specific pathogen free

Table 4.

ARRIVE checklist for methods (item number 12 and 13), results (item number 14)

Item	Number	Recommendation	Yes (%)	No (%)	95% CI
Methods
Experimental outcomes	12	Clearly define the primary and secondary experimental outcomes assessed (e.g., cell death, molecular markers, and behavioral changes)	161 (100)	0	1 (0.97-1)
Statistical methods	13	(a) Provide details of the statistical methods used for each analysis	161 (100)	0	1 (0.97-1)
		(b) Specify the unit of analysis for each dataset (e.g., single animal, group of animals, and single neuron)	161 (100)	0	1 (0.97-1)
		(c) Describe any methods used to assess whether the data met the assumptions of the statistical approach	22 (13.66)	139 (86.34)	0.13 (0.09-0.19)
Results
Baseline data	14	For each experimental group, report relevant characteristics and health status of animals (e.g., weight, microbiological status, and drug or test naive) before treatment or testing (this information can often be tabulated)	0	161 (100)	-

CI=Confidence interval

Discussion

In critical appraisal, the most common characteristic contemplated is internal validity.[15] If the outcomes derived from study population are indicative of the source population, then the study is assumed to be valid internally. The major hazard to internal validity is the risk of bias.[10,14] In in vivo AR, selection bias arises when nonrandom factors impact the distribution of animals to study interventions. Selection bias happens if detected variances between the groups in the study are attributable to the confounder rather than interventions. Randomization is the optimum tool for distribution of confounders across the intervention groups in equal proportion.[10] In this study, 45.34% of articles gave details about randomization. In a study by Kilkenny et al. 2009, 12% of all studies reported randomization of animals to experimental groups.[3] In a review by Hess 2011, 28% studies reported about randomization of animals.[16] In a methodological study of AR by Bara and Joffe 2014, 61% of all studies gave details about randomization.[1]

Performance bias happens when there are logical differences between experimental groups in susceptibility to determinant apart from the study intervention.[10] Allocation concealment can be used to prevent performance bias.[17] Random housing of the animal can also be used to prevent performance bias.[14] In this study, none of the studies reported allocation concealment mechanism. In a study by Bara and Joffe 2014, 6% of the studies reported allocation concealment.[1] Whereas in a study by Hirst et al. 2014, 15% of studies conveyed it.[18]

Detection bias can ensue if there are variances in outcome assessment methods between treatment groups.[10,17] It can be avoided by blinding of the persons evaluating the outcome.[10] In this study, 11.18% of the study reported blinding. In a study by Bara and Joffe 2014, 40% of all studies stated about blinding.[1] Macleod et al. 2015 reported about blinding in 29.5% of animal studies.[19] In a study by Kilkenny et al. 2009, only 14% of all included study reported about blinding.[3]

Attrition bias is a problem in clinical research but the likelihood for discretionary attrition bias in in vivo AR is practically nil. However, nondiscretionary attrition can unintentionally introduce bias.[10] The commonly used tool to prevent attrition bias is reporting dropouts. In this study, animal number in each experimental group included in the analysis was given by all studies. None of the article in this study gave details about any dropout or explain why they were not included in the analysis. In the report by Hotwani et al. 2017, 91% of the study reported the animal number in each experimental group included in the analysis.[12]

External validity defines whether outcomes of the study represent the target population. O'Connor and Sargeant stated that if the study results are internally valid, then it will represent the target population.[10] There are no guidelines for assessing external validity of AR.[10] This study does not offer any insight into the external validity of the studies analyzed. This study had several limitations. The included studies were diverse with respect to the study outcomes, animals used, and disease types. Some studies may have failed to report some methodological details when in fact these were used. This study only assesses the reporting quality of assessed studies and did not aim to evaluate the quality of conduct.

Conclusion

Overall, this study demonstrated good adherence of in vivo AR studies published in IJP with respect to the ARRIVE statement with few exceptions. It was found that adherence to only some criteria of ARRIVE guidelines was subpar. There is a necessity for optimal reporting of random distribution of animals to experimental groups, concealment of allocation, blinded outcome evaluation, computation of sample size, and reporting of attrition of animals for improving validity of AR.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.