Quality of reporting statistics in two Indian pharmacology journals

Abstract

Objective:

To evaluate the reporting of the statistical methods in articles published in two Indian pharmacology journals.

Materials and Methods:

All original articles published since 2002 were downloaded from the journals’ (Indian Journal of Pharmacology (IJP) and Indian Journal of Physiology and Pharmacology (IJPP)) website. These articles were evaluated on the basis of appropriateness of descriptive statistics and inferential statistics. Descriptive statistics was evaluated on the basis of reporting of method of description and central tendencies. Inferential statistics was evaluated on the basis of fulfilling of assumption of statistical methods and appropriateness of statistical tests. Values are described as frequencies, percentage, and 95% confidence interval (CI) around the percentages.

Results:

Inappropriate descriptive statistics was observed in 150 (78.1%, 95% CI 71.7–83.3%) articles. Most common reason for this inappropriate descriptive statistics was use of mean ± SEM at the place of “mean (SD)” or “mean ± SD.” Most common statistical method used was one-way ANOVA (58.4%). Information regarding checking of assumption of statistical test was mentioned in only two articles. Inappropriate statistical test was observed in 61 (31.7%, 95% CI 25.6–38.6%) articles. Most common reason for inappropriate statistical test was the use of two group test for three or more groups.

Conclusion:

Articles published in two Indian pharmacology journals are not devoid of statistical errors.

INTRODUCTION

Statistics is a tool in the hand of a researcher by which he can analyze his study findings. If statistics methods used in the study are inappropriate, the conclusions drawn from the study become questionable. Studies with poor methodological quality and poor statistics cannot prove or disprove study hypothesis with certainty. So conduction of these kind of studies raises many ethical issues like exposure of participants to risk of new intervention, deprivation of participants to established treatment, unnecessary use of animals in experimental studies, misuse of resources, and wrong clinical judgments on the basis of these studies once they get published.[1–5] Despite publication of various guidelines related to the reporting of various methodological and statistical parameters of a study, it has been observed that quality of statistical reporting is poor in various biomedical journals.[6,7] Various surveys done for the articles published in western medical journals indicate that statistical error in the published article is a common phenomenon and error rate may vary from 30% to 90%.[8–11] Although many surveys are done for statistical reporting in western journals, data are lacking for studies published in Indian medical journals. Some small studies done for articles published in Indian medical journals observed the same phenomenon of poor reporting of various statistical parameters.[12,13] It is observed that data related to the statistical reporting of articles published in pharmacology journals of India are lacking. So this study was designed with the aim of evaluating articles published in Indian pharmacology journals (Indian Journal of Pharmacology (IJP) and Indian Journal of Physiology and Pharmacology (IJPP)) for statistical reporting. These two pharmacology journals are widely circulated and Pubmed-indexed Indian pharmacology journals; hence they were selected for evaluation.

MATERIALS AND METHODS

All articles published in IJP and IJPP between 2002 to the latest issue of 2010 were downloaded from journals website (www.ijp-online.com and www.ijpp.com). In case of IJPP, articles published since 2002 were available on website. So to maintain uniformity for both journals, all articles which were published in or after 2002 were downloaded. Only original studies were considered for analysis. Short communications, research letters, and letter to editors were not taken into account. In case of IJPP, only articles related to pharmacology were downloaded. All articles were evaluated independently by first (J.K.) and second author (P.Y.). These articles were appraised for quality of reporting of descriptive statistics and quality of reporting of inferential statistics. Descriptive statistics is evaluated on the basis of appropriate reporting of data as mean, median, or frequency with the central tendencies. Inferential statistics was evaluated on the basis of reporting of assumptions of statistical tests and inappropriateness of statistical tests. Common methods of statistical analysis were also noted. Common reasons for inappropriate descriptive statistics and common reasons for inappropriate statistical tests were also noted. Any disagreements between two authors were resolved by consensus (k = 0.87 for inappropriate statistical tests). Appropriate method of descriptive statistics of ratio and interval data following the normal distribution is mean (SD) or mean ± SD. For ordinal data and for ratio and interval data not following the normal distribution, appropriate descriptive statistics is median and interquarantile range and for nominal data, frequency and percentage are appropriate. Appropriate statistical tests are selected on the basis of aim of the study and types of data. Once the statistical test is selected, all the assumptions for that particular statistical test should be checked before applying that statistical test.

Statistics

Values are described as frequencies, percentages, and 95% confidence interval around percentages.

RESULTS

Total 196 articles from various areas of research were downloaded from the journal sites. Major areas of research were diabetes (39 (19.8%) studies), central nervous system (17 (8.6%)), hepatoprotection (18 (9.1%)), and cardiovascular (17 (8.6%)). Other areas were inflammation (11 (5.6%)), antioxidants (9 (4.5%)), pain (7 (3.5%)), gastrointestinal (6 (3%)), and immunomodulation (4 (2%)). Most of the articles were dealing with animal studies (83% vs. 17%).

Descriptive statistics

Out of these 196 articles, information related to descriptive statistics was missing in four articles. Out of remaining 192 articles, inappropriate descriptive statistics was reported in 150 (78.1%, 95% CI 71.7–83.3%) articles. Out of these 150 studies 106/129 (82.1%) were from IJP and 44/63 (69.8%) from IJPP.

Most common reason for inappropriate reporting of descriptive statistics was the use of mean ± SEM at the place of “mean (SD)” or “mean ± SD” [Table 1].

Table 1.

Inappropriate descriptive statistics in articles published in two Indian pharmacology journals (n = no. of articles)

Inferential statistics

Statistical methods

Most common type of statistical method used in the articles of both pharmacology journals was “one-way analysis of variance (ANOVA)” [Table 2]. Out of 214 statistical methods only 10.7% were nonparametric methods.

Table 2.

Statistical methods used in articles published in two Indian pharmacology journals

Assumptions of statistical tests

Information related to fulfillment of assumptions of statistical tests was mentioned in only two articles. In one article, normal distribution was checked by Komolgorov-Smirnov test.

Inappropriate statistical tests

Out of 196 articles from both journals, information related to statistical test was missing in four articles. Out of remaining 192 articles inappropriate statistical tests were found in 61 (31.7%, 95% CI 25.6–38.6%) articles. Most common reason for inappropriate statistical test was use of two group test for analysis of three or more than three groups (22.9%) [Table 3].

Table 3.

Inappropriate statistical tests in articles published in two Indian pharmacology journals (n = no. of articles)

DISCUSSION

Main findings of this study are as follows: majority of articles published in two Indian pharmacology journals have inappropriate reporting of descriptive statistics, assumption of statistical tests were checked in only two article, and inappropriate statistical tests was used to analyzed data in 31.7% of articles.

One major finding was inappropriate use of “mean ± SEM” for description of data. The ideal method of reporting of these kinds of data is “mean (SD)” or “mean ± SD.” Although SD and SEM look similar, they give different information.[14] Standard deviation (SD) shows variability around the mean within the sample and standard error of mean (SEM) shows probability of proximity of sample mean around the population mean.[15] Readers and researchers are interested in knowing variability within the sample not the proximity of mean to the population mean. The value of SEM is always less than SD so when it is used as descriptive statistics readers may falsely conclude that variability of sample is small. To prevent confusion with CI in the place of “mean ± SEM” reporting as mean (SD) is a better method.[16] Similar findings were also observed in other studies done for western and Indian journals. In a study done by Negele (2001) for the articles published in four anesthesia journals, it was observed that inappropriate use of SEM was present in 23% articles.[17] In a similar study done for four Indian medical Journals by Saurabh et al. (2010), it was observed that inappropriate reporting of SEM was common in articles published in basic science journals but this inappropriateness was negligible in journals related to clinical practices.[18] In spite of highlighting this issue in various surveys, this practice of reporting the variability as SEM is common and is a matter of concern.[16,19,20] Ordinal data like scores or scales are sometimes described as “mean ± SEM” which is wrong as they should be reported as median (range).[16] This error was not much observed in this study as majority of data were in ratio scale but in some other studies this error found to be much more.[21]

In this study, majority of statistical tests were parametric tests. Nonparametric tests were used less frequently (10.7%). It has been observed that the use of nonparametric statistics is increasing regularly in articles published in medical journals.[22] Low proportion of nonparametric statistics may be because of ignoring of assumptions underlying parametric statistics by authors.[23] Most of the articles in this study were animal experiments where usually many groups are used for comparison; hence one-way ANOVA was most frequently used statistical method whereas in studies done for articles published in clinical journals student t test seems to be the most common method.[24] Most of the statistical methods were simple methods and sophisticated methods like survival analysis, multiple regressions were not observed. In this study, it is found that three statistical tests – one-way ANOVA, unpaired t test, and paired t test – cover about 82% of all statistical methods, so these are the most frequently used tests and interpretation of these tests should be taught in detail to postgraduate students and young researchers.

In this study, it was observed that fulfilling of assumptions of statistical tests was not reported in almost all the studies. One reason may be underreporting and second reason may be ignorance of researcher. Each statistical test has some assumptions and these assumptions need to be fulfilled before application of that statistical test. Information regarding fulfilling of these assumptions should be included in the manuscript. Similar observation was made in other studies.[25]

About 32% articles have at least one inappropriate statistical test and most frequent mistake was the use of two group test for comparison of three or more groups like use of unpaired t test for comparison of three unpaired groups. This problem was observed in other studies done for statistical reporting in western journals.[23,25,26] Frequency of statistical errors varies from journal to journal like for Chinese journals it is 46%,[23] for surgical journal it is 64%,[27] and for urology journals it is 28%.[28] Most common problem was the use of multiple unpaired t tests at the place of one-way ANOVA. Despite repeated recommendations, unpaired t test still continues to be used at the place of ANOVA,[29] which is a matter of concern. Another mistake observed was the use of parametric statistical tests for ordinal data like scores or scales. It is very important to understand that ordinal data do not follow the normal distribution. Hence the use of parametric tests for these kinds of data is not justifiable.[30] In a study, it was found that ordinal data were used in about one-third of articles and these data are appropriately presented and analyzed in 50% articles.[31] In this study, most of the articles were dealing with continuous variables so this finding is not as prominent as observed in other journals.

There may be various reasons for finding these kinds of statistical errors in the published articles like insufficient knowledge of statistics and research methodology in researcher,[32,33] insufficient ethical review of protocol submitted for permission from institutional ethics committee, insufficient peer review of submitted manuscript, and lessknowledge of statistics in journal editors. It is observed that in ethics committee statistical issues are not discussed in detail as members of ethics committee usually focus their attention on informed consent, etc. It is important to understand that poor-quality research is also unethical. So ethics committee should also have a qualified medical statistician who can give advice regarding the methodological and statistical aspects of the protocol.[34] Every article submitted to the journal should also be sent for statistical review and journals should have statistical advisors in their editorial board. It is observed that many journals do not have statistical advisors.[35] Postgraduate students and young researchers should be trained in research methodology and biostatistics. Research methodology should be incorporated in the curriculum of postgraduate course.

This study has some limitations. One of the major limitations is that focus of this study is very narrow. Only few but very important statistical parameters were observed. Parameters like post hoc power, adjustment of multiple endpoints, sample size calculation, confidence interval, use of exact P value etc. were not taken into consideration. Second limitation is only two pharmacology journals were considered for evaluation. As far as our perception goes, this is the first study done for articles published in Indian pharmacology journals and may be at international level.

This study shows that inappropriate statistics is very common in the articles published in Indian pharmacology journals. Measures should be taken by journal editors, ethics committee, and researchers to prevent these errors.