Pilot studies: Are they appropriately reported?

Abstract

Introduction:

Pilot studies play a pivotal role in deciding whether a main study can be undertaken thereby helping in appropriate framing of time, cost and study methods. However, they cannot be employed for testing a hypothesis and are underpowered in detecting clinically significant differences between the treatment arms. Literature from the west has shown serious lacunae on the part of researchers in reporting pilot studies. The present study assessed the reporting quality of pilot studies published from India.

Materials and Methods:

All the journal articles with a pilot study design published in Indian Journals between January and December 2013 were identified through PubMed search and were assessed for the following: Reason for undertaking the pilot study; report about intention of further work; mention about sample size calculation; statement on other studies evaluating the same hypothesis published elsewhere; whether any hypothesis was tested in the present study; use of inferential statistics including the total number of statistical analyses performed and whether confidence intervals were reported; post-hoc power calculation; application of randomization and/or blinding; total number of study participants and presence of a control group.

Results:

A total of 93 articles were considered in the present study. None of these reported reasons for undertaking the present pilot study and intention to carry of carrying out further work depending on their results. Also, none of them discussed the feasibility of conducting such studies in the given set-up. A total of 69/93 (67.7%) studies tested a hypothesis and had employed at least one of the statistical tests to infer whether any significant difference exist between various groups. None of the 93 articles mentioned confidence intervals and calculation of the sample size despite all mentioning the presence of previous studies evaluating a similar hypothesis. Similarly, none of these studies mentioned post-hoc analysis of power and median (range) of times of statistical analyses performed includes 5 (0–57).

Conclusion:

Pilot studies have been poorly reported in Indian biomedical journals, and more attention is required from all the stakeholders of research; researchers, peer reviewers and journal editors

INTRODUCTION

Generation of good quality evidence requires well designed and accurately performed clinical studies. Feasibility of conducting such studies requires an a priori estimate of both time and cost. Pilot studies, which are performed ahead of the main study[1] help us to narrow down the feasibility of a study by formulating same/similar hypothesis, calculating the sample size required and identifying potential detriment and benefit to the study participants. However, they are also limited in being underpowered in establishing the treatment effects and not providing a reliable estimate of sample size calculation.[2] Even for generating the final premarketing evidence of a new drug (phase III), the earlier drug development trials (phase I and II) serve as pilot studies. Considering their importance, many investigator-initiated studies also undertake pilot studies on few patients followed up for a short period; however, these are perhaps frequently unreported. Also, it is not mandatory that the endpoints of the pilot and the main study be the same. Pilot studies frequently involve surrogate outcome analyses, shorter follow-up period of the study patients and are underpowered. Hence, their results should always be interpreted with caution as they may not be clinically significant or appropriate. Even a clinical model has been developed to identify cancer chemo preventive agents for women with breast cancer from surrogate end points of a pilot study, to identify feasibility of performing phase II studies.[3] One has to understand that the pilot study just helps in learning the feasibility of conducting the main study, and the results are not confirmatory. Studies from the western world have reported poor quality of reporting pilot studies[4] following, which authors have drafted a 17 items checklist for reporting pilot studies.[5] Considering the lacunae of such data from Indian context, we evaluated the reporting quality of pilot studies published in a leading biomedical research journal from India.

MATERIALS AND METHODS

All the journal articles with a pilot study design published in Indian Journals between January and December 2013 were identified through PubMed search. The search strategy used was as follows - “pilot (tiab) AND India (PL) AND 2013 (All Fields)”. All types of articles (original/short communication/correspondence) were evaluated in the present study. The articles were assessed for the following: Reason for undertaking the pilot study; report of intention on further work; mention about sample size calculation; statement on other studies evaluating the same hypothesis published elsewhere; whether any hypothesis was tested in the present study; use of inferential statistics including the total number of statistical analyses performed and whether confidence intervals were reported; post-hoc power calculation; application of randomization and/or blinding; total number of study participants and presence of a control group. The studies were classified into following fields: Allopathic medicine, dentistry and complementary and alternative systems of medicine (CAM). Descriptive statistics was used.

RESULTS

Demographic details

The search revealed a total of 152 articles of which 93 articles were found to be relevant for the assessment in the present study (51 articles did not belong to biomedical science; 8-in vitro/animal studies). Majority of these (71/93, 76.3%) belonged to allopathic medicine, 13/93 (14%) were from dentistry and 9/13 (9.7%) were from CAM. Of the 93 studies, 64 (68.8%) were observational (62-cohort; 1-cross-sectional and 1-case control) and 29 (31.2%) were interventional (20-nonrandomized and 9-randomized) leading to a total of 73 (78.5%) studies with a control group.

Reporting quality

None of the identified studies reported the reason for undertaking the present pilot study and then a further main study depending on their results. Also, none of them discussed the feasibility of conducting such studies in the given set-up. The study size ranged between 4 and 600 with a median of 30. A total of 69/93 (67.7%) (allopathic-50/71 [70.4%]; dentistry-11/13 [84.6%]; and CAM-8/9 [88.9%]) studies tested a hypothesis and had employed at least one of the statistical tests to infer whether any significant difference exists between various groups. None of the 93 articles mentioned confidence intervals and calculation of the sample size despite all mentioning the presence of previous studies evaluating a similar hypothesis. Similarly, none of these studies mentioned post-hoc analysis of the power and median (range) of times of statistical analyses performed includes 5 (0–57) (allopathic medicine-5 [0–57]; dentistry-4 [0–51]; and CAM-10 [0–18]). Additionally, a pilot study was published in the field of allopathic medicine with a total sample size of only 42 study participants (21 in each of the two groups) but had applied a multivariate analysis to find out the association of 12 factors with the outcomes.

DISCUSSION

The present study assessed the quality of reporting pilot studies, published in biomedical research from India over a period of 1-year. We found major lacunae in terms of testing hypothesis, not mentioning the confidence intervals and sample size when inferential statistics was used irrespective of the field of medicine.

The main goal of pilot studies is to assess the feasibility of the main study, so that disastrous consequences both in terms of time and money shall be avoided in the main study. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention. Additionally, they also enhance the knowledge and confirm the competent skills of researchers/research team with a reasonable certainty. Also, presentation of results of pilot studies enhances the potential to obtain grants from funding agencies during the main study. Although they are fundamental before planning a major study, many researchers/journals/authors have always ignored them. One has to understand that a pilot study is not a hypothesis testing study where safety, efficacy or effectiveness can be assessed.[6] The pilot studies are underpowered in detecting any significant difference between the groups unless the sample size is based on an appropriate calculation. Pilot studies give information about the feasibility and modification required (if any) in the following larger, hypothesis testing main study. In the present study, we found that two-thirds of the pilot studies tested the hypothesis and inferred the significance of differences between the groups, but none of these have mentioned about sample size calculation. This is similar to the previous study where 81% of the pilot studies reported hypothesis testing and use of inferential statistics.[4] In fact, hypothesis testing and types of errors have to be differently interpreted in case of pilot studies. The null hypothesis for a pilot study will be that a definitive main study need not be performed and so type 1 error will be that a main study be falsely undertaken.[7] Also, post-hoc power analysis may throw light on the importance of statistical difference reported in a pilot study where an inferential statistic is used, and we found none of the articles reported the same. Nearly four-fifths of the studies had control group, but all of them did assess the significance of the differences in the outcome measures between groups. The purpose of having comparator groups in a pilot trial is to give clearer picture of issues related to recruitment, randomization, administration of interventions, evaluation of blinding procedures, and retention, when they are supposed to be there in the main study. However, it is an error to evaluate the significance of the differences between the groups in a pilot study without adequate power. In the absence of any previous estimate of the primary objective or in case when the expected results may vary hugely from what has been reported earlier, a pilot study with the least possible sample size may be initiated. The final estimate that is arrived at the end of such pilot studies may be used to calculate the sample size required for the main study. Despite the presence of previous studies, none of the pilot studies in the present study evaluating the hypothesis (10/13) calculated sample size; rather they have used inferential statistics. Further adding to the problem, multiple statistical analyses were performed without even adjusting the P values, and none of the studies did mention confidence intervals that are important to estimate the precision of study results. Even in a main study, the problems of using multiple statistical tests are well known, and a lot of methods have been proposed to correct the significant P value.[8] When it is so, authors of a pilot study with a small sample size should definitely restrict the use of a number of inferential statistics. We found that none of the authors of the pilot studies reported the intention of further work while Lancaster et al.[9] in 2001 reported the same in 50% and Arain et al.[4] in 2010 found it in 8% of the pilot studies. The results of a pilot study shall standalone without getting incorporated into the main study, which is referred to as “external pilot” and “internal pilot” studies are those where the pilot study population is included in the main study.[10] Also, it is important to mention confidence intervals in a study with a priori calculation of sample size. The significance is intuitive in a pilot study that indicates the precision of obtained estimates.[11] In fact, recently, Lee et al.[7] have shown the impact of various widths (75%, 85% and 95%) of confidence intervals in different pilot studies. Further reporting of confidence intervals, description of its relation to minimum clinically important difference is a must in pilot studies for the readers to correctly interpret the study results.

It is important that journals should have guidelines for reviewing/publication of pilot studies. While only 1/7 (14.2%) of the journals (new England Journal of Medicine, British Journal of Medicine, Journal of American Medical Association, British Journal of Surgery, British Journal of Cardiology, British Journal of Obstetrics and Gynecology and Lancet)[4] have been shown to possess publishing policy for pilot studies, none of the Indian journals associated with a professional body (Indian Journal of Pharmacology, Indian Journal of Neurology, Indian Journal of Ophthalmology, Journal of Postgraduate Medicine, Indian Journal of Surgery, Indian Journal of Obstetrics and Gynecology) instructions to authors/guideline for contributors did not reveal the report of any statements related to consideration of pilot studies. The editorial board should critically assess the statistical reporting in pilot studies, and it is preferred to have written guidelines for the same; earlier studies have found such guidelines existing in only around one-tenth of the Indian journals.[12] It is the need of the hour that Indian journals should adapt reviewing and publishing policy for pilot studies. A study should not be called as a pilot just because of the small sample size or lack of rigorous methodological evaluation.

CONCLUSION

Pilot studies have been poorly reported in Indian Biomedical Journals and more attention is required from all the stakeholders of research; researchers, peer reviewers and journal editors.