A Brief Overview of Pilot Studies and Their Sample Size Justification

Précis

Pilot studies, if properly designed and implemented, are an important tool that provide critical information for the development and potential success of a subsequent, larger trial. In fact, these small-scale studies are commonly used to assess the feasibility of whether a larger trial should be initiated. A popular investigator question is whether a pilot study requires a formal statistical power calculation. In general, the answer is “no”; however, the sample size needs to be justified based on the goal of the pilot study.

Introduction

Full-scale, hypothesis-driven, confirmatory trials can be very resource intensive and expensive to conduct and may not ultimately be successful (either positively or negatively) due to improper planning. To help ensure the successful conduct of the main, full-scale trial a pilot study may first be undertaken. The aims and methods of a pilot study should align with the goals of the subsequent main trial. Moore et al. define pilot studies as “preparatory studies designed to test the performance characteristics and capabilities of study designs, measures, procedures, recruitment criteria, and operational strategies that are under consideration for use in a subsequent, often larger, study.”¹ The terms pilot study and feasibility study often have been used interchangeably.² However, some advocate that feasibility studies are used to estimate important parameters needed to design the future main trial (e.g., variability of outcome measure, recruitment rates, willingness of patients to be randomized, subject retention rates); whereas, pilot studies are miniature versions of the main trials focused on whether the processes of the main trial are working well (e.g., randomization procedures, training of staff).^3,4 A pilot study can be either an internal study (i.e., the pilot data are included with the main trial study data) or an external study (i.e., the pilot data are independently assessed and not included in the main trial). Internal pilot studies usually incorporate an interim analysis at the end of the “pilot” stage, are often used to assess initial sample size assumptions in order to possibly re-estimate the sample size for the trial, and have proper accounting for the inflated type I error.⁵ Internal pilot studies must be thoroughly planned at the design stage of the main study as the pilot data are part of the main study itself. For the purpose of this brief overview, only external pilot studies and feasibility will be discussed.

Reasons for Conducting Pilot Studies

Before stating some valid reasons to perform a pilot study, let’s address a common improper justification many investigators use to conduct a pilot study. This elephant in the room is lack of funding. The lack of funding and resources is not justification in and of itself to conduct a small-scale study and simply label it as a “pilot” study. Matching the sample size for the study to the availability of funds without proper regard to goals is not sufficient justification.¹ Though lack of funding is definitely an issue, a proper pilot study still needs to have clear goals, details of how these goals will be assessed, justification for the sample size given the goals, and proper methodological rigor for how the pilot study will be conducted.

Thabane et al. breakdown the reasons to perform a pilot study into four broad classifications.⁶ The first classification is “process” which assesses the feasibility of the steps and processes for future success of the main trial. Some process examples include recruitment rates, participant retention rates, protocol compliance/adherence rates, assessment of eligibility criteria, assessing the order of procedures to be performed at a clinical visit, tracking shipments of biologic material from the clinic to an external laboratory, identifying issues resulting in missing data, and assessment of data collection forms and tools for understandability and completeness. The second classification is “resources” which assesses time and budget concerns. Some resource examples include the time required for participants to complete study forms, the cost and time to mail surveys, the time required to complete the patient visit, equipment availability and cost to use it, plans for when equipment breaks or is lost/stolen, and the investigators’ availability and time to perform all required tasks. The third classification is “management” which assesses staffing and data management concerns. Some management examples are potential issues entering data, matching data from different sources properly and effectively, concerns from personnel running the study, and assessing staffing capacity to perform the required tasks. The fourth classification is “scientific” which focuses on the impact of the intervention/treatment. A scientific example is determining an estimate of the intervention effect and variability for the primary outcome in order to inform the main trial.

Word of Caution for Powering the Main Trial

If estimation of intervention effect size and/or variability of measures are the goal of the pilot study in order to inform the sample size of the subsequent main trial, some caution must first be mentioned. It is advised not to use the actual effect size from the pilot study as the effect size for the main trial because the estimated effect size from the pilot’s small sample size is very likely overestimated and may mislead.⁷ In short, over-estimation of the intervention effect size for the main trial will result in an under-powered study because the true effect size is smaller than that observed in the pilot study. If possible, the main trial’s intervention effect size should be based on the minimally clinical important difference (MCID) for the outcome of interest. The MCID is the smallest intervention effect that would have a meaningful impact for the patient. Furthermore, the MCID is typically smaller than the observed pilot study intervention effect size which is based on a small number of participants. Determination of the MCID, however, is beyond the scope of this brief overview. Similarly, it is advised not to use the actual variability (i.e., standard deviation for a continuous outcome) from the pilot study as the variability measure for the main trial because it has been shown to often be under-estimated resulting in an underpowered main trial.⁸ Browne has suggested using the upper limit of the 80% confidence interval (CI) for the standard deviation from the pilot study to perform the power calculation of the main trial.⁸ Additionally, sensitivity analyses of the assumptions for the main trial could be performed to account for the uncertainty in both the effect size and variability. In other words, for the main trial perform a range of sample sizes based on plausible estimates informed by the pilot study.

Sample Size Justification

Pilot studies are not confirmatory trials and thus should not be hypothesis-driven (e.g., p-value centric). The analysis of a pilot study should be focused on descriptive statistics (e.g., means, standard deviations, and quantiles for continuous variables and frequencies and percentages for categorical variables) and estimates of precision (i.e., width of CI).⁵ Because pilot studies are not hypothesis-driven, there is no need for a formal statistical power calculation. However, justification of the sample size for a pilot study is still required and should be based on the primary goal of the study. Although strongly not advised, if any hypothesis tests are conducted in the pilot trial the results should be interpreted with extreme caution and a “significant” result should not stop proceeding to a larger confirmatory trial.⁵

An estimate of precision using a confidence interval approach, particularly a one-sided CI, is an ideal way to justify the sample size.^8,9,10 Based on the actual pilot goal such as feasibility, some estimates of precision are the 95% CI of the mean if the outcome of interest is continuous (e.g., time to complete a survey) or the 95% CI for a one-sample proportion if the outcome of interest is binary (e.g., protocol adherence rate). As an example, let’s assume the primary goal of a 6-month longitudinal pilot study is feasibility based on the retention of the participants at the end of 6 months. We anticipate the retention rate will be 90% and want the margin of error to be 5%, i.e., we are willing to accept a retention rate as low as 85% to consider the feasibility trial a success and move on to the main trial. As this is a binary endpoint, justifying the sample size using the 95% CI for a one-sample proportion $\left(p\pm 1.96\sqrt{\frac{p(1-p)}{n}}\right)$, where p is the estimate of the proportion and n is the sample size, is appropriate. In order to successfully assess this feasibility retention goal for this example, a pilot study sample size of 36 participants would be required to achieve a 95% CI of (0.85, 0.95) for a one-sample proportion.

Another approach to justify the sample size is to use and reference one of the multiple pilot study sample size flat rules of thumb proposed.^8,9,10,11,12 A flat rule of thumb is a single number that is suggested for the pilot study for all scenarios. For a binary outcome, such as an event rate, from 60 to 100 subjects per group should be sufficient for the pilot study.¹¹ For a continuous outcome, a sample size of anywhere from 12 per group to 35 per group have been proposed.^8,9,10,11,12 The larger the sample size for the pilot, the more precision the estimates will have for planning the main trial. It should be noted, however, that none of these flat rules of thumb take into consideration the size of the future main trial.

If the main trial’s target effect size is known a priori, stepped rules of thumb have been proposed which provide much more granularity for pilot study sample size justification.^13,14 These proposed stepped rules of thumb are based on the standardized effect size, also known as Cohen’s d, for the main trial. For a two independent arm trial, the standardized effect size is the difference in means between the two arms divided by the pooled standard deviation. Table 1 reproduces, with some slight edits, the stepped rules of thumb summarized by Bell et al.¹⁴ As an example using Table 1, suppose a pilot study is being planned to support moving forward to a main trial. The main trial will be a two-arm, randomized, controlled trial with an active drug arm and a placebo arm. The investigators know that for the main trial, based on expert opinion, a minimally important clinical difference for the primary outcome is 10 units and that from literature review the standard deviation could be reasonably estimated as 18 units resulting in a standardized effect size for the main trial of 10/18=0.56. The value of 0.56 is considered a medium standardized effect size. Using Table 1, for a 90% powered main trial, the pilot study sample size should be 15 per arm and the subsequent main trial will require somewhere between 44 and 235 participants per arm.

Table 1.

Pilot study sample size per arm for a continuous outcome using stepped rules of thumb as a function of the main trial’s target standardized effect size.^13,14

	80% power for main trial		90% power for main trial
Standardized Effect Size (d)*	Pilot study sample size per arm	Main trial sample size per arm	Pilot study sample size per arm	Main trial sample size per arm
Extra small (d<0.1)	50	>1571	75	>2103
Small (0.1≤ d<0.3)	20	176—1571	25	235—2103
Medium (0.3≤ d<0.7)	10	34—176	15	44—235
Large (d≥0.7)	10	≤34	10	≤44

^*Standardized effect size: $d=\frac{\left({\overline{x}}_1-{\overline{x}}_2\right)}{S{D}_{\mathit{\text{pooled}}}}$, where ${\overline{x}}_1=\text{ mean of arm }1$, ${\overline{x}}_2=\text{ mean of arm }2$, and the pooled standard deviation: $S{D}_{\mathit{\text{pooled }}}=\sqrt{\frac{\left(n_1-1\right)s_1^2+\left(n_2-1\right)s_2^2}{n_1+n_2-2}}$, where n₁ = number in arm 1, n₂ = number in arm 2, $s_1^2=\text{ variance in arm }1$, and $s_2^2=\text{ variance in arm }2$.

Summary

A pilot study is an important tool to assess the processes to be used in the main trial, identify problematic issues and determine if they can be corrected, and to determine the feasibility of conducting a main trial. Clear goals of the pilot study should be stated a priori with details as to what will be considered a successful pilot (e.g., feasibility success criteria). Analysis of pilot studies should be based on descriptive statistics and precision of estimates (CIs) and ideally not involve any inferential statistics (i.e., p-values). As the pilot study is not conducted for the purpose of inference, a power calculation is not needed. However, the sample size needs to be justified based on the goal of the pilot study.