Introduction of a pilot study

Abstract

A pilot study asks whether something can be done, should the researchers proceed with it, and if so, how. However, a pilot study also has a specific design feature; it is conducted on a smaller scale than the main or full-scale study. In other words, the pilot study is important for improvement of the quality and efficiency of the main study. In addition, it is conducted in order to assess the safety of treatment or interventions and recruitment potentials, examine the randomization and blinding process, increase the researchers' experience with the study methods or medicine and interventions, and provide estimates for sample size calculation. This review discusses with a focus on the misconceptions and the ethical aspect of a pilot study. Additionally how to interpret the results of a pilot study is also introduced in this review.

Introduction

To obtain high-quality outcomes, a good research study with relevant experimental design and accurate performance is required. Analyzing its feasibility prior to performing the main study (also known as the full study or large-scale main trial) can be very beneficial for this purpose. A pilot study is the first step of the entire research protocol and is often a smaller-sized study assisting in planning and modification of the main study [1,2]. More specifically, in large-scale clinical studies, the pilot or small-scale study often precedes the main trial to analyze its validity. Before a pilot study begins, researchers must fully understand not only the clear purpose and question of the study, but also the experimental methods and schedule. Researchers become aware of the procedures involved in the main study through the pilot study, which aids in the selection of the research method most suitable for answering the research question in the main trial. Despite the benefits and importance of the pilot study, researchers often are not interested.

A pilot study is performed either as an external pilot study independent of the main study or as an internal pilot study included in the research design of the main study. This article describes the core items of an external pilot study and misconceptions and ethical aspects of a pilot study and introduces the appropriate method for reporting the outcomes of the pilot study.

Objectives of a Pilot Study

Feasibility of the study protocol

A pilot study is performed reflecting all the procedures of the main study and validates the feasibility of the study by assessing the inclusion and exclusion criteria of the participants, preparation of the drugs and intervention, storage and testing of the instruments used for measurements in the study, as well as training of researchers and research assistants [3]. The researcher, as well as the research assistants, must fully understand the purpose, method, and procedures of the study [3,4,5]. In addition, the suitability of the method for data collection must be tested. Let us review the study by Youn and Hsu [6], where the authors compared the methods for pain reduction with a propofol injection. In the pilot study, the following four methods were tested in order to find the most appropriate method for the main trial: Group T received an injection of 0.5 mg/kg of 1% lidocaine after applying a tourniquet; Group H received an injection of 0.5 mg/kg of 1% lidocaine after the injection of 40–41℃ saline solution (200 ml); Group L received an injection of 0.5 mg/kg of 1% lidocaine after an injection of room temperature (23–24℃) saline solution (200 ml); and Group HL received an injection of 0.5 mg/kg of 1% lidocaine at 41℃.

Based on the verbal pain scores assessed after the injection of 0.5 mg/kg of propofol in each group, the final injection method was selected. In addition, changes in body temperature, based on the saline injection, and differences depending on body part used for the measurement of body temperature were also assessed.

Randomization and blinding

A pilot study assesses if the randomization and blinding are appropriately executed [7,8,9]. For example, to appropriately use the sealed opaque envelopes method, which is often used in the clinical trials, the detailed procedures for preparation, storage, and delivery are assessed [10]. Papers with randomized numbers are put into an opaque envelope and organized based on the order of the assignments. These envelopes are stored in the pharmacy department and provided by the department when needed. At the time of the provision, signatures from both the recipient and the provider are required. The study drugs are provided in an equal volume in identical syringes. Neither the patient nor the researcher has information on the study drug to which they are assigned. The appropriate drug that has been prepared in advance is used based on the randomized number in the envelope obtained from the pharmacy department.

Among the randomized groups, uniformity in the demographic characteristics and appropriate blinding, based on the researcher's plans, and the participants' understanding of randomization can be assessed. In addition, the most appropriate method used to explain randomization and obtain consent from the patients can be assessed. In the Benger et al. [3] study that compared three different methods to secure the airway of patients with out-of-hospital cardiac arrest (i-gel and Laryngeal Mask Airway Supreme versus current tracheal intubation), a cluster randomized design was used1), not based on the patient, but based on the paramedic. If randomization is performed based on the patient, the few following limitations need to be considered: the paramedic has to prepare and use all devices; it is difficult to perform randomization in an emergency situation; and there is a great chance that the device assigned to the patient would not actually be used.

Recruitment and consent

The researcher recruits the subjects and obtains consent for participation. Adequate information and time should be provided for the participants to make their decision and provide written consent. Thereafter, participants should be screened in order to confirm their suitability for the trial. The appropriateness of the consent form, recruitment rates, length of time to receive written consent, and the required number of researchers and research assistants is determined. In particular, the recruitment rate is directly related to the study period (duration) and success or failure. An insufficient number of participants results in lower statistical power, which can eventually lead to early termination of the trial in the worst-case scenario. Therefore, it is crucial to accurately identify the recruitment rate through a pilot study. Another solution to this problem is to set the statistical power at a higher level. Instead of using 80%, the minimum power needed in clinical studies, 90% power can result in a higher statistical power despite a low recruitment rate [11].

Recruitment rate can also be increased through modification of the experimental methods. In their study of urological pediatric patients, Vemulakonda and Jones [12] concluded that an observational cohort study, where caregivers were involved in deciding the treatment method, had higher recruitment rates than that of randomized clinical trials. In the study that utilized video clips with comic characters in order to reduce anxiety in pediatric patients, the interview method was changed, which that increased the retention rate from 20% to 72.5% [13].

Acceptability of intervention

Although the study drug or intervention may be significant and would be worth a try, whether the participants can accept the study drug or intervention is a separate issue [14]. It would be easier if the approved drugs or intervention were accepted for use, but difficult or new approaches or known side-effects of the drug or intervention can make it difficult for the patients to accept the treatment. Chow et al. [13] used either 1) the usual care providing information on anesthesia and operation room procedures to both the pediatric patients and their families on the day of the procedure or 2) story-telling medicine through a video clip that was approximately 20 minutes long with animation characters played on a tablet PC. Through this pilot study, feasibility and acceptability of the general experimental plan, as well as the potential for positive effects of story-telling medicine, have been assessed. The researchers suggested that story-telling medicine can be a possible and acceptable method for effectively reducing anxiety in pediatric patients. The results were used as the foundation to proceed with a full-scale randomized controlled study.

Selection of the most appropriate primary outcome measure

It is not easy to select the primary outcome that best reflects the intentions of the researchers. Furthermore, the primary outcome is directly related to the sample size calculation. If several primary outcomes are required, a sample size for each outcome is needed. Mouton et al. [5] performed a pilot study to assess if remote ischemic preconditioning can prevent organ damage in the patients undergoing abdominal aortic aneurysm repair and suggested cardiac events and renal injuries as primary outcomes.

Sample size calculation

One of the key reasons why a pilot study is needed is to obtain the required preliminary data for the calculation of a sample size for the primary outcome. For continuous outcomes, preliminary data such as the mean and standard deviations for the control group are needed. For categorical outcomes, preliminary data such as the success rate of the standard treatment are required [14]. When selecting more than one primary outcome, the preliminary data for each outcome needs to be obtained in order to calculate the sample size. If the sample size based on preliminary data varies, the largest sample size required is used as the sample size of the main trial. This is to maintain the statistical power for the primary outcome that requires the largest sample size while increasing the power for the remaining primary outcomes.

Common Misconceptions

Despite the fact that pilot studies are very useful, not many are reported. One of the key reasons is that results from these studies focus on statistical outcome rather than the feasibility of the study. Furthermore, the experimental design itself is not clear [2,15].

Effect size and sample size estimation

Many studies have a preceding pilot study in order to calculate the sample size. However, estimation of sample size required for the main trial needs to be performed with caution [2,14,16]. In order to determine the sample size of the main trial, the standardized effect size (i.e., Cohen's d) is required. Unfortunately, the standardized effect size calculated in the pilot study is an estimated value calculated from the sample and has a confidence interval. Due to an insufficient sample size, the confidence interval of the standardized effect size is extremely wide, and the corresponding sample size also has a wide range [16]. As a result, this could lead to errors in calculating the sample size or statistical power to be used in the main trial [17,18]. One of the ways to overcome this issue is to utilize the clinically meaningful difference. For this, the experience of the researcher is critical. For instance, there is a study comparing the mean arterial pressure after intubation. Assuming that the standard deviation of the mean arterial pressure measured after intubation in the treatment and control groups is 20 mmHg, the sample size varies depending on the set difference in the average arterial pressure between the two groups. If the difference is set at 5, 10, or 20 mmHg, the corresponding sample sizes are 253, 64, and 17, respectively. The value to be considered as the clinically meaningful value depends on the experience of the researcher.

Internal pilot study

Researchers have a strong desire to include the data collected from the pilot study into the main study because this allows the researchers to reduce both the number of participants required for the study and the duration of the study. However, this is only allowed in an internal pilot study that is not discussed in this text [1]. To perform an internal pilot study, it must be thoroughly planned at the study design stage of the main study and included in the study procedure. Furthermore, the researchers must consider the fact that changes in other categories associated with the main study, aside from calculating the sample size, cannot be made [1]. In addition, a slightly increased chance of a type 1 error due to the hypothesis stating that pilot study and main study are independent of each other is an important aspect to consider.

Analysis of a pilot study

There is the question of whether the hypothesis can be tested in the analysis of a pilot study. Considering that the appropriate power and sample size were not calculated for the pilot study, the researchers must recognize the fact that pilot studies are not for testing the hypothesis testing [7,19]. Therefore, they must be cautious about reporting the results of a pilot study. Furthermore, statistical significance in a pilot study does not mean that the main study or trial is not required.

Sample size for pilot studies

The primary purpose of pilot studies is not hypothesis testing and therefore sample size is often not calculated. Some studies recommend over 30 samples per group [20] while some suggest 12 per group [21]. An appropriate sample size needs to be determined, not for providing appropriate power for hypothesis testing, but to understand the feasibility of participant recruitment or study design. For instance, in the previously mentioned study of securing the airway in patients with out-of-hospital cardiac arrest, 30 paramedics who performed airway securing procedures at least twice a year were included in each group. Not all paramedics experience events where they need to secure the airway and applying the Poisson distribution predicted that roughly 17% of the paramedics experience 0 to 1 case. Therefore, this was taken into consideration when calculating the number of samples to be included. Furthermore, after considering around 25% dropout rate throughout the study, a total of 50 subjects per group was recruited [3]. An important point is that a sample in the pilot study needs to be identical to that of the main study; therefore, the inclusion and exclusion criteria should be identical [2].

Ethical Aspects

Considering that studies with inadequate statistical power are unethical [22], performing pilot studies without secured feasibility may be considered unethical as well. However, there are no descriptions about pilot studies even in the Good Clinical Practice guideline2). An important ethical point to consider, however, is to clearly explain the characteristics of a pilot study to the participants. In other words, the participants must be notified that based on the results of the pilot study, the main study may not be performed [2].

How to Interpret the Results of a Pilot Study

Conditions for a successful pilot study must be listed in advance. Depending on the fulfillment of these conditions, the researcher decides to proceed with the main study or to make modifications to the study design. Furthermore, results from the pilot study are described based on these conditions. Typical results from a pilot study can be described as one of the following [2]: 1) termination of the study (cannot proceed with the main study); 2) can proceed with the main study after modifying the study design; 3) not necessary to modify the study design, but requires thorough monitoring throughout the study procedures; or 4) can proceed without modifying the study design.

Thabane et al. [2] provided a checklist for pilot studies using the CONSORT statement. A brief description is provided below:

1. Must note that the study is a “pilot study” in the title.

2. In the introduction, background for the main study and rationale for performing the pilot study should be written.

3. In the methods section, categories for assessing the validity of the criteria and procedures to be applied in the main study should be defined, and the criteria to determine validity should be established. Inclusion and exclusion criteria of the participants, detailed administration and treatment method, definitions of the primary and secondary outcomes, method and reasoning behind the determination of the sample size, and methods for appropriate statistical analysis should be written.

4. In the results section, the validity of the described points in the methods section and points to be modified are described and solutions are sought. Moreover, description of the baseline data and recruitment status of the participants is included. Information on the primary and secondary outcomes, such as the mean, standard deviation, 95% confidence interval, probabilities, etc., are also reported.

5. Discussion should be focused on determining whether or not the main study is feasible. Previously listed items and standards are summarized. Possible biases or experimental problems that can occur in the main study are listed.

6. Lastly, whether or not the main study is feasible based on the pilot study is determined and elaborated.

Summary

A pilot study provides necessary information not only for calculating the sample size, but also for assessment of all other aspects of the main study, minimizing unnecessary effort from the researchers and participants, as well as the dissipation of research resources. In order for the pilot study to play its role, factors introduced in the text must be clearly defined before proceeding with the pilot study, and demonstrate a high level of completion. Furthermore, a pilot study provides valuable information, not only for the researcher's main study, but also for other similar studies; therefore, it is crucial to include complete information on the feasibility of the study.