Application of sample size re-estimation in clinical trials: A systematic review

Abstract

Background

Sample size re-estimation (SSR) is a method used to recalculate sample size during clinical trial conduct to address a lack of adequate information and can have a significant impact on study size, duration, resources, and cost. Few studies to date have summarized the conditions and circumstances under which SSR is applied. We therefore performed a systematic review of the literature related to SSR to better understand its application in clinical trial settings.

Methods

PubMed was used as the primary search source, supplemented with information from ClinicalTrials.gov where necessary details were lacking from PubMed. A systematic review was performed according to a pre-specified search strategy to identify clinical trials using SSR. Features of SSR, such as study phase and study start year, were summarized.

Results

In total, 253 publications met the pre-specified search criteria and 27 clinical trials were subsequently determined as relevant in SSR usage. Among trials where the study phase was provided, 2 (7.4%) trials were Phase I, 5 (18.5%) trials were Phase II, 11 (40.7%) trials were Phase III, and 2 (7.4%) trials were Phase IV.

Conclusion

Our results showed that SSR is also used in Phase I and II, which involve earlier decision making. We expect that SSR will continue to be used in early-phase trials where sufficient prior information may not be available. Furthermore, no major trends were observed in relation to therapy area or type of SSR, meaning that SSR may become a feasible and widely applied method in the future.

1. Introduction

Clinical trials and research on drug or treatment development should include sample size calculations at the time of study planning, and details of these calculations should be specified in the trial protocol. If there is insufficient information on the components required to calculate the sample size at the time of study planning, the calculation should be performed using only the information available at the time of planning the study. Sample size re-estimation (SSR) has been proposed to address a lack of adequate information to calculate sample size, such as insufficient knowledge of the efficacy/safety profile of the drug or treatment, changes in the therapy area, or a lack of prior information from planning validation studies [1]. SSR is divided approximately into two types: blinded SSR (bSSR; performed without using information on the treatment groups) and unblinded SSR (ubSSR; performed using information on the treatment groups). While bSSR re-estimates the sample size based on the nuisance parameter, ubSSR uses the estimate of differences between treatment groups. For using ubSSR, it has been said that there are concerns that Type I error is inflated and operational bias, which are caused by unblinding. Although the method to preserve Type I error has been proposed, the concern about operational bias cannot totally be resolved [2,3]. On the other hand, for bSSR, since the blinding is maintained, there is no risk about operational bias caused by unblinding, but it has been suggested that Type I error would be inflated, and the method to resolve it has been proposed [4,5]. When using SSR, it is necessary to consider other issues in advance such as the overall development plan of the investigation drug or treatment, key parameters among the missing prior information, regulatory requirements, expected treatment efficiency, resource availability, feasibility, and/or avoidance of back calculation. It has been reported that SSR can be applied to studies such as early stopping based on interim analysis and seamless design. In addition, SSR also bring challenges to clinical operations, including randomizations, blinding, drug supply, and site activations. These operational challenges should be addressed by preplanning and clear communication between different functions [6].

ICH-E9 states that “when event rates are lower than anticipated or variability is larger than expected, methods for sample size re-estimation are available without unblinding data or making treatment comparisons” and “in long term trials there will usually be an opportunity to check the assumptions which underlay the original design and sample size calculations. This may be particularly important if the trial specifications have been made on preliminary and/or uncertain information.” [7]. Regulatory agencies such as the United States Food and Drug Administration (FDA) and European Medicines Agency (EMA) have issued guidance on adaptive design (AD) including SSR [[8], [9], [10]]. Regulatory concerns include the rationale for using SSR, control of Type 1 error, trial integrity, and outcomes before and after using SSR [11]. Consequently, the clinical trial sponsor has a number of factors to consider regarding SSR application [12]. The use of AD, including SSR, can be beneficial in many circumstances, but the benefit may not be applicable in all cases, meaning that factors such as the statistical impact and the trial cost need to be considered [13].

Although several reports to date have evaluated the application of SSR, few have summarized the actual clinical application of SSR [14]. Although several reports on AD are available [12,[15], [16], [17], [18]], the number of reports focusing on SSR is limited. Two surveys on AD have recently been reported. The Drug Information Association Adaptive Designs Scientific Working Group (DIA-ADSWG) have reported survey results on AD in 2012 and 2016 [15,16]. The 2016 survey involved sending questionnaires to 114 organizations (pharmaceutical companies, biotechnology companies, clinical research organizations, and academic institutes) via email, statistical and clinical journals, and clinical trial registration sites (TrialTrove). The survey included only trials with pre-planned ADs or ADs described in the statistical analysis plan before the unblinded data review was conducted, and with pre-specified adaptations. Twenty-eight organizations responded to the survey, of which 25 had experience in planning AD. Of 1158 trials conducted by these 25 organizations, 57% (654 trials) included group sequential design (GSD), bSSR, or early stopping. In addition, 19% (98 trials) of the remaining 504 trials that conducted AD other than GSD/bSSR/early stopping (ubSSR only: 74 trials; ubSSR + other adaptive: 24 trials) had planned to include ubSSR. From the literature search, out of 4085 studies published in statistical journals and 25854 studies published in clinical journals, 224 statistical studies and 58 clinical studies included the keyword AD. Of these, 10% (22 trials) used bSSR in statistical journals and 9% (5 trials) used bSSR in clinical journals, while 23% (51 trials) used ubSSR in statistical journals and 3% (2 trials) used ubSSR in clinical journals. The results from the questionnaire showed that use of SSR was not widely reported, although bSSR was reported in both statistical and clinical journals and ubSSR tended to be reported less frequently in clinical journals. Bothwell et al. conducted a survey of trials registered in EMBASE, PubMed, Cochrane Registry of Controlled Clinical Trials, and Web of Science as of 2014 and ClinicalTrials.gov (CTG) as of 2015 [17]. The included trials were those conducted in human subjects, systematic reviews, meta-analyses, conference abstracts, commentaries, editorials, statistical methods, or economics for completed studies other than Phase I and I/II; the 10 identified keywords (adaptive hypothesis, adaptive treatment-switching, biomarker adaptive, adaptive dose-finding, pick-the dose-finding, pick-the-winner/drop-the-loser, sample size re-estimation, adaptive randomization, adaptive group sequential, adaptive seamless, and multiple adaptive) were included in the survey. Of the 2711 trials identified in the databases, 142 met the above search criteria and were included in the study. Of these, 8% (11 trials) included SSRs. Cerqueira et al. also conducted a survey to compile the technical, statistical, and regulatory implications of the use of AD in clinical trials in Medline, PubMed, the EU Clinical Trials Register, and ClinicalTrials.gov [19]. Phase I and seamless Phase I/II trials were excluded from the survey. This research retrieved 336 results, from which 78 were selected for analysis. Sixty-seven results were published articles and 11 were guidelines, papers, and regulatory bills. Of these, 10.3% (8 trials) were related to SSR. As mentioned previously, surveys conducted in recent years indicate that many studies have involved AD, and if SSR is focused on without differentiating between bSSR and ubSSR, SSR is applied in approximately 8%–20% of all AD studies. However, the response rate to the questionnaire survey by DIA-ADSWG was only 24.6%. In addition, depending on the terminology and wording used in the trial registration data and journals, it is possible that some trials which used SSR may have been overlooked. Therefore, it is possible that previous reports do not reflect the real-world use of SSR.

The application of SSR can have a significant impact on study factors such as study size, duration, resources, and cost, and can therefore lead to substantial benefits to the sponsor. In addition, SSR potentially has significant benefits for patients by minimizing over-enrolment and ensuring that studies on promising treatments are not invalidated as a result of insufficient sample size. As mentioned previously, few studies to date have summarized the conditions and circumstances under which SSR is applied. In the current review we aimed to support the future application of SSR by describing the current status of SSR use in clinical trials for the ultimate benefit of sponsors and patients.

2. Methods

2.1. Data sources

AD methodologies have been proposed since the late 20th century, and applications have been reported since the 1990s. EMA guidance on AD was released in 2007 followed by FDA guidance in 2019, although draft FDA guidance had been disclosed online since 2010; we therefore searched for studies published from 2010 onward. Therefore, it is possible that there has been a change in AD application since the late 2000s. In recent years, clinical trials and clinical research have been registered in databases such as ClinicalTrials.gov and the European Union Clinical Trials Register, but it is unclear whether information on AD is accurately captured in these databases. In the case of SSR, various terms and expressions may be used. It is necessary to use sources that accurately indicate whether AD is applied or not, and that provide the details of AD where applied. In addition, previous survey results covered studies from the US and Europe only and we sought to expand the geographical scope in the present study. PubMed was used as the primary search source because of its robust and ubiquitous application for literature searches, supplemented by ClinicalTrials.gov where key information was not available and to identify publicly available data from registered clinical trials.

2.2. Search methodology

PubMed was searched for studies using SSR or related sample size recalculation methods referring to the search strategy described previously by Edwards et al. [20]. and using the following terms: “sample size re-estimation” OR “sample size reestimation” OR “sample size adjustment” OR “sample size readjustment” OR “sample size modification” OR “sample size recalculation” OR “sample size reassessment” OR “adaptive sample size”.

The search duration was from 2010/1/1 to 2022/8/31. Next, for the identified studies, the following information was collected: SSR planned/performed; indication; clinical research/trial; study start year/duration; study status; study phase; number of subjects (actual number or planned number); participating region; achievement of primary endpoint; SSR type; reason for using SSR; actual sample size percentage increase.

For studies identified using the above search strategy, we confirmed the trials in which SSR was applied using information from PubMed and ClinicalTrials.gov to understand the use of SSR. The variable “SSR planned/performed” could be used to understand whether a study only planned to use SSR (but did not perform SSR), or performed SSR with or without planned SSR. The variables “indication”, “clinical research/trial”, and “study start year/duration” were collected to understand the study features. We were particularly interested in date of study start/duration because this may relate to changes based on guidelines or related information that may have affected the use of SSR. The items “study status”, “study phase”, “number of subjects (actual number or planned number)”, and “participating region” were collected to determine study scale. Finally, “achievement of primary endpoint”, “SSR type”, “reason for using SSR” and “actual sample size percentage increase” were collected to understand the features of SSR and the outcomes of trials using SSR.

3. Results

Using the pre-specified search strategies, 253 publications were identified, of which 222 publications were excluded because they focused on statistical methods, and 31 clinical trials and clinical research studies were extracted. Each publication was scrutinized to identify the use of SSR, resulting in 27 clinical trials determined to be relevant. Fig. 1 shows the application of the search strategy to extract studies, while study information is summarized in Table 1 (detailed information on each study is provided as supplementary material).

Fig. 1.

Application of search strategy to extract studies. SSR, sample size re-estimation.

Table 1.

Summary of study information.

Category		Trials determined to be relevant to SSR (N = 27)
		n (%)
SSR planned/performed	Only planned	15 (55.6)
	Unplanned and performed	2 (7.4)
	Planned and performed	10 (37.0)
Indication	Infections and infestations	6 (22.2)
	Nervous system disorders	5 (18.5)
	Neoplasms benign, malignant and unspecified (incl cysts and polyps)	5 (18.5)
	Renal and urinary disorders	2 (7.4)
	Respiratory, thoracic and mediastinal disorders	2 (7.4)
	Cardiac disorders	1 (3.7)
	Metabolism and nutrition disorders	1 (3.7)
	Pregnancy, puerperium and perinatal conditions	1 (3.7)
	Reproductive system and breast disorders	1 (3.7)
	Skin and subcutaneous tissue disorders	1 (3.7)
	Vascular disorders	1 (3.7)
	Not Applicable	1 (3.7)
Clinical research/trial	Clinical research	0 (0)
	Clinical trial	27a (100)
Study start year	Prior to 2009	6 (22.2)
	2010–2016	8 (29.6)
	2017–2022	13 (48.1)
Study durationb	≤2	8 (29.6)
	3–5	10 (37.0)
	6–10	8 (29.6)
	≥11	1 (3.7)
Study status	Completed	18 (66.7)
	Ongoing	9 (33.3)
Study status among trials in which SSR was performed	Completed	11 (91.7)
	Ongoing	1 (8.3)
Study phase	I	2 (7.4)
	II	5c (18.5)
	III	11 (40.7)
	IV	2 (7.4)
	Not specified	7 (25.9)
Number of subjects (actual number or planned number) d	<100	4 (14.8)
	100–499	13 (48.1)
	≥500	10 (37.0)
Participating region	Europe	10 (37.0)
	Europe and other	1 (3.7)
	North America	6 (22.2)
	North America and Europe	1 (3.7)
	North America, Europe, Japan, and other	1 (3.7)
	North America, Europe, and other	2 (7.4)
	North America and other	1 (3.7)
	Japan	1 (3.7)
	Other	4 (14.8)
Achievement of primary endpoint	Yes	5 (18.5)
	No	13 (48.1)
	Ongoing	9 (33.3)
Achievement of primary endpoint among trials in which SSR was performed	Yes	5 (41.7)
	No	6 (50.0)
	Ongoing	1 (8.3)
SSR type	bSSR	9 (33.3)
	ubSSR	10 (37.0)
	Not specified	8 (29.6)
Reason for using SSR	Lack of prior information	23 (85.2)
	Analysis plan needed to be changed during the trial	1 (3.7)
	Not specified	3 (11.1)
Actual sample size percentage increasee	N Mean (min–max) Median (IQR)	12 27.1 (-17.8 – 180.9) 13.9 (-6.8 – 41.2)

SSR, sample size re-estimation; bSSR, blinded SSR; ubSSR, unblinded SSR; IQR, interquartile range.

^aOne trial is for device development.

^bCalculated as (end study year − start study year) + 1.

^cOne trial was Phase I/II, which is counted as Phase II. One trial was Phase II/III, which is counted as Phase II because the trial was completed without proceeding to Phase III.

^dActual number of subjects for completed trials or the planned number of subjects for ongoing trials.

^eCalculated as ((actual sample size − planned sample size)/planned sample size) *100.

All twenty-seven trials (100%) were clinical trials; no clinical research studies were identified. There were 18 (66.7%) completed trials and 9 (33.3%) ongoing trials. In most (92.6%) of the trials, SSR was planned at the time of study initiation. Ten trials (37.0%) planned for SSR and actually performed it, while 15 trials (55.6%) planned SSR but did not perform it and 2 (7.4%) trials performed SSR without initial planning. In most of the trials, SSR was planned prior to trial initiation. Among the trials where the study phase was mentioned, 2 (7.4%) trials were Phase I, 5 (18.5%) trials were Phase II, 11 (40.7%) trials were Phase III, and 2 (7.4%) trials were Phase IV, indicating that SSR is not only used in late-phase clinical development but also in earlier decision making. Indications included the following: “cardiac disorders”, “infections and infestations”, “neoplasms benign, malignant and unspecified (incl cysts and polyps)”, “nervous system disorders”, “pregnancy, puerperium and perinatal conditions”, “renal and urinary disorders”, “reproductive system and breast disorders”, “respiratory, thoracic and mediastinal disorders”, “vascular disorders”, “metabolism and nutrition disorders”, “skin and subcutaneous tissue disorders”. There was no trend toward SSR being implemented more frequently for specific indications. The oldest trial started in 2003. Six (22.2%) trials started before 2009 while 21 (77.8%) trials started after 2010, when the FDA draft guidance related to ADs was disclosed online. Of the trials initiated after 2010, 8 (38.1%) trials were initiated between 2010 and 2016 and 13 (61.9%) trials were initiated between 2017 and 2022. Of note, many trials on Covid-19 have been initiated since 2019. The duration of the trials was 2 or fewer years in 8 (29.6%) trials, 3–5 years in 10 (37.0%) trials, 6–10 years in 8 (29.6%) trials, and 11 or more years in 1 (3.7%) trial. SSR was thus applied for a variety of trial start dates and durations. Regarding the number of subjects (the planned number of subjects for ongoing trials or actual number of subjects for completed trials), there were 4 (14.8%) trials with fewer than 100 subjects, 13 (48.1%) trials with 100–499 subjects, and 10 (37.0%) trials with 500 subjects or more. The trials in which SSR was implemented were relatively large in terms of subject size. Regarding the actual sample size percentage increase, the median percent change was 13.9%, the minimum percent change was −17.8% [21], and the maximum percent change was 180.9% [22]. Fig. 2 shows a scatter plot with sample size at the time of study planning on the horizontal axis and sample size at the time of finalization on the vertical axis for trials in which SSR was performed. This plot shows that the sample size at study finalization was increased following the implementation of SSR.

Fig. 2.

Scatter plot of actual versus planned sample size at the time of study planning and finalization. “Minimum” and “Maximum” indicate the minimum and maximum percent change from the sample size at the time of study planning, respectively. Dot line (“Median”) indicates the place corresponding to the median percent change from the sample size at the time of study planning.

There were 5 (18.5%) trials in which the primary endpoint was achieved, 13 (48.1%) trials in which the primary endpoint was not achieved, and 9 trials (33.3%) still ongoing. Five trials in which SSR was planned and conducted achieved the primary endpoint; the details of the five trials are shown in Table 2. No trends of note were observed for participating region or indication. There were 9 (33.3%) trials that used bSSR and 10 (37.0%) trials that used ubSSR, meaning that no trend in SSR type was identified. Regarding the reason for using SSR, most trials (85.2%) used SSR because of a lack of prior information during study design. Most trials that planned to use SSR were conducted in North America or Europe, with 11 (40.7%) trials conducted in North America and 15 (55.6%) trials that were conducted in, or included, Europe.

Table 2.

Summary of information from trials with SSR that achieved the primary endpoint.

Trial identifier	Study phase	Duration of the trial (years)	Planned number of subjects	Actual number of subjects
NCT01485185 [23]	Phase I	2	24–48a	30
NCT02741557 [24]	Phase I	1	86	118
NCT00428948 [25]	Phase III	6	1200–1500a	1445
NCT02641730 [26]	Phase III	4	150	159
NCT02224755 [22]	Not specified	6	366	1028

^aPlotted the average of the range shown in Fig. 2.

4. Discussion

Although several studies on SSR methodology have been published, few have detailed the studies in which SSR was actually planned and/or performed. The present survey demonstrates that studies and publications on actual SSR use remain limited. It is a limitation that not all study information is described in publications obtained from PubMed or for entries on ClinicalTrials.gov. Furthermore, the number of trials using SSR may have been underestimated because some studies might not have been extracted using the specified search strategy, or studies might not have been published; this should be taken into account when interpreting the results. Since SSR is an important piece of information for the study design with the sample size change prior to the end, we hope that the appropriate terms by which SSR use is easily found, such as described in 2.2 search methodology, will be included in future papers. Although the number of studies extracted in this survey was smaller than expected, there was no significant difference between the number of studies identified in the present research and the number identified in a previous survey of studies described in the introduction section [16,17,19], meaning that the search strategy can be considered appropriate. In terms of the start year of identified trials, we found that many started in 2010 or later which may have resulted from the online publication of draft FDA guidance on conducting ADs in 2010. Most trials using SSR were conducted in North America and Europe. Some trials were conducted in South America and Asia, but only two study included Japan, one of the founding members of ICH. These results indicate that use of SSR is not limited to North America and Europe, but we anticipate that more clinical trials performing SSR will be conducted in Japan and elsewhere in the future. There were 5 trials in which the primary endpoint was achieved among the trials in which SSR was conducted. Two of the five successful trials were Phase I trials with fewer than 100 planned subjects, suggesting that SSR was applied early in the development process and led to positive outcomes. In one trial, the actual number of subjects was more than twice the planned number of subjects to compensate for the lack of prior information. Increasing the sample size during a trial based on SSR can thus be recognized as an important factor in the success of that trial. Regarding SSR type, 2 of the 5 trials were bSSR, meaning that the use of unblinded information did not contribute to trial success. The two trials in which SSR was performed but not planned in advance both failed to achieve their primary endpoints, indicating the importance of careful planning when conducting SSR. No phase II trials were included among those in which the primary endpoint was achieved because 4 out of 5 Phase II trials are still ongoing and it remains uncertain whether the primary endpoint will be achieved. Prior to conducting this survey, we expected that SSR would be used primarily in later phases, but we found that SSR was also used in Phase I and II, representing earlier decision making. Successful trials varied in size and phase, demonstrating the broad applicability of SSR. Although the achievement of the primary endpoint is an important factor in determining whether a trial is successful or not, there are limitations to discussing the usefulness of SSR based on the achievement of the primary endpoint. In fact, some trials would have met their primary endpoint without the use of SSR, while some trials would have failed even with SSR due to no treatment effect. Especially in early phase trials such as Phase I and II, since the trial results come from the limited number of data compared to Phase III, so it is necessary to be cautious for interpretation of trial results, that is, even if the primary endpoint is achieved after SSR, it cannot easily be determined that it was met by SSR. However, as the number of subjects can be recalculated using information up to the middle of the trial by conducting SSR, the estimation would be more accurate than that assumed at the beginning of the trial, resulting in a reduction of information uncertainty. Our survey results mean, SSR has the potential to be used in any phase in situations where there is a lack of prior information considering the limitation. Therefore, it is expected that SSR can be used in Phase I and II trials where such information is frequently unavailable. A scatter plot of sample size at planning and at trial completion (Fig. 2) showed that sample size was increased with SSR, although the increases were generally not substantial. The maximum sample size is typically set in advance to prevent an unnecessary increase in the number of required subjects. Examples of “lack of prior information” as a reason for using SSR include insufficient information on effect size at the start of the study, limited information available on single-arm studies during study design, and the need to obtain information for designing the sample size for the next stage of clinical development. In addition to compensating for a lack of prior information, SSR may be useful from an ethical perspective such as that noted in the FDA guidance: “the ability to stop a trial early if it becomes clear that the trial is unlikely to demonstrate effectiveness can reduce the number of patients exposed to the unnecessary risk of an ineffective investigational treatment and allow subjects the opportunity to explore more promising therapeutic alternatives” [9]. This description applies to clinical trials performing SSR, given that many such trials plan to conduct an interim analysis when applying SSR. Therefore, when designing trials that are difficult to conduct, such as those with long enrollment periods or other recruitment challenges, trials for rare diseases, or those conducted in children, elderly patients, and pregnant women, SSR can be regarded as a feasible method to mitigate such challenges.

5. Conclusions

Prior to conducting this survey, we expected that SSR would predominantly be used in later phases of clinical development, such as in Phase III clinical trials, but our results showed that SSR is also used in Phase I and II, which involve earlier decision making. We expect that SSR will continue to be used in early-phase trials where sufficient prior information may not be available. Furthermore, no major trends were observed in relation to therapy area or type of SSR, meaning that SSR may become a feasible and widely applied method in the future.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article:

Data availability

The information used in this study is presented in the supplementary material.