Adaptive Gaussian process search for simulation-based sample size estimation in clinical prediction models: validation of the pmsims R package

Abstract

Background

Determining an adequate sample size is essential for developing reliable and generalisable clinical prediction models, yet practical guidance on selecting appropriate sample size calculation methods remains limited. Existing analytical and simulation-based tools impose restrictive assumptions and focus on mean-based criteria. We present and validate pmsims, an R package that uses Gaussian process (GP) surrogate modelling to provide a flexible and efficient simulation-based framework for sample size determination, validated here across binary, continuous, and survival prediction modelling contexts.

Methods

We conducted a comprehensive simulation study with two aims. Aim 1 compared three search engines implemented in pmsims, a GP surrogate-based adaptive procedure (gp), a deterministic bisection method (bisection), and a hybrid GP-bisection approach (gp-bs), across binary, continuous, and survival outcomes. Scenarios varied outcome prevalence or event rate, predictor dimensionality (), target performance metric (discrimination and calibration slope), aggregation criterion (mean vs 80% assurance), and total simulation budget (B = 200-2000). Each scenario was replicated 100 times; estimator stability was assessed via the coefficient of variation (CV). Aim 2 benchmarked the best-performing pmsims engine against pmsampsize (analytical) and samplesizedev (simulation-based) across a wider range of realistic scenarios, evaluating recommended sample sizes, computational time, and achieved model performance on independent validation datasets of 30,000 observations.

Results

The GP-based search engine consistently yielded the most stable sample size estimates (lowest CV) across all outcome types, ranking highest in 9/12 outcome-aggregation metric configurations. Its advantage was most pronounced in low-signal, high-dimensional settings, and was accentuated with κ = 20 replications per evaluation and a budget of B ≈ 1000, after which gains were minimal. In benchmark comparisons, pmsims (mean) achieved performance deviations within of the prespecified target across binary, continuous, and survival outcomes, comparable to samplesizedev and substantially outperforming pmsampsize in high-discrimination settings (deviations up to -9.84%).

Conclusions

The pmsims package, using the GP-based search engine with replications per evaluation and a budget of B ≈ 1000, provides a computationally efficient and flexible framework for principled sample size planning in clinical prediction modelling. It reliably achieves performance targets across the range of standard-model scenarios evaluated while requiring fewer model evaluations than non-adaptive simulation approaches, offering a compelling alternative to both analytical formulae and exhaustive simulation-based search.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12874-026-02935-9.

Introduction

Determining an adequate sample size is an essential prerequisite for developing stable, reliable, and generalisable clinical prediction models. Development samples that are too small can produce overfitted models with poor calibration and reduced discrimination when applied to new data, undermining clinical utility and potentially causing harm in decision-making contexts [1].

Sample size determination for prediction models differs fundamentally from the more familiar problem of sample size calculation for hypothesis testing. In a conventional hypothesis-testing or study-design setting, the calculation targets statistical power: the sample size is chosen so that a test of a single prespecified effect attains a desired probability of detecting that effect at a fixed type I error rate, and the key inputs are the assumed effect size, the significance level, and the desired power. Prediction modelling has no analogous null hypothesis or single effect to detect. Instead, the goal is to estimate many parameters jointly well enough that the resulting model predicts accurately in new individuals, that is, to control over-fitting and ensure adequate calibration and discrimination of the whole risk function rather than to reject a hypothesis [1, 2]. Consequently, the quantities that drive the required sample size are different: the number of candidate predictors, the outcome prevalence or event rate, the anticipated overall model strength (e.g. the large-sample C-statistic or ), and the acceptable degree of overfitting (shrinkage), rather than a target effect size and power [2, 3]. Performance is also multi-dimensional, spanning discrimination, calibration, and overall accuracy, so a single sample size may satisfy one criterion while failing another. These features make prediction-model sample size calculations both conceptually and computationally more demanding, and motivate flexible simulation-based tools of the kind described here.

In prediction modelling, sample size estimation seeks the minimum development sample size needed for model performance to meet or exceed a prespecified threshold on a chosen metric [4]. Targets can be defined using either a ‘mean’ criterion, where expected performance exceeds the threshold, or an ‘assurance’ criterion, where performance exceeds the threshold with high probability (e.g., 80%). The assurance approach explicitly accounts for training-sample variability and is therefore more robust to uncertainty in model development [5, 6].

Analytic tools have been developed to estimate minimum sample sizes for traditional regression-based prediction models. For example, the pmsampsize R library and Stata modules target an acceptable degree of overfitting (by shrinkage) or ensure a sufficiently precise estimation of the average outcome in the population [2]. However, these approaches are limited to linear or generalised linear models estimated using maximum likelihood and are not generalisable to a wider class of machine learning models or alternative estimation methods such as penalised maximum likelihood. Moreover, they rely on assumptions that are often unmet in practice. In particular, they typically presume a correctly specified linear predictor, normally distributed covariates, and properties of maximum likelihood estimation assumptions that can be violated in real-world clinical data characterised by non-linearities, interactions, and complex correlation structures [7]. A further limitation of existing analytical methods is that they focus on the ‘mean’ criterion, which does not capture variability in performance arising from finite training samples. Consequently, a sample size that meets a mean target may still imply a substantial probability of underperformance in the risk to the target population not captured by closed-form formulae. Simulation studies have also shown that analytic approaches can underestimate the required sample size when the model strength is high (C-statistic ≥ 0.8) [8].

To address the limitations of analytic approaches, simulation-based sample size tools have been developed. samplesizedev is a simulation-based R package that estimates sample sizes for binary and survival outcomes, quantifying variability in model performance metrics and supporting probability-based assessment of achieving target performance across training samples (the ‘assurance’ criterion) [6]. More recently, Bayesian approaches have also been proposed to support probability-based (assurance-type) sample size criteria for prediction modelling [5, 6, 9]. samplesizedev only supports logistic and Cox regression models, simulates predictors independently from standard normal distributions, and is not readily extendable to other models or data generators. Other approaches to sample size determination include learning curve methods, which extrapolate model performance from pilot data; and hybrid approaches, in which estimated learning curves are summarised as approximate analytical formulae parametrised by the characteristics of the datasets used to derive them. These approaches are reviewed in [4].

Despite these advances, there is a lack of practical and accessible tools for determining assurance-type sample size that accommodate diverse model specifications and realistic data-generating mechanisms. This gap presents researchers with an unhelpful choice: rely on simplified approaches that may not reflect the complexities of the intended application or omit sample size justifications altogether. Consistent with this, systematic reviews show that most prediction modelling studies do not report an explicit sample size justification [10].

In this paper, we present pmsims, an R package that provides a flexible simulation-based framework for estimating minimum sample sizes for developing clinical prediction models. The package addresses two core limitations of existing analytical approaches: limited flexibility in model and data-generating specifications, and the computational burden of naïve simulation-based searches over large sample size spaces [4].

The pmsims package incorporates two key innovations. First, it uses Gaussian process (GP) surrogate modelling of the sample size–performance relationship [11, 12]. Rather than exhaustively evaluating every candidate sample size via computationally expensive model fitting, the GP-based engine uses a Gaussian process surrogate model to approximate performance as a function of sample size. It iteratively selects the next sample sizes to evaluate by choosing points where the current Gaussian process shows the highest posterior uncertainty (or where an acquisition function that balances predicted performance and uncertainty is maximised), runs a limited number of Monte Carlo simulations at those points, updates the GP posterior with the new results, and repeats until the surrogate provides a smooth, reliable approximation of the entire performance curve [4]. This approach, grounded in Bayesian optimisation principles, dramatically reduces the number of required model fits compared to traditional grid search or bisection algorithms [11]. Second, pmsims provides a flexible framework for sample size calculations across different model types, data-generating mechanisms, and performance metrics of interest. This design moves beyond the narrower focus of existing tools, in principle enabling researchers to tailor calculations to their specific prediction modelling context rather than fitting their problem to available software. We emphasise, however, that the empirical validation reported here is confined to the standard regression models and idealised data-generating processes described below; the broader flexibility of the framework is a design property that remains to be characterised under more realistic conditions in our future work.

This study presents the comprehensive validation of the pmsims package (version 0.5.0) through two complementary objectives. First, we evaluate the performance and computational efficiency of the GP-based engine against traditional simulation-based search algorithms. Second, we assess empirical agreement between sample size estimates from pmsims and those from established analytical and simulation-based tools, specifically pmsampsize (implementing the Riley et al. methodology for logistic, linear and survival outcomes) [13] and samplesizedev (a simulation-based approach motivated by limitations of analytical formulae in certain scenarios) [8], across a broad spectrum of realistic prediction modelling scenarios.

The pmsims workflow

The pmsims package implements a flexible simulation framework for sample size estimation, described by three components: (i) a data-generating process, (ii) a model-fitting procedure, and (iii) a performance metric. These components are combined with a search engine to estimate the minimum sample size n that satisfies a prespecified performance criterion, either on average (the ‘mean’ criterion) or with a specified level of certainty (the ‘assurance’ criterion). Further details of the conceptual framework are provided in [4].

Although pmsims supports arbitrary combinations of data-generating processes, model-fitting procedures, and performance criteria, we restricted the scope of this validation study to enable direct comparison with existing sample size tools such as pmsampsize and samplesizedev. Specifically, we focused on binary, continuous, and survival outcomes; simple data-generating processes; and logistic regression, linear regression, and proportional hazards models. These are defined formally below and are provided in the package as pre-defined functions. By design, this validation therefore evaluates the search engines using simple, well-specified scenarios that allow any shortcomings of the search algorithms to be identified more readily, without confounding from additional sources of complexity. A separate forthcoming benchmarking study will assess pmsims under more realistic and challenging data-generating mechanisms (for example, mixed discrete and continuous covariates, correlated and noise predictors, and model misspecification); the present results should be interpreted within the well-specified settings evaluated here.

Data-generating process

Let be a p-dimensional vector of predictors, where . The data-generating process is defined by the joint distribution , where the outcome Y is generated conditionally on . For each outcome type, the linear predictor

is constructed, where contains coefficients for the signal predictors (all set to a common value ) and zeros for noise predictors. This structure reflects standard assumptions in simulation studies for evaluating prediction models [14]. The current version of the pmsims package (0.5.0) provides three data-generating processes:

Binary outcomes	, where . The baseline probability (when ) is controlled by the intercept , which is tuned along with to achieve a specified outcome prevalence and discrimination, key drivers of required sample size in binary prediction models [2].
Continuous outcomes	, with . The signal strength is tuned to achieve a target large-sample , a common measure of explained variance in continuous prediction settings [15].
Survival outcomes	Event times , where . Right-censoring is introduced at a time point such that the censoring rate matches a specified value. The linear predictor coefficients are tuned to achieve a target large-sample C-index, the most widely used discrimination measure for time-to-event prediction models [16, 17].

The predictors are simulated as standardised continuous normal variables, reflecting common design choices in methodological simulation studies [1]. These data-generating processes are implemented internally within the package, with separate routines handling binary, continuous, and survival outcomes, respectively.

Data generator tuning for continuous outcomes

The parameters governing each data-generating process are tuned to achieve two user-specified targets: (i) the desired outcome prevalence in the population, and (ii) the maximum achievable performance (e.g., C-statistic for binary/survival, for continuous) of a correctly specified model, as advocated in principled simulation-based evaluations [3, 8]. It should be noted that these tuning strategies are implemented specifically for the default prediction models (logistic, linear, and Cox regression); researchers wishing to employ alternative model classes would need to develop corresponding tuning functions tailored to those models, which remains an avenue for future development.

The continuous outcome tuning function in the package analytically determines the common coefficient for all signal predictors. Given a target large-sample , number of candidate features p, and number of noise features , it calculates

where is the number of true signal predictors. This derivation follows directly from the relationship between regression coefficients, predictor variance, and explained variance in linear models [18]. This ensures that a linear regression model fitted to a very large sample will achieve the expected .

Data generator tuning for binary outcomes

The tuning of the signal coefficient to match the required performance targets follows an approach similar to that of [19]. The intuition is as follows. Because the predictors are standardised normal variables and the coefficients are common across signal predictors, the linear predictor is itself (approximately) normally distributed, . Under this binormal structure, two population quantities of interest are fully determined by the mean μ and variance of η: (i) the marginal event prevalence, , which depends mainly on the location μ (controlled through the intercept ); and (ii) the large-sample C-statistic, which is the probability that a randomly chosen case has a higher linear predictor than a randomly chosen control and depends on the spread σ of η (controlled through ). A larger separates the case and control distributions of η and thereby increases discrimination, whereas μ shifts the overall event rate.

Tuning therefore reduces to finding the pair that makes both quantities equal to their user-specified targets simultaneously. Because and the C-statistic are smooth but coupled functions of (changing σ to fix discrimination also perturbs the prevalence, and vice versa), the two targets cannot be solved independently and are instead matched jointly by a numerical optimisation that minimises the combined squared error between the estimated and target values:

where and is computed via numerical integration over the bivariate normal distribution of the linear predictor for cases and controls, consistent with the binormal model for the ROC curve [19]. Once the optimised is obtained, the common signal coefficient is set as

which (given unit-variance, independent standardised predictors) makes the variance of equal to the optimised , so that a model fitted to a very large sample attains the target prevalence and C-statistic.

Data generator tuning for survival outcomes

The survival outcome tuning function performs optimisation to find parameters for a proportional hazards model. It searches for the log of the baseline hazard and the log of the standard deviation of the linear predictor that minimises the squared error between the actual event rate (1 − censoring rate) and C-index and their respective targets. The optimisation uses simulated survival data (with exponential event times) to evaluate the objective function, building on established links between the linear predictor variance and concordance in proportional hazards models [19]. The resulting σ is then used to set analogously to the binary case.

Model-fitting procedure

Given a training dataset

a prediction model is fitted. The package supports logistic regression for binary outcomes, linear regression for continuous outcomes, and Cox proportional hazards models for survival outcomes as default model classes. The model-fitting function returns a fitted model object , which is then used to generate predictions on an independent validation set.

Model evaluation and performance metrics

Model performance is evaluated on a large, independent validation set of size to approximate the expected performance in the population. Let denote the predicted probability (or risk score) for the i-th observation. The package supports multiple metrics:

• Discrimination
  For binary outcomes, the area under the ROC curve (AUC) is computed. For survival outcomes, Harrell’s C-index is used. For continuous outcomes, the out-of-sample is calculated as [15]

  
• Calibration

  The calibration slope γ is estimated by regressing the observed outcome on the linear predictor (or log-odds) of the predictions. A slope of 1 indicates perfect calibration.

Sample size search engines

In this study, we compare three search engines to find the minimum sample size such that the expected performance (or a specified quantile thereof) exceeds a target τ. Let denote the simulation process: generate training data of size n, fit the model, and compute the performance metric G on the validation set.

Gaussian process search

The primary search engine in pmsims is a Gaussian process (GP)-based adaptive procedure implemented via the mlpwr package [11]. This approach treats the relationship between sample size and model performance as an unknown smooth function,

and uses GP surrogate modelling to emulate the function using a limited number of simulation evaluations. The mlpwr package implements the surrogate-modelling and GP optimisation to the observed (sample size, performance) pairs and proposing the next evaluation point via an acquisition rule. pmsims builds on this engine to make it directly usable for clinical prediction model planning: it provides the outcome-specific data-generating processes and tuning routines (Data-generating process section), the default model fitting functions and the prediction-model performance metrics (AUC, C-index, out-of-sample , and calibration slope; Model evaluation and performance metrics section), the adaptive starting value search procedure that supplies data-driven search bounds (Adaptive starting value search for pmsims engines section), and aggregation under both the mean and assurance (quantile) criteria, together with the alternative bisection and hybrid gp-bs engines for comparison. In other words, mlpwr provides the surrogate optimiser, while pmsims adapts and extends it into a self-contained package tailored to sample size determination for prediction models.

Algorithm 1 begins by determining a data-driven lower bound on the sample size using outcome-specific heuristics informed by the number of predictors and, where relevant, the outcome prevalence or censoring rate. A small set of initial sample sizes is then evaluated to initialise the GP surrogate. At each iteration, the GP guides the selection of a new candidate sample size, targeting the smallest n at which the desired performance threshold τ is achieved.

Algorithm 1 Gaussian process–based sample size search (gp engine)

Model performance is aggregated either by its mean (mean-based criterion) or by a lower quantile of its sampling distribution (assurance criterion). For the assurance criterion, the target is the 20th percentile of the performance distribution across training sets, estimated empirically across simulation replications at each candidate n. In the GP-based engines, simulation noise is explicitly modelled using a bootstrap-based variance estimator, and the search continues until the simulation budget is exhausted.

Bisection search (bisection)

The bisection search uses a classical deterministic bisection algorithm. This engine maintains an interval that brackets the unknown minimum sample size . At each iteration, the algorithm evaluates model performance at the midpoint

Multiple simulation replicates are generated at , and performance is aggregated using either the mean-based or assurance criterion. If the aggregated performance estimate , the upper bound is updated to ; otherwise, the lower bound is updated. The interval is repeatedly halved until its width falls below a predefined tolerance or the simulation budget is exhausted.

Hybrid GP–bisection search (gp-bs)

The hybrid engine (gp_bs) combines bisection and Gaussian process surrogate modelling in a three-stage strategy. First, adaptive bound estimation is performed as in the other engines. Second, a coarse bisection search is run with a fixed budget of replications to rapidly narrow the search interval to , reducing the risk of poor GP initialisation that can occur when the search space is wide or the performance function is highly variable. The refined interval is then passed to the GP search stage, which conducts an adaptive, model-guided search within these tighter bounds. This engine may be particularly useful in scenarios with high performance variance or complex data-generating mechanisms.

Adaptive starting value search for pmsims engines

All three engines share a common first stage: an adaptive procedure that automatically determines the sample size bounds supplied to the main search algorithm (as described in Gaussian process search–Hybrid GP–bisection search (gp-bs) sections). The goal is to find two sample sizes and such that

where τ is the user-specified target performance. The search begins from a heuristic starting value determined by the outcome type and the number of predictors p, and proceeds by repeated doubling or halving until the target is bracketed. This stage uses a fixed pilot budget of replications (distinct from the main evaluation budget B), allocated as replications per candidate sample size, where K is the maximum number of iterations. A tolerance δ is applied to avoid unnecessary iterations when the performance estimate is already sufficiently close to τ.

Adaptive starting value search algorithm

The procedure is summarised in Algorithm 4, using the same conventions as the search engines in Algorithms 1–3.

Algorithm 2 Bisection-based sample size search (bisection engine)

Algorithm 3 Hybrid bisection–Gaussian process search (gp-bs engine)

Algorithm 4 Adaptive starting-value search

The doubling and halving strategy locates a bracket around τ without requiring prior knowledge of the sample size–performance relationship. The procedure is intentionally conservative, using few replications per candidate to keep the pilot cost low. The method assumes that the performance metric is monotonically non-decreasing in n; if the metric is non-monotonic, the algorithm may fail to bracket the target correctly, and manual specification of is recommended.

Simulation scenarios for pmsims package validation

This simulation study had two aims, following standard operating simulation design principles [20]. Aim 1 evaluated the statistical performance and computational efficiency of the Gaussian process–based adaptive search gp engine, benchmarking it against two reference engines: its hybrid (gp-bs) and a bisection search (bisection). Aim 2 benchmarked pmsims, using the engine identified as best-performing in Aim 1, against two existing approaches: pmsampsize [2] and samplesizedev [8].

Aim 1: Comparison of pmsims search engines

To systematically compare the performance of the three search engines (gp, bisection, and gp-bs), we conducted an extensive simulation study covering a range of realistic prediction modelling scenarios, consistent with best-practice guidance for simulation-based methodological research [21, 22]. The scenarios were varied along five key dimensions: (1) outcome type, (2) number of candidate predictors, (3) outcome distribution or model strength, (4) target performance metric and threshold, and (5) total simulation budget. For each outcome type—binary, continuous, and survival—we defined a set of base scenarios reflecting common characteristics encountered in clinical prediction research [2, 23]. Each scenario was independently replicated 100 times to ensure robust estimation of the precision and stability of the sample size estimates produced by each engine, in line with recommendations for Monte Carlo precision assessment [22].

The levels of specific factors for each outcome type are summarised in Table 1. For binary outcomes, we considered two event prevalences (5% and 20%) and two target metrics: the C-statistic (AUC) with a target set 0.05 below the large-sample value and the calibration slope with a target of 0.9. For continuous outcomes, we varied the expected large-sample (0.2 and 0.7) and similarly targeted either or a calibration slope of 0.9. For survival outcomes, we considered two event rates (40% and 80%) with targets on the C-index or the calibration slope (0.9).

Table 1.

Simulation scenarios for the comparative evaluation of search engines

Outcome	p	Model strength	Target metric	Target value
Binary	10, 100	Prevalence = 0.05, C = 0.80	AUC	0.75
		Prevalence = 0.05, C = 0.80	Calibration slope	0.90
		Prevalence = 0.20, C = 0.80	AUC	0.75
		Prevalence = 0.20, C = 0.80	Calibration slope	0.90
Continuous	10, 100			0.15
			Calibration slope	0.90
				0.65
			Calibration slope	0.90
Survival	10, 100	Event rate = 0.40, C = 0.80	C-index	0.75
		Event rate = 0.40, C = 0.80	Calibration slope	0.90
		Event rate = 0.80, C = 0.80	C-index	0.75
		Event rate = 0.80, C = 0.80	Calibration slope	0.90

Across all outcome types, we examined both a modest (10) and a larger (100) number of candidate predictors, with no noise predictors. The total simulation budget B was varied from 200 to to assess how algorithmic efficiency scales with available computational resources. The budget was allocated in blocks of independent simulation replicates per candidate sample size.

Aim 2: Benchmark comparison of pmsims with pmsampsize and samplesizedev

To benchmark the simulation-based approach implemented in pmsims against existing methods, we compared the sample size estimates from the best-performing engine (gp) with those provided by pmsampsize (for binary and continuous outcomes) and samplesizedev (for binary and survival outcomes). We evaluated a range of realistic prediction-modelling scenarios, systematically varying the model strength, outcome prevalence (or event rate), number of predictors, and target performance metric.

For binary outcomes, we considered large-sample C-statistics (AUC) of 0.8 and 0.9; outcome prevalences of 5% and 20%; and predictor counts ranging from 5 to 100 in seven steps (5, 10, 20, 40, 60, 80, 100). For continuous outcomes, the large-sample was set at 0.2, 0.5, and 0.7, with the same range of predictor counts. For survival outcomes, we used large-sample C-indices of 0.7, 0.8, and 0.9; event rates of 40%, 50%, and 80%; and the same predictor counts.

Within each scenario, two target metrics were examined: a discrimination metric set to 0.05 below the large-sample value, and the calibration slope set to 0.90. The gp engine was run with a fixed simulation budget of 1,000 total model evaluations, using 20 replications per candidate sample size, under both mean-based and assurance-based criteria (80% assurance). For each scenario, we recorded the recommended sample size for each method and the corresponding computational time. To assess validity, we generated an independent validation dataset of 30,000 observations and evaluated the achieved performance of a model developed on a sample of the recommended size. Each scenario was repeated 100 times. The complete set of evaluated scenarios is summarised in Table 2.

Table 2.

Simulation scenarios for comparing pmsims (gp engine) with existing sample size tools (pmsampsize, samplesizedev). All scenarios were evaluated for both mean-based and assurance-based criteria (80% assurance)

Outcome type	Large-sample performance	Prevalence/event rate	Target metric	Target value
Binary	C-statistic = 0.8, 0.9	0.05, 0.20	AUC	|C - 0.05|
	C-statistic = 0.8, 0.9	0.05, 0.20	Calibration slope	0.90
Continuous		–
		–	Calibration slope	0.90
Survival	C-index = 0.7, 0.8, 0.9	0.40, 0.50, 0.80	C-index	|C - 0.05|
	C-index = 0.7, 0.8, 0.9	0.40, 0.50, 0.80	Calibration slope	0.90

Note: For binary and survival outcomes, the large-sample performance is the expected C-statistic (AUC) or C-index, respectively. The target for discrimination metrics is set 0.05 below this value to reflect a practically acceptable margin for finite-sample optimism [23]. The calibration-slope target is 0.90, representing a commonly accepted threshold for good calibration in clinical prediction models [1]. All predictors are continuous and truly associated with the outcome.

Simulation estimands

In each simulation scenario, the performance of the sample size determination procedures is summarised using Monte Carlo estimands computed across S = 100 independent simulation runs [21, 22]. We computed the mean and standard deviation of the estimated minimum sample sizes across runs, as well as the coefficient of variation,

which provides a scale-free measure of relative dispersion and is used to assess estimator stability across simulation replicates [21].

The number of simulation replicates (S = 100) was selected to balance Monte Carlo accuracy against the substantial computational burden imposed by the nested design, in which each replicate entails a complete adaptive search involving up to model fits across κ replications per candidate sample size [22]. To render the resulting Monte Carlo uncertainty explicit, and in view of the fact that the ranking of search engines is driven primarily by the CV, we report a Monte Carlo standard error (MC SE) for each CV estimate [22].

For Aim 2, performance is evaluated based on the achieved model performance when fitting a model using the sample size recommended by each method. Let denote the performance estimated on an independent validation dataset in replicate s, and let denote the corresponding target performance value. Performance deviation is defined as

Results

Aim 1: Comparison of search engines

Binary outcomes

Figure 1 compares precision (coefficient of variation; CV) and computational time for the three search procedures across the calibration slope metric under varying design parameters: number of simulation replicates per evaluation (κ = 10, 20), outcome prevalence (0.05, 0.20), number of predictors (p = 10, 100), and total simulation budget (B = 200–2000). Increasing the number of replicates per evaluation to 20 substantially reduced the CV for the gp engine. Across all remaining conditions, gp consistently produced lower and more stable CV estimates than bisection and gp-bs, with the advantage most pronounced in the low-prevalence setting. Performance improvements were more consistent under mean-aggregation, which yielded smoother convergence trajectories. On average, with κ = 20, CV plateaued at a total simulation budget of approximately B = 1000 under both aggregation schemes, indicating that B = 1000 constitutes an efficient lower bound. Comparable patterns were obtained for the AUC metric (Fig. S1, Additional File 1).

Fig. 1.

Aim 1 (Binary outcome): Comparison of CV and computational time across search engines gp, bisection and gp-bs for calibration slope metric under varying number of simulation replicates per evaluation (κ = 10, 20), prevalence (0.05, 0.20), number of predictors (p = 10, 100) and total simulation budget (B = 200–2000)

At B = 1000 (Table 3), gp consistently achieved the lowest CV for AUC estimation, most notably in challenging settings characterised by low prevalence (0.05) and high dimensionality (p = 100). For calibration slope estimation, gp-bs occasionally outperformed the other methods under assurance-based aggregation in low-prevalence scenarios, though gp generally delivered competitive performance under mean aggregation.

Table 3.

Estimated minimum sample size () and stability (CV) for binary outcomes using assurance aggregation (20% quantile) and mean aggregation, stratified by target metric, event prevalence and number of predictors (p), at a fixed total budget B = 1000 and κ = 20. Results are averaged over 100 simulation replicates. Values in parentheses after each CV are the Monte Carlo standard error of the CV

Metric	Prevalence	p	Engine	Mean		CV (MC SE)
				Assurance	Mean agg.	Assurance	Mean agg.
AUC	0.05	10	bisection	444	320	17.05 (1.25)	11.79 (0.85)
			gp	431	315	13.39 (0.97)	3.01 (0.21)
			gp-bs	428	308	11.72 (0.84)	10.11 (0.73)
		100	bisection	3671	3190	11.67 (0.84)	8.17 (0.58)
			gp	3691	3168	5.43 (0.39)	1.52 (0.11)
			gp-bs	3650	3127	10.08 (0.72)	9.90 (0.71)
	0.2	10	bisection	152	111	12.68 (0.92)	10.89 (0.78)
			gp	154	110	4.51 (0.32)	2.91 (0.21)
			gp-bs	152	110	6.17 (0.44)	7.03 (0.50)
		100	bisection	1311	1145	7.43 (0.53)	6.76 (0.48)
			gp	1307	1138	2.61 (0.19)	0.82 (0.06)
			gp-bs	1289	1145	6.09 (0.43)	6.35 (0.45)
Calib. Slope	0.05	10	bisection	3778	1641	35.72 (2.84)	30.30 (2.34)
			gp	3515	1566	27.62 (2.11)	24.94 (1.88)
			gp-bs	3577	1596	29.20 (2.24)	23.37 (1.75)
		100	bisection	23654	18314	19.94 (1.47)	26.38 (2.00)
			gp	23955	17958	6.53 (0.47)	14.82 (1.08)
			gp-bs	22669	18879	18.28 (1.34)	25.52 (1.93)
	0.2	10	bisection	1389	594	26.20 (1.98)	16.48 (1.20)
			gp	1311	592	20.11 (1.49)	8.57 (0.61)
			gp-bs	1325	583	19.03 (1.40)	12.86 (0.93)
		100	bisection	8704	6696	18.50 (1.36)	14.39 (1.04)
			gp	8654	6432	11.15 (0.80)	2.19 (0.16)
			gp-bs	8308	6595	15.98 (1.16)	11.44 (0.82)

Continuous outcomes

Results for continuous outcomes were consistent with the binary findings. The gp engine consistently produced the lowest and most stable CV across all experimental conditions, with the advantage particularly pronounced in high-dimensional settings (p = 100) and under low-signal conditions (), where gp yielded CV reductions of up to 70–80% relative to bisection and gp-bs. With κ = 20, CV plateaued at approximately B = 1000. Figures S2 and S3 (Additional File 1) display CV and computational time for the calibration slope and metrics respectively, and the full numerical results are reported in Table 4.

Table 4.

Estimated minimum sample size () and stability (CV) for continuous outcomes using assurance aggregation (20% quantile) and mean aggregation, stratified by target metric, large-sample , and number of predictors (p), at B = 1000 and κ = 20. Results are averaged over 100 simulation replicates. Values in parentheses after each CV are the Monte Carlo standard error of the CV

Metric		p	Engine	Mean		CV (MC SE)
				Assurance	Mean agg.	Assurance	Mean agg.
	0.2	10	bisection	243	191	11.20 (0.81)	10.46 (0.75)
			gp	237	191	6.65 (0.47)	3.62 (0.26)
			gp-bs	242	188	8.51 (0.61)	6.71 (0.48)
		100	bisection	1934	1741	7.04 (0.50)	8.33 (0.60)
			gp	1954	1721	1.85 (0.13)	3.46 (0.25)
			gp-bs	1907	1735	7.97 (0.57)	7.18 (0.51)
	0.7	10	bisection	99	77	8.56 (0.61)	11.31 (0.81)
			gp	100	79	2.52 (0.18)	6.22 (0.44)
			gp-bs	99	78	6.17 (0.44)	6.37 (0.45)
		100	bisection	786	710	6.30 (0.45)	6.26 (0.45)
			gp	785	707	0.86 (0.06)	0.61 (0.04)
			gp-bs	778	711	5.72 (0.41)	5.08 (0.36)
Calib. Slope	0.2	10	bisection	86	190	17.56 (1.29)	8.03 (0.57)
			gp	91	191	10.21 (0.73)	3.55 (0.25)
			gp-bs	84	187	12.02 (0.87)	7.01 (0.50)
		100	bisection	4676	1741	12.59 (0.91)	8.50 (0.61)
			gp	4530	1732	6.30 (0.45)	4.34 (0.31)
			gp-bs	4594	1745	12.54 (0.90)	8.46 (0.61)
	0.7	10	bisection	598	77	7.51 (0.54)	7.71 (0.55)
			gp	582	78	3.71 (0.26)	1.81 (0.13)
			gp-bs	580	77	5.13 (0.37)	5.28 (0.38)
		100	bisection	572	715	8.56 (0.61)	5.94 (0.42)
			gp	568	706	5.35 (0.38)	0.49 (0.03)
			gp-bs	583	714	4.33 (0.31)	5.50 (0.39)

Survival outcomes

Figure 2 compares sample size precision (CV) and computational time for the calibration slope metric across design configurations: , event rate (0.4, 0.8), , and B = 200–2000. The gp engine consistently yielded lower and more stable CV values than both bisection and gp-bs, with the relative advantage most marked in the more challenging low-event-rate scenarios (0.4). Performance improvements were more systematic under mean-based aggregation. CV values approached a plateau at B ≈ 1000 for κ = 20 under both aggregation schemes. The corresponding results for the C-index metric are reported in Fig. S4 (Additional File 1).

Fig. 2.

Aim 1 (Survival outcome): Comparison of CV and computational time across search engines gp, bisection and gp-bs for calibration slope metric under varying (κ = 10, 20), event rate (0.4, 0.8), number of predictors (p = 10, 100) and total simulation budget (B = 200–2000)

Under B = 1000 (Table 5), gp consistently achieved the lowest CV for C-index estimation, often by a substantial margin, with the advantage most evident in low-event-rate (0.4), high-dimensional (p = 100) scenarios. The only exception arose under assurance-based aggregation with p = 10 and event rate 0.4, where gp-bs produced a slightly smaller CV. Calibration slope estimation was more demanding, yet gp still consistently achieved the lowest CV across all scenarios. The bisection procedure systematically underperformed under nearly all evaluated conditions.

Table 5.

Estimated minimum sample size and stability (CV) for survival outcomes using assurance aggregation (20% quantile) and mean aggregation, stratified by target metric, event rate and number of predictors (p), at B = 1000 and κ = 20. Results are averaged over 100 simulation replicates. Values in parentheses after each CV are the Monte Carlo standard error of the CV

Metric	Event rate	p	Engine	Mean		CV (MC SE)
				Assurance	Mean agg.	Assurance	Mean agg.
C-index	0.4	10	bisection	60	47	11.74 (0.85)	10.80 (0.78)
			gp	62	48	8.48 (0.61)	4.77 (0.34)
			gp-bs	62	48	6.77 (0.48)	4.82 (0.34)
		100	bisection	540	489	4.55 (0.32)	3.58 (0.25)
			gp	543	491	2.38 (0.17)	0.47 (0.03)
			gp-bs	541	491	3.22 (0.23)	2.44 (0.17)
	0.8	10	bisection	38	31	12.05 (0.87)	10.71 (0.77)
			gp	41	32	4.69 (0.33)	4.43 (0.32)
			gp-bs	40	32	4.78 (0.34)	3.65 (0.26)
		100	bisection	334	304	5.04 (0.36)	3.12 (0.22)
			gp	334	304	0.86 (0.06)	0.48 (0.03)
			gp-bs	334	304	2.39 (0.17)	1.97 (0.14)
Calib. Slope	0.4	10	bisection	486	268	25.79 (1.95)	20.96 (1.55)
			gp	479	270	21.00 (1.56)	12.94 (0.93)
			gp-bs	471	267	17.49 (1.28)	12.67 (0.91)
		100	bisection	3524	2992	11.84 (0.85)	8.00 (0.57)
			gp	3403	2990	5.92 (0.42)	3.78 (0.27)
			gp-bs	3504	2941	8.37 (0.60)	6.53 (0.47)
	0.8	10	bisection	328	178	22.46 (1.67)	17.43 (1.28)
			gp	324	171	10.53 (0.76)	5.63 (0.40)
			gp-bs	315	175	12.86 (0.93)	11.48 (0.83)
		100	bisection	2277	1863	9.07 (0.65)	7.85 (0.56)
			gp	2198	1846	5.36 (0.38)	3.88 (0.28)
			gp-bs	2225	1878	6.78 (0.48)	5.06 (0.36)

The Monte Carlo standard errors (Tables 3, 4 and 5) are small relative to the differences in CV between the search engines, indicating that S = 100 replicates yield sufficient precision to distinguish their performance reliably. The fluctuations in CV across the simulation budget B that are visible in some panels of (Figs. 1, 2, S2) around B = 500 are non-systematic and are most pronounced when the number of training datasets simulated per candidate sample size is small (κ = 10). These fluctuations arise from the instability of the surrogate-based search in the low-budget regime, where the number of evaluations is insufficient to fit the Gaussian process reliably, rather than reflecting any substantive feature of the underlying sample-size–performance relationship. This instability is an inherent characteristic of the low-budget setting and therefore manifests across many scenarios simultaneously, even though the direction and magnitude of any particular fluctuation are random. As B and κ increase, the trajectories become smoother, achieving approximate stability by B ≈ 1000 at κ = 20.

Overall ranking

Table 6 summarises the comparative performance of the three search algorithms across all 12 outcome–aggregation–metric configurations. gp achieves the lowest mean rank in 11 of the 12 configurations and is overall the best-performing search engine, achieving a perfect average rank of 1.00 in nine configurations. The sole exception arises for the calibration slope under binary outcome with assurance-based aggregation, where gp-bs attains the best rank. The bisection procedure is ranked last or tied for last in every configuration, consistent with the inefficiency of deterministic bisection in stochastic optimisation settings.

Table 6.

Average rank of search engines across outcomes, aggregation methods, and target metrics at B = 1000. Lower average rank indicates better overall performance (CV)

Outcome	Aggregation method	Metric	bisection	gp	gp-bs
Binary	Assurance	AUC	3.00	1.00	2.00
		Calib. Slope	2.75	1.75	1.50
	Mean	AUC	3.00	1.00	2.00
		Calib. Slope	2.75	1.25	2.00
Continuous	Assurance		2.75	1.00	2.25
		Calib. Slope	3.00	1.00	2.00
	Mean		2.75	1.00	2.25
		Calib. Slope	3.00	1.00	2.00
Survival	Assurance	C-index	2.50	1.25	2.25
		Calib. Slope	2.75	1.00	2.25
	Mean	C-index	3.00	1.00	2.00
		Calib. Slope	2.50	1.00	2.50

Aim 2: Benchmark comparison

Binary outcome

Table 7 reports the percentage deviation of achieved performance from target for binary outcome models with p = 20. pmsims (mean) and samplesizedev exhibited the most stable performance, with deviations typically within of the target. pmsampsize generally suggested the smallest sample sizes; although it occasionally achieved performance close to the target, it showed substantial negative deviations in several scenarios, most pronounced at the highest target discrimination (AUC 0.9), where deviations reached -7.24% (prevalence 5%) and -9.84% (prevalence 20%). pmsims (assurance) recommended the largest sample sizes and produced consistently minor negative deviations (ranging from -1.05% to -2.00%). Supplementary figures showing minimum sample size requirements, coefficient of variation, computational time, and achieved calibration performance across all predictor counts and scenarios are provided in Figs. S5–S8 (Additional File 1).

Table 7.

Comparison of sample size requirements and achieved performance deviations for binary outcomes across two prevalence levels (0.05, 0.20) and two target large-sample AUC values (0.8, 0.9), with p = 20. Performance evaluated for calibration slope

Metric	Prevalence	Large-sample AUC	Engine			% Deviation
Calib. slope	0.05	0.8	pmsims (assurance)	5562	0.882	−2.00
			pmsims (mean)	3403	0.906	0.70
			pmsampsize	2788	0.888	−1.34
			samplesizedev	3346	0.903	0.34
		0.9	pmsims (assurance)	4080	0.890	−1.16
			pmsims (mean)	2352	0.911	1.18
			pmsampsize	1303	0.835	−7.24
			samplesizedev	2331	0.908	0.91
	0.20	0.8	pmsims (assurance)	2024	0.884	−1.75
			pmsims (mean)	1239	0.898	−0.25
			pmsampsize	882	0.863	−4.14
			samplesizedev	1245	0.902	0.21
		0.9	pmsims (assurance)	1674	0.891	−1.05
			pmsims (mean)	986	0.898	−0.24
			pmsampsize	509	0.811	−9.84
			samplesizedev	1005	0.904	0.42

Continuous outcome

Table 8 presents the percentage deviation of observed performance from prespecified targets for continuous outcome models with p = 20. pmsims (mean) achieved high precision, with deviations within at and 0.7, and a modest positive deviation of +0.98% at . The assurance-based variant produced slight negative deviations (-0.12% to -0.42%). In contrast, pmsampsize showed systematic positive deviations that increased strongly with target : from +0.93% at to +7.53% at , reflecting the fact that in higher-signal settings the pmsampsize sample size is not driven by shrinkage, resulting in calibration slopes that exceed the target. Supplementary figures are provided in Figs. S9–S12 (Additional File 1).

Table 8.

Comparison of sample size requirements and achieved performance deviations for continuous outcomes across three target levels (0.2, 0.5, 0.7), with p = 20. Performance evaluated for calibration slope

Metric	Large-sample	Engine			% Deviation
Calib. slope	0.2	pmsims (assurance)	1196	0.897	−0.39
		pmsims (mean)	702	0.909	0.98
		pmsampsize	716	0.908	0.93
	0.5	pmsims (assurance)	320	0.899	−0.12
		pmsims (mean)	189	0.899	−0.09
		pmsampsize	254	0.928	3.09
	0.7	pmsims (assurance)	151	0.896	−0.42
		pmsims (mean)	95	0.900	0.03
		pmsampsize	254	0.968	7.53

Survival outcome

Table 9 compares the sample size requirements and achieved calibration slope deviations for survival outcome models with p = 20. All three methods achieved calibration slopes close to the target of 0.90 across the full range of event rates and target C-index values. pmsims (mean) produced the smallest deviations overall, consistently within across all scenarios. pmsims (assurance) showed modest negative deviations, ranging from -0.41% to -1.47%, reflecting its conservative sample size recommendations. samplesizedev performed comparably at event rates of 0.4 and 0.5, with deviations between -0.94% and +3.19%, though positive deviations at C-index = 0.9 (up to +3.19% at event rate 0.4 and +2.78% at event rate 0.5) suggest mild overestimation of the required sample size in high-discrimination settings. At event rate 0.8, all three methods performed well, with deviations within . Supplementary figures are provided in Figs. S13–S16 (Additional File 1).

Table 9.

Comparison of sample size requirements and achieved performance deviations for survival outcomes across three event rates (0.4, 0.5, 0.8) and three target C-index values (0.7, 0.8, 0.9), with p = 20. Performance evaluated for calibration slope

Metric	Event rate	Large-sample C-index	Engine			% Deviation
Calib. slope	0.4	0.7	pmsims (assurance)	1569	0.888	−1.34
			pmsims (mean)	963	0.896	−0.41
			samplesizedev	925	0.892	−0.94
		0.8	pmsims (assurance)	889	0.891	−1.03
			pmsims (mean)	584	0.908	0.87
			samplesizedev	544	0.895	−0.52
		0.9	pmsims (assurance)	680	0.893	−0.79
			pmsims (mean)	478	0.901	0.16
			samplesizedev	656	0.929	3.19
	0.5	0.7	pmsims (assurance)	1283	0.887	−1.47
			pmsims (mean)	808	0.902	0.26
			samplesizedev	802	0.899	−0.11
		0.8	pmsims (assurance)	756	0.894	−0.72
			pmsims (mean)	493	0.898	−0.21
			samplesizedev	472	0.897	−0.33
		0.9	pmsims (assurance)	590	0.896	−0.41
			pmsims (mean)	408	0.903	0.33
			samplesizedev	535	0.925	2.78
	0.8	0.7	pmsims (assurance)	919	0.891	−1.01
			pmsims (mean)	607	0.901	0.10
			samplesizedev	604	0.899	−0.11
		0.8	pmsims (assurance)	574	0.901	0.12
			pmsims (mean)	353	0.895	−0.60
			samplesizedev	374	0.901	0.11
		0.9	pmsims (assurance)	436	0.896	−0.48
			pmsims (mean)	292	0.901	0.15
			samplesizedev	340	0.915	1.67

Discussion

Researchers developing clinical prediction models face the critical challenge of justifying their chosen sample sizes, yet practical guidance on which sample size calculation method to use remains limited. In this study, we evaluated three simulation-based search algorithms implemented in the pmsims package—gp (Gaussian process-based), a deterministic bisection procedure, and a hybrid gp-bs approach—for estimating the minimum development sample size required for prediction models with binary, continuous, and time-to-event outcomes. We assessed each method under both mean-based and assurance-based criteria across a wide range of scenarios varying outcome prevalence or event rate, predictor dimensionality, and target performance metrics. We also benchmarked the best-performing method against established R packages for sample size calculation.

Simulation findings and recommendations

In all scenarios, the GP-based gp search engine produced the most precise estimates of the minimum sample size (lowest CV) when compared with a classical bisection search and the hybrid gp-bs approach. The advantage was especially large in challenging scenarios characterised by low signal (low or low event prevalence) and high dimensionality. This finding aligns with recent methodological advances in stochastic root-finding, where adaptive sampling methods like the Probabilistic Bisection Algorithm maintain near-optimal convergence rates even in low-signal regimes [24, 25].

Increasing the number of simulation replicates per candidate sample size (κ) substantially stabilised the surrogate model updates and reduced the CV of the estimated ; κ = 20 (rather than κ = 10) yielded markedly lower CV for gp. A total simulation budget of roughly B ≈ 1000 was an efficient trade-off between precision and cost, with only modest performance improvements beyond this threshold. This efficiency gain reflects the sequential nature of the GP-based search, which automatically concentrates simulation effort in regions where greater precision is needed [25].

As expected, the assurance criterion produced more conservative recommended sample sizes than mean-based targets. gp produced reliable assurance-based recommendations but required higher simulation effort to achieve comparable precision to mean-based estimates. The hybrid gp-bs sometimes achieved marginally better precision than gp for calibration targets under assurance in limited binary settings, reflecting the benefit of a coarse deterministic range-finding step when the performance curve is particularly noisy.

The deterministic bisection search consistently underperformed, requiring many more model evaluations to achieve comparable stability. This supports the view that deterministic interval halving is not well suited to stochastic sample size searches unless a very large evaluation budget is available [25].

When compared with pmsampsize (analytical) and samplesizedev (simulation-based), the pmsims gp engine produced recommended sample sizes that achieved the prespecified discrimination and calibration targets when evaluated on large independent validation sets. Analytical formulae can be sensitive to modelling assumptions and may under- or overestimate the required sample sizes in some high model strength settings [2], whereas simulation-based approaches can more directly reflect performance variability under the assumed data-generating mechanism. Our findings suggest that the GP-based adaptive search implemented in gp offers a compelling middle ground: it achieves the precision of simulation-based approaches while maintaining comparable sample efficiency through sequential learning.

Limitations and implications for future research

Our validation study focused on data-generating mechanisms with continuous standardised predictors and seven levels of model dimensionality (p = 5, … , 100), where all candidate predictors corresponded to true signal variables, and on three standard model classes (logistic, linear, and Cox regression). This design does not fully capture real-world complexities such as mixtures of discrete and continuous covariates, correlated noise features, missing data, or heavy-tailed distributions. Accordingly, our conclusions, in particular the relative ranking of the search engines and the recommended budget and replication settings, should be read as applying to these idealised, well-specified conditions, and the broader flexibility of pmsims across model types and data generators, while supported by its design, has not been empirically established here. Additional data-generating mechanisms incorporating realistic correlation structures and explicit noise predictors will be required to more fully characterise the generalisability of the findings, and form the focus of a separate forthcoming benchmarking study.

The specification of the initial value can affect numerical stability. Analytical procedures such as pmsampsize, which rely on closed-form expressions, are invariant to the choice of starting values. In contrast, simulation-based procedures can be sensitive to their initial search region. The pmsims gp engine employs adaptive, data-driven initial values (see Adaptive starting value search for pmsims engines section) to concentrate the search within informative regions of the sample-size domain; early iterations may nevertheless exhibit instability when the performance surface is flat or multimodal.

An additional consideration, relevant to all the three pmsims engines, pertains to the preliminary adaptive search for a starting sample size. This step assumes that model performance does not deteriorate as the size of the development sample increases; informally, that a larger sample size is never detrimental and that, beyond a certain point, the target performance level is achieved and subsequently maintained (Adaptive starting value search for pmsims engines section). For the correctly specified regression models investigated in this work, this assumption is plausible: as n increases, the maximum-likelihood coefficient estimates become more precise, and the fitted model converges towards the true data-generating process. Consequently, expected discrimination and calibration approach their large-sample limits [3, 26, 27]. However, this remains an assumption rather than a formal guarantee, and it need not hold for all performance metrics or model classes. For more flexible or misspecified models, for instance, a linear model fitted to data generated by a strongly non-linear process, or tree-based and other machine-learning methods out-of-sample performance may display non-monotonic or otherwise irregular behaviour as a function of n [28].

To reduce the impact of such violations, the package incorporates a basic safeguard: the search for the starting value terminates after a fixed number of doubling/halving iterations, or when the candidate sample size can no longer be updated. This ensures that, when the target performance is never attained, the procedure returns a well-defined boundary solution rather than continuing an unbounded search (Algorithm 4).

Conclusion

The pmsims package uses a GP-based gp search engine to provide a computationally efficient and flexible framework for principled sample size planning in clinical prediction modelling. Our evaluation shows that adaptive surrogate-based search substantially outperforms deterministic bisection approaches and achieves comparable performance relative to established analytical and simulation-based tools while requiring fewer evaluations. With pragmatic adaptive starting limits, sufficient replication per evaluation (), and transparent reporting of uncertainty around the estimated minimum n, simulation-based planning can improve the reliability of prediction models developed in practice.

Future work should prioritise: (1) more robust surrogate modelling under non-monotonic or highly variable performance curves, (2) improved assurance-based estimation to reduce simulation budgets for quantile targets, and (3) broader validation across more realistic data-generating mechanisms and model classes, including correlated covariates, noise predictors, missing data, and predictive uncertainty.

Authors' contributions

ORO, GF, and EC designed the simulation study. ORO conducted the simulation study, analysed the results, and drafted the manuscript. ORO, DSh, FZ, GF, and EC developed the pmsims package. DSh, SM, FZ, DSt, GF, and EC contributed to the conceptualisation and study design, interpretation of results, and critical revision of the manuscript. GF and EC provided oversight of the project.

Funding

The pmsims project, including this study, is funded by the NIHR Research for Patient Benefit programme (NIHR206858). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. ORO is supported by NIHR206858 at King’s College London. This study is part-funded by the National Institute for Health and Care Research (NIHR) Maudsley Biomedical Research Centre (BRC). DSt, DSh, GF, and EC received financial support from the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust (NIHR203318) and King’s College London.

Data availability

No empirical datasets were generated or analysed in this study. All simulation results are reproducible using the pmsims R package (version 0.5.0) with the simulation scenarios and parameter settings described in the Methods section.

Declarations

Ethics approval and consent to participate

Not applicable. This study involved no human participants, human data, or animal subjects; all data were generated by computer simulation.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.