Sensitivity Analyses of Clinical Trial Designs: Selecting Scenarios and Summarizing Operating Characteristics

Abstract

The use of simulation-based sensitivity analyses is fundamental for evaluating and comparing candidate designs of future clinical trials. In this context, sensitivity analyses are especially useful to assess the dependence of important design operating characteristics with respect to various unknown parameters. Typical examples of operating characteristics include the likelihood of detecting treatment effects and the average study duration, which depend on parameters that are unknown until after the onset of the clinical study, such as the distributions of the primary outcomes and patient profiles. Two crucial components of sensitivity analyses are (i) the choice of a set of plausible simulation scenarios and (ii) the list of operating characteristics of interest. We propose a new approach for choosing the set of scenarios to be included in a sensitivity analysis. We maximize a utility criterion that formalizes whether a specific set of sensitivity scenarios is adequate to summarize how the operating characteristics of the trial design vary across plausible values of the unknown parameters. Then, we use optimization techniques to select the best set of simulation scenarios (according to the criteria specified by the investigator) to exemplify the operating characteristics of the trial design. We illustrate our proposal in three trial designs.

1. Introduction

Clinical trial designs are becoming increasingly complex to meet the multifaceted needs and goals of precision medicine. Examples of complex designs include adaptive seamless phase i/ii designs for evaluating, early in the treatment development process, the dosing, safety, and activity of new drugs (Hobbs et al., 2019). Also, adaptive randomized trials with frequent interim looks at the data can evaluate one or more therapies simultaneously while attempting to minimize trial duration and resources (Thorlund et al., 2018; Berry et al., 2010). Additional examples of complex designs have been implemented in biomarker-stratified trials to evaluate the efficacy of a therapy and possible variations of treatment effects across patient subgroups (Mehta et al., 2019).

When planning a new trial, it is necessary to predict and evaluate several operating characteristics. Relevant operating characteristics can include the likelihood of selecting an effective dose with low toxicity in a phase i/ii study, the probability of detecting treatment effects in a randomized study, the expected trial duration, costs, and other metrics to evaluate designs that often enroll patients from different subgroups. Multiple operating characteristics typically need to be examined jointly in order to evaluate the relevant trade-offs achieved by candidate designs, such as balancing the accuracy in estimating treatment effects and the expected study duration.

The obvious challenge for evaluating a candidate design is that the vector of operating characteristics of the study design is not known and it is difficult to estimate before the onset of the trial. Indeed, the operating characteristics are usually a function of a vector of unknown parameters that identify the distribution of all relevant variables to be captured during the trial. For example, unknown parameters can include the enrollment and drop-out rates, the magnitude of treatment effects, and the prevalence of predictive biomarkers in the trial population. Uncertainty on these parameters makes it non-trivial to evaluate whether a candidate design is appropriate for implementing the new study.

Sensitivity analyses are commonly used to account for uncertainty on unknown parameters and operating characteristics when evaluating a candidate design. They typically proceed in three steps. First, a set of plausible scenarios, i.e., specific values of the vector of unknown parameters, is selected. Next, the corresponding operating characteristics are computed using trial simulations or analytic results. Finally, based on the computed operating characteristics and their variations across the set of scenarios, the investigators evaluate whether the candidate design is appropriate to achieve the aims of the study. Throughout the manuscript, we use the terms sensitivity analysis or simulation report to indicate a set of scenarios and the associated operating characteristics which are computed to illustrate how the operating characteristics vary across plausible values of unknown parameters.

Producing a simulation report to effectively evaluate a study design has been recommended as one of the key supporting documents for interacting with the FDA (Mayer et al., 2019; Food et al., 2020). However, it can be difficult to select the set of unknown parameters, especially if the dimension of the vector of unknown parameters is moderate to high (say ≥ 5). For the investigators, it might be unclear if the selected scenarios are adequate to illustrate the variations of the operating characteristics across potential values of the unknown parameters. Similarly, for regulators, there may be skepticism as to whether the selected scenarios are chosen to highlight positive aspects of the trial design without pointing at its limitations and negative aspects (Razavi et al., 2021). Another subtle challenge is the choice of the number of scenarios. Indeed, a large number of scenarios (say 100) may simplify the task of representing how the operating characteristics vary across potential values of the unknown parameters, but a simulation report that contains too many scenarios makes it difficult to interpret and communicate the included results.

We propose a method to choose an optimal set of scenarios for a simulation report that will provide relevant operating characteristics. This decision is based on a utility criterion, which formalizes the ability of any set of scenarios to represent the map between the unknown parameters and the operating characteristics. In some cases, we will consider a restriction of the parameter space to focus only on regions of plausible values of the unknown parameters. The utility criterion assigns high (low) utility to a set of scenarios if the table of potential unknown parameters and operating characteristics is an accurate (inaccurate) summary of how the design's operating characteristics vary across the considered parameter space. We call the set of scenarios that maximizes the utility criterion the Representative and Optimal Sensitivity Analysis (ROSA) scenarios. To select the ROSA scenarios, we introduce a computational procedure that leverages (i) flexible regression methods like neural networks (NNs) (Goodfellow et al., 2016) and (ii) optimization algorithms like simulated annealing (Bélisle, 1992). Our approach is applicable to any trial design, regardless of the number of unknown parameters and the number of operating characteristics.

In summary, we propose ROSA as a computational tool that allows one to examine any clinical trial design by selecting a parsimonious set of simulation scenarios with the goal of representing the variations of the operating characteristics across plausible values of unknown parameters. To illustrate this approach, we conduct sensitivity analyses for three trial designs. The first is a two-arm randomized design that aims to test and estimate the effects of an experimental treatment compared to the standard of care (SOC). The second is a multi-stage randomized trial that leverages an auxiliary/surrogate outcome $S$ measured shortly after randomization for interim decisions and a primary outcome $Y$ with a longer ascertainment time (Niewczas et al., 2019). The third is a biomarker-adaptive enrichment design similar to the design of the TAPPAS trial (Mehta et al., 2019), a randomized phase iii trial comparing TRC105 and pazopanib versus pazopanib alone in patients with advanced angiosarcoma (Jenkins et al., 2011; Jones et al., 2017). In the first design, we consider a single unknown parameter and a single operating characteristic, whereas for the latter two designs we consider multiple unknown parameters and multiple operating characteristics.

2. Selecting sensitivity scenarios

2.1. Notation and problem set-up

We introduce our procedure to select $K$ sensitivity scenarios ${\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\in \Theta$, where $\Theta$ is the set of potential values of the unknown parameters $\mathit{\theta }$. We assume that $\Theta$ is a bounded subset of ${ℝ}^d$ and use the notation ${‖\cdot ‖}_2$ to indicate the Euclidean norm on ${ℝ}^d$. We will restrict $\Theta$ to a subset ${\Theta }^{\prime }$ when there is sufficient prior information from completed studies or clinical experience. We identify ROSA scenarios ${\mathit{\theta }}_1^{\text{*}},\dots ,{\mathit{\theta }}_K^{\text{*}}$ as the scenarios that maximize a utility criterion $\mathcal{U}$

$${\mathit{\theta }}_1^{\text{*}},\dots ,{\mathit{\theta }}_K^{\text{*}}=\underset{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K}{\text{argmax}\hspace{0.17em}}\mathcal{U}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right),$$ (1)

where

$$\mathcal{U}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)=-\underset{{\mathit{\theta }}^{\prime }\in \Theta }{\text{max}}\left\{\underset{k=1,\dots ,K}{\text{min}}D\left[f\left({\mathit{\theta }}^{\prime }\right),f\left({\mathit{\theta }}_k\right)\right]\right\}.$$ (2)

We can symmetrically define the corresponding loss function $\mathcal{L}=-\mathcal{U}$ by inverting the sign in equation (2). Here, $D\left[f\left({\mathit{\theta }}^{\prime }\right),f\left({\mathit{\theta }}_k\right)\right]$ is a metric between the operating characteristics $f\left({\mathit{\theta }}^{\prime }\right)=\left(f_1\left({\mathit{\theta }}^{\prime }\right),\dots f_R\left({\mathit{\theta }}^{\prime }\right)\right)$ and $f\left({\mathit{\theta }}_k\right)=\left(f_1\left({\mathit{\theta }}_k,\right)\dots ,f_R\left({\mathit{\theta }}_k\right)\right)$. We will consider metrics of the form

$$D\left[f\left({\mathit{\theta }}^{\prime }\right),f\left({\mathit{\theta }}_k\right)\right]=\sum _{r=1}^R{w}_r{‖f_r\left({\mathit{\theta }}^{\prime }\right)-f_r\left({\mathit{\theta }}_k\right)‖}_2,$$

where $w_1,\dots ,w_R$ are non-negative weights that sum to one. The weights can be user-specified to calibrate the relative importance of different operating characteristics. Setting the weights to $1/R$ results in equal weighting for each operating characteristic.

We can now provide an explicit interpretation of the utility function $\mathcal{U}$ in equation (2). Consider a set of scenarios $\left\{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right\}$ – the order of the entries is not relevant – and an arbitrary scenario ${\mathit{\theta }}^{\prime }$ in $\Theta$. For $1\le k\le K$, the metric $D\left[f\left({\mathit{\theta }}^{\prime }\right),f\left({\mathit{\theta }}_k\right)\right]$ is a summary of the differences between the operating characteristics at ${\mathit{\theta }}^{\prime }$ and the same operating characteristics when we consider the $k$-th scenario ${\mathit{\theta }}_k$. Therefore, ${\text{min}}_{k=1,\dots ,K}D\left[f\left({\mathit{\theta }}^{\prime }\right),f\left({\mathit{\theta }}_k\right)\right]$ can be viewed as an approximation error between $f\left({\mathit{\theta }}^{\prime }\right)$ and a similar vector of operating characteristics selected among our $K$ options $f\left({\mathit{\theta }}_1\right),\dots ,f\left({\mathit{\theta }}_K\right)$. Expression (2) identifies through the maximization operator the worst-case (with highest approximation error) that we can obtain by varying ${\mathit{\theta }}^{\prime }$ in $\Theta$. We maximize the utility function $\mathcal{U}$ and use ${\mathit{\theta }}_1^{\text{*}},\dots ,{\mathit{\theta }}_K^{\text{*}}$ to indicate the ROSA scenarios. Alternative utility criteria and loss functions are described later in the manuscript. A table of notation used throughout the paper is provided in Table 1 below.

Table 1.

Notation

$\mathbf{\Theta }$	≜	Unknown parameter space in ${ℝ}^d$
${\mathbf{\Theta }}^{\prime }$	≜	Restricted unknown parameter subspace by prior knowledge in ${ℝ}^d$
${\mathbf{\Theta }}_{re}^{\prime }$	≜	Restricted unknown parameter subspace by prior knowledge and fixing certain dimensions in ${ℝ}^d$
${\mathbf{\Theta }}^F$	≜	Diffuse and finite unknown parameter subspace in ${ℝ}^d$
$\mathit{\theta }=\left({\theta }_1,\dots ,{\theta }_d\right)$	≜	$d$-dimensional vector of unknown parameters
${\mathit{\theta }}^{\mathit{t}}=\left({\mathit{\theta }}_1^t,\dots ,{\theta }_d^t\right)$	≜	$d$-dimensional training vector of unknown parameters
${\mathit{\theta }}^{\mathit{v}}=\left({\theta }_1^v,\dots ,{\theta }_d^v\right)$	≜	$d$-dimensional validation vector of unknown parameters
$\left\{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right\}$	≜	A set of $K$ sensitivity scenarios
$\mathcal{S}=\left\{{\mathit{\theta }}_1^{*},\dots ,{\mathit{\theta }}_K^{*}\right\}$	≜	The ROSA set of $K$ sensitivity scenarios optimizing loss $\mathcal{L}$
${\mathcal{S}}_r=\left\{{\mathit{\theta }}_{1,r}^{*},\dots ,{\mathit{\theta }}_{K,r}^{*}\right\}$	≜	The ROSA set of $K$ sensitivity scenarios optimizing marginal loss ${\mathcal{L}}_r$
$\mathbf{\text{f}}\left(\mathit{\theta }\right)$	≜	$R$-vector of operating characteristics for unknown parameters $\mathit{\theta }$
$\widehat{f}(\mathit{\theta })$	≜	Estimated $R$ vector of operating characteristics for unknown parameters $\mathit{\theta }$
$\overline{f}(\mathit{\theta })$	≜	Average across $M$ simulations of the $R$ vector of operating characteristics for unknown parameters $\mathit{\theta }$
$\phi \left(Z_{j,m},{\mathit{\theta }}_j\right)$	≜	Generic function to capture if a null hypothesis has been rejected, where $Z_{j,m}$ is the $m$th trial under the $j$th scenario, ${\mathit{\theta }}_j$
$\mathcal{L}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)$	≜	Loss Function
$\mathcal{U}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)$	≜	Utility criterion
$w_1,\dots ,w_r$	≜	Fixed non-negative weights for operating characteristics $f_1,\dots ,f_r$
${\omega }_1,{\omega }_2$	≜	Weights for stage 1 and 2 p-values
$D\left[\cdot ,\hspace{0.17em}\cdot \right]$	≜	Pre-specified distance metric
$z_1^i,\dots ,z_K^i$	≜	Gaussian noise in iteration $i$ of simulated annealing
${\rho }_i$	≜	Acceptance probability in iteration $i$ of simulated annealing
$\Theta$	≜	Unknown parameter space in ${ℝ}^d$
$T_0,T_1,\dots T_I$	≜	Decreasing sequence of positive numbers (cooling schedule of simulated annealing)
$r$	≜	Multiplicative reduction factor for simulated annealing in (0, 1)
$U_i$	≜	Random variable distributed Uniform(0,1) for simulated annealing
$e$	≜	Enrollment rate in (0,∞)
$N_a$	≜	Planned number of patients on arm $a=0,1$ at the final analysis
$n_a$	≜	Planned number of patients on arm $a=0,1$ at the interim analysis
$\mathcal{S}$	≜	Binary auxiliary outcome
$Y$	≜	Primary outcome
${\rho }_a$	≜	Response probability $P\left(Y=1|A=a\right)$
$\text{Δ}=p_1-p_0$	≜	Treatment effect on $Y$
$q_a$	≜	Response probability $P\left(S=1|A=a\right)$
${\rho }_a$	≜	Correlation between $Y$ and $\mathcal{S}$ in $A=a$

2.2. An example with a geometric interpretation

To provide a geometric interpretation of the utility criterion $\mathcal{U}$, we illustrate how one set of $K$ scenarios can be preferable to a different set of $K$ scenarios (Figure 1 ). Specifically, suppose we aim to design a single-arm trial with an interim analysis that allows for early-stopping for futility. The goal of the trial is to compare the response rate of an experimental drug ${\theta }_1$ with that of the $\text{SOC}\hspace{0.17em}{\theta }_0$ at the end of the study. However, because study patients only receive the experimental drug, the response rate under the $\text{SOC}\hspace{0.17em}{\theta }_0$ is estimated $\left({\widehat{\theta }}_0\right)$ before the onset of the study, for example using data from a previous trial. At the interim analysis, the trial may stop for futility if the preliminary evidence of positive treatment effects ${\text{Δ}}_{interim}$ is insufficient to continue the study. During the final analysis, the null hypothesis $H_0:{\theta }_1\le {\widehat{\theta }}_0$ (the experimental therapy is not superior to the historical control) is tested against the alternative hypothesis $H_1:{\theta }_1>{\widehat{\theta }}_0$ (the experimental therapy is superior to the historical control). In this design, $\mathit{\theta }=\left({\theta }_0,{\theta }_1\right)$ are the unknown parameters, and $\Theta ={\left[0,1\right]}^2$. Suppose that there are two operating characteristics of interest: (i) $f_1$, the probability of a positive result ($H_0$ is rejected) and (ii) $f_2$, the expected sample size.

Fig. 1.

Geometric representation of an arbitrary scenario ${\mathit{\theta }}^{\prime }$ and two proposed sets of scenarios. (Left) Parameter space $\mathbf{\Theta }={[0,1]}^2$ with arbitrary scenario ${\mathit{\theta }}^{\prime }$ (orange triangle) and two sets of proposed scenarios $\left\{{\mathit{\theta }}_1^1,\dots ,{\mathit{\theta }}_6^1\right\}$ (blue points) and $\left\{{\mathit{\theta }}_1^2,\dots ,{\mathit{\theta }}_6^2\right\}$ (red points). (Right) The set of operating characteristics $\mathbf{\text{f}}\left(\mathbf{\Theta }\right)$ coincides with the irregular shape. The operating characteristics of ${\mathit{\theta }}^{\prime }$ and two sets of scenarios are illustrated. The radius of the dotted circles (with blue points as centers) is the value of the loss $\mathcal{L}$ associated with the blue points. ROSA scenarios minimize the loss $\mathcal{L}$, which in turn is equal to the radius of the dotted circles that cover the operating characteristic surface $\mathbf{\text{f}}\left(\mathbf{\Theta }\right)$.

The left panel of Figure 1 is a representation of $\Theta$. We are interested in the two operating characteristics of the single-arm design. Two sets of $K=6$ scenarios are proposed. The first set of scenarios $\left\{{\mathit{\theta }}_1^1,\dots ,{\mathit{\theta }}_6^1\right\}$ (blue points) is chosen by varying both unknown parameters at the same time, while the second set $\left\{{\mathit{\theta }}_1^2,\dots ,{\mathit{\theta }}_6^2\right\}$ (red points) is chosen by varying only ${\mathit{\theta }}_0$ while fixing the value of ${\mathit{\theta }}_1$. The two sets of scenarios, the corresponding operating characteristics, and associated loss $\mathcal{L}=-\mathcal{U}$ are represented in the right panel of Figure 1. The first set of scenarios (blue points) is preferred over the second set (red points) because it is more representative of the variation of the operating characteristics over $\Theta$. Geometrically, the loss $\mathcal{L}\left({\mathit{\theta }}_1^1,\dots ,{\mathit{\theta }}_6^1\right)$ associated with the blue points is identical to the minimum radius of the circles with centers $f\left({\mathit{\theta }}_1^1\right),\dots ,f\left({\mathit{\theta }}_6^1\right)$ (see Figure 1) necessary to cover the operating characteristics surface $f\left(\Theta \right)$.

2.3. Estimating the operating characteristics

We describe an algorithm to numerically approximate the operating characteristics $f\left(\mathit{\theta }\right)$ for every $\mathit{\theta }\in \Theta$. This is necessary to solve the optimization problem in equation (2). Indeed, in most cases the function $f\left(\mathit{\theta }\right)$ cannot be computed in closed form.

We briefly outline our four-step procedure. In the first step, we choose a large number $J$ (say $J=1000$) of training scenarios ${\mathit{\theta }}_1^t,\dots ,{\mathit{\theta }}_J^t$. In the second step, we use Monte Carlo simulations to obtain estimates $\overline{f}\left({\mathit{\theta }}_1^t\right),\dots ,\overline{f}\left({\mathit{\theta }}_J^t\right)$ of $f\left({\mathit{\theta }}_1^t\right),\dots ,f\left({\mathit{\theta }}_J^t\right)$. In the third step, we train a flexible regression model – we use NNs in our implementation – based on the data points $\left({\mathit{\theta }}_1^t,\overline{f}\left({\mathit{\theta }}_1^t\right)\right),\dots ,\left({\mathit{\theta }}_J^t,\overline{f}\left({\mathit{\theta }}_J^t\right)\right)$. The output of this step is a regression function $\widehat{f}\left(\mathit{\theta }\right)$ that is easy to compute at any $\mathit{\theta }\in \Theta$ and that approximates $f\left(\mathit{\theta }\right)$. In the fourth step, we validate the regression model based on $J^{\prime }$ (say $J^{\prime }=200$) independent simulations $\left({\mathit{\theta }}_1^v,\overline{f}\left({\mathit{\theta }}_1^v\right)\right),\dots ,\left({\mathit{\theta }}_{J^{\prime }}^v,\overline{f}\left({\mathit{\theta }}_{J^{\prime }}^v\right)\right)$. Steps 1–3 of this procedure are summarized in Algorithm 1. Step 4 is described in Algorithm 2.

In more detail, in step 1, to select the training scenarios ${\mathit{\theta }}_1^t,\dots ,{\mathit{\theta }}_J^t$, we randomly select $J$ scenarios in $\Theta$ using Latin hypercube sampling (LHS) (McKay et al., 2000). LHS generates $J$ scenarios by first partitioning the $d$ unknown parameter dimensions into $J$ non-overlapping intervals and selecting one value from each interval at random. The $J$ values obtained for the first unknown parameter ${\mathit{\theta }}_1$ are randomly paired with the $J$ values obtained for the second ${\mathit{\theta }}_2$, and so on, for all $d$ unknown parameters to form $J\hspace{0.17em}d$-tuples, which constitute the training scenarios ${\mathit{\theta }}_1^t,\dots ,{\mathit{\theta }}_J^t$.

In step 2, we estimate the operating characteristics of the trial design. In the paper, we focus on operating characteristics that can be defined as expected values, which are often of great interest, e.g., bias, power, duration of the trial, etc. One approach to handling unbounded operating characteristics (e.g. median squared error) is to apply simple transformations in such a way that these operating characteristics can be expressed as expected values, i.e., so that we can write down

$$\mathbf{\text{f}}\left(\mathit{\theta }\right)={\mathbb{E}}_{\mathit{\theta }}\left[\phi \left(Z,\mathit{\theta }\right)\right]$$

for some function $\phi$, where the random vector $Z$ represents the data generated during the trial – including the collection of treatment assignment indicators and realized patient outcomes – under scenario $\mathit{\theta }$. In practice, to estimate $\mathbf{\text{f}}\left(\mathit{\theta }\right)$, we proceed as follows. First, for each of the training scenarios ${\mathit{\theta }}_j^t,1\le j\le J$, we simulate $M$ (say $M=200$) clinical trials following the trial design. We then use the $M$ scenario-specific simulated trials to compute the estimate

$$\overline{\mathbf{\text{f}}}\left({\mathit{\theta }}_j^t\right)=M^{-1}\sum _{m=1}^M\phi \left(Z_{j,m},{\mathit{\theta }}_j^t\right),\hspace{1em}1\le j\le J,$$

where $Z_{j,m}$ is the $m$^th trial dataset simulated under the $j$^th training scenario ${\mathit{\theta }}_j^t$. Throughout the manuscript $\phi \left(Z,\mathit{\theta }\right)$ will take values in a compact set. For example, $\phi$ can be the indicator that captures if a null hypothesis of interest has been rejected at the end of the study, or the duration of the simulated trial. One possibility for handling unbounded operating characteristics (e.g. median squared error) is to apply monotone transformations, from the real line to the unit interval, that rescale the operating characteristics. In this case, the selection of representative scenarios would be influenced by the specific monotone map used to express the operating characteristic.

In step 3, we have only two inputs, the scenarios ${\mathit{\theta }}_j^t$ and the estimates $\overline{\mathbf{\text{f}}}\left({\mathit{\theta }}_j^t\right),\hspace{0.33em}1\le j\le J$, to fit a function $\widehat{\mathbf{\text{f}}}\left(\mathit{\theta }\right)$. For example, one could use NNs, splines (Bookstein, 1989), or Gaussian processes (Rasmussen, 2003). We use NN regression functions in our applications because these are easy to compute using widely available software and have been demonstrated to have good performances (Leshno et al., 1993; Hornik, 1991; Goodfellow et al., 2016).

$$\begin{array}{l}\overline{\underset{¯}{\begin{array}{l}\mathbf{\text{Algorithm 1}}\text{:Obtaining a function }\widehat{f}\text{ that approximates the operating characteristic}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\\ \text{function}\hspace{0.17em}\mathbf{\text{f}}\end{array}}}\\ \mathbf{\text{1}} \mathbf{\text{Input:}}\text{ Trial design, Parameter space}\hspace{0.17em}\mathbf{\text{Θ}},J,M\\ \mathbf{\text{2}} \text{Step} \mathrm{1:}\hspace{0.33em}\text{Select}\hspace{0.17em}J\hspace{0.17em}\text{scenarios}\hspace{0.17em}{\mathit{\theta }}_1^t,\dots ,{\mathit{\theta }}_J^t\in \mathbf{\text{Θ}}\\ 3\text{ Step }2:\text{for} \hspace{0.17em}j=\mathit{1}\hspace{0.17em}\text{to}\hspace{0.17em}J \text{do}\\ 4\text{ Simulate }M\hspace{0.17em}\text{trials}\\ 5\text{ Obtain approximate operating characteristics}\\ \overline{f}\left({\mathit{\theta }}_j^t\right)=M^{-1}\sum _{m=1}^M\phi \left(Z_{j,m},{\mathit{\theta }}_j^t\right),\hspace{0.17em}\text{where}\hspace{0.17em}\hspace{0.17em}\phi \hspace{0.17em}\text{is a function of }{Z}_{j,m},\hspace{0.17em}\hspace{0.17em}\text{the }{m}^{th}\hspace{0.17em}\text{trial}\hspace{0.17em}\text{dataset}\\ \text{simulated under the }{j}^{th}\text{ scenario}\\ \mathbf{\text{6}} \mathbf{\text{end}}\\ \text{7 Step 3: Obtain an approximation of the operating characteristics }f\text{ by training a}\\ \text{regression algorithm, for rexample a NN model, and using the data points}\hspace{0.33em}\\ \left({\mathit{\theta }}_j^t,\overline{\mathit{f}}\left({\mathit{\theta }}_j^t\right)\right),1\le j\le J\\ \underset{¯}{\mathbf{\text{8}} \mathbf{\text{Output:}}\text{Function}\hspace{0.17em}\widehat{\mathbf{\text{f}}}\left(\mathit{\theta }\right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}\end{array}$$

In step 4 (Algorithm 2), we investigate the differences between $f$ and $f$. Specifically, we first select at random $J^{\prime }$ validation scenarios ${\mathit{\theta }}_1^v,\dots ,{\mathit{\theta }}_{J^{\prime }}^v$ independently with respect to previous computations (step 1–3) and simulate $M^{\prime }$ trials (say $M^{\prime }=500$) for each ${\mathit{\theta }}_{j^{\prime }}^v,1\le j^{\prime }\le J^{\prime }$. Based on the results of the simulated trials, for each $J^{\prime }$, we then compute Monte Carlo estimates $\overline{\mathbf{\text{f}}}\left({\mathit{\theta }}_{j^{\prime }}^v\right)={M^{\prime }}^{-1}{\displaystyle \sum _{m^{\prime }=1}^{M^{\prime }}\phi \left(Z_{j^{\prime },m^{\prime }},{\mathit{\theta }}_{j^{\prime }}^v\right)}$ of the operating characteristics $\mathbf{\text{f}}\left({\mathit{\theta }}_{j^{\prime }}^v\right)$. For several important operating characteristics (e.g., average sample size, expected duration, power, type 1 error), the estimator $\overline{\mathbf{\text{f}}}\left({\mathit{\theta }}_{j^{\prime }}^v\right)={M^{\prime }}^{-1}{\displaystyle \sum _{m^{\prime }=1}^{M^{\prime }}\phi \left(Z_{j^{\prime },m^{\prime }},{\mathit{\theta }}_{j^{\prime }}^v\right)}$ is unbiased. Finally, we compare the estimates $\overline{\mathit{f}}\left({\mathit{\theta }}_{j^{\prime }}^v\right)$ and the independent estimates $\mathit{f}\left({\mathit{\theta }}_{j^{\prime }}^v\right)$. We use summary statistics and graphs to evaluate the differences $\mathit{f}\left({\mathit{\theta }}_{j^{\prime }}^v\right)-\overline{\mathit{f}}\left({\mathit{\theta }}_{j^{\prime }}^v\right)$. If the approximation $f\left({\mathit{\theta }}_{j^{\prime }}^v\right)$ is not adequate, we can use a different regression methodology, increase the number $\left(M,M^{\prime }\right)$ of trials, or increase the number $J$ of training scenarios in Algorithm 1.

$$\begin{array}{l}\overline{\underset{¯}{\mathbf{\text{Algorithm 2:}}\text{Validating the approximation of the operating characteristics}\hspace{0.17em}f\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}}\\ \underset{¯}{\begin{array}{l}\mathbf{1}\mathbf{\text{ Input:}}\text{Approximation of the operating characteristics}\hspace{0.17em}f\hspace{0.17em}\text{Trial design},\hspace{0.17em}{J}^{\prime }\\ \mathbf{2 }\text{Randomly select}\hspace{0.17em}{J}^{\prime }\hspace{0.17em}\text{scenarios}\hspace{0.17em}{\mathit{\theta }}_1^v,\dots ,{\mathit{\theta }}_{J^{\prime }}^v\in \mathbf{\Theta }\hspace{0.17em}\text{independently from previous}\\ \text{computations }\left(\text{Algorithm 1}\right)\\ \mathbf{3}\mathbf{\text{ for}}\hspace{0.17em}{j}^{\prime }=1\hspace{0.17em}to\hspace{0.17em}{J}^{\prime }\hspace{0.17em}\mathbf{\text{do}}\\ 4\text{ Simulate}\hspace{0.17em}{M}^{\prime }\hspace{0.17em}\text{trials}\hspace{0.17em}{Z}_{j^{\prime },m^{\prime }}\\ 5\text{ Compute}\hspace{0.17em}\overline{f}\left({\mathit{\theta }}_{j^{\prime }}^v\right)={M^{\prime }}^{-1}\sum _{m^{\prime }=1}^{M^{\prime }}\phi \left(Z_{j^{\prime },m^{\prime }},{\mathit{\theta }}_{j^{\prime }}^v\right)\\ 6\text{ Compute}\hspace{0.17em}f\left({\mathit{\theta }}_{j^{\prime }}^v\right)\\ \mathbf{\text{7}} \mathbf{\text{end}}\\ \mathbf{\text{8}} \mathbf{\text{Output:}}\text{Set of differences}\hspace{0.17em}\mathit{f}\left({\mathit{\theta }}_{j^{\prime }}^v\right)-\overline{\mathit{f}}\left({\mathit{\theta }}_{j^{\prime }}^v\right)\hspace{0.17em}\text{and scatterplots to jointly visualize the }\\ \text{operating characteristic estimates}\hspace{0.17em}\overline{\mathit{f}}\left({\mathit{\theta }}_{j^{\prime }}^v\right)\hspace{0.17em}\text{and the independent estimates}\hspace{0.17em}\\ \mathit{f}\left({\mathit{\theta }}_{j^{\prime }}^v\right),1\le j^{\prime }\le J^{\prime }\hspace{0.17em}\text{Compute summaries of the differences (e}.\text{g}.,\hspace{0.17em}\text{median, range, or }\\ \text{other descriptive statistics)}.\\ \mathbf{9}\mathbf{\text{ Interpretation:}}\text{Differences between}\hspace{0.17em}\overline{\mathit{f}}\left({\mathit{\theta }}_{j^{\prime }}^v\right)\hspace{0.17em}\text{and the independent estimates}\hspace{0.17em}\\ \mathit{f}\left({\mathit{\theta }}_{j^{\prime }}^v\right),1\le j^{\prime }\le J^{\prime },\hspace{0.17em}\text{consistently close to zero provide evidence}\hspace{0.17em}\text{that}\hspace{0.17em}\mathit{f}\hspace{0.17em}\text{is an accurate }\\ \text{approximation of}\hspace{0.17em}f\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\end{array}}\end{array}$$

2.4. Approximating the loss function

After computing $f$ (Algorithm 1 ) and validating its accuracy (Algorithm 2), we use it to approximate the loss function $\mathcal{L}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)$. To proceed, we choose a diffuse and finite subset of the parameter space ${\mathbf{\Theta }}^F\subset \mathbf{\Theta }$. For example ${\mathbf{\Theta }}^F$ can include 100,000 random points from a distribution with support $\mathbf{\Theta }$. When ${\mathbf{\Theta }}^F$ contains a large number of random points that are distributed over $\mathbf{\Theta }$, under minimal assumptions (e.g., compact $\mathbf{\Theta }$ and operating characteristics with bounded range),

$$\begin{array}{l}\mathcal{L}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)=\underset{\mathit{\theta }\in \mathbf{\Theta }}{\text{max}}\left\{\underset{k=1,\dots ,K}{\text{min}}D\left[\mathit{f}\left(\mathit{\theta }\right),\mathit{f}\left({\mathit{\theta }}_k\right)\right]\right\}\\ \approx \underset{{\mathit{\theta }}^{\prime }\in {\mathbf{\Theta }}^F}{\text{max}}\left\{\underset{k=1,\dots ,K}{\text{min}}D\left[\widehat{\mathit{f}}\left({\mathit{\theta }}^{\prime }\right),\widehat{\mathit{f}}\left({\mathit{\theta }}_k\right)\right]\right\}=\mathcal{L}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right).\end{array}$$

To summarize, we can approximate the loss function $\mathcal{L}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)$ over the entire parameter space $\mathbf{\Theta }$ by $\mathcal{L}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)$ using a diffuse and finite subset ${\mathbf{\Theta }}^F$.

2.5. Optimization by simulated annealing

We now aim to approximately minimize the loss function $\mathcal{L}$. To illustrate the need for approximate solutions, consider the setting of a single unknown parameter $\left(d=1\right)$, a finite $\mathbf{\Theta }$, and an easy-to-compute loss function $\mathcal{L}$. Even in this simple setting identifying ${\mathit{\theta }}_1^{\text{*}},\dots ,{\mathit{\theta }}_K^{\text{*}}\in \mathbf{\Theta }$ can be challenging. For example, to select $K=10$ representative scenarios ${\mathit{\theta }}_1^{\text{*}},\dots ,{\mathit{\theta }}_K^{\text{*}}$ from 1000 points $\left\{{\mathit{\theta }}_j;1\le j\le 1000\right\}=\mathbf{\Theta }$, the loss function $\mathcal{L}$ would need to be calculated for $2.63\times {10}^{23}$ different possible sets $\left\{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right\}$. In what follows, we describe the use of simulated annealing (Algorithm 3 ), a simple strategy to reduce the outlined computational burden, regardless if $\mathbf{\Theta }$ is finite or not Kirkpatrick et al. (1983); Bélisle (1992); Spall (2005).

The simulated annealing algorithm proceeds as follows. First, initial scenarios ${\mathit{\theta }}_1^1,\dots ,{\mathit{\theta }}_K^1$ are proposed, for example by sampling ${\mathit{\theta }}_1^1,\dots ,{\mathit{\theta }}_K^1$ from a probability distribution with support $\mathbf{\Theta }$. Then, iteratively For $1\le i\le I$, the current scenarios ${\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i$ are perturbed by adding to them zero-mean noise variables $z_1^i,\dots ,z_K^i$, thus obtaining new proposed scenarios ${\mathit{\theta }}_1{ }^{\prime },\dots ,{\mathit{\theta }}_K{ }^{\prime }$ (this step is represented by the "Perturb" operator in Algorithm 3). At each iteration, the proposed scenarios ${\mathit{\theta }}_1{ }^{\prime },\dots ,{\mathit{\theta }}_K{ }^{\prime }$ can either be accepted (i.e., $\left[{\mathit{\theta }}_1^{i+1},\dots ,{\mathit{\theta }}_K^{i+1}\right]\leftarrow \left[{\mathit{\theta }}_1{ }^{\prime },\dots ,{\mathit{\theta }}_K{ }^{\prime }\right]$) or rejected (i.e., $\left[{\mathit{\theta }}_1^{i+1},\dots ,{\mathit{\theta }}_K^{i+1}\right]\leftarrow \left[{\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right]$). The acceptance or rejection of the proposed scenarios is stochastic, with probability ${\rho }_i$ (defined below), which is a function of $\mathcal{L}\left({\mathit{\theta }}_1{ }^{\prime },\dots ,{\mathit{\theta }}_K{ }^{\prime }\right)$ and $\mathcal{L}\left({\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right)$.

The acceptance probability ${\rho }_i$ is equal to 1 when $\mathcal{L}\left({\mathit{\theta }}_1^{\prime },\dots ,{\mathit{\theta }}_K^{\prime }\right)<\mathcal{L}\left({\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right)$. That is, if the proposed scenarios decrease the current loss value, then the proposed scenarios are accepted. If instead $\mathcal{L}\left({\mathit{\theta }}_1^{\prime },\dots ,{\mathit{\theta }}_K^{\prime }\right)\ge \mathcal{L}\left({\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right)$, then ${\rho }_i$ is

$${\rho }_i=\text{exp}\left(\frac{\mathcal{L}\left({\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right)-\mathcal{L}\left({\mathit{\theta }}_1^{\prime },\dots ,{\mathit{\theta }}_K^{\prime }\right)}{T_i}\right),$$

where $T_i,0\le i\le I$, is a decreasing sequence of positive real numbers often called the "cooling schedule" of the algorithm. A common cooling schedule is $T_i=T_0\cdot r^{i-1}$, where $T_0$ is a constant and $r\in \left(0,1\right)$ is a multiplicative contraction, but other forms are possible (Spall, 2005). In our applications, we use a piecewise-constant cooling schedule (Husmann et al., 2017).

After simulating the outlined Markov Chain for a fixed number $I$ of iterations, the final set of scenarios $\left\{{\mathit{\theta }}_1^{I+1},\dots ,{\mathit{\theta }}_K^{I+1}\right\}$ approximately minimizes the loss function $\mathcal{L}$ (Bélisle, 1992). In our ROSA implementation, we use multiple independent replicates of Algorithm 3, with different initial scenarios ${\mathit{\theta }}_1^1,\dots ,{\mathit{\theta }}_K^1$, to investigate convergence of the random trajectory $\left[\left({\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right);i\ge 1\right]$. Intuitively, we evaluate if the replicated trajectories, with different starting values, terminates with nearly identical final vectors $\left[{\mathit{\theta }}_1^{I+1},\dots ,{\mathit{\theta }}_K^{I+1}\right]$ and negligible differences in the $\mathcal{L}\left({\mathit{\theta }}_1^{I+1},\dots ,{\mathit{\theta }}_K^{I+1}\right)$ values.

$$\begin{array}{l}\overline{\underset{¯}{\mathbf{\text{Algorithm 3:}}\text{Pseudocode for simulated annealing to obtain ROSA scenarios}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}}\\ \mathbf{\text{1}} \text{Initialize the values of}\hspace{0.17em}{\mathit{\theta }}_1^1,\dots ,{\mathit{\theta }}_K^1,\hspace{0.17em}\text{e}.\text{g}., \text{by sampling from a distribution over}\hspace{0.17em}\mathbf{\Theta }\\ \mathbf{2} \mathbf{\text{for}}\hspace{0.17em}i=\mathit{1}\hspace{0.17em}to/\mathbf{\text{do}}\\ 3\text{ New proposal}\hspace{0.17em}{\mathit{\theta }}_1^{\prime },\dots ,{\mathit{\theta }}_K{ }^{\prime }\leftarrow \text{Perturb}\left({\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right)\\ \mathbf{\text{4}} \mathbf{\text{if}}\hspace{0.17em}\mathcal{L}\left({\mathit{\theta }}_1{ }^{\prime },\dots ,{\mathit{\theta }}_K{ }^{\prime }\right)\le \mathcal{L}\left({\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right)\hspace{0.17em}\mathbf{\text{then}}\\ 5\text{ Define}\hspace{0.17em}{\mathit{\theta }}_j^{i+1}={\mathit{\theta }}_j{ }^{\prime }\hspace{0.17em}\text{for every}\hspace{0.17em}j=1,\dots ,K;\\ \mathbf{\text{6}} \mathbf{\text{else}}\\ 7\text{ Compute the acceptance probability}\hspace{0.17em}{\rho }_i=\text{exp}\left(\left[\mathcal{L}\left({\mathit{\theta }}_1^i,\dots ,{\mathit{\theta }}_K^i\right)-\mathcal{L}\left({\mathit{\theta }}_1^{\prime },\dots ,{\mathit{\theta }}_K^{\prime }\right)\right]/T_i\right)\\ 8\text{ Sample}\hspace{0.17em}{U}_i\sim \text{Uniform}\left(0,1\right)\\ 9\text{ If}\hspace{0.17em}{U}_i\le {\rho }_i,\hspace{0.17em}\text{define}\hspace{0.17em}{\mathit{\theta }}_j^{i+1}={\mathit{\theta }}_j{ }^{\prime }\hspace{0.17em}\text{for every}\hspace{0.17em}j=1,\dots ,K;\\ 10\text{ Otherwise}\hspace{0.17em}{\mathit{\theta }}_j^{i+1}={\mathit{\theta }}_j^i\hspace{0.17em}\text{for every}\hspace{0.17em}j=1,\dots ,K.\\ \mathbf{11}\mathbf{\text{ end}}\\ \underset{¯}{\mathbf{12}\mathbf{\text{ Output:}}\hspace{0.17em}{\mathit{\theta }}_1^{I+1},\dots ,{\mathit{\theta }}_K^{I+1},\mathcal{L}\left({\mathit{\theta }}_1^{I+1},\dots ,{\mathit{\theta }}_K^{I+1}\right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}\end{array}$$

3. Applications: Sensitivity analyses of trial designs

We illustrate the ROSA approach by performing sensitivity analyses for three designs of different complexity levels. In each example, we describe the design of the trial, the unknown parameters, and the operating characteristics of interest. By illustrating the ROSA methodology in three trial designs, we show its flexibility with potential applications to evaluate nearly any clinical trial design. Indeed, ROSA only requires the possibility of simulating the trials under potential unknown parameters $\mathit{\theta }\in \mathbf{\Theta }$ and the definition of the operating characteristics of interest.

3.1. Application 1: Two-arm RCT

In the first example, we will only consider a single unknown parameter (i.e., $\mathit{\theta }\in ℝ$) and a single operating characteristic $f\left(\mathit{\theta }\right)$ that can be computed analytically. In this case, the optimal set of scenarios $\left\{{\mathit{\theta }}_1^{\text{*}},\dots ,{\mathit{\theta }}_K^{\text{*}}\right\}$ can be computed exactly, without resorting to approximation methods. This simple and stylized setting is useful to highlight the similarity of the approximations and selected scenarios computed by ROSA with their exact counterparts.

3.1.1. Trial design

We consider the design of a two-arm randomized trial (1:1 randomization ratio) with a sample of $n=30$ patients. For each $i=1,\dots ,n$, we let $A_i=0$ or 1 if the $i$-th study patient is assigned to the control or experimental arm. The outcomes of the $n$ study patients are $Y_1,\dots ,Y_n$, which we assume to be independent and normally distributed. If $A_i=a$ then $Y_i$ has mean ${\mu }_a$ and standard deviation $\sigma$ equal to 30. In the analysis of the study, a $z$-statistic will be used to test the null hypothesis $H_0:{\mathit{\mu }}_1-{\mathit{\mu }}_0\le 0$ against the alternative $H_1:{\mathit{\mu }}_1-{\mathit{\mu }}_0>0$ at 5% significance level.

3.1.2. Aim of the sensitivity analysis

The goal of the sensitivity analysis is to summaze the variation of the probability of rejecting $H_0$, a function $f\left(\mathit{\theta }\right)$ of the unknown treatment effect $\mathit{\theta }={\mathit{\mu }}_1-{\mathit{\mu }}_0\in \mathbf{\Theta }=ℝ$. For example, if we knew that $\mathit{\theta }=13.5$, then $f\left(\mathit{\theta }\right)=0.80$, but in general $\mathit{\theta }$ is an unknown value. Suppose we aim to identify $K=3$ scenarios ${\mathit{\theta }}_1^{\text{*}},{\mathit{\theta }}_2^{\text{*}},{\mathit{\theta }}_3^{\text{*}}$ that maximize the utility $\mathcal{U}$, i.e.,

$${\mathit{\theta }}_1^{\text{*}},{\mathit{\theta }}_2^{\text{*}},{\mathit{\theta }}_3^{\text{*}}=\underset{{\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3\in \mathbf{\Theta }}{\text{argmax}}\mathcal{U}\left({\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3\right),$$ (3)

where $\mathcal{U}\left({\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3\right)=-{\text{max}}_{{\mathit{\theta }}^{\prime }\in \mathbf{\Theta }}{\text{min}}_{k=1,2,3}\left|f\left({\mathit{\theta }}^{\prime }\right)-f\left({\mathit{\theta }}_k\right)\right|$.

In this trial, we have a single unknown parameter $\left(\mathbf{\text{Θ}}=ℝ\right)$, and the operating characteristic of interest is monotone, continuous, invertible, and ranges from 0 to 1. Therefore, it is straightforward to see that the optimal scenarios ${\mathit{\theta }}_1^{\text{*}},{\mathit{\theta }}_2^{\text{*}},{\mathit{\theta }}_3^{\text{*}}$ correspond to the operating characteristic values that evenly divide the interval (0,1). To be precise, $\left\{f\left({\mathit{\theta }}_1^{\text{*}}\right),f\left({\mathit{\theta }}_2^{\text{*}}\right),f\left({\mathit{\theta }}_3^{\text{*}}\right)\right\}=\left\{1/6,3/6,5/6\right\}$; these are the three values of a regular grid on the interval (0, 1). Figure 2A illustrates the optimal set of scenarios when $K=\left\{3,5,10\right\}$. Since $f\left(\mathit{\theta }\right)$ can be calculated exactly, the optimal scenarios ${\mathit{\theta }}_1^{\text{*}},{\mathit{\theta }}_2^{\text{*}},{\mathit{\theta }}_3^{\text{*}}$ can be obtained by computing the inverse function $f^{-1}$ at the values $1/6$, $3/6$, and 5/6. Specifically,

$$\left\{{\mathit{\theta }}_1^{\text{*}},{\mathit{\theta }}_2^{\text{*}},{\mathit{\theta }}_3^{\text{*}}\right\}=\left\{\frac{\sigma \left(z_{f\left({\mathit{\theta }}_1^{\text{*}}\right)}+z_{1-\alpha /2}\right)}{\sqrt{n}},\frac{\sigma \left(z_{f\left({\mathit{\theta }}_2^{\text{*}}\right)}+z_{1-\alpha /2}\right)}{\sqrt{n}},\frac{\sigma \left(z_{f\left({\mathit{\theta }}_3^{\text{*}}\right)}+z_{1-\alpha /2}\right)}{\sqrt{n}}\right\},$$

where $z_{1-\alpha /2}$ is the $1-\alpha /2$ quantile of the standard normal distribution. The corresponding optimal scenarios are illustrated as red asterisks in Figure 2B.

Fig. 2.

Sensitivity analysis of a RCT (operating characteristic: probability of rejecting $H_0$). Panel A: Exact solutions when $K=\left\{3,5,10\right\}$. Panel B: Comparison of $K=3$ scenarios selected through exact calculation (red asterisks) and by 20 ROSA implementations with different initial proposals (blue points). Panel C: Graphical tool to choose the number $K$ of sensitivity scenarios.

3.1.3. Implementing and benchmarking ROSA

The exact computation of the optimal set of scenarios provides a solid benchmark for an initial evaluation of ROSA (Algorthms 1–3). We can compare the exact solution with the results from ROSA, which has the advantage of being applicable to other designs and operating characteristics that are not available in closed form.

We implement our ROSA approach to identify $K=3$ scenarios. We randomly select $J=1000$ scenarios ${\mathit{\theta }}_1^t,\dots ,{\mathit{\theta }}_{1000}^t$ with independent samples from the Uniform(−5, 25) distribution. Note that $f\left(-5\right)\approx 0$ and $f\left(25\right)\approx 1$. For each ${\mathit{\theta }}_j^t,1\le j\le 1000$, we simulate $M=200$ trials to compute the estimate $\overline{f}\left({\mathit{\theta }}_j^t\right)={200}^{-1}{\displaystyle \sum _{m=1}^{200}\phi \left(Z_{j,m},{\mathit{\theta }}_j^t\right)}$, where $\phi \left(Z_{j,m},{\mathit{\theta }}_j^t\right)\in \left\{0,1\right\}$ either accepts or rejects the $H_0:{\mathit{\theta }}_j^t\le 0$ for trial $m$ and scenario $j$. Then, we compute a continous function $\widehat{f}\left(\mathit{\theta }\right)$ using the independent estimates $\overline{f}\left({\mathit{\theta }}_j\right)$ and a NN with 3 hidden layers (8, 64, and 64 neurons respectively) and ReLU activation functions. Finally, to select three sensitivity scenarios, we use a simulated annealing algorithm based on an initial parameterization $T_1=1000$ and final parameterization $T_{\text{min}}=0.1$ (c.f. Algorithm 3). We repeat these three steps (selection of scenarios, use of the NN, and optimization with simulated annealing) 20 times, each time initializing ${\mathit{\theta }}_1$, ${\mathit{\theta }}_2$, ${\mathit{\theta }}_3$ with independent random draws from the Uniform (−5, 25) distribution. The results of the exact approach (red asterisks) compared with ROSA (blue points) are shown in Figure 2B. The scenarios ${\mathit{\theta }}_1^{\text{*}}$, ${\mathit{\theta }}_2^{\text{*}}$, ${\mathit{\theta }}_3^{\text{*}}$ selected by simulated annealing (blue dots) are close to the exact solution (red asterisks).

3.1.4. Choice of number $K$ of scenarios

In practice, the decision regarding the number $K$ of scenarios to report is left to the analyst. This choice can be supported by a graph like Figure 2C, which allows the investigator to determine the minimum number $K$ of scenarios needed to guarantee a loss $\mathcal{L}\left({\mathit{\theta }}_1^{\text{*}},\dots ,{\mathit{\theta }}_K^{\text{*}}\right)$ no larger than a targeted threshold. For example, to guarantee a loss no larger than 0.050 in this example, we need to select at least 10 scenarios for the simulation report.

We ran ROSA with $K\text{ = 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, or 30,}$ and compared the loss $\mathcal{L}$ in the resulting set of scenarios with that of the exact solution. The difference in the loss $\mathcal{L}$ of the exact and approximate optima was less than 1% across all $K$ values that we considered (Figure 2C). Table 2 indicates that the computation time of the simulated annealing algorithm scales well as $K$ increases and that, as expected, the loss $\mathcal{L}$ decreases as $K$ increases. All analyses were run on a Windows laptop with an Intel(R) Core(TM) i7-7700HQ 2.80 GHz processor, 16GB RAM, and 6MB of cache memory.

Table 2.

ROSA computation time, ROSA loss $\mathcal{L}$, minimum (exact) loss $\mathcal{L}$, and relative difference in loss of ROSA scenarios compared to the exact solutions.

Number $K$ of Scenarios	Time (seconds)	ROSA Loss $\mathcal{L}$	Min. Loss $\mathcal{L}$	Rel. Diff.
5	8.8	0.101	0.100	1.0%
6	8.8	0.084	0.083	0.7%
7	9.1	0.072	0.071	0.8%
8	9.2	0.062	0.0625	0.7%
9	9.1	0.056	0.056	0.6%
10	9.1	0.050	0.050	0.2%
20	10.1	0.025	0.025	0.5%
30	10.2	0.017	0.0167	0.8%

3.2. Application 2: Interim decisions based on auxiliary outcomes

In the second example, we consider sensitivity analyses with multiple unknown parameters and two operating characteristics. We illustrate the use of our computational procedures, including the operating characteristics approximation procedure (Algorithm 1), the validation procedure (Algorithm 2), and the simulated annealing optimization procedure (Algorithm 3). We investigate whether it is appropriate to fix the value of some of the unknown parameters across all sensitivity scenarios. Identical values for a subset of the unknown parameters can simplify the interpretation of the sensitivity analysis but can also introduce severe limitations in faithfully representing how the operating characteristics vary across plausible values of the unknown parameters.

3.2.1. Trial design

We consider a two-arm, two-stage randomized trial with a binary primary outcome $Y$ and a binary auxiliary outcome $\mathcal{S}$ (Niewczas et al., 2019). The primary outcome $Y$ is available $T_Y$ months after randomization, while the auxiliary outcome $\mathcal{S}$ is available after $T_{\mathcal{S}}<T_Y$ months. For example, in glioblastoma trials, 12-month progression-free survival (PFS) and 24-month overall survival (OS) have been used as auxiliary and primary outcomes, respectively (Han et al., 2014). The approach that we illustrate is applicable for any value of $T_Y$ and $T_{\mathcal{S}}<T_Y$.

We let $N_a$ be the planned number of patients for arms $a=0,\hspace{0.17em}1$ (i.e., control and experimental arms) and indicate with $p_a$ the response probability $P\left(Y=1|A=a\right)$. Similarly, let $n_a$ be the planned number of patients assigned to arm a before the interim analysis, and $q_a$ indicate the response probability $P\left(S=1|A=a\right)$. The difference $\text{Δ}=p_1-p_0$ is the treatment effect on $Y$. The primary aim of the trial is to test $H_0:\text{Δ}\le 0$ versus $H_1:\text{Δ}>0$, at level $\alpha$. The final analysis of the study involves only the primary outcome $Y$, and the trial will use a standard $Z$-test, $Z_Y=\frac{{\hat{p}}_1-{\hat{p}}_0}{\sqrt{\overline{p}\left(1-\overline{p}\right)\left(N_1^{-1}+N_0^{-1}\right)}}$, where ${\widehat{p}}_a$ is the estimate of $p_a$ and $\overline{p}$ is a weighted average of ${\widehat{p}}_1$ and ${\widehat{p}}_0$.

An interim analysis is conducted after the auxiliary outcomes $\mathcal{S}$ become available for $n_a$ patients for arms $a=0$ and 1 (i.e., $T_{\mathcal{S}}$ months after the enrollment of $n_a$ patients on arms $a=0$ and 1), with early-stopping for futility or continuation based on a summary of the auxiliary outcomes $\mathcal{S}$. In several clinical settings, the treatment effect on $\mathcal{S}$ tends to be more pronounced than the treatment effect on $Y$. The interim analysis is based on the summary $Z_S=\frac{{\widehat{q}}_1-{\widehat{q}}_0}{\sqrt{\overline{q}\left(1-\overline{q}\right)\left(n_1^{-1}+n_0^{-1}\right)}}$, where ${\widehat{q}}_a$ is the estimate of $q_a$ and $\overline{q}$ is a weighted average of ${\widehat{q}}_1$ and ${\widehat{q}}_0$. We replicate the design of Niewczas et al. (2019), which calculates at the interim analysis the conditional power (CP) using the auxiliary outcome $\mathcal{S}$ to determine whether to stop the trial for futility or not. Specifically, the CP is calculated based on $Z_{\mathcal{S}}$ and the information fraction $t_s=\frac{N_1^{-1}+N_0^{-1}}{n_1^{-1}+n_0^{-1}}$ as

$$CP\left(t_S\right)=1-\text{Φ}\left(\frac{z_{1-\alpha }-Z_S{t}_S^{1/2}}{\sqrt{1-t_S}}\right),$$

where $z_{1-\alpha }$ is the $1-\alpha$ quantile of the standard normal distribution and $\text{Φ}\left(\cdot \right)$ is the cumulative distribution function of the standard normal distribution. Here, we set the cut-off point to be 0.5 so that the trial continues when $CP\left(t_s\right)\ge 0.5$.

3.2.2. Aim of the sensitivity analysis

The complexity of the simulation report increases with $K$ (the number of scenarios), $d$ (the number of entries of the unknown parameters $\mathit{\theta }$), and $R$ (the number of operating characteristics $\mathbf{\text{f}}\left(\mathit{\theta }\right)$). Here the full set of unknown parameters $\mathbf{\Theta }\subset {ℝ}^7$ include the enrollment rate $e\in \left(0,\infty \right)$, the response rates $p_a\in \left(0,1\right)$ for $Y$ in $A=a$, the response rates $q_a\in \left(0,1\right)$ for $\mathcal{S}$ in $A=a$, and the correlation between $Y$ and $\mathcal{S}$ in $A=a$, ${\rho }_a\in \left(-1,1\right)$.

Controlling the complexity of the simulation report is important to ensure high interpretability of the report, which will be discussed by several stakeholders. There are a few potential strategies to reduce the complexity of the simulation report. First, it is often possible to consider only a subset of the parameter space ${\mathbf{\Theta }}^{\prime }\subset \mathbf{\Theta }$ based on prior knowledge of plausible values of the unknown parameters. For example, previous clinical studies can indicate a plausible range for the enrollment rate $e$, the response rates $p_0$ under the SOC, and other parameters that are expected to have minimal variations across trials. In addition, we can also consider fixing multiple entries of the $K$ vectors ${\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K$ to some reference values. In this case the space from which we select scenarios ${\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K$ is further reduced to ${\mathbf{\Theta }}_{re}^{\prime }\subset {\mathbf{\Theta }}^{\prime }$. For example, if the operating characteristics have low sensitivity with respect to the correlation parameters ${\rho }_a$ or the enrollment rate $e$ of the study, then we can fix these unknown parameters to common values (i.e., estimates) across all $K$ scenarios.

ROSA allows us to evaluate whether it is appropriate to assign the same value to one or more unknown parameters (e.g., ${\rho }_0$ and ${\rho }_1$) across all $K$ scenarios. In other words, we evaluate a simulation report with all scenarios in a restricted subset ${\mathbf{\Theta }}_{re}^{\prime }\subset {\mathbf{\Theta }}^{\prime }$. A simulation report with scenarios in ${\mathbf{\Theta }}_{re}^{\prime }$ can potentially be easier to interpret compared to a report in which all $d$ entries of $\mathit{\theta }$ vary across scenarios by reducing the number of dimensions $d$ of the unknown parameters and pointing to the most relevant unknown parameters when discussing the variations of the operating characteristics across ${\mathbf{\Theta }}^{\prime }$. We can select scenarios from the restriction ${\mathbf{\Theta }}_{re}{ }^{\prime }\subset {\mathbf{\Theta }}^{\prime }$ only if the capability of the simulation report of representing the operating characteristics variations across ${\mathbf{\Theta }}^{\prime }$ is preserved. Our case study investigates this aspect. The operating characteristics of interest $\text{f}$ in our case study are the probability of rejecting the null hypothesis of no treatment effect on $Y$ at the end of the study and the average sample size.

3.2.3. Implementing and benchmarking ROSA

Using our ROSA procedure, we randomly select $J=1000$ training scenarios using LHS and conduct $M=500$ Monte Carlo simulations for each of the $J$ training scenarios to obtain estimates of the operating characteristics across ${\mathbf{\Theta }}^{\prime }$. Here ${\mathbf{\Theta }}^{\prime }$ is a product space with the enrollment rate $e\in \left(0.2,1\right)$, the response rates $p_a\in \left(0.2,0.4\right)$ for $Y$ in $A=a$, the response rates $q_a\in \left(0.2,0.4\right)$ for $\mathcal{S}$ in $A=a$, and the correlation between $Y$ and $\mathcal{S}$ in $A=a,{\rho }_a\in \left(0,0.6\right)$. For ${\mathbf{\text{Θ}}}_{re}^{\prime }$, we fix the enrollment rate $e=0.5$ and the response rates $p_0=q_0=0.3$ in the control groups.

We use a NN to obtain an interpolation of the operating characteristics. As described in Algorithm 4, to evaluate if the estimates of the operating characteristics are accurate, we compare them to independent Monte Carlo estimates of size $\text{M}=100,000$ on a set of $J^{\prime }=200$ uniformly-distributed validation points spanning the plausible parameter space ${\mathbf{\Theta }}^{\prime }$. The coefficients of determination $R^2$ in this comparison are above 0.96. This suggests that the NN accurately estimates the operating characteristics.

We compare two simulation reports, and our goal is to provide stakeholders the simplified version if it accurately describes the operating characteristics. The first one includes scenarios from ${\mathbf{\Theta }}^{\prime }\subset {ℝ}^7$ restricted by prior knowledge from completed studies and clinical experience and the second includes scenarios from ${\mathbf{\Theta }}_{re}^{\prime }\subset {\mathbf{\Theta }}^{\prime }$ further restricted by fixing the value of some entries of $\mathit{\theta }$ as described above. We use simulated annealing to identify two sets of scenarios in ${\mathbf{\Theta }}_{re}^{\prime }$ and ${\mathbf{\Theta }}^{\prime }$, respectively. In both cases we minimize the same loss function $\mathcal{L}$ defined over K-tuples of ${\mathbf{\Theta }}^{\prime }$ points. We also calculate the loss $\mathcal{L}$ associated with these two optimal sets of scenarios from ${\mathbf{\Theta }}^{\prime }$ and ${\mathbf{\Theta }}_{re}^{\prime }$. In Figure 3, we illustrate the difference in loss $\mathcal{L}$ between these two optimal sets; as expected, the loss $\mathcal{L}$ decreases as $K$ increases. We observe in Figure 3 that for any value of $K$, the loss $\mathcal{L}$ associated with the optimal set of scenarios restricted to ${\mathbf{\Theta }}_{re}^{\prime }$ is larger compared to the optimal scenarios in ${\mathbf{\Theta }}^{\prime }$. However, the difference is modest, and the gain in interpretability of a sensitivity analysis report with fewer unknown parameters may be worth the slightly larger loss. For example, if an investigator requires the loss to be under a threshold of $\mathcal{L}=0.2$, then it is sufficient to consider $K=10$ scenarios, regardless of whether we consider scenarios selected from ${\mathbf{\Theta }}^{\prime }$ or ${\mathbf{\Theta }}_{re}^{\prime }$

Fig. 3.

Clinical trial design with an interim analysis and an auxiliary endpoint. A graphical representation to choose the number of sensitivity scenarios $K\in \left\{2,5,10,15\right\}$. We compare optimal sets of scenarios selected from ${\mathbf{\Theta }}^{\prime }\subset {ℝ}^7$ and from the lower-dimensional restriction ${\mathbf{\Theta }}_{re}^{\prime }\subset {\mathbf{\Theta }}^{\prime }$.

3.3. Application 3: Biomarker-driven adaptive enrichment

In the third example, we discuss sensitivity analyses dedicated to an adaptive trial with subpopulations defined by biomarkers, considering multiple unknown parameters and multiple operating characteristics of interest. As a motivating example, in several oncology trials, a major decision is whether to restrict patient enrollment to a targeted subgroup of patients (e.g., biomarker-positive subgroup) or to enroll a broader patient population. Enrolling only a biomarker-positive subgroup may deny a substantial number of patients access to an effective therapy, whereas enrolling a larger population may compromise the power to detect positive treatment effects. Several trial designs discussed in the literature attempt to address the outlined problem through interim looks at the data.

3.3.1. Trial design

We consider an adaptive two-stage enrichment trial design with one-to-one randomization (Jenkins et al., 2011; Jones et al., 2017; Mehta et al., 2019). The design is applicable in the setting where a biomarker-positive subgroup of patients is hypothesized to benefit more from the experimental treatment than the rest of the study population. The design includes a single interim analysis, and it uses progression-free survival (PFS) for interim decision-making, while overall survival (OS) is the endpoint for the final analysis, which occurs when a pre-specified number of events is reached. The interim analysis uses the estimated PFS hazard ratio $\left(\text{HR}\right)$ to capture potential early signals of treatment effects. In the implementation of Jenkins et al. (2011), which we replicate, the HR is estimated for both the overall population $\left({\widehat{\mathit{\theta }}}_{HR}\right)$ and the biomarker-positive subgroup $\left({\widehat{\mathit{\theta }}}_{HR}^{+}\right)$. An interim decision determines which group is enrolled and tested during the second stage of the trial:

A –. Promising results in the biomarker-positive population.

If the HR estimate ${\widehat{\mathit{\theta }}}_{HR}^{+}<0.6$ but ${\widehat{\mathit{\theta }}}_{HR}\ge 0.8$, then the trial will continue enrolling only biomarker-positive patients and the final analysis will test $H_0^{+}$. Here $H_0^{+}$ is the null hypothesis of no differences in OS between treatment and control groups in the biomarker-positive population. The null hypothesis is rejected if ${\omega }_1{\text{Φ}}^{-1}\left(1-p_1^{+}\right)+{\omega }_2{\text{Φ}}^{-1}\left(1-p_2^{+}\right)<1.96$, where $p_1^{+}\left(p_2^{+}\right)$ is a log-rank p-value computed using only OS data from patients randomized during the first (second) stage of the trial. The weights $\left({\omega }_1,{\omega }_2\right)$ and the standard normal cumulative distribution function $\text{Φ}$ are used to summarize evidence of treatment effects from the two stages of the trial. We refer to Jenkins et al. (2011) for details on the choice of $\left({\omega }_1,{\omega }_2\right)$ and other aspects of the final analysis.

B –. Promising results in the overall population only.

If ${\widehat{\mathit{\theta }}}_{HR}^{+}\ge 0.6$ but ${\widehat{\mathit{\theta }}}_{HR}<0.8$, then the trial will continue enrolling all patients and the final analysis will only test $H_0^O$, the null hypothesis of no differences in OS in the overall population. In this case the null hypothesis is tested using stage-specific OS log-rank p-values $\left(p_1^O,p_2^O\right)$ and combining evidence from the two stages of the trial.

C –. Unpromising results.

If ${\widehat{\mathit{\theta }}}_{HR}^{+}\ge 0.6$ and ${\widehat{\mathit{\theta }}}_{HR}\ge 0.8$, then the trial stops early for futility.

D –. Promising early results for both populations.

Lastly, if the estimated HR in the biomarker-positive subgroup ${\widehat{\mathit{\theta }}}_{HR}^{+}<0.6$ and the overall population ${\widehat{\mathit{\theta }}}_{HR}<0.8$, then the trial will continue enrolling all patients and testing efficacy both in the overall population and in the biomarker-positive subgroup.

The potential conclusion at the final analysis are (i) to recommend the new treatment for biomarker-positive patients, (ii) recommend the new treatment for both biomarker-positive and biomarker-negative patients, or (iii) not recommend the experimental treatment for future patients.

3.3.2. Aims of the sensitivity analysis

We focus on the following three operating characteristics: (i) $f_1$, the probability of enrolling only biomarker-positive patients in the second stage, (ii) $f_2$, the probability of enrolling both biomarker-positive and biomarker-negative patients in the second stage, and (iii) $f_3$, the probability of no evidence of positive treatment effects, which is equal to the probability of not rejecting the null hypotheses.

We choose plausible intervals for the unknown parameters based on prior literature. Specifically, the recruitment rate ${\mathit{\theta }}_1\in \left(0.5,1\right)$ per week, the prevalence of the biomarker-positive subgroup ${\mathit{\theta }}_2\in \left(0.15,0.25\right)$, the PFS HR comparing the treatment and control groups in the biomarker-positive subgroup ${\mathit{\theta }}_3\in \left(0.5,1.2\right)$, the PFS HR comparing treatment and control in the biomarker-negative subgroup ${\mathit{\theta }}_4\in \left(0.6,1.2\right)$, the OS HR comparing treatment and control in the biomarker-positive subgroup ${\mathit{\theta }}_5\in \left(0.7,1.2\right)$, the OS HR comparing treatment and control groups in the biomarker-negative subgroup ${\mathit{\theta }}_6\in \left(0.8,1.2\right)$, the correlation between OS and PFS in the biomarker-positive subgroup ${\mathit{\theta }}_7\in \left(0.3,0.6\right)$, and the correlation between OS and PFS in the biomarker-negative subgroup ${\mathit{\theta }}_8\in \left(0.2,0.7\right)$. Marginal exponential distributions and latent frailty terms were used for simulating correlated OS and PFS times (Michael and Schucany, 2002). More flexible models such as the Weibull distribution can be considered.

3.3.3. Implementing and benchmarking ROSA

For the outlined two-stage trial with biomarker populations, our ROSA pipeline can be used to compute multiple simulation reports, varying both the list of operating characteristics $\mathit{f}$ and the definition of ${\mathbf{\Theta }}^{\prime }$. For example, one can fix the OS HRs in the biomarker-positive and negative populations to focus on the design sensitivity to other parameters, such as the PFS HRs. Similarly, the set of unknown parameters ${\mathbf{\Theta }}^{\prime }$ can be restricted to $\mathit{\theta }$ values with positive effects only for the biomarker-positive population. Importantly, one set of training simulations can be re-utilized to compute multiple sensitivity tables where the definitions of $\mathit{f}$ and ${\mathbf{\Theta }}^{\prime }$ vary.

We examine the difference in the marginal losses

$${\mathcal{L}}_r\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)=\underset{\mathit{\theta }\in \mathbf{\Theta }}{\text{max}}\left\{\underset{k=1,\dots ,K}{\text{min}}{‖f_r\left(\mathit{\theta }\right)-f_r\left({\mathit{\theta }}_k\right)‖}_2\right\}, 1\le r\le R,$$ (4)

when the set of scenarios are chosen by optimizing different loss functions. For example, let ${\mathcal{S}}_r$ be the set of scenarios that minimize the marginal loss ${\mathcal{L}}_r$ in (4). Similarly, let $\mathcal{S}$ be the set of scenarios that minimize the joint loss $\mathcal{L}=-\mathcal{U}$ in (2). Then it is intuitive that ${\mathcal{L}}_r\left({\mathcal{S}}_r\right)\le {\mathcal{L}}_r\left(\mathcal{S}\right),1\le r\le R$. In different words, the marginal losses ${\mathcal{L}}_{\text{r}}$ tend to be smaller when the set of scenarios is chosen to minimize ${\mathcal{L}}_{\text{r}}$ compared to a set of scenarios that minimizes $\mathcal{L}$ with the aim of representing multiple operating characteristics. If the discrepancy ${\mathcal{L}}_r\left({\mathcal{S}}_r\right)-{\mathcal{L}}_r\left(\mathcal{S}\right),1\le r\le R$, is relatively small for all $R$ total operating characteristics, then this indicates that it is reasonable to select a single set of scenarios $\mathcal{S}$ to illustrate how the $R$ operating characteristics vary jointly across $\mathbf{\text{Θ}}$. We describe the difference between the marginal losses ${\mathcal{L}}_r,r\hspace{0.17em}=\hspace{0.17em}1,2,3$, when scenarios ${\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K$ in ${\mathbf{\text{Θ}}}^{\prime }$ are chosen by optimizing ${\mathcal{L}}_r$ in (4) – optimum: ${\mathcal{S}}_r={\mathit{\theta }}_{1,r}^{\text{*}},\dots ,{\mathit{\theta }}_{K,r}^{\text{*}}$- or by optimizing $\mathcal{L}$ as in (2) - optimum: $\mathcal{S}={\mathit{\theta }}_1^{\text{*}},\dots ,{\mathit{\theta }}_K^{\text{*}}$. Recall that $\mathcal{S}$ is computed with the goal of illustrating how multiple operating characteristics vary across ${\mathbf{\Theta }}^{\prime }$ while ${\mathcal{S}}_r$ optimizes the representation of a single operating characteristic $f_r$. The weights in (2) are $w_1=w_2=w_3=1/3$. In Figure 4 panel 1, we plot ${\mathcal{L}}_1\left({\mathcal{S}}_1\right)$ in red and ${\mathcal{L}}_1\left(\mathcal{S}\right)$ in blue. Similarly, in panel 2 we compare ${\mathcal{L}}_2\left({\mathcal{S}}_2\right)$ and ${\mathcal{L}}_2\left(\mathcal{S}\right)$, and in panel 3 we compare ${\mathcal{L}}_3\left({\mathcal{S}}_3\right)$ and ${\mathcal{L}}_3\left(\mathcal{S}\right)$. Our results indicate that for all three operating characteristics, ${\mathcal{L}}_r\left(\mathcal{S}\right)>{\mathcal{L}}_r\left({\mathcal{S}}_r\right),r=1,2,3$; as expected, there is an increase of the marginal losses ${\mathcal{L}}_r$ when the set of scenarios is selected to illustrate jointly the variations of multiple operating characteristics across ${\mathbf{\Theta }}^{\prime }$. However, this difference is small $(<10\text{\%})$ for all $K\in \left\{2,5,10,15\right\}$. Furthermore, for each $K\in \left\{2,5,10,15\right\}$, the relative difference is similar across the three operating characteristics $f_1,f_2,f_3$ (Figure 4). This result supports the use of identical weights and of a single sensitivity table, with the same set of scenarios $\mathcal{S}$ to illustrate jointly all three operating characteristics.

Fig. 4.

Marginal losses ${\mathcal{L}}_r,r=1,2,3$ of different sets of scenarios ${\mathcal{S}}_r$ (red) and $\mathcal{S}$ (blue).

4. Discussion

The evaluation of complex designs such as dose-finding studies (lasonos et al., 2015), factorial trials (Green et al., 2002), and response-adaptive trials (Pallmann et al., 2018) focuses on multiple operating characteristics, such as the level of toxicities, the probability of selecting the correct treatment arm, or frequentist operating characteristics, including power and false positive probabilities. During the design stage of a complex clinical trial, simulation reports are typically produced to discuss sample size, interim analyses, and other major decisions with various stakeholders. The simulation report consists of one or a few tables dedicated to showcasing how major operating characteristics $\mathbf{\text{f}}\left(\mathit{\theta }\right)$ vary across potential values of unknown parameters in $\mathbf{\Theta }$. In most cases, the analyst focuses on subsets of plausible parameters ${\mathbf{\Theta }}^{\prime }\subset \mathbf{\Theta }$, for example, values concordant with previous studies, or subsets of potential $\mathit{\theta }$ values of particular interest, for example with positive and clinically relevant treatment effects.

Simulations are fundamental in the design of complex trials since operating characteristics can rarely be obtained analytically and are crucial in the assessment of study designs for regulators, pharmaceutical companies and other stakeholders (Food et al., 2020). However, a limited number of scenarios or poorly chosen scenarios could be inadequate to highlight variations of the operating characteristics across plausible unknown parameters and can result in sub-optimal decisions. We propose ROSA as a useful tool that can support investigators at this design stage when selecting which and how many scenarios to include in these simulation reports.

We focus on choosing an informative number $K$ of scenarios ${\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K$ among the plausible unknown parameters to summarize the variations of key operating characteristics. Our approach minimizes an explicit loss function and uses established techniques for functional approximation (NNs) and numerical optimization (simulated annealing). We showcase our approach in three trials. Importantly, our approach is general and can be applied to nearly any clinical trial design. It only requires simulations to mimic the clinical trial under hypothetical scenarios.

Although our approach is general, we focused on loss functions $\mathcal{L}$ of a specific form (2). It is possible to consider different loss functions. For example, one could consider the loss function $\mathcal{L}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_K\right)=E_{{\mathit{\theta }}^{\prime }\sim g\left(\cdot \right)}\left\{{\text{min}}_{k=1,\dots ,K}D\left[\mathbf{\text{f}}\left({\mathit{\theta }}^{\prime }\right),\mathbf{\text{f}}\left({\mathit{\theta }}_k\right)\right]\right\}$, where $g\left(\cdot \right)$ is a probability distribution on $\mathbf{\text{Θ}}$ (e.g., a posterior distribution obtained from previous data). The distribution $g$ could be used to incorporate prior information about the unknown parameters in the selection of sensitivity scenarios. Moreover, the metric $D:{\mathbf{\text{Θ}}}^2\to ℝ$ can be extended to capture both differences between operating characteristics at plausible values $\mathit{\theta },{\mathit{\theta }}^{\prime }\in \mathbf{\Theta }$ and other aspects, such as the difference between expected values of the outcomes $Y$ at $\mathit{\theta }$ and ${\mathit{\theta }}^{\prime }$.

One major challenge in the presentation of simulation reports is the need for simplicity and interpretability of the results. To this end, we considered fixing one or more unknown parameters to identical values across the $K$ scenarios, which may be reasonable when there is a priori knowledge of certain unknown parameters. There are other ways to simplify a simulation report, such as removing operating characteristics that do not vary across plausible unknown parameters, or reporting only the range of the operating characteristics across $\mathbf{\text{Θ}}$ instead of presenting the operating characteristics for each representative scenario.

Variations of the ROSA approach may also consider optimization algorithms other than simulated annealing and regression methods alternative to NN for approximating the operating characteristics across $\mathbf{\text{Θ}}$. The methodology that we proposed here can be used to handle other relevant problems, such as missing data. Indeed, during the design of the trial, there is often uncertainty on whether the analyses will involve missing data or not and the potential consequences of the missingness pattern. Probability models that include pre-treatment variables, outcomes, and missing data patterns are useful to explore the robustness of the design. In this case, ROSA can support the selection of scenarios with different missing data patterns.