Dynamic treatment regimens in small n, sequential, multiple assignment, randomized trials: An application in focal segmental glomerulosclerosis

Abstract

Focal segmental glomerulosclerosis (FSGS) is a rare kidney disease with an annual incidence of 0.2–1.8 cases per 100,000 individuals. Most rare diseases like FSGS lack effective treatments, and it is difficult to implement clinical trials to study rare diseases because of the small sample sizes and difficulty in recruitment. A novel clinical trial design, a small sample, sequential, multiple assignment, randomized trial (snSMART) has been proposed to efficiently identify effective treatments for rare diseases. In this work, we review and expand the snSMART design applied to studying treatments for FSGS. The snSMART is a multistage trial that randomizes participants to one of three active treatments in the first stage and then re-randomizes those who do not respond to the initial treatment to one of the other two treatments in the second stage. A Bayesian joint stage model efficiently shares information across the stages to find the best first stage treatment. In this setting, we modify the previously presented design and methods (Wei et al. 2018) such that the proposed design includes a standard of care as opposed to three active treatments. We present Bayesian and frequentist models to compare the two novel therapies to the standard of care. Additionally, we show for the first time how we should estimate and compare tailored sequences of treatments or dynamic treatment regimens (DTRs) and contrast the results from our methods to existing methods for analyzing DTRs from a SMART. We also propose a sample size calculation method for our snSMART design when implementing the frequentist model with Dunnett’s correction.

1. Introduction

Focal segmental glomerulosclerosis (FSGS) is a rare kidney disease with an annual incidence of 0.2–1.8 cases per 100,000 individuals [9]. FSGS has traditionally been diagnosed in patients with persistent proteinuria based on characteristic lesions in a kidney biopsy specimen [1]. The identification of an effective treatment for FSGS is generally by trial and error, i.e., try a therapy, assess response, move to alternate treatment option in treatment failures, and then repeat these steps. There is little evidence to guide the choice of initial therapy or the selection of subsequent therapies dependent on initial treatment response patterns.

Nephrologists are confronted with several questions when caring for patients with FSGS. How should treatments targeting a specific mechanism of disease be selected, implemented, and assessed? Which of these treatments can provide the best short- and long-term response rate? What is the best sequence to introduce therapy when there are several options? These questions are not unique to FSGS, but similar types of questions are shared across many rare diseases and have proved difficult to answer in clinical trials with small samples.

One clinical trial design that could address these questions in FSGS and other rare diseases is the small n sequential, multiple assignment, randomized trial design (snSMART) [10,12]. An snSMART design is a multi-stage trial where participants are first randomized to one of the treatment arms and those who do not respond to the initial treatment are re-randomized to one of the other treatment options. We show here for the first time that 1) snSMART designs may be used in settings with two active treatments and a standard of care, 2) snSMART designs may also be used to estimate and compare dynamic treatment regimens (DTRs), and 3) sample size can be calculated via simulation study when the frequentist model is used.

First, we describe standard SMART [6] and snSMART designs in more detail. We differentiate the standard SMART from an snSMART in terms of design characteristics and goals of interest. We present both frequentist and Bayesian models to estimate and compare first-stage treatment effects sharing information across stages and to estimate and compare DTRs where interest is in the longer-term course of care. Bias and efficiency of the presented models and standard SMART methods of analysis are compared via simulations, which are motivated by the FSGS setting. Further, we demonstrate and provide the code to find the sample size required in an snSMART to compare novel treatments to a standard of care implementing the frequentist model with Dunnett’s correction.

2. snSMART and SMART designs

The snSMART design is a SMART that is specifically applied to rare diseases, and differs from a standard SMART with respect to both the primary goal of the trial and some trial characteristics. Fig. 1 contains an illustrative snSMART design that can be applied to FSGS. The first randomization is assignment to either an anti-fibrotic therapy (A), anti-inflammatory therapy (B) or standard of care (C) with equal probability. At the completion of the initial treatment stage at the fourth month, all individuals are classified as responders or non-responders to their initial treatment. In the second stage, the responders to any therapy continue that therapy for another 4 months. The non-responders to any therapy are randomized equally to the other two treatment options. In this example snSMART, the primary outcome and definition of response is defined as the achievement of a 40% reduction in urine protein:creatinine ratio and and proteinuria value less than 1.5 [11]. Thus, for the evaluation at 4 months, the protein:creatinine ratio is compared to the baseline level and for the assessment at 8 months, the ratio is compared to the 4-month level.

Fig. 1.

A small n, sequential, multiple assignment, randomized trial (snSMART) design. Subjects are allocated to one of the three first stage treatment groups A, B, C at time 0. R represents equal randomization to the following treatments. Based on the response status at time t, patients either continue the initial treatment or are re-randomized to one of the other two treatments. Subgroups 1 through 9 denote the treatment paths that any one patient may follow. Second stage responses can be obtained at time 2t. The combination of two treatment paths, one for responders and another for non-responder sharing the same first stage treatment defines a DTR.

The primary aim of this snSMART design is to find the best treatment that induces 4-month response in individuals with FSGS. Thus, we are most interested in identifying a difference between either the anti-fibrotic therapy and standard of care or the anti-inflammatory therapy and standard of care. Through an snSMART design, information across stages can be pooled to estimate the response rates of each treatment, which can be compared to that of the standard of care.

Unlike snSMARTs, standard SMARTs are featured by a relatively large sample size with the aim of developing an optimal DTR. The DTRs of interest are embedded within a SMART design, and these DTRs are guidelines that provide decision rules to choose treatments based on individualized measures, such as response, over the course of a disease. Due to heterogeneity among individuals, the severity of symptoms, the presence of comorbidities, and lack of adherence, treatment effectiveness may differ substantially. Moreover, heterogeneity within individuals can lead to a difference in the effectiveness of a treatment for an individual over time, especially for disorders that repeatedly flare and then resolve [7,4]. DTRs are particularly useful in these settings, because alternative treatment can be provided to the individuals who fail to respond or adhere to the initial treatment. A two-stage DTR includes a guideline for the initial treatment and for subsequent treatment for those who respond to the initial treatment and for those who do not respond. Explicitly, a DTR AAB denotes the stage 1 treatment A and two possible stage 2 treatments A and B for stage 1 responders and non-responders, respectively. Ideally, SMARTs are followed by confirmatory randomized controlled trials, where the best (or several best) performing DTR(s) would be compared to the standard of care. For more information on standard SMARTs, we refer readers to Lei et al. [4].

The snSMART design was initially developed to find the best first-stage treatment among three active treatment options (e.g., the treatment that has the best 4-month response rate) pooling information from both stages [10,12]. snSMARTs, however, do not necessarily require three active investigational treatments, so that one treatment may be a standard of care. Although the primary goal of an snSMART is to find the best first-stage treatment, DTRs (i.e., first and second stage treatment guidelines that depend on response) are still embedded within an snSMART design and can also be estimated and compared, which is the secondary aim of an snSMART. While existing SMART analytic methods to estimate and compare DTRs can be used on an snSMART, we present more efficient methods in this paper.

3. Bayesian and frequentist analysis of an snSMART

We present Bayesian and frequentist models that can estimate both first stage treatment effects and DTR effects using data from both the first and second stages of an snSMART. Specifically, we present a Bayesian joint stage regression model (BJSM) and a frequentist joint stage regression model (JSRM) using generalized estimating equations (GEE). These models are extensions of those in Wei et al. [12] allowing for estimates of DTRs in the presence of potential first and second stage treatment interactions.

3.1. Bayesian joint stage model

We define the model where the first stage response rate to treatment j is denoted by π_j, j = A, B, C. The second stage response rate of the first stage responders to treatment j is denoted by β_1jπ_j, and the second stage response rate of the non-responders to first stage treatment j who receive treatment j′ in the second stage is denoted by β_0jπ_j′. Here, β_1j and β_0j are the linkage parameters since they link the first stage response rate to the second stage response rate. We assume that first stage non-responders are less likely to respond to either of the two other treatments in second stage (β_0j < 1). We also assume that linkage parameters only depend on the first stage treatment.

To estimate the first stage response rates, π_j, j = A, B, C and the response rates of all DTRs specified as π_jjj′, j ≠ j′ = A, B, C, we first obtain the posterior draws of π_j, β_1j, and β_0j through the BJSM as follows:

$$Y_{i1}^j|{\pi }_j~\text{Bernoulli}\left({\pi }_j\right)$$ (1)

$$Y_{i2}^{j\prime }|Y_{i1}^j,{\pi }_j,{\pi }_{j\prime },{\beta }_{1j},{\beta }_{0j}~\text{Bernoulli}\left({\left({\beta }_{1j}{\pi }_j\right)}^{Y_{11}^j}{\left({\beta }_{0j}{\pi }_{j\prime }\right)}^{1-Y_{i1}^j}\right)$$ (2)

$${\pi }_j~\text{Beta}\left({\theta }_1,{\delta }_1\right)$$ (3)

$${\beta }_{0j}~\text{Beta}\left({\theta }_2,{\delta }_2\right)$$ (4)

$${\beta }_{1j}~\text{Gamma}\left({\theta }_3,{\delta }_3\right)$$ (5)

Eqs. (1) and (2) show the distributions of the first and second stage responses. The prior distributions for the parameters π_j, β_0j and β_1j are given in Eqs. (3), (4) and (5). The hyperparameters of the prior distributions should be based on prior knowledge from investigators. Specifically, for π_j, we assigned the values of θ₁ = 0.4 and δ₁ = 1.6, which gives a prior mean of 0.2 for the response rates since we believe that an ineffective treatment or standard of care would have a response rate of 20%. Similarly, for β_0k, we have assigned the values of θ₂ = 1.6 and δ₂ = 0.4, so that the average prior response rate for the second stage treatment for non-responders was assumed a priori to be reduced by 20% compared to the first-stage response rate of the same treatment. For β_1j, we assigned the values of θ₃ = 2 and δ₃ = 2, so that the prior mean of 1 indicates that the first stage responders in the first stage are assumed to have the same response rate to the same treatment in the second stage. We note the change in the prior distribution of β_1j from Wei et al. [12]. We made the prior distribution more flexible here such that β_1j can range from zero to infinity as opposed to one to infinity.

Next, we compute the posterior draws for each DTR π_jjj′ from the following equation using the the posterior draws of β_0j, β_1j, and π_j:

$${\pi }_{jjj\prime }={\pi }_j\left({\pi }_j{\beta }_{1j}\right)+\left(1-{\pi }_j\right)\left({\pi }_{j\prime }{\beta }_{0j}\right)$$ (6)

As a result, it is easy to calculate the means and standard deviations of π_jjj′ from their posterior draws.

3.2. Joint stage regression model

A joint-stage regression model (JSRM) is a frequentist modeling approach that incorporates the responses of both stages as repeated measurements for each subject. Hence, generalized estimating equations (GEE) are used to estimate the response rates of each treatment and from these estimates, we can compute the marginal response rates for each DTR. For binary outcomes, the logit link is most commonly applied to estimate the response rate [3,4]. However, in small samples, the standard errors of parameters tend to be underestimated if we fit the model with the logit link function [5]. Instead of applying the bias-corrected variance estimator of Mancl and DeRouen [5], we use a log link in our GEE model [13].

For subject i = 1, …, N, and stage k = 1, 2, we let Y_ik be the response of subject i in stage k. The JSRM with six linkage parameters is as follows:

$$\begin{gathered} \text{log}\left(P\left(Y_{ik}=1\right)\right)={\alpha }_{1}1\left(j_{ik}=A\right)+{\alpha }_{2}1\left(j_{ik}=B\right)+{\alpha }_{3}1\left(j_{ik}=C\right) \\ +\left[{\alpha }_{4}1\left(Y_{i1}=1\right)+{\alpha }_{5}1\left(Y_{i1}=0\right)\right]1\left(j_{i1}=A,k=2\right) \\ +\left[{\alpha }_{6}1\left(Y_{i1}=1\right)+{\alpha }_{7}1\left(Y_{i1}=0\right)\right]1\left(j_{i1}=B,k=2\right) \\ +\left[{\alpha }_{8}1\left(Y_{i1}=1\right)+{\alpha }_{9}1\left(Y_{i1}=0\right)\right]1\left(j_{i1}=C,k=2\right) \end{gathered}$$ (7)

where j_ik is the treatment indicator of subject i in stage k and 1(·) is an indicator function. In Eq. (7), α₁ to α₃ correspond to the first stage response rates of the treatments A, B, and C, or π_A, π_B, and π_C, respectively. The second stage response rates start with the first stage response rates, but are then augmented by an amount depending on both (i) the treatment received in stage 1, and (ii) whether or not a response occurred in stage 1. For example, consider the individuals who receive treatment B in stage 2 after not responding to treatment A in stage 1. Their second stage response rate would be a function of α₂, in the first line of Eq. (7), and then augmented by α₅ to reflect the non-response to treatment A in stage 1.

We assume that the dependency of Y_i2 on Y_i1 is already taken into account with the covariates in the JSRM, so we fit this GEE model with the independence working covariance structure and use a robust “sandwich” estimator to estimate $\text{Cov}\left(\widehat{\mathit{\alpha }}\right)$, where α = (α₁, …, α₉)^T. Even if there is additional correlation of Y_i1 and Y_i2 that is not captured by the working covariance matrix, the use of sandwich estimator guarantees the unbiasedness of parameter estimates, and it is flexible to change the working covariance matrix to other type, such as compound symmetry, if a strong additional within-subject correlation can be identified.

To estimate the first stage response rates denoted by π_j, j = A, B, C, we exponentiate the first three coefficients in Eq. (7). Specifically, ${\widehat{\pi }}_A=\text{exp}\left({\widehat{\alpha }}_1\right)$, ${\widehat{\pi }}_B=\text{exp}\left({\widehat{\alpha }}_2\right)$ and ${\widehat{\pi }}_C=\text{exp}\left({\widehat{\alpha }}_3\right)$. To estimate the response rates of the six embedded DTRs, we use linear combinations of estimates from Eq. (7). For example, from Fig. 1, DTR AAB contains subgroups 1 and 2 and the estimated response rate is ${\widehat{\pi }}_{AAB}={\widehat{\pi }}_A\times {\widehat{\beta }}_{1A}{\widehat{\pi }}_A+\left(1-{\widehat{\pi }}_A\right)\times {\widehat{\beta }}_{0A}{\widehat{\pi }}_B$, where ${\widehat{\pi }}_A=\text{exp}\left({\widehat{\alpha }}_1\right)$, ${\widehat{\pi }}_B=\text{exp}\left({\widehat{\alpha }}_2\right)$, ${\widehat{\beta }}_{1A}=\text{exp}\left({\widehat{\alpha }}_4\right)$, ${\widehat{\beta }}_{0A}=\text{exp}\left({\widehat{\alpha }}_5\right)$. The response rates of the other DTRs can be estimated through a similar approach. The standard error of each estimated DTR response rate can be obtained from $\text{Cov}\left(\widehat{\mathit{\alpha }}\right)$ using the Delta method.

3.3. Weighted and replicated regression model

In the weighted and replicated regression model (WRRM), the observations are weighted to account for over/under-representation of the individuals due to design Nahum-Shani et al. [7]. For example, in Fig. 1, the non-responders are under-represented because they are not all assigned the same treatment in the second stage. To be specific, when we estimate the response rate of DTR AAB, we assume that all non-responders to initial treatment A receive B at the second stage. However, some non-responders are randomized to C, resulting in the under-representation of non-responders receiving B. Therefore, individuals are weighted by the inverse of the probability of their treatment. In the snSMART design in Fig. 1, the responders are weighted by 3, which is the inverse of the product of their first and second stage randomization probabilities (13 × 1), and non-responders are weighted by 6, which is the inverse of (1/3) × (1/2).

To estimate the response rates of different DTRs simultaneously using standard software, we then need to implement replication. In general, the second stage outcomes for subjects who are consistent with more than one DTR are replicated. For example, the subjects who respond to A in the first stage and continue the same treatment are consistent with two DTRs AAB and AAC, meaning that their second stage responses are used in estimation of both DTRs AAB and AAC. Thus, we replicate the data of these subjects who are consistent with two DTRs and assign these two sets of data to two DTRs, respectively. For non-responders to the first stage treatments, they are only consistent with one DTR, so no replication is required for them, and their second stage outcomes are only used for the estimation of the corresponding DTRs. As a result of replication of data for the first stage responders, the data are now considered as repeated measurements, which is the reason that the estimation of DTR effects is conducted under the framework of GEE [7].

Parallel to the JSRM approach, we use a log link function in our model. We follow a model parametrization so that dummy variable coding indicates the first and second stage treatments, where the DTR AAB is chosen as a reference DTR. Thus, if we let Y_i2r, r = 1, 2, be the second-stage response of subject i, the WRRM is:

$$\text{log}\left(P\left(Y_{i2r}=1\right)\right)={\alpha }_0^{\prime }+{\alpha }_1^{\prime }1\left(j_i=B\right)+{\alpha }_2^{\prime }1\left(j_i=C\right)+{\alpha }_3^{\prime }1\left(j_i=A,j_{ir}^{\prime }=C\right)+{\alpha }_4^{\prime }1\left(j_i=B,j_{ir}^{\prime }=C\right)+{\alpha }_5^{\prime }1\left(j_i=C,j_{ir}^{\prime }=B\right)$$ (8)

where j_ir′ is the r-th second-stage treatment of the subject i. For the non-responders to first stage treatment who are consistent with only one DTR, r = 1, and for the responders to first stage treatment who are consistent with two DTRs, r = 1, 2. We note that WRRM uses only second stage responses to estimate the response rates of DTRs, in contrast to the previous two methods which use both first and second stage outcomes.

After the model is fit, we estimate the response rates for each DTR by considering linear combinations of the regression parameters. DTR AAB is estimated by $\text{exp}\left({\widehat{\alpha }}_{0\prime }\right)$, DTR AAC is estimated by $\text{exp}\left({\widehat{\alpha }}_{0\prime }+{\widehat{\alpha }}_{3\prime }\right)$, DTR BBA is estimated by $\text{exp}\left({\widehat{\alpha }}_{0\prime }+{\widehat{\alpha }}_{1\prime }\right)$, DTR BBC is estimated by $\text{exp}\left({\widehat{\alpha }}_{0\prime }+{\widehat{\alpha }}_{1\prime }+{\widehat{\alpha }}_{4\prime }\right)$, DTR CCA is estimated by $\text{exp}\left({\widehat{\alpha }}_{0\prime }+{\widehat{\alpha }}_{2\prime }\right)$, and DTR CCB is estimated by $\text{exp}\left({\widehat{\alpha }}_{0\prime }+{\widehat{\alpha }}_{2\prime }+{\widehat{\alpha }}_{5\prime }\right)$. The variances of the estimated response rates for each DTR are calculated using the Delta method, where the variances and covariances of the estimated parameters in the models can be computed by the robust (sandwich) variance estimators.′

4. Simulation results

4.1. Scenarios

In order to compare the performance of the methods described in Section 3, we conducted simulation studies where we estimated the response rates of treatments or DTRs and their variances from the two-stage snSMART design shown in Fig. 1. We outline the data generation process here. Each arm in stage 1 contains exactly one-third of the subjects. Subject responses to first-stage treatments A, B and C are generated from Bernoulli distributions with parameters π_A, π_B and π_C, respectively. Second-stage responses for the responders to first-stage treatments are generated from Bernoulli distributions with parameters specified as the products of first-stage response rates and the corresponding linkage parameters: β_1Aπ_A, β_1Bπ_B, and β_1Cπ_C. Similarly, second-stage responses for the non-responders to first-stage treatments are generated from Bernoulli distributions with parameters specified as the products of β_0j and π_j′, j = A, B, C. For example, second-stage responses for the subjects who do not respond to treatment A in the first stage and receive the treatment B in the second stage (subgroup 2 in Fig. 1) are generated from Bernoulli(β_0Aπ_B).

Four sets of scenarios are considered in our study with different true treatment response rates. In scenarios 1a-c and 2a-c, two potentially active treatments A and B have the same response rates, but the treatments A and B in scenarios 2a-c have even higher response rate. The true response rates in these two sets of scenarios resemble potential settings in FSGS. In scenarios 3a-c, only one of the potentially active treatments is truly better than the standard of care in terms of response rate. In scenarios 4a-c, both potentially active treatments have higher response rates than that of the standard of care, but A has a even higher response rate. In each set of scenarios, there are three different combinations of linkage parameters β_0j and β_1j, j = A, B, C, with different assumptions. The true parameter values of each scenario are shown in Table 1. In the scenarios ending with a, β_1j, j = A, B, C, are assumed equal, and β_0j, j = A, B, C, only depend on the first stage treatment. For example, for the non-responders to treatment A, their linkage parameters are 0.8 regardless of which alternative treatments they receive in the second stage. In the scenarios ending with b, both β_0j and β_1j depend on the first stage treatements. In the scenarios ending with c, the linkage parameters for non-responders depend on both first and second stage treatments, which violates the assumption on β_0j of the BJSM and JSRM.

Table 1.

Response rates and linkage parameter values used to generate data for all scenarios.

Scenario		πA	πb	πC	β1A	β1B	β1C	β0AB	β0AC	β0BA	β0AC	β0CA	β0CB
1	a	0.40	0.40	0.20	1.0	1.0	1.0	0.8	0.8	0.6	0.6	0.4	0.4
	b				1.5	1.0	0.5	0.8	0.8	0.6	0.6	0.4	0.4
	c				1.5	1.0	0.5	0.65	0.75	0.7	0.6	0.75	0.45
2	a	0.45	0.45	0.20	1.0	1.0	1.0	0.8	0.8	0.6	0.6	0.4	0.4
	b				1.5	1.0	0.5	0.8	0.8	0.6	0.6	0.4	0.4
	c				1.5	1.0	0.5	0.65	0.75	0.7	0.6	0.75	0.45
3	a	0.45	0.20	0.20	1.0	1.0	1.0	0.8	0.8	0.6	0.6	0.4	0.4
	b				1.5	1.0	0.5	0.8	0.8	0.6	0.6	0.4	0.4
	c				1.5	1.0	0.5	0.65	0.75	0.7	0.6	0.75	0.45
4	a	0.45	0.30	0.20	1.0	1.0	1.0	0.8	0.8	0.6	0.6	0.4	0.4
	b				1.5	1.0	0.5	0.8	0.8	0.6	0.6	0.4	0.4
	c				1.5	1.0	0.5	0.65	0.75	0.7	0.6	0.75	0.45

π_j (j = A, B, C) is the response rate of treatment J in the first stage. β_1j is the linkage parameter for first stage responders to j, and β_0jj′ is the linkage parameter for first stage non-responders to j who receive j′ in the second stage.

4.2. Estimation with Bayesian and frequentist methods

We evaluate the response rate estimates of treatments obtained using different methods within each set of scenarios. Since the models here differ from those in Wei et al. [12], we compare the estimates from the BJSM, JSRM and first stage maximum likelihood estimate (FSMLE, the MLE of response rates using only first stage outcomes). For each scenario, we simulate 1000 replications and obtain first stage treatment effect estimates using the BJSM, JSRM and FSMLE. Table 3 shows the biases and rMSEs of the estimated treatment response rates in all six scenarios with the given sample sizes. The sample sizes used in these scenarios were calculated using the method in Section 4.3.

Table 3.

The bias and root mean squared error (rMSE) of the treatment response rate estimates using Bayesian Joint Stage Model (BJSM), Joint Stage Regression Model (JSRM), and first stage MLE (FSMLE).

Scenario		BJSM		JSRM		FSMLE
		Bias	rMSE	Bias	rMSE	Bias	rMSE
1a	πA	−0.031	0.068	−0.001	0.069	0.000	0.072
	πB	−0.021	0.065	0.001	0.069	0.000	0.072
	πC	−0.009	0.047	−0.002	0.052	−0.003	0.058
1b	πA	−0.020	0.062	−0.001	0.069	0.000	0.072
	πB	−0.020	0.065	0.001	0.069	0.000	0.072
	πC	−0.014	0.048	−0.002	0.052	−0.003	0.058
1c	πA	−0.001	0.056	0.020	0.071	0.000	0.072
	πB	−0.040	0.072	−0.021	0.071	0.000	0.072
	πC	−0.016	0.049	−0.002	0.052	−0.003	0.058
2a	πA	−0.040	0.083	−0.001	0.086	−0.001	0.089
	πB	−0.029	0.080	0.001	0.086	0.000	0.089
	πC	−0.010	0.057	−0.003	0.064	−0.004	0.071
2b	πA	−0.026	0.075	−0.001	0.086	−0.001	0.089
	πB	−0.029	0.080	0.001	0.086	0.000	0.089
	πC	−0.016	0.057	−0.003	0.064	−0.004	0.071
2c	πA	−0.004	0.068	0.022	0.088	−0.001	0.089
	πB	−0.047	0.086	−0.023	0.087	0.000	0.089
	πC	−0.017	0.058	−0.003	0.064	−0.004	0.071
3a	πA	−0.042	0.077	−0.002	0.076	0.000	0.078
	πB	−0.009	0.051	0.000	0.057	0.000	0.062
	πC	−0.009	0.049	−0.002	0.056	−0.003	0.062
3b	πA	−0.030	0.069	−0.002	0.076	0.000	0.078
	πB	−0.009	0.051	0.000	0.057	0.000	0.062
	πC	−0.015	0.050	−0.002	0.056	−0.003	0.062
3c	πA	−0.009	0.060	0.014	0.076	0.000	0.078
	πB	−0.019	0.052	−0.013	0.056	0.000	0.062
	πC	−0.017	0.050	−0.004	0.055	−0.003	0.062
4a	πA	−0.039	0.076	−0.002	0.075	0.000	0.078
	πB	−0.016	0.061	0.000	0.067	−0.001	0.071
	πC	−0.009	0.049	−0.002	0.055	−0.003	0.062
4b	πA	−0.028	0.069	−0.002	0.075	0.000	0.078
	πB	−0.016	0.061	0.000	0.067	−0.001	0.071
	πC	−0.015	0.050	−0.002	0.055	−0.003	0.062
4c	πA	−0.007	0.060	0.018	0.077	0.000	0.078
	πB	−0.031	0.065	−0.018	0.067	−0.001	0.071
	πC	−0.017	0.051	−0.004	0.055	−0.003	0.062

The sample sizes for scenarios 1a-c, 2a-c, 3a-c, and 4a-c, are 135, 90, 120 and 120, respectively.

In all scenarios, the BJSM has the largest biases among the three models, while the biases for JSRM and FSMLE are negligible. The only exceptions are scenario c’s where the biases for JSRM are larger than that of the FSMLE due to the fact that model assumptions on the linkage parameters for non-responders are violated, i.e., the linkage parameters for non-responders depend on both first and second stage treatments. When looking at the rMSEs, we find that BJSM performs slightly better than JSRM, and the rMSE of the FSMLE is the highest, which can be expected because only first-stage outcomes are used. Thus, the BJSM is more efficient than the other methods, but efficiency comes at a price of computational intensiveness and some bias. The BJSM and JSRM are preferred over the FSMLE based on efficiency. The choice between the BJSM and JSRM may depend on the bias-variance tradeoff and computational resources.

We can evaluate the response rate estimates of DTRs obtained using different methods within each set of scenarios as well. We compare the estimates from the BJSM, JSRM and WRRM. The expected DTR response rates are shown in Table 2. Fig. 2 shows the absolute values of mean biases and mean rMSEs of the estimated DTR response rates in all six scenarios with the calculated sample sizes. The detailed results for Scenarios 1a-c through 4a-c are tabulated in supplementary materials. Estimates from the BJSM have the largest biases among the three methods, while the biases from the other two methods are negligible. However, estimates from BJSM have the smallest rMSEs, and estimates from WRRM have the largest rMSEs because only second stage outcomes are used. Similar to the results from first stage treatment effect estimation, BJSM and JSRM are preferred over the WRRM based on the efficiency, and the choice between these two methods need to account for the bias-variance tradeoff and computational resources.

Table 2.

The expected response rate of dynamic treatment regimens (DTRs) for each scenario in Table 1.

Scenario		πAAB	πAAC	πBBA	πBBA	πCCA	πCCB
	a	0.352	0.256	0.304	0.232	0.168	0.168
	b	0.432	0.336	0.304	0.232	0.148	0.148
	c	0.396	0.330	0.328	0.232	0.260	0.164
2	a	0.401	0.291	0.351	0.268	0.184	0.184
	b	0.502	0.392	0.351	0.268	0.164	0.164
	c	0.465	0.386	0.376	0.268	0.290	0.182
3	a	0.291	0.291	0.256	0.136	0.184	0.104
	b	0.392	0.392	0.256	0.136	0.164	0.084
	c	0.375	0.386	0.292	0.136	0.290	0.092
4	a	0.334	0.291	0.279	0.174	0.184	0.136
	b	0.436	0.392	0.279	0.174	0.164	0.116
	c	0.411	0.386	0.310	0.174	0.290	0.128

π_AAB corresponds to DTR AAB, and the rests are similar.

Fig. 2.

Left column: The absolute values of means of bias of DTR response rate estimates across Scenarios 1 to 4. Right column: The means of root mean squared error (rMSE) of DTR response rate estimates across Scenarios 1 to 4.

4.3. Sample size calculation via Dunnett’s method

A sample size calculation for an snSMART is available for comparing the first stage response rates [10]. This sample size calculation, however, is based on a frequentist method that does not use all of the second stage data and thus is not efficient [12]. Here, we present a simulation-based sample size calculation using the JSRM with six linkage parameters when interest is in comparing two active treatments to a control or standard of care with a specified family wise error rate and power. We use the JSRM for sample size calculation based on its computation speed and frequentist operating characteristics.

We apply Dunnett’s approach under GEE [2,8] to identify a significant difference between the two drugs of interest, in our setting the novel antifibrotic drug (A) and novel anti-inflammatory drug (B), with the standard of care (C). The detailed steps of the approach can be found in Orelien et al. [8]. Simulations are conducted to obtain the total sample size to achieve a family-wise type I error rate (α) of 10% and 80% power. Since we are performing two pair-wise comparisons (A vs. C and B vs. C), type I error rate is defined as the probability that either or both of the two p-values are smaller than the nominal α when all three drugs have same response rates, and power is defined as the probability that either or both of the two p-values are smaller than the nominal α if both drugs of interest truly have higher response rates than the that of the standard of care. One thousand replicates have been performed for each sample size. We show power curves in Fig. 3 under scenarios 1a, 2a, 3a and 4a given in Table 1. We find that the appropriate total sample sizes for these four scenarios are about 135, 90, 120 and 120, respectively. The total sample sizes for scenarios ending in “b” and “c” resemble that of the corresponding scenarios ending in “a” (results not shown). The result indicates that an snSMART comparing two active treatments to a control is feasible for rare disease studies because the sample size can be controlled at the level of about 100 individuals, and the comparison of DTRs can be performed simultaneously. Specifically, if the difference in the response rates between active treatments and the control is 0.25, the sample size of this snSMART can be as small as 90.

Fig. 3.

Power curve using JSRM with Dunnett’s approach. Two pair-wise comparisons (active treatment A vs. standard of care C and active treatment B vs. standard of care C) are performed for each run. Power is estimated by the proportion of runs in which one or both of the p values from the two pair-wise comparisons after Dunnett’s correction are smaller than the nominal α.

The R code for generating the power curves for sample size calculation and the estimation of response rates using the BJSM, JSRM and WRRM can be found in the supplementary material.

5. Discussion

Using FSGS as an illustrative example, we have outlined how an snSMART design can be implemented to test the efficacy of novel therapies for rare diseases. An snSMART design can address important clinical issues regarding the optimal agent for the disease and the individual patient as well as how treatments can be sequenced and tailored to produce long-term responses. Moreover, an snSMART design maximizes the amount of information that can be learned from each patient and is likely to enhance acceptance of clinical research by patients and their families, and therefore promote participation in clinical trials.

We extended the BJSM and JSRM beyond the work of Wei et al. [12] and focused on estimating both the first stage response rates and DTR effects from models that included six linkage parameters. In general, the estimators of response rates from the BJSM and JSRM are slightly biased relative to estimators from WRRM because the WRRM does not involve any assumptions on the linkage parameters. However, when we consider rMSE, which involves both bias and variance, the BJSM is the best among the all three models under every scenario and sample size. The BJSM and JSRM may be preferred over WRRM for studying the rare diseases under a SMART design because of the general low bias and high efficiency. The estimators of response rates of WRRM are least efficient because only second-stage responses are used.

An assumption of this snSMART design is that the disease of interest should be relatively stable, and that the disease status of individuals does not wax and wane dramatically if there is no change in intervention. In some diseases where this assumption might be violated, this snSMART design, as well as other multi-stage designs, may not be appropriate because the observed outcomes from an individual might reflect the random fluctuation of the disease status rather than the actual treatment effects.

Future work includes improving the BJSM and JSRM to include baseline and time-varying covariates. The WRRM can include baseline and/or intermediate variables to potentially improve the efficiency of the estimated effects. However, the BJSM does not easily lend itself to controlling for covariates, and the JSRM model can only control for baseline measures. Future research will focus on applying precision medicine in the Bayesian and JSRM methods so that variables, such as age, sex or adherence to the initial treatment, can be successfully incorporated into our models.

Bayesian analysis has been a recommended approach for trials in the rare disease setting since the analysis incorporates prior information, more can be gained from smaller sample sizes. Bayesian analysis, however, requires a shift in the expectations of results such that p-values are not generated at the end of a Bayesian analysis. The analysis instead can provide estimates of the response rates, credible intervals (similar to confidence intervals) and probabilities that the treatments differ in their efficacy (e.g., the probability that the standard of care results in 20% less efficacy than the anti-fibrotic therapy is 90%). The results from Bayesian analyses are often more intuitive and interpretable than frequentist results.