Statistical Evaluation of Absolute Change versus Responder Analysis in Clinical Trials

Abstract

In clinical trials, the primary analysis is often either a test of absolute/relative change in a measured outcome or a corresponding responder analysis. Though each of these tests may be reasonable, determining which test is most suitable for a particular research study is still an open question. These tests may require different sample sizes, define different clinically meaningful differences, and most importantly, lead to different study conclusions. This paper aims to compare a typical non-inferiority test using absolute change as the study endpoint to the corresponding responder analysis in terms of sample size requirements, statistical power, and hypothesis testing results. From numerical analysis, using absolute change as an endpoint generally requires a larger sample size; therefore, when the sample size is the same, the responder analysis has higher power. The cut-off value and non-inferiority margin are critical which can meaningfully impact whether the two types of endpoints yield conflicting conclusions. Specifically, an extreme cut-off value is more likely to cause different conclusions. However, this impact decreases as population variance increases. One important reason for conflicting conclusions is that the population distribution is not normal. To eliminate conflicting results, researchers should pay attention to the population distribution and cut-off value selection.

Graphical Abstract

1. Introduction

In clinical trials, an analysis of a primary study endpoint is often conducted to determine whether the intended studies will achieve study objective with a desired statistical power. In practice, investigators can consider four different kinds of primary endpoints or outcomes based on this single study objective: (i) absolute change (i.e., endpoint absolute change from baseline), (ii) relative change (e.g., endpoint percent change from baseline), (iii) responder analysis based on absolute change (i.e., an individual subject is defined as a responder if his/her absolute change in the primary endpoint has exceeded a pre-specified threshold known as a clinically meaningful improvement), and (iv) responder analysis based on relative change. Although analyses based on these endpoints all sound reasonable, the following statements are often of great concern to the principal investigators (Chow, 2011). First, clinically meaningful differences (improvement) of these derived endpoints may not directly translate to one another. Second, these derived endpoints generally have different sample size requirements. Third, and most importantly, these derived endpoints may not arrive the same statistical conclusion (based on the same data set). As a result, it is of particular interest to determine which type of primary endpoint is most appropriate and can best inform the disease status and treatment effect.

Some scholars criticized responder analysis due to loss of information, i.e., statistical power of a trial will be reduced if we categorize a continuous outcome into a binary variable (Snapinn and Jiang, 2007; Henschke et al., 2014). Though responder analysis may cost in power, it still has its own implement. For example, we need to use the original scale (continuous) outcome to make binary decision, such as whether the patient should be hospitalized. In a heterogeneous disease, a subset of patients may have more benefit than others, and then the distribution of the outcome variable would be not normal (Jones et al., 2016). If the trial is to investigate an additional second agent and the proportion of patients with more benefit may be of greater interest, a responder analysis is more suitable (Jones et al., 2016). According to its the benefits and unfavorable drawbacks, Henschke et al. (2014) suggested to use responder analysis as the secondary analysis to better interpret findings from the main analysis. However, it should be noted that analysis using absolute change as endpoint and corresponding responder analysis have different statistical properties. Hence, it is of great importance to investigate their differences in terms of statistical power, sample size and conclusion.

To study the relative performance of these derived endpoints, in addition to mathematical derivations, we further conduct numerical study and real case study, using data from a recent rehabilitation program study in lung transplant candidates and recipients (Byrd et al., 2022). One of the commonly used clinical indicators for patients with pulmonary disease is 6-minute walk distance (6MWD), which can be used as not only a prognostic factor but also as a health outcome variable (Tuppin et al., 2008). For example, 6MWD has been used to measure functional status and exercise capacity of lung transplant recipients or patients (Martinu et al, 2008; Munro et al, 2009). Some studies have used endpoint as change from baseline (absolute change) of 6MWD as the outcome variable to evaluate the performance of pulmonary disease treatment (Ryerson et al, 2014), while some considered a responder analysis using 6MWD (Gilbert et al., 2009; Stoilkova-Hartmann et al, 2015). An individual may be defined as a responder if he/she meets a pre-specified threshold of improvement in 6MWD, otherwise he/she will be defined as a non-responder. As an example, Stoilkova-Hartmann et al. (2015) considered classifying patient performance after rehabilitation using the following criteria: 6MWD increment ≥ 50m is considered good, ≥ 25 to < 50m is moderate, and < 25m a non-responder. However, in another study, Holland et al. (2014) reported that the minimal important difference of change of 6MWD in chronic respiratory disease was 25 to 33m. Holland et al. (2017) used 25m as the threshold for equivalence in the change of 6MWD. Though the wide range of 6MWD is generally accepted as 25 to 30m, the exact threshold of change in 6MWD which is considered of clinically meaningful is debatable.

In this case study, for simplicity, we will focus on statistical evaluation of rehabilitation program in lung transplant candidates and recipients in term of absolute change of 6MWD and the responder analysis based on a pre-specified threshold (improvement) of 6MWD using absolute change. A comparison between the absolute change and responder analysis with various pre-specified thresholds is made in terms of sample size requirement and statistical power. In the next section, we present statistical methods for an analysis using absolute change as the study endpoint as well as the corresponding responder analysis. Additionally, we compare the performances of these study endpoints in terms of statistical power, sample size and study results/conclusions. In Section 3, we discuss a numerical analysis of the comparison between absolute change and responder analysis and the case study of the rehabilitation program in lung transplant candidate and recipients. Brief concluding remarks and recommendations are given in the last section of this article.

2. Methods

2.1. Hypothesis Testing for Efficacy

In a randomized clinical trial evaluating the performance of a new drug or a new treatment as compared to an active control (e.g., standard of care), non-inferiority testing is commonly considered. The success of a non-inferiority trial depends upon the selection of study endpoint and the non-inferiority margin. As indicated earlier, for a given study endpoint, there are four types of primary endpoints, namely, absolute change (e.g., endpoint change from baseline), relative change (e.g., endpoint percent change from baseline), responder analysis based on a pre-specified improvement (threshold) of absolute change, and responder analysis based on a pre-specified improvement (threshold) of relative change. As a result, the inference from a responder analysis is very sensitive to the pre-specified threshold (cutoff) value (Chow and Song, 2015). For simplicity and illustration purposes, in this artice, we will examine the performaces of the first two primary endpoints: absolute change and a corresponding responder analysis.

We assume a two-arm parallel randomized clinical trial comparing a test treatment (T) and an active control (C) with 1:1 treatment allocation ratio. Let W_1ij and W_2ij be the original response of ith patient in jth treatment group at baseline and post-treatment, where i = 1, …, n_j and j = C, T, respectively. Furthermore, W_1ij is assumed to follow lognormal distribution $LN({\mu }_j,{\sigma }_j^2)$, and W_2ij = W_1ij(1 + Δ_ij), where ${\Delta }_{ij}~LN({\mu }_{{\Delta }_{ij}},{\sigma }_{{\Delta }_{ij}}^2)$. Hence, the absolute change from baseline is

$$W_{2ij}-W_{1ij}=W_{1ij}{\text{Δ}}_{ij}~LN\left({\mu }_j+{\mu }_{{\Delta }_j},{\sigma }_j^2+{\sigma }_{{\Delta }_j}^2\right),$$ (1)

where W_1ij and Δ_ij are assumed to be independent. Let X_ij = log(W_2ij − W_1ij) represents the log absolute change, then $X_{ij}~N({\mu }_j+{\mu }_{{\Delta }_j},{\sigma }_j^2+{\sigma }_{{\Delta }_j}^2)$. Let x_ij denote the observations of random variable X_ij. The reason to use W_1ij and W_2ij instead directly use X_ij following normal distribution is that the same notation can be used to denote relative change. For example, $Y_{ij}=\text{log}(\frac{W_{2ij}-W_{1ij}}{W_{1ij}})$ can represent log relative change, which follows $N({\mu }_{{\Delta }_{ij}},{\sigma }_{{\Delta }_{ij}}^2)$. Though relative change endpoint is not the focus of this paper, this notation will benefit future studies.

The outcome variable for responder analysis based on a pre-specified absolute change is then given by $r_{A_j}=\frac{\#\{x_{ij}>c_1\}}{n_j}$, where c₁ is the cutoff value. Then the endpoint for the responder analysis becomes p_Aj = E[r_Aj]. For sufficiently large sample size, it can be verified that r_Aj asymptotically follows $N\left(p_{A_j},\frac{p_{A_j}\left(1-p_{A_j}\right)}{n_j}\right)$ (Chow, 2011). According to the definition of p_Aj,

$$\begin{gathered} p_{A_j}=E\left[r_{A_j}\right]=P\left(X_{ij}>c_1\right)=P\left(\frac{X_{ij}-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}>\frac{c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}\right) \\ =1-\text{Φ}\left(\frac{c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}\right), \end{gathered}$$ (2)

where Φ(·) is the cumulative distribution function (CDF) of standard normal distribution. The hypotheses for non-inferority testing based on the derived endpoint of absolute change and the corresponding responder analysis can be set up as follows.

1. Absolute change:

   $${\text{H}}_0:\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\ge {\text{δ}}_1\text{v.s.}{\text{H}}_{\text{A}}:\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)<{\text{δ}}_1,$$ (3)

   where δ₁ is the non-inferiority margin in hypothesis teting using absolute change.
2. Responder analysis based on a pre-specified threshold (improvement) of absolute change:

   $${\text{H}}_0:{\text{p}}_{{\text{A}}_{\text{C}}}-{\text{p}}_{{\text{A}}_{\text{T}}}\ge {\text{δ}}_2\text{v.s.}{\text{H}}_{\text{A}}:{\text{p}}_{{\text{A}}_{\text{C}}}-{\text{p}}_{{\text{A}}_{\text{T}}}<{\text{δ}}_2.$$ (4)

   where δ₂ is the non-inferiority margin in hypothesis testing using responder analysis.

For a non-inferority test based on the derived endpoint of absolute difference, the Z test statistic under null hypothesis in Equation (3) is given by

$$Z_1=\frac{{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}+{\delta }_1}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2}{n_1}+\frac{{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n_1}}}=\frac{{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}+{\delta }_1}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n_1}}}~N\left(0,1\right),$$ (5)

where ${\stackrel{-}{x}}_{\mathrm{T}}$ and ${\stackrel{-}{x}}_{\mathrm{C}}$ are the sample mean of absolute change in treatment and control group, and n₁ is the sample size of the treatment or control group, assuming the allocation ratio is 1:1. Let δ_1A denote the true sample mean difference. The corresponding statistical power can be written as

$$\begin{gathered} {\text{power}}_1=\text{P}\left({\text{Reject H}}_0{\text{|H}}_{\text{A}}\text{is true}\right)=P\text{(}{Z}_1>z_{1-\alpha }\text{|}{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}={\delta }_{1A}) \\ =\text{P}\left({\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}>z_{1-\alpha }\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n_1}}-{\delta }_1|{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}={\delta }_{1A}\right) \\ =P\left(\frac{{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}-{\delta }_{1A}}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n_1}}}>\frac{z_{1-\alpha }\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n_1}}-{\delta }_1-{\delta }_{1A}}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n_1}}}\right) \\ =1-\text{Φ}\left(z_{1-\alpha }-\frac{{\delta }_1+{\delta }_{1A}}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n_1}}}\right). \end{gathered}$$ (6)

The sample size requirement for the non-inferiority test using absolute difference can then be obtained as follows

$$n_1=\frac{2{\left(z_{1-\alpha }+z_{\beta }\right)}^2\left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}{{\left[\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)-{\delta }_1\right]}^2}.$$ (7)

For a non-inferiority test of responder analysis based on a pre-specified threshold (improvement) of absolute difference, the Z test statistic under null hypothesis in Equation (4) can be derived as follows

$$Z_2=\frac{{\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}+{\delta }_2}{\sqrt{\frac{{\text{r}}_{{\text{A}}_{\text{T}}}\left(1-{\text{r}}_{{\text{A}}_{\text{T}}}\right)}{n_2}+\frac{{\text{r}}_{{\text{A}}_{\text{C}}}\left(1-{\text{r}}_{{\text{A}}_{\text{C}}}\right)}{n_2}}}=\frac{{\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}+{\delta }_2}{\sqrt{\frac{{\text{r}}_{{\text{A}}_{\text{T}}}\left(1-{\text{r}}_{{\text{A}}_{\text{T}}}\right)+{\text{r}}_{{\text{A}}_{\text{C}}}\left(1-{\text{r}}_{{\text{A}}_{\text{C}}}\right)}{n_2}}}~N\left(0,1\right),$$ (8)

where r_AT and r_AC are the sample proportions in the treatment and the control group, respectively, and n₂ is the sample size of the treatment or control group, assuming the allocation ratio is 1:1. Similarly, let δ_2A denote the true proportion difference, then the corresponding statistical power is

$$\begin{gathered} powe{r}_2=P\left(Reject{H}_0\text{|}{H}_{A}istrue\right)=P\text{(}{Z}_2>z_{1-\alpha }|{\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}={\delta }_{2A}) \\ =\text{P}\left({\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}>z_{1-\alpha }\sqrt{\frac{{\text{r}}_{{\text{A}}_{\text{T}}}\left(1-{\text{r}}_{{\text{A}}_{\text{T}}}\right)+{\text{r}}_{{\text{A}}_{\text{C}}}\left(1-{\text{r}}_{{\text{A}}_{\text{C}}}\right)}{n_2}}-{\delta }_2|{\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}={\delta }_{2A}\right) \\ =1-\text{Φ}\left(z_{1-\alpha }-\frac{{\delta }_2+{\delta }_{2A}}{\sqrt{\frac{{\text{r}}_{{\text{A}}_{\text{T}}}\left(1-{\text{r}}_{{\text{A}}_{\text{T}}}\right)+{\text{r}}_{{\text{A}}_{\text{C}}}\left(1-{\text{r}}_{{\text{A}}_{\text{C}}}\right)}{n_2}}}\right) \\ \approx 1-\text{Φ}\left(z_{1-\alpha }-\frac{{\delta }_2+{\delta }_{2A}}{\sqrt{\frac{{\text{p}}_{{\text{A}}_{\text{T}}}\left(1-{\text{p}}_{{\text{A}}_{\text{T}}}\right)+{\text{p}}_{{\text{A}}_{\text{C}}}\left(1-{\text{p}}_{{\text{A}}_{\text{C}}}\right)}{n_2}}}\right), \end{gathered}$$ (9)

where the last approximate equation holds using Slutsky’s theorem (Chow, 2011). The sample size requirement for non-inferiority test for the responder analysis based on a pre-specified threshold (improvement) of absolute difference is then given by

$$n_2=\frac{2{\left(z_{1-\alpha }+z_{\beta }\right)}^2\left(p_{A_C}\left(1-p_{A_C}\right)+p_{A_T}(1-p_{A_T})\right)}{{\left(p_{A_C}-p_{A_T}-{\delta }_2\right)}^2}.$$ (10)

2.2. Statistical Power Comparison in Non-inferiority Tests

Many previous studies have suggested avoiding relative difference due to statistical inefficiency (Vickers, 2001). Following their ideas, we instead consider a comparison of non-inferiority tests using absolute change and a responder analysis using absolute change in terms of statistical power. The required sample sizes and conclusion comparison for non-inferiority tests are also shown in this section. Here, let AC denote absolute change, and PAC denote responder analysis using absolute change.

From the formula of statistical power of non-inferiority test shown in Section 2, the power difference can be computed using the cumulative distribution function (CDF) of N(0,1). Using Taylor expansion, the CDF of N(0,1) Φ(·) can be written as

$$\text{Φ}\left(x\right)=\frac{1}{\sqrt{2\pi }}\sum _{i=0}^n\frac{{\left(-1\right)}^n}{n!2^n\left(2n+1\right)}{x}^{2n+1}+\frac{1}{2}.$$ (11)

Keeping the first term of Taylor expansion in Equation (11), then Φ(x₁) − Φ(x₂) can be simplified as

$$\begin{gathered} \text{Φ}\left(x_1\right)-\text{Φ}\left(x_2\right)=\frac{1}{\sqrt{2\pi }}\left(\sum _{i=0}^n\frac{{\left(-1\right)}^n}{n!2^n\left(2n+1\right)}{x}_1^{2n+1}-\sum _{i=0}^n\frac{{\left(-1\right)}^n}{n!2^n\left(2n+1\right)}{x}_2^{2n+1}\right) \\ =\frac{1}{\sqrt{2\pi }}\sum _{i=0}^n\frac{{\left(-1\right)}^n}{n!2^n\left(2n+1\right)}\left(x_1^{2n+1}-x_2^{2n+1}\right)\approx \frac{1}{\sqrt{2\pi }}\left(x_1-x_2\right). \end{gathered}$$ (12)

To compare the statistical power of a non-inferiority test using the absolute change endpoint with the statistical power for a responder analysis using absolute change endpoint, we start with first simplify p_Aj. Using Equation (11), p_Aj can be written as

$$\begin{gathered} p_{A_j}=1-\text{Φ}\left(\frac{c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}\right)=1-\frac{1}{\sqrt{2\pi }}\cdot \frac{c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}-\frac{1}{2} \\ =\frac{1}{2}-\frac{1}{\sqrt{2\pi }}\cdot \frac{c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}. \end{gathered}$$ (13)

And

$$\begin{gathered} p_{A_j}\left(1-p_{A_j}\right)=\left(\frac{1}{2}-\frac{1}{\sqrt{2\pi }}\cdot \frac{c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}\right)\left(\frac{1}{2}+\frac{1}{\sqrt{2\pi }}\cdot \frac{c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}\right) \\ =\frac{1}{4}-\frac{1}{2\pi }{\left(\frac{c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)}{\sqrt{{\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2}}\right)}^2=\frac{1}{4}-\frac{{\left[c_1-\left({\text{μ}}_{\text{j}}+{\text{μ}}_{{\text{Δ}}_{\text{j}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{j}}^2+{\sigma }_{{\text{Δ}}_j}^2\right)}. \end{gathered}$$ (14)

Hence,

$$\begin{gathered} p_{A_C}-p_{A_T}=\frac{1}{4}-\frac{{\left[c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}-\left(\frac{1}{4}-\frac{{\left[c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2\right)}\right) \\ =\frac{{\left[c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2\right)}-\frac{{\left[c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}, \end{gathered}$$ (15)

and

$$\begin{gathered} p_{A_C}\left(1-p_{A_C}\right)+p_{A_T}\left(1-p_{A_T}\right)=\frac{1}{4}-\frac{{\left[c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}+\frac{1}{4}-\frac{{\left[c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2\right)} \\ =\frac{1}{2}-\frac{{\left[c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2\right)}-\frac{{\left[c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}. \end{gathered}$$ (16)

If we assume the sample sizes of non-inferority test using absolute change and corresponding responder analysis are the same, denoted as n, using Equation (12), the difference between power₁ and power₂ can be written as

$$\begin{gathered} powe{r}_1-powe{r}_2 \\ =1-\text{Φ}\left(z_{1-\alpha }-\frac{{\delta }_1+{\delta }_{1A}}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n}}}\right)-1+\text{Φ}\left(z_{1-\alpha }-\frac{{\delta }_2+{\delta }_{2A}}{\sqrt{\frac{{\text{p}}_{{\text{A}}_{\text{T}}}\left(1-{\text{p}}_{{\text{A}}_{\text{T}}}\right)+{\text{p}}_{{\text{A}}_{\text{C}}}\left(1-{\text{p}}_{{\text{A}}_{\text{C}}}\right)}{n}}}\right) \\ =\text{Φ}\left(z_{1-\alpha }-\frac{{\delta }_2+{\delta }_{2A}}{\sqrt{\frac{{\text{p}}_{{\text{A}}_{\text{T}}}\left(1-{\text{p}}_{{\text{A}}_{\text{T}}}\right)+{\text{p}}_{{\text{A}}_{\text{C}}}\left(1-{\text{p}}_{{\text{A}}_{\text{C}}}\right)}{n}}}\right)-\text{Φ}\left(z_{1-\alpha }-\frac{{\delta }_1+{\delta }_{1A}}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n}}}\right) \\ =\frac{1}{\sqrt{2\pi }}\left(z_{1-\alpha }-\frac{{\delta }_2+{\delta }_{2A}}{\sqrt{\frac{{\text{p}}_{{\text{A}}_{\text{T}}}\left(1-{\text{p}}_{{\text{A}}_{\text{T}}}\right)+{\text{p}}_{{\text{A}}_{\text{C}}}\left(1-{\text{p}}_{{\text{A}}_{\text{C}}}\right)}{n}}}-z_{1-\alpha }+\frac{{\delta }_1+{\delta }_{1A}}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n}}}\right) \\ =\frac{1}{\sqrt{2\pi }}\left(\frac{{\delta }_1+{\delta }_{1A}}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n}}}-\frac{{\delta }_2+{\delta }_{2A}}{\sqrt{\frac{{\text{p}}_{{\text{A}}_{\text{T}}}\left(1-{\text{p}}_{{\text{A}}_{\text{T}}}\right)+{\text{p}}_{{\text{A}}_{\text{C}}}\left(1-{\text{p}}_{{\text{A}}_{\text{C}}}\right)}{n}}}\right) \\ =\sqrt{\frac{n}{2\pi }}\left(\frac{{\delta }_1+{\delta }_{1A}}{\sqrt{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}}-\frac{{\delta }_2+{\delta }_{2A}}{\sqrt{\frac{1}{2}-\frac{{\left[c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2\right)}-\frac{{\left[c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}}}\right). \end{gathered}$$ (17)

2.3. Sample Size Comparison in Non-inferiority Tests

From Equation (15) and (16), the sample size for the responder analysis using absolute change endpoint in Equation (10) can be written as

$$n_2=\frac{2{\left(z_{1-\alpha }+z_{\beta }\right)}^2\left(\frac{1}{2}-\frac{{\left[c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2\right)}-\frac{{\left[c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}\right)}{{\left(\frac{{\left[c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2\right)}-\frac{{\left[c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}-{\delta }_2\right)}^2}.$$ (18)

When the significance level and desired statistical power are the same, we can compare the necessary sample size for a responder analysis to a test from absolute change with the ratio

$$\frac{n_2}{n_1}=\frac{\frac{2{\left(z_{1-\alpha }+z_{\beta }\right)}^2\left(p_{A_C}\left(1-p_{A_C}\right)+p_{A_T}\left(1-p_{A_T}\right)\right)}{{\left(p_{A_C}-p_{A_T}-{\delta }_2\right)}^2}}{\frac{2{\left(z_{1-\alpha }+z_{\beta }\right)}^2\left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}{{\left[\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)-{\delta }_1\right]}^2}}=\frac{\frac{p_{A_C}\left(1-p_{A_C}\right)+p_{A_T}\left(1-p_{A_T}\right)}{{\left(p_{A_C}-p_{A_T}-{\delta }_2\right)}^2}}{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{{\left[\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)-{\delta }_1\right]}^2}}=\frac{p_{A_C}\left(1-p_{A_C}\right)+p_{A_T}\left(1-p_{A_T}\right)}{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}\cdot {\left[\frac{\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)-{\delta }_1}{p_{A_C}-p_{A_T}-{\delta }_2}\right]}^2=\frac{\frac{1}{2}-\frac{{\left[c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2\right)}-\frac{{\left[c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)\right]}^2}{2\pi \left({\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2\right)}}{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{\left[\frac{\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)-{\delta }_1}{\frac{c_1-\left({\text{μ}}_{\text{T}}+{\text{μ}}_{{\text{Δ}}_{\text{T}}}\right)}{\sqrt{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2}}-\frac{c_1-\left({\text{μ}}_{\text{C}}+{\text{μ}}_{{\text{Δ}}_{\text{C}}}\right)}{\sqrt{{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}}-{\delta }_2}\right]}^2.$$ (19)

2.4. Conflict Probability in Non-inferiority Tests

In this section, we aim to investigate the probabilities of a non-inferority test using absolute change as the endpoint and the corresponding responder analysis having similar or different conclusions. We assume the samples used to conduct these two types of non-inferority test are the same. Thus, there are four possible types of events:

• Both AC and PAC reject H₀

  $$\text{P}({\text{AC reject H}}_0{\text{and PAC reject H}}_0)=P(Z_1>z_{1-\alpha },Z_2>z_{1-\alpha })=P\left(\frac{{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}+{\delta }_1}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n}}}>z_{1-\alpha },\frac{{\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}+{\delta }_2}{\sqrt{\frac{{\text{r}}_{{\text{A}}_{\text{T}}}\left(1-{\text{r}}_{{\text{A}}_{\text{T}}}\right)+{\text{r}}_{{\text{A}}_{\text{C}}}\left(1-{\text{r}}_{{\text{A}}_{\text{C}}}\right)}{n}}}>z_{1-\alpha }\right).$$ (20)
• AC fail to reject H₀, whereas PAC reject H₀

  $$\text{P}({\text{AC fail to reject H}}_0{\text{and PAC reject H}}_0)=P(Z_1\le z_{1-\alpha },Z_2>z_{1-\alpha })=P\left(\frac{{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}+{\delta }_1}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n}}}\le z_{1-\alpha },\frac{{\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}+{\delta }_2}{\sqrt{\frac{{\text{r}}_{{\text{A}}_{\text{T}}}\left(1-{\text{r}}_{{\text{A}}_{\text{T}}}\right)+{\text{r}}_{{\text{A}}_{\text{C}}}\left(1-{\text{r}}_{{\text{A}}_{\text{C}}}\right)}{n}}}>z_{1-\alpha }\right).$$ (21)
• AC reject H₀, whereas PAC fail to reject H₀

  $$\text{P}({\text{AC reject H}}_0{\text{and PAC fail to reject H}}_0)=P(Z_1>z_{1-\alpha },Z_2\le z_{1-\alpha })=P\left(\frac{{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}+{\delta }_1}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n}}}>z_{1-\alpha },\frac{{\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}+{\delta }_2}{\sqrt{\frac{{\text{r}}_{{\text{A}}_{\text{T}}}\left(1-{\text{r}}_{{\text{A}}_{\text{T}}}\right)+{\text{r}}_{{\text{A}}_{\text{C}}}\left(1-{\text{r}}_{{\text{A}}_{\text{C}}}\right)}{n}}}\le z_{1-\alpha }\right).$$ (22)
• Both AC and PAC fail to reject H₀

  $$\text{P}({\text{AC fail to reject H}}_0{\text{and PAC fail to reject H}}_0)=P(Z_1\le z_{1-\alpha },Z_2\le z_{1-\alpha })=P\left(\frac{{\overline{x}}_{\text{T}}-{\overline{x}}_{\text{C}}+{\delta }_1}{\sqrt{\frac{{\text{σ}}_{\text{T}}^2+{\sigma }_{{\text{Δ}}_T}^2+{\text{σ}}_{\text{C}}^2+{\sigma }_{{\text{Δ}}_C}^2}{n}}}\le z_{1-\alpha },\frac{{\text{r}}_{{\text{A}}_{\text{T}}}-{\text{r}}_{{\text{A}}_{\text{C}}}+{\delta }_2}{\sqrt{\frac{{\text{r}}_{{\text{A}}_{\text{T}}}\left(1-{\text{r}}_{{\text{A}}_{\text{T}}}\right)+{\text{r}}_{{\text{A}}_{\text{C}}}\left(1-{\text{r}}_{{\text{A}}_{\text{C}}}\right)}{n}}}\le z_{1-\alpha }\right).$$ (23)

3. Results

In this section, a numerical analysis using simulated data is conducted to investigate the difference between using absolute change as an endpoint and the corresponding responder analysis in terms of sample size requirement, statistical power, and non-inferiority test conclusion. Responses are assumed to follow a normal distribution. The allocation ratio is 1:1. The simulation is conducted 1000 times. Additionally, a case study is established to investigate the difference between a typical non-inferiority test and responder analysis using real clinical data from Byrd et al. (2022). Again, AC denotes typical non-inferiority test using absolute change as the endpoint, and PAC denotes the corresponding responder analysis. The significance level is 0.05, and the desired power is 0.80.

3.1. Numerical analysis

According to Equation (7), the sample size of AC is associated with the population mean, population variance, and the non-inferiority margin. Treatment group population mean is set to 0.2 and 0.3, control group population mean is set to 0, and population variance of both groups as 1.0, 2.0 and 3.0. Table 1 presents the required sample size of AC to achieve 80% statistical power. The sample size is associated with the effect size and the non-inferiority margin. When the effect size is fixed, a larger non-inferiority margin will lead to a smaller sample size in AC; when the non-inferiority margin is fixed, a larger effect size will lead to a smaller sample size in AC. Similarly, from Equation (10), the sample size of PAC is additionally related to the cut-off value (threshold) used to determine responders. As shown in Table 2, the impact of effect size and non-inferiority margin on sample size is the same as Table 1, when cut-off value is fixed. However, the impact of the cut-off value on the sample size calculation is quite complex, since its impact is associated with not only its absolute value but also the population mean and variance.

Table 1.

Sample sizes for non-inferiority test using absolute change endpoint (AC).

	μT + μΔT = 0.2									μT + μΔT = 0.3
${\sigma }_T^2+{\sigma }_{{\Delta }_T}^2$	1.0			2.0			3.0			1.0			2.0			3.0
${\sigma }_C^2+{\sigma }_{{\mathrm{\Delta }}_C}^2$	1.0	2.0	3.0	1.0	2.0	3.0	1.0	2.0	3.0	1.0	2.0	3.0	1.0	2.0	3.0	1.0	2.0	3.0
δ1 = 0.25	246	368	490	368	490	612	490	612	734	164	246	328	246	328	410	328	410	492
δ1 = 0.30	198	298	396	298	396	496	396	496	594	138	208	276	208	276	344	276	344	414
δ1 = 0.35	164	246	328	246	328	410	328	410	492	118	176	236	176	236	294	236	294	352
δ1 = 0.40	138	208	276	208	276	344	276	344	414	102	152	202	152	202	254	202	254	304
δ1 = 0.45	118	176	236	176	236	294	236	294	352	88	132	176	132	176	220	176	220	264
δ1 = 0.50	102	152	202	152	202	254	202	254	304	78	116	156	116	156	194	156	194	232
δ1 = 0.55	88	132	176	132	176	220	176	220	264	70	104	138	104	138	172	138	172	206
δ1 = 0.60	78	116	156	116	156	194	156	194	232	62	92	124	92	124	154	124	154	184
δ1 = 0.65	70	104	138	104	138	172	138	172	206	56	84	110	84	110	138	110	138	166
δ1 = 0.70	62	92	124	92	124	154	124	154	184	50	76	100	76	100	124	100	124	150

Table 2.

Sample sizes for responder analysis using absolute change endpoint (PAC).

	μT + μΔT = 0.2									μT + μΔT = 0.3
	1.0			2.0			3.0			1.0			2.0			3.0
${\sigma }_C^2+{\sigma }_{{\Delta }_C}^2$	1.0	2.0	3.0	1.0	2.0	3.0	1.0	2.0	3.0	1.0	2.0	3.0	1.0	2.0	3.0	1.0	2.0	3.0
cut-off value = 0.1
δ2 = 0.25	114	122	126	122	132	136	126	136	142	90	38	100	104	110	114	110	118	122
δ2 = 0.30	86	92	94	92	98	100	94	100	104	70	96	76	80	84	86	84	88	92
δ2 = 0.35	68	72	74	72	76	78	74	78	80	56	74	60	62	66	68	66	70	72
δ2 = 0.40	54	58	58	58	60	62	58	62	64	46	60	50	50	54	54	54	56	56
δ2 = 0.45	44	46	48	46	50	50	48	50	52	90	48	40	42	44	44	44	46	46
cut-off value = 0.2
δ2 = 0.25	114	132	142	114	132	142	114	132	142	90	104	110	96	110	118	100	114	122
δ2 = 0.30	86	98	104	86	98	104	86	98	104	70	80	84	74	84	88	76	86	92
δ2 = 0.35	68	76	80	68	76	80	68	76	80	56	62	66	60	66	70	60	68	72
δ2 = 0.40	54	60	62	54	60	62	54	60	62	46	50	54	48	54	56	50	54	56
δ2 = 0.45	44	48	52	44	48	52	44	48	52	38	42	44	40	44	46	40	44	46
cut-off value = 0.3
δ2 = 0.25	112	142	158	104	132	146	102	126	140	90	110	122	90	110	122	90	110	122
δ2 = 0.30	84	104	114	80	98	106	78	94	104	70	84	92	70	84	92	70	84	92
δ2 = 0.35	66	80	86	64	74	82	62	74	80	56	66	70	56	66	70	56	66	70
δ2 = 0.40	54	62	68	52	60	64	50	58	62	46	54	56	46	54	56	46	54	56
δ2 = 0.45	44	50	54	42	48	52	42	48	50	38	44	46	38	44	46	38	44	46
cut-off value = 0.4
δ2 = 0.25	110	150	176	96	130	150	92	122	140	88	118	134	84	110	124	82	106	120
δ2 = 0.30	84	108	124	74	96	108	70	90	102	68	88	100	66	82	94	64	80	90
δ2 = 0.35	64	82	92	58	74	82	56	70	78	56	68	76	52	66	72	52	64	70
δ2 = 0.40	52	64	72	48	58	64	46	56	62	46	56	60	44	52	58	42	52	56
δ2 = 0.45	42	52	58	40	48	52	38	46	50	38	46	50	36	44	48	36	42	46

Comparing sample sizes in Table 1 and Table 2, we find that when the non-inferiority margin is fixed, the required sample size of AC is much larger than the one of PAC. One important assumption we used here is that the non-inferiority margins of two tests are the same. The reason for us to make this assumption is that many scholars have suggested to use responder analysis as a secondary analysis (Henschke et al., 2014), i.e., the sample size is computed based on the primary analysis. In other words, statistical analysis of a typical non-inferiority test and the responder analysis will be conducted using the same dataset. However, in practice, it is more likely that the non-inferiority margins in these two tests are different, since these two tests have different meanings. Therefore, while conducting responder analysis as the secondary analysis, it would be possible that we do not have enough power for this secondary analysis.

Next, we compare the statistical power of AC and PAC using Equation (6) and (9), when the sample size is fixed. In the simulation process, the sample size used to generate random samples is the minimal of all possible sample sizes, given the population mean and standard deviation. Here, the population mean of treatment and control group are 0.2 and 0, the population variance of treatment group is 2, the population variance of control group ranges from 1 to 3, and the cut-off value ranges from 0.1 to 0.8. To make the power comparable, we assume the non-inferiority margin of AC and PAC are the same. As shown in Figure 1, with a fixed sample size, the statistical power of AC is the smallest, suggesting it requires a larger sample size to achieve desired power than PAC. This finding is consistent with results in Table 1 and Table 2. Additionally, in PAC, the statistical power when using different cut-off values become closer to each other as the population variance increases. The statistical power of PAC is either slightly lower than 80% or over 80%, regardless of cut-off value. The statistical power of AC is always below 60%. Hence, if the researchers conduct a sample size calculation based on responder analysis but end up with using typical non-inferiority test, they will not be able to achieve enough statistical power.

Figure 1.

Statistical power comparison of non-inferiority test using absolute change as endpoint (AC) and corresponding responder analysis (PAC).

To illustrate the relationship among required sample sizes, we assume the non-inferiority margin of using absolute change endpoint and responder analysis are the same. The setting is the same as the one in Figure 1. In Figure 2, the ratio of PAC sample size to AC is used to represent the relationship between AC and PAC’s sample size, where N₁ denote the sample size of AC, and N₂ denote the sample size of PAC. Under the setting used here, N₂/N₁ is always smaller than 0.35, suggesting the sample size AC is much larger than the one of PAC. When the non-inferiority margin increases, the ratios with different cut-off values not only become smaller but also closer to each other. Comparing Figure 2 (A), (B) and (C), we find that the ratio of sample size decreases, and the sample size ratios with different cut-off value become closer to each other when the variance of control group increases.

Figure 2.

Sample size comparison of non-inferiority test using absolute change as endpoint (AC) and corresponding responder analysis (PAC).

Another essential parameter of interest in responder analysis is the cut-off value (threshold) to determine whether an observation is responder or not. Let the population mean of treatment group range from 0.10 to 0.30. To make the results comparable, the non-inferiority margin in AC and PAC are set as 0. The range of cut-off value is set larger than previously, which is from −3 to 3. The simulation process is randomly generated continuous samples from normal distribution at first, where the sample size is computed using AC’s sample size formula in Equation (7). Then using the cut-off value, we label each subject as either a responder or a non-responder. As shown in Figure 3, the cut-off value can indeed drive the conclusion in a different direction. In Figure 3 (A), a negative cut-off value will provide conflict results; in Figure 3 (B), a more extreme cut-off value will provide conflict results; the same findings are found in Figure 3 (C). Additionally, the influence of cut-off value on the hypothesis test result is related to the population mean and variance; however, the overall pattern is the similar. Hence, a more extreme cut-off value, i.e., a cut-off which is further away from the population mean, is more likely to lead to conflict conclusions.

Figure 3.

Non-inferiority test results comparison of typical test using absolute change as endpoint (AC) and corresponding responder analysis (PAC).

3.2. Case study

In Section 3.1, we study the impact of essential parameters on sample size requirement, statistical power, and test conclusions using simulated data. To have a clearer illustration of the impact of cut-off value on non-inferiority test results, we conduct a case study using real clinic data from an observational study about rehabilitation in lung transplant patients (Byrd et al., 2022). The primary aim of Byrd et al. (2022) is to compare the performance of individual rehabilitation to group rehabilitation in both pre-operative and post-operative participants, measured by primary outcome variable, change in 6-minute walk distance (6MWD). Detailed change in 6MWD information of pre-operative and post-operative patients are presented in Table 3.

Table 3.

Change of 6MWD of pre-operative and post-operative participants in Byrd et al. (2022).

	Pre-operative		Post-operative
Rehabilitation	Group	Individual	Group	Individual
Sample size	93	81	110	105
Mean (SD)	51.6 (81.3)	56.6 (62.9)	174 (97.6)	160 (89.4)
Median [Q1, Q3]	44.5 [6.40,102]	59.7 [25.0,93.9]	168 [106,232]	159 [104,208]

In this section, the non-inferiority test is used to study under what circumstances AC and PAC may lead to different conclusions. According to previous studies (Holland et al. 2014; Holland et al, 2017), a clinically meaningful change in 6MWD is between 25m and 33m. The cut-off value used in here ranges from 20m to 35 m to have a more comprehensive understanding about the impact of cut-off value selection on study conclusions. The non-inferiority margin of AC ranges from −0.3 to 0.3, and the non-inferiority margin of PAC ranges from 0 to 0.03. As shown in Figure 4, for pre-operative patients, some cut-off values may lead to different conclusions. For instance, in Figure 4 (A), a cut-off value larger than 27 will yield conflicting results. However, for post-operative patients, if the cut-off value is between 20 and 35, both AC and PAC will always give consistent results. Having a closer look at the data, we find that for most post-operative patients, their change in 6MWD either extremely large (larger than 35) or extremely small (smaller than 20). In other words, in this scenario, an extreme cut-off value (ranging from 20 to 35) cannot significantly impact the proportion of responders in post-operative patients. It suggests that cut-off value selection may cause responder analysis and typical non-inferiority test to provide conflicting findings only under certain circumstances.

Figure 4.

Non-inferiority test results comparison of typical test using absolute change as endpoint (AC) and corresponding responder analysis (PAC) in Rehabilitation Program in Lung Transplant Study (Byrd et al, 2022).

As we mentioned in Section 1, a responder analysis answers a different question from the typical non-inferiority test. Specifically, if we pick an extreme cut-off value, responder analysis investigates whether the test treatment could bring substantially clinical benefit to patients. For example, in Figure 4 (A), AC gives an insignificant conclusion, i.e., individual rehabilitation is inferior to group rehabilitation, whereas PAC gives significant conclusion when the cut-off value is large. It suggests that individual rehabilitation is non-inferior to group rehabilitation only for a small proportion of patients and benefits them to have great improvement. In other words, the large cut off value allows us to focus on a smaller proportion of patients who had a substantial improvement; this difference may not be detectable in typical non-inferiority tests, yielding conflicting findings.

4. Discussion

One of the most important steps of any clinical trial is to determine the primary study endpoint, which may influence the process of establishing hypotheses, selecting statistics models, calculating sample size etc. Generally speaking, there are four types of study endpoints: (i) absolute change, (ii) relative change, (iii) responder analysis using absolute change, and (iv) responder analysis using relative change. This paper focuses on the comparison of endpoint (i) and (iii) in non-inferiority test in terms of sample size requirement, statistical power and whether different endpoints may lead to different conclusions, as example to illustrate how to compare different study endpoints. The comparison process in this study can also be generalized to compare any two study endpoints mentioned above.

In the numerical study section, both simulation study and case study using data in Byrd et al (2022) are conducted. According to the simulation study, the required sample size of a non-inferiority test using absolute change endpoint (AC) is associated with the population mean and variance of treatment and control group and the non-inferiority margin. The sample size of the corresponding responder analysis (PAC) is additionally related to the cut-off value used to determine responders. Fixing all parameters, we find that PAC requires a smaller sample size compared to AC. In other words, when the sample size is the same, PAC will always have a larger statistical power than AC, as shown in Figure 1. When the desired statistical power is the same, the sample size ratio of PAC to AC is always smaller than 1, which is also related to the non-inferiority margin and cut-off value. However, the impact of these two parameters decreases as the population variance increases. As the cut-off value becomes more extreme, the likelihood of obtaining conflicting conclusions from a non-inferiority hypothesis test increases. This was seen both in the simulation study and the case study. We find that the cut-off value selection is of great importance, which may cause conflicting results when the mean and median in treatment and control groups are closer to the cut-off value.

Without loss of generalizability, similar conclusions could be found in superiority and equivalence test. The fundamental reason for typical non-inferiority/superiority/equivalence test using absolute change as endpoint and corresponding responder analysis provide conflict conclusion is the distribution for the target population. If the samples follow normal distribution, it is very likely that typical test and responder analysis give the same conclusion, when the cut-off value is close to the population mean. Otherwise, these two types of analysis would provide conflict results, especially when the cut-off value is further away from the population mean.

Due to the great importance of cut-off value selection and the possibility of obtaining conflict conclusions, we suggest determining a cut-off value using domain knowledge in combination with statistics of the collected sample, while conducting responder analysis. Though clinically important difference (MCID) is always used as the cut-off value (Jones et al., 2016), some literatures have proposed some guidance or approaches on cut-off value selection (Farrar et al., 2006; Harrell, 2017). Additionally, since the sample size requirement of AC and PAC are different, it is necessary to check whether the sample size is large enough to achieve the desired statistical power. It should be noted that, not only may the typical test and responder analysis require different sample sizes, yield different power, and result in different study conclusions, but a test using absolute instead of relative change as study endpoints are prone to the same challenges. Some studies reported that absolute and relative change endpoints may lead to conflict conclusions (Chow, 2011; Curran-Everett and Williams, 2015). In addition, these endpoints are viewed differently by by drug approval administrations. According to the non-inferiority test guidance from the US Food and Drug Administration (FDA), constancy assumption of a study has been expected to based on constancy of relative effects, not absolute effects. (FDA, 2016). However, European Medicines Agency’s (EMA) guidance on non-inferiority test used absolute difference to illustrate instructions on non-inferiority test (EMA 2005). Hence, it is possible that one drug approved by the FDA may not be approved by EMA or wise versa, since the required sample size and statistical power of using absolute change and relative change as study endpoint are different (Chow, 2011). Hence, it would be useful to further provide the confidence interval of cut-off values, where the typical non-inferiority test and responder analysis may lead to consistent conclusions, and investigate under what circumstance both absolute and relative change endpoint will provide the same non-inferiority test results.