Evaluating Personalized (N-of-1) Trials in Rare Diseases: How Much Experimentation Is Enough?

Abstract

For rare diseases, conducting large, randomized trials of new treatments can be infeasible due to limited sample size, and it may answer the wrong scientific questions due to heterogeneity of treatment effects. Personalized (N-of-1) trials are multi-period crossover studies that aim to estimate individual treatment effects, thereby identifying the optimal treatments for individuals. This article examines the statistical design issues of evaluating a personalized (N-of-1) treatment program in people with amyotrophic lateral sclerosis (ALS). We propose an evaluation framework based on an analytical model for longitudinal data observed in a personalized trial. Under this framework, we address two design parameters: length of experimentation in each trial and number of trials needed. For the former, we consider patient-centric design criteria that aim to maximize the benefits of enrolled patients. Using theoretical investigation and numerical studies, we demonstrate that, from a patient’s perspective, the duration of an experimentation period should be no longer than one-third of the entire follow-up period of the trial. For the latter, we provide analytical formulae to calculate the power for testing quality improvement due to personalized trials in a randomized evaluation program and hence determine the required number of trials needed for the program. We apply our theoretical results to design an evaluation program for ALS treatments informed by pilot data and show that the length of experimentation has a small impact on power relative to other factors such as the degree of heterogeneity of treatment effects.

1. Introduction

When managing chronic diseases and conditions, patients commonly try different treatments over time before finding the right treatments. The practice of N-of-1 trials operationalizes this type of patient-centered experimentation by randomizing treatments to single patients in multiple crossover periods, often in a balanced fashion. N-of-1 trials can be used to identify the optimal personalized treatment for single patients in situations involving evidence for heterogeneity of treatment effects (HTE) or the lack of a cure (Davidson et al. 2021). As such, these trials are sometimes called single-patient trials or personalized trials. First introduced by (Hogben and Sim 1953), N-of-1 trials have recently been applied to treat rare diseases (Roustit et al. 2018), as well as common chronic conditions such as hypertension (Kronish et al. 2019)(Samuel et al. 2019). The use of personalized (N-of-1) trials in treating rare diseases is particularly appealing because demonstrating comparative effectiveness of treatments at the population level via parallel-group randomized trials is often infeasible.

In this article, we consider personalized (N-of-1) trials of treatments for people with amyotrophic lateral sclerosis (ALS). ALS is a rare neurodegenerative disease that affects motor neurons in the brain and spinal cord. Despite the fact that two modestly effective disease-modifying medications have been approved for the treatment of ALS (Edaravone [MCI-186] ALS 19 Study Group 2017), the disease has no cure, and thus, symptomatic treatments remain an important strategy to improve the quality of life in people with ALS (Mitsumoto, Brooks, and Silani 2014). In particular, muscle cramps are disabling symptoms affecting over 90% of ALS patients, with demonstrated between-patient variability and yet stable manifestation of symptoms in a patient (Caress et al. 2016). Several treatments targeting muscle cramps have been evaluated and have shown mixed results, suggesting the presence of HTE or inadequate statistical power for definitive conclusions (Baldinger, Katzberg, and Weber 2012). Furthermore, ALS itself has been considered markedly heterogeneous in its pathogeneses, disease manifestations, and disease progression (Al-Chalabi and Hardiman 2013)(van den Berg et al. 2019). These are the clinical situations in which personalized (N-of-1) trials can help patients identify the best treatments for themselves (n.d.a).

Despite renewed interest in N-of-1 trials and numerous recent applications, the literature has offered little discussion on the evaluation of the usefulness of N-of-1 trials. As N-of-1 trials typically require active physician involvement, intense monitoring, and frequent data collection compared with usual care, these additional costs and resources warrant careful evaluation of effectiveness before said trials are used in practice as regular clinical service. The primary evaluation question is “Does the practice of N-of-1 trials in clinical care improve outcomes on the standard of care?” However, presuming the quality of treatment decisions based on N-of-1 trials is higher than what standard of care would prescribe, reports of N-of-1 trials often describe only the applications and results of the trials without plans to address the evaluation question. An exception is (Kravitz et al. 2018) who compare N-of-1 intervention against the usual care for patients with musculoskeletal pain in a randomized fashion using data collected after experimentation ends and find no evidence of superior outcomes among participants undergoing N-of-1 trials. However, when planning the study, the authors had not considered the underlying model that accounts for variability and correlation in the longitudinal observations and the assumptions on the effect size, which would in turn drive the appropriate sample size of an evaluation program for N-of-1 trials. A design issue related to sample size determination is the duration of experimentation in N-of-1 trials. In this article, we propose a framework to evaluate the quality and effectiveness of N-of-1 trials and develop specific guidance to address these design issues. We will introduce the evaluation framework in Section 2 and define the basic analytical model for analyzing N-of-1 trials in Section 3. The main findings on the experimentation duration and sample size are derived and described in Section 4 and applied to the ALS treatment program in Section 5. The article ends with a discussion in Section 6. All technical details are provided in the Appendices.

2. An Evaluation Framework for Personalized (N-of-1) Trials

2.1. The Anatomy of a Personalized (N-of-1) Trial

We consider an evaluation program comparing the effectiveness of personalized (N-of-1) trials in treating muscle cramps in people with ALS relative to the institutional standard of care. Under the program, people with ALS will be randomized to receive personalized (N-of-1) trials that compare two standard drugs prescribed for muscle cramps, mexiletine and baclofen. In each trial, a patient will be given the two drugs sequentially over $T=18$ two-week treatment periods in two phases. The first phase consists of $m$ treatment periods (with $m<T$) when the two drugs are randomized in a multiple crossover fashion. This phase shall be referred to as the experimentation phase. In the remaining $T-m$ treatment periods, the patient will continue with a drug treatment selected based on data in the experimentation phase. This phase shall be referred to as the validation phase (Figure 1).

Figure 1.

Schema of an evaluation program for personalized (N-of-1) trials comparing treatment A and treatment B. Under the evaluation program, patients are randomized to either an N-of-1 trial or the standard of care.

During the treatment periods, the Columbia Muscle Cramp Scale (MCS) will be collected weekly to result in two MCS measurements for each period: one at the end of week 1 and one at the end of week 2. The MCS is a validated, composite score summarizing the frequency, severity, and clinical relevance of cramps in people with ALS (Mitsumoto et al. 2019). While the study does not include washout periods between treatments, only the measurement at the end of each two-week period will be used in the primary analysis in order to avoid carryover effects of the drugs.

Sandwiched between the two treatment phases is a feedback period where the MCS data in the experimentation phase are reviewed with the treating physician and the patient. The feedback period enables data-driven treatment decisions by providing the stakeholders with data visualization as well as numerical comparison (Davidson et al. 2021).

2.2. Standard of Care

In this article, we focus on a randomized controlled evaluation program where patients are randomized between an N-of-1 trial and standard of care (SOC). As depicted in Figure 1, a patient under SOC will be given either mexiletine or baclofen for 36 weeks, corresponding to the 18 two-week treatment periods in the N-of-1 trials, and will have the same follow-up schedule as the N-of-1 trial patients. Treatments in the ‘experimentation phase’ will be determined by the treating physicians. The ‘feedback period’ in the SOC arm may be viewed as a sham intervention and be conducted as a regular clinic visit before the patient continues into the ‘validation phase’ with the same drug in the remaining $T-m$ treatment periods. By the virtue of randomization, MCS collected in the validation phase under SOC will serve as the control data and allow for an unbiased comparison with the validation phase in the N-of-1 trial patients.

Let $p_0$ denote the probability that mexiletine will be prescribed under SOC and $p_1$ the probability baclofen will be prescribed such that $p_0+p_1=1$. The special case $p_0=1$ and $p_1=0$ corresponds to a clinical scenario where mexiletine is considered the standard treatment. Generally, the program probability parameters $p_0,p_1$ are somewhere between 0 and 1 when no clear best treatment exists. A program equipoise may be defined as when the treating physicians will give either of the drugs with equal likelihood, that is, $p_0=p_1=0.5$. These program parameters apparently affect the quality of treatment under standard of care, and hence the advantage of N-of-1 trials over standard of care. At the end of the evaluation program, these parameters can be estimated using the control data.

2.3. Design Parameters

While the study duration (or the number of treatment periods $T$) is determined based on feasibility and how long a patient can be followed in the evaluation program, an N-of-1 trial under the evaluation framework is defined by the length $m$ of the experimentation phase, and hence the length $T-m$ of the validation phase. Intuitively, the quality of the treatment decision by an N-of-1 trial improves with a larger $m$ as more data will be available during the feedback period. On the other hand, a long experimentation phase may place excessive burden on patients without benefitting them and imply a short validation phase for a given $T$. Rather than maximize accuracy, the experimentation length $m$ will respond to the question “How much experimentation is needed for an N-of-1 trial to be beneficial to an individual?”

A second design parameter is the specification of an analytical plan used to guide treatment selection during the feedback period. Principled statistical or data science methods should be employed to ensure the analysis is rigorous, while a prespecified plan entails preprogrammed algorithms that in turn facilitate quick feedback to the stakeholders.

Finally, as in conventional randomized controlled trials, the number of patients randomized in an evaluation program will need to be determined to ensure adequate statistical power for the primary evaluation question on whether N-of-1 trials improve outcomes.

To summarize, the design parameters that need to be prespecified at the planning stage of an evaluation program are the primary analysis plan used in the feedback period, the experimentation length ($m$) for each individual, and the number of individuals required. These will be discussed in next two sections.

3. An Analytical Model for N-of-1 Trials

Let $y_{it}$ be the outcome of patient $i$ in treatment period $t$ and $x_{it}\in \{-1,1\}$ be the corresponding treatment for $i=1,\dots ,n$ and $t=1,\dots ,T$. Without loss of generality, we assume a large value of the outcome $y_{it}$ is desirable. To put the notation in the context of our study, we let $y_{it}$ denote the negative value of MCS at the end of each two-week treatment period. For the treatments, baclofen is coded as $x_{it}=1$ and mexiletine as $x_{it}=-1$. In this article, we focus on balanced sequences between baclofen and mexiletine in the experimentation phase, that is, assuming

$$\sum _{t=1}^m{x}_{it}=0.$$ (3.1)

Consider the outcome model

$$y_{it}={\alpha }_i+{\beta }_i{x}_{it}+{ϵ}_{it}$$ (3.2)

where ${\beta }_i$ is the patient-specific treatment effect and the noise ${ϵ}_{it}\text{s}$ are mean zero normal with cov $\left({ϵ}_{it},{ϵ}_{is}\right)={\rho }_{st}{\sigma }^2$ and ${\rho }_{tt}=1$. To reflect heterogeneous symptoms and HTE among the patients, we postulate ${\alpha }_i\sim N\left({\mu }_A,{\sigma }_A^2\right)$ and ${\beta }_i\sim N\left({\mu }_B,{\sigma }_B^2\right)$. The mean ${\mu }_B$ indicates the average treatment effect and the variance ${\sigma }_B^2$ indicates the extent of HTE in the disease population. While ${\mu }_B=0$ represents the null scenarios where there is no average treatment effects, a large value of ${\sigma }_B^2$ indicates the needs for personalizing treatments.

Under model (3.2), the optimal treatment for patient $i$ can be expressed as $2I\left({\beta }_i>0\right)-1$, where $I\left(\cdot \right)$ is an indicator function. During the feedback period, we may present to patient $i$ an estimated treatment effect ${\widehat{\beta }}_i$ based on the experimentation phase data $\left\{\right(x_{it},y_{it}):t=1,\dots ,m\}$ along with the estimated optimal personalized treatment for the patient:

$$x_i^{\mathrm{*}}=2I\left({\widehat{\beta }}_i>0\right)-1.$$ (3.3)

Subsequently, in the event of perfect adherence to analysis result, the patient will receive the estimated optimal treatment (3.3) in the validation phase, that is, $x_{it}\equiv x_i^{\mathrm{*}}$ for $t=m+1,\dots ,T$.

Some practical notes on the choice of ${\widehat{\beta }}_i$ are in order. For the purposes of providing quick feedback, a broad range of estimators can be considered. The theoretical results derived in the following sections will hold for any estimators that are approximately normally distributed with mean ${\beta }_i$ and some finite variance ${\tau }_i^2$. A simple example is the the patient-specific least squares estimator ${\widehat{\beta }}_i^{LS}=\sum _{t=1}^m{x}_{it}{y}_{it}/m$ for patient $i$. The least squares estimator is unbiased for the patient-specific treatment effect ${\beta }_i$ regardless of the variance-covariance structure of $\left\{ϵ\right\}$ with variance

$${\tau }_i^2=\text{var}\left({\widehat{\beta }}_i^{LS}|{\alpha }_i,{\beta }_i\right)={\lambda }_i{\sigma }^2/m\text{where}{\lambda }_i=1+\sum _{s\ne t}{x}_{is}{x}_{it}{\rho }_{st}/m.$$ (3.4)

Note that the conditional variance (3.4) is free of the patient-specific parameters ${\alpha }_i$ and ${\beta }_i$. For the purposes of planning an N-of-1 trial, we will focus on the use of least squares. However, in the actual analysis, if additional information is available to inform the appropriate correlation structure of the data, likelihood-based estimation or weighted least squares accounting for such structure may improve efficiency.

4. How Much Experimentation Is Enough?

4.1. Patient-Centric Criteria and Length of Experimentation Phase

In this subsection, we discuss the choice of the experimentation length $m$ of an N-of-1 trial with respect to two different patient-centric criteria, both of which aim to maximize the benefits to patients on N-of-1 trials.

The first criterion is defined as the expected number of periods where a patient receives the optimal treatment. Mathematically, this criterion is denoted as $E\left(z_i\right)$, where $z_i$ is the number of periods in which patient $i$ receives the optimal treatment over the $T$ treatment periods.

Proposition 1.

Suppose ${\widehat{\beta }}_i\sim N\left({\beta }_i,{\tau }_i^2\right)$ under a balanced experimentation phase (3.1). Then for $0<m\le T$,

$$E\left(z_i\right)=\frac{m}{2}+\left(T-m\right)\hspace{0.17em}\text{Pr}\left(W\le \frac{|{\mu }_B+{\sigma }_{B}U|}{{\tau }_i}\right)$$

where $W,U$ are independent standard normal variables. Furthermore, if ${\mu }_B=0$, then

$$E\left(z_i\right)=\frac{m}{2}+\left(T-m\right)\hspace{0.17em}G\left({\sigma }_B/{\tau }_i\right)$$ (4.1)

where $G$ is the cumulative distribution function of $W/|U|$, which is a pivotal distribution.

The second patient-centric criterion is defined as the expected average outcome of a patient during an N-of-1 trial. This criterion is denoted as $E\left({\overline{y}}_i\right)$, where ${\overline{y}}_i=\sum _{t=1}^T{y}_{it}/T$ is the average outcome of the patient in all $T$ treatment periods.

Proposition 2.

Under the same condition as in Proposition 1, for $0<m\le T$,

$$E\left({\overline{y}}_i\right)={\mu }_A+\left(1-\frac{m}{T}\right)\left[{\mu }_B\left\{2\Phi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-1\right\}+\frac{2{\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\varphi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)\right]$$

where $\Phi$ and $\varphi$ respectively denote the standard normal distribution function and density.

We can derive a few practical principles from Proposition 1 and Proposition 2. First, conducting an N-of-1 trial with an experimentation length $m<T$ is generally beneficial for the patient compared to experimentation in all $T$ period. Specifically, we can derive from Proposition 1 that the patient will receive at least half of the time, that is, $E\left(z_i\right)\ge T/2$ for all $m$, and attain the minimum when $m=T$. Analogously from Proposition 2, the expected average outcome will be no smaller than the population average, that is, $E\left({\overline{y}}_i\right)\ge {\mu }_A$ for all $m$, and equality holds when $m=T$.

Second, we can derive from the propositions that $E\left(z_i\right)$ and $E\left({\overline{y}}_i\right)$ are increasing in ${\sigma }_B$ under the null ${\mu }_B=0$. In other words, an N-of-1 trial becomes more beneficial to the patient when there is a larger variability in the treatment effects across patients.

Third, and importantly, considering the null case where ${\mu }_B=0$ and when the least squares ${\widehat{\beta }}_i^{LS}$ is used to make inference based on the experimentation phase data is instructive. Under these conditions, we can derive from Proposition 2 that the criterion $E\left({\overline{y}}_i\right)$ is maximized at

$$m^{\mathrm{*}}=\frac{2T}{\sqrt{9+8{\xi }_{i}T}+3}$$ (4.2)

where ${\xi }_i=\frac{{\sigma }_B^2}{{\lambda }_i{\sigma }^2}$ and ${\lambda }_i$ is defined in (3.4). While Equation (4.2) gives the optimal length $m^{\mathrm{*}}$ as a function ${\sigma }_B,\sigma ,$ and ${\lambda }_i$, it provides some general guidance:

Main Result 1.

The optimal experimentation length $m^{*}$ is less than one-third of the total N-of-1 trial duration from a patient’s perspective, that is, $m^{*}\lesssim T/3$.

4.2. Sample Size

In this subsection, we discuss how much experimentation is adequate in terms of the sample size enrolled to the evaluation program. We first define the quality of an N-of-1 trial as the expected health outcome under the estimated optimal treatment $x_i^{\mathrm{*}}$ (3.3). Assuming perfect adherence to the analysis results in the feedback period, the quality of an N-of-1 trial can be defined as $E\left(y_i^{\mathrm{*}}\right)$ where $y_i^{\mathrm{*}}=\sum _{t=m+1}^T{y}_{it}/\left(T-m\right)$ and the expectation is taken with respect to the distributions of ${\alpha }_i,{\beta }_i,x_i^{\mathrm{*}}$, and $\left\{{ϵ}_{it}\right\}$. Analogously, we can define the quality of standard of care as $E\left(y_i\prime \right)$ where $y_i\prime$ is the average outcome observed in the validation phase for a patient under SOC and the expectation is taken under the assumption that the treatment in the experimentation phase continues to the validation phase. The primary objective of the evaluation program is to compare the quality of an N-of-1 trial and the quality of SOC. This can be formulated into a hypothesis testing problem with

$$H_0:\Delta :=E\left(y_i^{\mathrm{*}}\right)-E\left(y_i\prime \right)\le 0\text{versus}{H}_1:\Delta >0$$ (4.3)

where $\Delta$ measures the degree of quality improvement due to N-of-1 trials defined over the patient population. The hypotheses (4.3) can in turn be tested using the regular $Z$-statistic:

$$Z=\frac{\sqrt{n}\left({\overline{y}}^{\mathrm{*}}-\overline{y}\prime \right)}{\sqrt{v^{\mathrm{*}}+v\prime }}$$ (4.4)

where ${\overline{y}}^{\mathrm{*}}$ and $v^{\mathrm{*}}$ are respectively the sample mean and the sample variance of $\left\{y_i^{\mathrm{*}}\right\}$ in the $n$ patients randomized to an N-of-1 trial and $\overline{y}\prime$ and $v\prime$ are the sample mean and sample variance of $\{y_i\prime \}$ in the $n$ patients randomized to SOC. Using standard arguments gives the power of the $Z$-test

$$\text{Pr}\left(Z>c_{\alpha }|\Delta \right)\approx \Phi \left(\frac{\sqrt{n}\Delta }{\sqrt{\text{var}\left(y_i^{\mathrm{*}}\right)+\text{var}\left(y_i\prime \right)}}-c_{\alpha }\right)$$ (4.5)

where $c_{\alpha }$ is the upper $\alpha \text{th}$ percentile of standard normal. In Appendix C, we derive the expressions for $\Delta$, var $\left(y_i^{\mathrm{*}}\right)$, and var $\left(y_i\prime \right)$ in (4.5) under the condition that ${\tau }_i^2\equiv {\tau }^2$ for all $i$. This condition is met when the N-of-1 trial patients receive the same sequence $x_{it}$ in the experimentation phase or when $\left\{{ϵ}_{it}\right\}$ has a specific variance-covariance structure. For example, under a compound symmetry, that is, ${\rho }_{st}\equiv \rho$, we can show ${\lambda }_i\equiv \lambda =1-\rho$, that is, having ${\tau }_i^2\equiv \left(1-\rho \right){\sigma }^2/m$ for all $i$. Specifically, under the assumption that the SOC treatment $x_i\prime$ for a given patient is independent of the patient-specific treatment effect ${\beta }_i$, we have

$$\Delta =2{\mu }_B\left\{\Phi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-p_1\right\}+\frac{2{\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\varphi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right),$$ (4.6)

$$\text{var}\left(y_i^{\mathrm{*}}\right)={\sigma }_A^2+{\sigma }_B^2+{\mu }_B^2-\{\Delta +{\mu }_B(2p_1-1){\}}^2+\frac{{\sigma }^2}{T-m},$$ (4.7)

and

$$\text{var}\left(y_i\prime \right)={\sigma }_A^2+{\sigma }_B^2+{\mu }_B^2-{\mu }_B^2{\left(2p_1-1\right)}^2+\frac{{\sigma }^2}{T-m}.$$ (4.8)

The above expressions account for population-level information about the treatments through the program parameter $p_1$. For example, if emerging evidence in the literature suggests slight advantage of $x_i=1$ over $x_i=-1$, we may assume the physicians in the program will select $x_i\prime =1$ with $p_1>0.5$. In Appendix C, we provide expressions analogous to (4.6)–(4.8) for the situations where the physician may prescribe treatment $x_i\prime$ with patient-specific knowledge in addition to the population-level parameter $p_1$. However, we note that using expressions (4.6)–(4.8) may adequately reflect the standard of care where treatments are chosen based on population-level information rather than patient-specific knowledge. Furthermore, under the independence assumption of $x_i\prime$ and ${\beta }_i$, the power expression depends only on model parameters $\left({\sigma }_A,{\sigma }_B,{\mu }_B,\sigma ,{\tau }_i\right)$ for which information may be available to provide preliminary estimates and the known design parameters $\left(p_1,m,n,T\right)$. Finally, under the null case ${\mu }_B=0$:

Main Result 2.

All else being equal, the power to demonstrate quality improvement due to N-of-1 trials (vs SOC) increases as heterogeneity of treatment effects ${\sigma }_B^2$ increases.

5. Numerical Illustrations: Application to ALS Patients

5.1. Optimal Length of Experimentation

We use the MCS natural history data in (Mitsumoto et al. 2019) to inform the design of the evaluation program for people with ALS. Specifically, we fitted a random effects model to the data and obtained an estimate of ${\sigma }_A=4.8$ and $\sigma =1.6$. For simplicity in illustration, we further assume that the within-subject noise is conditionally independent given the population-level parameters. Figure 2 plots the patient-centric criteria against different values of ${\mu }_B,{\sigma }_B,$ and $m$ for $T=18$. While the two criteria adopt different metrics, they are maximized when $m$ is relatively small. In Figure 2 and in all $\left({\mu }_B,{\sigma }_B\right)$ that we have considered (not shown here), the optimal values of $m$ range from 2 to 6 for both criteria. This is consistent with what Main Result 1 implies: $m^{\mathrm{*}}\lesssim T/3=6$.

Figure 2.

Patient-centric criteria vs experimentation length $m$ under different values of ${\mu }_B$ and ${\sigma }_B$. Left: Expected number of optimal treatment periods vs $m$. Right: Expected average outcome (negative of MCS) of patient vs $m$.

5.2. Sample Size and Effect Size

Main Result 2 implies that ${\sigma }_B^2$ may be viewed as an effect size in power calculation, while the power also depends on other model parameters and design parameters. As in conventional practice, the choice of an effect size should be based on a clinically meaningful difference, whereas the other model parameters (e.g., ${\sigma }_A,\sigma$, etc) may be based on pilot data if available. Figure 3 plots the power against $\left(n,m\right)$ for three different effect sizes ${\sigma }_B$ for a one-sided test at 5% significance. Under each effect size, we identify the smallest $n$ that achieves 80% power for any $m$ and obtain that the required $\left(n,m\right)$ are $\left(210,12\right),\left(60,6\right)$, and $\left(34,4\right)$ respectively for ${\sigma }_B=1.6,3.2,$ and 4.8. We note that under a small effect size ${\sigma }_B=1.6$, the required $m=12$ is greater than $T/3$. In light of Main Result 1, we may instead adopt $\left(n,m\right)=\left(210,6\right)$ in order to maximize the benefits of the N-of-1 trials to the patients. The power of this modified design is 78%, which is slightly lower than the target 80%. Generally, we observe from Figure 3 that the impact of $m$ on power is relatively small compared to that of $n$ and ${\sigma }_B$ except when the effect size is small (${\sigma }_B=1.6$).

Figure 3.

Power vs $(n,m)$ for different values of ${\sigma }_B$ with ${\mu }_B=0$, ${\sigma }_A=4.8$, $\sigma =1.6$, $\rho =0$, and $T=18$.

To determine if a specific value of ${\sigma }_B$ corresponds to a clinically meaningful effect size, relating ${\sigma }_B$ to $\Delta$ using (4.6) may be useful, as $\Delta$ lives on the same scale as the measurement outcomes. In our application, a 3- to 4-point change on the MCS will represent a clinical meaningful shift. Based on the pilot data and assumptions, the effect size ${\sigma }_B=1.6,3.2,4.8$ translate to a degree of quality improvement $\Delta =1.2,2.5,3.8$ respectively. Thus, we set the sample size for this evaluation program at $n=34$ with four treatment periods (two on mexiletine and two on baclofen) based on the results for ${\sigma }_B=4.8$. Generally, the minimally clinically important heterogeneity (MCIH ${\sigma }_{B,\mathrm{m}\mathrm{i}\mathrm{n}}$) may be determined relating to the minimally clinically important change (MCID, ${\Delta }_{\mathrm{m}\mathrm{i}\mathrm{n}}$) using (4.6).

5.3. Power for Comparing to Fully Informed SOC

The calculations in the previous subsection assume the null case ${\mu }_B=0$ under which the power (4.5) does not depend on the parameter $p_1$. Under a non-null case, the value of $p_1$ reflects how informed the practice is about the population-level treatment effect. For example, if evidence in the literature suggests ${\mu }_B>0$, an informed practice will prescribe $x_i\prime =1$ with $p_1>0.5$. Specifically, standard of care that is fully informed by the literature may correspond to $p_1=\text{Pr}\left({\beta }_i>0\right)=\Phi \left({\mu }_B/{\sigma }_B\right)$.

Table 1 shows that as the average treatment effect ${\mu }_B$ grows larger and a fully informed SOC practice prescribes $x_i\prime =1$ more often (i.e., larger $p_1$), the power to demonstrate quality improvement in (4.3) becomes smaller. On the one hand, this suggests that if there is overwhelming evidence favoring $x_i\prime =1$ over $x_i\prime =-1$ in the literature, conducting N-of-1 trials will have diminished effect provided that the standard of care is fully informed. On the other hand, even with a large average treatment effect ${\mu }_B=2.4=0.5{\sigma }_B$, quality improvement due to N-of-1 trials $\Delta >3$, which is still clinically meaningful and the power is still reasonable high (68%) in this sensitivity analysis. This suggests that evaluating N-of-1 trials is a worthwhile endeavor unless there is overwhelming evidence of a large average treatment effect.

Table 1.

Quality improvement $\Delta$ and power for comparing to a fully informed SOC (standard of care) with $p_1=\Phi ({\mu }_B/{\sigma }_B)$ with $n=34,$, $m=4,$ $T=18,$ ${\sigma }_A=4.8,$ ${\sigma }_B=4.8,\sigma =1.6$ and $\rho =0$.

${\mu }_B$	$p_1=\Phi \left({\mu }_B/{\sigma }_B\right)$	$\Delta$	Power
0	0.50	3.8	80%
1.2	0.60	3.7	77%
1.6	0.63	3.6	75%
2.4	0.69	3.3	68%
4.8	0.84	2.3	39%

The numerical results in this article, and power and the patient-centric criteria in general, can be computed using tools are available at: https://roadmap2health.io/hdsr/n1power/.

6. Discussion

N-of-1 trials have been increasingly used as a design tool to bridge practice and science in rare diseases (Müller et al. 2021)(n.d.b). However, the literature is missing concrete guidelines on N-of-1 designs as to how much experimentation is appropriate. A fundamental issue is the articulation of a framework that will facilitate the evaluation of the usefulness of N-of-1 trials. In this article, we introduce an evaluation framework and outline the basic elements in an evaluation program for N-of-1 trials—namely, an experimentation phase, a feedback period, and a validation phase. In the literature, the reporting of N-of-1 trials mostly focuses only on the results of the experimentation phase, where patients explore the different treatments sequentially under a rigorous clinical protocol such as randomization, blinding, and scheduled follow-up. The feedback period and the validation phase are the critical elements in the planning and the conduct of N-of-1 trials but are, unfortunately, often omitted in the description of the design and the analytical plan.

Specifically, the length of the validation phase, relative to that of the experimentation phase, should be given careful consideration. We have demonstrated theoretically and numerically that the optimal length of experimentation from the patient’s perspective should be no greater than one-third of the entire study duration. This implies a relatively long validation phase, suggesting the importance of reproducing the quality of the decisions due to N-of-1 trials with additional follow-up. Our theoretical results also provide guidance on how many patients are needed in order to adequately power for testing quality improvement. Importantly, the relative length of experimentation and validation has minimal impact on the power. In other words, little conflict exists between the goal of maximizing patient benefits and maximizing power.

The feedback period facilitates evidence-based treatment decisions using data measured in the experimentation phase. Summarizing the relative benefits of the treatments via a single numerical statistic is a pragmatic way to present such evidence, because the information can be objectively presented and quickly digested by stakeholders. We have developed design calculus based on the model-based least squares estimation, which is quick to compute and produces unbiased estimates of patient-specific treatment effects under a broad range of scenarios. Other more sophisticated model-based methods may be used to deal with the more complex situations. For example, when we observe high volume of outcome measures via wearable devices, we could extend model (3.2) to an autoregressive model with multiple observations per treatment period (Kronish et al. 2019). In practice, treatment decisions are likely determined based on the totality of evidence. For example, in situations where a treatment that apparently benefits a patient may have side effects, a possibly less effective treatment may be preferred if it is more easily tolerated. Considerations of multiple outcomes in the analysis during the feedback period will likely increase adherence and will warrant further empirical, domain-specific research. Overall, as the feedback period potentially changes the treatment decisions—and hence, the outcomes—in the validation phase, it can be viewed as an integral part of the intervention component. We may thus experiment in a randomized fashion different elements in the feedback period for different individuals: we may consider presenting different endpoints (e.g., muscle cramp or safety), using a single endpoint, a composite outcome, or multivariate endpoints, using different types of analyses (e.g., intent-to-treat vs per-protocol), and asking patients for their satisfaction and preference (Cheung et al. 2020).

Some considerations, assumptions, and limitations for power calculation in conventional randomized controlled trials also apply for N-of-1 trials. First, power calculation involves the inputs of a number of nuisance model parameters (e.g., ${\sigma }_A,\sigma$) as well as the effect size (${\sigma }_B^2$). While the effect size ${\sigma }_B^2$ should be determined based on clinical relevance shift, the other parameters ideally can be based on estimates from pilot data. However, in situations where robust pilot data are not available, a potential useful strategy is to leverage the concepts of adaptive designs (U.S. Food & Drug Administration 2019) whereby the model parameters are updated using interim data in the evaluation program and the updates in turn inform a reassessment of the degree of quality improvement and the sample size required.

Second, our derivations assume that patients in both arms comply with their treatments in the following sense: patients in the N-of-1 trials adhere to the estimated optimal treatments based on the experimentation phase data, and patients in the SOC continue with the same treatment as in the experimentation phase. If there is prior information about noncompliance rate, power expressions can be derived accordingly under the proposed framework. However, from the viewpoint that the feedback period is part of the N-of-1 trial intervention, it should be designed to maximize adherence by choosing the outcomes and analyses that most reflect patient preference as discussed in the previous paragraph. Third, approaches to deal with missing data should be prespecified and implemented during the feedback period. An advantage of using model-based estimation is that the model can also serve as the basis for multiple imputations. That being said, no statistical approach can replace a well-conducted trial that is characterized by good compliance to treatment and minimal missing data.

Appendices

Appendix A. Theoretical Results Concerning $\mathit{E}\left({\mathit{z}}_{\mathit{i}}\right)$

A.1. Proof of Proposition 1.

First, consider the case ${\beta }_i>0$ for patient $i$ so the optimal treatment is $x_{it}=1$. Then, the number of periods the patient is on the optimal treatment equals

$$z_i=\frac{m}{2}+\left(T-m\right)I\left(x_i^{\mathrm{*}}=1\right)=\frac{m}{2}+\left(T-m\right)I\left({\widehat{\beta }}_i>0\right).$$ (A.1)

The first term in the right-hand-side of (A.1) is the number of optimal treatment periods received in the experimental phase, and the second term is the number in the validation phase. Since ${\widehat{\beta }}_i\sim N\left({\beta }_i,{\tau }_i^2\right)$, we have

$$E\left\{I\left({\widehat{\beta }}_i>0\right)|{\alpha }_i,{\beta }_i\right\}=\text{Pr}\left({\widehat{\beta }}_i>0|{\alpha }_i,{\beta }_i\right)=\Phi \left({\beta }_i/{\tau }_i\right),$$

and therefore,

$$E\left(z_i|{\alpha }_i,{\beta }_i\right)=\frac{m}{2}+\left(T-m\right)\Phi \left({\beta }_i/{\tau }_i\right)\text{when}{\beta }_i>0.$$ (A.2)

Next, under the case ${\beta }_i<0$, we can analogously derive that

$$E\left(z_i|{\alpha }_i,{\beta }_i\right)=\frac{m}{2}+\left(T-m\right)\Phi \left(-{\beta }_i/{\tau }_i\right)\text{when}{\beta }_i<0.$$ (A.3)

Combining (A.2) and (A.3) gives

$$E\left(z_i|{\alpha }_i,{\beta }_i\right)=E\left(z_i|{\beta }_i\right)=\frac{m}{2}+\left(T-m\right)\Phi \left(|{\beta }_i|/{\tau }_i\right),$$ (A.4)

which is free of ${\alpha }_i$. Since $E\left(z_i\right)=E\left\{E\right(z_i|{\beta }_i\left)\right\}$, the expectations of both sides in (A.4) are to be taken with respect to the distribution of ${\beta }_i\sim N\left({\mu }_B,{\sigma }_B^2\right)$ to complete the proof. By change of variable, we have

$$\begin{gathered} E\hspace{0.25em}\Phi \left(|{\beta }_i|/{\tau }_i\right)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{\frac{|b|}{{\tau }_i}}\frac{1}{{\sigma }_B}\varphi \left(w\right)\varphi \left(\frac{b-{\mu }_B}{{\sigma }_B}\right)dwdb \\ ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{\frac{|{\mu }_B+{\sigma }_{B}u|}{{\tau }_i}}\varphi \left(w\right)\varphi \left(u\right)dwdu \\ =\text{Pr}\left(W\le \frac{|{\mu }_B+{\sigma }_{B}U|}{{\tau }_i}\right). \end{gathered}$$ (A.5)

The proof is completed by substituting (A.5) into (A.4).

Appendix B. Theoretical Results Concerning $\mathit{E}\left({\overline{\mathit{y}}}_{\mathit{i}}\right)$

B.1. Lemma 1

Derivations of $E\left({\overline{y}}_i\right)$ will be facilitated by first noting the following lemma:

Lemma 1.

Let $V\sim N\left({\mu }_V,{\sigma }_V^2\right)$. Then,

$$E\left\{V\Phi \left(V\right)\right\}={\mu }_V\Phi \left(\frac{{\mu }_V}{\sqrt{{\sigma }_V^2+1}}\right)+\frac{{\sigma }_V^2}{\sqrt{{\sigma }_V^2+1}}\varphi \left(\frac{{\mu }_V}{\sqrt{{\sigma }_V^2+1}}\right)$$

where $\Phi$ and $\varphi$ respectively denote the standard normal distribution function and density function.

Proof of Lemma 1:

Using definition of expectation, we derive

$$\begin{gathered} E\left\{V\Phi \right(V\left)\right\}={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{v}v\varphi \left(u\right)\frac{1}{{\sigma }_V}\varphi \left(\frac{v-{\mu }_V}{{\sigma }_V}\right)dudv \\ ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{{\mu }_V+{\sigma }_{V}w}\left({\mu }_V+{\sigma }_{V}w\right)\varphi \left(u\right)\varphi \left(w\right)dudw \\ ={\mu }_V\text{Pr}\left(U<{\mu }_V+{\sigma }_{V}W\right)+{\sigma }_V{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}w\Phi \left({\mu }_V+{\sigma }_{V}w\right)\varphi \left(w\right)dw \\ ={\mu }_V\Phi \left(\frac{{\mu }_V}{\sqrt{1+{\sigma }_V^2}}\right)+{\sigma }_V^2{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\varphi \left(w\right)\varphi \left({\mu }_V+{\sigma }_{V}w\right)dw \end{gathered}$$ (B.1)

where $U,W$ are independent standard normal variables. Thus, $U-{\sigma }_{V}W\sim N\left(0,1+{\sigma }_V^2\right)$, and the first term in (B.1) can be evaluated as

$${\mu }_V\text{Pr}\left(U<{\mu }_V+{\sigma }_{V}W\right)={\mu }_V\Phi \left(\frac{{\mu }_V}{\sqrt{1+{\sigma }_V^2}}\right).$$ (B.2)

Next, the single integral in the second term in (B.2) can be evaluated using integration by parts

$$\begin{gathered} {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}w\Phi \left({\mu }_V+{\sigma }_{V}w\right)\varphi \left(w\right)dw={\sigma }_V{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\varphi \left({\mu }_V+{\sigma }_{V}w\right)\varphi \left(w\right)dw \\ ={\sigma }_V\frac{1}{\sqrt{{\sigma }_V^2+1}}\varphi \left(\frac{{\mu }_V}{\sqrt{{\sigma }_V^2+1}}\right). \end{gathered}$$ (B.3)

Equation (B.3) can be derived by straightforward derivation. The proof of Lemma 1 is thus completed by plugging (B.2) and (B.3) into (B.1).

B.2. Proof of Proposition 2

Recall that ${\overline{y}}_i$ denotes the average outcome of patient $i$ in all $T$ treatment periods in an N-of-1 trial. Hence,

$$\begin{gathered} E\left({\overline{y}}_i\right)=\frac{1}{T}\sum _{t=1}^{T}E\left({\alpha }_i+{\beta }_i{x}_{it}+{ϵ}_{it}\right)={\mu }_A+\frac{1}{T}\sum _{t=1}^{T}E\left({\beta }_i{x}_{it}\right) \\ ={\mu }_A+\left(1-\frac{m}{T}\right)E\left({\beta }_i{x}_i^{\mathrm{*}}\right). \end{gathered}$$ (B.4)

Equation (B.4) holds as because of balanced design $\sum _{i=1}^m{x}_{it}=0$. Next, since ${\widehat{\beta }}_i\sim N\left({\beta }_i,{\tau }_i^2\right)$, we have

$$\begin{gathered} E\left({\beta }_i{x}_i^{\mathrm{*}}\right)=E\left\{{\beta }_{i}E\left(x_i^{\mathrm{*}}|{\beta }_i\right)\right\}=E\left[{\beta }_{i}E\left\{2I\left({\widehat{\beta }}_i>0\right)-1|{\beta }_i\right\}\right]=E\left\{2{\beta }_i\Phi \left(\frac{{\beta }_i}{{\tau }_i}\right)-{\beta }_i\right\} \\ =2E\left\{{\beta }_i\Phi \left(\frac{\sqrt{m}{\beta }_i}{\sigma }\right)\right\}-{\mu }_B \\ =2{\tau }_{i}E\left\{\frac{{\beta }_i}{{\tau }_i}\Phi \left(\frac{{\beta }_i}{{\tau }_i}\right)\right\}-{\mu }_B \\ ={\mu }_B\left\{2\Phi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-1\right\}+\frac{2{\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\varphi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right). \end{gathered}$$ (B.5)

Expression (B.5) is obtained by applying Lemma 1 with $V={\beta }_i/{\tau }_i$. Putting (B.5) into (B.4) gives

$$E\left({\overline{y}}_i\right)={\mu }_A+\left(1-\frac{m}{T}\right)\left[{\mu }_B\left\{2\Phi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-1\right\}+\frac{2{\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\varphi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)\right]$$ (B.6)

thus completing the proof of Proposition 2.

B.3. Derivation of optimal experimentation length ${\mathit{m}}^{\mathbf{*}}$ and Main Result 1

For least squares ${\widehat{\beta }}_i^{LS}$, the variance ${\tau }_i^2={\lambda }_i{\sigma }^2/m$, where ${\lambda }_i=1+\sum _{s\ne t}{x}_{is}{x}_{it}{\rho }_{st}/m$. Further supposing ${\mu }_B=0$ simplifies (B.6) to

$$E\left({\overline{y}}_i\right)={\mu }_A+\left(1-\frac{m}{T}\right)\frac{2{\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\lambda }_i{\sigma }^2/m}}\varphi \left(0\right).$$

Hence, maximizing $E\left({\overline{y}}_i\right)$ as a function of $m$ is equivalent to maximizing the function

$$h\left(m\right)=\left(1-\frac{m}{T}\right)\frac{1}{\sqrt{{\xi }_i+1/m}}$$

where ${\xi }_i={\sigma }_B^2/\left({\lambda }_i{\sigma }^2\right)$ is free of $m$. Using standard calculus arguments, we can show that the maximizer $m^{\mathrm{*}}$ of $h\left(m\right)$ solves the equation $2{\xi }_i{m}^{\mathrm{*}2}+3m^{\mathrm{*}}-T=0$ or equivalently,

$$m^{\mathrm{*}}=\frac{\sqrt{9+8{\xi }_{i}T}-3}{4{\xi }_i}.$$ (B.7)

The derivation of $m^{\mathrm{*}}$ is completed by multiplying $\sqrt{9+8{\xi }_i}+3$ in the numerator and the denominator of (B.7), which gives

$$m^{\mathrm{*}}=\frac{2T}{\sqrt{9+8{\xi }_{i}T}+3}.$$

Now, since ${\xi }_i\ge 0$, we have $m^{\mathrm{*}}\le 2T/\sqrt{9}+3=T/3$. As a practical note, due to discreteness in $m$, the optimal $m$ may be a result of rounding up $m^{\mathrm{*}}$. Hence a slightly less sharp inequality would be $m^{\mathrm{*}}\lesssim T/3<T/3+1$.

Appendix C. Theoretical Results Concerning Power

In this section, we derive the expressions involved in the power of the $Z$-test—namely, $\Delta$, var $\left(y_i^{\mathrm{*}}\right)$, and var $\left(y_i\prime \right)$.

Recall that $p_0$ and $p_1$ respectively denote the probabilities that the treating physicians will prescribe mexiletine $\left(x_{it}=-1\right)$ and baclofen $\left(x_{it}=1\right)$ under the treatment program. Based on model (3.2), we can express the quality of an N-of-1 trial as:

$$\begin{array}{rrrrr}E\left(y_i^{\mathrm{*}}\right)& =& \frac{1}{T-m}\sum _{t=m+1}^{T}E\left({\alpha }_i+{\beta }_i{x}_{it}+{ϵ}_{it}\right)& =\frac{1}{T-m}\sum _{t=m+1}^{T}E\left({\alpha }_i+{\beta }_i{x}_i^{\mathrm{*}}+{ϵ}_{it}\right)& ={\mu }_A+E\left({\beta }_i{x}_i^{\mathrm{*}}\right).\end{array}$$

and analogously $E\left(y_i\prime \right)={\mu }_A+E\left({\beta }_i{x}_i\prime \right)$ where $x_i\prime$ is the treatment given to patient $i$ in SOC. Hence $\Delta =E\left({\beta }_i{x}_i^{\mathrm{*}}\right)-E\left({\beta }_i{x}_i\prime \right)$. Under the independence assumption of $x_i\prime$ and ${\beta }_i$, we further obtain $E\left(y_i\prime \right)={\mu }_A+\left(2p_1-1\right){\mu }_B$, and

$$\begin{gathered} \Delta =E\left({\beta }_i{x}_i^{\mathrm{*}}\right)-{\mu }_B\left(2p_1-1\right) \\ ={\mu }_B\left\{2\Phi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-1\right\}+\frac{2{\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\varphi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-{\mu }_B\left(2p_1-1\right) \\ =2{\mu }_B\left\{\Phi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-p_1\right\}+\frac{2{\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\varphi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right) \end{gathered}$$ (C.1)

where $E\left({\beta }_i{x}_i^{\mathrm{*}}\right)$ is given in (B.5).

Next,

$$\begin{gathered} \text{var}\left(y_i^{\mathrm{*}}\right)=\text{var}\left\{{\alpha }_i+{\beta }_i{x}_i^{\mathrm{*}}+\sum _{t=m+1}^T{ϵ}_{it}/\left(T-m\right)\right\} \\ ={\sigma }_A^2+\text{var}\left({\beta }_i{x}_i^{\mathrm{*}}\right)+\frac{{\sigma }^2}{T-m}={\sigma }_A^2+{\sigma }_B^2+{\mu }_B^2-\left\{E\right({\beta }_i{x}_i^{\mathrm{*}}){\}}^2+\frac{{\sigma }^2}{T-m} \\ ={\sigma }_A^2+{\sigma }_B^2+{\mu }_B^2-\{\Delta +{\mu }_B\left(2p_1-1\right){\}}^2+\frac{{\sigma }^2}{T-m}.:={\sigma }^{\mathrm{*}2} \end{gathered}$$

The last equality is a result of (C.1). Similarly, we can show

$$\begin{gathered} \text{var}\left(y_i\prime \right)={\sigma }_A^2+{\sigma }_B^2+{\mu }_B^2-{\mu }_B^2{\left(E{x}_i\prime \right)}^2+\frac{{\sigma }^2}{T-m} \\ ={\sigma }_A^2+{\sigma }_B^2+{\mu }_B^2-{\mu }_B^2{\left(2p_1-1\right)}^2+\frac{{\sigma }^2}{T-m}. \end{gathered}$$

Finally, under the null ${\mu }_B=0$, we have

$$\frac{\Delta }{\sqrt{\text{var}\left(y_i^{\mathrm{*}}\right)+\text{var}\left(y_i\prime \right)}}=\frac{2{\sigma }_B^2\varphi \left(0\right)}{\sqrt{{\sigma }_B^2+{\tau }_i^2}\sqrt{2{\sigma }_A^2+2{\sigma }_B^2-\frac{4{\sigma }_B^4{\varphi }^2\left(0\right)}{{\sigma }_B^2+{\tau }_i^2}+2{\sigma }^2/\left(T-m\right)}}.$$

Main Result 2 is proved by dividing ${\sigma }_B^2$ on the numerator and the denominator in the above expression, as a result of which the numerator will be a constant and the denominator will be a decreasing function ${\sigma }_B^2$.

For the situations where the physicians have patient-specific knowledge to inform treatments under the SOC, we may postulate that

$$x_i\prime =\{\begin{array}{ll}2I\left({\beta }_i>0\right)-1& \text{with probability}{\theta }_C\\ 1& \text{with probability}\left(1-{\theta }_C\right)p_1\\ -1& \text{with probability}\left(1-{\theta }_C\right)\left(1-p_1\right).\end{array}$$ (C.2)

The parameter ${\theta }_C$ indicates how perfect the knowledge the physicians have about the specific best treatments for their patients, with ${\theta }_C=1$ indicating perfect knowledge and ${\theta }_C=0$ indicating no additional knowledge beyond the population-level information $p_1$. Under the SOC treatment system (C.2), we have

$$E\left({\beta }_i{x}_i\prime \right)=E\left\{{\beta }_{i}E\left(x_i\prime |{\beta }_i\right)\right\}=2{\theta }_{C}E\left\{{\beta }_{i}I({\beta }_i>0\right\}-{\theta }_C{\mu }_B+\left(1-{\theta }_C\right){\mu }_B\left(2p_1-1\right)$$ (C.3)

where

$$E\left\{{\beta }_{i}I\right({\beta }_i>0\}={\sigma }_B\varphi ({\mu }_B/{\sigma }_B)+{\mu }_B\Phi ({\mu }_B/{\sigma }_B).$$ (C.4)

Using (B.5), (C.3), and (C.4), after some algebra, we have

$$\begin{gathered} \Delta =E\left({\beta }_i{x}_i^{\mathrm{*}}\right)-E\left({\beta }_i{x}_i\prime \right) \\ =2\left(1-{\theta }_C\right){\mu }_B\left\{\Phi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-p_1\right\}+\frac{2\left(1-{\theta }_C\right){\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\varphi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)+ \\ 2{\theta }_C{\mu }_B\left\{\Phi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-\Phi \left(\frac{{\mu }_B}{{\sigma }_B}\right)\right\}+2{\theta }_C\left[\frac{{\sigma }_B^2}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\varphi \left(\frac{{\mu }_B}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}\right)-{\sigma }_B\varphi \left(\frac{{\mu }_B}{{\sigma }_B}\right)\right] \end{gathered}$$

It is instructive to consider the null case ${\mu }_B=0$, under which

$$\Delta =\frac{2{\sigma }_B^2\varphi \left(0\right)}{\sqrt{{\sigma }_B^2+{\tau }_i^2}}-2{\theta }_C{\sigma }_B\varphi \left(0\right).$$

Hence, quality improvement $\Delta$ due to N-of-1 trials diminishes as the standard of care (SOC) patient-specific knowledge ${\theta }_C$ increases. Also under this special case, $\Delta >0$ if and only if ${\theta }_C^2<{\sigma }_B^2/\left({\sigma }_B^2+{\tau }_i^2\right)$

Similarly, we can obtain $\text{var}\left(y_i^{\mathrm{*}}\right)$ and $\text{var}\left(y_i\prime \right)$ under the SOC treatment system (C.2) by plugging (B.5) and (C.3) respectively in the followings:

$$\text{var}\left(y_i^{\mathrm{*}}\right)={\sigma }_A^2+{\sigma }_B^2+{\mu }_B^2-{\left\{E\left({\beta }_i{x}_i^{\mathrm{*}}\right)\right\}}^2+\frac{{\sigma }^2}{T-m}$$

and

$$\text{var}\left(y_i\prime \right)={\sigma }_A^2+{\sigma }_B^2+{\mu }_B^2-{\left\{E\left({\beta }_i{x}_i\prime \right)\right\}}^2+\frac{{\sigma }^2}{T-m}$$

Then under the null ${\mu }_B=0$, we have

$$\frac{\Delta }{\sqrt{\text{var}\left(y_i^{\mathrm{*}}\right)+\text{var}\left(y_i\prime \right)}}=\frac{2{\sigma }_B^2\varphi \left(0\right)-2{\theta }_C{\sigma }_B\sqrt{{\sigma }_B^2+{\tau }_i^2}\varphi \left(0\right)}{\sqrt{{\sigma }_B^2+{\tau }_i^2}\sqrt{2{\sigma }_A^2+2{\sigma }_B^2-\frac{4{\sigma }_B^4{\varphi }^2\left(0\right)}{{\sigma }_B^2+{\tau }_i^2}-4{\theta }_C^2{\sigma }_B^2{\varphi }^2\left(0\right)+2{\sigma }^2/\left(T-m\right)}}.$$

By dividing ${\sigma }_B^2$ on the numerator and the denominator in the above expression, we can see that the numerator is an increasing function in ${\sigma }_B^2$ and the denominator is a decreasing function in ${\sigma }_B^2$. Hence, Main Result 2 holds under this general case.