A permutation procedure to detect heterogeneous treatments effects in randomized clinical trials while controlling the Type-I error rate

Abstract

Background/Aims:

Secondary analyses of randomized clinical trials often seek to identify subgroups with differential treatment effects. These discoveries can help guide individual treatment decisions based on patient characteristics and identify populations for which additional treatments are needed. Traditional analyses require researchers to prespecify potential subgroups to reduce the risk of reporting spurious results. There is a need for methods that can detect such subgroups without a priori specification while allowing researchers to control the probability of falsely detecting heterogeneous subgroups when treatment effects are uniform across the study population.

Methods:

We propose a permutation procedure for tuning parameter selection that allows for Type-I error control when testing for heterogeneous treatment effects framed within the Virtual Twins procedure for subgroup identification. We verify that the Type-I error rate can be controlled at the nominal rate and investigate the power for detecting heterogeneous effects when present through extensive simulation studies. We apply our method to a secondary analysis of data from a randomized trial of very low nicotine content cigarettes.

Results:

In the absence of Type-I error control, the observed Type-I error rate for Virtual Twins was between 99 and 100%. In contrast, models tuned via the proposed permutation were able to control the Type-I error rate and detect heterogeneous effects when present. An application of our approach to a recently completed trial of very low nicotine content cigarettes identified several variables with potentially heterogeneous treatment effects.

Conclusions:

The proposed permutation procedure allows researchers to engage in secondary analyses of clinical trials for treatment effect heterogeneity while maintaining the Type-I error rate without pre-specifying subgroups.

Introduction

The primary objective of a randomized controlled trial (RCT) is typically to estimate and test the marginal treatment effect (i.e., the average treatment effect aggregated across the entire population). However, identifying subgroups with a differential response to treatment has long been an important scientific and secondary aim, which has grown in importance in the era of personalized medicine. This focus is motivated by the idea that the treatment effect may vary from individual to individual across the population, commonly referred to as treatment effect heterogeneity. Towards this aim, researchers are interested in identifying characteristics that can explain differences in the treatment effect and subgroups for which the treatment effect is different than the effect in the broader population. Characterizing treatment effect heterogeneity can help effectuate personalized medicine, deepen our understanding of possible treatment mechanisms, and suggest subgroups which may benefit from different or more intensive intervention. For example, in studies evaluating proposed regulations that would affect all members of a population, such as trials of very low nicotine content cigarettes under the broader umbrella of tobacco regulatory science¹ which motivate our work here, it is important to identify subgroups that may not have an ideal response to the policy so that additional targeted interventions can be developed in support of these populations².

Traditionally, subgroup analyses in an RCT must be pre-specified in the statistical analysis plan. Although pre-specification permits easier control of the family-wise error rate, the number of pre-specified subgroups is typically limited, involving only a small number of covariates, and any categorization of a continuous variable must be pre-specified as well.

Given the limitations of pre-specifying subgroups, many statistical methods for evaluating effect heterogeneity and discovering subgroups using flexible, data-adaptive methods have been proposed and studied. Existing tree-based methods include interaction trees³, honest causal trees⁴, and GUIDE⁵. Moreover there exist ensemble methods that leverage the estimates of many models such as random forests of interaction trees⁶ and STIMA⁷, which combines a multiple linear regression model with a regression tree to detect interaction effects. Bayesian approaches have also been proposed^8,9.

While identifying heterogeneous subgroups is an important scientific aim, there is a long history of finding spurious subgroups that cannot be replicated in subsequent trials^10–14. Thus, researchers are naturally concerned with the a priori probability of incorrectly detecting subgroups with differential treatment effects when there is a uniform treatment effect across the study population and whether the aforementioned probability can be controlled while maintaining sufficient power to detect heterogeneity when present. This is a particular concern for many data-adaptive approaches for which such control may be difficult to implement.

In this paper, we propose a permutation procedure for identifying treatment effect heterogeneity with Type-I error rate control. We frame our approach in the context of Virtual Twins¹⁵, which is a popular two stage approach to subgroup detection that has been widely used and discussed since its original publication^16–20. Despite the method’s wide usage, there is currently little guidance on how to select penalty parameters needed to fit such models. Many researchers are left using default software settings in their applications which are typically selected with predictive performance in mind. We address this limitation by conceptualizing Virtual Twins as a hypothesis testing procedure and showing how parameters can be tuned to accurately control the associated Type-I error rate.

The rest of the paper proceeds as follows. First, we establish notation and review Virtual Twins as originally proposed. Next, we propose a permutation procedure to assist tuning parameter selection and Type-I error control. Then, we detail several simulation studies to assess our proposed method’s performance in a variety of scenarios. Finally, we apply this method to data from an RCT of very low nicotine content cigarettes to describe patient characteristics that may impact smokers’ individual responses to the intervention.

Methods

Notation

Consider the data (Y_i, T_i, X_i), i = 1,…, n from an RCT with response Y_i, binary treatment indicator T_i, and covariates X_i = (X_1i,…,X_pi) (which may be continuous or categorical). We write the conditional mean of Y_i given T_i and X_i as

$$\text{E}\left(Y_i\mid T_i,{\mathit{X}}_i\right)=h\left({\mathit{X}}_i\right)+T_{i}g\left({\mathit{X}}_i\right)$$ (1)

We assume that T_i is associated with Y_i only through its first moment and that it does not affect the conditional variance or any higher moments of the response. However, we do not require such assumptions about the relationship between all other covariates and the response. The conditional average treatment effect for each subject is $\text{E}\left(Y_i\mid T_i=1,{\mathit{X}}_i\right)-\text{E}(Y_i\mid T_i=0,{\mathit{X}}_i)=g\left({\mathit{X}}_i\right)$, which we denote as Z_i.

Virtual Twins

Virtual Twins is a two-step approach that first estimates Z_i, typically using flexible regression techniques, and then models the estimated ${\widehat{Z}}_i$ using parsimonious and interpretable models. Although in principle an analyst could fit the main effect and interpretable treatment model simultaneously, the two-step procedure provides maximum flexibility to explain variability in the main effects in Step 1, while providing an interpretable model in Step 2.

Step 1

Step 1 consists of estimating subjects’ outcomes under both the control and treatment arms. This is accomplished by splitting the data based on the value of T_i and independently fitting flexible regression models ${\widehat{f}}_0\left({\mathit{X}}_i\right)$ and ${\widehat{f}}_1\left({\mathit{X}}_i\right)$ to estimate $\text{E}\left(Y_i\mid T_i=0,{\mathit{X}}_i\right)$ and $\text{E}\left(Y_i\mid T_i=1,{\mathit{X}}_i\right)$ respectively. Each subject’s estimated conditional average treatment effect is then given by ${\widehat{Z}}_i={\widehat{f}}_1\left(X_i\right)-{\widehat{f}}_0\left({\mathit{X}}_i\right)$. The original paper proposed using random forests²¹ to estimate these quantities but other authors²² have investigated the use of additional approaches to estimating the response surface in Step 1 including linear models fit using the lasso²³, MARS²⁴ and super learner²⁵.

Step 2

In Step 2 the analyst uses a simple and interpretable model such as a regression tree to model the estimated conditional average treatment effect ${\widehat{Z}}_i$ as a function of the covariates X_i. Variables included in this model are used to determine which covariates modify the treatment effect and identify subgroups of patients with homogeneous treatment effects. The original paper supports both regression and classification trees. Others²² have explored using the lasso and conditional inference trees as possible Step 2 methods.

Type-I error rate control

The original presentation of Virtual Twins encourages fitting the Step 2 models using a fixed list of tuning parameters. While this approach has shown acceptable performance, data-adaptive methods for parameter selection based on performance metrics may be advantageous. However, standard data-adaptive methods typically select tuning parameters to maximize predictive performance which may not be optimal in this context. First, such an approach is not guaranteed to preserve any Type-I error of detecting heterogeneous treatment effects. Second, the estimated conditional average treatment effect (i.e., ${\widehat{Z}}_i$) is a deterministic function of the features and, therefore, data-adaptive methods are likely to overfit the data. To address these limitations, we propose a permutation based framework to identify appropriate penalty parameters for a variety of Step 2 methods to maintain the Type-I error rate for concluding heterogeneity.

Framing as a hypothesis test

Controlling the Type-I error rate requires that we first frame Virtual Twins as a hypothesis test with a null hypothesis of a homogeneous treatment effect. This null hypothesis implies that each subject’s treatment effect is equal and g(X_i) = Δ for all i, so an individual’s conditional average response can be simplified to:

$$\text{E}\left(Y_i\mid T_i,{\mathit{X}}_i\right)=h\left({\mathit{X}}_i\right)+T_i\text{Δ}$$ (2)

We will reject the null hypothesis if the Step 2 model estimating g(X_i) includes any covariates (e.g., a tree with at least one split). Thus, a Type-I error corresponds to rejecting the null hypothesis when g(X_i) is constant.

Permutation procedure

We consider a class of methods for Step 2 which are fit by specifying one penalty parameter where for any fixed data set there exists a sufficiently large penalty parameter such that the fitted model is constant for all inputs. Examples of such methods include regression trees²⁶, conditional inference trees²⁷, and the lasso²³. We will henceforth refer to the smallest penalty parameter that achieves this constant model for a fixed data set as the minimal null penalty parameter. Formally, we let ${\widehat{\theta }}_N=\min \left\{\theta :\widehat{g}\left({\mathit{X}}_i,\theta \right)=d\mathrm{for\; all}i\right\}$ be the minimal null penalty parameter of a given data set where $\widehat{g}\left({\mathit{X}}_i,\theta \right)$ is the fitted Step 2 model given penalty parameter θ for a given data set.

We wish to identify the penalty parameter θ_α such that the procedure’s Type-I error rate is at most α for any fixed 0 ≤ α ≤ 1. We estimate this parameter using a permutation procedure. Permutation tests are typically used to achieve exact inference in small sample sizes by deriving the null distribution of the test statistic without model-based assumptions. Our procedure slightly departs from this framework to describe the null distribution of the minimal null penalty parameter, ${\widehat{\theta }}_N$, which is then used for parameter selection.

The procedure can be summarized as first permuting the treatment indicators to preserve the covariate main effects while eliminating potential treatment by covariate interactions, then fitting the Step 1 model to this permuted data to estimate the conditional average treatment effect under the null model, and finally fitting the Step 2 model to calculate ${\widehat{\theta }}_N$ for the permuted data.

The proposed algorithm is as follows. First calculate the estimated mean treatment effect $\widehat{\text{Δ}}$ and obtain ${\tilde{Y}}_i=Y_i-\widehat{\text{Δ}}I(T_i=1)$ to set the mean treatment effect to zero before permuting the treatment indicators. Note that testing for heterogeneity with Y_i is equivalent to doing so with ${\tilde{Y}}_i$. Then, for each j where j = 1,…,m for some large m,

1. Randomly permute the treatment indicator variables of the original data set to obtain $\left({\tilde{Y}}_i,T_i^{\ast (j)},{\mathit{X}}_i\right)$.

2. Fit the Step 1 model on the permuted data to estimate ${\widehat{Z}}_i^{\ast (j)}$ for each i.

3. Let ${\widehat{\theta }}_N^{(j)}$ be the minimal null penalty parameter for the Step 2 model fit for ${\widehat{Z}}_i^{\ast (j)}$.

Then, let ${\widehat{\theta }}_{\alpha }$ be the 1 − α percentile of ${\widehat{\theta }}_N^{(1)},\dots ,{\widehat{\theta }}_N^{(m)}$ and fit the Step 2 model for ${\widehat{Z}}_i$ estimated through Step 1 on the original data using the penalty parameter ${\widehat{\theta }}_{\alpha }$.

This permutation procedure can be reframed as a permutation test to leverage existing theory. Consider the test statistic ${\widehat{\theta }}_N$ and the accompanying hypothesis test that rejects the null hypothesis of no effect heterogeneity if ${\widehat{\theta }}_N>{\theta }_{\alpha }$ . We wish to identify the largest critical value θ_α such that $\mathrm{Pr}({\widehat{\theta }}_N>{\theta }_{\alpha }\mid H_0)\le \alpha$. This can be accomplished by taking the 1 − α percentile of the null distribution of ${\widehat{\theta }}_N$ which we generate via the permutation procedure. This permutation test is valid because after permuting we have data generated equivalent in distribution under the null hypothesis that $\text{E}\left({\tilde{Y}}_i\mid T_i,{\mathit{X}}_i\right)=h\left({\mathit{X}}_i\right)$ (recall the the main effect of treatment is 0 with ${\tilde{Y}}_i$).

Simulation studies

Simulation study design

Data were generated from the model Y_i = h(X_i) + T_ig(X_i) + ϵ_i where ${ϵ}_i\stackrel{\text{iid}}{\sim }N\left(0,{\sigma }^2\right)$ for i = 1,…, 1000. Covariate vectors X_i consisted of p covariates with p = 10, 20, 50 with continuous and binary variables in a 4:1 ratio. Continuous covariates were generated from a multivariate normal distribution with an AR(1) correlation structure, and binary covariates were simulated from independent Bernoulli distributions. Treatment indicators were randomly assigned to give a 1:1 allocation ratio with 500 patients per arm.

Each subject’s conditional average treatment effect was given via g(X_i). We considered a null scenario where g(X_i) was constant for all X_i. Under this scenario we expected our permutation procedure to falsely detect heterogeneity with probability α. We also simulated scenarios where g was a linear and nonlinear function of X_i to assess our permutation procedure’s power. The nonlinear g determined the conditional average treatment effect by partitioning the covariate space through several splits at the covariates’ true mean values along with one or two interaction terms, depending on the number of covariates. The function h(X_i) can be viewed as a patient’s expected outcome under the control. We examined scenarios where h was both a linear and nonlinear function of the covariates. The nonlinear h consisted of linear and quadratic terms, binary indicators for whether a covariate was above its true mean value, and covariate by covariate interactions.

We completed 2000 simulated trials for every combination of p, g, and h. Full simulation details are available in Table 1. The average R² when ignoring any treatment by covariate interactions was about 0.7 in each simulation to resemble our motivating data. Cohen’s f², which measures the effect size of the interaction, ranged from 0.01 to 0.03 under simulations with effect heterogeneity. Additional details are provided in the supplementary materials. Table S1 displays the average R² for each scenario both when ignoring and including treatment by covariate interactions as well as Cohen’s f². In addition, we ran a similar simulation study with increased residual variance to achieve an R² of about 0.2 in all simulated datasets.

Table 1.

Simulation study details. We carried out 2000 simulations under every combination of h(X_i), g(X_i) and number of covariates p. Patient outcomes were generated through the model Y_i = h(X_i) + T_ig(X_i) + ϵ_i where ${ϵ}_i\stackrel{\text{iid}}{\sim }N\left(0,4\right)$ and n = 1000.

	p = 10	p = 20	p = 50
Covariates
Continuous	$\begin{array}{l}{\left(X_{i1},\dots X_{i8}\right)}^T\stackrel{\text{iid}}{\sim }N(\mu ,\Sigma )\\ \mathit{\mu }\sim N\left(\mathbf{0},3{\mathit{I}}_8\right)\\ \mathbf{\text{Σ}}=\mathrm{AR}(1,\rho =0.7)\end{array}$	$\begin{array}{l}{\left(X_{i1},\dots X_{i16}\right)}^T\stackrel{\text{iid}}{\sim }N(\mu ,\Sigma )\\ \mathit{\mu }\sim N\left(\mathbf{0},3{\mathit{I}}_{16}\right)\\ \mathbf{\text{Σ}}=\mathrm{AR}(1,\rho =0.7)\end{array}$	$\begin{array}{l}{\left(X_{i1},\dots X_{i40}\right)}^T\stackrel{\text{iid}}{\sim }N(\mu ,\Sigma )\\ \mathit{\mu }\sim N\left(\mathbf{0},3{\mathit{I}}_{40}\right)\\ \mathbf{\text{Σ}}=\mathrm{AR}(1,\rho =0.7)\end{array}$
Binary	$X_{i9},X_{i10}\stackrel{\text{iid}}{\sim }\mathrm{Bernoulli}(0.7)$	$X_{i17},\dots ,X_{i20}\stackrel{\text{iid}}{\sim }\mathrm{Bernoulli}(0.7)$	$X_{i41},\dots ,X_{i50}\stackrel{\text{iid}}{\sim }\mathrm{Bernoulli}(0.7)$
Covariate Main Effects
Linear	$\begin{array}{l}{h}_1\left({\mathit{X}}_i\right)={\mathit{X}}_i\beta \\ {\beta }_j=1.25\text{for}j=1,\dots ,10\end{array}$	$\begin{array}{l}{h}_1\left({\mathit{X}}_i\right)={\mathit{X}}_i\beta \\ {\beta }_j=\{\begin{array}{ll}1& \mathrm{for}j=1,\dots ,10,17,18\\ 0& \mathrm{otherwise}\end{array}\end{array}$	$\begin{array}{l}{h}_1\left({\mathit{X}}_i\right)={\mathit{X}}_i\beta \\ {\beta }_j=\{\begin{array}{ll}1& \mathrm{for}j=1,\dots ,12,41,42,43\\ 0& \mathrm{otherwise}\end{array}\end{array}$
Nonlinear	$\begin{array}{l}{h}_2\left({\mathit{X}}_i\right)=X_{i1}+X_{i2}+X_{i9}+X_{i10}+\\ 2\sum _{j=3}^5{\left(X_{ij}-{\mu }_j\right)}^2+\\ 2\sum _{j=6}^{8}I\left(X_{ij}>{\mu }_j\right)+\\ 1/2\left(X_{i1}-{\mu }_1\right)\left(X_{i2}-{\mu }_2\right)-\\ \left(X_{i1}-{\mu }_1\right)X_{i9}\end{array}$	$\begin{array}{l}{h}_2\left({\mathit{X}}_i\right)=X_{i1}+X_{i2}+X_{i17}+X_{i18}+\\ 5/4\sum _{j=3}^6{\left(X_{ij}-{\mu }_j\right)}^2+\\ 5/4\sum _{j=7}^{10}I\left(X_{ij}>{\mu }_j\right)+\\ 1/2\left(X_{i1}-{\mu }_1\right)\left(X_{i2}-{\mu }_2\right)-\\ \left(X_{i1}-{\mu }_1\right)X_{i17}\end{array}$	$\begin{array}{l}{h}_2\left({\mathit{X}}_i\right)=X_{i1}+X_{i2}+X_{i41}+X_{i42}+X_{i43}+\\ 5/4\sum _{j=3}^7{\left(X_{ij}-{\mu }_j\right)}^2+\\ 5/4\sum _{j=8}^{12}I\left(X_{ij}>{\mu }_j\right)+\\ 1/2\left(X_{i1}-{\mu }_1\right)\left(X_{i2}-{\mu }_2\right)-\\ \left(X_{i1}-{\mu }_1\right)X_{i42}\end{array}$
Conditional Average Treatment Effect
Null	$g_0\left({\mathit{X}}_i\right)=2$	$g_0\left({\mathit{X}}_i\right)=2$	$g_0\left({\mathit{X}}_i\right)=2$
Linear	$\begin{array}{l}{g}_1\left({\mathit{X}}_i\right)=m+{\mathit{X}}_i\beta \\ {\beta }_j=\{\begin{array}{ll}1/2& \text{for}j=1,9\\ 0& \text{otherwise}\end{array}\\ m=2-\frac{1}{n}\sum _{i=1}^n{\mathit{X}}_i\beta \end{array}$	$\begin{array}{l}{g}_1\left({\mathit{X}}_i\right)=m+{\mathit{X}}_i\beta \\ {\beta }_j=\{\begin{array}{ll}1/2& \text{for}j=1,2,10,17\\ 0& \text{otherwise}\end{array}\\ m=2-\frac{1}{n}\sum _{i=1}^n{\mathit{X}}_i\beta \end{array}$	$\begin{array}{l}{g}_1\left({\mathit{X}}_i\right)=m+{\mathit{X}}_i\beta \\ {\beta }_j=\{\begin{array}{ll}1/2& \text{for}j=1,2,10,41\\ 0& \text{otherwise}\end{array}\\ m=2-\frac{1}{n}\sum _{i=1}^n{\mathit{X}}_i\beta \end{array}$
Nonlinear	$\begin{array}{l}{g}_2\left({\mathit{X}}_i\right)=m+\gamma \left({\mathit{X}}_i\right)\\ \gamma \left({\mathit{X}}_i\right)=1/2X_{i9}+I\left(X_{i1}>{\mu }_1\right)+\\ 1/8\left(X_{i1}-{\mu }_1\right)X_{i9}\\ m=2-\frac{1}{n}\sum _{i=1}^n\gamma \left({\mathit{X}}_i\right)\end{array}$	$\begin{array}{l}{g}_2\left({\mathit{X}}_i\right)=m+\gamma \left({\mathit{X}}_i\right)\\ \gamma \left({\mathit{X}}_i\right)=1/2X_{i17}+I\left(X_{i1}>{\mu }_1\right)+\\ 1/2I\left(X_{i2}>{\mu }_2\right)+1/4I\left(X_{i10}>{\mu }_{10}\right)+\\ 1/4\left(X_{i1}-{\mu }_1\right)X_{i17}\\ m=2-\frac{1}{n}\sum _{i=1}^n\gamma \left({\mathit{X}}_i\right)\end{array}$	$\begin{array}{l}{g}_2\left({\mathit{X}}_i\right)=m+\gamma \left({\mathit{X}}_i\right)\\ \gamma \left({\mathit{X}}_i\right)=1/2X_{i41}+I\left(X_{i1}>{\mu }_1\right)+\\ 1/2I\left(X_{i2}>{\mu }_2\right)+1/4I\left(X_{i10}>{\mu }_{10}\right)+\\ 1/4\left(X_{i1}-{\mu }_1\right)X_{i41+}\\ 1/8\left(X_{i2}-{\mu }_2\right)\left(X_{i10}-{\mu }_{10}\right)\\ m=2-\frac{1}{n}\sum _{i=1}^n\gamma \left({\mathit{X}}_i\right)\end{array}$

Methods Considered

We implemented our permutation procedure using various methods for fitting models for Steps 1 and 2, the performance of which have been evaluated in the context of Virtual Twins²².

In Step 1 we considered using random forests and super learner²⁵. Random forests consisted of 1000 trees. The library for the super learner models used a linear model tuned using a lasso (L1) penalty, MARS²⁴, and a random forest.

We considered the lasso²³, regression trees²⁶, and conditional inference trees²⁷ as Step 2 methods, all of which require the specification of some penalty parameter before they can be fit, which we tuned to control the Type-I error rate. The lasso is a linear regression method that penalizes the L¹-norm of the non-intercept regression coefficients to perform variable selection and regularization. Our permutation procedure tuned the penalty term’s weight. Regression trees recursively partition the covariate space to identify subgroups of the data with similar response values. A dense tree is fit and then the optimal subtree that minimizes a weighted loss function that combines the mean squared error and the number of terminal nodes is selected. The weight assigned to the number of terminal nodes is the complexity parameter which we tuned through our procedure. Like regression trees, conditional inference trees also recursively partition the covariate space to identify groups with similar responses. Unlike regression trees, which are fit using measures of information, conditional inference trees use a significance test for variable selection; the tree only will split if a significance test comparing the mean outcome between both considered subgroups has test statistic greater than some pre-specified threshold, which we tuned through our permutation procedure.

We tested all combinations of the discussed Step 1 and Step 2 models using our permutation procedure with α = 0.2 and α = 0.05 and using standard parameter selection techniques. Additional details are available in Table S2.

Summaries of performance

We evaluated the performance of our permutation procedure using the following metrics:

• Type-I error rate and power: For each simulated trial we recorded whether the Step 2 model included any covariates or not. If the model included at least one covariate, we concluded that the model detected treatment effect heterogeneity. The average of this value across many simulations corresponds to the Type-I error rate or the power, depending on the absence or presence of treatment effect heterogeneity, respectively.

• Sensitivity and specificity: Consider the partition ${\mathit{X}}_i=\left({\mathit{X}}_i^{\text{Heterogeneous}},{\mathit{X}}_i^{\text{Constant}}\right)$ such that $g\left({\mathit{X}}_i\right)=g\left({\mathit{X}}_i^{\text{Heterogeneous}}\right)$ for all i. We calculated the proportion of covariates in ${\mathit{X}}_i^{\text{Heterogeneous}}$ that were included in the Step 2 model (sensitivity) as well as the proportion of covariates in ${\mathit{X}}_i^{\text{Constant}}$ that were not included (specificity).

• Individual treatment effect mean squared error: We assessed accuracy in modeling the conditional average treatment effect by calculating the mean squared error: $\sum _{i=1}^n{\left[\widehat{g}\left({\mathit{X}}_i\right)-Z_i\right]}^2/n$ where $\widehat{g}\left({\mathit{X}}_i\right)$ is the fitted Step 2 model.

Results

Simulation Results

Table 2 summarizes the proportion of simulated trials in which at least one covariate was included in the Step 2 model across different data generating models for g(X_i), h(X_i) and number of covariates. In scenarios with a homogeneous treatment effect, this corresponds to the Type-I error rate. Implementations using standard approaches to selecting tuning parameters had Type-I error rates of nearly 100%. In contrast, when using our permutation procedure, we observed empirical Type-I error rates approximately equal to the targeted values. This metric is a model’s empirical power in scenarios with effect heterogeneity. Across all such scenarios, the highest power was obtained when using a super learner model in Step 1 and the lasso to fit the Step 2 model (regardless of whether g(X_i) was linear or nonlinear). These trends held regardless of the number of covariates but the power for a specific combination of methods tended to increase with the number of covariates (See Table S3 for results when p = 20). We observed similar trends with less power in supplemental simulations with a lower overall R² value (Table S4).

Table 2.

Proportion of simulations in which at least one covariate was included in the Step 2 model across all tested combinations of Step 1 (columns) and Step 2 (rows) methods. This corresponds to the Type I error rate for scenarios with homogeneous treatment effects and the power otherwise.

	p = 10		p = 50
	RF	SL	RF	SL
Homogeneous Treatment Effects; Linear Main Effects
LASSO(α = 0.2)	0.22*	0.21	0.19	0.18*
R.Tree(α = 0.2)	0.21	0.21	0.20	0.21
C.Tree(α = 0.2)	0.22*	0.22*	0.19	0.20
LASSO(α = 0.05)	0.05	0.06	0.05	0.05
R.Tree(α = 0.05)	0.05	0.06	0.06	0.06
C.Tree(α = 0.05)	0.05	0.06	0.05	0.04
Homogeneous Treatment Effects; Nonlinear Main Effects
LASSO(α = 0.2)	0.19	0.22	0.21	0.20
R.Tree(α = 0.2)	0.19	0.20	0.21	0.20
C.Tree(α = 0.2)	0.19	0.21	0.21	0.19
LASSO(α = 0.05)	0.06	0.05	0.06*	0.06
R.Tree(α = 0.05)	0.06	0.06	0.06	0.06
C.Tree(α = 0.05)	0.06	0.06	0.06*	0.05
Linear Treatment Effects; Linear Main Effects
LASSO(α = 0.2)	0.45	0.55	0.76	0.92
R.Tree(α = 0.2)	0.39	0.34	0.68	0.71
C.Tree(α = 0.2)	0.44	0.39	0.74	0.79
LASSO(α = 0.05)	0.20	0.30	0.48	0.77
R.Tree(α = 0.05)	0.15	0.09	0.40	0.33
C.Tree(α = 0.05)	0.19	0.15	0.48	0.47
Linear Treatment Effects; Nonlinear Main Effects
LASSO(α = 0.2)	0.32	0.43	0.66	0.83
R.Tree(α = 0.2)	0.22	0.25	0.46	0.59
C.Tree(α = 0.2)	0.31	0.41	0.64	0.80
LASSO(α = 0.05)	0.09	0.19	0.34	0.64
R.Tree(α = 0.05)	0.06	0.08	0.18	0.29
C.Tree(α = 0.05)	0.09	0.16	0.32	0.57
Nonlinear Treatment Effects; Linear Main Effects
LASSO(α = 0.2)	0.57	0.71	0.57	0.77
R.Tree(α = 0.2)	0.52	0.46	0.54	0.61
C.Tree(α = 0.2)	0.55	0.42	0.56	0.60
LASSO(α = 0.05)	0.30	0.44	0.31	0.52
R.Tree(α = 0.05)	0.28	0.16	0.28	0.29
C.Tree(α = 0.05)	0.27	0.15	0.30	0.29
Nonlinear Treatment Effects; Nonlinear Main Effects
LASSO(α = 0.2)	0.39	0.65	0.46	0.68
R.Tree(α = 0.2)	0.25	0.36	0.34	0.45
C.Tree(α = 0.2)	0.38	0.59	0.45	0.64
LASSO(α = 0.05)	0.12	0.36	0.19	0.41
R.Tree(α = 0.05)	0.06	0.11	0.11	0.18
C.Tree(α = 0.05)	0.12	0.28	0.19	0.35

^*For simulations with homogeneous treatment effects, 95% CI does not include α

Abbreviations: RF: random forest; SL: super learner; R.Tree: regression tree; C.Tree: conditional inference tree

Table S5 shows the sensitivity and specificity of each combination of Step 1 and Step 2 methods for each scenario. When the conditional average treatment effect was linear, the controlled lasso had the highest sensitivity of all controlled methods. When modeling a nonlinear conditional average treatment effect with Type-I error control the lasso tended to have the highest sensitivity of all methods for a given error rate. The sensitivity was relatively constant regardless of the number of covariates for all combinations of methods. Nearly all methods that controlled the Type-I error rate demonstrated near perfect specificity for all scenarios. When the Type-I error rate was not controlled, the specificity ranged from 0.01 to 0.96 and was lowest when the lasso was used in Step 2.

Table S6 displays the estimated mean squared error for the subject-specific conditional average treatment effect for all combinations of Step 1 and Step 2 methods. In the absence of treatment effect heterogeneity, the methods that did not attempt to control the Type-I error rate had mean squared errors substantially higher than their counterparts which controlled the error rate. When the treatment effect was heterogeneous, methods that fixed the Type-I error rate at α = 0.2 tended to have the lowest mean squared error when compared to models without Type-I error control and with α = 0.05 for all Step 1 methods. The mean squared error increased with the number of covariates except for models controlling the Type-I error when the treatment effect was homogeneous, for which the mean squared error remained constant as the number of covariates increased. Across all 18 simulation scenarios, using super learner and the lasso to fit the Step 1 and 2 models, respectively resulted in the smallest mean squared error out of all method combinations which control the Type-I error in Step 2.

Application

Smoking remains the leading cause of preventable death in the United States. Currently, researchers are considering the impact of multiple regulatory interventions to reduce the negative health effects of cigarette smoking, including reducing the nicotine content of cigarettes^1,2,28–31. Reducing the nicotine content of cigarettes would impact all smokers in the United States. While RCTs have investigated the benefit of such regulations on the U.S. smoking population, on average, it is also important to identify potential subgroups that receive less benefit from or are potentially harmed by such regulations to design additional targeted interventions to reduce smoking or minimize unintended consequences.

A recent RCT¹ evaluated the impact of nicotine reduction in a randomized, double-blind trial that assigned subjects to one of three interventions: 1) immediate reduction in nicotine content, 2) gradual reduction in nicotine content, and 3) maintenance of standard tobacco cigarettes (i.e., the control condition) following a 2:2:1 allocation ratio. Subjects were provided with cigarettes with nicotine content matching their treatment assignment for a 20-week intervention period, and the impact of nicotine reduction was evaluated by comparing the change in average number of cigarettes smoked per day from baseline to the last four weeks of the intervention.

Our application focused on comparisons between the gradual nicotine reduction group and the immediate nicotine reduction group as well as between the immediate reduction group and the control. We used the same modeling approaches in Step 1 and Step 2 as in our simulation study using 40 covariates that included demographic information and baseline smoking characteristics (Table S7 summarizes the study population along these covariates). We recorded which covariates were identified as having differential treatment effects in the Step 2 model.

Table 3 displays the covariates included in each model with Type-I error control when comparing the immediate and gradual groups and the immediate and control groups. The number of covariates included in each Step 2 model when the Type-I error rate was not controlled is presented in Table S8.

Table 3.

Variables determined to contribute to treatment effect heterogeneity for a given comparison of treatments on change in cigarettes per day. Covariates which had nonzero effects in the Step 2 Virtual Twins model fit for subjects’ estimated individual treatment effects with a given Step 1 model are marked with a ✓. Variables with no estimated effect for all presented models are omitted.

		Immediate vs. Gradual					Immediate vs. Control
Step 1	Step 2	TNE	CEMA	CPD	NNAL	Total	Age	Total
RF	LASSO(α = 0.2)	✓		✓		2		0
	R.Tree(α = 0.2)	✓				1	✓	1
	C.Tree(α = 0.2)	✓				1		0
	LASSO(α = 0.05)	✓		✓		2		0
	R.Tree(α = 0.05)	✓				1		0
	C.Tree(α = 0.05)	✓				1		0
SL	LASSO(α = 0.2)	✓	✓		✓	3		0
	R.Tree(α = 0.2)		✓			1		0
	C.Tree(α = 0.2)	✓				1		0
	LASSO(α = 0.05)	✓	✓			2		0
	R.Tree(α = 0.05)		✓			1		0
	C.Tree(α = 0.05)					0		0

Abbreviations: RF: random forest; SL: super learner; R.Tree: regression tree; C.Tree: conditional inference tree; TNE: total nicotine equivalents; CEMA: cyanoethyl mercapturic acid; CPD: cigarettes per day; NNAL: 4-(methylnitrosamino)-1-(3-pyridyl)-1-butanol

Immediate versus control

Models exploring the effect of immediate reduction compared to the control detected at most one covariate (age) when controlling the Type-I error at 20%. Traditional approaches detected as many as 27 covariates. We note that due to the study design, this model had fewer observations (n = 538) than when comparing gradual to immediate reduction (n = 723) and had less power to detect differential effects.

Immediate versus gradual

When modeling the effect of immediate versus gradual reduction using a random forest for Step 1 and a regression tree for Step 2 (as done in the original Virtual Twins paper) the covariates included are dependent on whether the model was tuned to control the Type-I error or not. When the error rate was not controlled six covariates were found to modify the the treatment effect. However, when controlling the error rate at either 20% or 5%, only total nicotine equivalents was found to modify the treatment effect. All other method combinations found at least one covariate in all but one case.

Given the combination’s superior performance in simulation studies, we report the results found when when a super learner model in Step 1 was paired with the lasso in Step 2. We observed statistically significant treatment effect heterogeneity at the 0.05 significance level with total nicotine equivalents (nmol/ml) and cyanoethyl mercapturic acid/creatinine urine (nmol/mg) identified as likely treatment effect modifiers with estimated conditional average treatment effect associated with immediate nicotine reduction of $\widehat{g}\left({\mathit{X}}_i\right)=5.78+0.187X_{i1}+0.009X_{i2}$, where X_i1 and X_i2 are centered and scaled (by the IQR) measures of total nicotine equivalents and cyanoethyl mercapturic acid, respectively. The results have important implications for tobacco regulatory science. While we observed significant treatment effect heterogeneity, the average treatment effect is 5.78 cigarettes per day. In contrast, the difference in the treatment effect associated with a difference equivalent to the interquartile range for total nicotine equivalents and cyanoethyl mercapturic acid is less than one cigarettes per day, which implies that the heterogeneity is small relative to the average treatment effect and that all smokers are likely to benefit from the intervention.

Discussion

We developed a permutation procedure that selects a tuning parameter which simultaneously regularizes the treatment effect heterogeneity and controls the Type-I error rate for detecting treatment effect heterogeneity in the Virtual Twins framework. This method both tests for heterogeneity and fits an estimated model for the conditional average treatment effect if there is heterogeneity. Our simulation results indicate that this procedure can control the Type-I error under a variety of null scenarios (e.g., with both linear and nonlinear covariate main effects, different numbers of measured covariates, etc.) and can detect treatment effect heterogeneity when it is present. Application to data from a recent RCT shows that when the Type-I error is not controlled, models tend to include far too many covariates to have face-validity or be useful to construct policy. Conversely, when controlling the error rate, our approach is able to detect covariates that are likely to modify the treatment effect based on our biological understanding of the intervention.

While many methods such as GUIDE, STIMA, and interaction trees have been proposed to detect subgroups with heterogeneous treatment effects or model the treatment effect, most if not all require the selection of some penalty parameter a priori. Proper specification of this parameter can control the Type-I error rate to a desired level; however, there is little to no guidance on how to select this parameter, and existing guidance often focuses on the model’s mean squared error and not its Type-I error rate.

Our approach differs by offering explicit guidelines on how to select this parameter and control the overall Type-I error rate. Moreover, we note that our permutation procedure could be adjusted to aid parameter selection for GUIDE, STIMA, and interaction trees to facilitate Type-I error control.

Other methods control the Type-I error rate when testing the existence of treatment by covariate interactions. Some only offer a test for treatment effect heterogeneity and do not offer a method for describing the treatment by covariate interaction^32,33. Additional work developed sophisticated permutation procedures to obscure treatment by covariate interactions while maintaining all other effects³⁴. Although our proposed permutation procedure correctly controls the Type-I error of interest, these alternative permutation strategies may be more efficient and are worth investigating within our framework. Additionally, some have leveraged these permutation procedures to propose a method with the aim of identifying the appropriate complexity parameter for regression trees for the conditional average treatment effect estimated through propensity score matching such that the Type-I error rate is maintained and the conditional average treatment effect can be modeled if detected³⁵. While our proposed method shares the goal of tuning regression trees’ complexity parameters, it is far more general and can support any Step 2 method with a single tuning parameter.

While our method showed strong performance on both simulated and real data, some drawbacks are worth noting. First, although our approach controlled the Type-I error rate regardless of the methods used in Steps 1 and 2, the power depends on how well the model was matched to the true data generative process. For example, although using the lasso for Step 2 yielded the highest power in a majority of scenarios, the power of tree-based methods tended to be closer to the (optimal) power of methods using the lasso when the true treatment effect was a nonlinear rather than linear function of the covariate space. Additionally, we note that permutation based tests are often not efficient and that there is space to develop more powerful tests that can maintain the Type-I error rate. Moreover, while our approach maintains the Type-I error rate when testing for treatment effect heterogeneity, it does not offer any probabilistic statements about the specific variables included in the model. That is, while the probability of including any covariates under the null hypothesis is controlled, the a priori probability of selecting a specific covariate when it does not contribute to effect heterogeneity is unknown. Future work could develop a hierarchical testing procedure to first assess if there are heterogeneous effects via our proposed method and then, if that null hypothesis is rejected, devise a way to individually test the candidate covariates detected in the final model for treatment interactions while controlling the family-wise error rate. Additionally, the permutation process itself imposes moderate computational costs. Because the data must be permuted before the Step 1 model is fit (which is often a dense ensemble method that requires multiple fittings such as a random forest), the model must be refit for each permutation, which can lead to nontrivial computational demands. Finally, while our proposed method is situated within the context of RCTs, future work exploring the method’s performance when applied to observational studies and extending the method to address any limitations in that context would be beneficial.

Although traditional guidelines recommend identifying potential subgroups a priori and testing for interactions to control the family-wise Type-I error rate³⁶, there is space for data-driven discovery. Our proposed approach allows researchers to control the Type-I error rate of an overall test for treatment effect heterogeneity and identify covariates and/or subgroups possibly associated with differential effects for future investigation. We note that this does not alleviate the need to account for conducting multiple hypothesis tests (e.g., also testing for the marginal treatment effect). However, our approach moves us towards a principled yet data-driven approach to discovery.