Don’t let your analysis go to seed: on the impact of random seed on machine learning-based causal inference

Abstract

Machine learning techniques for causal effect estimation can enhance the reliability of epidemiologic analyses, reducing their dependence on correct model specifications. However, the stochastic nature of many machine learning algorithms implies that the results derived from such approaches may be influenced by the random seed that is set prior to model fitting. In this work, we highlight the substantial influence of random seeds on a popular approach for machine learning-based causal effect estimation, namely doubly robust estimators. We illustrate that varying seeds can yield divergent scientific interpretations of doubly robust estimates produced from the same dataset. We propose techniques for stabilizing results across random seeds and, through an extensive simulation study, demonstrate that these techniques effectively neutralize seed-related variability without compromising the statistical efficiency of the estimators. Based on these findings, we offer practical guidelines to minimize the influence of random seeds in real-world applications, and we encourage researchers to explore variability due to random seed when implementing any method that involves random steps.

Introduction

As dataset size and complexity increase across scientific fields, so too does the importance of methods such as machine learning that can handle complex and high-dimensional data. As opposed to many traditional parametric regression techniques, whose results tend to be relatively stable, an important aspect of many machine learning approaches is that their results may vary based on the random seed that is set prior to fitting the model. This reliance on seed is sometimes because the algorithm inherently includes a random process. For example, in the random forest algorithm, many trees are grown using covariates selected at random from the full set of covariates. Other machine learning algorithms that include randomness in the training process are stochastic gradient boosting and neural networks, among others.^1–3 Even when randomness is not an inherent part of the algorithm’s training process, random steps may be involved in other aspects of estimation e.g. hyperparameter tuning, ensemble approaches, and/or cross-fitting.^4–6 Thus, we may conclude that many popular machine-learning approaches may be vulnerable to an over-reliance of results on the random seed that is set and that it may be possible to obtain meaningfully different predictions and/or scientific conclusions based on which seed is selected.

Random seed dependence has been acknowledged within literatures pertaining to reproducibility and machine learning. Previous works have noted that sharing data and code (including initial random seed) is critical for reproducing statistical results from the same data² and there has been a recent push in the machine learning literature for more thorough reporting on variation in algorithm performance, including variation due to random seeds.^7,8 There have also been recent efforts to stabilize the machine learning methods themselves across random seeds.¹ Neural network and natural language processing literatures even suggests treating random seed as a hyperparameter to be tuned during the learning process.^9,10 In this work we consider initial random seed to be a completely arbitrary decision which ideally should have minimal impact on results.

In our research, we are specifically interested in using machine learning to quantify causal effects of a treatment or exposure on an outcome. To estimate causal effects, machine learning may be combined with a doubly robust framework for estimation such as augmented inverse probability of treatment weighting, targeted maximum likelihood estimation (TMLE), and cross-fit versions of both of these methods.^5,11–13 Although these doubly robust frameworks were traditionally combined with parametric regression techniques, combining these estimation frameworks with machine learning has become increasingly popular, presenting a potential problem for the resulting estimator(s) in terms of stability.

Within the literature specific to estimation of causal effects, the study of random seed dependence has been mostly limited to specific application areas such as variable selection and conditional average treatment effect estimation. Several solutions have been proposed to account for variation in results due to random splitting of the data, including aggregating results from multiple splits^13–15 and sensitivity analyses.¹⁶ Zivich and Breskin explore variation in average treatment effect point estimates due to random seed after aggregating a double cross-fit estimator over multiple analytical runs using a single dataset.¹² Despite this previous work, it is still common to report results from implementation with a single random seed, and clear guidelines for stabilizing inference across multiple random seeds is not available.

In this paper, we focus on doubly robust machine learning-based estimators of the average treatment effect (ATE) and consider two proposals for stabilizing inference across random seeds.¹⁷ We demonstrate that in small samples, inference based on doubly robust, machine learning-based estimators can be alarmingly dependent on the seed selected. We study the proposed techniques in an extensive simulation study to develop guidelines for applied researchers. Finally, we apply the proposed methods to the analysis of a real dataset.

METHODS

Background

We consider the case where the observed data consist of $W$, a vector of potential confounders, $A$, a binary treatment or intervention, and $Y$, a clinical outcome of interest. Causal inference often involves potential outcomes, where $Y\left(a\right)$ describes the outcome that would have been seen if given treatment $A=a$. The ATE, $E\left[Y\left(1\right)\right]-E\left[Y\left(0\right)\right]$, describes the difference in population-level average outcomes if everyone in the population receives $A=1$ versus $A=0$. We use $\psi \left(1\right)$ and $\psi \left(0\right)$ to denote the treatment-specific means $E\left[Y\left(1\right)\right]$ and $E\left[Y\left(0\right)\right]$.

Under an assumption of no unmeasured confounding and sufficient experimentation the ATE is identified as $\psi \left(1\right)-\psi \left(0\right)=E\left[E\left[Y|A=1,W\right]-E\left[Y|A=0,W\right]\right]$. ¹⁸ We focus on estimating this identifying parameter using so-called doubly robust estimators, which are a straight-forward and popular approach for integrating machine learning into causal effect estimation.

Doubly robust estimators of the ATE

We present results for three estimators: augmented inverse probability of treatment weighting (augmented IPTW), targeted maximum likelihood estimate (TMLE), and doubly robust TMLE. Each requires, as an intermediate step, the estimation of at least two regressions: the propensity score and outcome regression. The propensity score describes the conditional probability of treatment $A$ given covariates $W$. The outcome regression describes the conditional mean of the outcome $Y$ given $A$ and $W$. We use ${\overline{Q}}_n\left(a,w\right)$ to denote the outcome regression estimate of $E\left[Y|A=a,W=w\right]$ and $g_n\left(a|w\right)$ to denote the propensity score estimate of $P\left(A=a|W=w\right)$. The outcome regression and/or propensity score may be estimated via parametric regression models or regression-based machine learning procedures. In this work the outcome regression and propensity score are estimated with machine learning procedures, which introduce random steps into the estimation process.

Given outcome regression and propensity score estimates, the AIPTW estimate of $\psi \left(a\right)$ is^19,20

$${\psi }_{n,AIPTW}\left(a\right)=\frac{1}{n}{\sum }_{i=1}^n{\overline{Q}}_n\left(a,W_i\right)+\frac{1}{n}{\sum }_{i=1}^n\frac{I\left(A_i=a\right)}{g_n\left(a|W_i\right)}\left\{Y_i-{\overline{Q}}_n\left(a,W_i\right)\right\},$$ (1)

and the estimate of the ATE is ${\psi }_{n,AIPTW}\left(1\right)-{\psi }_{n,AIPTW}\left(0\right)$. For brevity, we focus in the main paper on this AIPTW estimator, with results for the two additional doubly robust estimators included in the eAppendix.^21,22

When implementing doubly robust estimators of the ATE, influence curve-based standard error estimation may be used for confidence intervals (CIs) and hypothesis tests. The estimate

$${\widehat{\sigma }}_{n,AIPTW}^2=\frac{1}{n^2}{\sum }_{i=1}^n{\left\{{\overline{Q}}_n\left(1,W_i\right)-{\overline{Q}}_n\left(0,W_i\right)-{\psi }_{n,AIPTW}\left(1\right)+{\psi }_{n,AIPTW}\left(0\right)+\frac{2A_i-1}{g_n\left(A_i|W_i\right)}\left(Y_i-{\overline{Q}}_n\left(A_i,W_i\right)\right)\right\}}^2$$ (2)

can be used to construct 95% CIs for the ATE, $\left({\psi }_{n,AIPTW}\left(1\right)-{\psi }_{n,AIPTW}\left(0\right)\pm 1.96{\widehat{\sigma }}_{n,AIPTW}\right)$, as well as to test the null hypothesis of no treatment effect. Under regularity conditions, the test statistic $\left\{{\psi }_{n,AIPTW}\left(1\right)-{\psi }_{n,AIPTW}\left(0\right)\right\}/{\widehat{\sigma }}_{n,AIPTW}$ can be compared to appropriate quantiles of a standard Normal distribution to determine p-values for hypothesis tests.

Cross-fit estimators of the ATE

One common technical assumption needed to derive the statistical properties of the augmented IPTW and related doubly robust estimators is a Donsker assumption that imposes constraints on the complexity of outcome regression and propensity score estimators and their limiting values.²³ Some machine learning algorithms, e.g., the highly adaptive LASSO,²⁴ satisfy this assumption by construction; however, for many algorithms this assumption is difficult to scrutinize in practical applications and may be expected to fail.^23,25 This motivates the use of cross-fitting, an idea first proposed by Hasminskii and Ibragimov (1979).²⁶ Cross-fitting re-emerged in the causal effects literature with the proposals of Cross-Validated TMLE²⁷ and Double/Debiased Machine Learning,¹³ with these authors noting that cross-fitting removes the Donsker assumption and opens the door to a broader class of machine learning approaches for causal effect estimation.

Cross-fitting involves randomly splitting the dataset into $V$ partitions of approximately equal size. The outcome regression and propensity score are estimated $V$ times using data from all but one of the partitions. The $V-1$ partitions used to estimate the regressions are referred to as the training set, the left-out partition as the validation set. We denote by $V_i\in \left\{1,2,\dots ,V\right\}$ a variable indicating the partition to which observation $i$ belongs and denote by ${\overline{Q}}_{n,v}$ and $g_{n,v}$ the outcome regression and propensity score estimates when the $v^{th}$ partition is left out of the training set. The cross-fit augmented IPTW estimator is

$${\psi }_{n,CAIPTW}\left(a\right)=\frac{1}{n}{\sum }_{i=1}^n\left[{\overline{Q}}_{n,V_i}\left(a,W_i\right)+\frac{I\left(A_i=a\right)}{g_{n,V_i}\left(a|W_i\right)}\left(Y_i-{\overline{Q}}_{n,V_i}\left(a,W_i\right)\right)\right]$$ (3)

with variance estimated by

$${\widehat{\sigma }}_{n,CAIPTW}^2=\frac{1}{n^2}{\sum }_{i=1}^n{\left\{{\overline{Q}}_{n,V_i}\left(1,W_i\right)-{\overline{Q}}_{n,V_i,}\left(0,W_i\right)-{\psi }_{n,CAIPTW}\left(1\right)+{\psi }_{n,CAIPTW}\left(0\right)+\frac{2A_i-1}{g_{n,V_i}\left(A_i|W_i\right)}\left(Y_i-{\overline{Q}}_{n,V_i}\left(A_i,W_i\right)\right)\right\}}^2.$$ (4)

Other versions of cross-fitting have been recommended in the literature,²⁸ which also involve randomly partitioning the data and thus are also susceptible to sensitivity to random seed.

Dependence of doubly robust estimators on random seed

When estimating the outcome regression/propensity score to construct augmented and cross-fit augmented IPTW estimators of the ATE, there may be random steps at multiple points in the estimation process. For example, in our study we use random forest, LASSO, super learning, and cross-fitting, all of which involve random steps. Random forest uses bagging and random sampling of features.³ LASSO tunes hyperparameters using an internal layer of cross-validation, a process where the dataset is randomly partitioned into training and validation folds. Super learning, an ensemble method that combines multiple learners using a weighted sum, also uses a layer of cross-validation to determine the optimal weights for each individual learner.^4,29 Lastly, when cross-fitting is used, the dataset is randomly partitioned for the purposes of separating the training and evaluation data. Due to these random steps, the final realization of the outcome regression and propensity score will depend on the random seed set prior to estimation.

The propensity score and outcome regression estimators play a critical role in all downstream inference pertaining to causal effects – they determine not only the point estimate of the causal effect but also the confidence interval (CI) and/or hypothesis test statistics. Thus, the particular realization(s) of the estimated outcome regression and propensity score may have a substantial impact on the interpretation of the results and subsequent scientific conclusions. Researchers, funders, and policy makers may be uncomfortable with the fact that conclusions could hinge on something as arbitrary as the random seed that was used in the analysis. Thus, while machine learning and cross-fitting are often viewed as means of increasing the rigor and robustness of an analysis, the practical impact may be just the opposite.

This phenomenon is laid bare in the following simulation of a setting with no treatment effect (details in eAppendix 1). We simulated 200 observations with a 4-dimensional $W$, a binary $A$, and binary $Y$. We obtained cross-fit augmented IPTW estimates and standard errors under 1000 different initial random seeds using super learning to estimate both the outcome regression and the propensity score. For each seed we constructed a nominal 95% CI resulting in 1000 different CIs for this single data set. The distribution of these CIs highlights the dramatic sensitivity of inference to the choice of random seed (Figure 1). Because there is no treatment effect, we should hope that CIs contain 0 and this was indeed the case for 85% of the intervals. However, 14.9% of the intervals were uniformly greater than 0, and one CI was uniformly less than 0. The implication is alarming. Given the same data and analysis plan, two researchers could arrive at completely opposite scientific conclusions simply due to the choice of random seed: one analyst would conclude that there is statistical evidence that the intervention is harmful, the other that it is helpful.

Figure 1.

Confidence intervals for the average treatment effect (ATE) based on 1000 analyses of a single dataset that differ only in the initial random seed. The true ATE is zero (dashed black line). Point estimates are indicated by a black dot, and confidence intervals are colored according to whether they contain the true ATE (red) or not (blue).

Proposed Solutions

To stabilize doubly robust machine learning-based estimators in this context, we propose two ways of averaging results over multiple initial random seeds.¹⁷ A sketch of the theoretical justification for the approaches and example R code is in eAppendix 2.

1. Averaging on Intermediate Regressions: calculate the average predicted value for the initial outcome regressions and propensity scores estimates from $n_{seed}$ repeated applications of the machine learning training process. Define ${\overline{Q}}_{n,k}$ and $g_{n,k}$ as the outcome regression and propensity score estimates from machine learning algorithm trained after setting the $k^{th}$ initial random seed. Define the outcome regression and propensity score estimates to be used in augmented IPTW as

$${\tilde{Q}}_n^{n_{seed}}=\frac{1}{n_{seed}}{\sum }_{k=1}^{n_{seed}}{\overline{Q}}_{n,k}and{\tilde{g}}_n^{n_{seed}}=\frac{1}{n_{seed}}{\sum }_{k=1}^{n_{seed}}{g}_{n,k},$$ (5)

respectively. Using these estimates, build an augmented IPTW estimate using (1). We denote the resulting point estimate ${\tilde{\psi }}_{n,AIPTW}^{n_{seed}}\left(a\right)$. The averaged outcome regression and propensity score estimates can also be used to estimate the standard error of the estimator.

1. Averaging on the Final Estimate: calculate the augmented IPTW estimate $n_{seed}$ times and average:

$${\overline{\psi }}_{n,AIPTW}^{n_{seed}}\left(a\right)=\frac{1}{n_{seed}}{\sum }_{k=1}^{n_{seed}}{\psi }_{n,k,AIPTW}\left(a\right),$$ (6)

where ${\psi }_{n,k,AIPTW}\left(a\right)$ is the augmented IPTW estimate of $\psi \left(a\right)$ obtained after setting the $k^{th}$ random seed. An estimate of the standard error is

$${\widehat{\sigma }}_{n,AIPTW}^2=\frac{1}{n_{seed}}{\sum }_{k=1}^{n_{seed}}{\widehat{\sigma }}_{n,k,AIPTW}^2,$$ (7)

where ${\widehat{\sigma }}_{n,k,AIPTW}^2$ is the estimate in (2) computed using outcome regression and propensity score estimates from the $k^{th}$ seed.

Proposed Solutions and Cross-fitting

In the case of cross-fitting, solution one alone would not be sufficient to stabilize inference as it would not account for variability due to the cross-fit sample splitting. The averaging procedure would need to be repeated using multiple cross-fit splits, and the resulting ATE estimates averaged over to obtain a single, stabilized point estimate, as in our second proposal. However, studying the stability of such estimators would be extremely computationally intensive and is not included in this study. Conversely, our second proposal for averaging is tractable in this setting and corresponds with recommendations in Chernozhukov et al. (2018) in terms of point estimation, although their recommendations for estimating standard error differ.¹³

SIMULATION STUDY

Simulation Study Methods

We conducted a simulation study to evaluate the performance of the proposed solutions under 72 different scenarios, defined by the features of the data generating mechanism and the statistical analysis plan adopted by a hypothetical analyst.

Properties of the Simulated Data

We considered two data generating mechanisms, a low- and a high-dimensional mechanism, and three sample sizes, $N=100,500,\text{and}1000$. For the low-dimensional data generating mechanism, we generated $A$ and $W$ using the same mechanism as in the above illustrative example, but we updated the mechanism to include a non-zero treatment effect. For $i=1,2,\dots ,N,$

$$W_{i1}~Uniform\left(0,2\right),$$

$$W_{i2},W_{i3},W_{i4}~Bernoulli\left(0.5\right),$$

$$A_i|W_i~Bernoulli\left(g\left(1|W_i\right)\right),$$

$$Y_i|A,W_i~Bernoulli\left(\overline{Q}\left(A_i,W_i\right)\right)$$

Where $g\left(1|W_i\right)=expit\left(W_{i1}+W_{i2}\times W_{i3}-2W_{i4}\right)$, $\overline{Q}\left(A_i,W_i\right)=expit\left(W_{i1}+W_{i2}\times W_{i3}+W_{i4}\times {\text{A}}_{\text{i}}-3\right)$. Under this data generating mechanism, $\psi \left(1\right)\approx 0.254$, $\psi \left(0\right)\approx 0.170$, and $\psi \left(1\right)-\psi \left(0\right)\approx 0.084$.

For the high-dimensional data-generating mechanism, we generated 20 covariates, introduced dependence between covariates, and included variables that are predictive of neither the treatment nor outcome (details in eAppendix 1).

Properties of the Statistical Analysis Plan

We defined 12 baseline Statistical Analysis Plans for estimation of the ATE based on all combinations of the following: (i) augmented IPTW, TMLE, or doubly robust TMLE estimation of the ATE based on (ii) two-fold cross-fit or non-cross-fit versions of (iii) random forest or super learner for estimation of outcome regression/propensity score (see eAppendix 3 for details of TMLE and doubly robust TMLE). All statistical analysis plans included building a 95% CI for the ATE and a two-sided test of $H_0:\psi \left(1\right)-\psi \left(0\right)=0$.

When we used super learning to estimate the outcome regression or propensity score, we used stratified cross-validation. For candidate learners for the super learner, we chose popular regression approaches with robust implementations in R, including: logistic regression with interactions, random forest³⁰ and multivariate adaptive regression splines.³¹ The number of cross-validation folds was 10 or the number of events, whichever was less. For the high-dimensional data generating mechanism case, we also included LASSO³².

When using random forest to estimate the outcome regression and propensity score, we used the default hyperparameters in the SL.ranger function of the SuperLearner package.²⁹ The SL.ranger function does not use cross-validation to select any tuning parameters and thus random seed dependence is driven by the feature bagging and bootstrap resampling used by the algorithm. It is of note that tree-based methods such as random forest are generally not recommended for non cross-fit estimation of treatment effects (either alone or within the super learner)³³ ; nevertheless, random forest serves in our simulation as a prototypical algorithm that includes random steps in the model training process.

For each baseline statistical analysis plan, defined by modeling choices described above, we implemented additional statistical analysis plans that applied our proposed averaging solutions with $n_{seed}=5$, 10, 20, 40, 60, and 80.

Simulation Process

For each of the scenarios considered, we simulated 200 datasets (using initial random seeds 1 through 200) and implemented our statistical analysis plans 150 times for each simulated dataset (using initial random seeds 1 through 150) (Figure 2). After the initial random seed was set, we estimated the outcome regression or propensity score 80 times (or 80x2 = 160 times for cross-fit estimators). Using these 80 (160) estimates of the outcome regression and propensity score we implemented our proposed strategies by averaging at the level of the intermediate regressions and/or at the level of the cross-fit augmented IPTW estimates.

Figure 2.

Diagram of the simulation study process. The process involved the analysis of each of 200 simulated datasets. For each data set, we set 150 different initial seeds. For each initial seed, we created cross-fit and non-cross-fit estimates of the average treatment effect based on differing choices of $n_{seeds}$ for both proposed averaging strategies.

By repeatedly performing the analysis on each dataset, we can study whether and to what extent estimates and inference change based on the seed that is set under each statistical analysis plan. By using different values of $n_{seed}$ for our proposed strategies, we can also study whether and how the choice of $n_{seed}$ impacts estimator performance.

Performance Measures

We used the following metrics to evaluate each estimators’ stability.

• Within-Dataset Variability of ATE Point Estimates and CI Width: We produced box-plots displaying the distribution of the ATE point estimates and centered confidence interval bounds over the 150 initial seeds to visualize variability due to random seed in both point estimates and confidence interval width, within each dataset.

• Relative Within-Dataset Variability of ATE Point Estimates and CI Widths: For each dataset we divided the estimated within-dataset variance of point estimates by the estimated between dataset variance of point estimates. The between dataset variance was estimated using point estimates from the first analysis (random seed = 1) of each dataset. We repeated these calculations with CI width instead of point estimates. These measures give an idea of whether random seed is a substantial source of variability relative to expected sampling variability.

• Maximum, Relative Within-Dataset Range of CI Bounds: To quantify how random seed influences the values contained in a CI, we calculated the range for both the upper and lower CI bounds over the 150 initial seeds and divided the larger range by the mean CI width over the 150 initial seeds. This measure quantifies variability in CI limits relative to the width of the overall interval. The motivation for scaling the range by the mean CI width is that highly variable CI limits are more problematic in settings where CIs are narrow.

• Unstable CIs: We counted the number of datasets with >10% relative range for either the upper or lower CI bounds to measure the number of datasets experiencing large shifts in CI bounds due to random seed. We also counted the number of datasets with non-overlapping CIs across the 150 initial random seeds. This metric looks for the dramatic result wherein two random seeds lead to entirely incompatible inferences.

• Unstable Hypothesis Testing: We computed the proportion of times over the 150 seeds that the null hypothesis of no treatment effect was rejected, which we call the rejection proportion. Hypothesis testing within a dataset is considered stable when this measure is zero or one, indicating that all 150 analyses of the dataset led to the same hypothesis testing conclusion (reject or fail to reject). We also counted the number of datasets with statistically significant point estimates in the wrong direction and the number of datasets with statistically significant point estimates in opposite directions which we call discordant point estimates. Discordant point estimates indicate the extreme scenario where two analyses of the same dataset both reject the null but draw opposite scientific conclusions in terms of the direction of the treatment effect.

In addition to the above stability-related metrics, we also evaluated standard statistical performance metrics for each estimator, including bias, variance, mean squared error, coverage probability of 95% Wald CIs, and power to determine whether and to what extent the proposed averaging strategies affect these performance metrics. These metrics were calculated for each estimator using the results from the first analysis performed (under initial random seed = 1) for each of the 200 datasets.

Analysis was carried out using R³⁴ v4.0.2 with packages drtmle³⁵ and SuperLearner.²⁹

Simulation Study Results

We present results from the low-dimensional data generating mechanism when super learning is used to estimate the outcome regression and propensity score. Complete results are available in eAppendix 4.

Within-Dataset Variability of ATE Estimates and CI Width

When no or minimal averaging over seed was performed, we observed substantial within-dataset variability in both point estimates and CI width in small sample sizes (Figures 3 and 4). As expected, this variability decreased with both $n$ and $n_{seed}$ (eFigures 1.1 and 1.2). Averaging over seeds was particularly important for cross-fit estimators (Figure 3 and 4, panel B), where with no averaging many datasets exhibited high within-dataset variability in both point estimates and CI widths. At the smallest sample size with $n_{seed}=1$ the median relative within-dataset variability was 0.7% and 10.5% for non cross-fit point estimates and CI bounds, respectively; 58% and 85% for cross-fit point estimates and CI bounds, respectively. For both measures relative within-dataset variability tended to decrease as sample size and the value of $n_{seed}$ increased (Figures 5 and 6).

Figure 3.

Vertical boxplots of average treatment effect (ATE) point estimates from 150 analyses of each of the 200 datasets using augmented inverse probability of treatment weighting (AIPTW) estimated (A) without and (B) with cross-fitting at different values of $n_{seed}$. Results for all datasets are displayed in the first row and 7 select datasets are displayed in the second row. Each box-plot represents point estimates from analyses that differed only in initial random seed. The height of a box-plot visualizes the within-dataset variability of ATE point estimates due to random seed. Results displayed are from the low-dimensional and small sample size data generating mechanism when super learning was used to estimate the outcome regression and propensity score, and only results from $n_{seed}=1,10,\text{and}80$ are shown for clarity. The 200 simulated datasets are ordered by the mean ATE estimate over the 150 analyses when only one seed was used in the analysis. The black dashed line indicates the true ATE value. For the second row only, y-axes differ to display more detail for select datasets. We selected the datasets with rank 25, 50, 75, 100, 125, 150, and 175, and outliers are excluded if they are below the 0.001 percentile or above the 0.999 percentile of all point estimates for the datasets displayed.

Figure 4.

Vertical boxplots of centered confidence interval (CI) bounds from 150 analyses of each of the 200 datasets using augmented inverse probability of treatment weighting (AIPTW) estimated (A) without and (B) with cross-fitting at different values of $n_{seed}$. Results for all datasets are displayed in the first row and 7 select datasets are displayed in the second row. Each box-plot represents centered CI bounds (upper or lower) from analyses that differed only in initial random seed. The height of a box-plot visualizes the within-dataset variability of centered CI limits due to random seed. Results displayed are from the low-dimensional and small sample size data generating mechanism when super learning was used to estimate the outcome regression and propensity score, and only results from $n_{seed}=1,10,\text{and}80$ are shown for clarity. The 200 simulated datasets are ordered by the mean ATE estimate over the 150 analyses when only one seed was used in the analysis. For the second row only CI upper bound results are displayed and y-axes differ to display more detail for select datasets. We selected the datasets with rank 25, 50, 75, 100, 125, 150, and 175, and outliers are excluded if they are below the 0.001 percentile or above the 0.999 percentile of all CI upper bounds for the datasets displayed.

Figure 5.

Vertical boxplots of the relative within-dataset variability of average treatment effect (ATE) point estimates from the 200 datasets when using augmented inverse probability of treatment weighting (AIPTW) estimated (A) without and (B) with cross-fitting at different values of $n_{seed}$. Results displayed are from the low-dimensional data generating mechanism when super learning was used to estimate the outcome regression and propensity score. The y-axis is on the log-10 scale.

Figure 6.

Vertical boxplots of the relative within-dataset variability of confidence interval (CI) width from the 200 datasets when using augmented inverse probability of treatment weighting (AIPTW) estimated (A) without and (B) with cross-fitting at different values of $n_{seed}$. Results displayed are from the low-dimensional data generating mechanism when super learning was used to estimate the outcome regression and propensity score. The y-axis is on the log-10 scale.

Maximum, Relative Within-Dataset Range of CI Bounds and non-overlapping CIs:

The maximum, relative within-dataset range of CI bounds was considerable for all sample sizes when not averaging over multiple seeds (Figure 7). As expected, averaging over seeds led to a decrease in the maximum relative range of CI bounds and, generally, a decrease in the number of datasets with unstable CIs (Figures 7 and 8). For cross-fit augmented IPTW in small or medium sample sizes, all datasets had unstable CIs across the values of $n_{seed}$ that we used (Figure 8). When $n_{seed}=1$, 73 (36.5%) and 5 (2.5%) datasets had non-overlapping CIs in the small and medium sample sizes with cross-fitting, but averaging over multiple seeds did eliminate non-overlapping CIs (eTable 1).

Figure 7.

Jittered scatterplots of the maximum relative range of confidence interval (CI) bounds calculated from 150 analyses of each of 200 datasets using augmented inverse probability of treatment weighting (AIPTW) estimated (A) without cross-fitting and (B) with cross-fitting at different values of $n_{seed}$. The maximum range of CI bounds is the range of lower CI bounds or the range of upper CI bounds, whichever is larger, from the 150 analyses of a given dataset. This range is divided by the average CI width from the analyses to obtain the maximum relative range. A maximum relative range greater than 1 indicates that two analyses of the same dataset yielded an upper or lower CI limit that differed by more than the average width of the CIs across all 150 analyses. A maximum relative range of 0 indicates that CIs across seeds were all identical. Generally, a low maximum relative range of CI bounds is preferred, as it indicates a more consistent confidence intervals across random seeds. Results displayed are from the low-dimensional data generating mechanism when super learning was used to estimate the outcome regression and propensity score.

Figure 8.

Line graphs displaying the relationship between $n_{seed}$ and the proportion of data sets with unstable confidence intervals (CIs) as indicated by having a maximum relative range of CI bounds > 10% for (A) non cross-fit and (B) cross-fit augmented inverse probability of treatment weighting (AIPTW) estimates, in the low-dimensional data generating scenario when super learning was used to estimate the outcome regression and propensity score.

Unstable Hypothesis Testing

Hypothesis testing instability was present in all scenarios where no averaging was performed (Figure 9). As $n_{seed}$ increased the rejection proportions tended to cluster around zero or one (Figure 9) and the number of datasets with unstable hypothesis test results also decreased (Figure 10). For example, in the smallest sample size without cross-fitting, 47 (24%) datasets had unstable test results when $n_{seed}=1$. When $n_{seed}=40$, only 8 (4%) datasets had unstable test results. Point estimates in the wrong direction and discordant point estimates were primarily an issue with cross-fit estimators, and their frequency decreased as sample size and $n_{seed}$ increased (eTable 1).

Figure 9.

Jittered scatterplots of rejection proportion $\left(p\right)$ for each of 200 data sets. The rejection proportion is the fraction of the 150 analyses of a given dataset that rejected the null hypothesis: $p=0$ or $p=1$ indicates respectively that none or all of the 150 initial seeds led to rejection of the null hypothesis; $0<p<1$ indicates that testing conclusions differ based on random seeds, with some seeds leading to rejection of the null and others not rejecting the null. Results shown for the augmented inverse probability of treatment weighting (AIPTW) estimated (A) without cross-fitting and (B) with cross-fitting at different values of $n_{seed}$. Results displayed are from the low-dimensional data generating mechanism when super learning was used to estimate the outcome regression and propensity score.

Figure 10.

Line graphs displaying the relationship between $n_{seed}$ and the proportion of data sets with unstable hypothesis testing results as indicated by a rejection proportion not equal to zero or one for (A) non cross-fit and (B) cross-fit augmented inverse probability of treatment weighting (AIPTW) estimates, in the low-dimensional data generating scenario when super learning was used to estimate the outcome regression and propensity score.

Statistical Performance of Averaged Estimators

Generally, the estimators, CIs, and hypothesis tests that averaged over multiple seeds had similar performance to those that did not. Averaging over multiple seeds altered performance of the cross-fit augmented IPTW in the smallest sample size, with an observed decrease in bias and standard deviation and increase in CI coverage (to the point of extremely conservative CI coverage) with increasing values of $n_{seed}$ (eTable 2). These trends for this estimator persisted in larger sample sizes, though differences in performance were less dramatic. As expected, confidence interval coverage was low for non cross-fit augmented IPTW and increased with cross-fitting and larger sample sizes.

Additional Results

Additional simulation results are summarized in eAppendix 4. Across scenarios, the proposed strategies improved stability.

REAL DATA ANALYSIS

We applied our proposed strategy to a prospective observational study of 94 patients with multidrug-resistant (MDR) tuberculosis (TB) in the country of Georgia to compare treatment regimens including two recently approved drugs for treating MDR-TB, Bedaquiline and Delamanid. The outcomes of the study were binary six-month sputum culture conversion and a binary end of treatment clinical treatment outcome (treatment success vs. any other outcome)³⁶. We estimated the ATE, controlling for 17 covariates in analyses. First, we explored sensitivity to random seed by repeating the analysis with $n_{seed}=1$, 150 times with a different initial random seed each time. Next, we implemented our proposed strategy of averaging at the level of the point estimates for different values of $n_{seed}$, with an arbitrary initial random seed (i.e. 789). We present results of the cross-fit augmented IPTW estimator (results from other estimators are in eAppendix 5).

As expected, results did exhibit variability due to random seed when the analysis was repeated with $n_{seed}=1$. For both outcomes of interest both negative and positive point estimates were obtained depending on the random seed. Some analyses rejected the null hypothesis while other analyses failed to reject the null hypothesis. All rejections of the null hypothesis were accompanied by a positive point estimate, in favor of Bedaquiline (Figure 11). When implementing the proposed strategy, for the final clinical outcome, the ATE point estimates appear to be converging around 0.8 and the p-value to around 0.27 as the number of seeds averaged over increases (Table 1). For the sputum culture conversion outcome, similar ATE point estimates and CI bounds were observed across the number of seeds, but when $n_{seed}=1$ the p-value was <0.05, while $n_{seed}\ge 5$ led to p-values > 0.05 (Table 1). Although the results of this analysis consistently indicate that bedaquiline is associated with better outcomes than delamanid, these results illustrate how the point estimate, level of uncertainty, and hypothesis testing conclusions may change when analysis are repeated with $n_{seed}=1$ and after implementing our proposed strategies at different values of $n_{seed}$.

Figure 11.

Average treatment effect (ATE) point estimates and 95% confidence intervals (CI) results from the real data analysis comparing Delamanid and Bedaquiline regimens for multi-drug resistant tuberculosis in terms of the two primary outcomes of interest: A) final clinical outcome and B) binary six-month sputum culture conversion. Analyses used cross-fit augmented inverse probability of treatment weighting (AIPTW) with $n_{seed}=1$ and super learning used to estimate the outcome regression and propensity score. The analysis was repeated 150 different times on the same dataset, under a different initial random seed each time. Analysis results are ordered by p-values on the x-axis (larger to smaller).

Table 1:

Cross-fit augmented inverse probability of treatment weighting (AIPTW) point and interval estimation of average treatment effects comparing the effects of Bedaquiline versus Delamanid regimens on two clinical outcomes in patients with multi-drug resistant tuberculosis. The two outcomes studied were final clinical outcome and binary six-month sputum culture conversion (SCC). Results are summarized over different averaging levels, $n_{seed}$.

${\mathit{n}}_{\mathit{seed}}$	Treatment Effect	95% Confidence Interval		Confidence Interval Width	p-value
Final Clinical Outcome
1	0.43	−0.85	1.71	2.56	0.51
5	1.23	−0.36	2.81	3.18	0.13
10	1.00	−0.45	2.46	2.91	0.18
20	0.73	−0.74	2.20	2.94	0.33
40	0.87	−0.61	2.35	2.96	0.25
60	0.80	−0.70	2.29	2.99	0.30
80	0.84	−0.63	2.31	2.95	0.26
SCC
1	0.19	0.01	0.37	0.35	0.04
5	0.18	−0.03	0.37	0.40	0.10
10	0.17	−0.02	0.36	0.38	0.08
20	0.18	−0.01	0.37	0.38	0.07
40	0.18	−0.01	0.38	0.38	0.06
60	0.18	−0.04	0.40	0.45	0.12
80	0.18	−0.04	0.40	0.43	0.11

DISCUSSION

In this study we used simulation techniques to assess the sensitivity of statistical analyses to initial random seed. Our results illustrate that inference derived from doubly robust, machine learning-based estimators of the ATE can be heavily impacted by initial random seed, especially when the sample size is small and when cross-fitting is used. Our proposed averaging strategies led to improved robustness of results to random seed. Our simulation study suggests that these strategies can allow researchers to realize the flexibility offered by machine learning for estimating causal effects while mitigating concerns pertaining to random seed dependence. Our real data analysis demonstrates how random seed dependence may manifest in practice. Although point estimates of the ATE of Bedaquiline vs. Delamanid were reasonably consistent across seeds, testing conclusions were still susceptible to variation due to random seed. Binary decision making based on p-values, without considering the statistical and scientific evidence as a whole, is generally discouraged by the statistical community.³⁷ The variation in testing conclusions demonstrated in our study may further reinforce the idea that a binary decision rule (e.g p-value < 0.05) is inappropriate in this context. These results are of particular importance for regulatory settings where hypothesis testing is often considered critical for decision making, and additional care should be taken in machine learning contexts to ensure that statistically significant results are not spurious findings.

The number of seeds needed to sufficiently stabilize results in our simulation study changed depending on the data generating mechanism and analysis scenario. We attempted to develop guidelines for the number of seeds needed to stabilize inference by identifying the value of $n_{seed}$ that led to <5% of datasets with unstable results. When augmented IPTW estimators were implemented without cross-fitting in small samples, 80 seeds stabilized both CIs and tests; at sample sizes of 500, 20 seeds was sufficient; at sample size 1000, only 10 seeds were required. When we implemented cross-fitting, although averaging over multiple seeds improved stability, we did not achieve the desired levels of stability in most scenarios for either CIs or hypothesis testing results, indicating that $n_{seed}>80$ may be needed. It is not clear the extent to which these results will generalize across data generating mechanisms, particularly for data sets with practical positivity violations.³⁸ We also expect that, in the case of binary outcomes, sensitivity to random seed will be driven by the number of observed events as opposed to sample size.

In small samples, averaging over more seeds sometimes led to inflated standard errors, over-coverage of CIs, and decreased power of tests. We hypothesize that considering many seeds increases the chance of one seed yielding extreme standard error estimates that inflate the proposed variance estimate. This issue might be alleviated by considering alternative strategies for combining estimates over multiple seeds. For example, Chernozhukov and colleagues suggest stabilizing cross-fit estimators using the median.¹³ Methods for estimating variance that incorporate both the variability due to sampling data and the variability attributable to randomness in the training process have also been suggested in the literature.^13,14 We did not incorporate this variability in our standard error estimates. Nevertheless, we tended to see conservative or nominal CI coverage for cross-fit estimators.

Zivich and Breskin (2021) implemented the proposed approach of Chernozhukov et al.¹³ of stabilizing double cross-fit estimators by aggregating results over multiple partitions using the median and implementing a formula for variance that incorporates the variability in point estimates between partitions.¹² Although variability due to random seed was not the focus of their work, they did explore the appropriate number of partitions to aggregate over by re-analyzing a single simulated dataset, and found that 100 reiterations led to low within-dataset variability of the point estimates in that particular dataset. These results are promising, but the number of seeds needed to sufficiently stabilize point estimates and associated inference will depend on the particular dataset and analysis plan and more research is warranted for both double cross-fit estimators²⁸ and the single cross-fit estimators that we studied.

There may be additional means by which we may increase the stability of results to random seed. For example, increasing the number of folds in cross-validation (for super learning and/or for cross-fitting) may increase stability. Furthermore, certain algorithms may be modified to gain additional stability, such as increasing the number of trees in random forest. Analysts should carefully consider the specific details of the selected estimation approach to minimize the impact of random seed dependence.

Major limitations of the present work are that the desired level of stability was not achieved in many of our simulation scenarios for cross-fit estimators and the proposed methods may be computationally prohibitive at high values of $n_{seed}$ for both cross-fit and non-cross-fit estimators. More research is needed exploring alternative stabilization solutions with a particular focus on both cross-fit estimators as well as computational burden. Ideally, an adaptive approach would be developed so that additional seeds are implemented only when necessary.

Although we limited the scope of this work to the stability of doubly robust estimators of the ATE, the issue of instability due to random seed may arise in a multitude of analytical scenarios, including for other parameters of interest as well as other estimation frameworks. For any procedure involving random steps, we encourage applied researchers to explore variability in their results due to random seed,¹⁶ to report transparently, and to consider practical solutions such as averaging when appropriate. We also encourage additional methodologic research to explore the impact of random seed in other settings.

CONCLUSION

As machine learning techniques continue to gain popularity, the property of robustness to initial random seed should also become a focus in the research literature. Stability of estimators should be formally studied and reported transparently in methodological research alongside standard statistical performance criteria. Our research indicates that for doubly robust ATE estimators, 20 seeds can be expected to stabilize inference across a number of domains, with more seeds required in the presence of cross-fitting and in small samples. Additional evidence is needed to solidify this rule of thumb for different causal estimands and estimators.

Abbreviations:

AIPTW

Augmented Inverse Probability of Treatment Weighting

ATE

Average Treatment Effect

CI

Confidence Interval

DRTMLE

Doubly robust Targeted Maximum Likelihood Estimation

SCC

binary six-month sputum culture conversion

TMLE

Targeted Maximum Likelihood Estimation