BatMan: Mitigating Batch Effects Via Stratification for Survival Outcome Prediction

Abstract

Reproducible translation of transcriptomics data has been hampered by the ubiquitous presence of batch effects. Statistical methods for managing batch effects were initially developed in the setting of sample group comparison and later borrowed for other settings such as survival outcome prediction. The most notable such method is ComBat, which adjusts for batches by including it as a covariate alongside sample groups in a linear regression. In survival prediction, however, ComBat is used without definable groups for survival outcome and is done sequentially with survival regression for a potentially batch‐confounded outcome. To address these issues, we propose a new method called BATch MitigAtion via stratificatioN (BatMan). It adjusts batches as strata in survival regression and uses variable selection methods such as the regularized regression to handle high dimensionality. We assess the performance of BatMan in comparison with ComBat, each used either alone or in conjunction with data normalization, in a resampling-based simulation study under various levels of predictive signal strength and patterns of batch-outcome association. Our simulations show that (1) BatMan outperforms ComBat in nearly all scenarios when there are batch effects in the data and (2) their performance can be worsened by the addition of data normalization. We further evaluate them using microRNA data for ovarian cancer from the Cancer Genome Atlas and find that BatMan outforms ComBat while the addition of data normalization worsens the prediction. Our study thus shows the advantage of BatMan and raises caution about the use of data normalization in the context of developing survival prediction models. The BatMan method and the simulation tool for performance assessment are implemented in R and publicly available at LXQin/PRECISION.survival—GitHub.

A new method BatMan is developed to deal with transcriptomics batch effects in the setting of survival prediction.

INTRODUCTION

Recent decades have seen accumulating transcriptomics data produced by high-throughput technologies such as microarrays and sequencing.^1-3 Their translation into useful biomedical knowledge, however, has been complicated by the ubiquitous presence of batch effects, namely systematic data variations between data collection batches that undergo varied experimental handling conditions such as regents, equipment, and personnel.^4-6 Failure to properly manage batch effects has led to subpar and irreproducible interpretations of the data.^7-9

Statistical methods for managing batch effects have been developed in the setting of differential expression analysis for group comparison.^10-13 The predominantly popular method is ComBat, which uses an Empirical Bayes approach to pool feature-specific estimates of batch effects across features.¹³ Despite its popularity, ComBat has recently been shown to be unreliable when data batches and sample groups are confounded.^14,15 In this case, it may not only induce bias in the estimated group mean differences for individual features but also inflate correlations among molecular features (because of the deduction of estimated batch effects that are shared by features), leading to negative impact on downstream analysis.

Since its initial development, ComBat has also been used in the setting of survival prediction, an important goal in biomedical studies for disease management and trial design.^16-18 In this setting, transcriptomics data are first adjusted for batches using ComBat without sample groups and then used as covariates in survival regression.^19-21 As shown in this article, such sequential operation of batch adjustment and survival analysis is inadequate, especially when batches and survival outcome are confounded. Therefore, we propose a new method for managing batch effects in survival prediction. Our method enables concurrent batch management and survival prediction by adjusting batches as strata in Cox proportional hazards regression.²² In stratified Cox regression models, the baseline hazard is stratum-specific, and a common hazard ratio for each covariate is aggregated over strata.^23,24 It has been previously used in low-dimensional data to handle covariates that deviate from the proportional hazards assumption and to estimate the interaction between a covariate and the stratification variable.²⁵ For managing batch effects in survival analysis of high-dimensional transcriptomics data, we propose to combine batch stratification and variable selection to simultaneously mitigate batch effects and build survival predictors. We call this new method BATch MitigAtion via stratificatioN (BatMan).

To investigate the performance of BatMan in comparison with ComBat for survival prediction, we conducted a simulation study using data simulated from a pair of microRNA (miRNA) microarray data sets.^26-28 Two data sets were previously collected for the same set of tumor samples, where arrays in one data set were processed with uniform handling in a single batch and arrays in the other data set were generated with disparate handling in multiple batches.^27,28 Leveraging the paired data sets, we developed a resampling-based approach for simulating array data and a permutation-based procedure for generating outcome data under prespecified levels of association.^29,30 Here, we use this approach to generate data under various scenarios for assessing the performance of BatMan and ComBat, each either applied alone or together with data normalization.⁶ Our simulations show that (1) BatMan outperforms ComBat in nearly all scenarios when there are batch effects in the training data or test data, and (2) their performance can be worsened by the addition of data normalization.

We further evaluate the performance of BatMan and ComBat using the miRNA array data for ovarian cancer from the Cancer Genome Atlas (TCGA).³¹ Ovarian cancer is the fifth most common cause of cancer deaths in women in the United States.³² The majority of ovarian carcinomas are of high-grade serous histology and associated with poor prognosis; nearly one third of these patients will not respond to initial treatment with surgery and chemotherapy and die within one year of diagnosis.³³ Molecular predictors are thus needed to improve patient stratification and treatment selection in this challenging disease.³⁴

METHODS

BatMan

We propose to use the stratified Cox regression model in combination with variable selection methods to build a model for survival probability prediction while mitigating the negative impact of batch effects. Suppose there are $B$ batches in the data for $n$ samples. Let $T_{bi}$ and $C_{bi}$ be the time to the outcome of interest and the censoring time, respectively, for sample $i$ in batch $b$ ($i=1,\dots ,n_b,b=1,\dots ,B$). Let ${\Delta }_{bi}=I\left(T_{bi}\le C_{bi}\right)$ be the censoring indicator, where $I(\cdot )$ denotes the indicator function. Suppose $G$ molecular features are measured for each sample. Further denote the $G$-dimensional vector of true expression levels of the measured molecular features for sample $i$ in batch $b$ by $X_{bi}$ and the vector of random batch effects by $W_b=\left(W_{1b},\dots ,W_{Gb}\right)$, where $W_{gb}$ ($g=1,\dots ,G$) is the feature-specific batch effect for feature $g$ in batch $b$. Thus, we allow the random batch effects to vary across features and shared across samples in the same batch. The observed expression levels that are contaminated by batch effects can be written as $Z_{bi}=X_{bi}+W_b$. Let ${\beta }_{\mathrm{}}$ be the true $G$-dimensional regression coefficients of $X$. Let $h_{b0}\left(t\right)$ be the batch-specific baseline hazard function of the outcome event. Under the proposed batch-stratified Cox regression model, the hazard function can be written as

$$h\left(t|Z_{bi}\right)=h_{b0}\left(t\right)\exp \left(Z_{bi}^{\prime }\beta \right).$$

Suppose there are $m_b$ unique event times in batch $b$, denoted by $t_{bi}$ ($i=1,\dots ,m_b$). Denote the batch-specific risk set at time $t_{bi}$ by $R\left(t_{bi}\right)$. The partial likelihood function for the batch-stratified Cox model is

$$L_s\left(\beta \right)={\prod }_{b=1}^B{\prod }_{i=1}^{m_b}\frac{\exp \left(Z_{bi}^{\prime }\beta \right)}{{\sum }_{j\in R\left(t_{bi}\right)}\exp \left(Z_{bj}^{\prime }\beta \right)}$$

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\prod }_{b=1}^B{\prod }_{i=1}^{m_b}\frac{\exp \left(\left(X_{bi}^{\prime }+{W_b}^{\prime }\right)\beta \right)}{{\sum }_{j\in R\left(t_{bi}\right)}\exp \left(\left(X_{bj}^{\prime }+{W_b}^{\prime }\right)\beta \right)}$$

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\prod }_{b=1}^B{\prod }_{i=1}^{m_b}\frac{\exp \left(X_{bi}^{\prime }\beta \right)}{{\sum }_{j\in R\left(t_{bi}\right)}\exp \left(X_{bj}^{\prime }\beta \right)}$$

Thus, it is clear that batch effects cancel out because of the structure of the stratified partial likelihood function and the, by definition, batch-specific batch effects. In this article, we consider a batch to be either an array slide (containing multiple arrays) or an experimental batch (containing multiple array slides).

Since $G$ is typically much larger than the sample size $n$ in a transcriptomic study, a variable selection method is needed to build a parsimonious model of features that are strongly associated with the survival outcome. We implement the stratified Cox regression with two commonly used methods for variable selection: (1) univariate variable selection on the basis of P values and (2) regularized regression with a penalty term.³⁵

• 1. In the univariate method, we assess the significance of survival outcome association for each feature using a univariate stratified Cox regression model and select features with a P value less than or equal to a given cutoff. The selected features are then included in a multivariate stratified Cox model. The P value cutoff is selected via a grid search (eg, from 0 to .01 by .0005) to minimize the Akaike information criterion (AIC) in the multivariate model.³⁶

• 2. In the regularized method, the regression coefficient $\beta$ is estimated on the basis of the penalized logarithm of the stratified partial likelihood function in the following form:

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\log \left(L_s\left(\beta \right)\right)-n\lambda {\sum }_{g=1}^{G}J\left({\beta }_g\right),$$

where $\lambda$ is a tuning parameter that controls the magnitude of the penalty and ${\beta }_g$ is the $g$th component of $\beta$. In this article, we focus on the use of the Least Absolute Shrinkage and Selection Operator (lasso) penalty where $J\left({\beta }_g\right)=\left|{\beta }_g\right|$ and the adaptive lasso penalty where $J\left({\beta }_g\right)=\left|{\beta }_g\right|/|~{\beta }_g|$, with ${\stackrel{\sim }{\beta }}_g$ being a consistent estimator of the true value ${\beta }_{0g}$.³⁷ The adaptive lasso penalty is a modified version of the lasso penalty that renders it the oracle property.³⁸ They both can provide sparse estimates of $\beta$ with some components being exactly zero, thereby achieving variable selection. Cross-validation is used to select the tuning parameter $\lambda$ via grid search. The coordinate descent algorithm is modified by incorporating stratification and then used for estimating the regularized regression coefficients.³⁹ To speed up the computation, we precede fitting the regularized regression model with a variable screening step, which uses the univariate stratified Cox model to select the top [n₀/4] features with the largest univariate partial likelihood, where n₀ is the number of events in the data and [.] denotes the nearest integer. This two-step variable selection strategy has been extensively studied in high-dimensional data analysis literature and showed satisfactory performance.^40-42

ComBat

This section provides a brief overview of the ComBat method.¹³ ComBat adopts a location and scale model. Following similar notations as in BatMan, the model underlying ComBat can be written as:

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{Z}_{big}={\alpha }_{0g}+V{\alpha }_{1g}+{\gamma }_{bg}+{\delta }_{bg}{\epsilon }_{big},$$

where $Z_{big}$ is the observed expression level of feature $g$ in sample $i$ from batch $b$, ${\alpha }_{0g}$ is the baseline expression level of feature $g$, and ${\alpha }_{1g}$ is the regression coefficient vector corresponding to the design matrix $V$ that contains covariates thought to be associated with feature expression levels. Note that we do not consider any such covariates in this study. The batch-related parameters ${\gamma }_{bg}$ and ${\delta }_{bg}$ represent the additive and multiplicative batch effects of batch $b$ for feature $g$, respectively. The error term ${\epsilon }_{big}$ is assumed to follow a normal distribution with mean zero and variance ${\sigma }_g^2$.

The first step of ComBat is to calculate the standardized data $S_{big}$ that only contains information on batch effects:

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{S}_{big}=\frac{{Z_{big}-\widehat{\alpha }}_{0g}-V{\widehat{\alpha }}_{1g}}{{\widehat{\sigma }}_g},$$

where ${\widehat{\alpha }}_{0g}$, ${\widehat{\alpha }}_{1g}$, and ${\widehat{\sigma }}_g$ are estimated from the feature-wise univariate ordinary linear regression with the intercept, $V$, and the batch indicator. On the basis of a location-and-scale model, $S_{big}$ follows the $N\left({\gamma }_{bg}^{*},{\delta }_{bg}^2\right)$ distribution, where ${\gamma }_{bg}^{*}={\gamma }_{bg}/{\widehat{\sigma }}_g$. Next, ${\gamma }_{bg}^{*}$ and ${\delta }_{bg}^2$ are assumed to follow prior distributions of $N\left({\gamma }_b,{\tau }_b^2\right)$ and Inverse Gamma (${\lambda }_b$, ${\theta }_b$), respectively. Hyperparameters ${\gamma }_b,{\tau }_b^2$, ${\lambda }_b$, ${\theta }_b$ are estimated empirically from the standardized data using the method of moments. When there is evidence that the normal and inverse gamma prior distributions assumed for ${\gamma }_{bg}^{*}$ and ${\delta }_{bg}^2$, respectively, do not fit the data well, a nonparametric prior method can be used instead. Finally, the Empirical Bayes estimates of parameters, ${\widehat{\gamma }}_{bg}^{*}$ and ${\widehat{\delta }}_{bg}^{\mathrm{}}$, are used to calculate the batch-adjusted data $Z_{big}^{*}$:

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{Z}_{big}^{*}=\frac{{\widehat{\sigma }}_{0g}}{{\widehat{\delta }}_{bg}}\left(S_{big}-{\widehat{\gamma }}_{bg}^{*}\right)+{\widehat{\alpha }}_{0g}+V{\widehat{\alpha }}_{1g}.$$

The batch-adjusted data are then used for subsequent analyses such as survival prediction.

Simulation Study

Collection of the empirical data.

Details on data collection can be found in Qin et al.^27-29 Briefly, 96 high-grade serous ovarian cancer samples and 96 endometroid endometrial cancer samples were collected at Memorial Sloan Kettering Cancer Center between 2000 and 2012. Their miRNA expression levels were measured twice using Agilent microarrays (Release 16.0; Agilent Technologies, Santa Clara, CA), one with uniform handling that minimized batch effects and the other with typical practices where the arrays were processed in multiple batches resulting in batch effects. More specifically, uniform handling refers to the fact that the microarrays for all 192 samples were processed in one experimental run by a single experienced technician and that the arrays were assigned to the samples using block randomization. We call the first design the uniformly handled design and the second the nonuniformly handled design. The raw array data were preprocessed with log2 transformation and median summarization across replicate probes for each feature.⁴³

Estimation of biological effects.

We use the data collected with the uniformly handled design from the 96 ovarian cancer samples as a best approximate for the biological effects of these samples. We call them virtual samples.

Estimation of handling effects.

Assuming that handling effects are additive, we use the differences between the uniformly handled and the nonuniformly handled data sets to estimate the handling effects of the arrays in the latter data set. We call them virtual arrays and split them (by whole slides, each of which contains eight arrays) into two 96-array sets, one for prediction model training and another for validation. The additivity assumption has been deemed reasonable for microarray data and has been adopted in published methods on microarray data normalization and analysis.^44,45

Elicitation of regression coefficients.

We use the virtual samples to assess each feature's association with progression-free survival (PFS), an important survival outcome variable in ovarian cancer.³³ PFS is defined as the time from primary surgery to disease progression, death, or loss of follow-up, whichever occurs first. The censoring rate was 23% (22/96) in the data. Univariate Cox regression analysis identifies six features with P < .005 whose log hazard ratio ranged from 0.26 to 0.78.⁴⁶ Three sets of true regression coefficients from the estimated regression coefficients are generated with various signal levels: (1) moderate signal with a few strong biomarkers, which uses the quadruple of the univariately estimated log hazard ratios for the six most significant features and zero for the other features; (2) weak signal with many weak biomarkers, which uses 0.35 for 30 randomly selected features (irrespective of the six features used in moderate signal scenario and fixed selection for all simulation replications) and zero for the other features; and (3) null signal, which uses zero for all features. We set the L1 norm of the moderate-signal coefficient vector to be equal to that of the weak-signal coefficient vector so that the two vectors have the same total effect that is distributed differently across features.

Simulation of survival outcome.

We use a previously developed permutation-based method to simulate the new PFS outcome on the basis of the true regression coefficients and biological effects of features. The simulated PFS maintains the same marginal distribution as the observed PFS. Briefly, the observed PFS times are sorted in ascending order and then sequentially and probabilistically paired with the virtual samples on the basis of multinomial distributions whose parameters are the partial likelihood of each unpaired virtual sample assuming it fails at the given PFS time. Detailed description of this algorithm can be found in Ni and Qin.³⁰

Simulation of miRNA data.

To avoid overfitting, we simulate both training data and test data to build prediction models and evaluate their prediction performance, respectively. Training and test data are both simulated with virtual rehybridization to preserve the complex across-feature correlation structure of biological effects and handling effects.^27,29 Namely, the 96 virtual samples (along with their PFS times) are reassigned to the 96 virtual arrays allocated for training and test, respectively. Handling effects for each virtual array are then added to biological effects of the assigned virtual sample. We consider two scenarios for the reassignment: (1) randomized and (2) sorted by simulated PFS time so that PFS times are associated with handling effects.

Batch adjustment.

We adjust handling effects that can be attributed to the data collection batches in the simulated data using BatMan or ComBat, either alone or following data normalization (median normalization, quantile normalization,⁴⁷ and variance stabilizing normalization [VSN]⁴⁸). ComBat is applied using the ComBat function in the R package sva⁴⁹; quantile normalization is applied using the normalize.quantiles function in the R package preprocessCore⁵⁰; VSN uses the vsn2 function in the R package vsn.⁴⁷ Similar methods are applied to the training data and test data, except that the frozen version of a corresponding normalization method is used for the test data.⁵¹ That is, frozen quantile normalization is used for the test data where quantiles derived from the training data are used to normalize the test data; VSN parameters that are estimated in the training data are used to normalize the test data.

Prediction model training.

To reduce computational burden in simulations and alleviate collinearity in model fitting, we prefilter the features using two criteria: (1) high abundance (that is, mean expression on the log2 scale among the 96 samples ≥8) and (2) no strong inter‐feature correlation (that is, Pearson correlation coefficient <0.9). These criteria are applied to each simulated data set, resulting in varied sets of features that pass the filtering across simulation runs. Typically, 100-200 features are kept for the model training step. The same two methods for variable selection (univariate variable selection and regularized regression) introduced for BatMan are also used for the regression model fitting following ComBat. As a reference, we add a third method, which contains the true predictive features, referred to as the oracle method. Although the oracle model is not obtainable in practice, these results are nevertheless helpful for assessing the impact of batch effects and the effectiveness of batch adjustment methods on prediction accuracy.

Prediction model evaluation.

Harrell's C-index is used to evaluate prediction accuracy.⁵² It is a function of the test data and regression coefficients of a trained model. For stratified Cox models, C-index is calculated as the weighted average of stratum-specific C-indices with the weight being the number of comparable pairs in each stratum. The mean, standard deviation, and 2.5th and 97.5th percentiles of the C-index among the 400 runs for each simulation scenario are calculated.

Simulation scenarios.

Table 1 summarizes the eight scenarios investigated in our simulations. The scenarios are arranged in such a way that batch effects are increasingly prevalent and involved (ie, associated with the survival outcome).

TABLE 1.

Summary of Simulation Scenarios

To summarize, our simulations assess the performance of 19 approaches for batch effects management in combination with four methods for prediction model development under eight scenarios for data simulation.

Application to the TCGA Ovarian Cancer Study

The proposed BatMan method, in comparison with the ComBat method, is applied to miRNA microarray data from the TCGA ovarian cancer study.³¹ The miRNA data along with the PFS data for 466 patients with stage II to IV primary, high-grade, untreated ovarian cancer were previously downloaded from the TCGA data portal.⁵³ Among the 466 patients, 342 (73.3%) had disease progression or died during the study follow-up. The batch variable, defined as the first shipment from the Biospecimen Core Resource, was downloaded at the TCGA Batch Viewer website.⁵⁴ There are 98 batches with two to eight samples per batch. The same strategies for managing batch effects and building prediction models in our simulation study are applied to build a prediction model for PFS using the miRNA data. The 466 samples are split, by whole batches, into four folds, with three folds used for model training and the remaining fold for testing as measured by the C-index; this procedure is repeated 50 times, and the average C-index is reported.

RESULTS

Simulation With Moderate Signals

We present the results of the oracle, lasso-penalized, and univariate selection methods in the main text. In addition, we include in the appendix the results of the adaptive lasso-penalized method, which is very similar to that of the lasso-penalized method (Appendix Fig A1).

Oracle method.

Figure 1 shows the simulation results for the oracle method in terms of the prediction accuracy measured by the test data C-index.

FIG 1.

Test data Harrell's C-index of the prediction model developed by the oracle method under moderate signals. Vertical bars represent the 2.5th and 97.5th percentiles. Symbols in the bars represent the means. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization. Panel legend: see notations in Table 1.

When batch effects are absent (BE00Asso00), BatMan (alone) leads to the best prediction accuracy when either batch or slide stratification is used (C-index, 0.94 and 0.95, respectively); ComBat by batches is nearly as good (C-index, 0.94) while ComBat by slides is slightly worse (C-index, 0.90); the three normalization methods, when used alone, have varied accuracy, with median normalization being close to BatMan and ComBat (C-index, 0.92) and VSN and quantile normalization significantly worse (C-index 0.80 and 0.74, respectively); the combinations of normalization with BatMan or ComBat show similar accuracy to that of normalization alone (C-index, 0.73-0.92).

When batch effects are present but not associated with survival time in the test data (BE10Asso00, BE10Asso10, BE11Asso00, and BE11Asso10), BatMan by slides offers notably better accuracy (C-index, 0.79-0.89) than any other method under investigation. Its performance worsens slightly when used with median normalization (C-index, 0.78-0.88). BatMan by batches and ComBat by slides, either used alone or combined with median normalization, also offer fairly high C-indices ranging from 0.75 to 0.86. ComBat by batches, either used alone or with median normalization, exhibits lower accuracy (C-index, 0.72-0.83). Again, VSN and quantile normalization, either used alone or together with BatMan or ComBat, are the worst performers (C-index, 0.69-0.76).

When batch effects are present and associated with survival time in the test data (BE11Asso01, BE11Asso11, and BE11Asso1-1), the performance of all methods deteriorates (C-index, 0.57-0.77). In particular, BatMan and ComBat decline more than the normalization methods. As a result, their performance becomes comparable with that of the normalization methods. Of note, when batch effects are negatively associated with survival time (BE1Asso1-1), all three normalization methods outperform BatMan and ComBat; however, this performance comes with the price of spurious predictive signals as we will show in the null-signal simulation scenario.

Comparing across the eight panels, prediction accuracy is highest when there are no batch effects. It decreases for all methods when batch effects are present and further so when batch effects confound survival time. This overall trend reveals the chilling reality that batch effects negatively affect prediction modeling despite the use of batch adjustment. It also supports that BatMan is consistently a top performer, followed by ComBat and median normalization, while quantile normalization, currently the most popular strategy, does not satisfactorily remedy batch effects in survival prediction. Moreover, none of the batch adjustment methods under consideration can salvage the negative impact of confounding between survival time and batch effects on survival prediction.

Penalized regression method.

Figure 2 displays the simulation results of lasso-penalized Cox regression. The general patterns are very similar to those of the oracle method, with BatMan alone still a top performer across the eight simulation scenarios. A new observation is that, when batch effects are associated with survival time in the training data (BE10Asso10, BE11Asso10, AssoBE11Asso11, and BE11Asso1-1), prediction accuracy is highly variable for quantile normalization and VSN as evidenced by exceptionally wide ranges between the 2.5th and 97.5th percentiles of the C-index, when used alone or in combination. This makes these methods even less desirable.

FIG 2.

Test data Harrell's C-index of the prediction model developed by the lasso method under moderate signals. Vertical bars represent the 2.5th and 97.5th percentiles. Symbols in the bars represent the means. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization. See notations in Table 1.

Univariate selection method.

Figure 3 presents the results when the univariate selection method is used to build the prediction model. In general, prediction accuracy becomes lower and substantially more variable across most adjustment strategies when compared with the oracle and lasso methods. Notably, when batch effects are only present in training data and associated with survival time (BE10Asso10 and BE11Asso10), ComBat by batches (either alone or combined with median normalization) slightly outperforms BatMan (C-index, 0.73-0.79 v 0.72-0.76). However, as the univariate selection method results in lower prediction accuracy than the lasso method across the board, the latter should be the preferred modeling strategy.

FIG 3.

Test data Harrell's C-index of the prediction model developed by the univariate method under moderate signals. Vertical bars represent the 2.5th and 97.5th percentiles. Symbols in the bars represent the means. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization. See notations in Table 1.

Simulation With Weak Signals

Simulation results under weak signals are included in Appendix Figures A2-A5. The general observations are consistent with those under moderate signals, demonstrating the robustness of our findings to the strength of prognostic signals.

Simulation With Null Signal

We further examine the performance of adjustment strategies under the null model where no feature is associated with the survival outcome. Note that the oracle method is not available under the null model. The results are presented in Appendix Figures A6-A8. Under the null model, as expected, no modeling approach offers any predictive value regardless of the choice of adjustment strategies in general, which is reflected by the C-index being around 0.5 (BE00Asso00, BE10Asso00, BE10Asso10, BE11Asso00, BE11Asso10, and BE11Asso01). However, when batch effects exist in both training and test data with positive association with the survival outcome (BE11Asso11), normalization and ComBat lead to small but notable predictive values beyond a random guess (C-index, 0.52-0.56); conversely, when the association between batch effects and the survival outcome is negative (BE11Asso1-1), normalization and ComBat exhibit slightly predictive values in the reverse direction reflected by C-index being below 0.5 (C-index, 0.45-0.48). These observations suggest that batch effects that confound survival outcome can induce spurious predictive values, either positive or negative depending on the direction of the confounding association.

Application to the TCGA Ovarian Cancer Study

The 466 samples in the TCGA study were collected in 98 batches, and as shown in Appendix Figure A9, they exhibited strong evidence of batch effects. Figure 4 depicts the average test data C-index from the repeated splits of the TCGA ovarian cancer data. The prediction accuracy of BatMan exceeds that of ComBat for all three modeling approaches (univariate selection, lasso, and adaptive lasso) regardless of the use of normalization. Moreover, its performance is worsened by the addition of VSN or quantile normalization. These results are consistent with our findings in the simulation study.

FIG 4.

Test data Harrell's C-index of the prediction model developed on the basis of the TCGA ovarian cancer microRNA microarray data. Boxplots represent the distributions of the test data C-index from the 50 repeated random splitting of the original data set. MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

DISCUSSION

Although stratified regression is proven useful for analyzing low-dimensional data, it has been underutilized for high-dimensional data and has not been applied for the purpose of managing batch effects in transcriptomics data. In this article, we propose to use stratification along with variable selection for simultaneously mitigating batch effects and building survival predictors. We demonstrated with both simulations and real-data analysis that BatMan is more effective than ComBat, especially when there are batch effects in the data. As pointed out by Nygaard et al¹⁵ and Li et al¹⁴ ComBat-adjusted data may possess biased group differences and inflated correlations among samples, leading to adverse impacts on downstream data analysis. In contrast, BatMan cancels out batch effects by taking advantage of the partial likelihood structure and hence avoids estimating and subtracting batch effects.

The performance of BatMan depends on the size of the strata (ie, the batches). Our simulation study examines two stratum sizes, array slides (each of size eight) and array batches (each of size 32-40). Slides offer smaller stratum size than batches, which leads to lower efficiency and greater variance as well as a higher likelihood of having zero-event strata, especially when censoring rate is high. Stratification by slides outperforms by batches in our simulations, but this may not always be the case. Overall, there is a bias‐variance tradeoff in the choice of strata. In practice, one could use cross-validation to choose the batch variable that offers the best prediction. The performance of BatMan also depends on the use of a preceding normalization step. Adding such a step worsens the prediction especially when quantile normalization or VSN is used, which is consistent with our recent study that evaluated the operating characteristics of data normalization for survival prediction.³⁰

We simulate array data with virtual rehybridization and outcome data with permutation to avoid unrealistic parametric assumptions and to mimic empirical data as much as possible. The resampling-based methods help preserve marginal distributions and complex correlation structures in biologic variations and batch effects and the association between biologic signals and survival outcome. However, these methods are limited in their ability to single out the impact of one factor while holding the other factors constant. For example, they cannot isolate whether the negative impact of batch effects is due to the altered marginal distributions of features or the induced interfeature correlations.

This article illustrates the benefits of the proposed BatMan method and assesses its performance along with ComBat and data normalization in the context of survival risk prediction. When developing a survival risk prediction model using transcriptomics data that possess batch effects, we recommend the use of BatMan without normalization. Even better, whenever possible, we encourage the use of uniform handling in data collection to prevent batch effects or at least confounding batch effects and generate high-quality data so that the data-to-knowledge translation can be accurate and reproducible.

APPENDIX

FIG A1.

Test data Harrell’s C‐index of the prediction model developed by the adaptive lasso method under moderate signals. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

FIG A2.

Test data Harrell’s C‐index of the prediction model developed by the oracle method under weak signals. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

FIG A3.

Test data Harrell’s C‐index of the prediction model developed by the lasso method under weak signals. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

FIG A4.

Test data Harrell’s C‐index of the prediction model developed by the adaptive lasso method under weak signals. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

FIG A5.

Test data Harrell’s C‐index of the prediction model developed by the univariate method under weak signals. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

FIG A6.

Test data Harrell’s C‐index of the prediction model developed by the lasso method under null signal. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

FIG A7.

Test data Harrell’s C‐index of the prediction model developed by the adaptive lasso method under null signal. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

FIG A8.

Test data Harrell’s C‐index of the prediction model developed by the univariate method under null signal. BatMan_B, stratification by slide batches; BatMan_S, stratification by array slides; ComBat_B, ComBat by slide batches; ComBat_S, ComBat by array slides; MN, median normalization; QN, quantile normalization; VSN, variance stabilizing normalization.

FIG A9.

Boxplots of miRNA microarray data for the 466 samples in the TCGA ovarian cancer study. Each boxplot represents a tumor sample. Alternating shading and non‐shading indicate experimental batches.

DATA SHARING STATEMENT

Human tumor tissues used in this study were obtained from participants who provided informed consent, and their use in our study was approved by the Memorial Sloan Kettering Cancer Center Institutional Review Board. The data and code supporting the conclusions of this article can be found at https://github.com/LXQin/PRECISION.survival.

AUTHOR CONTRIBUTIONS

Conception and design: Ai Ni, Li-Xuan Qin

Financial support: Li-Xuan Qin

Administrative support: Li-Xuan Qin

Collection and assembly of data: Li-Xuan Qin

Data analysis and interpretation: All authors

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).