Abstract
A common objective of biomarker studies is to develop a predictor of patient survival outcome. Determining the number of samples required to train a predictor from survival data is important for designing such studies. Existing sample size methods for training studies use parametric models for the high-dimensional data and cannot handle a right-censored dependent variable. We present a new training sample size method that is non-parametric with respect to the high-dimensional vectors, and is developed for a right-censored response. The method can be applied to any prediction algorithm that satisfies a set of conditions. The sample size is chosen so that the expected performance of the predictor is within a user-defined tolerance of optimal. The central method is based on a pilot dataset. To quantify uncertainty, a method to construct a confidence interval for the tolerance is developed. Adequacy of the size of the pilot dataset is discussed. An alternative model-based version of our method for estimating the tolerance when no adequate pilot dataset is available is presented. The model-based method requires a covariance matrix be specified, but we show that the identity covariance matrix provides adequate sample size when the user specifies three key quantities. Application of the sample size method to two microarray datasets is discussed.
Keywords: Conditional score, Cox regression, High-dimensional data, Risk prediction, Sample size, Training set
1. Introduction
Modern biological assays are often expensive and complex laboratory procedures. Many assays have moved beyond the initial “proof of principle” phase, and their predictive strength is being evaluated. An example is the National Cancer Institute’s Director’s Challenge Lung Study (DCLS), which was designed to study gene expression signatures in lung cancer based on previous smaller studies that had shown a prediction signal existed that could be used to separate patients with good prognosis from those with poor prognosis (e.g. Beer and others, 2002). The first author took part in the sample size calculations for the DCLS, which used parametric assumptions and a simplified model. This paper’s objective is to develop a more careful approach to sample size estimation.
Determining sample size in high-dimensional studies is critical to prevent either undersized studies—that lead to inconclusive or erroneously negative findings—or wasteful oversized studies. The sample size method developed here is appropriate for studies utilizing right-censored survival data and biological assay measurements, and applying a predictor development algorithm (PDA) that can be used to predict risk of death or disease recurrence. In some applications, these are called prognostic predictors. The sample size is chosen so that the trained predictor will, when applied to future samples from the same population, produce risk scores with a mean Cox regression slope within a user-specified tolerance of an optimally trained predictor. An optimally trained predictor is one trained on the whole population. The method can be used with any prediction algorithm that satisfies a minimal set of criteria.
We are aware of no current sample size methods for developing risk predictors in high dimensions. Sample size methods for detecting a Cox regression slope (e.g. Hsieh and Lavori, 2000) are not applicable because training a predictor is fundamentally different than detecting a univariate slope. Development of a risk predictor is closely related to the development of a classifier. Lachenbruch (1968) developed a sample size method for multivariate normal classification. The objective function used in this paper is a natural Cox regression analog of Lachenbruch’s.
The basic idea of our approach is to treat the estimated risk predictor as an error contaminated “observation” for the true (i.e. optimally trained) risk predictor, and to adopt measurement error techniques for sample size calculation. Several errors-in-variables (EIVs) methods have been developed for Cox proportional hazards regression, including regression calibration (Prentice, 1982), likelihood-based methods (e.g. DeGruttola and Tu, 1994; Wulfsohn and Tsiatis, 1997; Henderson and others, 2000; Xu and Zeger, 2001; Song and others, 2002a), simulation extrapolation (SIMEX; Cook and Stefanski, 1994), conditional score (Tsiatis and Davidian, 2001; Song and others, 2002b), and corrected score (Nakamura, 1992; Huang and Wang, 2000; Song and Huang, 2005). See Carroll and others (2006) for a good review. Among these approaches, the conditional score approach is consistent and simple to compute and shows satisfactory finite sample performance (Song and Huang, 2005). As a result, we use the conditional score approach in the sample size calculation.
EIVs regression requires an estimate of error variance. The non-parametric leave-one-out bootstrap (LOOBS) (Efron and Tibshirani, 1997) can be used to estimate the prediction error variance. Bootstrap methods for the context of Cox regression have been developed (Efron and Tibshirani, 1986; Davison and Hinkley, 1997). But, LOOBS estimates of prediction error variance are known to exhibit overdispersion (e.g. Efron, 1983), which can be very large in high dimensions (e.g. Dobbin, 2009). Methods to adjust for overdispersion have been developed based on resubstitution (Efron, 1983; Efron and Tibshirani, 1997) and nested resampling (Jiang and Simon, 2007). However, resubstitution-based methods cannot be used here because they are based on error rates relative to true values, but the true asymptotic values are never observed directly. Nested resampling-based methods can entail excessive computational overhead. This application uses an alternative approach, which we call the tuned LOOBS. The tuned LOOBS is computationally efficient, corrects much of the bias of the LOOBS, and does not require knowledge of the asymptotic risk scores.
Ultimately, sample size determination for predictor training studies requires estimating the learning rate of the predictor. The learning rate is the speed at which the predictor performance improves as the sample size increases. Previous researchers have estimated the learning rate by fitting a specific parametric learning curve to a plot of 
versus 
error (Duda and others, 2001; Mukherjee and others, 2003). We have found these parametric assumptions questionable (Dobbin and Simon, 2011). As a result, we develop here a novel method for estimating the learning rate, which we call the linear transformation method. The method permits more flexibility in model fitting.
The sample size method is applied to two cancer datasets. One dataset was generated to independently validate a 76-gene signature in node-negative breast cancer patients. Tumor sample gene expression was analyzed. This retrospective cohort included time to distant metastases for 198 women treated at the Bordet Institute. Desmedt and others (2007) did successfully validate the signature, and this signature is being used in a clinical trial (Bogaerts and others, 2006). The second dataset studied a retrospective set of primary ovarian cancer tumors from 185 women hospitalized at Memorial Sloan–Kettering Cancer Center. A univariate Cox regression was fit to each feature to establish association with survival, and a multi-feature signature using 572 genes associated with survival in the training set was validated in an independent dataset after binarizing the survival predictions. The existence of signatures in ovarian cancer is less well established than in breast cancer. The study identified a prognostic signature in only a subset of the patients.
This paper is organized as follows. Section 2 presents an overview of the method. Section 3 presents the statistical model, definitions, and assumptions. Section 4 presents the estimation procedures in general. Section 5 demonstrates implementation of the method. Sections 6 and 7 present simulation studies, resampling studies, and real data applications. Finally, Section 8 presents summaries and conclusions.
2. Overview of approach
A training set can be used to develop a risk predictor. For a future patient, the risk predictor will assign a numerical value to that patient indicating the risk of disease recurrence or death (e.g. larger values indicate higher risk). But this risk score is an estimated value, and it is conditional on the training set. If a different training set is used, a different risk score will likely be assigned to the patient. This variation is analogous to measurement error variation. The measurement error is just a prediction error due to the limited training set. Under regularity conditions (Section 3 of supplementary material available at Biostatistics online) the prediction error will decrease stochastically as the training sample size, say 
, increases (Section 4, Theorem 1 of supplementary material available at Biostatistics online). In the limit, as 
goes to infinity, the prediction error will go to zero. A zero prediction error is in this way analogous to the zero measurement error. EIVs regression methods are used to recover the regression relationship between a response and the predictor when the predictor is measured with error. So, to estimate the relation between survival (the response) and the optimal risk scores, EIVs regression can be used. This reasoning leads to the sample size method shown in Figure 1.
Figure 1.
Overview of sample size procedure.
As shown in Figure 1, the method is based on a pilot dataset. The steps shown are: (1) select 
at random (without replacement) from the pilot dataset; (2) estimate the prediction performance for a training sample of size 
using cross-validation (CV); (3) estimate the variance of the risk score estimates by the tuned LOOBS; (4) combine the results of steps (2) and (3) to estimate the optimal performance by EIVs regression; (5) combine steps (2) and (4) to estimate the tolerance (distance from optimal, defined below) for 
; (6) estimate the learning curve from the pilot dataset using subsets of different sizes—the learning curve describes the relation between the training set size and the expected predictor performance; (7) use the results of step (6) to determine the sample size that will guarantee that the expected predictor performance is within a specified tolerance of the asymptotic performance.
3. The statistical model
The pilot dataset will contain 
patients’ data, consisting of vectors 
, each of dimension 
. With all patients are associated failure times 
and censoring times 
, which are all mutually independent. The observed quantities are 
and failure indicators δi=1{Ti≤Ci}
i=1,…,n. The failure time 
is assumed to be completely described by the hazard rate
 |
The most commonly used model in this setting is the Cox proportional hazards model
 |
where 
is a scalar, and 
the baseline hazard function. In the Cox model, the slope parameter 
will change if the 
are rescaled (e.g. 
, 
). Also, a mean shift in the 
(e.g. 
, 
) results in a change in the definition of the nuisance baseline hazard 
. So, to ensure identifiability, one assumes the 
have mean 
and variance 
. In real data applications, the variance of the risk scores may not be unity without first rescaling them. The 
is the change in the log-hazard associated with an increase of one standard deviation in the asymptotic risk scores, and rescaling will change the standard deviation.
The 
above we call the asymptotic risk scores. They are the limit of the estimated risk scores of a risk prediction algorithm. Under regularity conditions given in supplementary material available at Biostatistics online (Section 3), a patient 
’s estimated risk score will converge in quadratic mean to a scalar value 
as the training sample size increases.
An algorithm applied to a training set, say 
, results in a function that maps 
to 
. The resulting function, say 
, is assumed to be deterministic, so that we can write 
(Section 3, condition 1 of supplementary material available at Biostatistics online).
Unlike traditional Cox regression contexts, the asymptotic risk scores 
are not observed directly. Define 
. Here 
is an unknown function that maps the high-dimensional data into the asymptotic risk scores. Let 
. We relate the asymptotic risk scores to the estimated risk scores by a simple additive error model:
 |
(3.1) |
Here 
represents prediction error with zero mean. The notation 
is used instead of 
to facilitate comparison with Carroll and others (2006).
3.1. Obtaining 
In this paper, we focus on the sample size method once the algorithm for 
is determined. But in practice, selection of an appropriate prognostic predictor training function 
itself is critical. The function 
must satisfy the following conditions: (1) the prediction scores produced by 
must converge in quadratic mean to the true scores (for details, see Section 3.1 of supplementary material available at Biostatistics online) and (2) the rate of convergence must be fast enough that the learning curve can be adequately estimated from the pilot dataset. Condition (2) implies that one should pick the algorithm with the fastest possible learning rate. There will typically be algorithms that are widely accepted. For microarrays, examples include (1) PDA1 described below, (2) Lasso Cox regression (Tibshirani, 1996), and (3) elastic net Cox regression (Simon and others, 2011). See Witten and Tibshirani (2010) for comparisons of procedures. These three produce linear predictors, use the survival information, and are suited to high-dimensional low sample size contexts.
3.2. Defining En, Varn, σ2n, and Tol(n)
Define the notation 
and 
as the expectation and variance, respectively, taken over training sets of size 
in the population. Intuitively, one imagines drawing independent samples of size 
at random repeatedly from the target population, and each time constructing a risk predictor. This results in 
—an infinite sequence of risk predictors, across which means and variances are taken.
Under this notation, the variance of an individual’s estimated risk score for a fixed training size 
is 
. We will make the simplifying assumption that the prediction error variances are the same for all individuals 
, that is, 
.
If, for a fixed training set 
, a Cox regression of survival on all the estimated risk scores 
in the population is performed, under certain regularity conditions (Section 4.3 of supplementary material available at Biostatistics online) this will produce a slope estimate 
. For two different training samples of size 
, say 
, the slopes may be different, that is, 
. Under regularity conditions (Section 4.3 of supplementary material available at Biostatistics online and Li and Ryan, 2004), 
exists and 
. 
is shrunken toward zero. The tolerance is defined as 
.
4. Estimation
Briefly, for subsets of different sizes, first 
is estimated by CV, then 
is estimated by the tuned LOOBS, and finally the tolerance is estimated by EIVs regression. For details, see Section 2 of supplementary material available at Biostatistics online.
The 
is the Cox regression slope associated with an infinite training set. The risk scores estimated from CV, 
, and the LOOBS variance estimate, 
, are combined with the survival data in an EIVs Cox regression. Similar to Tsiatis and Davidian (2001), the conditional score approach we use assumes that Uij|Xi,Ti,Ci∼N(0,σ2). As shown in Figure S1 of supplementary material available at Biostatistics online (Section 1.7), homogeneity of the error variance appeared reasonable, so the heterogeneous model was not investigated in depth. Suppose one wants to estimate 
using the 
th training set. Let 
be the counting process for the failure time, and 
be the at risk process. Let Sij(u,β,σ2)=(Wij+dNi(u)σ2β)T. The conditional score estimating equation is
 |
where 
(
).
The tolerance is the absolute value of the difference between the mean Cox regression slope for a sample of size 
and the asymptotic slope. The estimate of Tol(n*) is 
, where 
is the slope estimate from the conditional score regression using the 
in the working training set, and 
is from the CV (Section 2 of supplementary material available at Biostatistics online).
Note that predicting the sample size associated with a desired tolerance is analogous to a regression prediction problem. First transform the tolerance using a Box–Cox transformation (Box and Cox, 1964) to linearize the relationship. Let 
be the estimated transformation of the tolerance. Finally, fit the linear regression model 
producing the ordinary least-squares estimates 
and 
. Let 
be the targeted tolerance, which is specified by the user. The sample size estimate is 
. The sample size algorithm is given in Section 2.3 of supplementary material available at Biostatistics online.
4.1. Constructing a confidence interval for the tolerance
The sample size estimate 
does not reflect the uncertainty in the estimation procedure. This uncertainty can be assessed in a confidence interval for the tolerance. Let 
be the proposed sample size. Let 
be the tolerance. The Box–Cox regression model is 
, where 
. Let 
, and 
be the ordinary least-squares estimates from the Box–Cox regression. The confidence interval formula, derived in Section 5 of supplementary material available at Biostatistics online, is
 |
A similar result is presented in Collins (1991). We performed a simulation to evaluate the procedure. Results are also discussed in Section 5 of supplementary material available at Biostatistics online as well.
5. High-dimensional example
The risk predictor method used in this paper’s examples is similar to the method used in Beer and others (2002). Denote the method by PDA1. Roughly, in PDA1, features are selected based on univariate score test 
-values, then weighted by univariate Cox regression coefficients. Details are provided in Section 2.2 of supplementary material available at Biostatistics online.
5.1. The tuned LOOBS on PDA1
PDA1 selects features with score test 
-values below a cutoff stringency level, such as 
. Denote by 
the estimated slope from a univariate Cox regression on feature 
applied to a bootstrap sample. Let 
be the observed information for feature 
based on the univariate partial likelihood of the bootstrap sample. To approximately preserve the nominal significance level, feature 
is selected during bootstrap if 
. While maintaining approximate significance level (Sections 1.2 and 4.6 of supplementary material available at Biostatistics online), this approach results in loss of bootstrap power compared with cross-validated power. The power lost is that associated with multiplying the standard error by roughly 
.
5.2. Tolerance estimation without a pilot dataset
A pilot dataset may not be available or may not be adequate as discussed below. Our approach can be adapted to that setting by assuming high-dimensional data are multivariate normal and survival is exponential. A combination of mathematics and Monte Carlo are used to estimate the tolerance. A key step is to estimate the prediction error variance 
with 
where 
is the population covariance and 
is the covariance estimate of the linear predictor. Details are presented in Section 5 of supplementary material available at Biostatistics online. Table ST12 of supplementary material available at Biostatistics online shows that the identity covariance matrix produces reasonable or conservative tolerance estimates for other typical covariance matrices with the same values of 
, 
, and the average marginal effect size. The marginal effect size for feature 
is approximated by (Section 1.6 of supplementary material available at Biostatistics online),
 |
where 
is the 
th element of the covariance matrix 
, and 
is the 
th element of 
. Therefore, only the identity covariance matrix need be used in practice. Users of the program should restrict simulations to settings where there is at least 70% power to detect survival features (otherwise a warning message appears). The program can currently be used with PDA1 or the lasso (Tibshirani, 1996).
6. Simulation studies
For all simulations, survival data were generated with baseline hazard exponential with mean 
, or with Weibull where noted. Censoring times were exponential with mean 
. The follow-up time was stopped at 
. High-dimensional data for features not associated with survival were generated as multivariate normal (or T where noted), with zero mean and identity covariance. Survival-related features were generated as indicated in the text and supplementary material available at Biostatistics online. Dimension was set at 
except where noted differently. Computation was carried out in C
on a Borland 5.0 compiler using Optivec and IMSL vector and matrix libraries, and R version 2.15.2.
Simulation evaluation of the tuned LOOBS appears in Section 1.2 of supplementary material available at Biostatistics online, where it is shown to reduce the bias of the simple LOOBS.
Table 1 (and Section 1.1 tables of supplementary material available at Biostatistics online) shows the estimation of 
for a variety of settings. Table 1 presents a scenario with 30 survival-associated features in a block compound symmetric covariance structure. As can be seen from the tables, 
is close to unbiased in most cases. The 
estimate tends to do better in cases where the feature selection stringency is appropriate to the data, and to break down in cases where the stringency is too lax (e.g. first row of first table in Section 1 of supplementary material available at Biostatistics online) or too strict (e.g. Table 1 when 
). Overall, the slope estimates do well at recovering the asymptotic slope for these sample sizes (
and about 200 events/deaths).
Table 1.
Monte Carlo results for estimating the asymptotic slope
 |
 |
 |
 |
 |
|---|---|---|---|---|
| 0.01 | 2.0 | 2.06 | 0.03 | 1.90 |
| 0.01 | 1.5 | 1.53 | 0.04 | 1.41 |
| 0.01 | 1.0 | 1.05 | 0.09 | 0.92 |
| 0.01 | 0.5 | 0.53 | 0.34 | 0.39 |
| 0.01 | 0.0 |
 0.01 |
0.82 | 0.00 |
| 0.001 | 2.0 | 2.09 | 0.03 | 1.91 |
| 0.001 | 1.5 | 1.61 | 0.05 | 1.44 |
| 0.001 | 1.0 | 1.10 | 0.15 | 0.89 |
| 0.001 | 0.5 | 0.43 | 0.48 | 0.29 |
| 0.001 | 0.0 |
 0.03 |
0.49 |
 0.02 |
| 0.0001 | 2.0 | 2.35 | 0.07 | 1.89 |
| 0.0001 | 1.5 | 1.77 | 0.11 | 1.39 |
| 0.0001 | 1.0 | 1.13 | 0.26 | 0.80 |
| 0.0001 | 0.5 | 0.36 | 0.33 | 0.25 |
| 0.0001 | 0.0 | 0.00 | 0.08 | 0.00 |
Entries are means (standard errors in ST2 of supplementary material available at Biostatistics online) of 100 Monte Carlo. Simulation settings have 
patients. High-dimensional feature covariance matrix has 30 informative features, in three compound symmetric blocks of size 10, within-block correlation 0.7, between-block correlation 0. 
bootstraps in the LOOBS loop. Column headings are: 
column is the stringency used for feature selection; 
is the optimal Cox regression slope. 
is the estimate of the optimal slope produced by our method. 
is the estimate of prediction error variance from the tuned LOOBS. 
(where 
is the raw slope that comes out of CV) is the estimated slope from 10-fold CV for the full dataset predictor developed from 
samples.
Table 2 shows the estimated sample sizes for a range of simulation settings, along with a pure Monte Carlo evaluation of the adequacy of the sample size estimates (rightmost two columns). In all cases, the estimated sample size results in a mean slope that is within the specified tolerance. The raw slopes are also within the tolerance value 56–99% of the time (rightmost column). (The raw slope is analogous to actual prediction error, and the mean slope is analogous to expected prediction error. To clarify this distinction, see, e.g. Dobbin, 2009.) The method is most conservative in the two cases when there are 30 informative features and the tolerance is set to the largest level of 
. In other settings, however, the method seems only mildly conservative.
Table 2.
Monte Carlo results for estimating the sample size, and pure Monte Carlo evaluations thereof
Pure MC eval. of  |
|||||||
|---|---|---|---|---|---|---|---|
 |
 (events) |
SRF |  |
Tolerance | estimate ( ) |
Mean Tol | % within Tol |
| 2 | 300 (194) | 1 | 0.001 | 0.10 | 467 | 0.071 | 73 |
| 2 | 300 (194) | 1 | 0.001 | 0.20 | 219 | 0.144 | 70 |
| 2 | 300 (194) | 1 | 0.001 | 0.30 | 136 | 0.190 | 74 |
| 2 | 300 (196) | 30 | 0.010 | 0.10 | 321 | 0.058 | 86 |
| 2 | 300 (196) | 30 | 0.010 | 0.20 | 249 | 0.141 | 81 |
| 2 | 300 (196) | 30 | 0.010 | 0.30 | 207 | 0.120 | 99 |
| 2 | 160 (103) | 1 | 0.001 | 0.10 | 417 | 0.057 | 75 |
| 2 | 160 (103) | 1 | 0.001 | 0.20 | 199 | 0.167 | 66 |
| 2 | 160 (103) | 1 | 0.001 | 0.30 | 127 | 0.198 | 73 |
| 2 | 160 (103) | 30 | 0.010 | 0.10 | 537 | 0.045 | 76 |
| 2 | 160 (103) | 30 | 0.010 | 0.20 | 335 | 0.099 | 96 |
| 2 | 160 (103) | 30 | 0.010 | 0.30 | 246 | 0.123 | 98 |
| 1 | 300 (210) | 1 | 0.001 | 0.10 | 258 | 0.095 | 56 |
| 1 | 300 (210) | 1 | 0.001 | 0.20 | 115 | 0.127 | 70 |
| 1 | 300 (210) | 1 | 0.001 | 0.30 | 67 | 0.199 | 66 |
Features distributed multivariate normal. When SRF (survival-related 
then feature covariance matrix is identity; when 
, then feature covariance is block diagonal compound symmetric for the 30 survival-related features, within-block correlation 0.7, between-block correlation 0.0, independent features uncorrelated. 
bootstraps in LOOBS. For the “Pure MC Eval. of 
”, 400 samples of size 
were created and a risk predictor developed on each; then each risk predictor was applied to a separate set of 5000 samples to obtain estimates of the mean slope 
associated with future samples. Then “mean Tol” is average tolerance and “% within Tol” is the proportion of estimated slopes for which this difference was less than the “Tolerance” column. When 
, due to RAM memory issues simulation parameters were slightly different: 2000 samples of size 
were created, and in each case 500 independent samples used to obtain estimates of τ2j,indep and 
.
Robustness of our method to poor power to detect survival features in the pilot dataset was evaluated by systematically generating datasets with smaller and smaller effects. Results are shown and discussed in Section 1.3 of supplementary material available at Biostatistics online. Power of at least 70% to detect survival-related features is recommended, because without these features the asymptotic performance estimate cannot be relied upon.
Now we turn to evaluation of the model-based tolerance estimation method, investigating robustness to model violations using pure Monte Carlo and resampling studies. The Monte Carlo robustness studies generated data from high-dimensional multivariate T distributions with 5 and 10 degrees of freedom, and accelerated failure time survival distributions with increasing and decreasing hazard. The model-based estimates were robust to these model violations (Section 1.5 of supplementary material available at Biostatistics online). In most settings, the estimated tolerance is within 0.05 of the true tolerance, the exception being certain cases of a heavy-tailed multivariate T with 5 d.f. and correlated features. For resampling evaluation, we used two large survival datasets, the Rosenwald and others (2002) dataset of leukemia, and the Loi and others (2007) dataset of breast cancer. Table 3 shows the results. Since resampling will not provide a 
, we instead use the tolerance associated with large sample sizes. For the Rosenwald dataset, we estimate 
. For the Loi dataset, we estimate 
. We estimate each by pure resampling first (top row), and then by our method with an identity covariance (second row). As can be seen from the table, the estimated tolerance is greater than the true tolerance for all cases (that is, the entry in the second row is greater than the first). In some cases, the estimated tolerance is very conservative, but in others it is only mildly conservative. Results held up under AR1 and CS covariance (Section 1.8 of supplementary material available at Biostatistics online). The table also shows one of the simulations we performed to show that the identity covariance provides conservative sample size estimates for a wide range of covariances as long as the dimension, 
, and the marginal effect size are the same. Note that the tolerance estimates in the bottom row are greater than all the other rows when the observed power is adequate (over 70%), showing our method performs well with the identity matrix even for these non-identity covariance matrices. Note that the R program calculates and prints the power.
Table 3.
Robustness evaluations of model-based method
| Resampling evaluation on Rosenwald data |
||||
|---|---|---|---|---|
 (events) |
50 (29) | 100 (58) | 150 (86) | 200 (115) |
 by resamp |
0.20 | 0.09 | 0.04 |
 0.02 |
 identity |
0.30
|
0.24 | 0.23 | 0.24 |
| Resampling evaluation on GSE6532 data | ||||
 (events) |
100 (36) | 150 (54) | 200 (73) | 250 (91) |
 by Resamp |
0.198 | 0.150 | 0.124 | 0.074 |
 identity |
0.301 | 0.260 | 0.246 | 0.216 |
| Simulation evaluation | ||||
 (events) |
100 (65) | 200 (130) | 300 (195) | 400 (260) |
 |
1.38 | 0.63 | 0.28 | 0.19 |
 |
1.14 | 0.40 | 0.16 | 0.08 |
 |
1.36 | 0.61 | 0.29 | 0.22 |
 |
1.08 | 0.47 | 0.19 | 0.12 |
 identity |
 |
0.81 | 0.60 | 0.42 |
Resampling studies on Rosenwald dataset and Loi dataset, and simulation study of a variety of covariances and linear predictors. For the resampling studies: first row is sample size and number of deaths (
(events)); second row is tolerance by resampling (
by Resamp), comparing the full dataset to the specified sample size (
, where 
for the Loi dataset and 
for the Rosenwald dataset); third row is tolerance using identity covariance (
identity); AR1 and CS covariances are similar and appear in Section 1.18 of supplementary material available at Biostatistics online. For the simulation studies: first row is the sample size and number of deaths (
(events)). Rows 2–6 are different combinations of 
and 
that produce the same marginal effect sizes. Without loss of generality, the first 9–30 features are the survival-related features, and the rest are independent noise features. The survival features are correlated. 
is block compound symmetric (block CS), three blocks of size 3, and correlation parameter 0.4; 
is block CS, three blocks of size 10, and correlation parameter 0.55; 
is block AR1, three blocks of size 3, and correlation parameter 0.48; 
is block AR1, three blocks of size 10, and correlation parameter 0.83. The 
are set so each element is the same number and 
for 
. More results and details appear in Sections 1.6 and 1.8 of supplementary material available at Biostatistics online.
Observed power below 70%, so our method not recommended here.
7. Application to real datasets
The sample size method was applied to the two microarray datasets described in Section 1. Desmedt and others (2007) studied gene expression profiles from frozen tumor samples of node negative breast cancer patients. The survival endpoint was defined as time from diagnosis to death from any cause or distant metastasis. Bonome and others (2008) studied gene expression signatures of stage III, high-grade primary ovarian cancer tumors. The survival endpoint was time from surgery to death from any cause. Patients were categorized by whether their tumors had been optimally or suboptimally debulked during surgery—a major prognostic factor. Both studies used Affymetrix U133A arrays.
Results are shown in Table 4, and details are presented in Sections 4.14.2 and 4.14.3 of supplementary material available at Biostatistics online. For the breast cancer data of Desmedt and others (2007), the estimated sample size to achieve a tolerance of 
was 207 for the distant metastases-free survival endpoint, or slightly larger than the 190 used in the study. The upper bound on the 90% confidence interval when 
was 0.23. For the ovarian cancer dataset of Bonome and others (2008), the estimated sample size is 120 to obtain a tolerance of 
for an association with overall survival. However, there is greater uncertainty here; the upper bound on the 90% confidence interval is 0.33—much larger than the breast cancer dataset. This result makes biological sense because the ovarian tumors were a heterogeneous set of optimally and suboptimally debulked tumors, and a survival signature was only found associated with the suboptimal group.
Table 4.
Sample size method applied to two microarray datasets
| Dataset | GSE7390 | E-Geod-26712 |
|---|---|---|
| Disease | Breast cancer | Ovarian cancer |
Train 
|
190 | 180 |
| Events | 88 | 129 |
| Predictors | 22 283 | 22 283 |
 |
37 | 17 |
 |
48 | 20 |
Sample size 
|
207 | 120 |
 |
10 | 10 |
| 90% CI for Tol | ( 2,23) |
(10.7,33.9) |
GSE7390 is from the Desmedt and others (2007) study of breast cancer using Affymetrix U133A microarrays. E-Geod-26712 is from the Bonome and others (2008) study of ovarian cancer using Affymetrix U133A microarrays. Outcome is overall survival. 
for the microarray datasets. Row heading descriptions: “Train 
” is the number in the training set (original dataset); “Events” is the number of deaths in the training set; “Predictors” is the number of features; “
” is the 10-fold cross-validated estimate of the Cox regression slope; “
” is the estimated optimal Cox slope; “Tolerance” is the user-specified tolerance for the sample size calculation; “
” is the sample size estimate; bottom row is the confidence interval from our CI method.
In the PDA1 algorithm, each feature is centered at zero and scaled to have variance 1. This adjustment was found critical in the applications. The conditional score method for datasets of size 
sometimes would not converge and were omitted.
The proportional hazards assumption was evaluated for each dataset as described in Section 4.5 of supplementary material available at Biostatistics online (Song and others, 2002b) by introducing an interaction between time and the cross-validated risk prediction scores. For GSE7390, the 
-value for the interaction was 0
74, and for E-GEOD-26712, the interaction 
-value was 0.64.
8. Discussion
In this paper, a method for determining the number of samples required to train a risk predictor on high-dimensional data was developed. The method can be used in low dimensions as well. This approach can be used with any risk prediction algorithm that satisfies a set of conditions, and requires a pilot dataset in order to perform the estimation. If no adequate pilot dataset is available, then a method and associated programs are presented for estimating tolerance from a model. The sample size method produces an estimate of the optimal (asymptotic) risk prediction performance associated with an infinite training set. This estimate is itself potentially useful for study planning and evaluation of the clinical utility of an observed relationship between a bioassay (e.g. gene expression signature) and survival. The sample size method was shown to work well on simulated data. The method was applied to several real datasets and issues with hands-on use of the method were discussed. Similarly, the parametric estimate of tolerance was evaluated in simulations and resampling studies and performed well.
The method employs a novel LOOBS approach which is called tuned (LOOBS). Tuned LOOBS tunes the feature selection during bootstrap in order to nearly match the cross-validated stringency level. This reduces the bootstrap overdispersion. We are developing tuned methods for other prediction algorithms to make the bootstrap variance estimates more accurate, and this is an area of ongoing research.
The conditional score method was used to fit an EIV Cox regression. We investigated also the SIMEX method, but it was not feasible because fitting the extrapolation curve required manual inspection of plots. When the error is heterogeneous, the method of Li and Ryan (2004) can be adopted.
The sample size is chosen so that the tolerance is below a user-specified value. The standard deviation of the risk scores is 1. Thus, if the asymptotic risk score increases by one standard deviation, the increase in hazard will be 
. For example, if 
, then the increase is 2-fold. If the tolerance was set to 
, then 
, so 
. Thus, an increase of one standard deviation in the estimated risk scores would be expected to be associated with an 
increase in the hazard.
A limitation of our approach is the use of the Cox regression model. The model assumes that the asymptotic risk scores are linearly related to the log-hazard. This assumption is probably at best an approximation. On the other hand, simple linear models have proved effective in high-dimensional class prediction studies (Dudoit and others, 2002), and therefore seem reasonable for risk prediction. A less parametric approach is possible, such as one based on the overall C criterion (Harrell and others, 1996) instead of the Cox regression slope, as suggested by a referee. By estimating the C criterion repeatedly for a range of sample sizes (less than or equal to the pilot dataset size), the relationship between the training size and C could be estimated, perhaps using a parametric non-linear regression. Fitting the model can produce an estimate of the optimal C, the C for a specified sample size, and the “c-tolerance” difference between the two.
Our method requires average power be over 70% to detect the marginal effect sizes. A formula for calculating marginal effect sizes was provided. The method of Hsieh and Lavori (2000) can be used to calculate the power and average power using, say, significance level 0.001 for PDA1.
Supplementary material
Supplementary material is available at http://biostatistics.oxfordjournals.org.
Funding
K.K.D. and X.S. were partially supported by 1R21CA152460 from NCI. K.K.D. was partially supported by the Georgia Cancer Coalition Distinguished Cancer Clinicians and Scientists award. X.S. was partially supported by NSF DMS-1106816 and NIH R01ES017030.
Supplementary Material
Acknowledgements
Conflict of Interest: None declared.
References
- Beer D. G., Kardia S. L. R., Huang C., Giordano T. J., Levin A. M., Misek D. E., Lin L., Chen G., Gharib T. G., Thomas D. G. and others Gene-expression profiles predict survival in patients with lung adenocarcinoma. Nature Medicine. 2002;8:816–824. doi: 10.1038/nm733. [DOI] [PubMed] [Google Scholar]
- Bogaerts J., Cardoso F., Buyse M. and others. and others Gene signature evaluation as a prognostic tool: challenges in the design of the MINDACT trial. Nature Clinical Practice Oncology. 2006;3:10. doi: 10.1038/ncponc0591. [DOI] [PubMed] [Google Scholar]
- Bonome T., Levine D. A., Shih J., Randonovich M., Pise-Masison C. A., Bogomolniy F., Ozbun L., Brady J., Barrett J. C., Boyd J., Birrer M. J. A gene signature for survival in suboptimally debulked patients with ovarian cancer. Cancer Research. 2008;68:5478–5486. doi: 10.1158/0008-5472.CAN-07-6595. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Box G. E. P., Cox D. R. An analysis of transformations (with discussion) Journal of the Royal Statistical Society. 1964;26:211–252. [Google Scholar]
- Carroll R. J., Ruppert D., Stefanski L. A., Crainiceanu C. M. Measurement Error in Nonlinear Models: A Modern Perspective. 2nd edition. Boca Raton: Chapman & Hall; 2006. [Google Scholar]
- Collins S. Prediction techniques for Box–Cox regression models. Journal of Business and Economic Statistics. 1991;9:267–277. [Google Scholar]
- Cook J. R., Stefanski L. A. Simulation-extrapolation estimation in parametric measurement error models. Journal of the American Statistical Association. 1994;89:1314–1328. [Google Scholar]
- Davison A. C., Hinkley D. V. Bootstrap Methods and their Application. Cambridge, UK: Cambridge University Press; 1997. [Google Scholar]
- DeGruttola V., Tu X. Modeling progression of CD4-lymphocyte count and its relationship to survival time. Biometrics. 1994;50:1003–1014. [PubMed] [Google Scholar]
- Desmedt C., Piette F., Loi S., Wang Y., Lallemand F., Haibe-Kains B., Viale G., Delornzi M., Zhang Y., d’Assignies M. S. and others. Strong time dependence of the 76-gene prognostic signature for node-negative breast cancer patients in the TRANSBIG Multicenter Independent Validation Series. Clinical Cancer Research. 2007;13:3207–3214. doi: 10.1158/1078-0432.CCR-06-2765. [DOI] [PubMed] [Google Scholar]
- Dobbin K. K. A method for constructing a confidence bound for the actual error rate of a prediction rule in high dimensions. Biostatistics. 2009;10:282–296. doi: 10.1093/biostatistics/kxn035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Dobbin K. K., Simon R. M. Optimally splitting cases for training and testing high dimensional classifiers. BMC Medical Genomics. 2011;4:31. doi: 10.1186/1755-8794-4-31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Duda R. O., Hart P. E., Stork D. G. Pattern Classification. 7th edition. New York: John Wiley & Sons; 2001. [Google Scholar]
- Dudoit S., Fridlyand J., Speed T. P. Comparison of discrimination methods for the classification of tumors using gene expression data. Journal of the American Statistical Association. 2002;97:77–87. [Google Scholar]
- Efron B. Estimating the error rate of a prediction rule: improvement on cross-validation. Journal of the American Statistical Association. 1983;78:316–331. [Google Scholar]
- Efron B., Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science. 1986;1:54–77. [Google Scholar]
- Efron B., Tibshirani R. Improvements on cross-validation: the .632+ bootstrap method. Journal of the American Statistical Association. 1997;92:548–560. [Google Scholar]
- Harrell F. E., Lee K. L., Mark D. B. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine. 1996;28:361–387. doi: 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4. [DOI] [PubMed] [Google Scholar]
- Henderson R., Diggle P., Dobson A. Joint modelling of longitudinal measurements and event time data. Biostatistics. 2000;1:465–480. doi: 10.1093/biostatistics/1.4.465. [DOI] [PubMed] [Google Scholar]
- Hsieh F. Y., Lavori P. W. Sample-size calculations for the Cox proportional hazards regression model with nonbinary covariates. Controlled Clinical Trials. 2000;21:552–560. doi: 10.1016/s0197-2456(00)00104-5. [DOI] [PubMed] [Google Scholar]
- Huang Y., Wang C. Y. Cox regression with accurate covariates unascertainable: a nonparametric-correction approach. Journal of the American Statistical Association. 2000;95:1209–1219. [Google Scholar]
- Jiang W., Simon R. A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification. Statistics in Medicine. 2007;26:5320–5334. doi: 10.1002/sim.2968. [DOI] [PubMed] [Google Scholar]
- Lachenbruch P. A. On expected probabilities of misclassification in discriminant analysis, necessary sample size, and a relation with the multiple correlation coefficient. Biometrics. 1968;24:823–834. [Google Scholar]
- Li Y., Ryan L. Survival analysis with heterogeneous covariate measurement error. Biometrics. 2004;99:724–735. [Google Scholar]
- Loi S., Haibe-Kains B., Desmedt C., Lallemand F., Tutt A. M., Gillet C., Ellis P., Harris A., Bergh J., Foekens J. A. and others Definition of clinically distinct molecular subtypes in estrogen receptor-positive breast carcinomas through genomic grade. Journal of Clinical Oncology. 2007;25:1239–1246. doi: 10.1200/JCO.2006.07.1522. [DOI] [PubMed] [Google Scholar]
- Mukherjee S., Tamayo P., Rogers S., Rifkin R., Engle A., Campbell C., Golub R. R., Mesirov J. P. Estimating dataset size requirements for classifying DNA microarray data. Journal of computational biology. 2003;10:119–142. doi: 10.1089/106652703321825928. [DOI] [PubMed] [Google Scholar]
- Nakamura T. Proportional hazards model with covariates subject to measurement error. Biometrics. 1992;48:829–838. [PubMed] [Google Scholar]
- Prentice R. L. Covariate measurement errors and parametric estimation in a failure time regression model. Biometrika. 1982;69:331–342. [Google Scholar]
- Rosenwald A., Wright G., Chan W. C., Connors J. M., Campo E., Fisher R. I., Gascoyne R. D., Muller-Hermelink H. K., Smeland E. B., Staudt L. M. The use of molecular profiling to predict survival after chemotherapy for diffuse large-B-cell lymphoma. The New England Journal of Medicine. 2002;346:1937–1947. doi: 10.1056/NEJMoa012914. [DOI] [PubMed] [Google Scholar]
- Simon N., Friedman J., Hastie T., Tibshirani R. Regularization paths for Cox’s proportional hazards model via coordinate descent. Journal of Statistical Software. 2011;39(5):1–13. doi: 10.18637/jss.v039.i05. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Song X., Davidian M., Tsiatis A. A. a A semiparametric likelihood approach to joint modeling of longitudinal and time-to-event data. Biometrics. 2002a;58:742–753. doi: 10.1111/j.0006-341x.2002.00742.x. [DOI] [PubMed] [Google Scholar]
- Song X., Davidian M., Tsiatis A. A. b An estimator for the proportional hazards model with multiple longitudinal covariates measured with error. Biostatistics. 2002b;3:511–528. doi: 10.1093/biostatistics/3.4.511. [DOI] [PubMed] [Google Scholar]
- Song X., Huang Y. On corrected score approach for proportional hazards model with covariate measurement error. Lifetime Data Analysis. 2005;12:91–110. doi: 10.1111/j.1541-0420.2005.00349.x. [DOI] [PubMed] [Google Scholar]
- Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. 1996;58:267–288. [Google Scholar]
- Tsiatis A. A., Davidian M. A semiparametric estimator for the proportional hazards model with longitudinal covariates measured with error. Biometrika. 2001;88:447–458. doi: 10.1093/biostatistics/3.4.511. [DOI] [PubMed] [Google Scholar]
- Witten D. M., Tibshirani R. Survival Analysis with high-dimensional covariates. Statistical Methods in Medical Research. 2010;19:29–51. doi: 10.1177/0962280209105024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wulfsohn M. S., Tsiatis A. A. A joint model for survival and longitudinal data measured with error. Biometrics. 1997;53:330–339. [PubMed] [Google Scholar]
- Xu J., Zeger S. L. The evaluation of multiple surrogate endpoints. Biometrics. 2001;57:81–87. doi: 10.1111/j.0006-341x.2001.00081.x. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.