Biomarker Discovery for Heterogeneous Diseases

Abstract

Background

Modern genomic and proteomic studies reveal that many diseases are heterogeneous, comprising multiple different subtypes. The common notion that one biomarker can be predictive for all patients may need to be replaced by an understanding that each subtype has its own set of unique biomarkers, affecting how discovery studies are designed and analyzed.

Methods

We used Monte Carlo simulation to measure and compare the performance of eight selection methods with homogeneous and heterogeneous diseases using both single-stage and two-stage designs. We also applied the selection methods in an actual proteomic biomarker screening study of heterogeneous breast cancer cases.

Results

Different selection methods were optimal and more than 2-fold larger sample sizes were needed for heterogeneous diseases compared with homogeneous diseases. We also found that for larger studies, two-stage designs can achieve nearly the same statistical power as single-stage designs at significantly reduced cost.

Conclusions

We found that disease heterogeneity profoundly affected biomarker performance. We report sample size requirements and provide guidance on the design and analysis of biomarker discovery studies for both homogeneous and heterogeneous diseases.

Impact

We have shown that studies to identify biomarkers for the early detection of heterogeneous disease require different statistical selection methods and larger sample sizes than if the disease were homogeneous. These findings provide a methodological platform for biomarker discovery of heterogeneous diseases.

INTRODUCTION

One major and underappreciated challenge in pre-diagnostic biomarker discovery is disease heterogeneity. Modern molecular analysis methods are increasingly revealing that what were once considered monotypic diseases instead comprise multiple molecular diseases that share a common clinical presentation(1-4). Breast cancer, for example, has many known molecular subtypes including the major subtypes luminal A and B, basal-like, and Erb-B2 over-expressed(5). These subtypes affect prognosis, response to treatment and recurrence (5-10). We hypothesize that disease heterogeneity may largely explain the low sensitivity of identified autoantibody biomarkers for breast cancer(11, 12). Heterogeneous diseases may not have a single biomarker that is predictive for all patients; each subtype may have its own set of unique biomarkers. Therefore, a biomarker that is excellent for detecting a particular subtype (e.g., 98% sensitivity) might only have 20% sensitivity for the disease overall because its sensitivity is capped by the prevalence of that subtype among those with the disease(13). Disease heterogeneity introduces new challenges for the discovery of biomarkers, such as the need for greater sample sizes to ensure that relevant subtypes have adequate representation. However, the magnitude of these changes or how to compute them has not been addressed. Moreover, an important question is whether heterogeneous diseases will require different statistical selection methods for identifying biomarkers and if so, which ones? Studies that have examined methods to determine sample size requirements and the relative performance of statistical biomarker selection methods have not considered these questions in the context of heterogeneous diseases(14-21).

A typical biomarker discovery study will screen a large collection of candidate biomarkers, typically genes or proteins, using a set of known disease-positive cases and a set of known disease-negative controls. As the number of candidate biomarkers tested increases, the cost per test increases dramatically. It is often prohibitive to screen all of the available cases and controls against all possible candidate markers. To mitigate this cost, an alternate approach is to conduct screening in two stages(12, 22-24). The first stage, the pre-screen, uses a moderate number of patients and controls to screen the full collection of biomarker candidates, with a goal of eliminating from consideration candidates that show little promise as biomarkers. In the second stage, the remaining candidates are screened using the remaining patients and controls. One advantage of the two-stage design is that in the second stage, the number of biomarker candidates to be screened may be sufficiently small that more detailed studies including additional replicates of measurements may be feasible, all of which can reduce intra-assay variability and variation of measured quantities. While this two-stage screening process is intuitively a cost effective way to screen a large number of candidates when using a large number of patients and controls, few studies have examined the efficiency of this two-stage design or the optimal allocation of cases and controls across the two stages(25-28) and none have examined these issues in the context of a heterogeneous disease.

We used Monte Carlo simulation(29, 30) to examine the relative performance of statistical biomarker selection methods, determine sample size requirements and compare experimental designs One key metric was power, the probability that a true biomarker would be selected in a screening process. We compared the power of several biomarker selection methods with homogeneous and heterogeneous case populations and found dramatic differences in the optimal methods for the two populations. Further, larger sample sizes are required for the heterogeneous disease compared to the homogeneous disease. We report sample size requirements and suggested methods for both homogeneous and heterogeneous diseases. We examined the performance of selection methods in a two-stage screening process and found that for larger studies, two-stage screening can achieve nearly the same power as single-stage screening at significantly reduced cost. We illustrate the impact of using different selection methods by comparing the results from their use in a real screening study for biomarkers for the early detection of breast cancer with a heterogeneous case population.

MATERIALS AND METHODS

Single Stage Screening

We used simulation to estimate and compare the power of biomarker selection methods with homogeneous and heterogeneous case populations using a single stage screening process. We considered a study in which 10,000 candidate biomarkers are to be screened and 50 of these are true biomarkers. In both case populations true biomarkers were set to have only 20% sensitivity at 95% specificity. The choice of 20% sensitivity was based upon consideration of breast cancer, the observation that the best autoantibody biomarkers for breast cancer typically have only 20-30% sensitivity at high specificity(12), and the prevalence of several subtypes of breast cancer, including basal-like, and luminal A and B, are typically reported to be between 10% and 30% depending on the subtype and study population(31, 32). For the homogeneous disease, the biomarker performance arises from a small distributional shift between the control and case populations. For the heterogeneous disease, the performance is due to a large signal observed in a small subpopulation of cases (Figure 1a-c). Importantly, because the particular subpopulation of cases that differ from the remaining cases may depend on the individual biomarker, our heterogeneous disease model does not presuppose how many different subtypes there are for a disease. For each subject and each candidate biomarker we independently simulated a response, which typically would correspond to a normalized and transformed measure of gene expression or protein abundance. We simulated case and control responses to non-biomarkers independently from a standard normal distribution. We also simulated control responses to true biomarkers independently from a standard normal distribution.

Figure 1.

(a) Sampling distributions of normalized and transformed intensities for true biomarkers for controls (green) and cases (red) with normal samples for a homogeneous disease. (b) Sampling distribution of normalized and transformed intensities for true biomarkers for controls (green) and cases (red) with normal samples for a heterogeneous disease. (d) ROC curves for true biomarkers with normal samples for homogeneous (blue) and heterogeneous (magenta) diseases. (e) Sampling distributions of normalized and transformed intensities for true biomarkers for controls (green) and cases (red) with a t₃ homogeneous disease model. (f) Sampling distribution of normalized and transformed intensities for true biomarkers for controls (green) and cases (red) with a t₃ heterogeneous disease model. (g) ROC curves for true biomarkers in the t₃ homogeneous (blue) and t₃ heterogeneous (magenta) disease models. In panels (a), (b), (d) and (e), the dashed vertical line indicates the threshold for 20% sensitivity and 95% specificity. In panels (c) and (f), the 45 degree line (black) represents the ROC curve when there is no difference between the case and control populations.

For the homogeneous disease, we simulated case responses to true biomarkers independently from a normal distribution with mean 0.80 and standard deviation 1. See Figure 1a. Thus, in this homogeneous disease model, true biomarkers have a sensitivity of 20% at 95% specificity and the area under the receiver operator characteristic (ROC) curve (AUC) is 0.71. For the heterogeneous disease, we simulated responses for each biomarker such that only 20% of cases would have differential response (i.e., the responding subtype) and the remaining 80% of cases would resemble the controls. Specifically, with 20% probability we simulated the response for an individual case from a normal distribution with mean 2.49 and standard deviation of 1; otherwise, we simulated the response from a standard normal distribution. See Figure 1b. Under this mixture model the overall sensitivity is 20% at 95% specificity, which is the same sensitivity as for the homogeneous disease, while the AUC for comparing all cases to controls is only .59. The ROC curves for the homogeneous and heterogeneous diseases are shown in Figure 1c. Note that the heterogeneous model did not presuppose how many different subtypes there are for a disease. Instead, we assumed that each of the multiple and possibly overlapping disease subtypes, which are characterized by a differential response in a biomarker, had a prevalence of 20% in the clinical disease population.

The stochastic models for homogeneous and heterogeneous diseases are simple and transparent. While both models achieve 20% sensitivity at 95% specificity, the heterogeneous disease model uses a mixture distribution for the case responses while a simple normal distribution is used for the homogeneous disease. In other aspects the models are identical. Both models assume that the candidate biomarkers are independent. Although this independence assumption is unlikely to hold in most applications, we make this assumption here to keep the models simple and transparent and because we are focused on selection of individual biomarkers rather than construction of multi-marker panels where the correlation structure between multiple markers could play a significant role. Both models also assume independence across subjects and utilize normal distributions. In a robustness study we replaced the normal distributions in the models with heavier tailed t distributions.

We examined eight selection methods that can be organized into three groups. The first group comprises the t-tests: the ordinary one-sided two-sample t-test, the one-sided Welch’s t-test, and a one-sided empirical Bayes moderated t-test(33). In this study, the moderated t-test represents the best-case scenario for t-tests, as this version exploits the homoscedasticity of the responses across biomarker candidates. These t-tests evaluate whether the mean response for cases is larger than for controls. The second group comprises tests of stochastic dominance: the one-sided Kolmogorov-Smirnov test, the one-sided Mann-Whitney U test, and a permutation test on the AUC. These are nonparametric tests that evaluate whether case responses tend to be larger than control responses. The third group comprises a permutation test on the partial AUC (PAUC) in the region where specificity is greater than 95% and a permutation test on the sensitivity at 95% specificity. These are nonparametric tests that evaluate whether there is sufficient mass in the case response distribution in the right tail of the control response distribution. See Supplementary Data for details on these methods. Most biomarker studies use variants of t-tests, often times through linear models that can easily accommodate covariates into an analysis. The full AUC and Mann-Whitney tests are also commonly used in biomarker studies. While researchers have advocated for the use of the partial AUC and sensitivity tests based upon their focus on clinically-relevant portions of the ROC curve(19), their use in biomarker discovery studies remains limited. For all methods we used the Benjamini-Hochberg procedure to control the false discovery rate (FDR) at 10% for each method based upon their respective p-values(34). Also see Supplementary Data for results under 50% FDR control.

We ran simulations using an equal number (N) of cases and controls, with N = 25, 50, 75, 100, 150, and 200, which encompassed the number of samples commonly used in preliminary biomarker studies. We constructed twenty simulated data sets at each sample size and used the eight statistical methods on each data set to construct lists of significant candidates, which we refer to as hits. We calculated the power for each simulation by computing the proportion of the 50 true biomarkers that appear in the list of hits. We also calculated the actual false discovery rate as the proportion of hits that were not among the 50 true biomarkers.

We estimated the sample size requirements for both homogeneous and heterogeneous diseases for power values of 70%, 80%, 90%, 95%, and 98% by linearly interpolating our power estimates for the best method, excluding the empirical Bayes t-test. We calculated the fold increase in sample size requirements for heterogeneous diseases as the ratio of the number of samples required for heterogeneous diseases to the number of samples required for homogeneous diseases.

Two Stage Screening

Here we consider a common scenario in which the total number of case and control samples is fixed and researchers wish to make the most efficient use of these samples to discover meaningful biomarkers. Specifically, we consider a two-stage process where the first stage uses a subset of the available samples to identify the top 750 candidates out of the 10,000 candidates and the second stage uses the remaining samples to identify the statistically significant hits out of those top 750. Using Monte Carlo simulation, we examined the performance of biomarker selection methods with homogeneous and heterogeneous case populations using a two-stage process, and compared their performance to a single-stage screening process with the same total number of cases and control.

We first conducted a preliminary study to determine the optimal scoring algorithm (out of 8 considered) for the first stage of a two-stage screening study (see Supplementary Data). Based on the results of this study (see Figure S1), for the homogeneous disease model we used empirical Bayes moderated t-statistics when the sample size is at most 40 and the ordinary t-statistic when the sample size is greater than 40. For the heterogeneous disease model we used empirical Bayes moderated t-statistics when N=10 and PAUC otherwise.

We then used the scoring algorithms identified above in the first stage of a two-stage screening process and examined the performance of different second-stage selection methods with the normal homogeneous and heterogeneous disease models. Although greater replication may be possible in the second stage due to the screening of fewer candidates, we here ignored the possible reduced variation that would accompany replication. We fixed the number of total cases and total controls and examined the power across various allocations of those cases and controls to the two stages. With 100 cases and 100 controls, we considered (first stage) : (second stage) allocations ranging from 10:90 up to 50:50. With 200 cases and 200 controls, we considered allocations ranging from 10:190 up to 140:60. We simulated ten data sets for each allocation of 100 cases and 100 controls. With 200 cases and 200 controls, we simulated ten data sets when the number allocated to the first stage was between 30 and 100, inclusive and five data sets when the number allocated was less than 30 or greater than 100.

RESULTS

Single Stage Screening

We compared the performance of biomarker selection methods with homogeneous and heterogeneous case populations using Monte Carlo simulation. Power estimates are displayed graphically in Figure 2 for both the homogeneous and heterogeneous disease sampling models. Numerical power estimates are provided in Supplementary Tables S1 and S2. For the homogeneous disease model, the t-tests and the Mann-Whitney and AUC tests identified over 60% of the true biomarkers using only 50 cases and 50 controls. By contrast, Kolmogorov-Smirnov identified less than 40%, and PAUC and the sensitivity test identified fewer than 10% of the true biomarkers with the same sample size.

Figure 2.

Estimated power for single stage design using normal samples for a (a) homogeneous disease and a (b) heterogeneous disease. The horizontal axis indicates N, the equal number of cases and controls. The vertical axis indicates power, the proportion of the true biomarkers that are selected by each method. Standard error bars are shown.

For the heterogeneous disease model, the results were strikingly different. At least 100 cases and 100 controls were needed before any method exceeded 60% power, and only the PAUC achieved that level of performance. At that sample size, the empirical Bayes moderated t-test identified nearly 50% of the true biomarkers, the sensitivity test approximately 30%, the other t-tests (ordinary and Welch’s) approximately 20%, and the Mann-Whitney, AUC, and Kolmogorov-Smirnov tests less than 10% of the true biomarkers. In general, among the eight methods the empirical Bayes moderated t-test had the greatest power at small samples sizes (N≤50) whereas PAUC had the greatest power at larger sample sizes (N≥75).

Estimated sample size requirements to achieve a desired level of power are given in Table 1 for both homogeneous and heterogeneous diseases. For each power value, the sample size required for the heterogeneous disease is at least twice that required for the homogeneous disease.

Table 1. Sample Size Requirements for Biomarker Screening Studies.

Suggested methods and sample size requirements for desired power and type of disease. Sample size requirements are given as the equal number of cases and controls in the study and are based upon a linear interpolation of the numerical power estimates, which are provided in Supplementary Tables S1 and S2. Fold increase for heterogeneous disease is the ratio of the number of samples required for a heterogeneous disease to the number of samples required for a homogeneous disease.

Desired Power	Disease	Suggested Method	Required Sample Sizes	Fold Increase in Sample Sizes for Heterogeneous Disease
70%	Homogeneous	T-test	50	2.26
	Heterogeneous	PAUC	113
80%	Homogeneous	T-test	60	2.15
	Heterogeneous	PAUC	129
90%	Homogeneous	T-test	70	2.07
	Heterogeneous	PAUC	145
95%	Homogeneous	T-test	77	2.16
	Heterogeneous	PAUC	166
98%	Homogeneous	T-test	96	2.00
	Heterogeneous	PAUC	192

In a separate study (see Supplementary Data), we investigated the robustness of this result by replacing the normal distribution with a t distribution with 3 degrees of freedom (t₃). See Figures 1d-f. As with normal samples, the optimal statistical methods differ dramatically depending on whether heterogeneity is present and the sample sizes required under heterogeneity are much greater than under homogeneity. In fact, the difference is more dramatic with t₃ samples than with normal samples. See Tables S3 and S4, and Figure S2. Although 90% power can be achieved with 50 cases and 50 controls with homogeneous disease samples, only 5% power can be achieved with heterogeneous disease samples.

Results under 50% FDR control for both normal and t₃ samples are similar to those found under 10% FDR control. See Figures S3 and S4 for single stage power curves under 50% FDR control.

Two Stage Screening

The results for 100 cases and 100 controls are displayed in Figure 3a,b. For a homogeneous disease, a single-stage screening study achieved 98-99% power with 5 of the 8 methods (Kolmogorov-Smirnov, PAUC and the sensitivity tests excluded). The same 5 methods achieved approximately 96% power using a less costly two-stage design with 40 first-stage cases and controls and 60 second-stage cases and controls. For a heterogeneous disease, PAUC achieved nearly 62% power using 100 cases and 100 controls in a single-stage study. The greatest power achieved by any of the eight methods with a total of 100 cases and 100 controls was 51% by PAUC with an allocation of 30:70. Although a loss of 11% power is not insignificant, the single-stage study could easily cost twice that of a 30:70 two-stage study based on our own experience with Nucleic Acid Programmable Protein Arrays (NAPPA)(35). See Figure S5 for similar results under 50% FDR control.

Figure 3.

Estimated power for a two-stage design using 100 total patients and 100 total controls with normal samples for a (a) homogeneous disease and (b) heterogeneous disease. Estimated power for a two-stage design using 200 total patients and 200 total controls with normal samples for a (c) homogeneous disease and (d) heterogeneous disease. The horizontal axis indicates the allocation of the total number of patients and controls across the two stages. The vertical axis indicates power, the proportion of the true biomarkers that are selected by each method. The selection method used for the first stage is the optimal first stage method for the prescribed first stage sample size and is common across the eight methods. The dashed horizontal line indicates the estimated power of the best method in a single-stage design using the same number of total patients and controls. Standard error bars are shown.

Figure 3c,d displays the results for 200 cases and 200 controls. With a homogeneous disease, a single-stage screening study achieved 100% power with 6 of the 8 methods (PAUC and the sensitivity test excluded). The same 6 methods also achieved 100% power using a less costly two-stage design with 50 first-stage cases and controls and 150 second-stage cases and controls. For a heterogeneous disease, PAUC achieved nearly 99% power using 200 cases and 200 controls in a single-stage study. With an allocation of 60:140, PAUC identified 94.4% of the true biomarkers. See Figure S6 for similar results with 50% FDR control.

Actual Breast Cancer Screening Study

To compare the performance of the eight selection methods with a known heterogeneous case population, we analyzed published data from a breast cancer screening study(12). Specifically, 761 antigens were screened against sera from 77 controls and 102 early stage breast cancer patients using NAPPA protein microarrays(35). Normalized and log-transformed intensity data were analyzed using each of the eight selection methods. The numbers of hits with 5% FDR control ranged from 38 with Kolmogorov-Smirnov up to 78 with Welch’s t-test. Summary statistics for these hits are given in Supplementary Table S5. Not surprisingly, the t-tests and tests for stochastic dominance yielded proteins with slightly higher AUC values (e.g., median AUC of 0.63 for t-test versus 0.60 for partial AUC). However, the median sensitivity values at 95% specificity ranged from 14-16% for the t-tests and tests for stochastic dominance, while the median sensitivity improved to 21% for partial AUC and 23% for the sensitivity test. Further, we examined the best hits from across the methods. Supplementary Table S6 lists the 37 proteins that were highly significant (1% FDR control) using any of the eight methods, along with their q-values (the minimum FDR for which the protein would be significant). This list contains many proteins that have previously been identified as having important roles in breast cancer biology and pathogenesis. Some of these have low q-values across all eight methods (EIF3E, PITX1A, PDCD6IP). However, some are identified only using partial AUC or the sensitivity test (RAC3, CTBP1, BMX) and some only by t-tests and tests for stochastic dominance (ARF1).

DISCUSSION

Selecting the best analysis methods

Disease heterogeneity is a well-established phenomenon. However the statistical methodologies most often used to identify pre-diagnostic biomarkers are designed for homogeneous diseases where at most the only case heterogeneity is assumed to be explainable by measured covariates such as age and gender. We have shown here the substantial effect that disease subtype heterogeneity may have on the performance of selection methods. The implications are significant: studies to identify biomarkers for the early detection of heterogeneous diseases require different statistical selection methods and larger sample sizes than if the disease were homogeneous.

Regarding the statistical selection methods, we found that the optimal statistical methods (out of the eight considered) differed dramatically between the two disease models. For example, the PAUC was the worst of the eight for the homogeneous disease, but the best for the heterogeneous disease at larger sample sizes. Similarly, the Mann-Whitney and AUC tests were among the best methods for the homogeneous disease and among the worst for the heterogeneous disease. Only the empirical Bayes moderated t-test performed relatively well under both disease models; however, we believe that the performance is artificially inflated due to a limitation of our study in which we used a common variance across the candidate biomarkers. To test this hypothesis, we conducted a small study in which the variances of the candidate biomarkers were simulated from an inverted gamma distribution with mean 1.0 and standard deviations ranging from 0.1 up to 10. We found that while the empirical Bayes moderated t-test produces a real improvement in power when the candidate biomarkers are nearly homoscedastic, this improvement vanishes with greater heteroscedasticity. See Supplementary Data and Figures S7 and S8 for additional details and results.

The explanation for the differences in performance of these tests for the two types of disease lies in the nature of the tests themselves. In the homogeneous disease model the distribution of responses for cases is simply shifted to the right of the distribution of control responses, which results in a moderate difference in the means of the distributions. Therefore t-tests, which test for a difference in the means and tests for stochastic dominance (Kolmogorov-Smirnov, Mann-Whitney, and AUC) perform moderately well at detecting this difference between case and control responses. In the heterogeneous disease model the only difference between the distributions of cases and controls is that there is a small subset of cases with large responses. While these aberrant cases do induce stochastic dominance and do increase the mean of the entire case population, the difference from the control population is less pronounced than in the homogeneous disease model. What better differentiates cases and controls in the heterogeneous disease model are the tails of the distributions. It is therefore not surprising that methods that specifically focus on differences in the tails, such as the partial AUC test and the sensitivity test, outperform tests that focus on differences in means in the heterogeneous disease model.

Although the partial AUC and sensitivity test performed best with a heterogeneous disease, neither of these methods was designed specifically for heterogeneous case populations. It is likely that other methods such as mixture modeling and cluster analysis will perform better with heterogeneous diseases in certain circumstances. Our intent in this study was not to find the optimal method, which will depend on many factors including the distributions of case and control responses; instead we examined the relative performance of methods for homogeneous and heterogeneous diseases in order to highlight the critical role that disease heterogeneity plays in the analysis of biomarker screening studies.

Another valid strategy for biomarker selection is to use multiple selection criteria. The application of the 8 methods on the breast cancer screening data yielded different sets of proteins. RAC3, a protein with an important role in breast cancer biology, was highly significant according to PAUC, but not according to any of the t-tests or the full AUC. Similarly, ARF1 was highly significant according to all tests, except for PAUC and the sensitivity test. The results suggest that the signals from a biomarker, even in a heterogeneous case population, may or may not appear to be heterogeneous, and therefore different testing strategies may be necessary to identify different types of biomarkers.

We have also shown that much larger sample sizes are required when heterogeneity is present. Whereas over 70% power can be achieved using a sample size of 50 cases and 50 controls for the homogeneous disease, only 15% power can be achieved using the same sample sizes for the heterogeneous disease. Further, we showed that at least twice the number of samples is needed for a heterogeneous disease compared to a homogeneous disease in order to obtain the same power. If a power analysis is based upon an assumption that the underlying disease is homogeneous, the study may be severely underpowered if the disease is actually heterogeneous.

Optimizing two stage design

A two-stage design may reduce the cost of screening studies as fewer candidates need to be screened in the second stage. We have shown that for moderately-sized studies, there is some loss in power from the use of a two-stage design, e.g. 62% to 51% using PAUC with 100 cases and 100 controls, but that loss may be justifiable due to the potential cost savings. For larger studies the power of the two-stage design can approach that of the single stage design. We therefore recommend that a two-stage design be considered for moderate to large biomarker discovery studies, where cost is an issue.

Recommendations

We have demonstrated that disease heterogeneity and sample sizes impact both the design and analysis techniques of biomarker screening studies. We strongly recommend that a formal power analysis be conducted for any planned biomarker screening study that accounts for the suspected amount of disease heterogeneity, sample availability, magnitude of differences between cases and controls, sample variability, false discovery rate tolerable and analysis methods. Further, a cost-benefit analysis could be conducted to examine the tradeoffs between using single and two-stage designs. In lieu of such formal analyses, we make the following recommendations for the design and analysis of biomarker screening studies (see Figure 4). For homogeneous case populations, for smaller sample sizes (less than 75 cases or 75 controls) a single stage design should be used, and investigators should seriously consider a two-stage design for larger sample sizes. For heterogeneous case populations, a single stage design should be used if there are fewer than 100 cases or 100 controls, and serious consideration should be given to a two-stage design for larger sample sizes. As for selection methods, the t-tests, Mann-Whitney U test, and AUC test are all viable tests for homogeneous case populations, and analysts should consider both normality of data and inclusion of covariate information when deciding between these approaches. For heterogeneous case populations, we recommend the use of the PAUC or sensitivity test, except for very small studies (fewer than 25 cases or 25 controls). In such small studies with heterogeneous case populations, these nonparametric tests will lack sufficient power. Instead, the best that one can accomplish may be to identify biomarkers that have a similar effect across the disease sub-types. Therefore, for these very small studies, we recommend the use of t-tests, Mann-Whitney U test, or the AUC test.

Figure 4.

Informal recommendations for the design and analysis of biomarker screening studies in lieu of formal power and cost-benefit analyses. The sample size (N) is the minimum of the number of cases and number of controls available for screening. In the diagram, ‘T’ refers to t-tests and many other linear modeling approaches, ‘M-W U’ is the Mann-Whitney U test, ‘AUC’ is a test for a large area under the ROC curve, ‘PAUC’ is a test for a large partial area under the ROC curve, and ‘Sensitivity’ is a test for high sensitivity at a given level of specificity.

The results of this study and these recommendations are based upon a Monte Carlo analysis of biomarker screening and are therefore dependent on the underlying stochastic model. Here we chose a simple mixture model to introduce disease heterogeneity in an intuitive and transparent fashion. When planning a biomarker screening study, we recommend that a formal power analysis be conducted with careful consideration given to modeling assumptions. In particular, our recommendations are based on simulating biomarker responses using normal distributions. If after normalization and transformation data do not follow normal distributions, sample size requirements may be greater. We have shown that replacing normal distributions with heavier-tailed t₃ distributions leads to substantial changes in both power estimates and preferred selection methods. If data from a pilot study are available, bootstrap methods(36) can be used for power analyses of screening studies without requiring such distributional assumptions.

This study has shown that disease heterogeneity plays a significant role in the statistical selection of biomarkers. Disease heterogeneity may similarly impact drug discovery studies and drug trials. Examples abound of drugs for which only a subset of patients respond: trastuzumab (Herceptin) in HER2-overexpressed breast cancer, gefitinib (Iressa) for EGFR-overexpressed lung cancer, and cetuximab (Erbitux) for EGFR-overexpressed colorectal cancer. Accounting for disease heterogeneity in the design and analysis of drug studies may improve the efficiency and success rate of these studies.