On the Bias of Precision Estimation Under Separate Sampling

Abstract

Observational case-control studies for biomarker discovery in cancer studies often collect data that are sampled separately from the case and control populations. We present an analysis of the bias in the estimation of the precision of classifiers designed on separately sampled data. The analysis consists of both theoretical and numerical results, which show that classifier precision estimates can display strong bias under separating sampling, with the bias magnitude depending on the difference between the true case prevalence in the population and the sample prevalence in the data. We show that this bias is systematic in the sense that it cannot be reduced by increasing sample size. If information about the true case prevalence is available from public health records, then a modified precision estimator that uses the known prevalence displays smaller bias, which can in fact be reduced to zero as sample size increases under regularity conditions on the classification algorithm. The accuracy of the theoretical analysis and the performance of the precision estimators under separate sampling are confirmed by numerical experiments using synthetic and real data from published observational case-control studies. The results with real data confirmed that under separately sampled data, the usual estimator produces larger, ie, more optimistic, precision estimates than the estimator using the true prevalence value.

Introduction

Biomarker discovery is typically attempted by means of observational case-control studies where classification techniques are applied to high-throughput measurement technologies, such as DNA microarrays,^1,2 next-generation RNA sequencing (RNA-seq),³ or “shotgun” mass spectrometry.⁴ The validity and reproducibility of the results depend critically on the availability of accurate and unbiased assessment of classification accuracy.^5,6

The vast majority of published methods in the statistical learning literature make the assumption, explicitly or implicitly, that the data for training and accuracy assessment are sampled randomly, or unrestrictedly, from the mixture of the populations. However, observational case-control studies in biomedicine typically proceed by collecting data that are sampled with restrictions. The most common restriction, and the one that is studied in this article, is that the data are sampled separately from the case and control populations. That creates an important issue in the application of traditional statistical learning techniques to biomedical data, because there is no meaningful estimator of case prevalences under separate sampling. Therefore, any methodology that directly or indirectly uses estimates of case prevalence could be severely biased.

Precision and Recall have become very popular classification accuracy metrics in the statistical learning literature.^7–9 The recall does not depend on the prevalence, while the precision does. Therefore, we investigate in this article the bias of the precision estimator when the typical separate sampling design used in case-control studies is not properly taken into account.

A similar study was conducted previously into the accuracy of cross-validation under separate sampling.¹⁰ It was shown in that study that the usual “unbiasedness” property of k-fold cross-validation does not hold under separate sampling. In fact, the bias can in fact be substantial and systematic, ie, not reducible under increasing sample size. In Braga-Neto et al,¹⁰ modified k-fold cross-validation estimators were proposed for the class-specific error rates. In the case where the true case prevalence is known, those estimators can be combined into an estimator of the overall error rate, which satisfies the usual “unbiasedness” property of cross-validation.

By contrast, the present paper employs analytical and numerical methods to investigate precision estimation under separate sampling. We show that the usual precision estimator is asymptotically unbiased as sample size increases, under the condition that the classification rule has a finite Vapnik-Chervonenkis (VC) dimension. However, under separate sampling, we show that the usual precision estimator will in general display a systematic bias, which cannot be reduced by increasing sample size, if the observed prevalence of cases in the data is different from the true prevalence in the population of interest, and the bias is larger the more different they are. In particular, the bias tends to be large when the true prevalence is small but the training data contain an equal number of examples from both classes, which is a common scenario in practice. If the true case prevalence is known (eg, from public health records), then a modified precision estimator that uses the known prevalence is shown to be asymptotically unbiased in the separate sampling case, under the condition that the classification rule is sufficiently stable as sample size increases. All of these theoretical results, and the approximations used to derive them, are verified by numerical experiments using both synthetic and real data from published studies.

Materials and Methods

In this section, we define and study the various error rates of interest in this study, including precision and recall.

Population performance metrics

The feature vector $X\in R^d$ summarizes numerical characteristics of a patient (eg, blood concentrations of given proteins). The label $Y\in \{0,1\}$ is defined as $Y=0$ if the patient is from the control population, and $Y=1$ if the patient is from the case population.

The prevalence is defined by

$$\mathrm{prev}=P(Y=1)$$ (1)

ie, the probability that a randomly selected individual is a case subject. The prevalence plays a fundamental role in the sequel.

A classifier $\psi :R^d\to \{0,1\}$ assigns $\mathbf{X}$ to the control or case population, according to whether $\psi (\mathbf{X})=0$ or $\psi (\mathbf{X})=1$, respectively. The classification sensitivity and specificity are defined as follows:

$$\mathrm{sens}=P(\psi (\mathbf{X})=1|Y=1)$$ (2)

$$\mathrm{spec}=P(\psi (\mathbf{X})=0|Y=0)$$ (3)

The closer both are to 1, the more accurate the classifier is. A noteworthy property of the sensitivity and specificity is that they do not depend on the prevalence.

Other common performance metrics for a classifier are the false-positive (FP), false-negative (FN), true-positive (FP), and true-negative (FN) rates, given by

$$\mathrm{FP}=P(\psi (\mathbf{X})=1,Y=0)$$ (4)

$$=(1-\mathrm{spec})\times (1-\mathrm{prev})$$ (5)

$$\mathrm{FN}=P(\psi (\mathbf{X})=0,Y=1)=(1-\mathrm{sens})\times \mathrm{prev}$$ (6)

$$\mathrm{TP}=P(\psi (\mathbf{X})=1,Y=1)=\mathrm{sens}\times \mathrm{prev}$$ (7)

$$\mathrm{TN}=P(\psi (\mathbf{X})=0,Y=0)=\mathrm{spec}\times \mathrm{(1}-\mathrm{prev)}$$ (8)

Unlike sensitivity and specificity, the previous performance metrics do depend on the prevalence.

Note that

$$\mathrm{prev}=\mathrm{FN}+\mathrm{TP},1-\mathrm{prev}=\mathrm{FP}+\mathrm{TN}$$ (9)

$$\mathrm{sens}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\mathrm{,spec}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$$ (10)

Finally, we define the precision and recall accuracy metrics. Precision measures the likelihood that one has a true case given that the classifier outputs a case:

$$\mathrm{prec}=P(Y=1|\psi (\mathbf{X})=1)$$ (11)

Applying Bayes’ Theorem and using previously derived relationships reveal that

$$\mathrm{prec}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}=\frac{\mathrm{sens}\times \mathrm{prev}}{\mathrm{sens}\times \mathrm{prev}+(1-\mathrm{spec})\times (1-\mathrm{prev})}$$ (12)

On the other hand, recall is simply the sensitivity:

$$\mathrm{rec}=\mathrm{sens}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$ (13)

It follows that precision depends on the prevalence, but recall does not.

Estimated performance metrics

In practice, the performance metrics defined in the previous section need to be estimated from sample data $S_n=\{({\mathbf{X}}_1,Y_1),\dots ,{\mathbf{X}}_n,Y_n)\}$. Let $\widehat{P}$ denote the empirical probability measure defined by $S_n$. The estimator of prevalence is

$$\widehat{\mathrm{prev}}=\widehat{P}(Y=1)=\frac{1}{n}\sum _{i=1}^n{I}_{Y_i=1}$$ (14)

where $I_A=1$ if $A$ is true and $I_A=0$ if $A$ is false. Similarly,

$$\widehat{\mathrm{FP}}=\widehat{P}(\psi (\mathbf{X})=1,Y=0)=\frac{1}{n}\sum _{i=1}^n{I}_{\{\psi ({\mathbf{X}}_i)=1,Y_i=0\}}$$ (15)

$$\widehat{\mathrm{FN}}=\widehat{P}(\psi (\mathbf{X})=0,Y=1)=\frac{1}{n}\sum _{i=1}^n{I}_{\{\psi ({\mathbf{X}}_i)=0,Y_i=1\}}$$ (16)

$$\widehat{\mathrm{TP}}=\widehat{P}(\psi (\mathbf{X})=1,Y=1)=\frac{1}{n}\sum _{i=1}^n{I}_{\{\psi ({\mathbf{X}}_i)=1,Y_i=1\}}$$ (17)

$$\widehat{\mathrm{TN}}=\widehat{P}(\psi (\mathbf{X})=0,Y=0)=\frac{1}{n}\sum _{i=1}^n{I}_{\{\psi ({\mathbf{X}}_i)=0,Y_i=0\}}$$ (18)

The remaining performance metrics estimators are defined analogously, using equations (10), (12), and (13):

$$\widehat{\mathrm{spec}}=\frac{\widehat{\mathrm{TN}}}{\widehat{\mathrm{TN}}+\widehat{\mathrm{FP}}}=\frac{{\sum }_{i=1}^n{I}_{\{\psi ({\mathbf{X}}_i)=0,Y_i=0\}}}{{\sum }_{i=1}^n{I}_{Y_i=0}}$$ (19)

$$\widehat{\mathrm{prec}}=\frac{\widehat{\mathrm{TP}}}{\widehat{\mathrm{TP}}+\widehat{\mathrm{FP}}}=\frac{{\sum }_{i=1}^n{I}_{\{\psi ({\mathbf{X}}_i)=1,Y_i=1\}}}{{\sum }_{i=1}^n{I}_{\psi ({\mathbf{X}}_i)=1}}$$ (20)

$$\widehat{\mathrm{rec}}=\widehat{\mathrm{sens}}=\frac{\widehat{\mathrm{TP}}}{\widehat{\mathrm{TP}}+\widehat{\mathrm{FN}}}=\frac{{\sum }_{i=1}^n{I}_{\{\psi ({\mathbf{X}}_i)=1,Y_i=1\}}}{{\sum }_{i=1}^n{I}_{Y_i=1}}$$ (21)

Mixture and separate sampling

The usual scenario in Statistical Learning is to assume that $S_n=\{({\mathbf{X}}_1,Y_1),\dots ,({\mathbf{X}}_n,Y_n)\}$ is an independent and identically distributed (i.i.d.) sample from the true distribution of the pair $(\mathbf{X},Y)$. That makes $S_n$ a sample from the mixture of populations, where each label $Y_i$ is distributed as

$$\begin{array}{c}P(Y_i=1)=\mathrm{prev}\\ P(Y_i=0)=1-\mathrm{prev}\end{array}$$ (22)

for $i=1,\dots ,n$. Under mixture sampling, $N_0={\sum }_{i=0}^n{I}_{Y_i=0}$ and $N_1={\sum }_{i=1}^n{I}_{Y_i=1}=n-N_0$ are binomial random variables, with parameters $(n,1-\mathrm{prev})$ and $(n,\mathrm{prev})$, respectively.

By contrast, observational case-control studies in biomedicine typically proceed by collecting data from the populations separately, where the separate sample sizes $n_0$ and $n_1$, with $n_0+n_1=n$, are pre-determined and nonrandom, ie, sampling occurs with the restriction $N_0={\sum }_{i=0}^n{I}_{Y_i=0}=n_0$ (or, equivalently, $N_1={\sum }_{i=1}^n{I}_{Y_i=1}=n_1$). Therefore, all probabilities and expectations over the sample are conditional on $N_0=n_0$. The restriction means that the labels $Y_1,\dots ,Y_n$ are no longer independent, even though the feature vectors ${\mathbf{X}}_1,\dots ,{\mathbf{X}}_n$ are still independent given the labels. In fact, under separate sampling, only the order of the labels $Y_1,\dots ,Y_n$ may be random. Thus, $f(Y_1,\dots ,Y_n|N_0=n_0)$ is a discrete uniform distribution over all $\left(\begin{array}{c}n\\ n_0\end{array}\right)$ possible orderings. This can also be obtained by direct computation, as follows:

$$\begin{array}{l}f(Y_1,\dots ,Y_n|N_0=n_0)=\frac{f(Y_1,\dots ,Y_n,N_0=n_0)}{P(N_0=n_0)}\hfill \\ =(\begin{array}{ll}\frac{{\mathrm{prev}}^{n_1}{(1-\mathrm{prev})}^{n_0}}{\left(\begin{array}{c}n\\ n_0\end{array}\right){\mathrm{prev}}^{n_1}{(1-\mathrm{prev})}^{n_0}}=\frac{1}{\left(\begin{array}{c}n\\ n_0\end{array}\right)},\hfill & \mathrm{if}\sum _{i=1}^n{I}_{Y_i=0}=n_0\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\hfill \end{array}$$ (23)

It is not difficult to verify that under equation (23), the marginal distribution of each label $Y_i$ is given by

$$\begin{array}{l}P(Y_i=1|N_0=n_0)=\frac{n_1}{n}\stackrel{\Delta }{=}r\hfill \\ P(Y_i=0|N_0=n_0)=\frac{n_0}{n}=1-r\hfill \end{array}$$ (24)

for $i=1,\dots ,n$, where $r$ is the (fixed) sample size ratio under separate sampling. Comparing equations (22) and (24) reveals the main difference between mixture and separate sampling.

Bias of the precision estimator

In this section, we present a theoretical large sample analysis of the bias of the estimators discussed previously, focusing on the precision estimator. Estimation bias is defined as the expectation over the sample data $S_n$ of the difference between the estimated and true quantities.

The situation is clear with the estimator of the prevalence itself, given by equation (14). Under mixture sampling, we have

$$E[\widehat{\mathrm{prev}}]=\frac{1}{n}\sum _{i=1}^{n}E[I_{Y_i=1}]=P(Y_1=1)=\mathrm{prev}$$ (25)

so the estimator is unbiased (in addition, as $n$ increases, $\mathrm{Var}(\widehat{\mathrm{prev}})\to 0$ and $\widehat{\mathrm{prev}}\to \mathrm{prev}$ in probability, by the law of large numbers). However, under separate sampling,

$$\begin{array}{l}E[\widehat{\mathrm{prev}}|N_0=n_0]=\frac{1}{n}\sum _{i=1}^{n}E[I_{Y_i=1}|N_0=n_0]\hfill \\ =P(Y_1=1|N_0=n_0)=r\hfill \end{array}$$ (26)

according to equation (24). This also follows directly from the fact that $\widehat{\mathrm{prev}}$ becomes a constant estimator, $\widehat{\mathrm{prev}}\equiv r$, according to equation (14). Thus,

$$\begin{array}{l}{\mathrm{Bias}}_{\mathrm{sep}}(\widehat{\mathrm{prev}})=E[\widehat{\mathrm{prev}}-\mathrm{prev}|N_0=n_0]\hfill \\ =r-\mathrm{prev}\hfill \end{array}$$ (27)

Assuming that the sample size ratio $r=n_1/n$ is held constant as $n$ increases (eg, under the common balanced design case, $n_0=n_1=n/2$), then this bias cannot be reduced with increased sample size. Furthermore, the bias is larger the further away $\mathrm{prev}$ is from $r$. In particular, the bias tends to be large when $\mathrm{prev}$ is small and $r=1/2$, which is a common scenario in practice.

The situation for $\widehat{\mathrm{FP}}$, $\widehat{\mathrm{FN}}$, $\widehat{\mathrm{FP}}$, and $\widehat{\mathrm{TN}}$ is more complicated. First, we are interested in a classifier ${\psi }_n$ derived by a classification rule from the sample data $S_n=\{({\mathbf{X}}_1,Y_1),\dots ,{\mathbf{X}}_n,Y_n)\}$. Therefore, all expectations and probabilities in the previous sections are conditional on $S_n$. Under mixture sampling, the powerful Vapnik-Chervonenkis Theorem can be applied to show that all of these estimators are asymptotically unbiased, provided that classification rule has a finite VC Dimension.¹¹ This includes many useful classification algorithms such as Linear Discriminant Analysis (LDA), linear Support Vector Machines (SVMs), perceptrons, polynomial-kernel classifiers, certain decision trees, and neural networks, but it excludes nearest-neighbor classifiers, for example. Classification rules with finite VC dimension do not cut the feature space in complex ways and are thus generally robust against overfitting.

Assuming mixture sampling and a classification algorithm with finite VC dimension $V_{\mathcal{C}}$, it can be shown that (the details are omitted; see Braga-Neto and Dougherty⁶ for a similar argument)

$${\mathrm{Bias}}_{\mathrm{mix}}(\widehat{\mathrm{FP}})\le 8\sqrt{\frac{V_{\mathcal{C}}\log (n+1)+4}{2n}}$$ (28)

so that the bias vanishes as $n\to \infty$. Similar inequalities apply to $\widehat{\mathrm{FN}}$, $\widehat{\mathrm{FP}}$, and $\widehat{\mathrm{TN}}$. These are distribution-free results; hence, vanishingly small bias is guaranteed if $n\gg V_{\mathcal{C}}$, regardless of the feature-label distribution. For linear classification rules, $V_{\mathcal{C}}=d+1$, where $d$ is the dimensionality of the feature vector. In this case, the $\widehat{\mathrm{FP}}$, $\widehat{\mathrm{FN}}$, $\widehat{\mathrm{FP}}$, and $\widehat{\mathrm{TN}}$ estimators are essentially unbiased if $n\gg d$.

Next we consider the bias of the precision and recall estimators under mixture sampling (the analysis for the sensitivity and specificity estimators is similar; in fact, the former is just the recall estimator). We will make use of the following approximation for the expectation of a ratio of two random variables $W$ and $Z$ (see Appendix 1 for the derivation of this approximation and the conditions under which it is valid):

$$E\left[\frac{W}{Z}\right]\approx \frac{E[W]}{E[Z]}$$ (29)

The approximation is quite accurate if $W$ and $Z$ are around $E[W]$ and $E[Z]$, respectively (it is asymptotically exact as $W\to E[W]$ and $Z\to E[Z]$). For the precision estimator,

$$\begin{array}{l}E[\widehat{\mathrm{prec}}]=E\left[\frac{\widehat{\mathrm{TP}}}{\widehat{\mathrm{TP}}+\widehat{\mathrm{FP}}}\right]\approx \frac{E[\widehat{\mathrm{T}P}]}{E[\widehat{\mathrm{TP}}+\widehat{\mathrm{FP}}]}\hfill \\ \approx \frac{E[\mathrm{TP}]}{E[\mathrm{TP}+\mathrm{FP}]}\approx E\left[\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}\right]=E[\mathrm{prec}]\hfill \end{array}$$ (30)

for a sufficiently large sample, where we used the previously established asymptotic unbiasedness of $\widehat{\mathrm{TP}}$, $\widehat{\mathrm{TP}}$, and $\widehat{\mathrm{FN}}$. An entirely similar derivation shows that $E[\widehat{\mathrm{rec}}]=E[\mathrm{rec}]$. Hence, for “well-behaved” classification algorithms (those with finite VC dimension), both the precision and recall estimators are asymptotically unbiased under mixture sampling.

We are not aware of the existence of a VC theory for separate sampling at this time. To obtain approximate results for the separate sampling case, we will assume instead that at large enough sample sizes, the classifier $\psi$ is nearly constant, and invariant to the sample. This assumption is not unrelated to the finite VC dimension assumption made in the case of mixture sampling. Many of the same classification algorithms that have finite VC dimension, such as LDA and linear SVMs, will also become nearly constant as sample size increases. In this case, we have

$$\begin{array}{l}E[\widehat{\mathrm{TP}}|N_0=n_0]=\frac{1}{n}\sum _{i=1}^{n}E[I_{\{\psi (X_i)=1,Y_i=1\}}|N_0=n_0]\hfill \\ =P(\psi ({\mathbf{X}}_1)=1,Y_1=1|N_0=n_0)\hfill \\ =P(\psi ({\mathbf{X}}_1)=1|Y_1=1)P(Y_1=1|N_0=n_0)\hfill \\ =\mathrm{sens}\times r\hfill \end{array}$$ (31)

where we used the fact that the event $\{\psi ({\mathbf{X}}_1)=1\}$ is independent of $N_0$ given $Y_1$ and equation (24). Note that the equality $P(\psi ({\mathbf{X}}_1)=1|Y_1=1)=\mathrm{sens}$ depends on the fact that $\psi$ is assumed to be constant, so that $({\mathbf{X}}_1,Y_1)$ behaves as an independent test point (also because of a constant $\psi$, there is no expectation around $\mathrm{sens}$). Hence, $\widehat{\mathrm{TP}}$ is biased under separate sampling, with

$${\mathrm{Bias}}_{\mathrm{sep}}(\widehat{\mathrm{TP}})=\mathrm{sens}\times r-\mathrm{TP}=\mathrm{sens}\times \left(r-\mathrm{prev}\right)$$ (32)

As in the case with the bias of $\widehat{\mathrm{prev}}$ under separate sampling, the bias of $\widehat{\mathrm{TP}}$ cannot be reduced with increasing sample size. The bias is in fact larger the more sensitive the classifier is. One can derive similar results for $\widehat{\mathrm{FP}}$, $\widehat{\mathrm{FN}}$, and $\widehat{\mathrm{TN}}$.

The recall estimator is approximately unbiased under separate sampling:

$$\begin{array}{l}E[\widehat{\mathrm{rec}}|N_0=n_0]=E\left[\frac{\widehat{\mathrm{TN}}}{\widehat{\mathrm{TN}}+\widehat{\mathrm{FP}}}|N_0=n_0\right]\hfill \\ =E\left[\frac{\widehat{\mathrm{TP}}}{\widehat{\mathrm{prev}}}|N_0=n_0\right]=\frac{E[\widehat{\mathrm{TP}}|N_0=n_0]}{r}\hfill \\ =\frac{\mathrm{sens}\times r}{r}=\mathrm{sens}=\mathrm{rec}\hfill \end{array}$$ (33)

This is a consequence of recall’s not being a function of the prevalence. However, for the precision estimator,

$$\begin{array}{l}E[\widehat{\mathrm{prec}}|N_0=n_0]=E\left[\frac{\widehat{\mathrm{TP}}}{\widehat{\mathrm{TP}}+\widehat{\mathrm{FP}}}|N_0=n_0\right]\hfill \\ \approx \frac{E[\widehat{\mathrm{TP}}|N_0=n_0]}{E[\widehat{\mathrm{TP}}+\widehat{\mathrm{FP}}|N_0=n_0]}\hfill \\ =\frac{\mathrm{sens}\times r}{\mathrm{sens}\times r+(1-\mathrm{spec})\times (1-r)}\hfill \\ \ne \frac{\mathrm{sens}\times \mathrm{prev}}{\mathrm{sens}\times \mathrm{prev}+(1-\mathrm{spec})\times (1-\mathrm{prev})}=\mathrm{prec}\hfill \end{array}$$ (34)

The precision estimator is thus biased under separate sampling unless the true prevalence matches exactly the sample ratio $r=n_1/n$; the bias is larger the further away $\mathrm{prev}$ is from $r$.

In case the true prevalence is known, eg, from public health records and government databases, then we show below that the following estimator of the precision,

$$\begin{array}{l}{\widehat{\mathrm{prec}}}^{\mathrm{prev}}=\frac{\widehat{\mathrm{sens}}\times \mathrm{prev}}{\widehat{\mathrm{sens}}\times \mathrm{prev}+(1-\widehat{\mathrm{spec}})\times (1-\mathrm{prev})}\hfill \end{array}$$ (35)

which is based on equation (12), is an asymptotically unbiased estimator of the precision under either mixture or separate sampling. Asymptotic unbiasedness in the mixture sampling case can be shown by repeating the steps in the analysis of the ordinary precision estimator. Under separate sampling, we have

$$\begin{array}{l}E[{\widehat{\mathrm{prec}}}^{\mathrm{prev}}|N_0=n_0]\hfill \\ \approx \frac{E[\widehat{\mathrm{sens}}|N_0]\times \mathrm{prev}}{E[\widehat{\mathrm{sens}}|N_0]\times \mathrm{prev}+(1-E[\widehat{\mathrm{spec}}|N_0])\times (1-\mathrm{prev})}\hfill \\ =\frac{\mathrm{sens}\times \mathrm{prev}}{\mathrm{sens}\times \mathrm{prev}+(1-\mathrm{spec})\times (1-\mathrm{prev})}=\mathrm{prec}\hfill \end{array}$$ (36)

since $E[\widehat{\mathrm{sens}}|N_0=n_0]=\mathrm{sens}$ and $E[\widehat{\mathrm{spec}}|N_0=n_0]=\mathrm{spec}$, as can be easily shown. Hence, ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ is an asymptotically unbiased estimator of the precision under either mixture or separate sampling. The ordinary precision estimator $\widehat{\mathrm{prec}}$ should not be used under separate sampling, or large and irreducible bias may occur. On the other hand, in the impossibility of obtaining information on the true prevalence value, then no meaningful estimator of the precision is possible.

Results and Discussion

In this section, we employ synthetic and real-world data to investigate the accuracy of the analysis in the previous section and the performance of the precision estimator under separate sampling. Corresponding results for mixture sampling and the recall estimator can be found in the Supplementary Material.

Experiments with synthetic data

We performed a set of experiments employing synthetic data from a homoskedastic Gaussian model, consisting of three-dimensional class-conditional distributions $N({\mathit{\mu }}_i,\Sigma )$, for $i=0,1$, with $\mathit{\mu }0=(0,0,0)$, ${\mathit{\mu }}_1=(0,0,\theta )$, where $\theta >0$ is a parameter governing the separation between the classes, and $\Sigma =\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,{\sigma }_3^2)$ (ie, a matrix with ${\sigma }_1^2,{\sigma }_2^2,{\sigma }_3^2$ on the diagonal and zeros off the diagonal). We consider two sample sizes, $n=30$ and $n=200$, so that we can compare the results for small and large sample sizes. All experiments with separate sampling are performed with sample size ratio $r=n_1/n\in [0.1,0.9]$. The synthetic data parameters are summarized in Table 1.

Table 1.

Synthetic data parameters.

Parameter	Value
Dimensionality/feature size	$D=3$
Mean difference	$\theta =2$
Covariance matrix	${\sigma }_1^2=0.5,{\sigma }_2^2=0.5,{\sigma }_3^2=1$
Sample size	$n=30,200$
Sample size ratio $r$	$r=0.1,0.3,0.5,0.7,0.9$
True prevalence	$\mathrm{prev}=0.1,0.3,0.5,0.7,0.9$

For each value of $r$ and $\mathrm{prev}$, we repeat the following process 1000 times and average the results to estimate expected values:

1. Generate sample data $S_n$ of size $n$ according to $r$ (separate sampling) or $prev$ (mixture sampling);

2. Train a classifier using one of three classification rules:¹² LDA, 3-Nearest Neighbors (3NN), and a nonlinear Radial-Basis Function Support Vector Machine (RBF-SVM).

3. Obtain recall and precision estimates. Compute both the usual precision estimate $\widehat{\mathrm{prec}}$ and the modified precision estimate ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$.

4. Obtain accurate estimates of the true precision values using a test set of size 10 000.

Figure 1 displays the results of the experiment. Note that there is only one curve for the traditional precision estimator $\widehat{\mathrm{prec}}$ because it does not employ the actual value of $\mathrm{prev}$. The values of $\widehat{\mathrm{prec}}$ and ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ coincide when $\mathrm{prev}=r$, as expected. However, as the values of $\mathrm{prev}$ and $r$ become different, their values become quite different, and ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ displays much less bias, ie, it tracks the true precision much more closely, than $\widehat{\mathrm{prec}}$. At the small sample size $n=30$, both estimators display bias, which is however much larger overall for $\widehat{\mathrm{prec}}$ than for ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$. At the large sample size $n=200$, the bias of ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ nearly disappears for LDA and is reduced for the other classification rules. We note that among these classification rules, LDA is the only one with a finite VC dimension; the fact that the bias in this case shrinks to zero as sample size increases confirms the results of the theoretical analysis in the previous section (convergence is quite fast, and quite evident at $n=200$, due to the fact that the synthetic data are homoskedastic Gaussian). Note also that the bias of $\widehat{\mathrm{prec}}$ cannot be reduced by increasing sample size, which is also in agreement with the theoretical analysis (and so are the results in the Supplementary Material).

Figure 1.

Average true precision (solid curves), average usual precision estimate $\widehat{\mathrm{prec}}$ (dash-diamond curves), and average modified precision estimate ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ (dashed curves), for LDA, 3NN, and RBF-SVM, with sample sizes $n=30$ and $n=200$, and different prevalence values, as a function of the sample size ratio. LDA indicates Linear Discriminant Analysis; 3NN, 3-Nearest Neighbors; RBF-SVM, Radial-Basis Function Support Vector Machine.

To examine more closely the effect of the difference between $\mathrm{prev}$ and $r$ on precision estimation, Figure 2 plots bias estimates for $\widehat{\mathrm{prec}}$ and ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ as a function of the absolute difference between $\mathrm{prev}$ and $r$, using the same data employed in Figure 1. It can be seen that the bias is always positive, indicating optimistic precision estimates. In nearly all cases, ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ has a smaller bias than $\widehat{\mathrm{prec}}$, and when $\mathrm{prev}$ is far from $r$, the difference in bias becomes quite large.

Figure 2.

Estimated bias of the usual precision estimator $\widehat{\mathrm{prec}}$ (dotted curves), and the modified precision estimator ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ (dashed curves) for LDA, 3NN, and RBF-SVM, with sample sizes $n=30$ and $n=200$, and different prevalence values, as a function of the absolute difference between true prevalence and sample size ratio. LDA indicates Linear Discriminant Analysis; 3NN, 3-Nearest Neighbors; RBF-SVM, Radial-Basis Function Support Vector Machine.

Case studies with real data

Here we further investigate the bias of precision estimation under separate sampling using real data from three published studies.

Leukemia study

This publication¹³ used a tumor microarray data set containing two types of human acute leukemia: acute myeloid leukemia (AML) and acute lymphoblastic leukemia (ALL). Gene expression measurements were taken from $15154$ genes from 72 tissue specimens, 47 of which of ALL type (class 0), and 25 of AML type (class 1), so that $r=0.347$. The estimator ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ was computed using the value $\mathrm{prev}=0.222$, which is the incidence rate of ALL over AML in the US population.¹⁴

Breast cancer study

The second publication¹⁵ employed the Wisconsin Breast Cancer (Original) Dataset from the University of California-Irvine (UCI) Machine Learning Repository,^16,17 which has been used by several groups to investigate breast cancer classification methods.^18,19 The data set consists of 699 instances, 458 and 241 of which are from benign and malignant tumors, respectively, and 10 features corresponding to cytological characteristics of breast fine-needle aspirates. According to Wilkins,²⁰ fewer than 20% of breast lumps are malignant; therefore, we used used $\mathrm{prev}=0.2$ in the computation of the modified precision estimator ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$.

Liver disease study

The final publication²¹ employed a liver disease data set, also from the UCI Machine Learning Repository. This data set contains 5 blood test attributes and 345 records, of which 145 belong to individuals with liver disease (class 0) and 200 measurements are taken from healthy individuals (class 1), so that $r=0.42$. This data set was donated to UCI in 1990, when the prevalence rate for chronic liver disease in the United States was $\mathrm{prev}=0.1178$,²² which we use as the prevalence in the computation of the ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ estimator.

All three studies used libraries from the Weka machine learning environment²³ to compute usual precision estimates on separately sampled data, while ignoring true prevalences, for different classification rules: Naive Bayes (NB),²⁴ C4.5 decision tree,²⁵ Back-Propagated Neural Networks, 3NN, and Linear SVM.¹² We reproduced the analysis in all three papers using Weka, obtaining almost exactly the same $\widehat{\mathrm{prec}}$ estimates reported in those papers, and added for comparison the ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ using the prevalence values described above. The results, displayed in Figure 3, show that without exception, the usual precision estimates $\widehat{\mathrm{prec}}$ are larger than the more accurate ${\widehat{\mathrm{prec}}}^{\mathrm{prev}}$ estimates, in agreement with the previously observed fact that $\widehat{\mathrm{prec}}$ displays a larger (optimistic) bias. The bias is particularly large in the case of the liver disease study, reflecting the fact that among the three data sets, this is the one where the value of $\mathrm{prev}$ and $r$ differ the most.

Figure 3.

Precision estimates for different classification rules using separately sampled leukemia, breast cancer, and liver disease data. The white bars depict the usual estimated precision estimates, while the shaded bars are for the precision estimates using the true case prevalences. NB indicates Naive Bayes; 3NN, 3-Nearest Neighbors; SVM, Support Vector Machine.

Concluding Remarks

Accuracy and reproducibility in observational studies is critical to the progress of biomedicine, in particular, in the discovery of reliable biomarkers for disease diagnosis and prognosis. In this study, theoretical results confirmed by numerical experiments show that the usual estimator of precision can be severely biased under the typical separate sampling scenario in observational case-control studies. This will be true especially if the true disease prevalence differs significantly from the apparent prevalence in the data. If knowledge of the true disease prevalence is available, or can even be approximately ascertained, then it can be used to define a modified precision estimator, which is nearly unbiased at moderate sample sizes. In all the results using real data sets, we observed that the usual precision estimator produces values that are larger, ie, more optimistic, than the modified one using the true prevalence, which agrees with the results obtained with the synthetic data. Absence of knowledge about the true prevalence means simply that the precision cannot be reliably estimated in observational case-control studies and its use should be discouraged. Finally, we note that in our experiments, we considered the case where the prevalence is between 0.1 and 0.9, not without reason. If the prevalence is significantly under 0.1, as is the case in some rare diseases, then neither the precision, nor in fact the classification error, should be used as a criterion of performance, but rather the sensitivity and specificity need to be considered separately—otherwise, a large precision and small classification error can be achieved by biasing the classification rule to produce FP rates close to zero while ignoring the FN rate.

Appendix 1

Here we derive the asymptotic approximation in equation (29). If $f:{ℝ}^2\to ℝ$ is infinitely differentiable at point $(a,b)$, then it can be expanded by a bivariate Taylor series around $(a,b)$ as

$$\begin{array}{l}f(x,y)=f(a,b)+\frac{\partial f(a,b)}{\partial x}(x-a)+\frac{\partial f(a,b)}{\partial y}(y-b)\hfill \\ +\mathrm{second}\mathrm{and}\mathrm{higher}\mathrm{order}\mathrm{terms}\mathrm{in}x-a\mathrm{and}y-b\hfill \end{array}$$ (37)

Now let $X_n$ and $Y_n$ be sequences of random variables with means µ_X and µ_Y, with ${\mu }_Y\ne 0$. The ratio $x/y$ is infinitely differentiable at $(a,b)$ if $b\ne 0$; therefore, we can apply the previous result and get

$$\begin{array}{l}\frac{X_n}{Y_n}=\frac{{\mu }_X}{{\mu }_Y}+\frac{1}{{\mu }_Y}(X_n-{\mu }_X)-\frac{{\mu }_X}{{\mu }_Y^2}(Y_n-{\mu }_Y)\hfill \\ +\mathrm{second}\mathrm{and}\mathrm{higher}\mathrm{order}\mathrm{terms}\mathrm{in}{X}_n-{\mu }_X\mathrm{and}{Y}_n-{\mu }_Y\hfill \end{array}$$ (38)

Taking expectations on both sides gives

$$E\left[\frac{X_n}{Y_n}\right]=\frac{{\mu }_X}{{\mu }_Y}+E\left[\begin{array}{l}\mathrm{second}\mathrm{and}\mathrm{higher}\\ \mathrm{order}\mathrm{terms}\mathrm{in}{X}_n-{\mu }_X\mathrm{and}{Y}_n-{\mu }_Y\end{array}\right]$$ (39)

Except in pathological cases involving heavy-tailed distributions, the remainder in the previous equation becomes negligible as $X_n\to {\mu }_X$ and $Y_n\to {\mu }_Y$ in probability. Therefore, we write

$$E\left[\frac{X}{Y}\right]\approx \frac{E[X]}{E[Y]}$$ (40)

as long as $X$ and $Y$ are around $E[X]$ and $E[Y]$, respectively (ie, $\mathrm{Var}[X]$ and $\mathrm{Var}[Y]$ are small).