Evaluating the Predictiveness of a Continuous Marker

Summary

Consider a continuous marker for predicting a binary outcome. For example, serum concentration of prostate specific antigen (PSA) may be used to calculate the risk of finding prostate cancer in a biopsy. In this paper we argue that the predictive capacity of a marker has to do with the population distribution of risk given the marker and suggest a graphical tool, the predictiveness curve, that displays this distribution. The display provides a common meaningful scale for comparing markers that may not be comparable on their original scales. Some existing measures of predictiveness are shown to be summary indices derived from the predictiveness curve. We develop methods for making inference about the predictiveness curve, for making pointwise comparisons between two curves and for evaluating covariate effects. Applications to risk prediction markers in cancer and cystic fibrosis are discussed.

1. Background

The Early Detection Research Network (EDRN) is a national network of biomarker development laboratories and clinical centers sponsored by the National Cancer Institute (Vastag 2000). Two of its goals are to develop biomarkers for cancer screening and for cancer risk prediction. The statistical focus in cancer screening is on the capacity of a marker to accurately classify subjects as diseased or not (Pepe et al 2001). Classification performance parameters such as sensitivity and specificity are of key interest, because ultimately it is the proportion of diseased subjects detected (sensitivity) and the proportion of non-diseased subjects unnecessarily referred for work-up (1-specificity) that enter into decisions about screening policy. The evaluation of markers for cancer risk prediction, however, requires a different approach. In this context we need to quantify how well a marker identifies people at high or low risk for cancer. A cancer risk prediction marker might be used to select subjects for a prevention intervention or indeed for screening, but does not classify subjects directly. Indeed Gail and Pfeiffer (2005) note that criteria for cancer risk prediction markers will often be much less stringent than those required of screening markers.

In this paper we propose a graphical display to aid in the assessment of risk prediction markers. The predictiveness curve shows the predictive capacity of a marker. An important attribute is that it provides a common scale for comparing risk prediction markers. The predictiveness curve has been suggested previously by Bura and Gastwirth (2001) and Copas (1999), albeit with different terminology. However, their focus was on inference for summary indices. They did not address inference for the curve itself or comparisons between curves, nor did they consider curves for subpopulations defined by covariates. In this paper, in addition to addressing these points, we demonstrate the practical usefulness of the predictiveness curve in assessing the value of risk prediction markers. We illustrate the methodology using two datasets. The first is a non-cancer application. It concerns major pulmonary infections in children with cystic fibrosis and the capacity of measures of lung function and nutritional status to predict them. The second concerns prostate specific antigen (PSA), a widely used biomarker for prostate cancer.

2. Predictiveness of a Binary Marker

Let D denote the binary outcome and denote the marker by Y. The risk associated with marker value Y = y is

$$risk\left(y\right)\equiv P(D=1\mid Y=y).$$

Throughout most of this paper we assume that larger values of Y are associated with increasing risk, but generalize the ideas in Section 9. Although our interest is primarily in evaluating continuous markers, we first consider the simple setting when the marker is binary. In that case, subjects either have the lower risk level, risk(0) = P(D = 1|Y = 0), or the higher value, risk(1) = P(D = 1|Y = 1).

Frequently the relative risk is used to summarize the predictiveness of a marker. However, clearly the absolute levels of risk, not just their ratio, are important in describing the predictive capacity of a marker. For example, a marker with relative risk equal to 10 may correspond to absolute risks of (risk(0) = 0.1%, risk(1) = 1%) or (risk(0) = 1%, risk(1) = 10%). These two scenarios have very different implications if risks below 5% say are considered unimportant and risks above 5% are considered “high.”

Reporting of absolute risks, however, is not sufficient either. The proportions of the population who attain the lower and higher risk levels are also crucial components of predictiveness. For example, when seeking a marker that identifies subjects at “high risk”, a marker with larger high risk prevalence, P[Y = 1], may be preferable even if the absolute high risk value, risk(1), is somewhat smaller.

The definitions of “high” and “low” risk depend entirely on the clinical context and include consideration of overall prevalence, consequences of disease, and interventions for subjects in the low and high risk strata. In general, once appropriate thresholds have been decided upon, one marker is preferable to another if it leaves fewer subjects in the intermediate equivocal risk range.

In conclusion, to fully describe the predictiveness of a binary marker, we stipulate that one should report the absolute risks and the frequencies of those risks in the population. That is, one should report the distribution of risk in the population conferred by the marker. The remainder of this paper expands on this theme in relation to continuous markers.

3. The Predictiveness Curve

We propose the predictiveness curve, R(v) versus v, for describing the predictive capacity of a marker where R(v) is the risk associated with the v^th quantile of the marker:

$$R\left(v\right)=P[D=1\mid Y=F^{-1}\left(v\right)],$$

and F is the cumulative distribution function of the marker. Figure 1 displays curves for −FEV₁, a measure of lung function and −weight, a measure of nutritional status, for predicting serious lung infection in the following year among cystic fibrosis patients. The markers are negated to conform to our convention that increasing values are associated with increasing risk. Details of the data will be discussed in Section 7.

Observe that by using the scale v = F(Y) on the x-axis, the markers are transformed to a common scale. This facilitates their comparison, whereas on their original scales the markers are not comparable. We see for example that at the 90^th percentile of −FEV₁, the risk is 0.76, whereas the risk is only 0.58 at the 90^th percentile of −weight. Patients in the top 10% of the marker distribution are at greater risk when lung function rather than nutritional status is used as the risk prediction marker. Pulmonary function is also a better marker of low risk. The bottom 10% have risks in the range (0.25, 0.28) according to −weight but in a much lower range, (0.01,0.15) according to −FEV₁.

Another way of looking at the predictiveness curve is to consider the inverse function. We see that R⁻¹(p) is the proportion of the population with risks less than or equal to p. Suppose p_L is a threshold that defines ‘low risk’ and p_H is a threshold that defines ‘high risk.’ Then the proportions of the population with low, high, and equivocal risks are R⁻¹(p_L), 1−R⁻¹(p_H) and R⁻¹ (p_H)−R⁻¹(p_L), respectively. To illustrate in the cystic fibrosis example, suppose we take p_H = 0.75 and p_L = 0.25, then lung function is predictive of low risk in R⁻¹(0.25) = 29% of the population, of high risk in 1 − R⁻¹(0.75) = 12% and it leaves 59% of patients in the equivocal risk range. Nutritional status on the other hand is completely uninformative about high or low risk. Knowledge of weight leaves all 100% of patients with risks in the equivocal risk range, (0.25,0.75).

Interestingly, the predictiveness curve displays the distribution of risk(Y) in the population. As mentioned above, and most easily seen from the plot itself, R⁻¹(p) is the proportion of the population with risks less than p. Mathematically we write

$$R^{-1}\left(p\right)=P[risk\left(Y\right)<p]$$

is the cumulative distribution of risk(Y). Correspondingly R(v) is the 100×v^th percentile of risk(Y) in the population. We find the display simple and useful for describing the predictive capacity of a marker. It conveys the essential elements of our concept of predictiveness and leads us to a general definition for the predictiveness of a marker in the population.

$$Definition.\text{Predictiveness of}Y\equiv \text{the distribution of}risk\left(Y\right)$$

A marker that is uninformative about risk assigns equal risk to all subjects, risk(Y) = P[D = 1|Y] = P[D = 1] = ρ. The corresponding predictiveness curve is the horizontal line at R(v) = ρ, v ∈ (0, 1). On the other hand, a perfect marker assigns risk(Y) = 1 for the proportion ρ of subjects with D = 1 and risk(Y) = 0 for the proportion 1 − ρ with D = 0. Correspondingly, its predictiveness curve is the step function R(v) = I[(1 − ρ) < v], where I[.] is the indicator function. Most risk prediction markers are imperfect, lying between these extremes. Better markers have steeper curves corresponding to wider variation in risk. Note that ∫ R(v)dv = ∫ P[D = 1|Y]dF(Y) = ρ. Therefore ∫(R(v) − ρ)dv = 0, which implies that ${\int }_0^{v^{\ast }}(\rho -R\left(v\right))dv={\int }_{v^{\ast }}^1\left(R\left(v\right)\right)-\rho )dv$ where v* is the point where R(v*) = ρ. In other words, the areas between the curve and the horizontal line at ρ that are above and below the horizontal line are equal. The horizontal line at ρ serves as a useful benchmark and visual aid in evaluating predictiveness curves.

4. Estimation

We now turn to the task of estimating the predictiveness curve. Suppose data from a random sample of n independent identically distributed subjects are available, {(Y_i, D_i), i = 1, . . . , n}. We model the risk as a parametric increasing function of Y into (0,1):

$$P[D=1\mid Y]=G(\beta ,Y)$$

where G has the form of a cumulative distribution function (cdf). Assume that an asymptotically normal estimator of β is employed with $\text{var}\left(\sqrt{n}(\widehat{\beta }-\beta )\right)=\Sigma \left(\beta \right)$. For example, $\widehat{\beta }$ might be the maximum likelihood estimate from a linear logistic model G(β, Y) = exp(β₀ + βY)/{1 + exp(β₀ + βY)}. We prefer to employ more flexible models and use the 3 parameter Box-Cox family (Cole and Green, 1992) in our illustrations. Let F̂ be the empirical cdf of Y. Then

$$\widehat{R}\left(v\right)\equiv G(\widehat{\beta },{\widehat{F}}^{-1}\left(v\right))v∊(0,1).$$

Theorem 1 The asymptotic distribution of $\sqrt{n}(\widehat{R}\left(v\right)-R\left(v\right))$ is mean 0, normal, with variance σ²(v) where

$${\sigma }^2\left(v\right)={\left(\frac{\partial R\left(v\right)}{\partial \beta }\right)}^T\Sigma \left(\beta \right)\left(\frac{\partial R\left(v\right))}{\partial \beta }\right)+v(1-v){\left(\frac{\partial R\left(v\right)}{\partial v}\right)}^2$$ (1)

and 0 < v < 1.

The result indicates that the variance of R̂(v) is comprised of two additive components. The first is due to variability in $\widehat{\beta }$ while the second is due to variability in F̂⁻¹(v). Observe that the magnitude of the second component depends on the slope of the predictiveness curve at v. The variability due to F̂⁻¹(v) is more important when R(v) is steep. It makes sense intuitively that imprecision in F̂⁻¹(v) will have a greater impact on R̂(v) when (∂/∂v)R(v) is larger. Asymptotic theory for the inverse function is provided in the next result.

Theorem 2 $\sqrt{n}({\widehat{R}}^{-1}\left(p\right)-R^{-1}\left(p\right))$ has an asymptotically normal distribution with mean 0 and variance τ²(p) where

$${\tau }^2\left(p\right)={\left[\frac{\partial R^{-1}\left(p\right)}{\partial \beta }\right]}^T\Sigma \left(\beta \right)\left[\frac{\partial R^{-1}\left(p\right)}{\partial \beta }\right]+R^{-1}\left(p\right)(1-R^{-1}\left(p\right))$$ (2)

and p is in the range of {R(v) : vε(0, 1)}

The variance again is comprised of two additive components, one due to $\widehat{\beta }$ and the other due to F̂. If β were known precisely, then R̂⁻¹(p) is the binomial proportion of subjects with risk(Y) < p, so its variance is the binomial variance R⁻¹(p)(1 − R⁻¹(p))/n as indicated by (2). On the other hand, if F, the population distribution of Y, were known, then ${\widehat{R}}^{-1}\left(p\right)=F\left(G^{-1}(\widehat{\beta },p)\right)$, which only includes $\widehat{\beta }$ as a random variable, and its variance is given by the first component of (2). Observe also the simple relationship between τ²(p) and σ²(v) at v = R⁻¹(p): σ²(v) = ((∂/∂v)R(v))²σ²(p). Variability in R̂⁻¹(p) is magnified by (∂/∂v)R(v) on the scale for R̂(v).

Consistent estimates of σ²(v) and τ²(p) are obtained by substituting estimated quantities into expressions (1) and (2). Derivatives with respect to β of R̂(v) = G(β, F̂⁻¹(v)) and ${\widehat{R}}^{-1}\left(p\right)=\widehat{F}\left(G^{-1}(\widehat{\beta },p)\right)$ are easily obtained since G and G⁻¹ are simple differentiable functions of β. To estimate (∂/∂v)(R(v)) = [(∂/∂y)G(β, y)][(∂/∂v)F⁻¹(v)] where y = F⁻¹(v), the first component is straightforward while the second involves the density of Y, (∂/∂v)F⁻¹(v) = 1/f(F⁻¹(v)). We use a Gaussian kernel density estimate for f in our applications with bandwidth h optimal for normally distributed data (Silverman, 1986):h = n^−⅕ 1.06 min(standard deviation, interquartile range/1.349).

5. Simulation Studies

We performed a limited simulation study to investigate the use of large sample inference for R(v) and R⁻¹(p) in finite sample studies. Data were simulated according to two models. In the first, F is a standard normal distribution and the risk function is linear logistic, G(β, Y) = exp(β₀ + β₁Y)/(1 + exp(β₀ + β₁Y)). We chose (β₀, β₁) such that R(0.1) and R(0.9) were at pre-defined values. In the second, F is standard log normal and the risk function is from the Box-Cox family: G(β, Y) = Φ(β₀ + β₁Y^(β₂) where Y^(λ) = (Y^λ − 1)/λ when λ ≠ 0 and Y^(λ) = log Y when λ = 0. In this setting we fixed β₂ = −0.4, 0, or 0.4 and then chose (β₀, β₁) based on specified values for R(0.1) and R(0.9). We fit the predictiveness curves using the correct form for G(β, Y). That is, we did not investigate the robustness of these models to misspecification. In practice one should check for goodness of fit, investigating multiple model forms if necessary.

Tables 1 and 2 show the results of our simulations for one of the Box-Cox model simulations. We found that bias was minimal. Variance estimates reflected the actual sampling variability with sample sizes of n = 500 or more. Consequently the coverage of 95% confidence intervals was excellent in moderate to large samples but lower than the nominal level with n = 100. Problems occurred only at the edges of the predictiveness curve, at v = 0.1 and v = 0.9. Considering that with n = 100 only 10 observations lay beyond these points, the reduced performance under these circumstances seems reasonable. With larger sample sizes, (n ≥ 500), inference based on asymptotic inference appeared to work very well across all of the scenarios we studied (additional results not shown).

Table 1.

Results of 5,000 simulations to evaluate the application of inference based on asymptotic theory to finite sample studies. Box-Cox risk with β₀ = –0.486, β₁ = 0.793, β₂ = 0.4, the same as in Table 2. Shown are results for R̂(v)

	v = 0.1	v = 0.3	v = 0.5	v = 0.7	v = 0.9
R(v)	0.100	0.194	0.313	0.491	0.800
Bias
% bias in R̂(v)
n = 100	–1.636	–3.058	–0.969	–0.493	–0.749
n = 500	–0.164	–0.619	–0.468	–0.308	–0.031
n = 2000	–0.279	–0.240	–0.152	–0.095	–0.033
Variance
$\frac{\text{Asymptotic - Observed}}{\text{Observed}}\%$
n = 100	–7.660	–11.976	–4.646	–7.322	7.262
n = 500	3.276	1.103	–0.875	–1.694	–3.330
n = 2000	0.538	–0.200	1.739	0.047	5.910
95% Confidence Interval
coverage (%)
n = 100	86.529	92.094	92.914	92.874	89.992
n = 500	92.984	94.769	94.567	94.386	94.245
n = 2000	94.238	95.038	95.428	95.161	95.346

Table 2.

Results of simulations to evaluate the application of inference based on asymptotic theory to finite sample studies. Box-Cox risk with β₀ = –0.486, β₁ = 0.793, β₂ = 0.4, the same as in Table 1. Shown are results for R̂^–1(p)

	p = 0.1	p = 0.3	p = 0.5	p = 0.7	p = 0.9
R–1(p)	0.100	0.480	0.708	0.848	0.944
Bias
% bias in R̂–1(p)
n = 100	20.292	0.821	–0.892	–0.800	–0.746
n = 500	0.795	0.500	–0.096	–0.156	–0.199
n = 2000	0.316	0.154	0.0008	–0.034	–0.031
Variance
$\frac{\text{Asymptotic - Observed}}{\text{Observed}}\%$
n = 100	37.008	7.548	5.396	–9.196	–0.409
n = 500	11.167	1.173	–0.018	–1.133	3.857
n = 2000	1.249	1.730	1.134	4.121	6.095
95% Confidence Interval
coverage (%)
n = 100	73.46	92.11	92.01	94.06	91.23
n = 500	91.33	93.60	94.08	95.09	95.53
n = 2000	94.20	94.77	94.65	95.71	95.86

6. Further Inferential Techniques

6.1 Comparing Markers

A key attribute of the predictiveness curve is that it provides a common relevant scale for comparing risk prediction markers. Here we consider formal comparisons between two markers. Comparisons might be based on the difference in risk percentiles, R_A(v₀) − R_B(v₀), at some v = v₀ of interest where subscripts denote markers A and B. The Appendix summarizes asymptotic distribution theory for R̂_A(v) − R̂_B(v), assuming that both markers are measured on the same individuals. The estimated standard error of the difference is calculated using steps analogous to those described earlier for calculating the estimated standard error of R̂(v). A p-value can then be based on the Z-statistic: R̂_A(v) − R̂_B(v) divided by the estimated standard error.

A particularly compelling case can be made for comparing markers on the basis of R̂⁻¹(p) where p is a threshold that defines high (or low) risk. One marker of high risk would be preferred over another if it identifies a greater fraction of people at high risk. That is, if the high risk threshold is p_H and $R_A^{-1}\left(p_H\right)-R_B^{-1}\left(p_H\right)>0$ then marker B is a better marker of high risk. Asymptotic distribution theory for ${\widehat{R}}_A^{-1}\left(p\right)-{\widehat{R}}_B^{-1}\left(p\right)$, provided in the Appendix can serve as the basis of confidence intervals and hypothesis testing.

Our methodology concerns pointwise comparisons between markers. The simple clinically relevant interpretations for points on the predictiveness curve and its inverse motivate making such comparisons in practice. However, statistical power might be greater for statistics based on summary indices. Gail and Pfeiffer (2005) note that several indices of predictability are functionals of the predictiveness curve. For example, the average entropy (Shapiro 1977) is ∫{R(v) log R(v) + (1 − R(v)) log(1 − R(v))}dv and the risk variance is ∫(R(v) − ρ)²dv. Both give rise to measures of the proportion of explained variation (Mittlebock and Schemper, 1996) which are existing summary indices of predictiveness. Bura and Gastwrith (2001) suggest the total gain, TG = ∫|R(v) − ρ|dv, as a summary index. Methods for making formal comparisons between predictiveness curves based on summary indices, however, have not been studied yet. Bootstrapping could be used for inference in practice.

6.2 Covariate Specific Predictiveness Curves

The predictiveness of a marker can vary across populations. This can happen if the marker distribution varies or if there is an interaction between the marker and a covariate on risk. In addition, the usefulness of a risk prediction marker may vary with the overall risk. In a low risk population, the marker may not identify any high risk subjects while in a moderate risk population it might, even if the distribution of the marker and its association with risk are the same. In this section we consider how to estimate predictiveness curves in subpopulations defined by covariates denoted by Z.

For a discrete covariate, one can simply stratify and estimate stratum specific predictiveness curves, R_Z(v), as described earlier. More generally we model the effect of Z on risk with

$$P(D=1\mid Y,Z)=G(\beta ,Y,Z)$$

where, as before, G is monotone increasing in Y, and we use a semiparametric location-scale model (Heagerty and Pepe, 1999) for the distribution of Y given Z

$$F_z\left(y\right)=P(Y\le y\mid Z=z)=F_0\left(\frac{y-{\mu }_z}{{\sigma }_z}\right)$$

where μ_Z = γ^′U(Z) and log(σ_Z) = δ^′W(Z), and U(Z) and W(Z) are specified functionals of Z. For example, for binary Z, U(Z) and W(Z) could be (1, Z), while for continuous Z, U(Z) and W(Z) could be a B-spline basis for Z. Writing U_i = U(Z_i) and W_i = W(Z_i), the estimators $\widehat{\gamma }$ and $\widehat{\delta }$ are solutions to

$$\begin{array}{cc}\hfill & \sum _{i=1}^n{U}_i(Y_i-{\gamma }^{\prime }{U}_i)∕{\sigma }_{Z_i}^2=0\hfill \\ \hfill & \sum _{i=1}^n{W}_i[{(Y_i-{\gamma }^{\prime }{U}_i)}^2-{\sigma }_{Z_i}^2]∕{\sigma }_{Z_i}^2=0.\hfill \end{array}$$

Denoting the empirical cumulative distribution of standardized residuals by F̂₀(·), the covariate-specific marker, Z=z, distribution estimate is

$${\widehat{F}}_z\left(y\right)={\widehat{F}}_0\left(\frac{y-{\widehat{\gamma }}^{\prime }u}{e^{{\widehat{\delta }}^{\prime }w}}\right)$$

where u = U(z) and w = W(z). The corresponding v^th quantile is

$${\widehat{F}}_z^{-1}\left(v\right)={\widehat{\gamma }}^{\prime }u+e^{{\widehat{\delta }}^{\prime }w}{\widehat{F}}_0^{-1}\left(v\right).$$

Having fit the risk model, through maximum likelihood or otherwise, the covariate specific predictiveness curve estimate is

$${\widehat{R}}_z\left(v\right)=G(\widehat{\beta },{\widehat{F}}_z^{-1}\left(v\right),z).$$

Similarly, the estimated inverse is

$${\widehat{R}}_z^{-1}\left(p\right)={\widehat{F}}_z\left(G^{-1}(\widehat{\beta },p,z)\right).$$

Asymptotic distribution theory is provided in the appendix.

7. The Cystic Fibrosis Data

Cystic fibrosis is a genetic disorder that results in impaired ion transport across the cell membrane. Its effects on pulmonary and gastrointestinal systems lead to progressive deterioration. Predicted survival currently extends into the mid 30s (Cystic Fibrosis Foundation, 2004). The main culminating event that leads to death is acute pulmonary exacerbation, i.e., lung infection requiring intravenous antibiotics.

To illustrate our methodology we use data from the Cystic Fibrosis Registry, a database maintained by the Cystic Fibrosis Foundation, containing annually updated information on over 20,000 people diagnosed with CF and living in the USA. We consider weight and FEV₁ measured in 1995 to predict pulmonary exacerbations in 1996. Data for 11,960 patients are analyzed of whom 5094 (42%) had at least one pulmonary exacerbation. Patients 6 years of age and older are included. Weight is standardized for age and gender (Hamill et al., 1977) and FEV₁ is standardized for age, gender and height (Knudson et al., 1983). See Moskowitz and Pepe (2004) for more details. Figure 1 shows the estimated predictiveness curves for −FEV₁ and −weight. We have referred to them as population curves earlier in the paper when illustrating concepts but now acknowledge their sampling variability.

Table 3 shows point estimates and confidence intervals for R(v) and R⁻¹(p). We have tight confidence intervals for the estimates whose values were already mentioned in relation to this curve in Section 3. Confidence intervals calculated using bootstrap resampling were almost identical to those based on asymptotic theory. The second column shows the % variance in R̂(v) due to F̂, i.e., the second component of (1). Observe that it is larger for −FEV₁ than for −weight presumably because the predictiveness curve for −FEV₁ is more steep. An hypothesis test based on the difference in predictiveness estimates at v = .1 yields p-value< 0.01. This test uses the asymptotic variance expression given in the Appendix. At v = 0.9 the difference in R(v) for the two markers is also statistically significant (p-value < 0.01).

Table 3.

Point estimates and 95% confidence intervals for R(v) and R^–1(p) using – FEV₁ and –weight as markers of risk for subsequent pulmonary exacerbation in patients with cystic fibrosis.

	Estimate	Variance due to F̂	Confidence Interval Asymptotic	Confidence Interval Bootstrap
R(0.9)
FEV1	0.76	10.0%	(0.748,0.779)	(0.749,0.779)
weight	0.58	0.60%	(0.568,0.601)	(0.567,0.601)
R(0.1)
FEV1	0.15	8.91%	(0.133,0.157)	(0.133,0.157)
weight	0.28	0.72%	(0.262,0.293)	(0.261,0.294)
R–1(0.25)
FEV1	0.29	18.16%	(0.273,0.311)	(0.270,0.314)
weight	0	0%	(0,0.040)	(0,0.039)
R–1(0.75)
FEV1	0.88	13.58%	(0.865,0.897)	(0.864,0.898)
weight	1	0%	(1,1)	(1,1)

We next consider the predictiveness of lung function (Y = −FEV₁) in subpopulations defined by their nutritional status (Z = weight). We modeled the risk of pulmonary exacerbation with the Box-Cox form

$$P[D=1\mid Y,Z]=\Phi ({\beta }_0+{\beta }_1{Y}^{\left(\lambda \right)}+{\beta }_{2}Z)$$

and used a semiparametric location scale model for log(−Y) with mean a natural cubic spline function of Z having knots at the 0.30 and 0.70 quantiles of Z and log standard derivation also a natural cubic spline with knot at the median of Z. Fitted predictiveness curves are shown in Figure 2 for subjects with weight at the median, first and third quartiles. Observe that the incidence of pulmonary exacerbation varies across these three populations, from 58% in subjects at the first quartile of weight to 30% in subjects at the third quartile. In the latter population, FEV₁ identifies .41 of subjects below the low risk threshold of 25% (95% CI = (0.38,0.43)) but only 0.01 with risks above the high risk threshold of 75% (95% CI=(0.01,0.02)). On the other hand, in the high risk population FEV₁ identifies only 0.06 (95% CI=(0.05,0.07)) low risk subjects and 0.23 (95% CI=(0.20,0.25)) high risk subjects. Therefore FEV₁ is a particularly useful marker of high risk in subjects at already somewhat elevated risk due to their nutritional status and similarly it is more useful as a low risk marker in subjects at somewhat reduced risk based on their weight.

8. Markers for Prostate Cancer

The Prostate Cancer Prevention Trial was a randomized prospective study of men with PSA < 3.0 ng/mL, aged 55 years and older who were followed up for 7 years with annual PSA measurements. A biopsy was recommended for all men either during or at the end of the study. Thompson et al. (2006) identified 5519 men on the placebo arm of the trial who had undergone prostate biopsy and had a PSA and digital rectal exam (DRE) during the year prior to biopsy and at least 2 PSA values from the 3 years prior to biopsy. Prostate cancer risk was evaluated as a function of PSA, PSA velocity and several other variables including age, family history, DRE and prior prostate biopsy. Here we use the data to compare PSA and PSA velocity as predictors of prostate cancer risk. 21.9% of men were found to have prostate cancer. Figure 3 top panel displays the predictiveness curves for PSA and PSA velocity. The curve for PSA velocity is more shallow, indicating that it is a poorer marker of risk for this general class of prostate cancers. The 90^th percentile of risk is 0.291 according to PSA velocity while it is higher, 0.369, according to the absolute most recent PSA measurement (p < 0.01). At the low end of the scale, the 10^th risk percentiles based on PSA velocity and PSA are 0.149 and 0.091 (p < 0.01) respectively, again suggesting that PSA is the better marker of risk. According to PSA velocity 2.9% of men can be classified as having risk below 10% while far more, 13.6%, qualify as low risk when using most recent PSA as the marker (p < 0.01). In addition, a greater fraction are found to have risks above 30% with PSA, 1 − R̂⁻¹(.30) = 23.0%, than with PSA velocity 1 − R̂⁻¹(0.30) = 8.0%(p < 0.01).

Prostate cancer biopsy specimens are classified using the Gleason scoring system with higher scores associated with more aggressive disease. The bottom panel of Figure 3 shows the same markers as predictors of high-grade prostate cancer (Gleason score ≥ 7) which occurred in only 4.7% of men. Again, the 10^th and 90^th percentiles of risk are better for PSA than for PSA velocity. The 10^th percentiles are 0.4% versus 1.3% (p=0.05) and the 90^th percentiles are 11.1% versus 8.5% (p < .01) for the two markers. The proportion of subjects with risks greater than 10% is 0.063 according to the PSA velocity marker while the proportion is much larger, 0.132, according to the absolute PSA marker (p < 0.01). We conclude that absolute PSA is also a better risk prediction marker than PSA velocity for high grade prostate cancer.

9. Discussion

The predictiveness curve provides a complete and conceptually simple description of the capacity of a marker to predict risk. It is not an entirely new proposal. The idea of categorizing a continuous marker according to quartiles or quintiles say and documenting the proportions of subjects with D = 1 in each category is not uncommon in the applied literature (see Willett et al., 1987 for example). Our proposal builds on and formalizes the idea, allowing the marker scale to remain continuous and restricting the risk function to be monotone. Our method has the added advantage that it provides for inference about the inverse function, R⁻¹(p) (or 1− R⁻¹(p)) the proportion of the population with risks below (or above p), which are often of key interest.

Receiver operating characteristic curves generalize to continuous markers the notions of sensitivity, P(Y = 1|D = 1), and specificity, P(Y = 0|D = 0), that are defined for binary markers. The idea is to use a threshold c to define a series of binary markers, “Y > c”, and to plot the corresponding sensitivity versus 1−specificity for all values of c. In a similar vein, Moskowitz and Pepe (2004) proposed generalizing the binary marker notions of positive and negative predictive values, PPV = P(D = 1|Y = 1) and NPV = P(D = 0|Y = 0), to continuous markers using thresholds. They proposed the positive predictive value (PPV) curve, which is a plot of PPV(v) = P(D = 1|Y > F⁻¹(v)) versus v. These curves are mathematically related to the predictiveness curve

$$\mathrm{PPV}\left(v\right)={\int }_v^{1}R\left(u\right)du∕(1-v).$$

In words, the PPV of the decision rule F(Y) > v is the average risk(Y) for Y that satisfy the positivity criterion. We note that the PPV curve is concerned with the performance of classification rules, not directly with predictiveness. An estimated predictiveness curve gives rise to an estimate of the predictive value curve. However, in contrast the empirical nonparametric methods used by Moskowitz and Pepe (2004) to estimate the PPV curve do not give rise to a simple estimate of R(v). Numerical derivatives are required and mononicity cannot be guaranteed. Observe that neither ROC curves nor positive predictive value curves show the population distribution of risk that is displayed by predictiveness curves.

A key element of our proposal is that risk(Y) is monotone increasing. This allows us to interpret R⁻¹(p) as the proportion of the population with risks below p, and to say that a proportion v of the population have risks below R(v). Some markers may not have monotone increasing risk functions. To accommodate such markers a more general definition of the predictiveness curve can be provided:

$$R\left(v\right)=p:P[risk\left(Y\right)\le p]=v$$ (3)

That is, R(v) is the 100×v^th percentile of risk(Y) in the population. Equivalently, R⁻¹(p) is the cdf of the random variable risk(Y). The ordering on the x-axis is according to risk(Y) and is equivalent to that based on Y if risk(Y) is a monotone increasing function. For the more general case, one can fit a risk model that allows non-monotonicity and plot the empirical percentiles for R̂(v).

In most applications however, we expect that the risk function is monotone increasing with the marker. Efficiency is likely gained by incorporating this restriction into the model. We used the Box-Cox family of distributions to fit monotone predictiveness curves to data. This 3 parameter family is reasonably flexible. It includes a wide variety of shapes (Cole and Cole 1992). However, further research is required to determine if another parametric family would be preferable. For fitting non-monotone curves we prefer B-splines because of their local nature and numerical stability. However analogues for fitting monotone curves do not appear to be available.

We have considered simple continuous prediction markers in this paper. However Y could be a function of multiple markers and risk factors. It might be a clinical prediction score derived from fitting a risk model to data. We note that if the risk score is to be evaluated with predictiveness curves estimated from the same data as used to develop the score, then issues pertaining to shrinkage, (ie., overoptimism of R̂(v) estimated from the same data used to develop the linear combination) would need to be addressed. We leave that for future research.

Appendix: Large Sample Distributions

Let

$$\begin{array}{cc}\hfill {\Psi }_{1i}& =-\frac{\partial R\left(v\right)}{\partial v}(I[Y_i\le F^{-1}\left(v\right)]-v)\hfill \\ \hfill {\Psi }_{2i}& =I[G(\beta ,Y_i)\le p]-P[G(\beta ,Y)\le p]=I[Y_i\le G^{-1}(\beta ,p)]-R^{-1}\left(p\right)\hfill \end{array}$$

and assume $\sqrt{n}({\widehat{\beta }}_n-\beta )=n^{-\frac{1}{2}}{\sum }_{i=1}^n{\Psi }_{3i}+o_p\left(1\right)$, where Φ_3i, i = 1, ..., n are independently identically distributed variables with E(Φ_3i|Y_i) = 0 and var(Φ_3i) ≡ Σ(β). Observe that Φ_1i and Φ_2i are also mean zero random variables with variances $v(1-v){\left(\frac{\partial R\left(v\right)}{\partial v}\right)}^2$ and R⁻¹(p)(1 − R⁻¹(p)), respectively. When $v=R^{-1}\left(p\right),{\Psi }_{1i}=-\frac{\partial R\left(v\right)}{\partial v}{\Psi }_{2i}$.

Then

$$\begin{array}{cc}\hfill & \sqrt{n}\left(\widehat{R}\left(v\right)-R\left(v\right)\right)=n^{-1∕2}\left(\frac{\partial R\left(v\right)}{\partial \beta }\right)\sum _{i=1}^n{\Psi }_{3i}+n^{-1∕2}\sum _{i=1}^n{\Psi }_{1i}+o_p\left(1\right)\hfill \\ \hfill & \sqrt{n}\left({\widehat{R}}^{-1}\left(p\right)-R^{-1}\left(p\right)\right)=n^{-1∕2}\left(\frac{\partial R^{-1}\left(p\right)}{\partial \beta }\right)\sum _{i=1}^n{\Psi }_{3i}+n^{-1∕2}\sum _{i=1}^n{\Psi }_{2i}+o_p\left(1\right)\hfill \end{array}$$

A.1 Comparing Markers

Subscripts A and B are used to denote the markers.

Result A.1

The asymptotic distribution of $\sqrt{n}({\widehat{R}}_A\left(v\right)-{\widehat{R}}_B\left(v\right)-(R_A\left(v\right)-R_B\left(v\right)))$ is normal with mean 0 and variance

$$\text{var}\left(\frac{\partial R_A\left(v\right)}{\partial {\beta }_A}{\Psi }_{3A}-\frac{\partial R_B\left(v\right)}{\partial {\beta }_B}{\Psi }_{3B}\right)+\text{var}({\Psi }_{1A}-{\Psi }_{1B})$$

where β_A and β_B are the parameters in the risk models for P(D= 1|Y_A) and P(D = 1|Y_B) respectively.

Result A.2

The asymptotic distribution of $\sqrt{n}({\widehat{R}}_A^{-1}\left(p\right)-{\widehat{R}}_B^{-1}\left(p\right))-(R_A^{-1}\left(p\right)-R_B^{-1}\left(p\right))$ is normal with mean 0 and variance

$$\text{var}\left(\frac{\partial R_A^{-1}\left(p\right)}{\partial {\beta }_A}{\Psi }_{3A}-\frac{\partial R_B^{-1}\left(p\right)}{\partial {\beta }_B}{\Psi }_{3B}\right)+\text{var}({\Psi }_{2A}-{\Psi }_{2B})$$

Asymptotic variances can be estimated by substituting empirical or estimated quantities as necessary.

A.2 Covariate Specific Predictiveness Curves

Result A.3

Denote $\sqrt{n}(\widehat{\gamma }-\gamma )\equiv n^{-1∕2}{\sum }_{i=1}{\gamma }_i,\sqrt{n}\left(\widehat{\delta }-\delta \right)\equiv n^{-1∕2}{\sum }_{i=1}{\delta }_i$, and $\text{var}\left(\sqrt{n}(\widehat{\beta }-\beta )\right)\equiv \Sigma \left(\beta \right)$.

Let

$$\begin{array}{cc}\hfill {\Psi }_{1iz}& =-\frac{\partial R_z\left(v\right)}{\partial v}\left(I\left(\frac{Y_i-{\gamma }^{\prime }{U}_i}{e^{{\delta }^{\prime }{W}_i}}\le F_0^{-1}\left(v\right)\right)-v\right)\hfill \\ \hfill & +\frac{\partial G(\beta ,F_z^{-1}\left(v\right),z)}{\partial F_z^{-1}\left(v\right)}{\left(u-e^{{\delta }^{\prime }w}E(U∕e^{{\delta }^{\prime }W})\right)}^{\prime }{\gamma }_i\hfill \\ \hfill & +\frac{\partial G(\beta ,F_z^{-1}\left(v\right),z)}{\partial F_z^{-1}\left(v\right)}(F_z^{-1}\left(v\right)-{\gamma }^{\prime }u){(w-E\left(W\right))}^{\prime }{\delta }_i\hfill \end{array}$$

$\sqrt{n}({\widehat{R}}_z\left(v\right)-R_z\left(v\right))$ and $\sqrt{n}({\widehat{R}}_z^{-1}\left(p\right)-R_z^{-1}\left(p\right))$ converge in distribution to mean 0 normal random variables with variances

$$\begin{array}{c}\hfill {\sigma }_z^2\left(v\right)=\left(\frac{\partial R_z\left(v\right)}{\partial \beta }\right)\Sigma \left(\beta \right){\left(\frac{\partial R_z\left(v\right)}{\partial \beta }\right)}^T+\text{var}\left({\Psi }_{1z}\right),\text{and}\hfill \\ \hfill {\tau }_z^2\left(p\right)={\left(\frac{\partial R_z^{-1}\left(p\right)}{\partial p}\right)}^2{\sigma }_z^2\left(v\right),\hfill \end{array}$$

for $v=R_z^{-1}\left(p\right)$ respectively. These expressions reduce to those of Theorems 1 and 2 when no covariates are modeled.