A unified procedure for meta-analytic evaluation of surrogate end points in randomized clinical trials

Abstract

The meta-analytic approach to evaluating surrogate end points assesses the predictiveness of treatment effect on the surrogate toward treatment effect on the clinical end point based on multiple clinical trials. Definition and estimation of the correlation of treatment effects were developed in linear mixed models and later extended to binary or failure time outcomes on a case-by-case basis. In a general regression setting that covers nonnormal outcomes, we discuss in this paper several metrics that are useful in the meta-analytic evaluation of surrogacy. We propose a unified 3-step procedure to assess these metrics in settings with binary end points, time-to-event outcomes, or repeated measures. First, the joint distribution of estimated treatment effects is ascertained by an estimating equation approach; second, the restricted maximum likelihood method is used to estimate the means and the variance components of the random treatment effects; finally, confidence intervals are constructed by a parametric bootstrap procedure. The proposed method is evaluated by simulations and applications to 2 clinical trials.

1. INTRODUCTION

Surrogate end points are of great interest in randomized clinical trials when the clinically meaningful end point is expensive or it takes a long duration to occur. For a posttreatment intermediate outcome to qualify as a surrogate end point, it should reliably predict the treatment effect on the clinical end point. While few surrogates have been established (Fleming and DeMets, 1996), there is clearly a need for identifying surrogate markers to accelerate evaluation of new therapies and interventions (Rolan, 1997).

Statistical evaluation of surrogate end points has accumulated a large body of literature in the past 20 years. See, for example, Weir and Walley (2006), for a comprehensive review. In a landmark paper, Prentice (1989) defined a surrogate end point as “a response variable for which a test of the null hypothesis of no relationship to the treatment groups under comparison is also a valid test of the corresponding null hypothesis based on the true endpoint.” He proposed that surrogates should capture the full net effect of the treatment and should correlate with the true end points. While complete mediation is often hard to attain, Freedman and others (1992) developed a metric denoted as “proportion of treatment effect explained” (PTE), which is 1 minus the ratio of the treatment effect adjusted for surrogates and the net treatment effect, to quantify the degree of validity of a candidate surrogate end point. Concern has been expressed that the PTE measure is highly variable and may be outside of the range of 0 to 1.

Frangakis and Rubin (2002) noted that “statistical surrogates,” such as the PTE measure, involve assessing the treatment effect given the observed postrandomization surrogate markers and thus do not have a causal interpretation due to potential selection bias (Rosenbaum, 1984). They proposed a new definition of surrogacy, “principal surrogate,” based on the potential outcome framework (Rubin, 1974, Rubin, 1978). In essence, if one can identify individual-level treatment effects on the true outcome and individual-level treatment effects on the surrogate end point, it is sensible to evaluate surrogacy by assessing the association between these 2 sets of treatment effects. While individual-level treatment effects are generally not identifiable, Gilbert and Hudgens (2008) developed an estimation procedure by incorporating a baseline covariate(s) that predicts the unobserved potential outcomes in HIV vaccine trials. Li and others (2010) proposed a Bayesian estimation method to evaluate the probabilities of counterfactuals when both clinical and surrogate end points are binary.

Another thread of statistical evaluation of surrogacy is the meta-analytical approach, motivated by the lack of power to evaluate surrogacy in any single trial. In particular, Daniels and Hughes (1997) first considered Bayesian random-effects models to assess the association between treatment effects on the potential surrogate and the clinical outcome in multiple trials, using Markov chain Monte Carlo techniques. When both end points are normally distributed, Buyse and others (2000) recommended 2 coefficients of determination, one at the trial level (R _trial ²) and the other at the individual level (R _indiv ²). Both quantities can be estimated from meta-analysis using linear mixed models. Gail and others (2000) considered prediction accuracy of treatment effect on the clinical end point when a new trial comes in with only the surrogate end point measured.

For the nonnormal outcomes that are abundant in clinical trials, for example, binary and time- to-event end points, estimation of R _trial ² is not immediate due to modeling and computational difficulties in inferring variance components in a random-effects model with nonnormal outcomes. Several methods have been proposed, mostly dealing with special cases where the surrogate end point and the clinical end point have the same form. Notably, Renard and others (2002) proposed generalized linear mixed models (GLMMs) with latent variables in the binary–binary setting; for 2 failure time end points, Burzykowski and others (2001) proposed a two-stage procedure with copula models for the association of marginal survival functions. On the other hand, similar efforts have been devoted to extend the definition of the individual-level surrogacy, R _indiv ², to nonnormal settings. From an information theory perspective, Alonso and Molenberghs (2006) unified several measures previously proposed for the individual-level surrogacy, including a summary correlation measure R _Λ ² for repeated measurements (Alonso and others, 2006) and a general scaled likelihood reduction factor (Alonso and others, 2004).

Existing methods for nonnormal data entail that specialized models, if available, have to be invoked to handle the correlation of surrogate end point and clinical end point in the same participant, for example, copula models for 2 failure time outcomes, so that different settings require different models. None of these methods can model a surrogate end point and a clinical end point that have different forms, for example, a repeated binary surrogate and a failure time clinical end point, as in our data example in Section 6.1. In this article, we propose a unifying 3-step procedure for a broad range of outcome forms, including binary outcomes, failure time outcomes, and repeated measures. The joint distribution of the treatment effects is first obtained by an estimating equation approach, thereby circumventing the difficulty in modeling the correlation of clinical and surrogate end points, then variance components of between-trial treatment effects are estimated by a restricted maximum likelihood (REML) method, and finally, a parametric bootstrap procedure is employed to obtain their confidence intervals (CIs). Compared to existing methods, this 3-step procedure provides maximal flexibility for diverse forms of clinical and surrogate end points.

For each component of our proposed procedure, it is useful to emphasize its novelty relative to existing works. First, our proposed estimating equation approach is more general than that proposed by Gail and others (2000) since we consider the treatment effect directly in estimating equations rather than developing separate marginal models in the treatment arm and in the control arm. Semiparametric models, such as the Cox proportional hazard model widely used in randomized clinical trials, can thus be used instead of imposing distributional assumption on survival times. Daniels and Hughes (1997) proposed a Bayesian random-effects model that also accommodates general forms of outcomes. However, the estimation of the within-trial covariance between 2 treatment effects has to be obtained by a nonparametric bootstrap procedure. The estimation equation approach directly yields the within-trial covariance between 2 treatment effects. Second, the REML estimation of between-trial variance components is an adaptation of the bivariate random-effects model in the meta-analysis literature (van Houwelingen and others, 2002), which is easily implementable. Third, the CIs based on our proposed parametric bootstrap procedure provides better coverage than the likelihood-based CIs for the estimated correlation (see Section 4), for example, the two-stage estimation procedure in Burzykowski and others (2001) because the latter can be overly optimistic when a limited number of trials is available.

In Section 2, we define and discuss several metrics for the meta-analytical evaluation of surrogacy in a general regression setting that encompasses nonnormal outcomes. We describe the 3-step estimation procedure in Section 3 and present simulation results in Section 4, 2 data examples in Section 5, and close by a short discussion.

2. METRICS OF SURROGACY FOR PREDICTION

At the core of the requirement for a surrogate end point, “knowing the effect of treatment on the surrogate allows prediction of the effect of treatment on the more clinically relevant outcome” (Joffe and Greene, 2009), when a new trial comes in with only the surrogate end point measured. The meta-analytic evaluation of surrogate end points assesses empirically the predictability of treatment effects on the clinical end point by the surrogate in a series of studies, assuming that the new study employs the same type of treatments or drugs as the set of completed trials (Daniels and Hughes, 1997, Buyse and others, 2000, Gail and others, 2000). This amounts to quantifying the correlation of the 2 sets of treatment effects in multiple clinical trials. Buyse and others (2000) defined R _trial ² and R _indiv ² in the context of linear mixed models, where R _trial ² is the coefficient of determination for the 2 sets of trial-level treatment effects and R _indiv ² is the squared correlation between the surrogate and the clinical end points after adjustment for the trial effects and the treatment effect. We now generalize these concepts to any regression model that yields asymptotic linear estimators of treatment effects.

Consider data from J trials, j = 1,…,J, the jth trial of which contains n _j participants. For the ith subject in the jth trial, i = 1,…,n _j, a clinical end point Y _ij, a surrogate end point X _ij, and a binary treatment assignment Z _ij are measured. Let 𝒯 _1j be the effect of Z on X in the jth trial, and similarly, define 𝒯 _2j to be the effect of Z on Y in the jth trial. For outcomes that can be parameterized in a generalized linear model (GLM), the following models are assumed for the treatment effects:

(2.1)

(2.2)

where g ₁ and g ₂ are link functions, for example, the logit function for a binary outcome, α _0j and γ _0j define the intercepts in the jth trial. For failure time outcomes, Cox proportional hazard models can be constructed with trial-specific baseline hazard.

We hypothesize that (𝒯 _1j,𝒯 _2j) are random effects and their joint distribution is bivariate normal,

(2.3)

Because of randomization, we are able to estimate these trial-averaged treatment effects consistently by ${\widehat{𝒯}}_{1j}$ and ${\widehat{𝒯}}_{2j}$, the asymptotically linear estimators (ALEs) as we show in Section 2.2. Conditional on (𝒯 _1j,𝒯 _2j), the joint distribution of ${\widehat{𝒯}}_{1j}$, ${\widehat{𝒯}}_{2j}$ can be derived,

(2.4)

where Σ_ll^′j are elements of the asymptotic covariance matrix of $\left({\widehat{𝒯}}_{1j}\right)$, ${\widehat{𝒯}}_{2j}$ for l,l ^′ = 1,2. Marginally, the variance of consists of the between-trial variability in (2.3) and the within-trial variability in (2.4). The latter decreases with sample size in each trial, while the former is not affected by sample size. Note that differing from Buyse and others (2000), we do not treat the intercepts α _0j and γ _0j as random effects to reduce computation complexity for a nonnormal regression model. This should have little impact on the estimation of treatment effects since by randomization treatment effects are independent of the estimators of baseline parameters.

Now suppose a new trial with sample size n _{j + 1} has the surrogate end point measured but not the clinical end point. We assume that the pair of treatment effects for the new trial follows the bivariate normal distribution in (2.3). Suppose we estimate the treatment effect on the surrogate as and its estimated variability as , where Σ_{11j + 1} is the asymptotic variance. The bivariate random variables are asymptotic normal with mean (μ ₂,μ ₁), variance and covariance σ ₁₂. This leads to the expected treatment effect on the clinical end point and the variability,

The within-trial variability decreases with n _{j + 1} and is presumably much smaller than the between-trial variability, if heterogeneity of treatment effects across trials is evident. To accurately predict the treatment effect on the clinical end point in the new trial, it is thus of interest to quantify 3 metrics from the J existing trials:

1. the squared correlation between 2 sets of true treatment effects in the J trials. The higher this squared correlation is, the more accurate the prediction of the treatment effect on the clinical end point could be. In linear models, ρ ² is the R _trials(r) ² in the reduced random-effects model proposed by Buyse and others (2000), though ρ ² is defined in a general regression setting that covers nonnormal outcomes. For a good surrogate end point, we need ρ ² to be well over 0.5, ideally close to 1.

2. the slope of regressing 𝒯 _2j on 𝒯 _1j. It corresponds to the parameter β in Daniels and Hughes (1997), and in a single trial, it reduces to the relative effect proposed in Buyse and Molenberghs (1998). λ ₁ can be viewed as a summary of dose correspondence between 2 sets of treatment effects. If there is causal relationship between the surrogate and the clinical end points, one expect to see a bigger effect on the surrogate and a bigger effect on the clinical end point. This echoes the use of dose–response as evidence for causation in observational studies (Breslow and Day, 1980). Ghosh and others (2010) recently discussed the connection of the relative effect and the causal effect of the surrogate on the clinical outcome. Certainly the value of λ ₁ depends on the scales of 𝒯 _1j and 𝒯 _2j, for example, 𝒯 _1j may be in the linear scale and 𝒯 _2j may be in the scale of log odds ratio or log hazard ratio.

3. the expected treatment effect on the clinical end point when the treatment effect on the surrogate is 0. It corresponds to the parameter α in Daniels and Hughes (1997). Though the emphasis in predicting the treatment effect on the clinical end point is on the accuracy of such prediction (R _trial ²), it is informative to know whether the predicted treatment effect on the clinical end point is 0 when the treatment effect on the surrogate is 0 as it has ties to the Prentice's definition of surrogacy, as well as the “causal necessity” condition in principal surrogacy (Gilbert and Hudgens, 2008). In meta-analytical literature, a similar intercept measure has been discussed in Daniels and Hughes (1997).

The second and the third metric are complementary to the first one in making a prediction of the treatment effect on the clinical end points, knowing the treatment effect on the surrogate. A good surrogate would have a high ρ ² so that one can accurately make the prediction. It should also yield a small λ ₀ so that no effect on the surrogate suggests no effect on the clinical end point. A large λ ₁ is not required since its value is scale dependent but could be used for comparative purposes when several candidate surrogates are available and standardized to the same scale. These criteria essentially assess the association between 2 sets of treatment effects averaged across a number of trials. They are useful for the prediction, but they do not generally have a causal interpretation for the effect of the surrogate on the clinical end points in the individual level.

Although our focus in this paper is on the trial-level surrogacy, it is useful to discuss the implication of the models (2.1)–(2.4) to the individual-level surrogacy. Buyse and Molenberghs (1998) suggested the association between the surrogate and the clinical end points after adjustment for the treatment effect or R _indiv ² in linear models (Buyse and others, 2000), as an individual-level surrogacy measure. It is not required for prediction when a new trial comes in with only the surrogate measured, but a high R _indiv ² does improve prediction when the new trial has clinical end point partially collected (Li and Taylor, 2010). Extending the concept of R _indiv ² to repeated measures and nonnormal outcomes has motivated a series of works that are unified by information theory (Alonso and others, 2004, Alonso and others, 2006, Alonso and Molenberghs, 2006). Under our framework based on (2.3) and (2.4), it is straightforward to generalize R _indiv ² to the correlation of the estimated treatment effects. Define to be the asymptotic correlation of conditional on (𝒯 _1j,𝒯 _2j). It is easy to show that in the linear models, r _j ² is equivalent to R _indiv ², the residual correlation of X and Y after adjusting for the treatment. Thus, r _j ² extends this concept to the nonnormal settings. The estimation of r _j ², though not a focus of this article, can be easily attained by the estimation equation approach we describe later in Section 3.

3. A 3-STEP PROCEDURE TO ASSESS λ₀, λ₁ AND ρ

We start from the models in (2.1)–(2.4). For a series of estimated treatment effects there are 2 levels of variability: the within-trial variability from the estimation (2.4) and the between-trial variability (2.3). In the procedure we propose, we first estimate the treatment effects consistently in trials, along with the within-trial variability, that is, the asymptotic distribution of the estimated treatment effects. We then use an expectation–maximization (EM) algorithm to compute REML estimates for (μ ₁,μ ₂,σ ₁ ²,σ ₁₂,σ ₂ ²), and hence λ ₁, λ ₂, and ρ.

3.1. Within-trial joint distribution of

Denote by α the complete set of parameters in the regression model (2.1) and denote by γ the complete set of parameters in the regression model (2.2). (𝒯 _1j,𝒯 _2j) are contained in α and γ, respectively. The estimators for α and γ are often formulated by ALEs (Newey and Powell, 1990, Robins and others, 1994). Let n = ∑_j n _j, an estimator $\hat{\mathbf{\alpha }}$ is asymptotically linear if The function B ₁ is referred to as the “influence function” of $\hat{\mathbf{\alpha }}$ in the sense of Casella and Berger (2002). The influence function B ₂ of an ALE for γ is defined similarly. ALE can be obtained by solving a system of estimating equations that are sums of n independent score contributions. Let ∑_j∑_{i = 1} ^n_j U _1ij = 0 be the set of estimating equations solved for α, and let ∑_j∑_{i = 1} ^n_j U _2ij = 0 be the set of estimating equations to be solved for γ. Let A ₁ = E[∂ U _1ij/∂ α] and A ₂ = E[∂ U _2ij/∂ γ]. Thus, the influence functions can be written as B _1ij = A ₁ ^{− 1} U _1ij and B _2i = A ₂ ^{− 1} U _2ij.

The random vectors U _1ij and U _2ij are i.i.d. with zero mean, but for the same i and j, U _1ij and U _2ij are correlated. The joint distribution of $\hat{\mathbf{\alpha }}$ and $\hat{\mathbf{\gamma }}$ is established using the central limit theorem, Slutsky's theorem, and the Cramer–Wold device:

(3.5)

where E(B _lij ^′ B _l^′ij), l,l ^′ = 1,2, are 2J×2J submatrices of the full covariance matrix. The limiting distribution of can be retrieved from the joint distribution of $(\hat{\mathbf{\alpha }},\hat{\mathbf{\gamma }})$.

This estimating equation approach generalizes the marginal score approach in Gail and others (2000). In particular, Gail and others (2000) start the estimation from the marginal distribution of Y and X in the treatment arm and the control arm, separately. Thus, a distributional assumption is needed for Y and X, for example, a Weibull distribution for time-to-event outcomes. Our approach first estimates the marginal treatment effects separately. The joint distribution of treatment effects are obtained by estimation equation theory, thus applicable to a more general class of models, for example, a GLM, correlated data modeled by generalized estimating equations (GEEs), and censored time-to-event data fitted by a Cox proportional hazard model. The linearization of score functions and regularity conditions in each of these models can be found in respective literature (McCullagh, 1983, Liang and Zeger, 1986, Lin and Wei, 1989). Among these, the estimating functions and influence functions for a logistic regression or GEEs are immediate to compute, while the linearization of score functions for a Cox model requires extra calculation, see, for example, Theorem 2.1 of Lin and Wei (1989).

3.2. REML estimates of λ ₀, λ ₁, and ρ

Consider both the between-trial variability (2.3) and the within-trial variability (2.4), the marginal distribution of when sample sizes within trials are large, can be approximated as

The estimation of (μ ₁,μ ₂,σ ₁ ²,σ ₂ ²,σ ₁₂) is achieved by maximum (approximate) likelihood of J pairs of , assuming Σ_j is known. The likelihood is approximate in the sense that it is based on the estimated asymptotic joint distribution of not the probability of observing individual data. This strategy is commonly used in meta-analysis (van Houwelingen and others, 2002).

To proceed to estimation, it is convenient to express the model as a bivariate random-effects model with J independent observations, T _j = β + b _j + u _j, j = 1,…,J, where β is the fixed-effects parameter, b _j is the random effect, and u _j is the estimation error. All these variables are bivariate normal, T _j = (𝒯 _1j,𝒯 _2j), β = (μ ₁,μ ₂), b _j = (b _1j,b _2j), u _j = (ε _j,ε_j). The distribution of b _j are bivariate normal 𝒩(0,D) in (2.3), and they are independent of the pair of estimation error u _j = (ε _j,ε_j), which have the distribution in (2.4). Let V _j = D + Σ_j denote the variance of T _j. If we view T _j as observables, the model resembles a random-effects model for longitudinal data, except that we assume that the distributions of ε _j and ε_j are known. Maximum likelihood (ML) estimates and REML estimates can be computed by an EM algorithm (Laird and Ware, 1982, Laird, 1982), treating b _j as missing data. REML estimates of variance components are generally preferable to ML estimates, particularly when the sample size is small. We next describe the EM algorithm to estimate (μ ₁,μ ₂,σ ₁ ²,σ ₂ ²,σ ₁₂).

If we observe b _j, it is straightforward to estimate the variance components Thus, in the E-step, we compute the expectations of these sufficient statistics given current parameter estimates. In particular, these expectations for the REML estimates are computed as follows:

which includes the extra variability from estimating the fixed-effects parameters μ ₁ and μ ₂. In the M-step, the fixed-effects estimates and the random-effects estimates are updated,

as well as the variance components ${\hat{\sigma }}_1^2,{\hat{\sigma }}_2^2$, and ${\hat{\sigma }}_{12}$. From these estimates, we compute ${\widehat{\lambda }}_0$, ${\widehat{\lambda }}_1$, and $\widehat{\rho }$.

3.3. Parametric bootstrap

The variances for the estimates of fixed-effects parameters and variance components can be computed by inverting the information matrix, though the small number of trials or subgroups may render this approach inadequate. The profile likelihood approach typically yields a better coverage probability, though it assumes an asymptotic χ ² distribution for the likelihood ratio, and its computation is algebraically cumbersome for the compound parameters we have, for example, the correlation. We propose a parametric bootstrap procedure, which first samples J pairs of treatment effects from the bivariate normal distribution

and then reestimates (μ ₁,μ ₂,σ ₁ ²,σ ₂ ²,σ ₁₂) by the procedure in Section 3.2, assuming ${\hat{\Sigma }}_j$ is known. The 3 metrics we defined in Section 2 are computed correspondingly. The 2.5% and 97.5% quantiles of 1000 bootstrap samples were used as the lower bound and the upper bound of 95% CI.

4. SIMULATIONS

We evaluate the performance of the point estimates and the bootstrap CIs in small samples by simulations. We assume that both the surrogate and the clinical end points are binary. There are k trials with heterogeneity in treatment effects, each of which has equal sample size. We generate the individual-level data on treatment and surrogate and clinical end points in 2 steps. In the first step, the treatment effects on the surrogate and treatment effects on the clinical end point among K trials were generated by a bivariate normal distribution with mean (0.5,0.4), variance (σ ²,σ ²), and covariance σ ₁₂. In the second step, the treatment assignment was randomly assigned to each participant with probability 0.5. The probabilities of the clinical end point and the surrogate are generated by the logistic functions in (2.1) and (2.2), using varying intercepts in each trial and the treatment effects.

To study the performance of the 3 estimators, we vary 4 attributes of the total variability: the number of trials k, which represents the amount of information for estimating between-trial variability; the total sample size n, which for a fixed k controls the precision of the within-trial estimators; the between-trial variability σ ², which measures the magnitude of heterogeneity of treatment effects; and the between-trial covariance σ ₁₂, which determines the size of ρ. In Table 1, we evaluate the bias, the 95% coverage probability, and the type I error or power performance.

Table 1.

The small-sample performance of the 3 estimators in 1000 simulations. In each simulation, k trials were generated, each of which has n/k participants. The treatment effects for the surrogate and for the clinical end point among k trials are generated by the bivariate normal distribution with mean (0.5, 0.4), variance (σ ², σ ²), and correlation ρ. The numbers in bold font are the type I errors for respective parameters. CP, cyclophosphamide plus cisplatin

		k = 10						k = 20
		n = 2000			n = 8000			n = 2000			n = 8000
		λ 0	λ 1	ρ	λ 0	λ 1	ρ	λ 0	λ 1	ρ	λ 0	λ 1	ρ
σ 2 = 0.25, ρ = 0.0
	Bias	0.003	0.051	0.024	0.005	0.028	0.015	0.029	0.007	0.029	0.016	0.007	0.001
Likelihood	95% CP	0.956	0.952	0.867	0.891	0.911	0.844	0.980	0.981	0.940	0.937	0.943	0.907
Bootstrap	95% CP	0.953	0.970	0.970	0.912	0.940	0.940	0.973	0.981	0.981	0.951	0.964	0.964
	Type I error/power	0.168	0.030	0.030	0.322	0.060	0.060	0.171	0.019	0.019	0.469	0.036	0.036
σ 2 = 0.25, ρ = 0.4
	Bias	– 0.022	0.071	0.001	– 0.005	0.022	0.005	– 0.029	0.121	0.029	0.006	0.034	0.016
Likelihood	95% CP	0.964	0.948	0.873	0.906	0.902	0.832	0.978	0.970	0.942	0.947	0.944	0.892
Bootstrap	95% CP	0.963	0.984	0.983	0.920	0.935	0.935	0.966	0.981	0.978	0.972	0.966	0.950
	Type I error/power	0.064	0.082	0.082	0.120	0.213	0.213	0.078	0.099	0.099	0.147	0.279	0.279
σ 2 = 0.25, ρ = 0.8
	Bias	– 0.009	0.019	– 0.046	– 0.012	0.021	0.008	– 0.011	0.018	– 0.067	– 0.006	0.018	– 0.002
Likelihood	95% CP	0.977	0.950	0.936	0.946	0.925	0.823	0.986	0.958	0.977	0.972	0.946	0.876
Bootstrap	95% CP	0.953	0.962	0.953	0.953	0.970	0.970	0.976	0.977	0.991	0.970	0.970	0.984
	Type I error/power	0.047	0.296	0.296	0.053	0.747	0.747	0.023	0.229	0.229	0.030	0.893	0.893
σ 2 = 0.5, ρ = 0.0
	Bias	0.002	0.066	0.031	0.003	0.020	0.010	0.034	0.006	– 3 × 10−4	0.023	0.002	– 5 × 10−4
Likelihood	95% CP	0.941	0.923	0.853	0.903	0.905	0.848	0.967	0.962	0.904	0.944	0.945	0.907
Bootstrap	95% CP	0.936	0.944	0.944	0.899	0.941	0.941	0.957	0.953	0.953	0.940	0.965	0.965
	Type I error/power	0.189	0.056	0.056	0.303	0.059	0.059	0.276	0.047	0.047	0.477	0.035	0.035
σ 2 = 0.5, ρ = 0.4
	Bias	0.009	0.059	0.007	0.003	0.013	0.008	– 0.066	0.113	0.027	0.010	0.011	0.005
Likelihood	95% CP	0.956	0.937	0.856	0.903	0.900	0.835	0.978	0.934	0.904	0.944	0.941	0.898
Bootstrap	95% CP	0.941	0.952	0.953	0.904	0.926	0.926	0.963	0.937	0.956	0.957	0.961	0.960
	Type I error/power	0.074	0.159	0.159	0.129	0.223	0.223	0.114	0.181	0.181	0.169	0.294	0.294
σ 2 = 0.5, ρ = 0.8
	Bias	– 0.019	0.043	0.008	– 0.007	0.009	0.001	– 0.059	0.083	– 0.016	– 0.002	0.016	– 0.001
Likelihood	95% CP	0.982	0.940	0.848	0.948	0.914	0.846	0.992	0.922	0.914	0.974	0.947	0.877
Bootstrap	95% CP	0.967	0.982	0.994	0.930	0.948	0.920	0.943	0.937	0.986	0.961	0.964	0.953
	Type I error/power	0.033	0.576	0.576	0.070	0.807	0.807	0.057	0.590	0.590	0.039	0.966	0.966

The biases of the 3 estimators appear to be generally small and go to 0 when the sample size increases. For n = 2000 and k = 10, the bias can be at most 0.071 for ${\widehat{\lambda }}_1$ when σ ² = 0.25 and ρ = 0.4. The biases of $\widehat{\rho }$ and ${\widehat{\lambda }}_0$ are much smaller. When n = 8000, so that the within-trial variability is much smaller, the biases of all 3 estimators are negligible. A similar pattern is observed for the scenarios when k = 20. The number of trials, the magnitude of heterogeneity, and the size of ρ have little influence on the bias performance.

We compare the 95% coverage probabilities computed by 2 methods: one is the proposed bootstrap method and the other is the likelihood-based method used in Burzykowski and others (2001) that inverts information matrix of estimated variance components and uses the delta method. Almost in every setting, the likelihood-based method provides a smaller coverage for the correlation than the nominal value, while the coverages for λ ₀ and λ ₁ are much better behaved. The poor coverage for the correlation provided by the likelihood-based method is likely caused by the constraint ρ ² ≤ 1 as well as a small number of trials. For the bootstrap method, the 95% coverage probabilities are generally acceptable, sometimes conservative, given the random variation of 1000 simulations, except in a few settings for λ ₀, for example, σ ² = 0.5,ρ = 0,n = 8000,k = 10 and σ ² = 0.5,ρ = 0.4,n = 8000,k = 10. The main factor that impacts the performance of the coverage probability appears to be the number of trials: a larger number for k generally provides a better coverage probability as it should provide more information toward between-trial variance components.

The type I error of testing the hypothesis that a parameter is 0 appears to be satisfactory, see the numbers highlighted by the bold font in Table 1. Not surprisingly, the larger the effect size of ρ, the higher power one would get for testing ρ = 0 or λ ₁ = 0. For the same effect size, the between-trial variability appears to have a major influence on the power, particularly when the sample size is small per trial. The reason is that a larger between-trial variability will make it easier to separate between-trial and within-trial variability. The increment of the sample size acts similarly by reducing the within-trial variability. A larger number of trials will generally help to estimate the between-trial variability when the total sample size is fixed, except in one setting where σ ² = 0.25,ρ = 0.8,n = 2000. In this case, the increase of the number of trials is not able to offset the decrease of sample size in each trial.

In this particular example where both the surrogate and the clinical end points are binary, we also evaluate the potential of the likelihood approach using a GLMM. There are a number of procedures developed to avoid numerical integration involved in evaluating the likelihood in the presence of random effects. Among them, the popular penalized quasi-likelihood (PQL) method was chosen here since it is readily available in the S/R function glmmPQL() and it has been discussed in the meta-analysis context (Breslow and Clayton, 1993, van Houwelingen and others, 2002). Table 2 shows the percentage of converged simulations, the bias, and the 95% coverage probability among the converged simulations for the estimators computed by glmmPQL(). The simulation setup is exactly the same as that in Table 1. The convergence performance is poor when the sample size per trial is 100 or 200, so that it is hard to judge the bias and the coverage probability in these scenarios. The performance is much improved with sample size per trial reaching 400 or 800, perhaps more realistic in meta-analysis, and the biases for all 3 estimators appear to be small. However, the performance of the 95% coverage probability is not satisfactory, with too narrow coverage under a majority of simulation settings. This is particularly the case for the variance estimators of $\widehat{\rho }$ and ${\widehat{\lambda }}_1$. These observations are consistent with the previous reports on difficulties of PQL for mixed models for binary data (Rodríguez and Goldman, 1995, Engel, 1998).

Table 2.

The small-sample performance of the 3 estimators estimated by the glmmPQL() function in R. The simulation settings are exactly the same as those in Table 1. The bias and 95% coverage probability were evaluated among those simulations that attain convergence. CP, cyclophosphamide plus cisplatin

	k = 10						k = 20
	n = 2000			n = 8000			n = 2000			n = 8000
	λ 0	λ 1	ρ	λ 0	λ 1	ρ	λ 0	λ 1	ρ	λ 0	λ 1	ρ
σ 2 = 0.25, ρ = 0.0
% converged	23.8			93.3			6.3			89.3
Bias	0.032	– 0.006	– 0.008	– 0.001	0.014	0.010	0.057	0.007	6e-5	0.020	0.012	0.0001
95% CP	0.912	0.937	0.866	0.879	0.817	0.752	0.937	0.984	0.984	0.907	0.821	0.785
σ 2 = 0.25, ρ = 0.4
% converged	16.6			89.3			4.7			71.4
Bias	0.104	– 0.197	– 0.205	– 0.016	0.048	0.022	0.266	– 0.369	– 0.376	– 0.006	0.033	0.011
95% CP	0.892	0.813	0.903	0.908	0.802	0.739	0.617	0.596	0.979	0.940	0.959	0.814
σ 2 = 0.25, ρ = 0.8
% converged	3			70.2			1.3			15.0
Bias	0.178	– 0.383	– 0.471	– 0.043	0.081	0.072	0.471	– 0.799	– 0.800	– 0.005	0.005	– 0.005
95% CP	0.733	0.633	1.000	0.967	0.721	0.474	0.077	0.154	1.000	0.993	0.873	0.880
σ 2 = 0.5, ρ = 0.0
% converged	59.4			99.1			33.5			99.8
Bias	0.017	0.021	0.016	0.001	0.012	0.008	0.063	– 0.004	– 0.014	0.026	0.0002	– 0.001
95% CP	0.914	0.872	0.808	0.905	0.843	0.782	0.907	0.916	0.866	0.929	0.861	0.832
σ 2 = 0.5, ρ = 0.4
% converged	45.5			97.9			21.5			98.0
Bias	0.051	– 0.053	– 0.073	– 0.009	0.029	0.006	0.138	– 0.157	– 0.167	– 0.011	0.054	0.034
95% CP	0.925	0.846	0.831	0.933	0.839	0.779	0.907	0.916	0.866	0.956	0.855	0.809
σ 2 = 0.5, ρ = 0.8
% converged	10.5			85.6			1.0			47.7
Bias	0.089	– 0.130	– 0.162	– 0.021	0.046	0.022	0.250	– 0.231	– 0.279	– 0.009	0.017	0.007
95% CP	0.924	0.848	0.905	0.929	0.861	0.832	0.800	0.800	0.900	0.996	0.904	0.847

5. DATA EXAMPLES

5.1. The EXPLORE study

As alluded by Joffe and Greene (2009), if there is strong prior knowledge on subgroups among which treatment effects are different, we could use these subgroups as units of meta-analysis, based on the same conditions we elaborated for clinical centers. These analyses do not yield definitive evidence for evaluation of prediction for a new trial, yet they provide useful information of surrogacy when only a single trial is available. In HIV prevention trials, behavioral risk groups are often considered to have heterogeneity in prevention effects. We show an example in HIV prevention trials below.

The EXPLORE study was a multisite two-trial randomized controlled phase IIb trial designed to test the effect of a behavioral intervention in preventing acquisition of HIV infection among 4295 men who have sex with men in the United States (The EXPLORE Study Team, 2004). In addition to the primary HIV infection end point, secondary end points on behavioral risk-taking were assessed at twice-yearly visits, including serodiscordant unprotected receptive anal (SDURA) intercourse, serodiscordant unprotected anal (SDUA) intercourse, and unprotected anal (UA) intercourse. The HIV infection rate was 18.2% (95% CI − 4.7 to 36.0) lower in the intervention trial than the standard trial (The EXPLORE Study Team, 2004).

One of the secondary analyses planned is to evaluate self-reported changes in risk behaviors as a correlate of protection against acquisition of HIV. The primary end point is censored time-to-infection, and the surrogate is longitudinally collected behavior measures. Using our proposed methods, we assessed the correlation of 2 sets of treatment effects, one on time to HIV acquisition and the other on self-reported UA, SDUA, or SDURA, in subgroups defined by all combinations of alcohol consumption, depression, and noninjection-drug use (8 subgroups). These potential effect modifiers were prespecified before the primary analysis (The EXPLORE Study Team, 2004). Figure 1 shows the scatter plot of treatment effects on HIV acquisition in a Cox proportional hazard model and treatment effects on SDURA, the most sensitive one to the treatment among the 3 behavior measures, in a GEE model. The area of a circle is proportional to the size of the subgroup.

Fig. 1.

The EXPLORE study: the x-axis is the treatment effect (log odds ratio) on SDURA intercourse; the y-axis is the treatment effect (log hazard ratio) on time to HIV acquisition. Eight subgroups were formed based on alcohol consumption, depression, and noninjection-drug use. The sizes of circles are proportional to sample sizes in subgroups.

From Figure 1, there seems to be a linear pattern between 2 sets of treatment effects, except for one outlier group. SDURA might be a promising surrogate of HIV acquisition. However, our estimation yields ρ = 0.54 with 95% CI [ − 0.95,0.96], λ ₁ = 3.82 with 95% CI [ − 4.86,5.22], and λ ₀ = 0.5 with 95% CI [ − 1.13,0.85]. None of these criteria is statistically significant. The CIs for ρ and λ ₁ are unacceptably wide, possibly because there is insufficient sample size within subgroups or the heterogeneity of treatment effects is indeed small. If we perform a global test of varying treatment effects among 8 subgroups, neither end point yields a significant p-value. This result suggests that there is not enough evidence to support SDURA as a surrogate end point. Other than the sample size consideration, the lack of evidence may also come from reporting bias of behavioral risk or merely the fact that a single binary measure is not adequate to cover the entire profile of behavioral risk-taking in the past half-year.

5.2. A meta-analysis of advanced ovarian cancer trials

We also applied our method to the data from a meta-analysis of 4 randomized multicenter trials in advanced ovarian cancer (Ovarian Cancer Meta-analysis Project, 1991). These 4 trials compared 2 treatment modalities: cyclophosphamide plus cisplatin versus cyclophosphamide plus adriamycin plus cisplatin. The clinical end point is the survival time and the surrogate is the progression-free survival time. The data set was later updated to contain a minimum follow-up of 10 years in all trials (Ovarian Cancer Meta-analysis Project, 1998). Strong correlation (0.95) is observed between the survival time and the progression-free survival time.

A total of 50 clinical centers are available for analysis, including 2 smaller trials that are treated as 2 separate centers. These centers are used as units of the proposed meta-analysis. To allow within-center treatment effects to be estimated with reasonable precision, we first drop the centers with less than 3 patients on either treatment arm, then collapse the centers that have small sample sizes and similar treatment effects for progression-free survival time. The minimum sample size of a “center” was set to be 50. We thus used 12 “centers” in the analysis, with sample size ranges from 54 to 274. The rationale of collapsing smaller sites is 2-fold: first, the method relies on approximated likelihood of the asymptotic normal distribution; second, small sites will yield large within-center variability that leads to lack of precision for estimating between-center variability.

Figure 2 shows the treatment effects on the progression-free survival time and treatment effects on the survival time among 12 “centers.” This is clearly a linear trend between 2 sets of effects. The correlation is high (0.989) with a narrow 95% CI [0.896,0.999]. The slope is almost 1 with its 95% CI excluding 0. The intercept does not significantly differ from 0. These results suggest that the progression-free survival time is a valid surrogate for the survival time.

Fig. 2.

The ovarian cancer trials: the x-axis is the treatment effect (log hazard ratio) on time to progression or death; the y-axis is the treatment effect (log hazard ratio) on time to death. Twelve groups were formed by collapsing small clinical sites to have minimal within-group sample size 50. The sizes of circles are proportional to sample sizes in the groups.

Burzykowski and others (2001) assessed the surrogacy of the progression-free survival time, using copula models to describe the correlation between the survival time and the progression-free survival time. They considered a 2-step estimating procedure, though they used the Fisher information of the likelihood to compute the CI. They also exclude centers with less than 3 patients per arm, though they did not collapse small sites. In our analysis, if we do not collapse small centers, we found that the CIs are markedly wider, for instance, $\widehat{\rho }=0.67$ with 95% CI [ − 0.21,0.99], while their analysis yields R _trial ² = 0.95[0.76,1.14] in one of their models. The discrepancy may come from several sources: our estimating equation approach is different from, though more general than, the copula models they used; we used parametric bootstrap to obtain the CI, which is perhaps better fitted for a limited number of groups than the Wald CIs based on the Fisher information. The latter approach could yield the upper bound of the CI above 1.

6. DISCUSSION

The main contribution of our paper is that the proposed estimation method, based on estimating equations on marginal treatment effects, circumvents the difficulty of modeling the within-subject correlation of binary and survival outcomes in the meta-analysis approach. We propose a parametric bootstrap procedure to construct CI, which is particularly suitable to small numbers of subgroups or trials involved in meta-analysis. Such benefit can also be attained by applying a Bayesian approach like Daniels and Hughes (1997) to trial-level treatment effect estimates after the first step of our procedure.

One challenge to the meta-analytical evaluation of surrogacy is to have a number of previous trials on the study drug or “similar drugs” that presumably take action in the same biological pathway through the candidate surrogate. If there is significant heterogeneity among study populations, and a new trial is likely also conducted in these centers, it may be useful to use these clinical centers as units of meta-analysis, admitting that they do not provide the same level of evidence in surrogacy as multiple trials. In HIV prevention trials funded by National Institutes of Health clinical trial networks, study sites can be quite heterogeneous, for example, sites in United states and sites in Africa, representing different sexual risk groups and regional epidemics, and these sites are likely used again in the next prevention trial. In this context, it remains of interest to evaluate the potential of clinical sites as the units of the meta-analysis.

FUNDING

Fred Hutchinson Cancer Research Center Faculty Development Fund to J.Y.D.; grants from National Institute of Health (R01 AI089341, U01 AI068615, U01AI46749).

Two grants used for the funding of this paper were originally missed from the funding information. They have now been added.