Robust Estimation of the Proportion of Treatment Effect Explained by Surrogate Marker Information

Abstract

In randomized treatment studies where the primary outcome requires long follow-up of patients and/or expensive or invasive obtainment procedures, the availability of a surrogate marker that could be used to estimate the treatment effect and could potentially be observed earlier than the primary outcome would allow researchers to make conclusions regarding the treatment effect with less required follow-up time and resources. The Prentice criterion for a valid surrogate marker requires that a test for treatment effect on the surrogate marker also be a valid test for treatment effect on the primary outcome of interest. Based on this criterion, methods have been developed to define and estimate the proportion of treatment effect on the primary outcome that is explained by the treatment effect on the surrogate marker. These methods aim to identify useful statistical surrogates that capture a large proportion of the treatment effect. However, current methods to estimate this proportion usually require restrictive model assumptions that may not hold in practice and thus may lead to biased estimates of this quantity. In this paper, we propose a nonparametric procedure to estimate the proportion of treatment effect on the primary outcome that is explained by the treatment effect on a potential surrogate marker and extend this procedure to a setting with multiple surrogate markers. We compare our approach to previously proposed model-based approaches and propose a variance estimation procedure based on a perturbation-resampling method. Simulation studies demonstrate that the procedure performs well in finite samples and outperforms model-based procedures when the specified models are not correct. We illustrate our proposed procedure using a dataset from a randomized study investigating a group-mediated cognitive behavioral intervention for peripheral artery disease participants.

1. Introduction

Clinical trials aimed at identifying effective treatment and prevention strategies to reduce the risk of a clinical outcome often face a number of key challenges when estimating a treatment effect on outcome risk. In particular, studies often require long term follow-up of patients in order to observe a sufficient number of events to precisely estimate treatment effects [1, 2]. In such settings, the availability of a surrogate marker that could be used to estimate the treatment effect and could be observed earlier than the primary outcome or with less cost or invasiveness to the patient would potentially allow researchers to make conclusions regarding the treatment effect with less required follow-up time and/or less cost [3]. That is, validated surrogate markers could enable shorter randomized clinical trials and require smaller sample sizes, thus accelerating acquisition of clinical information [4].

In one of the most influential papers on the validation of surrogate markers, Prentice [5] defined a criterion for a valid surrogate marker which required that a test for treatment effect on the surrogate marker also be a valid test for treatment effect on the primary outcome of interest. Since his work, a substantial amount of research has led to the development of four major frameworks for evaluating and validating surrogate markers, as described in Joffe & Greene [6]: one based on conditioning on the observed surrogate marker, a second based on defining direct and indirect effects of the treatment on the primary outcome, a third based on a meta-analytic framework, and a fourth based on principal stratification, with the approach proposed by Prentice belonging to the first framework. Methods developed within the first two frameworks have often focused on defining and estimating the proportion of treatment effect on the primary outcome that is explained by the treatment effect on the surrogate marker. Motivated by the Prentice criterion, these methods aim to identify useful statistical surrogates (as opposed to principal surrogates [7]) as those which capture a large proportion of the treatment effect on the primary outcome, which is also the focus of this paper. However, available statistical methods to estimate this proportion have numerous limitations [8, 9, 10, 11]. In particular, current methods usually require restrictive model assumptions that may not hold in practice. For example, Freedman et al. [8] proposed to estimate this proportion by examining the change in the regression coefficient for treatment when the surrogate marker is added to a specified regression model. However, when this model is misspecified, the appropriate interpretation of this estimate is not clear [12, 13]. Wang & Taylor [9] propose a much more flexible approach to estimate the proportion of treatment effect explained by defining a quantity that attempts to capture what the effect of the treatment would be if the values of the surrogate marker in the treatment group were distributed as those in the control group. While modeling choices are still required, this approach accommodates various practical settings and has a causal interpretation under certain conditions [13].

It is of great interest to investigate estimation procedures that allow for more flexible assumptions and do not rely on the correct specification of multiple models. Furthermore, when there are multiple surrogate markers of interest, it is difficult to capture the complex relationships between the surrogate markers using restrictive model-based methods. The approach proposed by Freedman et al. [8] for the single marker setting can be easily extended to a multiple marker setting by examining the change in the regression coefficient for treatment when the surrogate markers are added to a specified regression model, though this approach still relies on the correct specification of the models. Xu & Zeger [14] have proposed methods to determine whether multiple markers can improve inference about the treatment effects on a clinical endpoint, but their approach requires specifying parametric models and using Markov Chain Monte Carlo to estimate model parameters.

In this paper, we propose a robust estimation procedure to estimate the proportion of treatment effect on the primary outcome that is explained by the treatment effect on one or more potential surrogate markers in order to identify useful statistical surrogates. For brevity, we will refer to ‘the proportion of treatment effect on the primary outcome that is explained by the treatment effect on a surrogate marker’ as ‘the proportion of treatment effect explained by a surrogate’. In Section 2 we introduce our setup and definitions in a potential outcomes framework and describe the quantity we aim to estimate. In Section 3 we propose our estimation procedure in a single marker and multiple marker setting. We first describe available approaches for estimating the quantity including the model-based approach proposed by Freedman et al. [8] and a more flexible though still model-based approach proposed by Wang & Taylor [9]. We propose to estimate the proportion of treatment effect explained by a single potential surrogate marker using a nonparametric approach and then extend this procedure to estimate the proportion of treatment effect explained by a combination of multiple potential surrogate markers. We focus on the setting in which the surrogate marker(s) and primary outcome are fully observed for all individuals in both treatment groups and the surrogate marker (at least one surrogate marker in the multiple surrogate case) is continuous, while the primary outcome can be any general fully observed (uncensored) outcome. In Section 4 we describe the asymptotic properties of our estimates and propose variance estimation procedures. In Section 5 we investigate the finite sample properties of our estimation procedure and compare our proposed procedure to other available methods using simulation studies. In Section 6 we illustrate this procedure using a dataset from a randomized study investigating a group-mediated cognitive behavioral intervention for peripheral artery disease participants.

2. Setup and Definitions in a Causal Inference Framework

Let G be the binary treatment indicator with G = T for treatment and G = C for control (or placebo) and we assume throughout that subjects are randomly assigned to treatment or control at baseline. Let Y and S denote the primary outcome measure and surrogate marker measure, respectively, observed for all subjects i, i = 1, ..., n. Suppose Y is used to estimate and test for a treatment effect, but that Y is expensive or invasive to obtain while S is less expensive or invasive or that S can be obtained earlier than Y. We aim to measure the surrogacy of S by estimating the proportion of treatment effect explained by S.

To define our quantity of interest, we use potential outcomes notation such that Y^(g) and S^(g) denote the primary outcome and surrogate marker under treatment G = g. That is, Y^(T), Y^(C), S^(T) and S^(C) denote the measures for the primary outcome under the treatment, primary outcome under the control, surrogate marker under the treatment and surrogate marker under the control, respectively. In practice, we only observe (Y, S) = (Y^(T), S^(T)) or (Y^(C), S^(C)) depending on whether G = T or C.

Throughout, we define the treatment effect, Δ, as the expected difference in Y under the treatment compared to Y under the control, Δ = E(Y^(T) − Y^(C)). We aim to measure the surrogacy of a potential surrogate marker using contrasts between the actual treatment effect on Y and the residual treatment effect that would be observed if the surrogate marker is not affected by the treatment. The residual treatment effect can be defined as

$${\Delta }_S={\int }_{-\infty }^{\infty }E(Y^{\left(T\right)}-Y^{\left(C\right)}\mid S^{\left(T\right)}=S^{\left(C\right)}=s)d{F}_C\left(s\right)={\int }_{-\infty }^{\infty }{\Delta }_S\left(s\right)d{F}_C\left(s\right),$$ (1)

where Δ_S(s) = E(Y^(T) − Y^(C)|S^(T) = S^(C) = s) and F_C(·) is the marginal cumulative distribution function of S^(C), the surrogate marker measure under the control. Note that F_C(s) could similarly be replaced by F_T(s), the marginal cumulative distribution function of S^(T) and we assume that the support of S_C and S_T are the same. However, Δ_S(s) is in general not identifiable since S^(T) and S^(C) can not be observed simultaneously. To circumvent this difficulty, we assume that

$$Y^{\left(T\right)}\perp S^{\left(C\right)}\mid S^{\left(T\right)}\text{and}{Y}^{\left(C\right)}\perp S^{\left(T\right)}\mid S^{\left(C\right)}.$$ (2)

That is, given the surrogate marker value in one group, the surrogate marker value in the other group becomes noninformative to the potential response in the current group. Under assumption (2),

$$\begin{array}{c}\hfill {\Delta }_S\left(s\right)=E(Y^{\left(T\right)}\mid S^{\left(T\right)}=s)-E(Y^{\left(C\right)}\mid S^{\left(C\right)}=s)\text{and}\hfill \\ \hfill {\Delta }_S={\int }_{-\infty }^{\infty }E(Y^{\left(T\right)}\mid S^{\left(T\right)}=s)d{F}_C\left(s\right)-{\int }_{-\infty }^{\infty }E(Y^{\left(C\right)}\mid S^{\left(C\right)}=s)d{F}_C\left(s\right),\hfill \end{array}$$ (3)

where the first term in (3) can be interpreted as the expected outcome under treatment if the treatment has no effect on the surrogate marker and the residual treatment effect Δ_S(·) can be used to measure the surrogacy of S. Note that Δ_S given in (3) only depends on observed quantities and can be viewed as a measure for the direct treatment effect beyond the surrogate marker even without assumption (2). Further discussions on the interpretation of Δ_S can be found in Section 7.

Thus, the proportion of treatment effect explained by the surrogate marker, which we denote by R_S, can be expressed using a contrast between Δ_S and Δ:

$$R_S=\{\Delta -{\Delta }_S\}∕\Delta =1-{\Delta }_S∕\Delta .$$ (4)

This definition is not new and has been proposed by Wang & Taylor [9]. In addition, Taylor et al. [13] provides a detailed description and assessment of the causal interpretation that is possible using the proportion of treatment effect explained by a surrogate marker as a measure of surrogacy. In this paper, we focus on nonparametrically estimating this proportion with a continuous surrogate marker. Informally, we use R_S to measure the extent to which the treatment effect on the surrogate marker captures information about the treatment effect on Y by comparing the total treatment effect with the hypothetical treatment effect when there is no effect of treatment on S i.e. when F_T(s) = F_C(s). In the next section we describe previously proposed methods to estimate this quantity when a single continuous surrogate marker is available which have generally focused on model-based estimators, and then describe our proposed estimation procedure in this setting. The following section then extends this methodology to a setting with multiple surrogate markers.

3. Estimation

3.1. Single Surrogate Marker

We first focus on a setting where a single continuous surrogate marker, S, is available and our goal is to estimate the proportion of treatment effect explained by S, R_S. Let the treatment effect measure of interest be the expected difference in outcome under treatment compared to control, Δ = E(Y^(T) − Y^(C)) = E(Y^(T)|G = T) − E(Y^(C)|G = C) assuming that we are in a randomized treatment setting. The observed data consist of n_T independent identically distributed (i.i.d) copies of (Y^(T), S^(T), {(Y_Ti, S_Ti), i = 1, ···, n_T}, from the treatment group G = T and n_C i.i.d copies of (Y^(C), S^(C)), {(Y_Ci, S_Ci), i = 1, ···, n_C}, from the treatment group G = C. One can then estimate the treatment effect as

$$\widehat{\Delta }=n_T^{-1}\sum _{i=1}^{n_T}{Y}_{Ti}-n_C^{-1}\sum _{i=1}^{n_C}{Y}_{Ci}.$$

Since R_S = 1 − Δ_S/Δ, we now focus on estimating Δ_S, the residual treatment effect. As expressed in (3), Δ_S aims to capture the expected difference in outcome if the distribution of the surrogate marker in the treatment group was the same as the distribution of the surrogate marker in the control group. One model-based approach to estimate Δ_S, proposed by Wang & Taylor [9], is to specify models for E(Y^(g)|S^(g)), g = T, C such as:

$$E(Y^{\left(C\right)}\mid S^{\left(C\right)})={\beta }_0+{\beta }_1{S}^{\left(C\right)}\text{and}E(Y^{\left(T\right)}\mid S^{\left(T\right)})=({\beta }_0+{\beta }_2)+{\beta }_1{S}^{\left(T\right)}$$ (5)

It can be shown that if this model is correctly specified, Δ_S = β₂. Thus, reasonable estimates for Δ_S and R_S could be ${\widehat{\beta }}_2$ and $1-{\widehat{\beta }}_2∕\widehat{\Delta }$, respectively. This approach is equivalent to that proposed by Freedman et al. [8] where an estimate for the proportion of treatment effect explained by a surrogate is obtained by fitting the following two regression models:

$$E(Y\mid G)={\gamma }_0+{\gamma }_{1}I(G=T)\text{and}E(Y\mid G,S)={\gamma }_{0S}+{\gamma }_{1S}I(G=T)+{\gamma }_{2S}S$$ (6)

and estimating R_S as ${\widehat{R}}_S^F=1-{\widehat{\gamma }}_{1S}∕{\widehat{\gamma }}_1$ with ${\widehat{\gamma }}_{1S}$ and ${\widehat{\gamma }}_1$ being the estimators of γ_1S and γ₁, respectively. Here we have used ^F to indicate that this is the estimate based on Freedman's proposed approach. When all the specified models are linear as in (5) and (6), ${\widehat{R}}_S^F=1-{\widehat{\gamma }}_{1S}∕{\widehat{\gamma }}_1=1-{\widehat{\beta }}_2∕\widehat{\Delta }$.

Alternatively, to allow for additional flexibility, Wang & Taylor [9] suggest that one could include an interaction term when specifying the models used to estimate Δ_S:

$$\begin{array}{c}\hfill E\left(S^{\left(C\right)}\right)={\alpha }_0\text{and}E\left(S^{\left(T\right)}\right)-E\left(S^{\left(C\right)}\right)={\alpha }_1,\hfill \\ \hfill E(Y^{\left(C\right)}\mid S^{\left(C\right)})={\beta }_0+{\beta }_1{S}^{\left(C\right)}\text{and}E(Y^{\left(T\right)}\mid S^{\left(T\right)})=({\beta }_0+{\beta }_2)+({\beta }_1+{\beta }_3)S^{\left(T\right)}.\hfill \end{array}$$ (7)

It can be shown that when these models hold, Δ_S = β₂ + β₃α₀. Thus, reasonable estimates for Δ_S, Δ, and R_S using this approach would be

$${\widehat{\Delta }}_S^M={\widehat{\beta }}_2+{\widehat{\beta }}_3{\widehat{\alpha }}_0,\widehat{\Delta }={\widehat{\beta }}_2+{\widehat{\beta }}_1{\widehat{\alpha }}_1+{\widehat{\beta }}_3({\widehat{\alpha }}_0+{\widehat{\alpha }}_1)\text{and}{\widehat{R}}_S^M=1-{\widehat{\Delta }}_S^M∕\widehat{\Delta },$$

where we have used ^M to denote this as the flexible model-based estimator.

While the latter approach is more flexible in terms of model specification given the interaction term, both model-based approaches rely on correct model specification to obtain a consistent estimate. In particular, the limitations of using ${\widehat{R}}_S^F$ as an estimate of the proportion of treatment effect explained by a surrogate have been widely noted and include the reliance of the estimate on the correct specification of both models in (6), which is often not practical, and the difficulty with interpreting the estimate when one or both models do not hold [10, 12, 9].

To develop a more robust approach, we instead propose an estimation procedure to estimate R_S that does not require any model specification and instead nonparametrically estimates Δ_S. Specifically, we propose to estimate μ_T(s) = E(Y^(T)|S^(T) = s) nonparametrically using kernel smoothing and we denote the resulting estimator as ${\widehat{\mu }}_T\left(s\right)$. That is,

$${\widehat{\mu }}_T\left(s\right)=\frac{{\sum }_{i=1}^{n_T}{K}_h(S_{Ti}-s)Y_{Ti}}{\sum _{i=1}^{n_T}{K}_h(S_{Ti}-s)}$$

where S_Ti is the observed S^(T) for person i, Y_Ti is the observed Y^(T) for person i, K(·) is a smooth symmetric density function with finite support, K_h(·) = K(·/h)/h and h is a specified bandwidth. As in most nonparametric functional estimation procedures, the choice of the smoothing parameter h is critical. To eliminate the impact of the bias of the conditional mean function on the resulting estimator, we require the standard undersmoothing assumption of $h=O\left(n_{n_T}^{-\delta }\right)$ with δ ∈ (1/4, 1/2). To obtain an appropriate h we first use the bandwidth selection procedure given by Scott [15] to obtain h_opt; and then we let $h=h_{opt}{n}_T^{-c_0}$ for some c₀ ∈ (1/20, 3/10) to ensure the desired rate for h. In all numerical examples, we chose c₀ = 0.25. We then estimate Δ_S as

$$\begin{array}{cc}\hfill {\widehat{\Delta }}_S=& n_C^{-1}\sum _{i=1}^{n_C}{\widehat{\mu }}_T\left(S_{Ci}\right)-n_C^{-1}\sum _{i=1}^{n_C}{Y}_{Ci}\hfill \\ \hfill =& \sum _{i=1}^{n_C}\frac{{\widehat{\mu }}_T\left(S_{Ci}\right)-Y_{Ci}}{n_C},\hfill \end{array}$$ (8)

where S_Ci is the observed S^(C) for person i and Y_Ci is the observed Y^(C) for person i. Note that the second term of (8) simply estimates the mean response among those with G = C while the first term attempts to estimate the hypothetical mean response that would have been observed if the distribution of S among those with G = T was the same as that of those with G = C. In this way, ${\widehat{\Delta }}_S$ is an estimate of Δ_S, the residual treatment effect after removing the treatment effect attributable to S. Lastly, we estimate R_S as

$${\widehat{R}}_S=1-{\widehat{\Delta }}_S∕\widehat{\Delta }.$$

Recall from above that it was assumed that the supports of S^(T) and S^(C) are the same. Since μ_T(s) is only identifiable on the support of S^(T) without additional parametric assumption, this assumption is necessary to estimate $\int {\mu }_T\left(s\right)d{F}_C\left(s\right)$. Otherwise, if the supports of S^(T) and S^(C) are not dramatically disparate, one may alternatively consider a modified version of this estimation method that uses appropriate spline or local linear smoothing methods [16, 17], which allow mild extrapolation of μ_T(s) beyond the support of S^(T).

3.2. Multiple Surrogate Markers

While work in the area of surrogate marker research has alluded to the need for a valid estimate of the proportion of treatment effect explained by a set of multiple surrogate markers, limited work has been done to propose and investigate robust estimates in this setting [12, 14]. One potential approach would be an extension of Freedman's estimate obtained by fitting the following two models:

$$E(Y\mid G)={\gamma }_0+{\gamma }_{1}I(G=T)\text{and}E(Y\mid G,\mathbf{S})={\gamma }_{0S}+{\gamma }_{1S}I(G=T)+{\gamma }_{2S}^{\prime }\mathbf{S}$$ (9)

where S is the vector of candidate surrogate markers, and estimating the surrogacy by

$${\widehat{R}}_{\mathbf{S}}^F=1-\frac{{\widehat{\gamma }}_{1S}}{\widehat{{\gamma }_1}}$$

where, again, ${\widehat{\gamma }}_1$ and ${\widehat{\gamma }}_{1S}$ are estimators for the corresponding regression coefficients.

Alternatively, one may generalize the definition of R_S for a single marker, described above, to multiple markers in a straightforward manner:

$$R_{\mathbf{S}}=1-\frac{{\Delta }_{\mathbf{S}}}{\Delta },$$ (10)

where

$${\Delta }_{\mathbf{S}}=\int E(Y^{\left(T\right)}-Y^{\left(C\right)}\mid {\mathbf{S}}^{\left(T\right)}={\mathbf{S}}^{\left(C\right)}=\mathbf{s})d{F}_C\left(\mathbf{s}\right)=\int E(Y^{\left(T\right)}\mid {\mathbf{S}}^{\left(T\right)}=\mathbf{s})d{F}_C\left(\mathbf{s}\right)-E\left(Y^{\left(C\right)}\right)$$

under the assumption that

$$Y^{\left(T\right)}\perp {\mathbf{S}}^{\left(C\right)}\mid {\mathbf{S}}^{\left(T\right)}\text{and}{Y}^{\left(C\right)}\perp {\mathbf{S}}^{\left(T\right)}\mid {\mathbf{S}}^{\left(C\right)}$$

where F_C(s) is the marginal cumulative distribution function of S^(C).

If the model (9) is correctly specified, it is not difficult to show that Freedman's estimate is a consistent estimator of R_S. However, since (9) may not be correctly specified, we consider alternative approaches. To estimate Δ_S, one could consider a flexible model-based approach where models for E(Y|G, S) and E(S_j|G) for each S_j in S = {S₁, ...S_p} (where p is the number of surrogate markers) are specified and obtain an estimate of Δ_S assuming these models are correct. Without loss of generality, consider the case where there are three surrogate markers, S = {S₁, S₂, S₃} and one specifies the following linear models:

$$E(Y^{\left(C\right)}\mid {\mathbf{S}}^{\left(C\right)})={\beta }_0+{\beta }_1{S}_1^{\left(C\right)}+{\beta }_2{S}_2^{\left(C\right)}+{\beta }_2{S}_3^{\left(C\right)},$$ (11)

$$E(Y^{\left(T\right)}\mid {\mathbf{S}}^{\left(T\right)})=({\beta }_0+{\beta }_4)+({\beta }_1+{\beta }_5)S_1^{\left(T\right)}+({\beta }_2+{\beta }_6)S_2^{\left(T\right)}+({\beta }_3+{\beta }_7)S_3^{\left(T\right)},$$ (12)

and

$$E\left(S_j^{\left(C\right)}\right)={\alpha }_j,j=1,2,3.$$ (13)

It can be shown that when these models hold

$${\Delta }_{\mathbf{S}}={\beta }_4+{\beta }_5{\alpha }_1+{\beta }_6{\alpha }_2+{\beta }_7{\alpha }_3.$$ (14)

Thus, reasonable estimates for Δ_S and R_S here would be easily obtained by replacing the unknown regression coefficients in (14) by their consistent estimators. We denote the resulting estimators by ${\widehat{\Delta }}_{\mathbf{S}}^M$ and ${\widehat{R}}_{\mathbf{S}}^M$, similar to the single marker setting.

With this approach, one could now define a single “pseudo-marker”

$$W=({\beta }_0+{\beta }_4)+({\beta }_1+{\beta }_5)S_1+({\beta }_2+{\beta }_6)S_2+({\beta }_3+{\beta }_7)S_3,$$

using (12). If the surrogacy of S is defined by (10), then the surrogacy of S is equivalent to that of W when W = μ_T(S) = E(Y^(T)|S^(T) = S). Here, the surrogacy of the one-dimensional “marker” W is defined by (3) and (4). The formal justification of this claim is given in Appendix A of the Supplementary Materials. Therefore, we can claim that W has the full surrogacy of all three markers combined, i.e., R_W = R_S, which can be estimated based on (14). However, the validity of this approach, no matter how flexible the relevant regression model is, still depends on the correct specification of the model. Under a misspecified model, R_W is different from R_S and, more importantly, the model-based estimator ${\widehat{R}}_{\mathbf{S}}^M$ is not a valid estimator of either of them.

An alternative way to estimate Δ_S is to employ a nonparametric procedure. However, when the dimension of S (the number of potential surrogate markers) is greater than two, a completely nonparametric procedure is infeasible due to the curse of dimensionality [18]. Instead, we propose to use a two-stage procedure combining the aforementioned model-based approach and the nonparametric estimation procedure proposed in Section 3.1. Specifically, our proposed two-stage procedure is based on a dimension reduction approach where we focus on the conditional distribution of Y^(T)|S^(T) first. That is, we employ a working semiparametric model such as

$$E(Y^{\left(T\right)}\mid {\mathbf{S}}^{\left(T\right)}=\mathbf{s})=g\left({\beta }^{\prime }\mathbf{s}\right)$$ (15)

to reduce the dimension of S at the first stage, where g(·) is a monotone increasing function given a priori. Even when the working model is misspecified, the resulting estimator $\widehat{\beta }$ based on {(Y_Ti, S_Ti), i = 1, ···, n_T} converges to a deterministic limit β₀ in probability as n_T → ∞ under general regularity conditions. Other commonly used regression models such as the generalized transformation model can also be employed to estimate the conditional expectation E(Y^(T)|S^(T)) either directly or indirectly. Without loss of generality, we consider $Q={\beta }_0^{\prime }\mathbf{S}$ as the new surrogate marker of interest and in the second stage, apply the approach proposed in the single marker setting to estimate its surrogacy. To be specific, we nonparametrically estimate E(Y^(T)|Q_T = q) where $Q_T={\beta }_0^{\prime }{\mathbf{S}}^{\left(T\right)}$ as ${\widehat{\mu }}_T\left(q\right)$ based on $\left\{(Y_{Ti},{\widehat{Q}}_{Ti}={\widehat{\beta }}^{\prime }{\mathbf{S}}_{Ti}),i=1,\cdots ,n_T)\right\}$ and then estimate Δ_S as

$${\widehat{\Delta }}_{\mathbf{S}}=\sum _{i=1}^{n_C}\frac{{\widehat{\mu }}_T\left({\widehat{Q}}_{Ci}\right)-Y_{Ci}}{n_C},$$

where ${\widehat{Q}}_{Ci}={\widehat{\beta }}^{\prime }{\mathbf{S}}_{Ci}$. We estimate the proportion of treatment effect captured by S via Q, R_S as ${\widehat{R}}_{\mathbf{S}}=1-{\widehat{\Delta }}_{\mathbf{S}}∕\widehat{\Delta }$. Regardless of the correct model specification of (15), ${\widehat{\Delta }}_{\mathbf{S}}$ will always be a consistent estimator of Δ_Q where

$${\Delta }_Q=\int E(Y^{\left(T\right)}\mid Q_T=q)d{F}_{Q_C}\left(q\right)-E\left(Y^{\left(C\right)}\right)$$

and F_QC(·) is the cumulative distribution function of $Q_C={\beta }_0^{\prime }{\mathbf{S}}^{\left(C\right)}$ and ${\widehat{R}}_{\mathbf{S}}$ will always be a consistent estimator of R_Q = 1 − Δ_Q/Δ, the surrogacy of $Q={\beta }_0^{\prime }\mathbf{S}$. While the estimator ${\widehat{R}}_{\mathbf{S}}$ is constructed to estimate the full surrogacy of S, it may not approximate the latter well if the working model (15) fails to characterize the dependence of Y^(T) on S^(T). Therefore, in practice one may want to fit a more flexible regression model than (15), e.g. (with a slight abuse of notation) one could assume a working regression model

$$E(Y^{\left(T\right)}\mid {\mathbf{S}}^{\left(T\right)})=g\left\{{\beta }^{\prime }\mathbf{Z}\left({\mathbf{S}}^{\left(T\right)}\right)\right\},$$

where Z(S) is a known transformation of the marker S including, for example, interactions and nonlinear transformations of the components of S. In the end, we may use $\widehat{Q}={\widehat{\beta }}^{\prime }\mathbf{Z}\left(\mathbf{S}\right)$ as the new surrogate marker for the treatment effect and ${\widehat{R}}_{\mathbf{S}}$ as the estimator for its surrogacy. Note that even though we may not capture the full surrogacy of S, $\widehat{Q}$ and ${\widehat{R}}_{\mathbf{S}}$ could still serve as a good surrogate marker and an accurate assessment of its surrogacy, respectively.

4. Inference and Variance Estimation

It can be shown that under suitable regularity conditions the proposed estimates ${\widehat{R}}_S$ and ${\widehat{R}}_{\mathbf{S}}$ are consistent estimators of R_S and R_Q, respectively. Furthermore, $n^{\frac{1}{2}}\{{\widehat{R}}_S-R_S\}$ and $n^{\frac{1}{2}}\{{\widehat{R}}_{\mathbf{S}}-R_Q\}$ converge weakly to respective normal distributions. The theoretical justification is provided in Appendix B (single marker setting) and Appendix C (multiple marker setting) of the Supplementary Materials. The variances associated with these asymptotic distributions are difficult to estimate empirically. Therefore, we propose to estimate the variability of our proposed estimators and construct confidence intervals using a perturbation-resampling method, which has been successfully used in many applications [19, 20, 21, 22, 23]. This perturbation-resampling method is similar to the wild bootstrap [24, 25, 26].

Specifically, $\mathrm{let}\{{\mathbf{V}}^{\left(b\right)}={(V_{T1}^{\left(b\right)},\dots V_{T{n}_T}^{\left(b\right)},V_{C1}^{\left(b\right)},\dots V_{C{n}_C}^{\left(b\right)})}^{⊺},b=1,....,D\}$ be n × D independent copies of a positive random variables V from a known distribution with unit mean and unit variance, such as the standard exponential distribution. In the single marker setting, let

$$\begin{array}{c}\hfill {\widehat{\mu }}_T^{\left(b\right)}\left(s\right)=\frac{{\sum }_{i=1}^{n_T}{V}_{Ti}^{\left(b\right)}{K}_h(S_{Ti}-s)Y_{Ti}}{{\sum }_{i=1}^{n_T}{V}_{Ti}^{\left(b\right)}{K}_h(S_{Ti}-s)},\hfill \\ \hfill {\widehat{\Delta }}_S^{\left(b\right)}=\frac{{\sum }_{i=1}^{n_C}{V}_{Ci}^{\left(b\right)}\{{\widehat{\mu }}_T^{\left(b\right)}\left(S_{Ci}\right)-Y_{Ci}\}}{{\sum }_{i=1}^{n_C}{V}_{Ci}^{\left(b\right)}},\hfill \\ \hfill {\widehat{\Delta }}^{\left(b\right)}=\frac{{\sum }_{i=1}^{n_T}{V}_{Ti}^{\left(b\right)}{Y}_{Ti}}{{\sum }_{i=1}^{n_T}{V}_{Ti}^{\left(b\right)}}-\frac{{\sum }_{i=1}^{n_C}{V}_{Ci}^{\left(b\right)}{Y}_{Ci}}{{\sum }_{i=1}^{n_C}{V}_{Ci}^{\left(b\right)}},\hfill \end{array}$$

and

$${\widehat{R}}_S^{\left(b\right)}=1-{\widehat{\Delta }}_S^{\left(b\right)}∕{\widehat{\Delta }}^{\left(b\right)}.$$

Then one can estimate the distribution of

$$\left(\begin{array}{c}\hfill {\widehat{\Delta }}_S-{\Delta }_S\hfill \\ \hfill \widehat{\Delta }-\Delta \hfill \end{array}\right)$$

by the empirical distribution of

$$\left(\begin{array}{c}\hfill {\widehat{\Delta }}_S^{\left(b\right)}-{\widehat{\Delta }}_S\hfill \\ \hfill {\widehat{\Delta }}^{\left(b\right)}-\widehat{\Delta }\hfill \end{array}\right),b=1,\dots ,D.$$

For example, the variance of former can be approximated by $\widehat{\Sigma }$, the empirical variance of the latter conditional on the observed data. To construct a 100(1 − α)% confidence interval for R_S, one can calculate the 100(α/2)^th and 100(1 − α/2)^th empirical percentile of ${\widehat{R}}_S^{\left(b\right)}$ or estimate the variance of ${\widehat{R}}_S-R_S$ by the empirical variance of ${\widehat{R}}_S^{\left(b\right)}-{\widehat{R}}_S$ and construct the corresponding Wald-type confidence interval. An alternative is to employ Fieller's method for making inference on the ratio of two parameters [27, 28] and obtain the 100(1 − α)% confidence interval for R_S as

$$\left\{1-r:\frac{{({\widehat{\Delta }}_S-r\widehat{\Delta })}^2}{{\widehat{\sigma }}_{11}-2r{\widehat{\sigma }}_{12}+r^2{\widehat{\sigma }}_{22}}\le c_{\alpha }\right\},$$

where $\widehat{\Sigma }={\left({\widehat{\sigma }}_{ij}\right)}_{1\le i,j\le 2}$ and c_α is the (1 − α)th percentile of

$$\left\{\frac{{\{{\widehat{\Delta }}_S^{\left(b\right)}-(1-{\widehat{R}}_S){\widehat{\Delta }}^{\left(b\right)}\}}^2}{{\widehat{\sigma }}_{11}-2(1-{\widehat{R}}_S){\widehat{\sigma }}_{12}+{(1-{\widehat{R}}_S)}^2{\widehat{\sigma }}_{22}},b=1,\cdots ,C\right\}.$$

In the multiple marker setting, let ${\widehat{Q}}_{Ti}^{\left(b\right)}$ and ${\widehat{Q}}_{Ci}^{\left(b\right)}$ be the perturbed estimates of ${\widehat{Q}}_{Ti}$ and ${\widehat{Q}}_{Ci}$ estimated using weights V^(b), respectively. Specifically, if $\widehat{\beta }$ is obtained via solving the estimating equation

$$n_T^{-1}\sum _{i=1}^{n_T}{\mathbf{S}}_{Ti}\{Y_{Ti}-g\left({\beta }^{\prime }{\mathbf{S}}_{Ti}\right)\}=0$$

then ${\widehat{\beta }}^{\left(b\right)}$ is the root of the perturbed estimating equation

$$n_T^{-1}\sum _{i=1}^{n_T}{V}_{Ti}^{\left(b\right)}{\mathbf{S}}_{Ti}\{Y_{Ti}-g\left({\beta }^{\prime }{\mathbf{S}}_{Ti}\right)\}=0$$

and ${\widehat{Q}}_{gi}^{\left(b\right)}={\mathbf{S}}_{gi}^{\prime }{\widehat{\beta }}^{\left(b\right)}$, for g = T or C. Furthermore, the perturbed counterparts of ${\widehat{\mu }}_T(\cdot )$ and ${\widehat{\Delta }}_{\mathbf{S}}$ are

$${\widehat{\mu }}_T^{\left(b\right)}\left(q\right)=\frac{{\sum }_{i=1}^n{V}_{Ti}^{\left(b\right)}{K}_h\left({\widehat{Q}}_{Ti}^{\left(b\right)}-q\right)Y_{Ti}}{{\sum }_{i=1}^n{V}_{Ti}^{\left(b\right)}{K}_h\left({\widehat{Q}}_{Ti}^{\left(b\right)}-q\right)}\text{and}{\widehat{\Delta }}_{\mathbf{S}}^{\left(b\right)}=\frac{{\sum }_{i=1}^n{V}_{Ci}^{\left(b\right)}\left\{{\widehat{\mu }}_T^{\left(b\right)}\left({\widehat{Q}}_{Ci}^{\left(b\right)}\right)-Y_{Ci}\right\}}{{\sum }_{i=1}^n{V}_{Ci}^{\left(b\right)}},$$

respectively, and ${\widehat{R}}_{\mathbf{S}}^{\left(b\right)}$ can be defined as $1-{\widehat{\Delta }}_{\mathbf{S}}^{\left(b\right)}∕{\widehat{\Delta }}^{\left(b\right)}$. Again the distribution of

$$\left(\begin{array}{c}\hfill {\widehat{\Delta }}_{\mathbf{S}}-{\Delta }_Q\hfill \\ \hfill \widehat{\Delta }-\Delta \hfill \end{array}\right)$$

can be approximated by the empirical distribution of

$$\left(\begin{array}{c}\hfill {\widehat{\Delta }}_{\mathbf{S}}^{\left(b\right)}-{\widehat{\Delta }}_{\mathbf{S}}\hfill \\ \hfill {\widehat{\Delta }}^{\left(b\right)}-\widehat{\Delta }\hfill \end{array}\right),b=1,\dots ,D,$$

when the sample size is large. Therefore, we may make inference for R_Q similarly and obtain confidence intervals as described in the single marker setting. The theoretical justification for the perturbation-resampling procedure is provided in Appendix D of the Supplementary Materials.

5. Simulation Studies

5.1. Single Surrogate Marker

We examined several simulation settings to assess the performance of our proposed estimator, ${\widehat{R}}_S$, and proposed variance estimation procedure in the single surrogate marker setting. In addition, we compared the performance of our proposed estimator to both model-based estimators described in Section 3.1, ${\widehat{R}}_S^F$ and ${\widehat{R}}_S^M$. When the models that are required to be specified to obtain ${\widehat{R}}_S^F$ and ${\widehat{R}}_S^M$ are correct, we would expect the proposed estimate and both model-based to be unbiased though the proposed estimate may not be as efficient. However, when these models are not correctly specified, we would expect the model-based estimators to potentially yield biased estimates of R while ${\widehat{R}}_S$ should remain unbiased. Therefore, we examine two main simulation settings with varying values of R_S: one where the models specified in the estimation procedure for ${\widehat{R}}_S^M$ are correct and one where these models are not correctly specified. In both simulation settings the model specification required for ${\widehat{R}}_S^F$ does not hold because we allow for an interaction between the treatment and the surrogate. Throughout all simulations we use a normal density kernel for our proposed estimate. For each simulation setting we present results when n_T = n_C = 200 and n_T = n_C = 1000 to assess performance and sensitivity to sample size. For all three estimates (${\widehat{R}}_S$, ${\widehat{R}}_S^M$ and ${\widehat{R}}_S^F$), we estimate variance using our proposed perturbation approach and construct confidence intervals using both the quantiles of the perturbed estimates and Fieller's method.

In the first simulation setting, Setting (i), we generated data such that the models specified in (7) are correct. That is,

$$\begin{array}{cc}\hfill S^{\left(T\right)}\sim & N({\alpha }_0+1,2)\hfill \\ \hfill S^{\left(C\right)}\sim & N({\alpha }_0,1)\hfill \\ \hfill Y^{\left(T\right)}\mid S^{\left(T\right)}=s\sim & N(3+6s,3)\hfill \\ \hfill Y^{\left(C\right)}\mid S^{\left(C\right)}=s\sim & N(2+5s,3)\hfill \end{array}$$

where α₀ = 23, 5, and −1/3 correspond to R_S = 0.2, 0.5 and 0.9, respectively. That is, when α₀ = 5, the treatment effect on the surrogate marker explains approximately 50% of the overall treatment effect. The top portion of Table 1 shows the performance of our proposed estimator ${\widehat{R}}_S$ and both model-based estimators, ${\widehat{R}}_S^M$ and ${\widehat{R}}_S^F$ in this setting when R_S = 0.2, 0.5, and 0.9. The proposed estimation procedure performs well with very small bias, an estimated standard deviation using the perturbation approach close to the empirical standard deviation and a coverage level close to the nominal level of 95%. As expected, both ${\widehat{R}}_S$ and ${\widehat{R}}_S^M$ have very small bias and the efficiency loss demonstrated by the proposed estimation approach when the model specified by the flexible model-based approach is correct, is fairly mild. However, ${\widehat{R}}_S^F$ has poor performance with inadequate coverage and large bias compared to the other two estimators as was expected given that this estimation approach incorrectly assumes that there is no interaction between the treatment and the surrogate marker. Performance in general is better when n_T = n_C = 1000 compared to when n_T = n_C = 200 as expected, with the flexible model-based approach being slightly superior to the proposed method when the smaller sample size is used.

Table 1.

Performance of the proposed (${\widehat{R}}_S$) and model-based estimators (${\widehat{R}}_S^M$ and ${\widehat{R}}_S^F$) in terms of bias, empirical standard deviation (ESD), average standard deviation (ASD) obtained using the proposed perturbation-resampling approach, mean squared error (MSE) and coverage of the 95% confidence intervals obtained using both the quantiles of the perturbed estimates (Quantile) and Fieller's method (Fieller), in Setting (i) where there is a single surrogate marker and the models assumed by the flexible model-based estimator, ${\widehat{R}}_S^M$, are correctly specified, and Setting (ii) where there is a single surrogate marker and the specified models are not correct with n_T = n_S = 200 and with n_T = n_S = 1000.

Setting (i) Model Correctly Specified
nT = 200, nC = 200
	RS = 0.2			RS = 0.5			RS = 0.9
	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$
Bias	−0.0023	−0.0005	−0.0072	−0.0066	−0.0021	−0.0189	−0.0072	0.0009	−0.0295
ESD	0.0260	0.0259	0.0251	0.0457	0.0445	0.0432	0.0559	0.0506	0.0488
ASD	0.0262	0.0258	0.0250	0.0466	0.0450	0.0434	0.0561	0.0513	0.0494
MSE	0.0007	0.0007	0.0007	0.0021	0.0020	0.0022	0.0032	0.0026	0.0033
Coverage (Quantile)	0.946	0.941	0.938	0.945	0.941	0.914	0.926	0.942	0.885
Coverage (Fieller)	0.951	0.947	0.942	0.946	0.947	0.915	0.934	0.944	0.885

nT = 1000, nC = 1000
	RS = 0.2			RS = 0.5			RS = 0.9
	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$
Bias	−0.0003	0.0003	−0.0065	−0.0014	−0.0001	−0.0169	−0.0031	−0.0007	−0.031
ESD	0.0118	0.0115	0.0111	0.0207	0.0196	0.0188	0.0241	0.0214	0.0205
ASD	0.0117	0.0115	0.0111	0.0203	0.0196	0.0189	0.0238	0.0216	0.0208
MSE	0.0001	0.0001	0.0002	0.0004	0.0004	0.0006	0.0006	0.0005	0.0014
Coverage (Quantile)	0.948	0.947	0.899	0.939	0.946	0.840	0.936	0.934	0.676
Coverage (Fieller)	0.948	0.950	0.900	0.940	0.951	0.835	0.941	0.941	0.682

Setting (ii) Model Misspecified
nT = 200, nC = 200
	RS = 0.2			RS = 0.5			RS = 0.9
	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$
Bias	−0.0022	−0.0141	−0.0241	−0.0064	−0.0364	−0.0612	−0.007	−0.0629	−0.0844
ESD	0.0250	0.0290	0.0275	0.0404	0.0521	0.0492	0.0161	0.0436	0.0404
ASD	0.0255	0.0292	0.0277	0.0417	0.0525	0.0497	0.0176	0.0441	0.0414
MSE	0.0006	0.0010	0.0013	0.0017	0.0040	0.0062	0.0003	0.0059	0.0088
Coverage (Quantile)	0.946	0.912	0.842	0.948	0.883	0.737	0.937	0.562	0.246
Coverage (Fieller)	0.947	0.908	0.832	0.947	0.873	0.715	0.933	0.537	0.224

nT = 1000, nC = 1000
	RS = 0.2			RS = 0.5			RS = 0.9
	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$	R̂S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$
Bias	−0.0008	−0.0134	−0.0234	−0.0022	−0.0335	−0.0585	−0.0018	−0.0587	−0.0805
ESD	0.0116	0.0133	0.0126	0.0185	0.0236	0.0223	0.0071	0.0192	0.0177
ASD	0.0116	0.0133	0.0126	0.0185	0.0236	0.0224	0.0072	0.0191	0.0177
MSE	0.0001	0.0004	0.0007	0.0003	0.0017	0.0039	0.0001	0.0038	0.0068
Coverage (Quantile)	0.950	0.826	0.528	0.944	0.681	0.214	0.936	0.079	0.005
Coverage (Fieller)	0.951	0.818	0.521	0.948	0.673	0.201	0.942	0.067	0.002

In the second simulation setting, Setting (ii), we generated data such that the models specified in (7) are not correct. Specifically,

$$\begin{array}{cc}\hfill S^{\left(T\right)}\sim & \exp \left\{N(1.7,0.2)\right\}\hfill \\ \hfill S^{\left(C\right)}\sim & \exp \left\{N(1.62,0.1)\right\}\hfill \\ \hfill Y^{\left(T\right)}\mid S^{\left(T\right)}=s\sim & {\alpha }_0+{\alpha }_1{s}^2+{\alpha }_2\exp (s∕5)+\exp \left\{N(0,0.3)\right\}\hfill \\ \hfill Y^{\left(C\right)}\mid S^{\left(C\right)}=s\sim & 0.8s^2+0.2\exp (s∕5)+\exp \left\{N(0,0.3)\right\}\hfill \end{array}$$

where (α₀, α₁, α₂) = (20, 1, 0.5) corresponds to R_S ≈ 0.2, (α₀, α₁, α₂) = (0.5, 1, 0.5) corresponds to R_S ≈ 0.5, and (α₀, α₁, α₂) = (0, 0.82, 0.22) corresponds to R_S ≈ 0.9. The bottom portion of Table 1 shows the performance of our proposed estimator ${\widehat{R}}_S$ and both model-based estimators, ${\widehat{R}}_S^M$ and ${\widehat{R}}_S^F$ in this setting. As expected when the models are misspecified, both model-based estimators have rather large bias and poor coverage with higher bias and poorer coverage as R_S increases, while the proposed estimate is unbiased and has coverage levels close to the nominal level of 95%. For all estimators, the perturbation approach produces standard deviation estimates that are close to the empirical estimates. As expected, when n_T = n_C = 200, the estimates from the proposed method have larger biases than those in the larger sample size setting but still outperform the two model-based approaches.

5.2. Multiple Surrogate Markers

We also examined several simulation settings to assess the performance of our proposed estimator, ${\widehat{R}}_{\mathbf{S}}$, and proposed variance estimation procedure in the multiple surrogate marker setting, and compared the performance of our proposed estimator to the extension of Freedman's estimator, ${\widehat{R}}_{\mathbf{S}}^F$, and the model-based estimator, ${\widehat{R}}_{\mathbf{S}}^M$, described in Section 3.2. Since we assumed a working linear regression model for $E(Y^{\left(T\right)}\mid {\mathbf{S}}^{\left(T\right)}=\mathbf{s})={\beta }_0^{\prime }\mathbf{s}$, both the proposed and model-based estimators aim to estimate R_Q, the surrogacy of $Q={\beta }_0^{\prime }\mathbf{S}$. On the other hand, it can be shown that Freedman's method aims to estimate R_F, the surrogacy of ${\gamma }_{2\mathbf{S}}^{\prime }\mathbf{S}$, where γ_2S is the limit of the estimated regression coefficient of model (9) as the sample size goes to infinity. As in the previous section, when the specified models are correct, we would expect the proposed estimate and both model-based estimates to be unbiased though we expect some efficiency loss with our proposed estimator. However, when these models are not correctly specified, we would expect the proposed estimator and its associated inference are still valid for estimating R_Q, while neither ${\widehat{R}}_{\mathbf{S}}^M$ nor ${\widehat{R}}_{\mathbf{S}}^F$ are consistent for their respective quantities (R_Q and R_F, respectively). Therefore, we examine two main simulation settings with varying values of R_S: one where the specified models are correct and one where these models are misspecified. In both simulation settings the model specification required for ${\widehat{R}}_{\mathbf{S}}^F$ does not hold because we allow for an interaction between the treatment and the surrogate. We will also compare R_Q and R_F to the true surrogacy of S, R_S. Throughout all simulations we use a normal density kernel for our proposed estimate and for each simulation setting we present results when n_T = n_C = 200 and n_T = n_C = 1000 to assess performance and sensitivity to sample size, as in the single marker settings. For all three estimates (${\widehat{R}}_{\mathbf{S}}$, ${\widehat{R}}_{\mathbf{S}}^M$ and ${\widehat{R}}_{\mathbf{S}}^F$), we estimate variance using our proposed perturbation approach and construct confidence intervals using both the quantiles of the perturbed estimates and Fieller's method.

In the third simulation setting, Setting (iii), we generated data such that the models specified in (12) and (13) are correct. That is,

$$\begin{array}{cc}\hfill S_1^{\left(T\right)}\sim & N(6,4),S_1^{\left(C\right)}\sim N(5,1)\hfill \\ \hfill S_2^{\left(T\right)}\sim & \mathrm{Binom}\left(0.6\right),S_2^{\left(C\right)}\sim \mathrm{Binom}\left(0.4\right)\hfill \\ \hfill S_3^{\left(T\right)}\sim & \mathrm{Exp}\left(1\right),S_3^{\left(C\right)}\sim \mathrm{Exp}\left(2\right)\hfill \\ \hfill Y^{\left(T\right)}\mid {\mathbf{S}}^{\left(T\right)}={(s_1,s_2,s_3)}^{\prime }\sim & N({\alpha }_0+2s_1+{\alpha }_1{s}_2+{\alpha }_2{s}_3,9)\hfill \\ \hfill Y^{\left(C\right)}\mid {\mathbf{S}}^{\left(C\right)}={(s_1,s_2,s_3)}^{\prime }\sim & N(2+1.92s_1+3.5s_2+2.3s_3,9)\hfill \end{array}$$

where (α₀, α₁, α₂) = (17.5, 4, 2.5) corresponds to R_S = 0.2, (α₀, α₁, α₂) = (5.35, 4, 2.5) corresponds to R_S = 0.5, and (α₀, α₁, α₂) = (1.99, 3.6, 2.3) corresponds to R_S = 0.9. The top portion of Table 2 shows the performance of our proposed estimator ${\widehat{R}}_{\mathbf{S}}$ and both model-based estimators, ${\widehat{R}}_{\mathbf{S}}^M$ and ${\widehat{R}}_{\mathbf{S}}^F$ when R_S = 0.2, 0.5, and 0.9. The proposed estimation procedure performs well with very small bias, an estimated standard deviation using the perturbation approach close to the empirical standard deviation and a coverage level close to the nominal level of 95%. As expected, both ${\widehat{R}}_{\mathbf{S}}$ and ${\widehat{R}}_{\mathbf{S}}^M$ have very small bias and the efficiency loss demonstrated by the proposed estimation approach when the model specified by the flexible model-based approach is correct, is fairly mild. However, the 95% confidence interval based on ${\widehat{R}}_{\mathbf{S}}^F$ has inadequate coverage due to relatively large bias compared to the other two estimators. It is expected given that this estimation approach assumes there is no interaction between the treatment and the surrogate marker. As in the single marker setting, performances of all methods are in general better when n_T = n_C = 1000 compared to when n_T = n_C = 200, with the flexible model-based approach being slightly superior to the proposed method especially when the smaller sample size is used.

Table 2.

Performance of the proposed (${\widehat{R}}_{\mathbf{S}}$) and model-based estimators (${\widehat{R}}_{\mathbf{S}}^M$ and ${\widehat{R}}_{\mathbf{S}}^F$) in terms of bias, empirical standard deviation (ESD), average standard deviation (ASD) obtained using the proposed perturbation-resampling approach, mean squared error (MSE) and coverage of the 95% confidence intervals obtained using both the quantiles of the perturbed estimates (Quantile) and Fieller's method (Fieller), in Setting (iii) where there are three surrogate markers and the models assumed by the flexible model-based estimator, ${\widehat{R}}_{\mathbf{S}}^M$, are correctly specified, and Setting (iv) where there are three surrogate markers and the specified models are not correct with n_T = n_S = 200 and with n_T = n_S = 1000.

Setting (iii) Model Correctly Specified
nT = 200, nC = 200
	RS = 0.20			RS = 0.50			RS = 0.90
	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$
Bias	−0.0023	−0.0003	−0.0045	−0.0049	0.000	−0.0103	−0.0047	0.0041	−0.0014
ESD	0.0201	0.0189	0.0182	0.0423	0.0385	0.0367	0.0846	0.0772	0.0745
ASD	0.0200	0.0191	0.0183	0.0412	0.0389	0.0367	0.0814	0.0781	0.0746
MSE	0.0004	0.0004	0.0003	0.0018	0.0015	0.0015	0.0072	0.006	0.0055
Coverage (Quantile)	0.946	0.952	0.937	0.931	0.946	0.926	0.924	0.943	0.934
Coverage (Fieller)	0.950	0.951	0.941	0.936	0.949	0.932	0.929	0.942	0.938

nT = 1000, nC = 1000
	RS = 0.20			RS = 0.50			RS = 0.90
	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$
Bias	−0.0001	0.0004	−0.0039	−0.0001	0.0011	−0.0096	0.0005	0.0027	−0.0036
ESD	0.0091	0.0086	0.0081	0.0190	0.0175	0.0164	0.0364	0.0338	0.0323
ASD	0.0089	0.0085	0.0082	0.0184	0.0173	0.0163	0.0357	0.0338	0.0322
MSE	0.0001	0.0001	0.0001	0.0004	0.0003	0.0004	0.0013	0.0011	0.0011
Coverage (Quantile)	0.948	0.947	0.922	0.945	0.946	0.907	0.948	0.949	0.943
Coverage (Fieller)	0.950	0.948	0.923	0.948	0.947	0.908	0.948	0.950	0.945

Setting (iv) Model Misspecified
nT = 200, nC = 200
	RS = 0.20			RS = 0.50			RS = 0.90
	RQ = 0.206		RF = 0.211	RQ = 0.523		RF = 0.546	RQ = 0.924		RF = 0.926
	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$
Bias	−0.0031	0.0190	−0.0169	−0.0095	0.0462	−0.0941	−0.0138	0.0887	−0.0850
ESD	0.0259	0.0360	0.0283	0.0458	0.0742	0.0528	0.0298	0.0952	0.0664
ASD	0.0260	0.0351	0.0277	0.0463	0.0720	0.0517	0.0313	0.0934	0.0655
MSE	0.0007	0.0017	0.0011	0.0022	0.0076	0.0116	0.0011	0.0169	0.0116
Coverage (Quantile)	0.943	0.909	0.888	0.935	0.894	0.501	0.912	0.826	0.699
Coverage(Fieller)	0.941	0.924	0.884	0.936	0.907	0.480	0.944	0.847	0.704

nT = 1000, nC = 1000
	RS = 0.20			RS = 0.50			RS = 0.90
	RQ = 0.206		RF = 0.211	RQ = 0.523		RF = 0.546	RQ = 0.924		RF = 0.926
	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$	R̂ S	${\widehat{R}}_{\mathbf{S}}^M$	${\widehat{R}}_{\mathbf{S}}^F$
Bias	−0.0008	0.0186	−0.0171	−0.0024	0.0468	−0.0930	−0.0035	0.0875	−0.0859
ESD	0.0114	0.0155	0.0121	0.0196	0.0316	0.0224	0.0125	0.0411	0.0289
ASD	0.0117	0.0159	0.0125	0.0203	0.0323	0.0232	0.0126	0.0411	0.0289
MSE	0.0001	0.0006	0.0004	0.0004	0.0032	0.0091	0.0002	0.0094	0.0082
Coverage (Quantile)	0.956	0.788	0.719	0.960	0.701	0.012	0.926	0.443	0.142
Coverage(Fieller)	0.957	0.808	0.714	0.960	0.715	0.010	0.939	0.464	0.134

In the last simulation setting, Setting (iv), we generated data such that the models specified in (12) and (13) are not correct. Specifically,

$$\begin{array}{cc}\hfill S_1^{\left(T\right)}\sim & N(6,4),S_1^{\left(C\right)}\sim N(5,1)\hfill \\ \hfill S_2^{\left(T\right)}\sim & \mathrm{Binom}\left(0.6\right),S_2^{\left(C\right)}\sim \mathrm{Binom}\left(0.4\right)\hfill \\ \hfill S_3^{\left(T\right)}\sim & \mathrm{Exp}\left(1\right),S_3^{\left(C\right)}\sim \mathrm{Exp}\left(2\right)\hfill \\ \hfill Y^{\left(T\right)}\mid {\mathbf{S}}^{\left(T\right)}={(s_1,s_2,s_3)}^{\prime }\sim & {\alpha }_0+{\alpha }_1{s}_1^2{s}_2+{\alpha }_2{s}_1{s}_2{s}_3+\exp \left({\alpha }_3{s}_1{s}_2\right)+\exp \left\{N(0,0.09)\right\}\hfill \\ \hfill Y^{\left(C\right)}\mid {\mathbf{S}}^{\left(C\right)}={(s_1,s_2,s_3)}^{\prime }\sim & {\alpha }_4+{\alpha }_5{s}_1^2{s}_2+{\alpha }_6{s}_1{s}_2{s}_3+\exp \left({\alpha }_7{s}_1{s}_2\right)+\exp \left\{N(0,0.09)\right\}\hfill \end{array}$$

where (α₀, α₁, α₂, α₃, α₄, α₅, α₆, α₇) = (47, 0.7, 0.4, 0.2, 0.5, 0.7, 0.39, 0.2) corresponds to R_S ≈ 0.2, (α₀, α₁, α₂, α₃, α₄, α₅, α₆, α₇) = (9.3, 0.7, 0.4, 0.2, 0.5, 0.5, 0.3, 0.1) corresponds to R_S ≈ 0.5, and (α₀, α₁, α₂, α₃, α₄, α₅, α₆, α₇) = (1.2, 0.52, 0.32, 0.1, 0.5, 0.5, 0.3, 0.1) corresponds to R_S ≈ 0.9. The lower portion of Table 2 shows the performance of ${\widehat{R}}_{\mathbf{S}}$, ${\widehat{R}}_{\mathbf{S}}^M$, and ${\widehat{R}}_{\mathbf{S}}^F$ in this setting when R_S ≈ 0.2, 0.5, and 0.9. As noted above, since we assume a working linear regression model for $E(Y^{\left(T\right)}\mid {\mathbf{S}}^{\left(T\right)}=\mathbf{s})={\beta }_0^{\prime }\mathbf{s}$, both the proposed and flexible model-based estimators aim to estimate R_Q, the surrogacy of $Q={\beta }_0^{\prime }\mathbf{S}$, while Freedman's method aims to estimate R^F, the surrogacy of ${\gamma }_{2\mathbf{S}}^{\prime }\mathbf{S}$. Therefore, in Table 2 we provide both the true R_S (0.2,0.5 or 0.9) and (R_Q, R_F) and calculate bias, MSE, and coverage with respect to the quantity each estimator aims to estimate. As discussed in Section 3.2, when the working model (15) in our proposed procedure is not correct, the proposed estimator, ${\widehat{R}}_{\mathbf{S}}$, will always be a consistent estimator of R_Q but will only approximate the true R_S. In contrast, when the models in the flexible model-based procedure, (11) and (12), are not correct, the model-based estimator ${\widehat{R}}_{\mathbf{S}}^M$ is not a consistent estimate of either R_Q or R_S. In this simulation setting, none of the specified models are correct [hatwide] i.e. the working model for the proposed procedure is not correct and the models specified by the flexible model-based procedure and Freedman's approach are also not correct. The proposed estimation procedure outperforms both model-based estimators in terms of bias and MSE and has better coverage for R_Q. When R_S = 0.90 and the working model is misspecified, the coverage of our proposed estimator is slightly lower than the nominal level which may due to the potential inadequacy of the normal approximation, particularly when R_S is close to 1.

6. Example

To illustrate our proposed estimation procedure we use data from a study of a 6-month group-mediated cognitive behavioral (GMCB) intervention for peripheral artery disease (PAD) participants. Previous results from this study described in McDermott et al. [29] showed that the GMCB intervention, which promoted home-based walking exercise, improved distance covered in a 6-minute walk, 12 months after completing the intervention, compared to a control group. The intervention consisted of weekly visits to an exercise facility and incorporated group support and self-regulatory skills (n_T = 81) while the control group condition involved weekly on-site group meetings at a medical center where participants received health educational lectures on topics not related to exercise (n_C = 85). The primary outcome was the distance the participant completed after 6 minutes of walking up and down a 100-foot hallway at 12 months after randomization. There were three potential surrogate outcomes of interest which were obtained from the Walking Impairment Questionnaire (WIQ), a PAD-specific measure of self-reported limitations with 3 domains: walking distance, walking speed, and stair climbing (all on a scale from 0-100) [30]. Given that measurement of the primary outcome, the distance covered in 6 minutes, requires special supervision and attendance by the patients, we were interested in the questionnaire measures as potential surrogates as they would require fewer resources to collect.

For illustration, we estimate the proportion of treatment effect explained by each of the three potential surrogates alone (walking distance, walking speed and stair climbing from the WIQ), and then estimate the proportion of treatment effect explained by all three surrogates together using our proposed procedure and the model-based procedures. The overall treatment effect, defined as the difference in the change in distance covered in six minutes in the intervention group (average gain of 25.6 meters) compared to the control group (average loss of 7.4 meters), was 33.9 meters. Table 3 shows the resulting estimates of R_S for S equal to each of the three surrogate measures using the proposed estimator, ${\widehat{R}}_S$, the flexible model-based estimator, ${\widehat{R}}_S^M$, and Freedman's estimator, ${\widehat{R}}_S^F$, and corresponding standard deviation estimates and 95% confidence intervals. The estimated proportion of treatment effect explained varied substantially depending on the estimation procedure used. While none of the measures appear to explain a substantial proportion of the treatment effect, self-reported walking speed from the WIQ appears to capture the largest proportion of the treatment effect. Specifically, walking speed explains about 48% of the treatment effect using our proposed estimation procedure, 37% using the flexible model-based procedure and 17% using Friedman's approach. Walking distance from the WIQ also appears to explain a reasonable proportion of the treatment effect while stair climbing explains little of the treatment effect. The last portion of Table 3 shows the estimated quantities for all three surrogate measures together. With the flexible model-based procedure, one could use the estimated model parameters to construct a single “pseudo-marker” as described in Section 3.2 which would be:

$$W=9.80+0.33\left(\text{Walking Distance}\right)+0.95\left(\text{Walking Speed}\right)+0.44\left(\text{Stair Climbing}\right).$$

Using our proposed procedure, 48% of the treatment effect is explained by the three measures together while the flexible model-based procedure and Freedman's approach estimate the proportion of treatment effect explained as 38% and 18%, respectively. It is important to note that all procedures produce rather wide confidence intervals in this illustration, we discuss this further below.

Table 3.

Estimates, standard deviations (SD), and 95% confidence intervals (CI) using both the quantile method (Quantile) and Fieller's method (Fieller) for estimates of the proportion of treatment effect explained by each of three potential surrogate markers alone (walking distance, walking speed and stair climbing obtained from the WIQ) and the proportion explained by all three together, where the outcome of interest is change in 6 minute walking distance from baseline to 12 months using the proposed estimator, ${\widehat{R}}_S$, the flexible model-based estimator, ${\widehat{R}}_S^M$, and Freedman's estimator, ${\widehat{R}}_S^F$

	R̂ S	${\widehat{R}}_S^M$	${\widehat{R}}_S^F$
Potential surrogate: walking distance from the WIQ
Estimate	0.4403	0.1701	0.0753
SE	0.22	0.1771	0.0976
95% CI (Quantile)	(0.04,0.92)	(−0.07,0.57)	(−0.03,0.31)
95% CI (Fieller)	(0.01,1.01)	(−0.11,0.55)	(−0.08,0.27)
Potential surrogate: walking speed from the WIQ
Estimate	0.4808	0.3722	0.1726
SE	0.2437	0.2839	0.1618
95% CI (Quantile)	(0.13,1.04)	(0.04,1.06)	(−0.02,0.53)
95% CI (Fieller)	(0.08,1.05)	(−0.04,1.11)	(−0.06,0.52)
Potential surrogate: stair climbing from the WIQ
Estimate	0.122	0.1392	0.0695
SE	0.2099	0.1581	0.0913
95% CI (Quantile)	(−0.34,0.47)	(−0.08,0.52)	(−0.02,0.3)
95% CI (Fieller)	(−0.43,0.46)	(−0.17,0.51)	(−0.11,0.28)
All three potential surrogates
Estimate	0.4835	0.3801	0.1771
SE	0.3236	0.3095	0.1785
95% CI (Quantile)	(−0.21,1.06)	(0.04,1.12)	(−0.02,0.61)
95% CI (Fieller)	(−0.28,1.04)	(−0.06,1.07)	(−0.09,0.57)

7. Discussion

We have proposed a nonparametric procedure to estimate the proportion of treatment effect explained by a single potential surrogate marker and have extended this procedure to a setting with multiple surrogate markers. Specifically, our procedure uses kernel smoothing to estimate the conditional mean of the primary outcome given the surrogate marker under treatment and applies this estimate to the control group to obtain an estimate of the residual treatment effect. In the multiple marker setting, we use a working model to obtain a single summary measure and again use kernel smoothing in an effort to obtain an estimate that is more robust to misspecification of the working model. We have compared our proposed approach to available model-based approaches which require specification of models describing the relationship between the surrogate and the primary outcome and demonstrated through simulations that the proposed procedure outperforms the model-based procedures when the specified models do not hold. In addition, we have proposed a variance estimation procedure based on perturbation-resampling and showed that the resulting variance estimates are close to empirical estimates.

As discussed in Molenberghs et al. [11], focusing on the proportion of treatment effect explained by a surrogate as the quantity of interest to capture surrogacy has some limitations. While our definition of the quantity in (4) improves upon the more common definition using coefficients from a linear regression model because it does not rely on correct model specification, the quantity will still tend to be unstable when the treatment effect is close to zero and confidence intervals for the quantity will tend to be very wide unless one has a very large sample size or the treatment effect is large, as we observed in our illustration with PAD participants. Therefore, use of the proportion of treatment effect explained quantity would not be advisable in a study where the treatment effect is small. While we have shown that our proposed estimators can greatly reduce bias by being robust to model misspecification, further research to develop more efficient estimators would be useful.

We note that causal interpretations of the estimated quantity should be approached with care since certain untestable assumptions are required to hold in order for this quantity to truly capture the proportion of treatment effect explained by the surrogate. However, independent of those assumptions, the defined surrogacy quantity could be used to quantify the ability of the surrogate marker to replace the primary outcome in estimating the treatment effect. Specifically, one could measure the treatment effect based on S only using

$$\tilde{\Delta }=\int \mu \left(s\right)d{F}_T\left(s\right)-\int \mu \left(s\right)d{F}_C\left(s\right)$$

where μ(s) is a prediction of Y based on S = s and F_T (·) is the marginal cumulative distribution function of S^(T), the surrogate marker under treatment. If μ(s) = E(Y^(T)|S^(T) = s), then the difference that would be observed when $\tilde{\Delta }$ is used as a measure of the treatment effect instead of Δ is exactly (3) i.e. Δ_S. This equivalence does not rely on the assumptions in (2) and suggests that the defined surrogacy quantity has an appropriate interpretation in terms of the bias in replacing the treatment effect on the primary outcome with the expected treatment effect given the surrogate information.

A limitation of our proposed approach is the requirement that the supports of S^(T) and S^(C) are equivalent. However, in most practical applications where one aims to estimate the proportion of treatment effect explained, it is more likely that there will be substantial regions of overlap between S^(T) and S^(C) and less likely that these two regions of support will be substantially separated. In addition, due to the use of kernel smoothing, care should be taken in using the proposed estimator in small sample size settings; as shown in Section 5, performance may be adequate with sample sizes of 200 in each group but in certain settings, performance can be undesirable.

Finally, we have assumed that both the surrogate marker and primary outcome are fully observed and that the surrogate marker (at least one in the multiple surrogate case) is continuous. In simple settings, where the surrogate marker is discrete, alternatives that do not involve nonparametric smoothing can be used. When the primary outcome of interest is a time-to-event outcome such as time to death or time to diabetes diagnosis, the methods proposed here would require further (nontrivial) extension to handle missing surrogate marker measurements and censoring of the primary outcome itself. Further research in these areas is warranted.

An R package implementing the methods described here, called Rsurrogate, is available on CRAN.