Integrating Information from Existing Risk Prediction Models with No Model Details

Abstract

Consider the setting where (i) individual-level data are collected to build a regression model for the association between an event of interest and certain covariates, and (ii) some risk calculators predicting the risk of the event using less detailed covariates are available, possibly as algorithmic black boxes with little information available about how they were built. We propose a general empirical-likelihood-based framework to integrate the rich auxiliary information contained in the calculators into fitting the regression model, to make the estimation of regression parameters more efficient. Two methods are developed, one using working models to extract the calculator information and one making a direct use of calculator predictions without working models. Theoretical and numerical investigations show that the calculator information can substantially reduce the variance of regression parameter estimation. As an application, we study the dependence of the risk of high grade prostate cancer on both conventional risk factors and newly identified molecular biomarkers by integrating information from the Prostate Biopsy Collaborative Group (PBCG) risk calculator, which was built based on conventional risk factors alone.

1. INTRODUCTION

In the era of data science, it is common that data are collected from several sources and all provide useful information for answering the same scientific question. An analysis based on a single data source may yield biases in estimation or results that are not accurate enough. Integrating data from multiple sources becomes essential in order to pull together different pieces of information to draw more accurate conclusions and to make more insightful decisions. A common issue for data integration is that different sources usually provide information in different forms: some studies release the actual collected individual-level data whereas others only release aggregate data after the analysis. Different methods are needed to integrate different forms of data and to conduct inference.

This paper considers the setting where a current study collects individual-level data and builds a regression model to study the association between the risk of experiencing an event of interest and certain covariates, and where some risk prediction models for the same event using less detailed covariates have been built by previous studies and are accessible as risk calculators with little model detail released. These risk calculators contain rich information about the association of interest and thus it is highly desirable to integrate such information into fitting the regression model in order to improve the estimation efficiency and better understand how the risk is affected by different covariates, especially when the sample size of the current-study data is not large.

We consider the situation where some of the covariates are conventional risk factors known to be associated with the event and/or are typically adjusted for, and others are new potential factors whose association with the event has not been well studied. An example is our data application in Section 4 that studies the risk for high grade prostate cancer. Most existing studies on prostate cancer do not consider risk factors that are related to the molecular mechanisms of prostate cancer progression. However, it has been shown that prostate cancer antigen 3 (PCA 3) and TMPRSS2:ERG gene fusions are two biomarkers that have better specificity than prostate-specific antigen (PSA), a well-known conventional predictor, for early detection of prostate cancer (Tomlins et al., 2016). Therefore, in our data application we will build an expanded regression model by including both biomarkers in addition to some conventional risk factors, such as age, race, PSA, digital rectal examination (DRE) findings, prior biopsy results, and family history. The setting we consider has a current study that collects individual-level data on both types of covariates. On the other hand, for the same event of interest, some existing studies have developed risk calculators that output the predicted risk of the event based on certain conventional factors. Many such risk calculators are accessible online. For instance, for prostate cancer, some widely used online risk calculators include the Prostate Cancer Prevention Trial (PCPT) risk calculator (Version 1, Thompson et al., 2006; Version 2, Ankerst et al., 2014) and the more recent Prostate Biopsy Collaborative Group (PBCG) risk calculator (Ankerst et al., 2018). Such risk calculators contain useful auxiliary information about the association of interest, but they are sometimes available as a black box with little detail released about the actual models used to build them, especially when they are built based on machine learning techniques (e.g. Mocellin et al., 2009).

The setting we consider is substantially different from the ones in the existing literature which assume that external study model details are available. Such details typically include the specification of the model, the estimated parameter values, and sometimes the corresponding standard errors (e.g. Imbens & Lancaster, 1994; Qin, 2000; Chatterjee et al., 2016; Cheng et al., 2018; Cheng et al., 2019; Han & Lawless, 2019; Huang and Qin, 2020; Sheng et al., 2021). It is also different from the settings where the population values of certain quantities are available and can be used as calibration factors to increase the efficiency of estimation (e.g. Deville & Sarndal, 1992; Chen & Qin, 1993; Chaudhuri et al., 2008; Lumley et al., 2011; Chen & Kim, 2014; Qin et al., 2015; Huang et al., 2016). The lack of such concrete information in our setting presents a unique challenge. Without assuming any detailed knowledge about the risk calculator models, we will develop methods to extract the auxiliary information contained in the risk calculators and integrate it into fitting the regression model of interest. Integrating such auxiliary information can considerably improve the estimation efficiency. Our setting is similar to that in Gu et al. (2019), who considered only one risk calculator and proposed a synthetic data method based on multiple imputation, fundamentally different from our development.

Our development is based on the empirical likelihood (EL) method (Owen, 1988, 2001; Qin & Lawless, 1994). The EL method has been widely adopted to integrate auxiliary information to improve the estimation efficiency, especially in survey sampling (e.g. Chen & Qin, 1993; Chen et al., 2002; Chaudhuri et al., 2008; Chen & Kim, 2014) and medicine and public health related research (e.g. Qin et al., 2015; Huang et al., 2016; Qin, 2017; Cheng et al., 2018; Han & Lawless, 2019). The constrained maximum likelihood estimation proposed in Chatterjee et al. (2016) has a strong connection to the EL method (Han & Lawless 2016). Compared to existing alternatives, such as those based on the generalized method of moments (e.g. Imbens & Lancaster, 1994), the generalized regression (e.g. Chen & Chen, 2000), weight calibration (e.g. Lumley et al., 2011), and others (e.g. Boonstra et al., 2013; Grill et al., 2015; Estes et al., 2018), the EL-type methods provide a likelihood-based framework for estimation and inference. Details about many superior properties of the EL method can be found in Owen (2001) and Qin (2017). Our development in this paper also extends the EL method to new settings of data integration and expands its applicability and effectiveness.

The rest of the paper is organized as follows. Section 2 contains our proposed methods, one based on postulating working models and one without. Section 3 gives simulation studies and Section 4 provides an application to prostate cancer research. Some discussions are given in Section 5. The Appendix includes some technical details for the results established in Section 2.

2. THE PROPOSED METHODS

2.1. Notation and Setup

The main interest is to study the association between the risk of experiencing an event and certain covariates, including both the conventional and the newly discovered risk factors. Let Y denote the binary outcome indicating if a subject experiences the event (Y = 1 if yes and Y = 0 if no), X the vector of conventional risk factors, and Z the vector of newly discovered risk factors. A regression model P(Y = 1 | X, Z; β) is specified for the risk P(Y = 1 | X, Z), where β is the vector of regression parameters and has the true value β₀ such that P(Y = 1 | X, Z; β₀) = P(Y = 1 | X, Z). A widely used regression model is the logistic regression

$$P(Y=1\mid \mathit{X},\mathit{Z};\mathit{\beta })=\exp \left({\beta }_c+{\mathit{\beta }}_{\mathit{X}}^{\text{T}}\mathit{X}+{\mathit{\beta }}_{\mathit{Z}}^{\text{T}}\mathit{Z}\right)/\left\{1+\exp \left({\beta }_c+{\mathit{\beta }}_{\mathit{X}}^{\text{T}}\mathit{X}+{\mathit{\beta }}_{\mathit{Z}}^{\text{T}}\mathit{Z}\right)\right\}.$$

Let (Y_i, X_i, Z_i), i = 1, . . . , n, denote the individual-level data collected based on a random sample of size n. Then the maximum likelihood estimator (MLE) for β₀ is

$${\widehat{\mathit{\beta }}}_{mle}\equiv \arg \underset{\mathit{\beta }}{\max }\prod _{i=1}^{n}f\left(Y_i\mid {\mathit{X}}_i,{\mathit{Z}}_i;\mathit{\beta }\right),$$ (1)

where f(Y | X, Z; β) = P(Y = 1 | X, Z; β)^{I(Y = 1)}P(Y = 0 | X, Z; β)^{I(Y = 0)} and I(·) is the indicator function. Write s(β) ≡ s(Y, X, Z; β) = ∂ log f(Y | X, Z; β)/∂β and S(β) = E{s(β)s(β)^T}. From the theory of MLE (e.g. Lehmann & Casella, 2003), as n → ∞, we know that ${\widehat{\mathit{\beta }}}_{mle}\stackrel{p}{\to }{\mathit{\beta }}_0$, $\sqrt{n}\left({\widehat{\mathit{\beta }}}_{mle}-{\mathit{\beta }}_0\right)\stackrel{d}{\to }N\left(0,S^{-1}\right)$ and ${\widehat{\mathit{\beta }}}_{mle}$ has the optimal efficiency when no auxiliary information about β₀ is available, where S ≡ S(β₀).

Previous studies on the same event of interest have usually produced rich auxiliary information about the association, and such information can be used to improve the estimation efficiency over the MLE ${\widehat{\mathit{\beta }}}_{mle}$. In this paper we consider the case where the results from previous studies are available through risk calculators. Suppose that there are J risk calculators for calculating the risk of experiencing the event. It is common that some risk factors collected in a current study were not used or available in previous studies, and, for those that were indeed used, the current study may take a finer measurement (e.g. a continuous measurement instead of a categorical one). To take these possibilities into consideration, let Z be the risk factors that are available only in the current study. For the jth calculator, j = 1, . . . , J, let X_(j) be a possibly coarsened version of X. For example, the variables in X_(j) may be a subset and/or a categorized version of those in X. In other words, X_(j) is a many-to-one function of X: the value of X_(j) can be completely determined by X but not the reverse. The jth calculator uses X_(j) as the predictors to calculate the risk of experiencing the event, P(Y = 1 | X_(j)), and let ${\widehat{p}}_{(j)}$ denote the corresponding predicted value for P(Y = 1 | X_(j)). Thus, based on the collected individual-level data, the jth calculator produces data $\left({\mathit{X}}_{(j)i},{\widehat{p}}_{(j)i}\right)$, i = 1, . . . , n. To keep the generality of our development, we do not assume any other knowledge about these risk calculators.

2.2. A Method Based on Working Models

To extract information from the data $\left({\mathit{X}}_{(j)i},{\widehat{p}}_{(j)i}\right)$ produced by the jth calculator, we postulate a working model p_(j)(X_(j); θ_(j)) ≡ P(Y = 1 | X_(j); θ_(j)) for P(Y = 1 | X_(j)) with parameter vector θ_(j). Such a working model represents our knowledge/belief about how the association between Y and X_(j) should be modeled. This working model may be different from the actual unknown model used to build the calculator and it may be incorrectly specified for P(Y = 1 | X_(j)), but this does not prevent the working model from providing a reasonable approximation to P(Y = 1 | X_(j)) and extracting the calculator information.

A challenge when fitting the working model p_(j)(X_(j); θ_(j)) based on the data $\left({\mathit{X}}_{(j)i},{\widehat{p}}_{(j)i}\right)$ is that the ${\widehat{p}}_{(j)}$ is a fixed function of X_(j) determined by the jth calculator with no randomness. If the output from the jth calculator were a random binary value ${\widehat{Y}}_{(j)}~\mathrm{Bernoulli}\left({\widehat{p}}_{(j)}\right)$, we could fit the working model p_(j)(X_(j); θ_(j)) by maximizing the likelihood

$$\prod _{i=1}^n{p}_{(j)}{\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)}^{{\widehat{Y}}_{(j)i}}{\left\{1-p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)\right\}}^{1-{\widehat{Y}}_{(j)i}},$$

which would lead to estimating θ_(j) by solving the corresponding score equation

$$\sum _{i=1}^n\frac{{\widehat{Y}}_{(j)i}-p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)}{p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)\left\{1-p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)\right\}}\frac{\partial p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)}{\partial {\mathit{\theta }}_{(j)}}=0.$$

With the fixed value ${\widehat{p}}_{(j)}$, we replace the ${\widehat{Y}}_{(j)}$ in the above equation by ${\widehat{p}}_{(j)}$ and estimate θ_(j) by ${\widehat{\mathit{\theta }}}_{(j)}$ that solves

$$\sum _{i=1}^n\frac{{\widehat{p}}_{(j)i}-p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)}{p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)\left\{1-p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)\right\}}\frac{\partial p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)}{\partial {\mathit{\theta }}_{(j)}}=0.$$ (2)

Therefore, the information contained in the jth calculator is summarized by the working model p_(j)(X_(j); θ_(j)) and the parameter estimate ${\widehat{\mathit{\theta }}}_{(j)}$ that satisfies (2).

Another justification for fitting the model p_(j)(X_(j); θ_(j)) by solving (2) is to consider the weighted least squares discrepancy

$$\sum _{i=1}^n{\left(\frac{{\widehat{p}}_{(j)i}-p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)}{\sqrt{{\widehat{p}}_{(j)i}\left(1-{\widehat{p}}_{(j)i}\right)}}\right)}^2$$

between p_(j)(X_(j); θ_(j)) and ${\widehat{p}}_{(j)}$, where ${\widehat{p}}_{(j)}\left(1-{\widehat{p}}_{(j)}\right)$ is an estimate of the variance of the binary outcome Y at X_(j). Minimizing this discrepancy amounts to solving

$$\sum _{i=1}^n\frac{{\widehat{p}}_{(j)i}-p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)}{{\widehat{p}}_{(j)i}\left(1-{\widehat{p}}_{(j)i}\right)}\frac{\partial p_{(j)}\left({\mathit{X}}_{(j)i};{\mathit{\theta }}_{(j)}\right)}{\partial {\mathit{\theta }}_{(j)}}=0.$$

Equation (2) is the same as this equation with ${\widehat{p}}_{(j)}\left(1-{\widehat{p}}_{(j)}\right)$ replaced by p_(j)(X_(j); θ_(j)){1 − p_(j)(X_(j); θ_(j))}, which is an estimate of the variance of Y based on the working model. Thus, ${\widehat{\mathit{\theta }}}_{(j)}$ actually makes p_(j)(X_(j); θ_(j)), the prediction of P(Y = 1 | X_(j)) based on the working model, as close as possible to ${\widehat{p}}_{(j)}$, the prediction based on the calculator.

To transform the summarized information into a form that can be integrated into the estimation of β₀, consider for a moment the hypothetical scenario where the calculators output the exact true risk of experiencing the event; i.e. ${\widehat{p}}_{(j)}\equiv P\left(Y=1\mid {\mathit{X}}_{(j)}\right)$. Then, because ${\widehat{\mathit{\theta }}}_{(j)}$ solves (2), we have $E_{(Y,\mathit{X},\mathit{Z})}\left\{h_{(j)}\left(Y,{\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}^{*}\right)\right\}=0$, where ${\widehat{\mathit{\theta }}}_{(j)}\stackrel{p}{\to }{\mathit{\theta }}_{(j)}^{*}$ as n → ∞ and

$${\mathit{h}}_{(j)}\left(Y,{\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)=\frac{Y-p_{(j)}\left({\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)}{p_{(j)}\left({\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)\left\{1-p_{(j)}\left({\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)\right\}}\frac{\partial p_{(j)}\left({\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)}{\partial {\mathit{\theta }}_{(j)}}.$$

Here and after, we occasionally use a subscript to explicitly indicate under which distribution is the expectation taken; e.g., E_(Y,X,Z)(·) is taken under the joint distribution of (Y, X, Z). Write

$${\mathit{u}}_{(j)}\left(\mathit{X},\mathit{Z};\mathit{\beta },{\mathit{\theta }}_{(j)}\right)=\frac{P(Y=1\mid \mathit{X},\mathit{Z};\mathit{\beta })-p_{(j)}\left({\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)}{p_{(j)}\left({\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)\left\{1-p_{(j)}\left({\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)\right\}}\frac{\partial p_{(j)}\left({\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}\right)}{\partial {\mathit{\theta }}_{(j)}},$$

we then have

$$\begin{array}{l}{E}_{(Y,\mathit{X},\mathit{Z})}\left\{h_{(j)}\left(Y,{\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}^{*}\right)\right\}=E_{(\mathit{X},\mathit{Z})}\left[E_{(Y\mid \mathit{X},\mathit{Z})}\left\{h_{(j)}\left(Y,{\mathit{X}}_{(j)};{\mathit{\theta }}_{(j)}^{*}\right)\mid \mathit{X},\mathit{Z}\right\}\right]\\ =E_{(\mathit{X},\mathit{Z})}\left\{{\mathit{u}}_{(j)}\left(\mathit{X},\mathit{Z};{\mathit{\beta }}_0,{\mathit{\theta }}_{(j)}^{*}\right)\right\},\end{array}$$

and thus

$$E_{(\mathit{X},\mathit{Z})}\left\{{\mathit{u}}_{(j)}\left(\mathit{X},\mathit{Z};{\mathit{\beta }}_0,{\mathit{\theta }}_{(j)}^{*}\right)\right\}=0.$$ (3)

Therefore, through fitting the working model p_(j)(X_(j); θ_(j)), (3) summarizes the information regarding β₀ contained in the jth calculator as a moment constraint under the marginal distribution of (X, Z).

To take the moment constraint (3) into account when estimating β₀, we consider a discrete distribution q_i = dF(X_i, Z_i) on the data points (X_i, Z_i), i = 1, . . . , n, and propose to estimate β₀ by ${\widehat{\mathit{\beta }}}_{el1}$ defined through

$$\begin{array}{l}\underset{\mathit{\beta },q_1,\dots ,q_n}{\max }\prod _{i=1}^n\left\{f\left(Y_i\mid {\mathit{X}}_i,{\mathit{Z}}_i;\mathit{\beta }\right)q_i\right\}\text{subject}\text{to}\\ q_i>0,\sum _{i=1}^n{q}_i=1,\sum _{i=1}^n{q}_i{\mathit{u}}_{(j)}\left({\mathit{X}}_i,{\mathit{Z}}_i;\mathit{\beta },{\widehat{\mathit{\theta }}}_{(j)}\right)=0(j=1,\dots ,J).\end{array}$$ (4)

Compared to (1), the maximization in (4) is over the joint distribution of (Y, X, Z), where the conditional distribution of Y | (X, Z) is parametrically modeled and the marginal distribution of (X, Z) is nonparametrically modeled subject to certain constraints that are a data version of (3). Therefore, ${\widehat{\mathit{\beta }}}_{el1}$ integrates the auxiliary information about β₀ and thus should have higher efficiency compared to the MLE ${\widehat{\mathit{\beta }}}_{mle}$. The maximization (4) is similar to that in Qin (2000).

With J calculators, write

$$\mathit{\theta }=\left(\begin{array}{c}{\mathit{\theta }}_{(1)}\\ {\mathit{\theta }}_{(2)}\\ ⋮\\ {\mathit{\theta }}_{(J)}\end{array}\right)\text{and}\mathit{u}(\mathit{\beta },\mathit{\theta })\equiv \mathit{u}(\mathit{X},\mathit{Z};\mathit{\beta },\mathit{\theta })=\left(\begin{array}{c}{\mathit{u}}_{(1)}\left(\mathit{X},\mathit{Z};\mathit{\beta },{\mathit{\theta }}_{(1)}\right)\\ {\mathit{u}}_{(2)}\left(\mathit{X},\mathit{Z};\mathit{\beta },{\mathit{\theta }}_{(2)}\right)\\ ⋮\\ {\mathit{u}}_{(J)}\left(\mathit{X},\mathit{Z};\mathit{\beta },{\mathit{\theta }}_{(J)}\right)\end{array}\right).$$

In the Appendix we show that ${\widehat{\mathit{\beta }}}_{el1}$ is the component of (${\widehat{\mathit{\beta }}}_{el1}$, $\widehat{\mathit{\rho }}$) that satisfies

$$\sum _{i=1}^n{s}_i\left({\widehat{\mathit{\beta }}}_{el1}\right)+\sum _{i=1}^n\frac{\partial u_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)/\partial {\mathit{\beta }}^{\text{T}}}{1-{\widehat{\mathit{\rho }}}^{\text{T}}{u}_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)}\widehat{\mathit{\rho }}=0,$$ (5)

$$\sum _{i=1}^n\frac{u_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)}{1-{\widehat{\mathit{\rho }}}^{\text{T}}{u}_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)}=0,$$ (6)

Equivalently, ${\widehat{\mathit{\beta }}}_{el1}$ is also the component of (${\widehat{\mathit{\beta }}}_{el1}$, $\widehat{\mathit{\rho }}$, $\widehat{\mathit{\theta }}$) that satisfies (5), (6) and (2) simultaneously.

When studying the asymptotic properties of ${\widehat{\mathit{\beta }}}_{el1}$, a complication is that, for each risk calculator, the output risk prediction is based on a fixed function of the input covariates and does not accommodate the uncertainty associated with the model fitting when building the risk calculators. If the fixed function of X_(j) used by the jth calculator is not the same as P(Y = 1 | X_(j)), ${\widehat{\mathit{\beta }}}_{el1}$ will not be theoretically consistent for β₀, although the bias may be small if the output ${\widehat{p}}_{(j)}$ is close to P(Y = 1 | X_(j)). This is intuitive because the calculator information needs to compatible with that of the current study to yield improvements after data integration. Incompatible information will bias the estimation. Therefore, for the purpose of establishing asymptotic properties of ${\widehat{\mathit{\beta }}}_{el1}$,e assume that

$${\widehat{p}}_{(j)}=P\left(Y=1\mid {\mathit{X}}_{(j)}\right),j=1,\dots ,J.$$ (7)

We then have the following theorem on ${\widehat{\mathit{\beta }}}_{el1}$.

Theorem 1

Under Regularity Conditions 1 specified in the Appendix, assuming (7) and as n → ∞, we have (i) ${\widehat{\mathit{\beta }}}_{el1}\stackrel{p}{\to }{\mathit{\beta }}_0$, and (ii) $\sqrt{n}\left({\widehat{\mathit{\beta }}}_{el1}-{\mathit{\beta }}_0\right)$ has an asymptotic normal distribution with mean 0 and variance

$${\left\{S+{\mathit{G}}_1^T{\mathbf{\Omega }}^{-1}{\mathit{G}}_1-{\mathit{G}}_1^T\left({\mathbf{\Omega }}^{-1}{U}_1-\mathit{I}\right){\left({\mathit{G}}_1{S}^{-1}{\mathit{G}}_1^T+U_1{\mathbf{\Omega }}^{-1}{U}_1\right)}^{-1}\left(U_1{\mathbf{\Omega }}^{-1}-\mathit{I}\right){\mathit{G}}_1\right\}}^{-1},$$ (8)

where G₁ ≡ E{∂u(β₀, θ^∗)/∂β}, $\mathbf{\Omega }\equiv E{\left[\mathit{u}\left({\mathit{\beta }}_0,\mathit{\theta }\ast \right)-\tilde{E}\left\{\mathit{u}\left({\mathit{\beta }}_0,\mathit{\theta }\ast \right)\right\}\right]}^{\otimes 2}$ with

$$\tilde{E}\{\mathit{u}(\mathit{\beta },\mathit{\theta })\}\equiv \left(\begin{array}{c}E\{{\mathit{u}}_{(1)}\left(\mathit{X},\mathit{Z};\mathit{\beta },{\mathit{\theta }}_{(1)}\right)\mid {\mathit{X}}_{(1)}\}\\ ⋮\\ E\left\{{\mathit{u}}_{(J)}\left(\mathit{X},\mathit{Z};\mathit{\beta },{\mathit{\theta }}_{(J)}\right)\mid {\mathit{X}}_{(J)}\right\}\end{array}\right)$$

and EA^⊗2 = E(AA^T) for any matrix A, U₁ ≡ E{u(β₀, θ^∗)u(β₀, θ^∗)^T}, I is the identity matrix, and $\mathit{\theta }*\equiv \left({\mathit{\theta }}_{(1)}^{*},\dots ,{\mathit{\theta }}_{(J)}^{*}\right)$.

The proof of Theorem 1 is given in the Appendix. Due to the complexity of (8), there is no general clear comparison to S⁻¹, the asymptotic variance of the MLE ${\widehat{\mathit{\beta }}}_{mle}$. But an efficiency improvement over ${\widehat{\mathit{\beta }}}_{mle}$ can be anticipated when the calculators are not poorly built and when the working models are reasonably postulated. Indeed, in such settings comprehensive simulation studies have shown that ${\widehat{\mathit{\beta }}}_{el1}$ provides a substantial efficiency gain over ${\widehat{\mathit{\beta }}}_{mle}$. In the next subsection we will propose an alternative estimator that is guaranteed to improve over ${\widehat{\mathit{\beta }}}_{mle}$.

2.3. A Method Without Working Models

The working model approach summarizes the auxiliary information contained in the calculators by fitting a working model to the data $\left({\mathit{X}}_{(j)i},{\widehat{p}}_{(j)i}\right)$. As seen from (8), the estimation of θ_(j) introduced by the working models has a very complex effect on the asymptotic variance of ${\widehat{\mathit{\beta }}}_{el1}$. In this section we consider an alternative method without working models, the efficiency of which will not be compromised by the estimation of any nuisance parameters.

For any arbitrary vector function d_(j)(X_(j)) of X_(j), assuming all relevant moments exist, we have

$$E_{(\mathit{X},\mathit{Z})}\left[d_{(j)}\left({\mathit{X}}_{(j)}\right)\left\{P(Y=1\mid \mathit{X},\mathit{Z})-P\left(Y=1\mid {\mathit{X}}_{(j)}\right)\right\}\right]=0.$$ (9)

Since P(Y = 1 | X, Z) = P(Y = 1 | X, Z; β₀) and the jth calculator outputs ${\widehat{p}}_{(j)}$ as a prediction for P(Y = 1 | X_(j)), (9) provides an alternative to (3) to summarize the auxiliary information contained in the jth calculator, again in the form of a moment equality under the distribution of (X, Z). To integrate this summary information into the estimation of β₀, let q_i = dF(X_i, Z_i) and define another estimator ${\widehat{\mathit{\beta }}}_{el2}$ through

$$\begin{array}{l}\underset{\mathit{\beta },q_1,\dots ,q_n}{\max }\prod _{i=1}^n\left\{f\left(Y_i\mid {\mathit{X}}_i,{\mathit{Z}}_i;\mathit{\beta }\right)q_i\right\}\text{subject}\text{to}{q}_i>0,\sum _{i=1}^n{q}_i=1,\\ \sum _{i=1}^n{q}_i{\mathit{d}}_{(j)}\left({\mathit{X}}_{(j)i}\right)\left\{P\left(Y_i=1\mid {\mathit{X}}_i,{\mathit{Z}}_i;\mathit{\beta }\right)-{\widehat{p}}_{(j)i}\right\}=0(j=1,\dots ,J).\end{array}$$ (10)

This estimator of β₀ directly uses the outputs from the calculators instead of postulating working models. For each calculator, a vector function d_(j)(X_(j)) needs to be chosen. For a chosen d_(j)(X_(j)), the constraints in (10) are similar to those in (4) when the working model p_(j)(X_(j); θ_(j)) is the logistic regression

$$\frac{\exp \left\{{\mathit{d}}_{(j)}{\left({\mathit{X}}_{(j)}\right)}^{\text{T}}{\mathit{\theta }}_{(j)}\right\}}{1+\exp \left\{{\mathit{d}}_{(j)}{\left({\mathit{X}}_{(j)}\right)}^{\text{T}}{\mathit{\theta }}_{(j)}\right\}},$$ (11)

because in this case the last constraint in (4) becomes

$$\sum _{i=1}^n{q}_i{\mathit{d}}_{(j)}\left({\mathit{X}}_{(j)i}\right)\left\{P\left(Y_i=1\mid {\mathit{X}}_i,{\mathit{Z}}_i;\mathit{\beta }\right)-p_{(j)}\left({\mathit{X}}_{(j)i};{\widehat{\mathit{\theta }}}_{(j)}\right)\right\},$$

and thus the only difference between (4) and (10) is in which prediction of P(Y = 1 | X_(j)) is used, the $p_{(j)}\left({\mathit{X}}_{(j)};{\widehat{\mathit{\theta }}}_{(j)}\right)$ from the working model or the ${\widehat{p}}_{(j)}$ from the calculator. In this case, intuitively, because the maximization in (10) does not involve estimation of any nuisance parameters, ${\widehat{\mathit{\beta }}}_{el2}$ should have a higher efficiency compared to ${\widehat{\mathit{\beta }}}_{el1}$, as the asymptotic variance of ${\widehat{\mathit{\beta }}}_{el2}$ will not have a component coming from the estimation of θ_(j). This intuition is confirmed by our theoretical results below.

For ease of notation, write

$$\mathit{u}(\mathit{\beta })\equiv \mathit{u}(\mathit{X},\mathit{Z};\mathit{\beta })=\left(\begin{array}{c}{\mathit{d}}_{(1)}\left({\mathit{X}}_{(1)}\right)\left\{P(Y=1\mid \mathit{X},\mathit{Z};\mathit{\beta })-{\widehat{p}}_{(1)}\right\}\\ {\mathit{d}}_{(2)}\left({\mathit{X}}_{(2)}\right)\left\{P(Y=1\mid \mathit{X},\mathit{Z};\mathit{\beta })-{\widehat{p}}_{(2)}\right\}\\ ⋮\\ {\mathit{d}}_{(J)}\left({\mathit{X}}_{(J)}\right)\left\{P(Y=1\mid \mathit{X},\mathit{Z};\mathit{\beta })-{\widehat{p}}_{(J)}\right\}\end{array}\right).$$

Based on derivations similar to those leading to (5) and (6), ${\widehat{\mathit{\beta }}}_{el2}$ is the component of (${\widehat{\mathit{\beta }}}_{el2}$, $\widehat{\mathit{\rho }}$) that satisfies (5) and (6) but with u(β, θ) replaced by u(β). Following the same proof as that for Theorem 1, the properties of ${\widehat{\mathit{\beta }}}_{el2}$ are given below.

Theorem 2

Under Regularity Conditions 2 specified in the Appendix, assuming (7) and as n → ∞, we have (i) ${\widehat{\mathit{\beta }}}_{el2}\stackrel{p}{\to }{\mathit{\beta }}_0$, and (ii) $\sqrt{n}\left({\widehat{\mathit{\beta }}}_{el2}-{\mathit{\beta }}_0\right)$ has an asymptotic normal distribution with mean 0 and variance

$${\left\{\mathit{S}+{\mathit{G}}_2^{\text{T}}{\mathit{U}}_2^{-1}{\mathit{G}}_2\right\}}^{-1},$$ (12)

where G₂ ≡ E{∂u(β₀)/∂β} and U₂ ≡ E{u(β₀)u(β₀)^T}.

It is clear that (12) is smaller than S⁻¹, the asymptotic variance of the MLE ${\widehat{\mathit{\beta }}}_{mle}$. Thus, ${\widehat{\mathit{\beta }}}_{el2}$ is expected to be more efficient than ${\widehat{\mathit{\beta }}}_{mle}$ when the calculators were not poorly built. There is no general comparison between (8) and (12) because (8) depends on the working models postulated and (12) depends on the d_(j)(X_(j)) chosen. However, in the setting where all working models for ${\widehat{\mathit{\beta }}}_{el1}$ are logistic regression as in (11) and all vector functions d_(j)(X_(j)) for ${\widehat{\mathit{\beta }}}_{el2}$ are taken to be the same as those in (11), a direct comparison between (8) and (12) is possible. In this case, some calculation shows that G₁ = G₂ and Ω = U₂. Therefore, (12) is smaller than (8) because the third term that is subtracted in (8) is semi-positive definite. In this case ${\widehat{\mathit{\beta }}}_{el2}$ has a higher efficiency than ${\widehat{\mathit{\beta }}}_{el1}$. Because of the popularity of logistic regression, this efficiency comparison is widely applicable in practice. Another important observation from (12) is that, since ${\mathit{G}}_2^{\text{T}}{U}_2^{-1}{\mathit{G}}_2$ becomes larger in the positive-definite sense as the dimension of u(β) increases, (12) becomes smaller as more calculators are integrated, as long as the calculators output correct predictions and do not use exactly the same predictors.

2.4. Some Discussion and Remarks

Because of the efficiency properties of ${\widehat{\mathit{\beta }}}_{el2}$ and the direct use of the output probabilities ${\widehat{p}}_{(j)}$ from the calculators without introducing any nuisance parameters, it is more desirable to implement ${\widehat{\mathit{\beta }}}_{el2}$ in practice, which requires specifying the d_(j)(X_(j)). In theory, the asymptotic variance (12) becomes smaller as more functions are included in d_(j)(X_(j)). In practice, however, a large dimension of d_(j)(X_(j)) may jeopardize the numerical performance. The specification of the d_(j)(X_(j)) can be guided by first building working models, such as logistic regression models, and then taking the d_(j)(X_(j)) to be the corresponding regressors. The working models should be specified based both on existing scientific knowledge about the association of interest and on the calculator data $\left({\mathit{X}}_{(j)},{\widehat{p}}_{(j)}\right)$. When partial information is available about the original models used to build the calculators, such as the inclusion of certain interactions, it should be accommodated when specifying the working models. Intuitively, working models that are close to the possibly unknown true calculator models should lead to good final performance, because the true models contain all the calculator information. After the working models are specified as in (11), the corresponding d_(j)(X_(j)) can be used to implement ${\widehat{\mathit{\beta }}}_{el2}$. The resulting ${\widehat{\mathit{\beta }}}_{el2}$ is guaranteed to be more efficient than both ${\widehat{\mathit{\beta }}}_{mle}$ and the ${\widehat{\mathit{\beta }}}_{el1}$ based on the specified working models. In this way the working models are specified only as a guideline for choosing the d_(j)(X_(j)), and the efficiency of ${\widehat{\mathit{\beta }}}_{el2}$ is not affected by the estimation of any nuisance parameters.

The large sample properties of ${\widehat{\mathit{\beta }}}_{el1}$ and ${\widehat{\mathit{\beta }}}_{el2}$ are established under the assumption that ${\widehat{p}}_{(j)}=P\left(Y=1\mid {\mathit{X}}_{(j)}\right)$. In reality, due to model specification and random errors when building the calculators, ${\widehat{p}}_{(j)}$ is not the same as P(Y = 1 | X_(j)). For example, the model of interest P(Y = 1 | X, Z; β) implicitly imposes restrictions on modeling P(Y = 1 | X_(j)), which may not be met by the calculator models. However, the proposed methods should still lead to small finite-sample bias and considerable efficiency gains compared to ${\widehat{\mathit{\beta }}}_{mle}$ if ${\widehat{p}}_{(j)}$ is a good approximation to P(Y = 1 | X_(j)). This should be the case when the calculators were built based on carefully specified models and decent sample sizes so that they captured most of the association between the risk of interest and corresponding covariates. In practice, there are ways to quickly check the quality of this approximation. One way is to compare the data $\left({\mathit{X}}_{(j)i},{\widehat{p}}_{(j)i}\right)$ produced by the calculators to the data (X_(j)i, Y_i) using some simple quantities, such as the means of ${\widehat{p}}_{(j)}$ and Y within each level of X_(j). Another way is to compare the coefficients of the regression of Y_i on X_(j)i to the coefficients of the regression of ${\widehat{p}}_{(j)i}$ on X_(j)i. The comparison can be made by constructing confidence intervals for the former coefficients and check if they cover the latter coefficients. In our data application in Section 4 we carry out such a comparison.

The proposed methods are very flexible in the sense that they apply to black-box-type calculators where little information is available about how the calculators were built, especially when they were built based on machine learning techniques. Some extra care may be needed when there are multiple calculators based on similar covariates. In this case, the auxiliary information provided by these calculators may be similar because the information is about the association between the outcome of interest and the corresponding covariates. Although in theory ${\widehat{\mathit{\beta }}}_{el2}$ will keep gaining efficiency when integrating more calculators, using all these calculators simultaneously might jeopardize numerical performance because some constraints in (4) and (10) may become highly correlated. Therefore, in the presence of multiple calculators with similar inputs, we would recommend using the one(s) based on large sample sizes and based on populations similar to the current study, which may be checked by the procedures mentioned in the previous paragraph.

A simple way to implement the proposed methods might seem to be to solve the equations in (5) and (6). However, this procedure is not recommended owing to its unstable behavior: Equation (6), viewed as an equation for ρ for fixed β and θ, typically has many roots (Han & Wang, 2013). Here we need the $\widehat{\mathit{\rho }}$ such that the ${\widehat{q}}_i$ maximizing (4) or (10) are between 0 and 1. Solving (5) and (6) directly can lead to an unwanted root. Please refer to the derivation of (5) and (6) in the Appendix for an expression for ${\widehat{q}}_i$. A more reliable implementation is to follow the Newton-Raphson-type algorithm provided in Han & Lawless (2019).

3. SIMULATION STUDIES

In this section we carry out simulation studies to investigate the finite-sample performance of the proposed methods. The covariates X = (X₁, X₂) and Z are generated from a three dimensional multivariate normal distribution, where the means are all zero, the variances are all one, and the correlations are all 0.4. Given the covariates, the response Y is generated from a Bernoulli distribution with

$$P(Y=1\mid \mathit{X},Z)=\mathrm{expit}\left(0.5-0.5X_1-0.5X_2+0.5Z+0.5X_{2}Z\right),$$

where expit(x) = e^x/(1 + e^x). Therefore we have ${\mathit{\beta }}_0=\left({\beta }_c,{\beta }_{X_1},{\beta }_{X_2},{\beta }_Z,{\beta }_{X_{2}Z}\right)=(0.5,-0.5,-0.5,0.5,0.5)$. We consider Calculators 1, 2 and 3 constructed based on the generated data using covariates X₍₁₎ = X₁, $X_{(2)}=\left(X_1,{\tilde{X}}_2\right)$, and X₍₃₎ = (X₁, X₂), respectively, where ${\tilde{X}}_2=I\left(X_2>0\right)$. Calculator 1 for P(Y = 1 | X₍₁₎) is based on the semiparametric single index model in Klein & Spady (1993) implemented in R (R Core Team, 2021) with package np (Hayfield & Racine, 2008). Because there is only one covariate, this semiparametric model becomes nonparametric. Calculator 2 for P(Y = 1 | X₍₂₎) is based on the same nonparametric model but for ${\tilde{X}}_2=1$ and ${\tilde{X}}_2=0$ separately. Calculator 3 for P(Y = 1 | X₍₃₎) is based on the random forest method (Breiman, 2001) implemented in the R package randomForest (Liaw & Wiener, 2002). The nonparametric nature of these models requires a large sample size to achieve a good quality for risk prediction, and we used a sample size of 50000 to build these calculators to ensure their good quality.

First, we compare estimators ${\widehat{\mathit{\beta }}}_{el1}$ based on working models, ${\widehat{\mathit{\beta }}}_{el2}$ without using working models, and the MLE ${\widehat{\mathit{\beta }}}_{mle}$ without integrating calculator information. In this comparison we fix the working models or the d_(j)(X_(j)) for each calculator j = 1, 2, 3. Specifically, for ${\widehat{\mathit{\beta }}}_{el1}$, the postulated working models for the calculators are all logistic regression as specified in (11), where d_(j)(X_(j)) contains the intercept and corresponding main effects; i.e., d_(j)(X_(j)) = (1, X_(j)), j = 1, 2, 3. These same d_(j)(X_(j)) are used for ${\widehat{\mathit{\beta }}}_{el2}$. For both ${\widehat{\mathit{\beta }}}_{el1}$ and ${\widehat{\mathit{\beta }}}_{el2}$, we consider seven versions, ${\widehat{\mathit{\beta }}}_{el}-1$, ${\widehat{\mathit{\beta }}}_{el}-2$, ${\widehat{\mathit{\beta }}}_{el}-3$, ${\widehat{\mathit{\beta }}}_{el}-12$, ${\widehat{\mathit{\beta }}}_{el}-13$, ${\widehat{\mathit{\beta }}}_{el}-23$, and ${\widehat{\mathit{\beta }}}_{el}-123$, where the numbers indicate which calculators are incorporated into the estimation.

Tables 1 and 2 contain the simulation results based on n = 400 and n = 1000, respectively, both using 1000 replications. Compared to ${\widehat{\mathit{\beta }}}_{mle}$, the empirical standard errors of ${\widehat{\mathit{\beta }}}_{el1}$ and ${\widehat{\mathit{\beta }}}_{el2}$ for those covariates also used by the corresponding calculators are substantially smaller, confirming our theoretical conclusion of efficiency gains by integrating calculator information. Note that, for ${\widehat{\mathit{\beta }}}_{el}-23$ and ${\widehat{\mathit{\beta }}}_{el}-123$, the empirical standard errors corresponding to those covariates not used by the calculators (i.e. Z and X₂Z) become larger compared to ${\widehat{\mathit{\beta }}}_{mle}$. This is caused by the estimation of nuisance parameters in the working models. As seen from (8), the effect of estimating nuisance parameters on the efficiency of ${\widehat{\mathit{\beta }}}_{el1}$ is quite complex. Although in general we anticipate efficiency gains for ${\widehat{\mathit{\beta }}}_{el1}$, the gains are typically small for the coefficients that do not appear in the calculators and may even be negative due to estimation of nuisance parameters.

Table 1:

Simulation results for comparisons between methods with and without using working models based on n = 400 and 1000 replications.

		Method 1: with working models					Method 2: without working models
		βc	β X 1	β X 2	βZ	β X 2 Z	βc	β X 1	β X 2	βZ	β X 2 Z
MLE	bias	0.003	−0.005	−0.014	0.006	0.015	0.003	−0.005	−0.014	0.006	0.015
	emp	0.123	0.137	0.139	0.140	0.143	0.123	0.137	0.139	0.140	0.143
	est	0.120	0.135	0.139	0.139	0.141	0.120	0.135	0.139	0.139	0.141
	cov	95.0	94.8	95.1	95.6	94.8	95.0	94.8	95.1	95.6	94.8
EL-1	bias	−0.001	0.018	−0.014	0.006	0.015	−0.001	0.019	−0.014	0.006	0.015
	emp	0.052	0.071	0.139	0.140	0.143	0.052	0.071	0.139	0.140	0.143
	est	0.052	0.073	0.138	0.138	0.140	0.052	0.073	0.138	0.138	0.140
	cov	95.4	94.2	94.9	95.3	94.8	95.4	94.0	95.0	95.3	94.7
EL-2	bias	−0.000	0.020	−0.035	0.006	0.016	0.000	0.020	−0.035	0.006	0.015
	emp	0.049	0.059	0.061	0.140	0.142	0.049	0.059	0.061	0.140	0.142
	est	0.048	0.062	0.059	0.138	0.140	0.048	0.061	0.059	0.138	0.140
	cov	95.4	92.3	94.4	95.4	95.0	95.5	92.0	94.1	95.4	95.0
EL-3	bias	−0.006	−0.002	−0.017	0.006	0.015	−0.005	−0.002	−0.015	0.006	0.013
	emp	0.064	0.072	0.076	0.140	0.143	0.063	0.071	0.075	0.140	0.143
	est	0.061	0.074	0.076	0.138	0.140	0.060	0.073	0.074	0.138	0.140
	cov	95.3	95.5	95.3	95.2	94.9	95.5	94.4	94.7	95.3	94.7
EL-12	bias	0.003	0.021	−0.036	0.005	0.015	0.000	0.022	−0.036	0.005	0.015
	emp	0.050	0.060	0.061	0.140	0.142	0.049	0.059	0.061	0.140	0.142
	est	0.049	0.062	0.060	0.138	0.140	0.048	0.061	0.059	0.138	0.140
	cov	94.9	91.7	94.1	95.4	95.4	95.8	91.5	93.8	95.4	95.3
EL-13	bias	0.002	0.004	−0.011	0.005	0.009	−0.001	0.017	−0.017	0.006	0.014
	emp	0.074	0.072	0.082	0.140	0.143	0.049	0.062	0.075	0.140	0.143
	est	0.073	0.074	0.085	0.138	0.140	0.049	0.062	0.074	0.138	0.140
	cov	94.6	94.8	96.1	95.2	95.1	95.5	92.5	94.5	95.3	94.6
EL-23	bias	−0.001	−0.003	−0.023	0.024	0.026	−0.003	0.016	−0.032	0.005	0.022
	emp	0.072	0.072	0.078	0.146	0.159	0.048	0.060	0.060	0.141	0.141
	est	0.068	0.072	0.079	0.141	0.149	0.047	0.061	0.058	0.138	0.138
	cov	93.6	95.1	95.0	94.9	92.7	95.5	92.3	94.3	94.8	94.5
EL-123	bias	0.003	−0.002	−0.019	0.027	0.020	−0.003	0.018	−0.033	0.005	0.022
	emp	0.078	0.075	0.080	0.149	0.166	0.048	0.060	0.059	0.141	0.141
	est	0.077	0.073	0.080	0.142	0.154	0.047	0.060	0.058	0.138	0.138
	cov	93.6	94.6	95.2	93.9	92.8	95.9	91.7	94.2	94.9	94.6

emp: empirical standard error. est: mean of estimated standard errors over 1000 replications. cov: percentage over 1000 replications that the 95% confidence intervals constructed based on asymptotic distributions cover the true value.

Table 2:

Simulation results for comparisons between methods with and without using working models based on n = 1000 and 1000 replications.

		Method 1: with working models					Method 2: without working models
		βc	β X 1	β X 2	βZ	β X 2 Z	βc	β X 1	β X 2	βZ	β X 2 Z
MLE	bias	0.004	−0.004	−0.003	0.003	0.008	0.004	−0.004	−0.003	0.003	0.008
	emp	0.075	0.087	0.087	0.086	0.091	0.075	0.087	0.087	0.086	0.091
	est	0.076	0.085	0.087	0.087	0.088	0.076	0.085	0.087	0.087	0.088
	cov	95.0	94.0	94.6	95.0	94.5	95.0	94.0	94.6	95.0	94.5
EL-1	bias	0.005	0.008	−0.003	0.003	0.008	0.005	0.008	−0.003	0.003	0.007
	emp	0.031	0.046	0.087	0.086	0.091	0.031	0.046	0.087	0.086	0.091
	est	0.032	0.045	0.087	0.087	0.088	0.032	0.045	0.087	0.087	0.088
	cov	95.7	93.8	94.6	95.1	94.6	95.7	93.8	94.6	95.1	94.6
EL-2	bias	0.005	0.016	−0.012	0.003	0.008	0.005	0.017	−0.012	0.003	0.008
	emp	0.029	0.039	0.035	0.086	0.090	0.029	0.039	0.035	0.086	0.090
	est	0.030	0.038	0.036	0.087	0.087	0.030	0.038	0.036	0.087	0.087
	cov	95.8	89.7	95.2	94.8	94.2	95.6	89.6	95.2	94.8	94.2
EL-3	bias	−0.001	−0.002	0.007	0.003	0.007	−0.001	−0.002	0.006	0.003	0.007
	emp	0.038	0.047	0.045	0.086	0.091	0.037	0.046	0.044	0.086	0.091
	est	0.038	0.046	0.047	0.087	0.087	0.037	0.045	0.046	0.087	0.087
	cov	95.5	94.6	95.2	94.9	94.2	95.3	93.9	94.7	95.0	94.3
EL-12	bias	0.002	0.016	−0.012	0.003	0.008	0.005	0.016	−0.012	0.003	0.007
	emp	0.030	0.039	0.035	0.086	0.090	0.029	0.039	0.035	0.086	0.090
	est	0.030	0.038	0.036	0.087	0.087	0.030	0.038	0.036	0.087	0.087
	cov	95.2	90.6	95.4	94.7	94.3	95.4	89.7	95.4	94.8	94.3
EL-13	bias	0.001	0.001	0.010	0.002	0.005	0.004	0.005	0.006	0.003	0.007
	emp	0.045	0.047	0.051	0.086	0.091	0.030	0.040	0.045	0.086	0.091
	est	0.046	0.046	0.052	0.087	0.087	0.030	0.039	0.046	0.087	0.087
	cov	95.7	94.2	93.6	95.0	93.8	95.8	93.6	94.7	95.1	94.3
EL-23	bias	−0.001	−0.001	0.001	0.009	0.028	0.001	0.014	−0.010	0.001	0.016
	emp	0.043	0.046	0.050	0.087	0.096	0.028	0.039	0.034	0.086	0.089
	est	0.043	0.044	0.049	0.088	0.093	0.029	0.038	0.036	0.087	0.086
	cov	94.7	93.1	94.5	95.1	94.2	95.6	90.5	96.0	94.8	93.9
EL-123	bias	−0.003	0.000	0.004	0.009	0.027	0.002	0.014	−0.009	0.001	0.016
	emp	0.046	0.047	0.050	0.088	0.098	0.028	0.039	0.034	0.086	0.089
	est	0.049	0.044	0.049	0.088	0.096	0.029	0.038	0.036	0.087	0.086
	cov	95.0	93.1	93.6	94.7	94.9	95.2	90.4	95.8	94.9	94.0

emp: empirical standard error. est: mean of estimated standard errors over 1000 replications. cov: percentage over 1000 replications that the 95% confidence intervals constructed based on asymptotic distributions cover the true value.

The comparison of empirical standard errors between ${\widehat{\mathit{\beta }}}_{el1}$ and ${\widehat{\mathit{\beta }}}_{el2}$ confirms that the latter is more efficient, especially when more than one calculator is used. Empirical standard errors for ${\widehat{\mathit{\beta }}}_{el2}$ either decrease or stay about the same as more calculators are integrated, consistent with our theory, whereas those for ${\widehat{\mathit{\beta }}}_{el1}$ may increase considerably due to the estimation of more nuisance parameters. It is also seen that, for all considered estimators, the mean of estimated standard errors over 1000 replications is very close to the corresponding empirical standard error, especially when n = 1000, confirming the derived asymptotic variances for both ${\widehat{\mathit{\beta }}}_{el1}$ and ${\widehat{\mathit{\beta }}}_{el2}$.

Second, we vary the d_(j)(X_(j)) for each calculator to assess its effects. We also make a comparison of the proposed method to that in Gu et al. (2019). Since ${\widehat{\mathit{\beta }}}_{el2}$ is superior to ${\widehat{\mathit{\beta }}}_{el1}$ both theoretically and numerically, we now only include ${\widehat{\mathit{\beta }}}_{el2}$. We focus on the three versions ${\widehat{\mathit{\beta }}}_{el2}-1$, ${\widehat{\mathit{\beta }}}_{el2}-2$ and ${\widehat{\mathit{\beta }}}_{el2}-3$ since the method in Gu et al. (2019) deals with one calculator. The three versions of Gu et al.’s estimator are ${\widehat{\mathit{\beta }}}_{GTCM}-1$, ${\widehat{\mathit{\beta }}}_{GTCM}-2$ and ${\widehat{\mathit{\beta }}}_{GTCM}-3$. For our estimators we consider the following specifications of the d_(j)(X_(j)). For ${\widehat{\mathit{\beta }}}_{el2}-1$, take d₍₁₎(X₍₁₎) to be (1, X₍₁₎) and $\left(1,X_{(1)},X_{(1)}^2\right)$, resulting in ${\widehat{\mathit{\beta }}}_{el2}-1-1$ and ${\widehat{\mathit{\beta }}}_{el2}-1-2$, respectively. For ${\widehat{\mathit{\beta }}}_{el2}-2$, take d₍₂₎(X₍₂₎) to be $\left(1,X_1,{\tilde{X}}_2\right)$, $\left(1,X_1,{\tilde{X}}_2,X_1{\tilde{X}}_2\right)$ and $\left(1,X_1,{\tilde{X}}_2,X_1{\tilde{X}}_2,X_1^2\right)$, resulting in ${\widehat{\mathit{\beta }}}_{el2}-2-1$ and ${\widehat{\mathit{\beta }}}_{el2}-2-2$ and ${\widehat{\mathit{\beta }}}_{el2}-2-3$, respectively. For ${\widehat{\mathit{\beta }}}_{el2}-3$, take d₍₃₎(X₍₃₎) to be (1, X₁, X₂), (1, X₁, X₂, X₁X₂) and $\left(1,X_1,X_2,X_1{X}_2,X_1^2,X_2^2\right)$, resulting in ${\widehat{\mathit{\beta }}}_{el2}-3-1$ and ${\widehat{\mathit{\beta }}}_{el2}-3-2$ and ${\widehat{\mathit{\beta }}}_{el2}-3-3$, respectively.

Table 3 contains the simulation results based on n = 400 and n = 1000 using 1000 replications. The MLE ${\widehat{\mathit{\beta }}}_{mle}$ is also included as the benchmark for comparison. When varying the d_(j)(X_(j)) for the proposed estimator ${\widehat{\mathit{\beta }}}_{el2}$ based on either Calculator 2 or Calculator 3, an inclusion of the interaction between the calculator covariates (i.e. ${\mathit{X}}_1{\tilde{\mathit{X}}}_2$ for Calculator 2 or X₁X₂ for Calculator 3) in addition to the main effects leads to further efficiency gains for estimating ${\mathit{\beta }}_{{\mathit{X}}_2\mathit{Z}}$ (i.e. comparing EL2–2-2 to EL2–2-1 or comparing EL2–3-2 to EL2–3-1). A further inclusion of the quadratic effects does not lead to additional efficiency gains. These observations suggest that interactions should be included in d_(j)(X_(j)) if the current study model includes interactions that involve the calculator covariates. We leave a detailed theoretical and numerical study of this as a future research topic. Overall, the estimators GTCM-1, GTCM-2 and GTCM-3 based on Gu et al. (2019) sometimes have small biases, and the latter two have roughly the same or slightly larger empirical standard errors compared to our estimators EL2–2-2 and EL2–3-2 that include interactions in d_(j)(X_(j)). For all the estimators under comparison, the efficiency for estimating β_Z does not change much from the MLE. This observation makes sense because the calculators provide little auxiliary information about the regression coefficients for the covariates not used by the calculators. This observation is also in full agreement with findings in the existing literature (e.g. Chatterjee et al., 2016; Cheng et al., 2018; Han & Lawless, 2019).

Table 3:

Simulation results for different choices of d_(j)(X_(j)) and comparisons to Gu et al. (2019) based on 1000 replications. The proposed method here is Method 2 without using working models.

		n = 400					n = 1000
		βc	β X 1	β X 2	βZ	β X 2 Z	βc	β X 1	β X 2	βZ	β X 2 Z
MLE	bias	0.010	−0.005	−0.013	0.006	0.016	0.006	−0.006	−0.003	0.006	0.002
	emp	0.122	0.136	0.144	0.140	0.139	0.076	0.084	0.087	0.083	0.085
GTCM-1	bias	0.036	0.015	0.008	0.044	−0.087	0.040	0.009	0.013	0.047	−0.093
	emp	0.059	0.081	0.138	0.134	0.096	0.037	0.049	0.084	0.082	0.060
EL2-1-1	bias	−0.000	0.018	−0.013	0.006	0.016	0.007	0.009	−0.003	0.006	0.002
	emp	0.052	0.076	0.144	0.140	0.139	0.033	0.045	0.087	0.083	0.085
EL2-1-2	bias	−0.000	0.018	−0.013	0.006	0.016	0.007	0.009	−0.002	0.006	0.003
	emp	0.050	0.075	0.144	0.141	0.130	0.031	0.044	0.087	0.084	0.078
GTCM-2	bias	0.028	−0.017	0.087	0.044	−0.089	0.032	−0.016	0.098	0.048	−0.093
	emp	0.054	0.071	0.076	0.133	0.095	0.035	0.042	0.047	0.081	0.060
EL2-2-1	bias	0.000	0.019	−0.033	0.006	0.015	0.006	0.018	−0.012	0.006	0.002
	emp	0.048	0.065	0.062	0.141	0.138	0.030	0.037	0.035	0.083	0.085
EL2-2-2	bias	−0.009	0.011	−0.034	0.008	0.049	−0.002	0.011	−0.013	0.008	0.028
	emp	0.042	0.060	0.062	0.141	0.104	0.026	0.034	0.036	0.084	0.061
EL2-2-3	bias	−0.010	0.011	−0.034	0.008	0.051	−0.002	0.011	−0.013	0.009	0.029
	emp	0.042	0.060	0.062	0.142	0.104	0.026	0.034	0.036	0.084	0.061
GTCM-3	bias	0.031	0.001	0.001	0.044	−0.087	0.034	−0.001	0.018	0.047	−0.093
	emp	0.064	0.081	0.081	0.134	0.095	0.041	0.049	0.049	0.082	0.060
EL2-3-1	bias	−0.005	0.000	−0.012	0.006	0.014	0.000	−0.003	0.007	0.006	0.001
	emp	0.059	0.076	0.075	0.140	0.139	0.039	0.044	0.045	0.083	0.085
EL2-3-2	bias	−0.008	−0.001	−0.013	0.006	0.024	−0.002	−0.006	0.005	0.007	0.013
	emp	0.057	0.073	0.073	0.140	0.114	0.036	0.043	0.044	0.083	0.069
EL2-3-3	bias	−0.010	−0.001	−0.017	0.008	0.032	0.000	−0.008	0.012	0.005	0.005
	emp	0.057	0.073	0.070	0.140	0.111	0.036	0.043	0.040	0.084	0.068

emp: empirical standard error. GTCM-1,2 or 3: the method in Gu et al. (2019) using Calculator 1, 2, or 3, respectively. EL2-j-1,2 or 3: the proposed Method 2 without working models using Calculator j and different specifications of d_(j)(X_(j)).

4. DATA APPLICATION

As an application, we fit an expanded regression model for high grade prostate cancer by including two biomarkers, prostate cancer antigen 3 (PCA 3) and TMPRSS2:ERG gene fusions, measured from urine, in addition to the conventional risk factors: prostate-specific antigen (PSA), age, race, digital rectal examination (DRE) findings, prior biopsy results, and family history. To improve efficiency, we will integrate information from an online accessible risk calculator, the Prostate Biopsy Collaborative Group (PBCG) risk calculator (Ankerst et al., 2018).

The PBCG risk calculator was built as a state-of-the-art risk prediction tool, an alternative to the widely used Prostate Cancer Prevention Trial (PCPT) risk calculator (Version 1, Thompson et al., 2006; Version 2, Ankerst et al., 2014). The PCPT calculator was the first online prostate cancer risk assessment tool and was built based on data collected in the 1990s from 5519 men, mostly white, in the placebo group of the PCPT. In contrast, the PBCG risk calculator was built on data from 15611 men undergoing prostate biopsies during 2006–2017 at eight North American institutions and three European institutions participating in the PBCG. The heterogeneity in the study cohorts in PBCG endows the PBCG risk calculator with a much wider applicability compared to the PCPT calculator.

The data set we use is the validation cohort from Tomlins et al. (2016), who investigated whether including two additional biomarkers, PCA 3 and TMPRSS2:ERG, could give a more accurate risk prediction compared to the PCPT calculator. This cohort consists of 1244 men presenting for diagnostic biopsy at seven community clinics throughout the United States. Since the PBCG risk calculator requires an input PSA level between 2 and 50 ng/ml and an input age between 40 and 90 years old, in our analysis we remove subjects with PSA and age outside those range. The final analysis is based on 1014 men.

The logistic regression model of interest is

$$\log \frac{\pi }{1-\pi }={\beta }_c+{\beta }_1{X}_1+{\beta }_2{X}_2+{\beta }_3{X}_3+{\beta }_4{X}_4+{\beta }_5{X}_5+{\beta }_6{X}_6+{\beta }_7{Z}_1+{\beta }_8{Z}_2,$$

where π is the probability of observing high grade prostate cancer, X₁ = log₂(PSA) (log transformation of the PSA level with base 2), X₂ = age, X₃ = DRE (a binary indicator of an abnormal digital rectal exam), X₄ = biopsy (a binary indicator of prior negative biopsy), X₅ = race (a binary indicator of African Ancestry) and X₆ = family history (a binary indicator of first-degree family history of prostate cancer) are the conventional risk predictors, and Z₁ = log₂(PCA3 + 1) and Z₂ = T2:ERG are the two biomarkers. Here, following Cheng et al. (2019), we take the log₂ transformation of PCA3 and dichotomize TMPRSS2:ERG by splitting at the median. The PBCG risk calculator uses X₁ through X₆ as the input and the exact formula for predicting the risk of high grade prostate cancer is given in the supplementary material of Ankerst et al. (2018). In the same supplementary Ankerst et al. (2018) also gave formulas for risk prediction when some or all of X₃, X₄ and X₆ are not available. Therefore, as an illustration, in our application we also consider predictions based only on X₁, X₂ and X₅ as input, and refer to this prediction as calculator PBCG-2. Note that, since these two calculators were built using the same data and the input of PBCG-2 is a subset of PBCG, we do not anticipate an efficiency improvement by integrating PBCG-2 in addition to PBCG.

As a checking of the assumption that ${\widehat{p}}_{(j)}$ should be close to P(Y = 1 | X_(j)), we construct 95% confidence intervals for the regression coefficients of X_(j) based on a logistic regression of Y on X_(j), and then check if these intervals cover the corresponding coefficients based on a logistic model for ${\widehat{p}}_{(j)}$ conditional on X_(j), where the latter coefficients are computed by solving equation (2). We found coverage for all coefficients for both calculators. When looking at the 68% confidence intervals instead, we found coverage for all coefficients but not that of X₁ for both calculators. These findings show that it is reasonable to assume ${\widehat{p}}_{(j)}$ is close to P(Y = 1 | X_(j)).

Table 4 contains the analysis results based on both ${\widehat{\mathit{\beta }}}_{mle}$ and ${\widehat{\mathit{\beta }}}_{el2}$. For ${\widehat{\mathit{\beta }}}_{el2}$, the d_(j)(X_(j)) is taken to be (1, X₁, X₂, X₃, X₄, X₅, X₆) and (1, X₁, X₂, X₅) for the PBCG and PBCG-2 calculators, respectively. Results based on ${\widehat{\mathit{\beta }}}_{el1}$ are very similar to those based on ${\widehat{\mathit{\beta }}}_{el2}$ and are thus omitted. A major observation is that, after integrating information from the PBCG calculator, the standard errors corresponding to X₁ through X₆, the covariates also used by the PBCG calculator, reduce to a third or a quarter of those for the MLE. Standard errors corresponding to Z₁ and Z₂ also become smaller. Similar observations can be made when integrating the PBCG-2 calculator. These are in full agreement with our theoretical results. Integrating both calculators simultaneously produces more or less the same results as integrating the PBCG calculator alone, which is what we anticipated. Another major observation is that integrating information from the PBCG calculator reveals a marginal significance of African Ancestry (p-value= 0.062) and a high significance of family history (p-value< 0.001) for their association with high grade prostate cancer. This significance is not detected without integrating the PBCG information.

Table 4:

Prostate cancer study analysis results based on logistic regression (n = 1014).

	Without calculators			With PBCG calculator			With PBCG-2 calculator			With both calculators
	est	s.e.	p	est	s.e.	p	est	s.e.	p	est	s.e.	p
Intercept	−7.421	0.769	< 0.001	−6.802	0.227	< 0.001	−7.276	0.315	< 0.001	−6.826	0.229	< 0.001
log2(PSA)	0.595	0.119	< 0.001	0.756	0.033	< 0.001	0.774	0.051	< 0.001	0.751	0.033	< 0.001
age	0.037	0.011	0.001	0.027	0.004	< 0.001	0.036	0.005	< 0.001	0.028	0.004	< 0.001
DRE	0.635	0.210	0.003	0.769	0.057	< 0.001	0.631	0.199	0.001	0.762	0.057	< 0.001
biopsy	−0.895	0.245	< 0.001	−0.938	0.058	< 0.001	−0.901	0.216	< 0.001	−0.943	0.058	< 0.001
race	0.090	0.341	0.793	0.192	0.103	0.062	−0.050	0.120	0.675	0.181	0.103	0.079
history	0.248	0.209	0.235	0.413	0.049	< 0.001	0.249	0.191	0.193	0.414	0.049	< 0.001
PCA3	0.347	0.062	< 0.001	0.347	0.057	< 0.001	0.347	0.056	< 0.001	0.345	0.057	< 0.001
T2ERG	0.565	0.183	0.002	0.567	0.166	0.001	0.568	0.165	0.001	0.582	0.166	< 0.001

est: estimated value. s.e.: standard error. p: p-value. DRE: a binary indicator of an abnormal digital rectal exam. biopsy: a binary indicator of prior negative biopsy. race: a binary indicator of African Ancestry. history: a binary indicator of first-degree family history of prostate cancer. PCA3: log₂(PCA3 + 1). T2ERG: dichotomized TMPRSS2:ERG split at the median

5. DISCUSSION

We have proposed two methods for integrating information contained in existing risk calculators into estimation of regression parameters. The first method relies on working models to extract information contained in the calculators and the second method directly uses the risk predictions without working models. The second method is recommended in practice because its efficiency gain is not compromised by the estimation of any nuisance parameters introduced by working models. Given that many risk calculators have been developed for various diseases and many of them are of black-box-type, our proposed methods have a broad range of applications.

A potential issue when integrating information from multiple calculators is that these calculators may target different populations, either because of an original design or because of the sample based on which they were built. When the population from which the current study sample is taken is different from the populations the existing calculators can be applied to, ${\widehat{p}}_{(j)}$ may no longer be a good approximation to P(Y = 1 | X_(j)), and in this case integrating information from such calculators will introduce bias. Thus, it is crucial to use calculators whose target population is the same as the current one. This indeed was the consideration in our data application when choosing the PBCG calculator over the PCPT calculator, as the former was built based on multiple heterogeneous cohorts and thus has a wider applicability. Recently Estes et al. (2018) and Sheng et al. (2021) proposed methods to address the population heterogeneity problem, under the setting where external models and parameter estimates are directly available. These methods can be combined with the techniques developed in this paper to deal with black-box calculators in the presence of population heterogeneity. New methods dealing with population heterogeneity could also be developed for data integration purposes. For example, as pointed out by one referee, the instance weighting method for domain adaption in the natural language processing literature may be adopted (e.g. Jiang & Zhai, 2007).

In this paper we considered binary outcomes that indicate whether experiencing an event or not. The proposed methods can be directly applied to continuous outcomes under linear regression models with calculators predicting the outcome values. A desirable extension is to the setting of survival outcomes where calculators are available that give the, say, 5-year survival probabilities based on a set of risk factors. Because of the importance of survival outcomes in many areas, including medicine and public health, such an extension is highly desirable and will be investigated in our future research.

Appendix

Derivation of (5) and (6)

Using the Lagrange multiplier method, the Lagrangian corresponding to the constrained optimization problem (4) is

$$\mathcal{L}=\sum _{i=1}^n\log f_i(\mathit{\beta })+\sum _{i=1}^n\log q_i+n{\mathit{\rho }}^{\text{T}}\sum _{i=1}^n{q}_i{\mathit{u}}_i(\mathit{\beta },\widehat{\mathit{\theta }})-\mu (\sum _{i=1}^n{q}_i-1),$$

where ρ and μ are the Lagrange multipliers. At the solution ${\widehat{\mathit{\beta }}}_{el1}$ and ${\widehat{q}}_i$ we must have $\partial \mathcal{L}/\partial q_i=0$ and $\partial \mathcal{L}/\partial \mathit{\beta }=0$ for some $\widehat{\mathit{\rho }}$ and $\widehat{\mu }$. Multiplying both sides of $\partial \mathcal{L}/\partial q_i=1/{\widehat{q}}_i+n{\widehat{\mathit{\rho }}}^{\text{T}}{\mathit{u}}_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)-\widehat{\mu }=0$ by ${\widehat{q}}_i$ and summing over i, the constraints in (4) lead to $\widehat{\mu }=n$, which, combined with $\partial \mathcal{L}/\partial q_i=0$ yields ${\widehat{q}}_i=1/\left[n\left\{1-{\widehat{\mathit{\rho }}}^{\text{T}}{\mathit{u}}_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)\right\}\right]$. Then $\partial \mathcal{L}/\partial \mathit{\beta }=0$ gives (5) and the constraint ${\sum }_{i=1}^n{q}_i{\mathit{u}}_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)=0$ gives (6).

Regularity conditions

Regularity Conditions 1.

(1) The parameter spaces $\mathcal{B}$ for β and Θ for θ are compact. β₀ is an interior point of $\mathcal{B}$ and θ^∗ is an interior point of Θ. (2) E[log f(Y|X, Z; β)] is uniquely maximized at β₀ and E[u(X, Z; β₀, θ)] = 0 has a unique solution θ^∗. (3) $E\left[{\sup }_{(\mathit{\beta },\mathit{\theta })\in \mathcal{B}\times \Theta }\Vert \mathit{u}(\mathit{X},\mathit{Z};\mathit{\beta },\mathit{\theta }){\Vert }^{\alpha }\right]<\infty$ for some α > 2. (4) E[∂² log f(Y|X, Z; β₀)/∂β∂β^T] and E{u(X, Z; β₀, θ^∗)u(X, Z; β₀, θ^∗)^T} are non-singular. (5) ${\sup }_{(\mathit{\beta },\mathit{\theta })\in \mathcal{B}\times \Theta }{n}^{-1/2}{\sum }_{i=1}^n\left\{l_i(\mathit{\beta },\mathit{\theta })-E[l(\mathit{\beta },\mathit{\theta })]\right\}=O_p(1)$ for l(β, θ) = log f(Y|X, Z; β) and u(X, Z; β, θ). (6) u(X, Z; β, θ) is continuously differentiable, $E\left[{\sup }_{(\mathit{\beta },\mathit{\theta })\in \mathcal{B}\times \Theta }\Vert \partial \mathit{u}(\mathit{\beta },\mathit{\theta })/\partial \mathit{\beta }\Vert \right]<\infty$, and $E\left[{\sup }_{(\mathit{\beta },\mathit{\theta })\in \mathcal{B}\times \Theta }\Vert \partial \mathit{u}(\mathit{\beta },\mathit{\theta })/\partial \mathit{\theta }\Vert \right]<\infty$. (7) log f(Y|X, Z; β) is twice continuously differentiable and $E\left[{\sup }_{\mathit{\beta }\in \mathcal{B}}\Vert \partial s(\mathit{\beta })/\partial \mathit{\beta }\Vert \right]<\infty$.

Regularity Conditions 2.

(1) The parameter space $\mathcal{B}$ for β is compact. β₀ is an interior point of $\mathcal{B}$. (2) E[log f(Y|X, Z; β)] is uniquely maximized at β₀. (3) $E\left[{\sup }_{\mathit{\beta }\in \mathcal{B}}\Vert \mathit{u}(\mathit{X},\mathit{Z};\mathit{\beta }){\Vert }^{\alpha }\right]<\infty$ for some α > 2. (4) E[∂² log f(Y|X, Z; β₀)/∂β∂β^T] and E{u(X, Z; β₀)u(X, Z; β₀)^T} are non-singular. (5) ${\sup }_{\mathit{\beta }\in \mathcal{B}}{n}^{-1/2}{\sum }_{i=1}^n\left\{l_i(\mathit{\beta })-E[l(\mathit{\beta })]\right\}=O_p(1)$ E[l(β)]} = O_p(1) for l(β) = log f(Y|X, Z; β) and u(X, Z; β). (6) u(X, Z; β) is continuously differentiable and $E\left[{\sup }_{\mathit{\beta }\in \mathcal{B}}\Vert \partial \mathit{u}(\mathit{\beta })/\partial \mathit{\beta }\Vert \right]<\infty$. (7) log f(Y |X, Z; β) is twice continuously differentiable and $E\left[{\sup }_{\mathit{\beta }\in \mathcal{B}}\Vert \partial s(\mathit{\beta })/\partial \mathit{\beta }\Vert \right]<\infty$.

Proof of Theorem 1

For (i), because E{u(β₀, θ^∗)} = 0, an application of the M-estimator theory (e.g., van der Vaart 1998) to (5) and (6) leads to $\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\rho }}\right)\stackrel{p}{\to }\left({\mathit{\beta }}_0,0\right)$ as n → ∞. Thus ${\widehat{\mathit{\beta }}}_{el1}\stackrel{p}{\to }{\mathit{\beta }}_0$.

For (ii), applying the mean-value theorem to (5) and (6) around (β₀, ρ = 0) leads to

$$0=\frac{1}{n}\sum _{i=1}^n\left(\begin{array}{c}{\mathit{s}}_i\left({\mathit{\beta }}_0\right)\\ {\mathit{u}}_i\left({\mathit{\beta }}_0,\widehat{\mathit{\theta }}\right)\end{array}\right)+\frac{1}{n}\sum _{i=1}^n\left(\begin{array}{cc}\frac{\partial {\mathit{s}}_i(\overline{\mathit{\beta }})}{\partial \mathit{\beta }},& \frac{\partial {\mathit{u}}_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)/\partial {\mathit{\beta }}^{\text{T}}}{1-{\overline{\mathit{\rho }}}^{\text{T}}{\mathit{u}}_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)}\\ \frac{\partial {\mathit{u}}_i(\overline{\mathit{\beta }},\widehat{\mathit{\theta }})/\partial \mathit{\beta }}{1-{\overline{\mathit{\rho }}}^{\text{T}}{\mathit{u}}_i\left({\widehat{\mathit{\beta }}}_{el1},\widehat{\mathit{\theta }}\right)},& \frac{{\mathit{u}}_i(\overline{\mathit{\beta }},\widehat{\mathit{\theta }}){\mathit{u}}_i{\left({\mathit{\beta }}_{el1},\widehat{\mathit{\theta }}\right)}^{\text{T}}}{{\left\{1-{\overline{\mathit{\rho }}}^{\text{T}}{\mathit{u}}_i\left({\mathit{\beta }}_{el1},\widehat{\mathit{\theta }}\right)\right\}}^2}\end{array}\right)\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{el1}-{\mathit{\beta }}_0\\ \widehat{\mathit{\rho }}\end{array}\right),$$

where $\overline{\mathit{\beta }}$ is some value between ${\widehat{\mathit{\beta }}}_{el1}$ and β₀ and $\overline{\mathit{\rho }}$ is some value between $\widehat{\mathit{\rho }}$ and 0. Then we have

$$\begin{array}{l}\sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{el1}-{\mathit{\beta }}_0\\ \widehat{\mathit{\rho }}\end{array}\right)\\ =-{\left(\begin{array}{c}-\mathit{S},{\mathit{G}}_1^{\text{T}}\\ {\mathit{G}}_1,{\mathit{U}}_1\end{array}\right)}^{-1}\frac{1}{\sqrt{n}}\sum _{i=1}^n\left(\begin{array}{c}{\mathit{s}}_i\left({\mathit{\beta }}_0\right)\\ {\mathit{u}}_i\left({\mathit{\beta }}_0,\widehat{\mathit{\theta }}\right)\end{array}\right)+o_p(1)\\ -{\left(\begin{array}{c}-\mathit{S},{\mathit{G}}_1^{\text{T}}\\ {\mathit{G}}_1,{\mathit{U}}_1\end{array}\right)}^{-1}\left\{\frac{1}{\sqrt{n}}\sum _{i=1}^n\left(\begin{array}{c}{\mathit{s}}_i\left({\mathit{\beta }}_0\right)\\ {\mathit{u}}_i\left({\mathit{\beta }}_0,{\mathit{\theta }}^{*}\right)\end{array}\right)+\left(\begin{array}{c}0\\ \left\{\frac{1}{n}{\sum }_{i=1}^n\frac{\partial {\mathit{u}}_i\left({\mathit{\beta }}_0,\overline{\mathit{\theta }}\right)}{\partial \mathit{\theta }}\right\}\sqrt{n}\left(\widehat{\mathit{\theta }}-{\mathit{\theta }}^{*}\right)\end{array}\right)\right\}+o_p(1)\\ =-{\left(\begin{array}{c}-\mathit{S},{\mathit{G}}_1^{\text{T}}\\ {\mathit{G}}_1,{\mathit{U}}_1\end{array}\right)}^{-1}\left\{\frac{1}{\sqrt{n}}\sum _{i=1}^n\left(\begin{array}{c}{\mathit{s}}_i\left({\mathit{\beta }}_0\right)\\ {\mathit{u}}_i\left({\mathit{\beta }}_0,{\mathit{\theta }}^{*}\right)\end{array}\right)+\left(\begin{array}{c}0\\ Q\sqrt{n}\left(\widehat{\mathit{\theta }}-{\mathit{\theta }}^{*}\right)\end{array}\right)\right\}+o_p(1)\end{array}$$

where Q ≡ E{∂u(β₀, θ^∗)/∂θ} and $\overline{\mathit{\theta }}$ is some value between $\widehat{\mathit{\theta }}$ and θ^∗. On the other hand, since ${\widehat{\mathit{\theta }}}_{(j)}$ solves (2) and we assume ${\widehat{p}}_{(j)}=P\left(Y=1\mid {\mathit{X}}_{(j)}\right)$, it is seen that $\widehat{\mathit{\theta }}$ actually solves ${\sum }_{i=1}^n{\tilde{E}}_i\left\{\mathit{u}\left({\mathit{\beta }}_0,\mathit{\theta }\right)\right\}=0$. Then some algebra shows that

$$\sqrt{n}\left(\widehat{\mathit{\theta }}-\mathit{\theta }*\right)=-Q^{-1}\frac{1}{\sqrt{n}}\sum _{i=1}^n{\tilde{E}}_i\left\{\mathit{u}\left({\mathit{\beta }}_0,\mathit{\theta }*\right)\right\}+o_p(1).$$

Therefore we have

$$\sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{el1}-{\mathit{\beta }}_0\\ \widehat{\mathit{\rho }}\end{array}\right)=-{\left(\begin{array}{c}-\mathit{S},{\mathit{G}}_1^{\text{T}}\\ {\mathit{G}}_1,{\mathit{U}}_1\end{array}\right)}^{-1}\frac{1}{\sqrt{n}}\sum _{i=1}^n\left(\begin{array}{c}{\mathit{s}}_i\left({\mathit{\beta }}_0\right)\\ {\mathit{u}}_i\left({\mathit{\beta }}_0,\mathit{\theta }*\right)-{\tilde{E}}_i\left\{\mathit{u}\left({\mathit{\beta }}_0,\mathit{\theta }*\right)\right\}\end{array}\right)+o_p(1).$$

Some calculation leads to

$$\mathrm{Var}\left(\begin{array}{c}\mathit{s}\left({\mathit{\beta }}_0\right)\\ \mathit{u}\left({\mathit{\beta }}_0,\mathit{\theta }*\right)-\tilde{E}\left\{\mathit{u}\left({\mathit{\beta }}_0,\mathit{\theta }*\right)\right\}\end{array}\right)=\left(\begin{array}{cc}\mathit{S}& 0\\ 0& \mathbf{\Omega }\end{array}\right),$$

and thus from the central limit theorem we have

$$\mathrm{asy}\mathrm{Var}\left\{\sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{el1}-{\mathit{\beta }}_0\\ \widehat{\mathit{\rho }}\end{array}\right)\right\}={\left(\begin{array}{c}-\mathit{S},{\mathit{G}}_1^{\text{T}}\\ {\mathit{G}}_1,{\mathit{U}}_1\end{array}\right)}^{-1}\left(\begin{array}{cc}\mathit{S}& 0\\ 0& \mathbf{\Omega }\end{array}\right){\left(\begin{array}{c}-\mathit{S},{\mathit{G}}_1^{\text{T}}\\ {\mathit{G}}_1,{\mathit{U}}_1\end{array}\right)}^{-1}.$$

Some further calculation then yields (8).