Efficient Evaluation of Prediction Rules in Semi-Supervised Settings under Stratified Sampling

Abstract

In many contemporary applications, large amounts of unlabeled data are readily available while labeled examples are limited. There has been substantial interest in semi-supervised learning (SSL) which aims to leverage unlabeled data to improve estimation or prediction. However, current SSL literature focuses primarily on settings where labeled data is selected uniformly at random from the population of interest. Stratified sampling, while posing additional analytical challenges, is highly applicable to many real world problems. Moreover, no SSL methods currently exist for estimating the prediction performance of a fitted model when the labeled data is not selected uniformly at random. In this paper, we propose a two-step SSL procedure for evaluating a prediction rule derived from a working binary regression model based on the Brier score and overall misclassification rate under stratified sampling. In step I, we impute the missing labels via weighted regression with nonlinear basis functions to account for stratified sampling and to improve efficiency. In step II, we augment the initial imputations to ensure the consistency of the resulting estimators regardless of the specification of the prediction model or the imputation model. The final estimator is then obtained with the augmented imputations. We provide asymptotic theory and numerical studies illustrating that our proposals outperform their supervised counterparts in terms of efficiency gain. Our methods are motivated by electronic health record (EHR) research and validated with a real data analysis of an EHR-based study of diabetic neuropathy.

1. Introduction

Semi-supervised learning (SSL) has emerged as a powerful learning paradigm to address big data problems where the outcome is cumbersome to obtain and the predictors are readily available [Chapelle et al., 2009]. Formally, the SSL problem is characterized by two sources of data: (i) a relatively small sized labeled dataset $\mathcal{L}$ with n observations on the outcome y and the predictors x and (ii) a much larger unlabeled dataset $\mathcal{U}$ with $N\gg n$ observations on only x. A promising application of SSL and the motivation for this work is in electronic health record (EHR) research. EHRs have immense potential to serve as a major data source for biomedical research as they have generated extensive information repositories on representative patient populations [Murphy et al., 2009, Kohane, 2011, Wilke et al., 2011]. Nonetheless, a primary bottleneck in recycling EHR data for secondary use is to accurately and efficiently extract patient level disease phenotype information [Sinnott et al., 2014, Liao et al., 2015]. Frequently, true phenotype status is not well characterized by disease-specific billing codes. For example, at Partner’s Healthcare, only 56% of patients with at least 3 International Classification of Diseases, Ninth Revision (ICD9) codes for rheumatoid arthritis (RA) have confirmed RA after manual chart review [Liao et al., 2010]. More accurate EHR phenotyping has been achieved by training a prediction model based on a number of features including billing codes, lab results and mentions of clinical terms in narrative notes extracted via natural language processing (NLP) [Liao et al., 2013, Xia et al., 2013, Ananthakrishnan et al., 2013, e.g.]. The model is traditionally trained and evaluated using a small amount of labeled data obtained from manual medical chart review by domain experts. SSL methods are particularly attractive for developing such models as they leverage unlabeled data to achieve higher estimation efficiency than their supervised counterparts. In practice, this increase in efficiency can be directly translated into requiring fewer chart reviews without a loss in estimation precision.

In the EHR phenotyping setting, it is often infeasible to select the labeled examples uniformly at random, either due to practical constraints or due to the nature of the application. For example, it may be necessary to oversample individuals for labeling with a particular billing code or diagnostic procedure for a rare disease to ensure that an adequate number of cases are available for model estimation. In some settings, training data may consist of a subset of individuals selected uniformly at random together with a set of registry patients whose phenotype status is confirmed through routine collection. Stratified sampling is also an effective strategy when interest lies in simultaneously characterizing multiple phenotypes. One may oversample patients with at least one billing code for several rare phenotypes and then perform chart review on the selected patients for all phenotypes of interest. Though it is critical to account for the aforementioned sampling mechanisms to make valid statistical inference, it is non-trivial in the context of SSL since the fraction of subjects being sampled for labeling is near zero. The challenge is further amplified when the fitted prediction models are potentially misspecified.

Existing SSL literature primarily concerns the setting in which $\mathcal{L}$ is a uniform random sample from the underlying pool of data and thus the missing labels in $\mathcal{U}$ are missing completely at random (MCAR) [Wasserman and Lafferty, 2008]. In this setting, a variety of methods for classification have been proposed including generative modeling [Castelli and Cover, 1996, Jaakkola et al., 1999], manifold regularization [Belkin et al., 2006, Niyogi, 2013] and graph-based regularization [Belkin and Niyogi, 2004]. While making use of both $\mathcal{U}$ and $\mathcal{L}$ can improve estimation, in many cases SSL is outperformed by supervised learning (SL) using only $\mathcal{U}$ when the assumed models are incorrectly specified [Castelli and Cover, 1996, Jaakkola et al., 1999, Corduneanu, 2002, Cozman et al., 2002, 2003]. As model misspecification is nearly inevitable in practice, recent work has called for ‘safe’ SSL methods that are always at least as efficient as the SL counterparts. For example, several authors have considered safe SSL methods for discriminative models based on density ratio weighted maximum likelihood [Sokolovska et al., 2008, Kawakita and Kanamori, 2013, Kawakita and Takeuchi, 2014]. Though the true density ratio is 1 in the MCAR setting, the efficiency gain is achieved through estimation of density ratio weight, a statistical paradox previously observed in the missing data literature [Robins et al., 1992, 1994]. More recently, Krijthe and Loog [2016] introduced a SSL method for least squares classification that is guaranteed to outperform SL. Chakrabortty and Cai [2018] proposed an adaptive imputation-based SSL approach for linear regression that also outperforms supervised least squares estimation. It is unclear, however, whether these methods can be extended to accommodate additional loss functions. Moreover, none of the aforementioned methods are applicable to settings where the labeled data is not a uniform random sample from the underlying data such as the stratified sampling design. Additionally, the focus of existing work has been on the estimation of prediction models, rather than the estimation of model performance metrics. Gronsbell and Cai [2018] recently proposed a semi-supervised procedure for estimating the receiver operating characteristic parameters, but this method is similarly limited to the standard MCAR setting.

This paper addresses these limitations through the development of an efficient SS estimation method for model performance metrics that is robust to model misspecification in the presence of stratified sampling. Specifically, we develop an imputation based procedure to evaluate the prediction performance of a potentially misspecified binary regression model. To the best of our knowledge, the proposed method is the first SSL procedure that provides efficient and robust estimation of prediction performance measures under stratified sampling. We focus on two commonly used error measurements, the overall misclassification rate (OMR) and the Brier score. The proposed method involves two steps of estimation. In step I, the missing labels are imputed with a weighted regression with nonlinear basis functions to account for stratified sampling and to improve efficiency. In step II, the initial imputations are augmented to ensure the consistency of the resulting estimators regardless of the specification of the prediction model or the imputation model. Through theoretical results and numerical studies, we demonstrate that the SS estimators of prediction performance are (i) robust to the misspecification of the prediction or imputation model and (ii) substantially more efficient than their SL counterparts. We also develop an ensemble cross-validation (CV) procedure to adjust for overfitting and a perturbation resampling procedure for variance estimation.

The remainder of this paper is organized as follows. In Section 2, we specify the data structure and problem set-up. We then develop the estimation and bias correction procedure for the accuracy measures in Sections 3 and 4. Section 5 outlines the asymptotic properties of the estimators and section 6 introduces the perturbation resampling procedure for making inference. Our proposals are then validated through a simulation study in Section 7 and a real data analysis of an EHR-based study of diabetic neuropathy is presented in Section 8. We conclude with additional discussions in Section 9.

2. Preliminaries

2.1. Data Structure

Our interest lies in evaluating a prediction model for a binary phenotype y based on a predictor vector $\mathbf{x}={\left(1,x_1,\dots ,x_p\right)}^{\top }$ for some fixed p. The underlying full data consists of $N={\sum }_{s=1}^S{N}_s$ independent and identically distributed random vectors

$$\mathcal{F}={\left\{{\mathbf{F}}_i={\left(y_i,{\mathbf{u}}_i^{\top }\right)}^{\top }\right\}}_{i=1}^N$$

where ${\mathbf{u}}_i={\left({\mathbf{x}}_i^{\top },{\mathcal{S}}_i\right)}^{\top }$, ${\mathcal{S}}_i\in \{1,2,\dots ,S\}$ is a discrete stratification variable that defines a fixed number of strata $S$ for sampling, and $N_s={\sum }_{i=1}^{N}I\left({\mathcal{S}}_i=s\right)$ is the sample size of stratum s. Throughout, we let ${\mathbf{F}}_0={\left(y_0,{\mathbf{x}}_0^{\top },{\mathcal{S}}_0\right)}^{\top }$ be a future realization of F.

Due to the difficulty in ascertaining y, a small uniform random sample is obtained from each stratum and labeled with outcome information. The observable data therefore consists of

$$\mathcal{D}={\left\{{\mathbf{D}}_i={\left(y_i{V}_i,{\mathbf{u}}_i^{\top },V_i\right)}^{\top }\right\}}_{i=1}^N$$

where $V_i\in \{0,1\}$ indicates whether y_i is ascertained. We let

$$P\left(V_i=1\mid \mathcal{F}\right)={\widehat{\pi }}_{{\mathcal{S}}_i},{\widehat{\pi }}_s=n_s/N_s\text{and}{n}_s=\sum _{j=1}^{N}I\left({\mathcal{S}}_j=s\right)V_j.$$

Without loss of generality, we suppose that the first $n={\sum }_{s=1}^S{n}_s$ subjects are labeled and $\left\{n_s,s=1,\dots ,S\right\}$ are specified by design. We assume that

$${\widehat{\rho }}_{1s}=n_s/n\stackrel{p}{\to }{\rho }_{1s}\in (0,1)\text{and}{\widehat{\rho }}_s=N_s/N\stackrel{p}{\to }{\rho }_s\in (0,1)$$

as n and $N\to \infty$ respectively. This ensures that

$${\widehat{\pi }}_s/{\widehat{\pi }}_t\stackrel{p}{\to }\left({\rho }_{1s}{\rho }_t\right)/\left({\rho }_s{\rho }_{1t}\right)\in (0,1)$$

as $n\to \infty$ for any pair of s and t [Mirakhmedov et al., 2014]. As in the standard SS setting, we further assume max_s ${\widehat{\pi }}_s\stackrel{p}{\to }0$ as $n\to \infty$ [Chakrabortty and Cai, 2018, Zhang et al., 2019]. This assumption distinguishes the current setting from (i) the familiar missing data setting where ${\widehat{\pi }}_s$ is bounded above 0 and (ii) standard SSL under uniform random sampling as y is MCAR conditional on $\mathcal{S}$ (i.e. $V\perp (y,\mathbf{x})\mid \mathcal{S}$) under stratified sampling.

2.2. Problem Set-Up

To predict y₀ based on x₀, we fit a working regression model

$$\text{P}(y=1\mid \mathbf{x})=g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)$$ (1)

where $\mathit{\theta }={\left({\theta }_0,{\theta }_1,\dots ,{\theta }_p\right)}^{\top }$ is an unknown vector of regression parameters and $g(\cdot ):(-\infty ,\infty )\to (0,1)$ is a specified, smooth monotone function such as the expit function. The target model parameter, $\overline{\mathit{\theta }}$, is the solution to the estimating equation

$$\mathbf{U}(\mathit{\theta })=\text{E}\left[\mathbf{x}\left\{y-g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)\right\}\right]=\mathbf{0}.$$

We let the predicted value for y₀ be $\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_0\right)$ for some function $\mathcal{Y}$. In this paper, we aim to obtain SS estimators of the prediction performance of $\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{\text{x}}}_0\right)$ quantified by the Brier score

$${\overline{D}}_1=\text{E}\left[{\left\{y_0-{\mathcal{Y}}_1\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_0\right)\right\}}^2\right]\text{with}{\mathcal{Y}}_1(x)=g(x),$$

and the overall misclassification rate (OMR) ${\overline{D}}_2=\text{E}\left[{\left\{y_0-{\mathcal{Y}}_2\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_0\right)\right\}}^2\right]$ with ${\mathcal{Y}}_2(x)=I\{g(x)>c\}$ for some specified constant c.

We focus on these two metrics as the convey distinct information about the performance of the prediction model. The OMR summarizes the overall discrimination capacity of the model while the Brier score summarizes the calibration of the model. More complete discussions regarding assessment of model performance can be found in Hand [1997, 2001], Gneiting and Raftery [2007] and Gerds et al. [2008].

To simplify presentation, we generically write

$$\overline{D}=D(\overline{\mathit{\theta }})\text{with}D(\mathit{\theta })=\text{E}\left[d\left\{y_0,\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_0\right)\right\}\right],$$

where $d(y,z)={(y-z)}^2,\mathcal{Y}(\cdot )={\mathcal{Y}}_1(\cdot )$ for Brier score, and $\mathcal{Y}(\cdot )={\mathcal{Y}}_2(\cdot )$ for the OMR. We will construct a SS estimator of $D(\overline{\mathit{\theta }})$ to improve the statistical efficiency of its SL counterpart, ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)$, where

$${\widehat{\mathit{\theta }}}_{\text{SL}}{\text{solves U}}_n(\mathit{\theta })=\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{x}}_i\left\{y_i-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}=\mathbf{0},{\widehat{D}}_{\text{SL}}(\mathit{\theta })=\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_{i}d\left\{y_i,\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\},$$

and the weights ${\widehat{w}}_i=V_i/{\widehat{\pi }}_{{\mathcal{S}}_i}$ account for the stratified sampling with ${\sum }_{i=1}^N{\widehat{w}}_i=N$. Since ${\widehat{w}}_i\stackrel{p}{\to }\infty$ for those with $V_i=1$, standard M-estimation theory cannot be directly applied to establish the asymptotic behavior of the SL estimators. We show in Appendix C that ${\widehat{\mathit{\theta }}}_{\text{SL}}$ is a root-n consistent estimator for $\overline{\mathit{\theta }}$ and derive the asymptotic properties of ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)$ in Appendix D. We also note that throughout the article we use the subscripts n or N to index estimating equations to clarify if they are computed with the labeled or full data, respectively.

3. Estimation Procedure

Our approach to obtaining a SS estimator of $D(\overline{\mathit{\theta }})$ proceeds in two steps. First, the missing outcomes are imputed with a flexible model to improve statistical efficiency. Next, the imputations are augmented so that the resulting estimators are consistent for $D(\overline{\mathit{\theta }})$ regardless of the specification of the prediction model or the imputation model. The final estimator of the accuracy measure is then estimated using the full data and the augmented imputations. This estimation procedure is detailed in the subsequent sections.

We comment here that the initial imputation step allows for construction of a simple and efficient SS estimator of $\overline{\mathit{\theta }}$. As efficient estimation of $\overline{\mathit{\theta }}$ may not be of practical utility in the prediction setting, we keep our focus on accuracy parameter estimation and defer some of the technical details of model parameter estimation to the Appendix. However, we do note that the estimator of $D(\overline{\mathit{\theta }})$ inherently has two sources of estimation variability. The dominating source of variation is from estimating the accuracy measure itself while the second source is from the estimation of the regression parameter. Therefore, by leveraging a SS estimator of $\overline{\mathit{\theta }}$ in estimating $D(\overline{\mathit{\theta }})$, we may further improve the efficiency of our SS estimator of the accuracy measure. These statements are elucidated by the influence function expansions of the SS and SL estimators of $D(\overline{\mathit{\theta }})$ presented in Section 5.

3.1. Step 1: Flexible imputation

We propose to impute the missing y with an estimate of $m(\mathbf{u})=P(y=1\mid \mathbf{u})$. The purpose of the imputation step is to make use of $\mathcal{U}$ as it essentially characterizes the covariate distribution due to its size. The accuracy metrics provide measures of agreement between the true and predicted outcomes and therefore depend on the covariate distribution. We thus expect to decrease estimation precision by incorporating $\mathcal{U}$ into estimation. In taking an imputation-based approach, we rely on our estimate of m(u) to capture the dependency of y on u in order to glean information from $\mathcal{U}$. While a fully nonparametric method such as kernel smoothing allows for complete flexibility in estimating m(u), smoothing generally does not perform well with moderate p due to the curse of dimensionality [Kpotufe, 2010]. To overcome this challenge and allow for a rich model for m(u), we incorporate some parametric structure into the imputation step via basis function regression.

Let $\mathbf{\text{Φ}}(\mathbf{u})$ be a finite set of basis functions with fixed dimension that includes x. We fit a working model

$$\text{P}(y=1\mid \mathbf{u})=g\left\{{\gamma }^{\top }\mathbf{\text{Φ}}(\mathbf{u})\right\}$$ (2)

to $\mathcal{L}$ and impute y as $g\left\{{\tilde{\gamma }}^{\top }\mathbf{\text{Φ}}(\mathbf{u})\right\}$ where $\tilde{\gamma }$ is the solution to

$${\widetilde{\mathbf{\text{Q}}}}_n(\gamma )=\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{\text{Φ}}}_i\left[y_i-g\left\{{\gamma }^{\top }{\mathbf{\text{Φ}}}_i\right\}\right]-{\lambda }_n\gamma =\mathbf{0}$$ (3)

${\mathbf{\text{Φ}}}_i=\mathbf{\text{Φ}}\left({\mathbf{u}}_i\right)$ and ${\lambda }_n=o\left(n^{-\frac{1}{2}}\right)$ is a tuning parameter to ensure stable fitting. Under Conditions 1–3 given in Section 5, we argue in Appendix D that $\tilde{\gamma }$ is a regular root-n consistent estimator for the unique solution, $\tilde{\gamma }$, to $\mathbf{\text{Q}}(\gamma )=\text{E}\left\{\mathbf{\text{Φ}}(\mathbf{u})\left[y-g\left\{{\gamma }^{\top }\mathbf{\text{Φ}}(\mathbf{u})\right\}\right]\right\}=\mathbf{0}$. We take our initial imputations as ${\tilde{m}}_{\text{I}}(\mathbf{u})=g\left\{{\tilde{\gamma }}^{\top }\mathbf{\text{Φ}}(\mathbf{u})\right\}$.

With y imputed as ${\tilde{m}}_{\text{I}}(\mathbf{u})$, we may also obtain a simple SS estimator for $\overline{\mathit{\theta }}$, ${\check{\mathit{\theta }}}_{\text{SSL}}$, as the solution to

$${\widehat{\mathbf{U}}}_N(\mathit{\theta })=\frac{1}{N}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\tilde{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}=\mathbf{0}.$$

The asymptotic behaviour of ${\check{\mathit{\theta }}}_{\text{SSL}}$ is presented and compared with ${\widehat{\mathit{\theta }}}_{\text{SL}}$ in the Appendix. When the working regression model (1) is correctly specified, it is shown that ${\widehat{\mathit{\theta }}}_{\text{SL}}$ is fully efficient and ${\check{\mathit{\theta }}}_{\text{SSL}}$ is asymptotically equivalent to ${\widehat{\mathit{\theta }}}_{\text{SL}}$. When the outcome model in (1) is not correctly specified, but the imputation model in (2) is correctly specified, we show in Appendix E that ${\check{\mathit{\theta }}}_{\text{SSL}}$ is more efficient than ${\widehat{\mathit{\theta }}}_{\text{SL}}$. When the imputation model is also misspecified, ${\check{\mathit{\theta }}}_{\text{SSL}}$ tends to be more efficient than ${\widehat{\mathit{\theta }}}_{\text{SL}}$, but the efficiency gain is not theoretically guaranteed. We therefore obtain the final SS estimator, denoted as ${\widehat{\mathit{\theta }}}_{\text{SSL}}={\left({\widehat{\theta }}_{\text{SSL},0},\dots ,{\widehat{\theta }}_{\text{SSL},p}\right)}^{\top }$, as a linear combination of ${\widehat{\mathit{\theta }}}_{\text{SL}}$ and ${\check{\mathit{\theta }}}_{\text{SSL}}$ to minimize the asymptotic variance. Details are provided in Appendices A and B.

3.2. Step 2: Robustness augmentation

To obtain an efficient SS estimator for $\overline{D}=D(\overline{\mathit{\theta }})$, we note that

$$d(y,\mathcal{Y})=y(1-2\mathcal{Y})+{\mathcal{Y}}^2$$ (4)

is linear in y when $y\in \{0,1\}$. With a given estimate of $m(\cdot )$, denoted by $\tilde{m}(\cdot )$, a SS estimate of $\overline{D}$ can be obtained as

$$\frac{1}{N}\sum _{i=1}^{N}d\left\{\tilde{m}\left({\mathbf{u}}_i\right),\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}.$$

However, $\tilde{m}(\mathbf{u})$ needs to be carefully constructed to ensure that the resulting estimator is consistent for $\overline{D}$ under possible misspecification of the imputation model. Using the expression in (4), a sufficient condition to guarantee consistency for $\overline{D}$ is that

$$\text{E}\left[\{y_0-\tilde{m}\left({\mathbf{u}}_0\right)\}\{1-2\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_0\right)\}\mid \tilde{m}(\cdot )\right]\stackrel{p}{\to }0\text{as}n\to \infty .$$ (5)

This condition implies that $\text{E}\left[d\left\{y_0,\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_0\right)\right\}-d\left\{\tilde{m}\left({\mathbf{u}}_0\right),\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_0\right)\right\}\right]\stackrel{p}{\to }0$. Unfortunately, ${\tilde{m}}_{\text{I}}(\mathbf{u})=g\left\{{\tilde{\gamma }}^{\top }\mathbf{\text{Φ}}(\mathbf{u})\right\}$ does not satisfy (5) when (2) is misspecified. To ensure that (5) holds regardless of the adequacy of the imputation model used for estimating the regression parameters, we augment the initial imputation ${\tilde{m}}_{\text{I}}(\mathbf{u})$ as

$${\tilde{m}}_{\text{II}}(\mathbf{u};\mathit{\theta })=g\left\{{\tilde{\gamma }}^{\top }\mathbf{\Phi }(\mathbf{u})+{\tilde{\mathit{\nu }}}_{\mathit{\theta }}^{\top }{\mathbf{z}}_{\mathit{\theta }}\right\}$$

where ${\mathbf{z}}_{\theta }={\left[1,\mathcal{Y}\left({\mathit{\theta }}^{\top }\mathbf{x}\right)\right]}^{\top }$ and ${\tilde{\mathit{\nu }}}_{\theta }$ is the solution to the IPW estimating equation

$${\mathbf{P}}_n(\mathit{\nu },\mathit{\theta })=\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i\left\{y_i-g\left({\tilde{\gamma }}^{\top }{\mathbf{\Phi }}_i+{\mathit{\nu }}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right)\right\}{\mathbf{z}}_{i\mathit{\theta }}=\mathbf{0}\text{for any given}\mathit{\theta }.$$ (6)

We let ${\overline{\mathit{\nu }}}_{\mathit{\theta }}$ be the limiting value of ${\tilde{\mathit{\nu }}}_{\mathit{\theta }}$ which solves the limiting estimation equation

$$\mathbf{R}(\mathit{\nu }\mid \mathit{\theta })=\text{E}\left({\mathbf{z}}_{\mathit{\theta }}\left[y-g\left\{{\overline{\gamma }}^{\top }\mathbf{\text{Φ}}(\mathbf{u})+{\mathit{\nu }}^{\top }{\mathbf{z}}_{\mathit{\theta }}\right\}\right]\right)=\mathbf{0}.$$ (7)

This estimating equation is monotone in z_θ for any θ and thus ${\overline{\mathit{\nu }}}_{\mathit{\theta }}$ exists under mild regularity conditions. It also follows from (7) that (i) $\text{E}\left(\left[y-g\left\{{\overline{\gamma }}^{\top }\mathbf{\text{Φ}}(\mathbf{u})+{\overline{\mathit{\nu }}}_{\mathit{\theta }}^{\top }{\mathbf{z}}_{\mathit{\theta }}\right\}\right]\right)=0$ and (ii) $\text{E}\left(\mathcal{Y}\left({\mathit{\theta }}^{\top }\mathbf{x}\right)\left[y-g\left\{{\overline{\gamma }}^{\top }\mathbf{\Phi }(\mathbf{u})+{\overline{\mathit{\nu }}}_{\mathit{\theta }}^{\top }{\mathbf{z}}_{\mathit{\theta }}\right\}\right]\right)=0$ which ensure that the sufficiency condition in (5) is satisfied. We thus construct a SS estimator of D(θ) as

$${\widehat{D}}_{\text{SSL}}(\mathit{\theta })=\frac{1}{N}\sum _{i=1}^{N}d\left\{{\tilde{m}}_{\text{II}}\left({\mathbf{u}}_i;\mathit{\theta }\right),\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}.$$

In Section 5.2, we present the asymptotic properties of ${\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)$ and ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)$ and compare ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)$ with its supervised counterpart, ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)$. Similar to the SS estimation of $\overline{\mathit{\theta }}$, it is shown in Appendix F that the unlabeled data helps to reduce the asymptotic variance of ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)$. Specifically, ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)$ is shown to be asymptotically more efficient than ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)$ when the imputation model is correct. In practice, however, we may want to use ${\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)$ instead of ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)$ to achieve improved finite sample performance.

4. Bias Correction via Ensemble Cross-Validation

Similar to the supervised estimators of the prediction performance measures, the proposed plug-in estimator uses the labeled data for both constructing and evaluating the prediction model and is therefore prone to overfitting bias [Efron, 1986]. K-fold cross-validation (CV) is a commonly used method to correct for such bias. However, it has been observed that CV tends to result in overly pessimistic estimates of accuracy measures, particularly when n is not very large relative to p [Jiang and Simon, 2007]. Bias correction methods such as the 0.632 bootstrap have been proposed to address this behavior [Efron, 1983, Efron and Tibshirani, 1997, Fu et al., 2005, Molinaro et al., 2005]. Here, we propose an alternative ensemble CV procedure that takes a weighted sum of the apparent and K-fold CV estimators.

We first construct a K-fold CV estimator by randomly partitioning $\mathcal{L}$ into K disjoint folds of roughly equal size, denoted by $\left\{{\mathcal{L}}_k,k=1,\dots ,K\right\}$. Since N is assumed to be sufficiently large, no CV is necessary for projecting to the full data. For a given k, we use $\mathcal{L}/{\mathcal{L}}_k$ to estimate $\overline{\gamma }$ and $\overline{\mathit{\theta }}$, denoted as ${\tilde{\gamma }}_{(-k)}$ and ${\widehat{\mathit{\theta }}}_{(-k)}$, respectively. The $n_k$ observations in ${\mathcal{L}}_k$ are used in the augmentation step to obtain ${\tilde{\mathit{\nu }}}_{{\widehat{\mathit{\theta }}}_{(-k)},k}$, the solution to ${\mathbf{P}}_{n_k}\left(\mathit{\nu },{\widehat{\mathit{\theta }}}_{(-k)}\right)=0$. For the k^th fold, we estimate the accuracy measure as

$${\widehat{D}}_k\left({\widehat{\mathit{\theta }}}_{(-k)}\right)=N^{-1}\sum _{i=1}^{N}d\left\{{\tilde{m}}_{\text{II},k}\left({\mathbf{u}}_i\right),\mathcal{Y}\left({\widehat{\mathit{\theta }}}_{(-k)}^{\top }{\mathbf{x}}_i\right)\right\},$$

where ${\tilde{m}}_{\text{II},k}\left({\mathbf{u}}_i\right)=g\left({\tilde{\gamma }}_{(-k)}^{\top }{\mathbf{\Phi }}_i+{\tilde{\mathit{\nu }}}_{{\widehat{\mathit{\theta }}}_{(-k)},k}^{\top }{\mathbf{Z}}_{i{\widehat{\mathit{\theta }}}_{(-k)}}\right)$, and take the final CV estimator as ${\widehat{D}}_{\text{SSL}}^{\text{cv}}=K^{-1}\sum _{k=1}^K{\widehat{D}}_k\left({\widehat{\mathit{\theta }}}_{(-k)}\right)$. In practice, we suggest averaging over several replications of CV to remove the variation due to the CV partition. We then obtain the weighted CV estimator with

$${\widehat{D}}_{\text{SSL}}^{\omega }=\omega {\widehat{D}}_{\text{SSL}}+(1-\omega ){\widehat{D}}_{\text{SSL}}^{\text{cv}},\text{where}\omega =K/(2K-1).$$

We may similarly obtain a CV-based supervised estimator, denoted by ${\widehat{D}}_{\text{SL}}^{\text{cv}}$, as well as the corresponding weighted estimator, ${\widehat{D}}_{\text{SL}}^{\omega }$. Note that the fraction of observations from stratum s in the kth fold, ${\widehat{\rho }}_{1s,k}$, deviates from ${\widehat{\rho }}_{1s}$ in the order of $O\left(\sqrt{n_s}/n\right)$. Although this deviation is asymptotically negligibile, it may be desirable to perform the K-fold partition within each strata to ensure that ${\widehat{\rho }}_{1s,k}={\widehat{\rho }}_{1s}$ when n_s is small or moderate.

Using similar arguments as those given in Tian et al. [2007], it is not difficult to show that $n^{\frac{1}{2}}\left({\widehat{D}}_{\text{SSL}}-\overline{D}\right)$ and $n^{\frac{1}{2}}\left({\widehat{D}}_{\text{SSL}}^{\text{cv}}-\overline{D}\right)$ are first-order asymptotically equivalent. Thus, the ensemble CV estimator ${\widehat{D}}_{\text{SSL}}^{\omega }$ reduces the higher order bias of ${\widehat{D}}_{\text{SSL}}$ and ${\widehat{D}}_{\text{SSL}}^{\text{cv}}$, but has the same asymptotic distribution. Although the empirical performance is promising, it is difficult to rigorously study the bias properties of ${\widehat{D}}_{\text{SSL}}^{\omega }$ as the regression parameter doesn’t necessarily minimize the loss function, $\widehat{D}(\mathit{\theta })$. We provide a heuristic justification of the ensemble CV method in Appendix H which assumes $\widehat{\mathit{\theta }}$ minimizes $\widehat{D}(\mathit{\theta })$.

5. Asymptotic Analysis

We next present the asymptotic properties of our proposed SS estimator of $\overline{D}$. To facilitate our presentation, we first discuss the properties of ${\check{\mathit{\theta }}}_{\text{SSL}}$ as the accuracy parameter estimates inherently depend on the variability in estimating $\overline{\mathit{\theta }}$. We then present our main result high-lighting the efficiency gain of our proposed SS approach for accuracy parameter estimation. We conclude our theoretical analysis with two practical discussions of (i) intrinsic efficient estimation in the SS setting and (ii) optimal allocation in stratified sampling.

For our asymptotic analysis, we let ${\mathbf{\text{Σ}}}_1\succ {\mathbf{\text{Σ}}}_2$ if ${\mathbf{\text{Σ}}}_1-{\mathbf{\text{Σ}}}_2$ is positive definite and ${\mathbf{\text{Σ}}}_1⪰{\mathbf{\text{Σ}}}_2$ if ${\mathbf{\text{Σ}}}_1-{\mathbf{\text{Σ}}}_2$ is positive semi-definite for any two symmetric matrices Σ₁ and Σ₂. For any matrix M and vectors v₁ and v₂, M_j· represents the j^th row vector, ${\mathbf{v}}_1^{\otimes 2}={\mathbf{v}}_1{\mathbf{v}}_1^{\top }$, and $\left\{{\mathbf{v}}_1,{\mathbf{v}}_2\right\}={\left({\mathbf{v}}_1^{\top },{\mathbf{v}}_2^{\top }\right)}^{\top }$ is the vector concatenating v₁ and v₂. To establish our theoretical results, we recall that ${\widehat{\rho }}_{1s}$ and ${\widehat{\rho }}_s$ converge to some fixed values ρ_1s and ρ_s in probability, as assumed in Section 2.1, and introduce the following three conditions.

Condition 1.

The basis Φ(u) contains x, has compact support, and is of fixed dimension. The density function for x, denoted by p(x), and P(y = 1 | u) are continuously differentiable in the continuous components of x and u, respectively. There is at least one continuous component of x with corresponding non-zero component in $\overline{\mathit{\theta }}$.

Condition 2.

The link function g(·) is continuously differentiable with derivative $\dot{g}(\cdot )$.

Condition 3.

(A) There is no vector γ such that $P\left({\gamma }^{\top }{\mathbf{\text{Φ}}}_1>{\gamma }^{\top }{\mathbf{\text{Φ}}}_2\mid y_1>y_2\right)=1$ and $\text{E}\left[{\mathbf{\text{Φ}}}^{\otimes 2}\dot{g}\left\{{\overline{\gamma }}^{\top }\mathbf{\text{Φ}}\right\}\right]>\mathbf{0}$. (B) There is a small neighborhood of $\overline{\mathit{\theta }},\mathbf{\Theta }=\left\{\mathit{\theta }:\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<\delta \right\}$ for some δ > 0, such that for any θ ∈ Θ, there is no vector r such that $P({\mathbf{r}}^{\top }\left\{{\mathbf{\text{Φ}}}_1,{\mathbf{z}}_{1\mathit{\theta }}\right\}>{\mathbf{r}}^{\top }\left\{{\mathbf{\Phi }}_2,{\mathbf{Z}}_{2\mathit{\theta }}\right\}\mid y_1>y_2)=1$ and $E\left[{\mathbf{z}}_{\mathit{\theta }}^{\otimes 2}\dot{g}\left\{{\overline{\gamma }}^{\top }\mathbf{\text{Φ}}+{\overline{\mathit{\nu }}}_{\mathit{\theta }}^{\top }{\mathbf{z}}_{\mathit{\theta }}\right\}\right]\succ \mathbf{0}$. (C) $\text{E}\left[{\mathbf{x}}^{\otimes 2}\dot{g}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\right]\succ \mathbf{0}$.

Remark 1.

Conditions 1–3 are commonly used regularity conditions in M-estimation theory and are satisfied in broad applications. Similar conditions can be found in Tian et al. [2007] and Section 5.3 of Van der Vaart [2000]. Condition 3(A) and 3(B) assume that there is no γ and ν such that ${\mathit{\gamma }}^{\top }\mathbf{\text{Φ}}+{\mathit{\nu }}^{\top }{\mathbf{z}}_{\mathit{\theta }}$ can perfectly separate the samples based on y. In our application of EHR data analysis, these conditions are typically satisfied as the outcomes of interest (i.e. disease status) do not perfectly depend on covariates such as billing codes, lab values, procedure codes, and other features extracted from free-text. Similar to Tian et al. [2007], Condition 3(A) ensures the existence and uniqueness of the limiting parameters $\overline{\mathit{\theta }}$ and $\overline{\gamma }$ and Condition 3(B) ensures the existence and uniqueness of ${\overline{\mathit{\nu }}}_{\mathit{\theta }}$.

5.1. Asymptotic Properties of ${\check{\mathit{\theta }}}_{\text{SSL}}$

The asymptotic properties of ${\check{\mathit{\theta }}}_{\text{SSL}}$ are summarized in Theorem 1 and the justification is provided in Appendix E.

Theorem 1.

Under Conditions 1–3, ${\check{\mathit{\theta }}}_{SSL}\stackrel{p}{\to }\overline{\mathit{\theta }}$, and

$${\widehat{\mathcal{W}}}_{SSL}=n^{\frac{1}{2}}\left({\check{\mathit{\theta }}}_{SSL}-\overline{\mathit{\theta }}\right)=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left(n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right){\mathbf{e}}_{SSLi}\right)+o_p(1)$$

which weakly converges to $N\left(\mathbf{0},{\mathbf{\Sigma }}_{SSL}\right)$ where

$${\mathbf{\Sigma }}_{SSL}=\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}E\left\{{\mathbf{e}}_{SSLi}^{\otimes 2}\mid {\mathcal{S}}_i=s\right\},{\mathbf{e}}_{SSLi}={\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\},\mathit{\text{and}}\mathbf{A}=\text{E}\left\{{\mathbf{x}}_i^{\otimes 2}\dot{g}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}.$$

Remark 2.

To contrast with the supervised estimator ${\widehat{\mathit{\theta }}}_{SL}$, we show in Appendix C that

$${\widehat{\mathcal{W}}}_{SL}=n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{SL}-\overline{\mathit{\theta }}\right)=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left(n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right){\mathbf{e}}_{SLi}\right)+o_p(1),$$

which weakly converges to $N\left(\mathbf{0},{\mathbf{\Sigma }}_{SL}\right)$ where

$${\mathbf{\Sigma }}_{SL}=\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}\text{E}\left\{{\mathbf{e}}_{SLi}^{\otimes 2}\mid {\mathcal{S}}_i=s\right\},\mathit{\text{and}}{\mathbf{e}}_{SLi}={\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}.$$

It follows that when the imputation model $P(y=1\mid \mathbf{u})=g\left\{{\overline{\gamma }}^{\top }\mathbf{\Phi }(\mathbf{u})\right\}$ is correctly specified, ${\mathbf{\text{Σ}}}_{SL}⪰{\mathbf{\text{Σ}}}_{SSL}$. When $P\left({\overline{\gamma }}^{\top }\mathbf{\text{Φ}}(\mathbf{u})\ne {\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)>0$, we have that ${\mathbf{\text{Σ}}}_{SL}\succ {\mathbf{\text{Σ}}}_{SSL}$.

5.2. Asymptotic Properties of ${\widehat{D}}_{\text{SSL}}\left({\check{\theta }}_{\text{SSL}}\right)$ and ${\widehat{D}}_{\text{SSL}}\left({\widehat{\theta }}_{\text{SSL}}\right)$

The asymptotic properties of ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)$ are summarized in Theorem 2 and the justification is provided in Appendix F.

Theorem 2.

Under Conditions 1–3, ${\widehat{D}}_{SSL}\left({\check{\mathit{\theta }}}_{SSL}\right)\stackrel{p}{\to }\overline{D}$, and ${\check{\mathcal{T}}}_{SSL}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{SSL}\left({\check{\mathit{\theta }}}_{SSL}\right)-D(\overline{\mathit{\theta }})\right\}$ is asymptotically Gaussian with mean zero and variance ${\sigma }_{SSL}^2$ given in Appendix F. Also, ${\check{\mathcal{T}}}_{SSL}$ is asymptotically equivalent to

$$n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left(n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right)\left[\left\{d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-d\left(m_{II,i},{\overline{\mathcal{Y}}}_i\right)\right\}+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{e}}_{SSLi}\right]\right),$$

where ${\overline{\mathcal{Y}}}_i=\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)$, $m_{II,i}=g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i+\overline{\nu }\overline{\mathit{\theta }}{\mathbf{z}}_{i\overline{\mathit{\theta }}}\right)$ is the imputation model based approximation to $\text{P}(y=1\mid \mathbf{u})$ and $\dot{\mathbf{D}}(\mathit{\theta })=\partial D(\mathit{\theta })/\partial \mathit{\theta }$.

Remark 3.

We also show that ${\widehat{\mathcal{T}}}_{SSL}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{SSL}\left({\widehat{\mathit{\theta }}}_{SSL}\right)-D(\overline{\mathit{\theta }})\right\}$ is asymptotically equivalent to

$$n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left(n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right)\left[\left\{d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-d\left(m_{II,i},{\overline{\mathcal{Y}}}_i\right)\right\}+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }\left\{\mathbf{W}{\mathbf{e}}_{SSLi}+(\mathbf{I}-\mathbf{W}){\mathbf{e}}_{SLi}\right\}\right]\right),$$

which is also asymptotically Gaussian with mean zero where W is a diagonal matrix defined in Appendix B.

Remark 4.

As shown in Appendix D, ${\widehat{\mathcal{T}}}_{SL}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{SL}\left({\widehat{\mathit{\theta }}}_{SL}\right)-D(\overline{\mathit{\theta }})\right\}$ is asymptotically Gaussian with mean zero and variance ${\sigma }_{SL}^2$ defined in Appendix D. It is equivalent to

$$n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left(n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right)\left[\left\{d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-D(\overline{\mathit{\theta }})\right\}+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{e}}_{SLi}\right]\right).$$

We verify in Appendix F that when the imputation model is correctly specified, the asymptotic variance of ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)$ is smaller than that of ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)$ regardless of the specification of the working regression model in (1). This is because the accuracy measures always depend on the marginal distribution of x and the proposed SS approach leverages $\mathcal{U}$. Therefore ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)$ is asymptotically more efficient than ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)$ even when model (1) is correctly specified and ${\widehat{\mathit{\theta }}}_{\text{SL}}$ is fully efficient.

While we cannot theoretically guarantee that the SS estimator is more efficient than the supervised estimator under misspecification of the imputation model, the first and dominating term in the influence function expansion corresponds to the variability from estimating the accuracy measure. Even when the imputation model is misspecified, it may still provide a close approximation to P(y = 1 | u) and therefore result in reduced variability relative to the supervised approach. The second term of the influence function corresponds to the variability from estimation of the regression parameter. As the SS estimator of the regression parameter is more efficient than its supervised counterpart under model misspecification, we also expect this term to have smaller variation than its supervised counterpart. In our simulation studies, we evaluate the performance of our proposals under various model misspecifications to assess whether this heuristic justification holds up empirically. We also further study this limitation from a theoretical viewpoint in the next section where we introduce a SS estimator with the intrinsic efficiency property from the semiparametric inference literature for comparison.

5.3. Intrinsic Efficient SS Estimation

For simplicity, we begin our discussion of intrinsic efficient estimation focusing on estimation of the regression parameter. Recall that the idea in Section 3.1 is to (i) solve $N^{-1}{\sum }_{i=1}^N{\widehat{w}}_i{\mathbf{\text{Φ}}}_i\left[y_i-g\left\{{\gamma }^{\top }{\mathbf{\text{Φ}}}_i\right\}\right]-{\lambda }_n\gamma =\mathbf{0}$ to obtain estimated coefficients $\tilde{\gamma }$ for imputation and then (ii) solve $N^{-1}{\sum }_{i=1}^N{\mathbf{x}}_i\left\{g\left({\tilde{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}=\mathbf{0}$ to obtain the SS estimator, ${\widehat{\mathit{\theta }}}_{\text{SSL}}$. By Theorem 1, for any $\mathit{e}\in {ℝ}^{p+1}\setminus \left\{\mathbf{0}\right\}$, the asymptotic variance of $n^{\frac{1}{2}}\left({\mathit{e}}^{\top }{\widehat{\mathit{\theta }}}_{\text{SSL}}-{\mathit{e}}^{\top }\overline{\mathit{\theta }}\right)$ can be expressed as:

$$\frac{1}{n}\sum _{i=1}^n{\zeta }_i{\left(e^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\right)}^2{\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)\right\}}^2,$$ (8)

where ${\zeta }_i={\sum }_{s=1}^S{\rho }_s^2{\rho }_{1s}^{-2}{V}_{i}I\left({\mathcal{S}}_i=s\right)$ for each $i\in \{1,2,\dots ,N\}$. When the imputation model $\text{P}(y=1\mid \mathbf{u})=g\left({\gamma }^{\top }\mathbf{\Phi }\right)$ is misspecified, an alternative estimating equation for γ may be used to directly reduce the asymptotic variance of the resulting SS estimator. Specifically, for a fixed Φ, we may find the estimating equation for γ that leads to the lowest asymptotic variance of the estimator for ${\mathit{e}}^{\top }\overline{\mathit{\theta }}$, a property referred to as “intrinsic efficiency” in the semiparametric inference literature [Tan, 2010]. We briefly propose estimation procedures for an estimator achieving this property with potential to improve upon our original proposal under potential misspecification of the imputation model.

To directly minimize the asymptotic variance of the SS estimator of ${\mathit{e}}^{\top }\overline{\mathit{\theta }}$ given by (8), we obtain the estimated coefficients for the imputation model with

$$\begin{gathered} {\tilde{\gamma }}^{(1)}=\underset{\gamma }{\text{argmin}}\frac{1}{2n}\sum _{i=1}^n{\widehat{\zeta }}_i{\left({\mathit{e}}^{\top }{\widehat{\mathbf{A}}}^{-1}{\mathbf{x}}_i\right)}^2{\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\Phi }}_i\right)\right\}}^2+{\lambda }_n^{(1)}\parallel \gamma {\parallel }_2^2, \\ \text{s.t.}\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{x}}_i\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\Phi }}_i\right)\right\}=\mathbf{0}, \end{gathered}$$ (9)

where ${\widehat{\zeta }}_i={\sum }_{s=1}^S{\widehat{\rho }}_s^2{\widehat{\rho }}_{1s}^{-2}{V}_{i}I\left({\mathcal{S}}_i=s\right)$, $\widehat{\mathbf{A}}=N^{-1}{\sum }_{i=1}^N{\mathbf{x}}_i^{\otimes 2}\dot{g}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}^{\top }{\mathbf{x}}_i\right)$ are the empirical estimates of ${\zeta }_i$ and A, respectively, and ${\lambda }_n^{(1)}=O\left(n^{-\frac{1}{2}}\right)$ is again a tuning parameter for stable fitting. We then solve $N^{-1}{\sum }_{i=1}^N{\mathbf{x}}_i\left\{g\left({\tilde{\gamma }}^{(1)\top }{\mathbf{\Psi }}_i\right)-g\left({\theta }^{\top }{\mathbf{x}}_i\right)\right\}=\mathbf{0}$ to obtain ${\widehat{\mathit{\theta }}}_{\text{intri}}$, and return ${\mathit{e}}^{\top }{\widehat{\mathit{\theta }}}_{\text{intri}}$ as the intrinsic efficient estimator for ${\mathit{e}}^{\top }\overline{\mathit{\theta }}$. The moment condition in (9) is used for calibrating the potential bias from a misspecified imputation model and ensuring the consistency of ${\widehat{\mathit{\theta }}}_{\text{intri}}$. This condition is explicitly imposed when constructing our original proposal.

To study the asymptotic properties of ${\widehat{\mathit{\theta }}}_{\text{intri}}$ and compare it with our original proposal, ${\widehat{\mathit{\theta }}}_{\text{SSL}}$, we let

$${\overline{\gamma }}^{(1)}=\underset{\gamma }{\text{argmin}}\text{E}\left[R{\left({\mathit{e}}^{\top }{\mathbf{A}}^{-1}\mathbf{x}\right)}^2{\left\{y-g\left({\gamma }^{\top }\mathbf{\Phi }\right)\right\}}^2\right],\text{s.t.}\text{E}\left[\mathbf{x}\left\{y-g\left({\gamma }^{\top }\mathbf{\Phi }\right)\right\}\right]=\mathbf{0},$$

be the limit of ${\tilde{\gamma }}^{(1}$, where $R={\sum }_{s=1}^{S}I(\mathcal{S}=s){\rho }_s/{\rho }_{1s}$. The proof of Theorem 3 is provided in Appendix G.2.

Theorem 3.

Under condition 1, and conditions A1 and A2 introduced in Appendix G.2, $n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{intri}-\overline{\mathit{\theta }}\right)$ converges weakly to a mean zero normal distribution, and is asymptotically equivalent to $\widehat{\mathcal{W}}\left({\overline{\gamma }}^{(1)}\right)$ where

$$\widehat{\mathcal{W}}(\gamma )=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right){\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\Phi }}_i\right)\right\}\right].$$

In addition: (i) when the imputation model $\text{P}(y=1\mid \mathbf{u})=g\left({\gamma }^{\top }\mathbf{\text{Φ}}\right)$ is correctly specified, ${\widehat{\mathit{\theta }}}_{intri}$ is asymptotically equivalent to ${\widehat{\mathit{\theta }}}_{SSL}$ and (ii) the asymptotic variance of $n^{\frac{1}{2}}\left({\mathit{e}}^{\top }{\widehat{\mathit{\theta }}}_{intri}-{\mathit{e}}^{\top }\overline{\mathit{\theta }}\right)$ is minimized among estimators with $\left\{{\mathit{e}}^{\top }\widehat{\mathcal{W}}(\gamma ):\text{E}\left[\mathbf{x}\left\{y-g\left({\gamma }^{\top }\mathbf{\Phi }\right)\right\}\right]=\mathbf{0}\right\}$. Consequently, the variance of the intrinsic efficient estimator is always less than or equal to the asymptotic variance of both $n^{\frac{1}{2}}\left({\mathit{e}}^{\top }{\widehat{\mathit{\theta }}}_{SL}-{\mathit{e}}^{\top }\overline{\mathit{\theta }}\right)$ and $n^{\frac{1}{2}}\left({\mathit{e}}^{\top }{\widehat{\mathit{\theta }}}_{SSL}-{\mathit{e}}^{\top }\overline{\mathit{\theta }}\right)$.

The details and theoretical analysis of the intrinsic efficient estimation procedure of the accuracy measure $\overline{D}$ is presented Appendices G.1 and G.2. Similar to Theorem 3, we show that ${\widehat{D}}_{\text{intri}}$ is asymptotically equivalent with our proposal, ${\widehat{D}}_{\text{SSL}}$, when the imputation model is correctly specified and has smaller asymptotic variance than ${\widehat{D}}_{\text{SSL}}$ when the imputation model is misspecified. However, it is important to note that estimation based on intrinsic efficiency is a non-convex problem and one may encounter numerical optimization issues which may limit its use in practice. We provide simulation studies comparing the intrinsic efficient estimator to the proposed approach in Section S4 Supplement.

5.4. Optimal Allocation in Stratified Sampling

Another important practical issue is how to select the strata and the corresponding selection probabilities. Here we provide here a detailed assessment of the optimal (or Neyman) allocation of the labeled data across the strata. Specifically, the general form of the influence function for our estimators is

$$n^{1/2}\sum _{s=1}^S{\rho }_s\left\{n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right)f\left({\mathbf{F}}_i\right)\right\}+o_p(1)$$

for a function f with ${\sigma }_s^2=\text{E}\left\{f^2\left({\mathbf{F}}_i\right)\mid {\mathcal{S}}_i=s\right\}$ and $\text{E}\left\{f\left({\mathbf{F}}_i\right)\right\}=0$. The asymptotic variance can then be expressed as

$$\begin{gathered} \sum _{s=1}^S{\rho }_s^2\left\{n_s^{-2}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right)\text{E}\left\{f^2\left({\mathbf{F}}_i\right)\mid {\mathcal{S}}_i=s\right\}\right\}=\sum _{s=1}^S{\rho }_s^2\frac{{\sigma }_s^2}{n_s} \\ =n^{-1}\sum _{s=1}^S{n}_s\sum _{s=1}^S\frac{{\left({\rho }_s{\sigma }_s\right)}^2}{n_s}⩾n^{-1}{\left(\sum _{s=1}^S{\rho }_s{\sigma }_s\right)}^2, \end{gathered}$$

by the Cauchy-Schwarz inequality, and equality holds if and only if

$$n_s=n\frac{{\rho }_s{\sigma }_s}{{\sum }_{s=1}^S{\rho }_s{\sigma }_s}\text{for}s=1,\dots ,S.$$ (10)

The optimal sampling probabilities are therefore proportional to (i) the relative stratum size and (ii) the variability within the stratum, with greater weight placed on large stratum with high variability. Consequently, stratified sampling leads to a more efficient estimator than uniform random sampling when the allocation in (10) is used.

Remark 5.

There is a rich body of survey sampling literature concerning model-assisted approaches that address the practically important question of how to select the strata and the corresponding selection probabilities [Neyman, 1934, Särndal et al., 2003, Nedyalkova and Tillé, 2008]. The optimal allocation given by (10) is in similar spirit with the sampling schemes used in Cai and Zheng [2012], Liu et al. [2012]. It is particularly useful for HER-based phenotyping studies such as the diabetic neuropathy example in Section 8 as it is often straightforward for domain experts to define a filter variable(s) that yields relatively large stratum with increased prevalence of y (e.g. patients with notes containing terms related to the disease, relevant lab values, or specialist visits) and thus increased variability.

We provide additional numerical studies to illustrate Remark 5 in Section S5 of the Supplement.

6. Perturbation Resampling Procedure for Inference

We next propose a perturbation resampling procedure to construct standard error (SE) and confidence interval (CI) estimates in finite samples. Resampling procedures are particularly attractive for making inference about $\overline{D}$ when $\mathcal{Y}={\mathcal{Y}}_2$ since ${\widehat{D}}_{\text{SSL}}(\mathit{\theta })$ is not differentiable in θ. To this end, we generate a set of independent and identically distributed (i.i.d) non-negative random variables, $\mathcal{G}=\left(G_1,\dots ,G_n\right)$, independent of $\mathcal{D}$, from a known distribution with mean one and unit variance.

For each set of $\mathcal{G}$, we first obtain a perturbed version of ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ as

$${\widehat{\mathit{\theta }}}^{\ast }={\widehat{\mathit{\theta }}}_{\text{SSL}}+{\widehat{\mathbf{A}}}^{-1}\sum _{k=1}^K\sum _{i\in {\mathcal{L}}_k}\frac{{\widehat{w}}_i\left(G_i-1\right)}{{\sum }_{j=1}^n{\widehat{w}}_j}\left[{\mathbf{x}}_i{y}_i-\widehat{\mathbf{W}}{\mathbf{x}}_{i}g\left({\tilde{\gamma }}_{(-k)}^{\top }{\mathbf{\text{Φ}}}_i\right)-(\mathbf{I}-\widehat{\mathbf{W}}){\mathbf{x}}_{i}g\left({\widehat{\mathit{\theta }}}_{(-k)}^{\top }{\mathbf{x}}_i\right)\right],$$

where $\widehat{\mathbf{A}}=N^{-1}{\sum }_{i=1}^N{\mathbf{x}}_i^{\otimes 2}\dot{g}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}^{\top }{\mathbf{x}}_i\right)$. We use CV to correct for variance underestimation due to overfitting. Next, we find the solution ${\tilde{\gamma }}^{\ast }$ to the perturbed objective function

$${\widetilde{\mathbf{\text{Q}}}}_n^{\ast }(\gamma )=\frac{{\sum }_{i=1}^n{\widehat{w}}_i{\mathbf{\text{Φ}}}_i\left[y_i-g\left({\gamma }^{\top }{\mathbf{\text{Φ}}}_i\right)\right]G_i}{{\sum }_{i=1}^n{\widehat{w}}_i{G}_i}-{\lambda }_n\gamma =\mathbf{0}$$ (11)

and the solution ${\tilde{\mathit{\nu }}}^{\ast }$ that solves

$${\mathbf{P}}_n^{\ast }\left(\mathit{\nu },{\widehat{\mathit{\theta }}}^{\ast }\right)=\frac{{\sum }_{i=1}^n{\widehat{w}}_i\left\{y_i-g\left({\tilde{\gamma }}^{\ast }{\mathbf{\Phi }}_i+{\mathit{\nu }}^{\top }{\mathbf{z}}_{i{\widehat{\mathit{\theta }}}^{\ast }}\right)\right\}{\mathbf{z}}_{i{\widehat{\mathit{\theta }}}^{\ast }}{G}_i}{{\sum }_{i=1}^n{\widehat{w}}_i{G}_i}=\mathbf{0}$$

to obtain perturbed counterparts of $\tilde{\gamma }$ and $\tilde{\mathit{\nu }}$, respectively. We then compute ${\tilde{m}}_{11}^{\ast }\left({\mathbf{u}}_i\right)=g\left({\tilde{\gamma }}^{\ast \top }{\mathbf{\text{Φ}}}_i+{\tilde{\mathit{\nu }}}^{\ast \top }{\mathbf{z}}_{i{\widehat{\mathit{\theta }}}^{\ast }}\right)$ and obtain the perturbed estimator of ${\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)$ as

$${\widehat{D}}_{\text{SSL}}^{\ast }\left({\widehat{\mathit{\theta }}}^{\ast }\right)=N^{-1}\sum _{i=1}^N\left[{\tilde{m}}_{11}^{\ast }\left({\mathbf{u}}_i\right)\left\{1-2\mathcal{Y}\left({\widehat{\mathit{\theta }}}^{\ast \top }{\mathbf{x}}_i\right)\right\}+{\mathcal{Y}}^2\left({\widehat{\mathit{\theta }}}^{\ast \top }{\mathbf{x}}_i\right)\right].$$

Following arguments such as those in Tian et al. [2007], one may verify that $n^{\frac{1}{2}}\{{\widehat{D}}_{\text{SSL}}^{\ast }\left({\widehat{\mathit{\theta }}}_{\text{SSL}}^{\ast }\right)-{\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)\}\mid \mathcal{D}$ converges to the same limiting distribution as ${\widehat{\mathcal{T}}}_{\text{SSL}}$. Additionally, it may be shown that ${\widehat{\mathcal{T}}}_{\text{SSL}}^{\text{cv}}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}^{\text{cv}}-D(\overline{\mathit{\theta }})\right\}$ and hence ${\widehat{\mathcal{T}}}_{\text{SSL}}^{\omega }=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}^{\omega }-D(\overline{\mathit{\theta }})\right\}$ converge to the limiting distribution of ${\widehat{\mathcal{T}}}_{\text{SSL}}$. We utilize these results to approximate the distribution of ${\widehat{\mathcal{T}}}_{\text{SSL}}^{\omega }$ with the empirical distribution of a large number of perturbed estimates using the above resampling procedure to base inference for $D(\overline{\mathit{\theta }})$ on the proposed bias-corrected estimator. The variance of ${\widehat{D}}_{\text{SSL}}^{\omega }$ can correspondingly be estimated with the sample variance and confidence intervals may be constructed accordingly.

7. Simulation Studies

We conducted extensive simulation studies to evaluate the performance of the proposed SSL procedures and to compare to existing methods. Throughout, we generated p = 10 dimensional covariates x from N(0, C) with ${\mathbf{C}}_{kl}=3{(0.4)}^{|k-l|}$. Stratified sampling was performed according to $\mathcal{S}$ generated from the following two mechanisms:

1. $\mathcal{S}\in \{1,S=2\}\text{with}\mathcal{S}=1+I\left(x_1+{\delta }_1⩽0.5\right)\text{and}{\delta }_1\sim N(0,1).$

2. $\mathcal{S}\in \{1,2,3,S=4\}\text{with}\mathcal{S}=1+I\left(x_1+{\delta }_1⩽0.5\right)+2I\left(x_3+{\delta }_2⩽0.5\right),{\delta }_1\sim N(0,1),{\delta }_2\sim N(0,1),\text{and}{\delta }_1\perp {\delta }_2.$

We let $\mathbb{S}={(I(\mathcal{S}=1),\dots ,I(\mathcal{S}=S-1))}^{\top }$. For both settings, we sampled n_s = 100 or 200 observations from each stratum. Throughout, we let v₁ be the natural spline of x with 3 knots and v₂ be the interaction terms $\left\{{\mathbf{x}}_1:{\mathbf{x}}_{-1},{\mathbf{x}}_2:{\mathbf{x}}_{-(1,2)}\right\}$, where ${\mathbf{x}}_1:{\mathbf{x}}_{-1}$ and ${\mathbf{x}}_2:{\mathbf{x}}_{-(1,2)}$ represent interaction terms of x₁ with the remaining covariates and x₂ with covariates excluding x₁ and x₂, respectively. With $\mathit{\theta }={\left\{0,1,1,0.5,0.5,{\mathbf{0}}_{(p-4)\times 1}\right\}}^{\top }$ and ${ϵ}_{\text{logistic}}$ and ${ϵ}_{\text{extreme}}$ denoting noise generated from the logistic and extreme valuep−2,0.3q distributions, we simulated y from the following models:

1. $\left({\mathcal{M}}_{\text{correct}},{\mathcal{I}}_{\text{correct}}\right)$ with correct outcome model and correct imputation model:

   $$y=I\left({\mathit{\theta }}^{\top }\mathbf{x}+{ϵ}_{\text{logistic}}>2\right)\text{and}\mathbf{\text{Φ}}={\left(1,{\mathbf{x}}^{\top },{\mathbf{v}}_1^{\top },{\mathbb{S}}^{\top }\right)}^{\top };$$
2. $\left({\mathcal{M}}_{\text{incorrect}},{\mathcal{I}}_{\text{correct}}\right)$ with incorrect outcome model and correct imputation model:

   $$y=I\left[{\mathit{\theta }}^{\top }\mathbf{x}+0.5\left\{x_1{x}_2+x_1{x}_5-x_2{x}_6-I(\mathcal{S}=1)\right\}+{ϵ}_{\text{logistic}}>0\right]\text{and}\mathbf{\Phi }={\left(1,{\mathbf{x}}^{\top },{\mathbf{v}}_2^{\top },{\mathbb{S}}^{\top }\right)}^{\top };$$
3. $\left({\mathcal{M}}_{\text{incorrect}},{\mathcal{I}}_{\text{incorrect}}\right)$ with incorrect outcome model and incorrect imputation model:

   $$y=I\left\{{\mathit{\theta }}^{\top }\mathbf{x}+x_1^2+x_3^2+\exp \left(-2-3x_4-3x_6\right){ϵ}_{\text{extreme}}>2\right\}\text{and}\mathbf{\Phi }={\left(1,{\mathbf{x}}^{\top },{\mathbf{v}}_1^{\top },{\mathbb{S}}^{\top }\right)}^{\top }.$$

While the outcome model is misspecified in both (ii) and (iii), the misspecification is more severe in (iii) due to the higher magnitude of nonlinear effects. These configurations are chosen to mimic EHR settings where the signals are typically sparse and S is small. The covariate effects of 1 represent the strong signals from the main billing codes and free-text mentions of the disease of interest. The two weaker signals 0.5 characterize features such as related medications, signs, symptoms and lab results relevant to the disease of interest.

Across all settings, we compare our SS estimators to both the SL estimator and the alternative density ratio (DR) method [Kawakita and Kanamori, 2013, Kawakita and Takeuchi, 2014]. The basis function $\phi (\mathbf{u})$ required in the DR method was chosen to be the same as $\mathbf{\text{Φ}}(\mathbf{u})$ in our method for all settings. The details and theoretical properties of the DR method are further discussed in the Supplement. We employed the ensemble CV strategy to construct a bias corrected DR estimator for $\overline{D}$, denoted as ${\widehat{D}}_{\text{DR}}^{\omega }$, to ensure a fair comparison to our approach. The three settings of outcome and imputation models under (i), (ii), and (iii) allow us to verify the asymptotic efficiency of the proposed SSL procedures relative to the SL and DR methods under various scenarios of misspecification.

For each configuration, results are summarized with 500 independent data sets. The size of the unlabeled data was chosen to be 20,000 across all settings. For all our numerical studies including the real data application, CV was performed with either K = 3 or K = 6 and averaged over 20 replications. The estimated SEs were based on 500 perturbed realizations and the OMR was evaluated with c = 0.5. We let ${\lambda }_n=\log (2p)/n^{1.5}$ when fitting the ridge penalized logistic regression. We focus primarily on results for S = 2 and K = 6, but include results for S = 4 and K = 3 in Section S3 of the Supplement as they show similar patterns. Additionally, our analyses concentrate on the performance of the accuracy metrics. Results for the regression parameter estimates can be found in Section S2 of the Supplement. The code to implement the proposed methods and run the simulation studies can be found at https://github.com/jlgrons/Stratified-SSL.

In Figure 1, we present the percent biases of the apparent, CV, and ensemble CV estimators of the accuracy parameters. Although all three estimators have negligible biases, the SSL exhibits slightly less bias than its supervised counterpart and the DR estimator under $\left({\mathcal{M}}_{\text{incorrect}},{\mathcal{I}}_{\text{incorrect}}\right)$. The ensemble CV method is effective in bias correction while the apparent estimators are optimistic and the standard CV estimator is pessimistic. For example, under $\left({\mathcal{M}}_{\text{incorrect}},{\mathcal{I}}_{\text{incorrect}}\right)$, n_s = 100 and S = 2, the percent bias of the SSL estimators for the OMR are −8.2%, 8.8% and −0.5% when we use the plug-in, 6-fold CV, and the ensemble CV methods, respectively. The efficiency of ${\widehat{D}}_{\text{SSL}}^w$ and ${\widehat{D}}_{\text{DR}}^{\omega }$ relative to ${\widehat{D}}_{\text{SL}}^w$ for both the Brier scoreSL and OMR are presented in Figure 2. Again, ${\widehat{D}}_{\text{SSL}}^w$ is substantially more efficient than ${\widehat{D}}_{\text{SL}}^w$ and ${\widehat{D}}_{\text{DR}}^{\omega }$, with efficiency gains of approximately 15%–30% under $\left({\mathcal{M}}_{\text{correct}},{\mathcal{I}}_{\text{correct}}\right)$, 40% under $\left({\mathcal{M}}_{\text{correct}},{\mathcal{I}}_{\text{correct}}\right)$, and 40%–80% under $\left({\mathcal{M}}_{\text{correct}},{\mathcal{I}}_{\text{correct}}\right)$. Results for K = 3 have similar patterns and are presented in Figure S1 and S2 of the Supplement. In Table 1, we present the results for the interval estimation obtained from the perturbation resampling procedure for the SS estimator ${\widehat{D}}_{\text{SSL}}^{\omega }$. The SEs are well approximated and the empirical coverage for the 95% CIs is close to the nominal level across all settings.

Figure 1:

Percent biases of the apparent (AP), CV, and ensemble cross validation (eCV) estimators of the Brier score (BS) and overall misclassification rate (OMR) for SL, SSL and DR under (i) $\left({ℳ}_{\text{correct}},{ℐ}_{\text{correct}}\right)$, (ii) $\left({ℳ}_{\text{incorrect}},{ℐ}_{\text{correct}}\right)$, and (iii) $\left({ℳ}_{\text{incorrect}},{ℐ}_{\text{incorrect}}\right)$. Shown are the results obtained with K = 6 fold for CV.

Figure 2:

Relative efficiency (RE) of ${\widehat{D}}_{\text{SSL}}^{\omega }$ (SSL) and ${\widehat{D}}_{\text{DR}}^{\omega }$ compared to ${\widehat{D}}_{\text{SL}}^{\omega }$ for the Brier score (BS) and OMR under (i) $\left({ℳ}_{\text{correct}},{ℐ}_{\text{correct}}\right)$, (ii) $\left({ℳ}_{\text{incorrect}},{ℐ}_{\text{correct}}\right)$, and (iii) $\left({ℳ}_{\text{incorrect}},{ℐ}_{\text{incorrect}}\right)$. Shown are the results obtained with K = 6 fold CV.

Table 1:

The 100×ESE of ${\widehat{D}}_{\text{SL}}^w$, ${\widehat{D}}_{\text{SSL}}^w$ and ${\widehat{D}}_{\text{DR}}^w$ under (i) $\left({ℳ}_{\text{correct}},{ℐ}_{\text{correct}}\right)$; (ii) $\left({ℳ}_{\text{incorrect}},{ℐ}_{\text{correct}}\right)$, and (iii) $\left({ℳ}_{\text{incorrect}},{ℐ}_{\text{incorrect}}\right)$. For ${\widehat{D}}_{\text{SSL}}^w$, we also show the average of the 100 × ASE as well as the empirical coverage probability (CP) of the 95% confidence intervals constructed based on the resampling procedure.

(i) $\left({ℳ}_{\text{correct}},{ℐ}_{\text{correct}}\right)$

	Brier score				OMR
	${\widehat{D}}_{\text{SL}}^w$	${\widehat{D}}_{\text{SSL}}^w$		${\widehat{D}}_{\text{DR}}^w$	${\widehat{D}}_{\text{SL}}^w$	${\widehat{D}}_{\text{SSL}}^w$		${\widehat{D}}_{\text{DR}}^w$
	ESE	ESEASE	CP	ESE	ESE	ESEASE	CP	ESE
S = 2, ns = 100	1.27	1.191.25	0.94	1.31	2.29	2.102.23	0.97	2.35
S = 2, ns = 200	0.97	0.870.86	0.93	0.95	1.67	1.501.60	0.97	1.67
S = 4, ns = 100	0.97	0.890.85	0.93	0.95	1.70	1.541.58	0.95	1.64
S = 4, n = 200	0.67	0.580.60	0.95	0.66	1.16	1.021.11	0.97	1.16

(ii) $\left({ℳ}_{\text{incorrect}},{ℐ}_{\text{correct}}\right)$

	Brier score				OMR
	${\widehat{D}}_{\text{SL}}^w$	${\widehat{D}}_{\text{SSL}}^w$		${\widehat{D}}_{\text{DR}}^w$	${\widehat{D}}_{\text{SL}}^w$	${\widehat{D}}_{\text{SSL}}^w$		${\widehat{D}}_{\text{DR}}^w$
	ESE	ESEASE	CP	ESE	ESE	ESEASE	CP	ESE
S = 2, ns = 100	1.72	1.471.39	0.93	1.85	3.09	2.562.65	0.94	3.42
S = 2, ns = 200	1.13	1.010.94	0.92	1.16	2.14	1.851.85	0.95	2.21
S = 4, ns = 100	1.21	1.070.96	0.92	1.24	2.13	1.871.88	0.95	2.18
S = 4, ns = 200	0.86	0.730.67	0.93	0.87	1.62	1.371.34	0.94	1.62

(iii) $\left({ℳ}_{\text{incorrect}},{ℐ}_{\text{incorrect}}\right)$

	Brier score				OMR
	${\widehat{D}}_{\text{SL}}^w$	${\widehat{D}}_{\text{SSL}}^w$		${\widehat{D}}_{\text{DR}}^w$	${\widehat{D}}_{\text{SL}}^w$	${\widehat{D}}_{\text{SSL}}^w$		${\widehat{D}}_{\text{DR}}^w$
	ESE	ESEASE	CP	ESE	ESE	ESEASE	CP	ESE
S = 2, ns = 100	1.57	1.311.29	0.94	1.53	2.68	2.152.31	0.96	2.61
S = 2, ns = 200	1.05	0.860.85	0.95	1.01	1.87	1.451.54	0.97	1.83
S = 4, ns = 100	1.11	0.870.88	0.96	0.99	1.97	1.461.57	0.96	1.87
S = 4, ns = 200	0.78	0.610.61	0.95	0.73	1.37	1.021.08	0.97	1.30

Remark 6.

While our simulation studies focus on the SSL estimators proposed in Section 3, we also investigated the finite sample performance of ${\widehat{\mathit{\theta }}}_{intri}$ and ${\widehat{D}}_{intri}$ and compared them with our original proposals. The numerical studies are described in Section S4 of the Supplement and demonstrate that when the estimated coefficients for the imputation model of the original estimators are equal or close to those of the intrinsic efficient estimator, these two methods perform equivalently with respect to mean square errors (MSE). In contrast, under a setting where the coefficients for the imputation model differ across these two methods, ${\widehat{\mathit{\theta }}}_{intri}$ and ${\widehat{D}}_{intri}$ have about 30% smaller MSE than ${\widehat{\mathit{\theta }}}_{SSL}$ and ${\widehat{D}}_{SSL}$, on average. The detailed results are presented in Table S6 of the Supplement.

Remark 7.

To illustrate the benefit of stratified sampling in both the supervised and SS settings, we provide numerical studies of the optimal allocation in S5 of the Supplement. Mimicking our example in Section 8, we let the risk of y significantly differ across the S = 2 sampling groups. The stratification variable is picked so that $\text{P}(y=1\mid \mathcal{S}=1)$ is much lower than $\text{P}(y=1\mid \mathcal{S}=2)$. We consider two sampling strategies: (i) uniform random sampling of n subjects, and (ii) stratified sampling of n/2 subjects from each stratum. Since $\text{P}(y=1\mid \mathcal{S}=1)$ is low and close to 0, the variability of this stratum ${\sigma }_1^2=E\left\{f^2(\mathbf{F})\mid \mathcal{S}=1\right\}$ is smaller than that of $\mathcal{S}$ = 2 with $\text{P}(y=1\mid \mathcal{S}=2)$ not near 0 and 1. Connecting this with (10), the stratified sampling strategy oversamples within $\mathcal{S}=2$ so that its allocation of n₂ is more close to the optimal choice. Consistent with this observation, our simulation results indicate that stratified sampling is more efficient than uniform random sampling in both the supervised and SS settings with an average relative efficiency > 1.45 across different setups. We further inspect the supervised estimator of $D(\overline{\mathit{\theta }})$ under setup (I) in Section S5, for which the optimal allocation is n₁ = 0.47n and n₂ = 0.53n and nearly coincides with our equal allocation of n across the two stratum.

8. Example: EHR Study of Diabetic Neuropathy

We applied the proposed SSL procedures to develop and evaluate an EHR phenotyping algorithm for classifying diabetic neuropathy (DN), a common and serious complication of diabetes resulting in nerve damage. The full study cohort consists of N = 16,826 patients over age 18 with one or more of 12 ICD9 codes relating to DN identified from Partners HealthCare EHR. An initial assessment of 100 charts by physicians revealed the prevalence of DN in the study cohort was approximately 17%. To obtain a labeled set with sufficient DN cases for model training and improve efficiency, the investigators decided to employ a stratified sampling scheme. To do so, a binary filter variable $\mathcal{S}$ indicating whether a patient had a neurological exam and a neurology note with at least 1,000 characters was created. The prevalence of DN in the “enriched” stratum with $\mathcal{S}=1$ was expected to be higher than that in the stratum with $\mathcal{S}=0$. As demonstrated in our theoretical analysis in Section 5.4 and our numerical studies in Section S5, oversampling within the enriched set can improve estimation efficiency relative to taking a uniform random sample and is a common approach taken in EHR-based analyses. For this study, the investigators sampled n₀ = 70 and n₁ = 538 patients from the N₀ = 13608 patients with $\mathcal{S}=0$ and the N₁ = 3218 patients with $\mathcal{S}=1$, respectively, for developing the phenotyping algorithm.

To train the model for classifying DN, a set of 11 codified and NLP features related to DN were selected from an original list of 75 via an unsupervised screening as described in Yu et al. [2015]. The codified features included $\mathcal{S}$, diagnostic codes for diabetes, type 2 diabetes mellitus, diabetic neuropathy, other idiopathic peripheral autonomic neuropathy, and diabetes mellitus with neurological manifestation as well as normal glucose lab values and prescriptions for anti-diabetic medications. The NLP features included mentions of terms related to DN in the patient record including glycosylated hemoglobin (HgA1c), diabetic, and neuropathy. As all these features (with the exception of $\mathcal{S}$) are count variables and tend to be highly skewed, we used the transformation $x\to \log (x+1)$ to stabilize model fitting.

We developed DN classification models by fitting a logistic regression with the above features based on ${\widehat{\mathit{\theta }}}_{\text{SL}},{\widehat{\mathit{\theta }}}_{\text{SSL}}$ and ${\widehat{\mathit{\theta }}}_{\text{DR}}$ obtained from density ratio weighted estimation. Since the proportion of observations with $\mathcal{S}=0$ is relatively low in the labeled data, we implemented 100 replications of 6-fold CV procedure by splitting the data randomly within each strata to improve the stability of the CV procedure. To construct the basis for ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ and ${\widehat{\mathit{\theta }}}_{\text{DR}}$, we used a natural spline with 3 knots on all covariates except $\mathcal{S}$. To improve training stability, we set the ridge tuning parameter λ_n = n⁻¹ when fitting the imputation model.

As shown in Table 2(a), the point estimates are reasonably similar which confirms the consistency and stability of SS estimator in a real data setting. As expected, we find that the two most influential predictors are the diagnostic code for diabetic neuropathy and anti-diabetic medications. Importantly, we note substantial efficiency gains of ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ compared to ${\widehat{\mathit{\theta }}}_{\text{SL}}$. The SSL estimates are > 50% more efficient than the SL estimates for several features including six diagnostic code features and one NLP feature for DN. Additionally, ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ is the most efficient estimator among all three approaches for nearly all variables.

Table 2:

Results from the diabetic neuropathy EHR study: (a) estimates (Est.) of the regression parameters for both codified (COD) and NLP features based on ${\widehat{\mathit{\theta }}}_{\text{SL}}$, ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ and ${\widehat{\mathit{\theta }}}_{\text{DR}}$ along with their estimated SEs and the coordinate-wise relative efficiencies (RE) of ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ compared to ${\widehat{\mathit{\theta }}}_{\text{SL}}$ and (b) ${\widehat{D}}_{\text{SL}}^{\omega }$, ${\widehat{D}}_{\text{SSL}}^{\omega }$ and ${\widehat{D}}_{\text{DR}}^{\omega }$ along with their estimated SEs and relative efficiencies (RE) of ${\widehat{D}}_{\text{SSL}}^{\omega }$ and ${\widehat{D}}_{\text{DR}}^{\omega }$ compared to ${\widehat{D}}_{\text{SL}}^{\omega }$.

(a) Estimates of the Regression Coefficients

		${\widehat{\mathit{\theta }}}_{\text{SL}}$		${\widehat{\mathit{\theta }}}_{\text{SSL}}$			${\widehat{\theta }}_{\text{DR}}$
		Est.	SE	Est.	SE	RE	Est.	SE	RE
	Intercept	−3.89	0.80	−3.85	0.73	1.21	−3.26	0.66	1.45
COD	Neurological exam & note	−1.09	0.81	−0.90	0.61	1.77	−2.38	1.16	0.48
	Diabetes	−0.24	0.57	0.02	0.43	1.79	−0.09	0.48	1.42
	Type 2 Diabetes Mellitus	−1.05	0.73	−0.52	0.60	1.62	−0.62	0.86	0.73
	Diabetic Neuropathy	1.99	0.86	1.79	0.68	1.58	1.66	1.08	0.63
	Anti-diabetic Meds	1.70	0.59	1.12	0.44	1.79	1.50	0.51	1.32
	Diabetes Mellitus with
	Neuro Manifestation	0.36	1.17	0.60	0.98	1.42	0.59	0.90	1.68
	Other Idiopathic Peripheral
	Autonomic Neuropathy	0.86	0.71	0.93	0.68	1.09	1.01	0.76	0.89
	Normal Glucose	−0.56	0.65	−0.20	0.51	1.64	−1.27	0.80	0.66
NLP	Diabetic	0.30	0.58	−0.57	0.49	1.39	0.14	0.65	0.80
	HgA1c	−0.52	0.75	−0.64	0.70	1.16	−0.67	0.85	0.79
	Neuropathy	0.27	0.58	0.30	0.47	1.55	0.37	0.54	1.15

(b) Estimates of the Accuracy Parameters (×100)

	${\widehat{D}}_{\text{SL}}^{\omega }$		${\widehat{D}}_{\text{SSL}}^{\omega }$			${\widehat{D}}_{\text{DR}}^{\omega }$
	Est.	SE	Est.	SE	RE	Est.	SE	RE
Brier score	8.97	2.09	9.59	1.68	1.55	9.60	1.87	1.26
OMR	12.87	3.25	14.01	2.55	1.63	12.29	3.04	1.14

In Table 2(b), we compare ${\widehat{D}}_{\text{SL}}^w,{\widehat{D}}_{\text{SSL}}^w$ and ${\widehat{D}}_{\text{DR}}^w$ for the Brier score and OMR with c = 0.5. While the point estimates for the accuracy measures based on these different approaches are relatively similar, ${\widehat{D}}_{\text{SSL}}^w$ is 55% more efficient than ${\widehat{D}}_{\text{SL}}^w$ for the Brier score and 63% more efficient for the OMR. Again, ${\widehat{D}}_{\text{SSL}}^w$ is substantially more efficient than the DR estimator ${\widehat{D}}_{\text{DR}}^w$. These results support the potential value of our method for EHR-based research as these gains in efficiency may be directly translated into requiring fewer labeled examples for model evaluation.

9. Discussion

In this paper, we focused on the evaluation of a classification rule derived from a working regression model under stratified sampling in the SS setting. In particular, we introduced a two-step imputation-based method for estimation of the Brier score and OMR that makes use of unlabeled data. Additionally, as a by-product of our procedure, we obtained an efficient SS estimator of the regression parameter. Through theoretical and numerical studies, we demonstrated the advantage of the SS estimator over the SL estimator with respect to efficiency. We also developed a weighted CV procedure to adjust for overfitting and a resampling procedure for making inference. Our numerical studies indicate that our proposed method outperforms the existing DR method for SSL in the finite sample studies and we provide further discussion of this finding in Section S6 of the Supplement. Importantly, this article is one of the first theoretical studies of labeling based on stratified sampling within the SSL literature. We focus on the stratified sampling scheme due to its direct application to a variety of EHR-based analyses, including the development of a phenotyping algorithm for diabetic neuropathy presented in the previous section.

In our numerical studies, we used spline functions with 3 or 4 knots and interaction terms for the imputation model. It would be possible to use more knots or add more features to the basis function for settings with a larger n. However, care must be taken to avoid overfitting and potential loss in the efficiency gain of the SS estimator in finite sample. Alternatively, other basis functions can be utilized provided that Φ(u) contains x in its components to ensure consistency of the regression parameter. In settings where nonlinear effects of x on y are present, it may be desirable to impose a more complex outcome model to improve the prediction performance. A potential approach is to explicitly include nonlinear basis functions in the outcome model. In Section S7 of the Supplementary Materials, we consider using the leading principal components (PCs) of x and Ψ(x) where Ψ(·) is a vector of nonlinear transformation functions under a variety of settings. This approach performs similarly or better than the commonly used random forest model with respect to predictive accuracy, suggesting the utility of our proposed methods in the presence of nonlinear effects. Our numerical results also illustrate the efficiency gain of the SS estimators of the Brier score and OMR relative to the SL and DR methods. It is important to note, however, that the dimensions of both x and Φ were assumed to be fixed in our asymptotic analysis. Accommodating more complex modeling with p not small relative to n requires extending the proposed SSL approach to settings where x and Φ are high dimensional.

For accuracy estimation, we proposed an ensemble CV estimator that eliminates first-order bias when the estimated regression parameter is the minimizer of the empirical performance measure. Though this condition may not hold when the outcome model is misspecified, we have found that the suggested weights perform well in our numerical studies. Such ensemble methods that accommodate the more general case in both the supervised and SS settings warrant further research. Additionally, an important setting where our proposed SSL procedure would be of great use is in drawing inferences about two competing regression models. As it is likely that at least one model is misspecified, we would expect to observe efficiency gains in estimating the difference in prediction error with the proposed method.

Lastly, while the present work focuses on the binary outcome along with the Brier score and OMR, the proposed SSL framework can potentially be extended to more general settings with continuous y and/or other accuracy parameters. In particular, for binary y and corresponding classification rule ${\mathcal{Y}}_2=I\left\{g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)>c\right\}$, it would be of interest to consider estimation of the sensitivity, specificity, and weighted OMR with different threshold values to analyze the costs associated with the false positive and false negative errors.

Appendix

Here we provide justifications for our main theoretical results. The following lemma confirming the existence and uniqueness of the limiting parameters $\overline{\mathit{\theta }},\overline{\gamma }$, and ${\overline{\mathit{\nu }}}_{\theta }$ will be used in our subsequent derivations.

Lemma A1.

Under conditions 1–3, unique $\overline{\mathit{\theta }}$ and $\overline{\gamma }$ exist. In addition, there exists δ > 0 such that a unique ${\overline{\mathit{\nu }}}_{\mathit{\theta }}$ exists for any θ satisfying $\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<\delta$.

Proof.

Conditions 3 (A) and (B) imply that there is no θ and γ such that with non-trivial probability,

$$I\left(Y_1>Y_2\right)=I\left({\mathit{\theta }}^{\top }{\mathbf{x}}_1>{\mathit{\theta }}^{\top }{\mathbf{x}}_2\right),$$

and

$$I\left(Y_1>Y_2\right)=I\left({\gamma }^{\top }{\mathbf{\text{Φ}}}_1>{\gamma }^{\top }{\mathbf{\text{Φ}}}_2\right).$$

It follows directly from Appendix I of Tian et al. [2007] that there exist finite $\overline{\mathit{\theta }}$ and $\overline{\gamma }$ that solve $U(\overline{\mathit{\theta }})$ and $Q(\overline{\gamma })$, respectively, and that $\overline{\mathit{\theta }}$ and $\overline{\gamma }$ are unique. For $\mathit{\theta }\in \mathbf{\text{Θ}}$, there exists no ν such that

$$I\left(Y_1>Y_2\right)=I\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_1+{\mathit{\nu }}^{\top }{\mathbf{z}}_{1\theta }>{\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_2+{\mathit{\nu }}^{\top }{\mathbf{z}}_{2\theta }\right),$$

or

$$I\left(Y_1>Y_2\right)=I\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_1+{\mathit{\nu }}^{\top }{\mathbf{z}}_{1\mathit{\theta }}<{\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_2+{\mathit{\nu }}^{\top }{\mathbf{z}}_{2\mathit{\theta }}\right),$$

which similarly implies there exists a finite ${\overline{\mathit{\nu }}}_{\mathit{\theta }}$ that is the solution to $\mathbf{R}(\mathit{\nu }\mid \mathit{\theta })=0$. The solution is also unique as $E\left[{\mathbf{z}}_{\theta }^{\otimes 2}\dot{g}\left\{{\overline{\gamma }}^{\top }\mathbf{\text{Φ}}+{\mathit{\nu }}^{\top }{\mathbf{z}}_{\theta }\right\}\right]\succ \mathbf{0}$ for any $\mathit{\nu }$. □

A. Estimation Procedure for ${\widehat{\mathit{\theta }}}_{\text{SSL}}$

We propose to obtain a simple SS estimator for $\overline{\mathit{\theta }},{\check{\mathit{\theta }}}_{\text{SSL}}$, as the solution to

$${\widehat{\mathbf{\text{U}}}}_N(\mathit{\theta })=\frac{1}{N}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\tilde{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}=\mathbf{0}.$$ (A.1)

Note that when u includes the stratum information $\mathcal{S}$ and the imputation model (2) is correctly specified, there is no need to use the inverse probability weighted (IPW) estimating equation with ${\widehat{w}}_i$ as in (3). However, under the general scenario where the imputation model may be misspecified, the unweighted estimating equation is not guaranteed to provide an asymptotically unbiased estimate of $\overline{\gamma }$ and necessitates the use of the IPW approach.

As detailed in the subsequent sections, when the imputation and outcome models are misspecified, the efficiency gain of ${\check{\mathit{\theta }}}_{\text{SSL}}$ relative to ${\widehat{\mathit{\theta }}}_{\text{SL}}$, is not theoretically guaranteed. We therefore obtain the final SS estimator, denoted as ${\widehat{\mathit{\theta }}}_{\text{SSL}}={\left({\widehat{\theta }}_{\text{SSL},0},\dots ,{\widehat{\theta }}_{\text{SSL},p}\right)}^{\top }$, as a linear combination of ${\widehat{\mathit{\theta }}}_{\text{SL}}$ and ${\check{\mathit{\theta }}}_{\text{SSL}}$ to minimize the asymptotic variance. For simplicity we consider here the component-wise optimal combination of the two estimators. That is, the jth component of $\overline{\mathit{\theta }},{\overline{\mathit{\theta }}}_j$, is estimated with

$${\widehat{\theta }}_{\text{SSL},j}={\widehat{W}}_{1j}{\tilde{\theta }}_{\text{SSL},j}+\left(1-{\widehat{W}}_{1j}\right){\tilde{\theta }}_{\text{SL},j}$$

where ${\widehat{W}}_{1j}$ is the first component of the vector ${\widehat{\mathbf{W}}}_j={\mathbf{1}}^{\top }{\widehat{\mathbf{\Sigma }}}_j^{-1}/\left({\mathbf{1}}^{\top }{\widehat{\mathbf{\Sigma }}}_j^{-1}\mathbf{1}\right)$ and ${\widehat{\mathbf{\text{Σ}}}}_j$ is a consistent estimator for $\text{cov}\left\{{\left({\check{\theta }}_{\text{SSL},j},{\tilde{\theta }}_{\text{SL},j}\right)}^{\top }\right\}$. To estimate the variance of ${\widehat{\mathit{\theta }}}_{\text{SSL}}$, one may rely on estimates of the influence functions of ${\check{\mathit{\theta }}}_{\text{SSL}}$ and ${\widehat{\mathit{\theta }}}_{\text{SL}}$. To avoid under-estimation in a finite sample, we obtain bias-corrected estimates of the influence functions via K-fold CV. Details on the K-fold CV procedure, as well as the computations for the aforementioned estimation of ${\widehat{\mathbf{\text{Σ}}}}_j$ and ${\widehat{W}}_{1j}$, are given in Appendix B.

B. Cross-validation Based Inference for ${\widehat{\mathit{\theta }}}_{\text{SSL}}$

Here we provide the details of the procedure to obtain ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ as well as an estimate of its variance. We employ K-fold CV in the proposed procedure to adjust for overfitting in finite sample and denote each fold of $\mathcal{L}$ as ${\mathcal{L}}_k$ for $k=1,\dots ,K$. First, we estimate ${\widehat{\mathbf{\text{Σ}}}}_j$ with

$$n^{-1}\sum _{k=1}^K\sum _{i\in {\mathcal{L}}_k}\left\{{\mathcal{W}}_j\left({\tilde{\gamma }}_{(-k)},{\mathbf{D}}_i\right),{\mathcal{V}}_j\left({\widehat{\mathit{\theta }}}_{(-k)},{\mathbf{D}}_i\right)\right\}{\left\{{\mathcal{W}}_j\left({\tilde{\gamma }}_{(-k)},{\mathbf{D}}_i\right),{\mathcal{V}}_j\left({\widehat{\mathit{\theta }}}_{(-k)},{\mathbf{D}}_i\right)\right\}}^{\top }$$

where ${\tilde{\gamma }}_{(-k)}$ is the estimator of $\overline{\gamma }$ based on $\mathcal{L}/{\mathcal{L}}_k,{\widehat{\mathit{\theta }}}_{(-k)}$ is the supervised estimator of $\overline{\mathit{\theta }}$ based on $\mathcal{L}/{\mathcal{L}}_k$,

$$\begin{gathered} \mathcal{W}\left({\tilde{\gamma }}_{(-k)},{\mathbf{D}}_i\right)={\widehat{\mathbf{A}}}^{-1}\left[{\varpi }_i{\mathbf{x}}_i\left\{y_i-g\left({\tilde{\gamma }}_{(-k)}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}\right],\mathcal{V}\left({\widehat{\theta }}_{(-k)},{\mathbf{D}}_i\right)={\widehat{\mathbf{A}}}^{-1}\left[{\varpi }_i{\mathbf{x}}_i\left\{y_i-g\left({\widehat{\theta }}_{(-k)}^{\top }{\mathbf{x}}_i\right)\right\}\right], \\ \widehat{\mathbf{A}}=N^{-1}\sum _{i=1}^N{\mathbf{x}}_i^{\otimes 2}\dot{g}\left({\check{\theta }}_{\text{SSL}}^{\top }{\mathbf{x}}_i\right)\text{and}{\varpi }_i={\widehat{w}}_{i}n/N \end{gathered}$$

In practice, ${\widehat{\mathbf{\text{Σ}}}}_j$ may be unstable due to the high correlation of ${\check{\mathit{\theta }}}_{\text{SSL}}$ and ${\widehat{\mathit{\theta }}}_{\text{SL}}$. One may use a regularized estimator ${\left({\widehat{\mathbf{\Sigma }}}_j+{\delta }_n\mathbf{I}\right)}^{-1}$ with some ${\delta }_n=O\left(n^{-\frac{1}{2}}\right)$ to stabilize the estimation and obtain ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ accordingly. The covariance of ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ may then be consistently estimated with

$$n^{-1}\sum _{k=1}^K\sum _{i\in {\mathcal{L}}_k}\left\{\mathcal{Z}\left({\tilde{\gamma }}_{(-k)},{\widehat{\mathit{\theta }}}_{(-k)},{\mathbf{D}}_i\right)\right\}{\left\{\mathcal{Z}\left({\tilde{\gamma }}_{(-k)},{\widehat{\mathit{\theta }}}_{(-k)},{\mathbf{D}}_i\right)\right\}}^{\top }\text{where}$$ (B.1)

$$\mathcal{Z}\left({\tilde{\gamma }}_{(-k)},{\widehat{\mathit{\theta }}}_{(-k)},{\mathbf{D}}_i\right)=\widehat{\mathbf{W}}\left\{\mathcal{W}\left({\tilde{\gamma }}_{(-k)},{\mathbf{D}}_i\right)\right\}+(\mathbf{I}-\widehat{\mathbf{W}})\left\{\mathcal{V}\left({\widehat{\mathit{\theta }}}_{(-k)},{\mathbf{D}}_i\right)\right\}$$ (B.2)

$\widehat{\mathbf{W}}=\text{diag}\left({\widehat{W}}_{10},\dots ,{\widehat{W}}_{1p}\right)$ is an estimate of $\mathbf{W}=\text{diag}\left(W_{10},\dots ,W_{1p}\right),{\widehat{W}}_{1j}$ is the first component of ${\widehat{\mathbf{W}}}_j={\mathbf{1}}^{\top }{\widehat{\mathbf{\Sigma }}}_j^{-1}/\left({\mathbf{1}}^{\top }{\widehat{\mathbf{\Sigma }}}_j^{-1}\mathbf{1}\right)$ and $W_{1j}$ is the first component of ${\mathbf{W}}_j={\mathbf{1}}^{\top }{\mathbf{\text{Σ}}}_j^{-1}/\left({\mathbf{1}}^{\top }{\mathbf{\text{Σ}}}_j^{-1}\mathbf{1}\right)$. The confidence intervals for the regression parameters can be constructed with the proposed variance estimates and the asymptotic normal distribution of the SS estimator.

C. Asymptotic Properties of ${\widehat{\mathit{\theta }}}_{\text{SL}}$

The main complication in deriving the asymptotic properties of the SL estimators arises from the fact that $P\left(V_i=1\mid \mathcal{F}\right)={\widehat{\pi }}_{{\mathcal{S}}_i}\to 0$ as $n\to \infty$ and hence ${\widehat{w}}_i=V_i/{\widehat{\pi }}_{{\mathcal{S}}_i}$ is an ill behaved random variable tending to infinity in the limit for those with V_i = 1. This substantially distinguishes the SS setting from the standard missing data literature. To overcome this complication, we note that for subjects in the labeled set, $N^{-1}{\widehat{w}}_i={\sum }_{s=1}^{S}I\left({\mathcal{S}}_i=s\right){\widehat{\rho }}_s{n}_s^{-1}{V}_i$ and $V_i\mid {\mathcal{S}}_i=s$, are independent identically distributed (i.i.d) random variables since the labeled observations are drawn randomly within each stratum. Also note that ${\widehat{\rho }}_s\stackrel{p}{\to }{\rho }_s$ as assumed in Section 2.1. Hence for any function f with $\mathrm{var}\left\{f\left({\mathbf{F}}_i\right)\mid S_i=s\right\}<\infty$,

$$\begin{gathered} N^{-1}\sum _{i=1}^N{\widehat{w}}_{i}f\left({\mathbf{F}}_i\right)=\sum _{s=1}^S{\widehat{\rho }}_s{n}_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right)f\left({\mathbf{F}}_i\right) \\ =\sum _{s=1}^S\left\{{\rho }_s+o_p(1)\right\}\left\{n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right)f\left({\mathbf{F}}_i\right)\right\}+o_p(1) \end{gathered}$$ (C.1)

$$=\sum _{s=1}^S{\rho }_s\text{E}\left\{f\left({\mathbf{F}}_i\right)\mid {\mathcal{S}}_i=s\right\}+o_p(1)=\text{E}\left\{f\left({\mathbf{F}}_i\right)\right\}+o_p(1).$$ (C.2)

We begin by verifying that ${\widehat{\mathit{\theta }}}_{\text{SL}}$ is consistent for $\overline{\mathit{\theta }}$. It suffices to show that (i) ${\sup }_{\mathit{\theta }\in \mathbf{\Theta }}\parallel {\mathbf{\text{U}}}_n(\mathit{\theta })-\mathbf{U}(\mathit{\theta }){\parallel }_2=o_p(1)$ and (ii) ${\inf }_{\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2>ϵ}{\Vert \mathbf{\text{U}}(\mathit{\theta })\Vert }_2>0$ for any $ϵ>0$ [Newey and McFadden, 1994, Lemma 2.8]. To this end, we write

$${\mathbf{U}}_n(\mathit{\theta })=\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{x}}_i\left\{y_i-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}=\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{V_i=1}I\left({\mathcal{S}}_i=s\right){\mathbf{x}}_i\left\{y_i-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\right]+o_p(1).$$

Under Conditions 1–3, x belongs to a compact set and $\dot{g}\left({\mathit{\theta }}^{\top }\mathbf{x}\right)$ is continuous and uniformly bounded for $\mathit{\theta }\in \mathbf{\text{Θ}}$. From the uniform law of large numbers (ULLN) [Pollard, 1990, Theorem 8.2], $n_s^{-1}{\sum }_{V_i=1}I\left({\mathcal{S}}_i=s\right){\mathbf{x}}_i\left\{y_i-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}$ converges to $E\left[{\mathbf{x}}_i\left\{y_i-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\mid {\mathcal{S}}_i=s\right]$ in probability uniformly as $n\to \infty$ and ${\sup }_{\mathit{\theta }\in \mathbf{\Theta }}{\Vert {\mathbf{U}}_n(\mathit{\theta })-\mathbf{U}(\mathit{\theta })\Vert }_2=o_p(1)$. Furthermore, (ii) follows directly from Lemma A1 and consequently ${\widehat{\mathit{\theta }}}_{\text{SL}}\stackrel{p}{\to }\overline{\mathit{\theta }}$ as $n\to \infty$.

Next we consider the asymptotic normality of ${\widehat{\mathcal{W}}}_{\text{SL}}=n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}-\overline{\mathit{\theta }}\right)$. Noting that ${\widehat{\mathit{\theta }}}_{\text{SL}}\stackrel{p}{\to }\overline{\mathit{\theta }}$ and ${\widehat{\rho }}_s\stackrel{p}{\to }{\rho }_s$, we apply Theorem 5.21 of Van der Vaart [2000] to obtain the Taylor expansion

$$\begin{gathered} {\widehat{\mathcal{W}}}_{\text{SL}}=n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}-\overline{\mathit{\theta }}\right)=n^{\frac{1}{2}}\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}+o_p(1) \\ =n^{\frac{1}{2}}\sum _{s=1}^S\left\{{\rho }_s+o_p(1)\right\}\left\{n_s^{-1}\sum _{V_i=1}I\left({\mathcal{S}}_i=s\right){\mathbf{e}}_{\text{LL}i}\right\}+o_p(1), \end{gathered}$$

where ${\mathbf{e}}_{\text{SL}i}={\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}$ and $\mathbf{A}=\text{E}\left\{{\mathbf{x}}_i^{\otimes 2}\dot{g}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}$. It then follows by the classical Central Limit Theorem that ${\widehat{\mathcal{W}}}_{\text{SL}}\to N\left(0,{\mathbf{\text{Σ}}}_{\text{SL}}\right)$ in distribution and

$${\widehat{\mathcal{W}}}_{\text{SL}}=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left\{n_s^{-1}\sum _{V_i=1}I\left({\mathcal{S}}_i=s\right){\mathbf{e}}_{\text{SL}i}\right\}+o_p(1),$$

where ${\mathbf{\text{Σ}}}_{\text{SL}}={\sum }_{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}E\left\{{\mathbf{e}}_{\text{SL}i}^{\otimes 2}\mid {\mathcal{S}}_i=s\right\}$.

D. Asymptotic Properties of ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)$

We begin by showing that ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)\stackrel{p}{\to }D(\overline{\mathit{\theta }})$ as $n\to \infty$. We first note that since ${\widehat{\rho }}_s\stackrel{p}{\to }{\rho }_s$,

$${\widehat{D}}_{\text{SL}}(\mathit{\theta })=\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{V_i=1}I\left({\mathcal{S}}_i=s\right)d\left\{y_i,\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\right]+o_p(1).$$ (D.1)

It follows by the ULLN that ${\sup }_{\mathit{\theta }\in \mathbf{\Theta }}\left|{\widehat{D}}_{\text{SL}}(\theta )-D(\mathit{\theta })\right|=o_p(1)$ since $d\{y,\mathcal{Y}\text{(}{\mathit{\theta }}^{\top }\mathbf{x}\text{}}$ is continuously differentiable in θ and uniformly bounded. The consistency of ${\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)$ for $D(\overline{\mathit{\theta }})$ then follows from the fact that ${\widehat{\mathit{\theta }}}_{\text{SL}}$ converges in probability to $\overline{\mathit{\theta }}$ as $n\to \infty$.

To establish the asymptotic distribution of ${\widehat{\mathcal{T}}}_{\text{SL}}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)-D(\overline{\mathit{\theta }})\right\}$, we first consider ${\tilde{\mathcal{T}}}_{\text{SL}}(\mathit{\theta })=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SL}}(\mathit{\theta })-D(\mathit{\theta })\right\}$. We verify that there exists δ > 0, such that the classes of functions indexed by θ:

$${\mathcal{B}}_1=\left\{I(\mathcal{S}=s)\left|y-I\left\{g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)>c\right\}\right|:\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<\delta \right\}$$

$$\text{and}{\mathcal{B}}_2=\left\{I(\mathcal{S}=s){\left[y-g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)\right]}^2:\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<\delta \right\}$$

are Donsker classes. For ${\mathcal{B}}_1$, we note that $\left\{I\left\{g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)>c\right\}:\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<\delta \right\}$ is a Vapnik-Chervonenkis class [Van der Vaart, 2000, Page 275] and thus

$${\mathcal{B}}_1=\left\{I\left(s=s_0\right)\left[I(y=0)I\left\{g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)>c\right\}+I(y=1)I\left\{g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)⩽c\right\}\right]:\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<\delta \right\}$$

is a Donsker class. For ${\mathcal{B}}_2,{\left[y-g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)\right]}^2$ is continuously differentiable in θ and uniformly bounded by a constant. It follows that ${\mathcal{B}}_2$ is a Donsker class [Van der Vaart, 2000, Example 19.7]. By Theorem 19.5 of Van der Vaart [2000] we then have

$$n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SL}}(\mathit{\theta })-D(\mathit{\theta })\right\}=\sum _{s=1}^S{\rho }_s{\rho }_{1s}^{-\frac{1}{2}}{n}_s^{-\frac{1}{2}}\sum _{V_i=1}I\left({\mathcal{S}}_i=s\right)\left[d\left\{y_i,\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}-D(\mathit{\theta })\right]+o_p(1),$$

which converges weakly to a mean zero Gaussian process index by θ. Thus, $n^{\frac{1}{2}}\{{\widehat{D}}_{\text{SL}}(\mathit{\theta })-D(\mathit{\theta })\}$ is stochastically equicontinuous at $\overline{\mathit{\theta }}$. In addition, note that D(θ) is continuously differentiable at $\overline{\mathit{\theta }}$ and ${\widehat{\rho }}_s\stackrel{p}{\to }{\rho }_s$. It then follows that

$$\begin{gathered} {\widehat{\mathcal{T}}}_{\text{SL}}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SL}}\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)-D\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)\right\}+n^{\frac{1}{2}}\left\{D\left({\widehat{\mathit{\theta }}}_{\text{SL}}\right)-D(\overline{\mathit{\theta }})\right\} \\ =n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left(n_s^{-1}\sum _{V_i=1}I\left({\mathcal{S}}_i=s\right)\left[d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-D(\overline{\mathit{\theta }})+\dot{D}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{e}}_{\text{SL}i}\right]\right)+o_p(1), \end{gathered}$$

which converges in distribution to $N\left(0,{\sigma }_{\text{SL}}^2\right)$ where

$${\sigma }_{\text{SL}}^2=\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}\text{E}\left[{\left\{d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-D(\overline{\mathit{\theta }})+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{e}}_{\text{SL}i}\right\}}^2\mid {\mathcal{S}}_i=s\right].$$

E. Asymptotic Properties of ${\check{\mathit{\theta }}}_{\text{SSL}}$

We first consider the asymptotic properties of $\tilde{\gamma }$. Under Conditions 1–3 and using that ${\lambda }_n=o\left(n^{-\frac{1}{2}}\right)$ and ${\widehat{\rho }}_s\stackrel{p}{\to }{\rho }_s$, we can adapt the same procedure as in Appendix C to show that $\tilde{\gamma }\stackrel{p}{\to }\overline{\gamma }$ as $n\to \infty$. We obtain the following Taylor series expansion

$$n^{\frac{1}{2}}(\tilde{\gamma }-\overline{\gamma })=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right){\mathbf{C}}^{-1}{\mathbf{\text{Φ}}}_i\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}\right]+o_p(1)$$

where $\mathbf{\text{C}}=\text{E}\left\{{\mathbf{\text{Φ}}}_i^{\otimes 2}\dot{g}\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}$. We then have that $n^{\frac{1}{2}}(\tilde{\gamma }-\overline{\gamma })$ converges to zero-mean Gaussian distribution. To verify that ${\check{\mathit{\theta }}}_{\text{SSL}}$ is consistent for $\overline{\mathit{\theta }}$, it suffices to show that (i) ${\sup }_{\mathit{\theta }\in \mathbf{\Theta }}\parallel {\widehat{\mathbf{U}}}_N(\mathit{\theta })-{\mathbf{U}}^0(\mathit{\theta }){\parallel }_2=o_p(1)$ and (ii) ${\inf }_{\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2⩾ϵ\mid }\parallel \mathbf{U}(\mathit{\theta }){\parallel }_2>0$, for any ϵ > 0 [Newey and McFadden, 1994, Lemma 2.8], where

$${\mathbf{U}}^0(\mathit{\theta })=\text{E}\left[{\mathbf{x}}_i\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\right].$$

For (i), we first note that since $\tilde{\gamma }\stackrel{p}{\to }\overline{\gamma },\dot{g}(\cdot )$ is continuous and Φ is bounded,

$${\widehat{\mathbf{\text{U}}}}_N(\mathit{\theta })=N^{-1}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\tilde{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}=N^{-1}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}+o_p(1).$$ (E.1)

Note that $g\left({\overline{\gamma }}^{\top }\mathbf{\Phi }\right)-g\left({\mathit{\theta }}^{\top }\mathbf{x}\right)$ is bounded and continuously differentiable in θ. We then apply the ULLN to have that ${\sup }_{\mathit{\theta }\in \mathbf{\Theta }}{\Vert {\widehat{\mathbf{U}}}_N(\mathit{\theta })-{\mathbf{U}}^0(\mathit{\theta })\Vert }_2=o_p(1)$. For (ii), we note

$${\mathbf{U}}^0(\mathit{\theta })=-\text{E}\left[{\mathbf{x}}_i\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)\right\}\right]+\text{E}\left[{\mathbf{x}}_i\left\{y_i-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\right]=\text{E}\left[{\mathbf{x}}_i\left\{y_i-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\right].$$

Therefore, (ii) holds by Lemma A1 and ${\check{\mathit{\theta }}}_{\text{SSL}}$ is consistent for $\overline{\mathit{\theta }}$.

Now we consider the weak convergence of $n^{\frac{1}{2}}\left({\check{\mathit{\theta }}}_{\text{SSL}}-\overline{\mathit{\theta }}\right)$. Under Conditions 1–3, we have the Taylor expansion

$$n^{\frac{1}{2}}\left({\check{\mathit{\theta }}}_{\text{SSL}}-\overline{\mathit{\theta }}\right)=n^{\frac{1}{2}}{\mathbf{A}}^{-1}\left[N^{-1}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}+\mathbf{B}(\tilde{\gamma }-\overline{\gamma })\right]+o_p(1),$$

where $\mathbf{B}=\text{E}\left\{{\mathbf{x}}_i{\mathbf{\text{Φ}}}_i^{\top }\dot{g}\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}$. This expansion coupled with the fact that $N^{-1}{\sum }_{i=1}^N{\mathbf{X}}_i\{g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\}=O_p\left(N^{-\frac{1}{2}}\right)=o_p\left(n^{-\frac{1}{2}}\right)$ imply that

$$n^{\frac{1}{2}}\left({\check{\mathit{\theta }}}_{\text{SSL}}-\overline{\mathit{\theta }}\right)=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^n{\mathbf{A}}^{-1}{\mathbf{\text{BC}}}^{-1}I\left({\mathcal{S}}_i=s\right){\mathbf{\text{Φ}}}_i\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}\right]+o_p(1).$$

Letting ${\mathbf{x}}_i={\left(x_{i1},\dots ,x_{ip}\right)}^{\top }$, we note that for $j=1,\dots ,p$,

$${\left[{\mathbf{\text{BC}}}^{-1}\right]}_j=\text{E}\left\{x_{ij}{\mathbf{\text{Φ}}}_i^{\top }\dot{g}\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}\text{E}{\left[\left\{{\mathbf{\text{Φ}}}_i^{\otimes 2}\dot{g}\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}\right]}^{-1}=\underset{\beta }{\text{argmin}}\text{E}\left\{\dot{g}\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right){\left(x_{ij}-{\beta }^{\top }{\mathbf{\text{Φ}}}_i\right)}^2\right\}.$$

Since x_ij is a component of the vector $\mathbf{\text{Φ}}(\mathbf{u})$, the minimizer β can be chosen such that $x_{ij}-{\beta }^{\top }{\mathbf{\text{Φ}}}_i=0$ for $i=1,\cdots ,N$ which implies that ${\mathbf{x}}_i=\mathbf{B}{\mathbf{C}}^{-1}{\mathbf{\text{Φ}}}_i$. Thus

$$n^{\frac{1}{2}}\left({\check{\mathit{\theta }}}_{\text{SSL}}-\overline{\mathit{\theta }}\right)=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left\{n_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right){\mathbf{e}}_{\text{SSL}i}\right\}+o_p(1)$$

where ${\mathbf{e}}_{\text{SSL}i}={\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}$. It then follows from the classical Central Limit Theorem that $n^{\frac{1}{2}}\left({\check{\mathit{\theta }}}_{\text{SSL}}-\overline{\mathit{\theta }}\right)\to N\left(0,{\mathbf{\Sigma }}_{\text{SSL}}\right)$ in distribution where

$${\mathbf{\text{Σ}}}_{\text{SSL}}=\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}E\left\{{\mathbf{e}}_{\text{SSL}i}^{\otimes 2}\mid {\mathcal{S}}_i=s\right\}.$$

We then see that

$$\begin{gathered} {\mathbf{\text{Σ}}}_{\text{SL}}-{\mathbf{\text{Σ}}}_{\text{SSL}}=\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}\left[\text{E}\left\{{\mathbf{e}}_{\text{SL}i}^{\otimes 2}\mid {\mathcal{S}}_i=s\right\}-\text{E}\left\{{\mathbf{e}}_{\text{SSL}i}^{\otimes 2}\mid {\mathcal{S}}_i=s\right\}\right]= \\ \sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}{\mathbf{A}}^{-1}\text{E}\left[{\mathbf{x}}_i^{\otimes 2}{\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}}^2+2{\mathbf{x}}_i^{\otimes 2}\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}\mid {\mathcal{S}}_i=s\right]. \end{gathered}$$

Therefore, when the imputation model is correctly specified, it follows that

$${\mathbf{\Sigma }}_{\text{SL}}-{\mathbf{\Sigma }}_{\text{SSL}}=\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}E\left[{\mathbf{A}}^{-1}{\mathbf{x}}_i^{\otimes 2}{\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}}^2\mid {\mathcal{S}}_i=s\right]⪰\mathbf{0}.$$

F. Asymptotic Properties of ${\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)$ and ${\widehat{D}}_{\text{sSL}}\left({\tilde{\mathit{\theta }}}_{\text{sSL}}\right)$

First note that by Lemma A1, there exists δ > 0 such that for all θ satisfying $\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<\delta$, so that ${\overline{\mathit{\nu }}}_{\mathit{\theta }}$ is unique. Then, similar to the derivations in Appendices C and E, we may show that ${\tilde{\mathit{\nu }}}_{\mathit{\theta }}$ is consistent for ${\overline{\mathit{\nu }}}_{\mathit{\theta }}$ and $n^{\frac{1}{2}}\left({\tilde{\mathit{\nu }}}_{\mathit{\theta }}-{\overline{\mathit{\nu }}}_{\mathit{\theta }}\right)$ is asymptotically Gaussian with mean zero.

Let ${\mathbf{\text{Θ}}}_{\delta }=\mathbf{\text{Θ}}\cap \left\{\mathit{\theta }:\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<\delta \right\}$. For the consistency of ${\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)$ for $D(\overline{\mathit{\theta }})$, we note that the uniform consistency of ${\tilde{\mathit{\nu }}}_{\mathit{\theta }}$ for ${\overline{\mathit{\nu }}}_{\mathit{\theta }}$ and $\tilde{\gamma }$ for $\overline{\gamma }$, together with the ULLN and under regularity Conditions 1–3 and ${\widehat{\rho }}_s\stackrel{p}{\to }{\rho }_s$, imply ${\sup }_{\mathit{\theta }\in {\mathbf{\Theta }}_{\delta }}\left|{\widehat{D}}_{\text{SSL}}(\mathit{\theta })-D(\mathit{\theta })\right|=o_p(1)$. It then follows from the consistency of ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ and ${\check{\mathit{\theta }}}_{\text{SSL}}$ for $\overline{\mathit{\theta }}$ that ${\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)\stackrel{p}{\to }D(\overline{\mathit{\theta }})$ and ${\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)\stackrel{p}{\to }D(\overline{\mathit{\theta }})$ as $n\to \infty$.

To derive the asymptotic distribution for ${\widehat{\mathcal{T}}}_{\text{SSL}}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)-D(\overline{\mathit{\theta }})\right\}$ and ${\breve{\mathcal{T}}}_{\text{SSL}}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)-D(\overline{\mathit{\theta }})\right\}$, we first consider

$${\tilde{\mathcal{T}}}_{\text{SSL}}(\mathit{\theta })=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}(\mathit{\theta })-D(\mathit{\theta })\right\}=n^{\frac{1}{2}}\left[N^{-1}\sum _{i=1}^{N}d\left\{g\left({\tilde{\gamma }}^{\top }{\mathbf{\Phi }}_i+{\tilde{\nu }}_{\mathit{\theta }}\right),\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}-D(\mathit{\theta })\right]$$

Under Conditions 1–3 and by Taylor series expansion about $\overline{\gamma }$ and $\overline{\mathit{\nu }}$ and the ULLN,

$${\tilde{\mathcal{T}}}_{\text{SSL}}(\mathit{\theta })=n^{\frac{1}{2}}\left[N^{-1}\sum _{i=1}^{N}d\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i+{\overline{\mathit{\nu }}}_{\mathit{\theta }}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right),\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}-D(\mathit{\theta })+{\mathbf{G}}_{\mathit{\theta }}(\tilde{\gamma }-\overline{\gamma })+{\mathbf{H}}_{\mathit{\theta }}\left({\tilde{\mathit{\nu }}}_{\mathit{\theta }}-{\overline{\mathit{\nu }}}_{\mathit{\theta }}\right)\right]$$

where ${\mathbf{G}}_{\mathit{\theta }}=\text{E}\left[{\text{Φ}}_i^{\top }\dot{g}\left({\overline{\gamma }}^{\top }{\text{Φ}}_i+{\overline{\mathit{\nu }}}_{\mathit{\theta }}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right)\left\{1-2\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\right]$ and ${\mathbf{\text{H}}}_{\theta }=\text{E}\left[{\mathbf{z}}_{i\mathit{\theta }}^{\top }\dot{g}\left({\overline{\gamma }}^{\top }{\text{Φ}}_i+{\overline{\mathit{\nu }}}_{\mathit{\theta }}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right)\left\{1-2\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\right]$. From the previous section we have

$$n^{\frac{1}{2}}(\tilde{\gamma }-\overline{\gamma })=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^n{\mathbf{C}}^{-1}I\left({\mathcal{S}}_i=s\right){\mathbf{\text{Φ}}}_i\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}\right]+o_p(1)$$

Similar arguments can be used to verify that

$$n^{\frac{1}{2}}{\left({\tilde{\mathit{\nu }}}_{\mathit{\theta }}-{\overline{\mathit{\nu }}}_{\mathit{\theta }}\right)}^{\top }=n^{\frac{1}{2}}{\mathbf{J}}_{\mathit{\theta }}^{-1}\left[\sum _{s=1}^S{\rho }_s{n}_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right){\mathbf{z}}_{i\mathit{\theta }}\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i+{\overline{\mathit{\nu }}}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right)\right\}+\mathbf{K}{(\tilde{\gamma }-\overline{\gamma })}^{\top }\right]+o_p(1)$$

where ${\mathbf{J}}_{\mathit{\theta }}=\text{E}\left\{{\mathbf{z}}_{i\mathit{\theta }}^{\otimes 2}\dot{g}\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i+{\overline{\mathit{\nu }}}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right)\right\}$ and ${\mathbf{K}}_{\mathit{\theta }}=-\text{E}\left\{{\mathbf{z}}_{i\mathit{\theta }}{\mathbf{\text{Φ}}}_i^{\top }\dot{g}\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i+{\overline{\mathit{\nu }}}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right)\right\}$. These results, together with the fact that $N^{-\frac{1}{2}}{\sum }_{i=1}^{N}d\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i+{\overline{\mathit{\nu }}}_{\mathit{\theta }}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right),\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}-D(\mathit{\theta }))$ converges weakly to zero-mean Gaussian process in θ, imply that

$$\begin{gathered} {\tilde{\mathcal{T}}}_{\text{SSL}}(\theta )=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s(n_s^{-1}\sum _{i=1}^{n}I({\mathcal{S}}_i=s)[\left({\mathbf{G}}_{\theta }+{\mathbf{H}}_{\theta }{\mathbf{J}}_{\theta }^{-1}{\mathbf{K}}_{\theta }\right){\mathbf{C}}^{-1}{\text{Φ}}_i\left\{y_i-g\left({\overline{\gamma }}^{\top }{\text{Φ}}_i\right)\right\} \\ +{\mathbf{H}}_{\theta }{\mathbf{J}}_{\theta }^{-1}{\mathbf{z}}_{i\theta }\left\{y_i-g\left({\overline{\gamma }}^{\top }{\text{Φ}}_i+{\overline{\nu }}_{\theta }^{\top }{\mathbf{z}}_{i\theta }\right)\right\}])+o_p(1). \end{gathered}$$

We may simplify the above expression by noting that $\left\{1-2\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}$ is a linear combination of z_iθ and hence $\left[{\mathbf{H}}_{\mathit{\theta }}{\mathbf{J}}_{\mathit{\theta }}^{-1}\right]{\mathbf{z}}_{i\mathit{\theta }}=\left\{1-2\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}$. Additionally, ${\mathbf{H}}_{\mathit{\theta }}{\mathbf{J}}_{\mathit{\theta }}^{-1}{\mathbf{K}}_{\mathit{\theta }}=-{\mathbf{G}}_{\mathit{\theta }}$ which implies that $\left({\mathbf{G}}_{\mathit{\theta }}+{\mathbf{H}}_{\mathit{\theta }}{\mathbf{J}}_{\mathit{\theta }}^{-1}{\mathbf{K}}_{\mathit{\theta }}\right){\mathbf{C}}^{-1}=0$. Thus,

$${\tilde{\mathcal{T}}}_{\text{SSL}}(\mathit{\theta })=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right)\left\{1-2\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i+{\overline{\mathit{\nu }}}^{\top }{\mathbf{z}}_{i\mathit{\theta }}\right)\right\}\right].$$

This combined with the fact that D(θ) is continuously differentiable at $\overline{\mathit{\theta }},\widehat{\text{W}}$ is consistent for its limiting value W introduced in Appendix B, and Conditions 1–3 then give that

$${\widehat{\mathcal{T}}}_{\text{SSL}}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)-D\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)\right\}+n^{\frac{1}{2}}\left\{D\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)-D(\overline{\mathit{\theta }})\right\}$$

$$\begin{gathered} =n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s(n_s^{-1}\sum _{i=1}^{n}I({\mathcal{S}}_i=s)[\left\{1-2{\overline{\mathcal{Y}}}_i\right\}\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i+{\overline{\mathit{\nu }}}_{\overline{\mathit{\theta }}}^{\top }{\mathbf{z}}_{i\overline{\mathit{\theta }}}\right)\right\} \\ +\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }\left\{\mathbf{W}{\mathbf{e}}_{\text{SSL}i}+(\mathbf{I}-\mathbf{W}){\mathbf{e}}_{\text{SL}i}\right\}])+o_p(1) \end{gathered}$$

$$\begin{gathered} =n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s(n_s^{-1}\sum _{i=1}^{n}I({\mathcal{S}}_i=s)[\left\{d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-d\left(m_{\text{II},i},{\overline{\mathcal{Y}}}_i\right)\right\} \\ +\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }\left\{\mathbf{W}{\mathbf{e}}_{\text{SSL}i}+(\mathbf{I}-\mathbf{W}){\mathbf{e}}_{\text{SL}i}\right\}])+o_p(1). \end{gathered}$$

Note that the existence of $\dot{\mathbf{\text{D}}}(\overline{\mathit{\theta }})$ is implied by Condition 1, namely, that the density function of ${\overline{\mathit{\theta }}}^{\top }\mathbf{\text{x}}$ is continuously differentiable in ${\overline{\mathit{\theta }}}^{\top }\mathbf{\text{x}}$ and $\text{P}(y=1\mid \mathbf{u})$ is continuously differentiable in the continuous components of u. We then have that $n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)-D(\overline{\mathit{\theta }})\right\}$ converges to a zero mean normal random variable by the classical Central Limit Theorem. Using similar arguments as those for ${\breve{\mathcal{T}}}_{\text{SL}}$, we have that ${\check{\mathcal{T}}}_{\text{SSL}}=n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}\left({\check{\mathit{\theta }}}_{\text{SSL}}\right)-D(\overline{\mathit{\theta }})\right\}$ can be expanded as

$$n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left(n_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right)\left[\left\{d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-d\left(m_{\text{II},i},{\overline{\mathcal{Y}}}_i\right)\right\}+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{e}}_{\text{ssL}i}\right]\right),$$

which also converges to a zero mean normal random variable.

Comparing the asymptotic variance of ${\check{\mathcal{T}}}_{\text{SSL}}$ with ${\widehat{\mathcal{T}}}_{\text{SL}}$, first note that

$${\widehat{\mathcal{T}}}_{\text{sL}}=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left(n_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right)\left[d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-D(\overline{\mathit{\theta }})+\dot{D}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{e}}_{\text{SL}i}\right]\right)+o_p(1)$$

$$\begin{gathered} =n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s(n_s^{-1}\sum _{i=1}^{n}I({\mathcal{S}}_i=s)[d\left(y_i,{\overline{\mathcal{Y}}}_i\right)-d\left(m_{\text{II},i},{\overline{\mathcal{Y}}}_i\right)+d\left(m_{\text{II},i},{\overline{\mathcal{Y}}}_i\right)-D(\overline{\mathit{\theta }}) \\ +\dot{D}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}])+o_p(1) \end{gathered}$$

$$\begin{gathered} =n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s(n_s^{-1}\sum _{i=1}^{n}I({\mathcal{S}}_i=s)[\left(1-2{\overline{\mathcal{Y}}}_i\right)\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i+{\overline{\mathit{\nu }}}_{\overline{\mathit{\theta }}}^{\top }{\mathbf{z}}_{i\overline{\mathit{\theta }}}\right)\right\}+d\left(m_{\text{II},i},{\overline{\mathcal{Y}}}_i\right)-D(\overline{\mathit{\theta }}) \\ +\dot{D}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)+g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}])+o_p(1). \end{gathered}$$

Letting $h_1\left({\mathbf{\text{Φ}}}_i\right)=1-2{\overline{\mathcal{Y}}}_i+\dot{D}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i$ and

$$h_2\left({\mathbf{\Phi }}_i\right)=d\left(m_{\text{II},i},{\overline{\mathcal{Y}}}_i\right)-D(\overline{\mathit{\theta }})+\dot{D}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{g\left({\overline{\gamma }}^{\top }{\mathbf{\Phi }}_i\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}.$$

we note that $h_1\left({\mathbf{\text{Φ}}}_i\right)$ and $h_2\left({\mathbf{\text{Φ}}}_i\right)$ are functions of Φ_i and do not depend on y_i. Thus, when $P(y=1\mid \mathbf{u})=g\left({\overline{\gamma }}^{\top }\mathbf{\text{Φ}}\right)$, we have $\overline{\mathit{\nu }}=\mathbf{0}$ and

$$\begin{gathered} {\sigma }_{\text{SL}}^2=\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}\text{E}\left[h_1^2\left({\mathbf{\text{Φ}}}_i\right){\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}}^2+2h_1\left({\mathbf{\text{Φ}}}_i\right)h_2\left({\mathbf{\text{Φ}}}_i\right)\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}+h_2^2\left({\mathbf{\text{Φ}}}_i\right)\mid {\mathcal{S}}_i=s\right] \\ =\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}\text{E}\left[h_1{\left({\mathbf{\text{Φ}}}_i\right)}^2{\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}}^2+h_2{\left({\mathbf{\text{Φ}}}_i\right)}^2\mid {\mathcal{S}}_i=s\right] \end{gathered}$$

while the asymptotic variance of ${\check{\mathcal{T}}}_{\text{SSL}}$ is

$${\sigma }_{\text{SSL}}^2=\sum _{s=1}^S{\rho }_s^2{\rho }_{1s}^{-1}\text{E}\left[h_1^2\left({\mathbf{\text{Φ}}}_i\right){\left\{y_i-g\left({\overline{\gamma }}^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}}^2\mid {\mathcal{S}}_i=s\right].$$

Therefore, when $P(y=1\mid \mathbf{u})=g\left({\overline{\gamma }}^{\top }\mathbf{\Phi }\right)$, it follows that ${\text{Δ}}_{aVar}:={\sigma }_{\text{SL}}^2-{\sigma }_{\text{SSL}}^2>0$. Additionally, when model (1) is correct and $P(y=1\mid \mathbf{u})=g\left({\overline{\gamma }}^{\top }\mathbf{\text{Φ}}\right)=g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)$, we have $h_2\left({\mathbf{\text{Φ}}}_i\right)=d\left(m_{\text{II},i},{\overline{\mathcal{Y}}}_i\right)-D(\overline{\mathit{\theta }})$ which is not equal to 0 with probability 1, so that again ${\text{Δ}}_{aVar}>0$.

G. Intrinsic Efficient Estimation

G.1. Intrinsic Efficient Estimator for $\overline{D}$

We first introduce the intrinsic efficient estimator of the accuracy measures. Without loss of generality, we set the imputation basis for both θ and D(θ) as ${\text{Ψ}}_{\mathit{\theta }i}={\left[{\mathbf{\text{Φ}}}_i^{\top },\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right]}^{\top }$, where θ is plugged in with some preliminary estimator for θ, denoted as $\tilde{\mathit{\theta }}$. In practice, one may take $\tilde{\mathit{\theta }}$ as either the simple SL estimator or the SSL estimator obtained following SectionT 3.1. We include $\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)$ in the imputation basis to simplify the notation and presentation in this section. Although this distinguishes the following discussion from the proposal in Section 3.2, it is straightforward to extend our results to the original proposal.

Recall that for the original SSL estimator of the regression parameter, one first obtains ${\tilde{\gamma }}_{\widehat{\mathit{\theta }}}$ as the solution to

$$N^{-1}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{\text{Ψ}}}_{{\tilde{\mathit{\theta }}}_i}\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\text{Ψ}}}_{{\tilde{\mathit{\theta }}}_i}\right)\right\}-{\lambda }_n\gamma =\mathbf{0}$$

and then solves $N^{-1}{\sum }_{i=1}^N{\mathbf{x}}_i\left\{g\left({\tilde{\gamma }}_{\tilde{\mathit{\theta }}}^{\top }{\mathbf{\Psi }}_{{\tilde{\mathit{\theta }}}_i}\right)-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}=\mathbf{0}$ to obtain the estimator of $\overline{\mathit{\theta }}$. Despite the change in basis, we still denote this estimator as ${\widehat{\mathit{\theta }}}_{\text{SSL}}$ with a slight abuse in the notation. Adapting the augmentation procedure in Section 3.2, we then find ${\tilde{\gamma }}_{{\widehat{\mathit{\theta }}}_{\text{SSL}}}$ as the solution to

$$N^{-1}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{\text{Ψ}}}_{{\widehat{\mathit{\theta }}}_{\text{SSL}}i}\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\text{Ψ}}}_{{\widehat{\mathit{\theta }}}_{\text{SSL}}i}\right)\right\}-{\lambda }_n\gamma =\mathbf{0},$$

and estimate $\overline{D}$ with ${\widehat{D}}_{\text{SSL}}={\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)$ where ${\widehat{D}}_{\text{SSL}}(\mathit{\theta })=N^{-1}{\sum }_{i=1}^{N}d\left\{g\left({\tilde{\gamma }}_{\mathit{\theta }}^{\top }{\mathbf{\text{Ψ}}}_{\mathit{\theta }i}\right),\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}$. Extending Theorem 2, the asymptotic variance of $n^{\frac{1}{2}}\left\{{\widehat{D}}_{\text{SSL}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}\right)-\overline{D}\right\}$ may be expressed as

$$\frac{1}{n}\sum _{i=1}^n\text{E}{\zeta }_i{\left\{1-2{\overline{\mathcal{Y}}}_i+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\right\}}^2{\left\{y_i-g\left({\overline{\gamma }}_{\overline{\mathit{\theta }}}^{\top }{\overline{\mathbf{\Psi }}}_i\right)\right\}}^2,$$ (G.1)

where ${\overline{\gamma }}_{\overline{\mathit{\theta }}}$ represents the limits of ${\tilde{\gamma }}_{{\widehat{\mathit{\theta }}}_{\text{SSL}}}\left(\text{or}{\tilde{\gamma }}_{\tilde{\mathit{\theta }}}\right)$, and ${\overline{\mathbf{\text{Ψ}}}}_i={\overline{\mathbf{\text{Ψ}}}}_{{\overline{\mathit{\theta }}}_i}={\left[{\mathbf{\text{Φ}}}_i^{\top },\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right]}^{\top }$. Analogous to the construction of ${\mathit{e}}^{\top }{\widehat{\mathit{\theta }}}_{\text{intri}}$, we consider minimizing the asymptotic variance given by (G.1) to estimate $\overline{D}$. Specifically, we first solve for ${\tilde{\gamma }}_{\tilde{\mathit{\theta }}}^{(2)}$ with

$$\begin{gathered} \underset{\gamma }{\text{argmin}}\frac{1}{2n}\sum _{i=1}^n{\widehat{\zeta }}_i{\left\{1-2\mathcal{Y}\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)+{\widehat{\dot{\mathbf{D}}}}^{\top }{\widehat{\mathbf{A}}}^{-1}{\mathbf{x}}_i\right\}}^2{\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\Psi }}_{{\tilde{\mathit{\theta }}}_i}\right)\right\}}^2+{\lambda }_n^{(2)}\parallel \gamma {\parallel }_2^2, \\ \text{s.t.}\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i{\left[{\mathbf{x}}_i^{\top },\mathcal{Y}\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right]}^{\top }\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\Psi }}_{{\tilde{\mathit{\theta }}}_i}\right)\right\}=0, \end{gathered}$$ (G.2)

where $\widehat{\dot{\mathbf{\text{D}}}}$ is an estimation of $\dot{\mathbf{D}}(\overline{\mathit{\theta }})$ and the tuning parameter ${\lambda }^{(2)}=o\left(n^{-\frac{1}{2}}\right)$. Similar to (3) and (9), moment constraints in (G.2) calibrate potential bias of the estimators for $\overline{\mathit{\theta }}$ and $\overline{\mathit{D}}(\mathit{\theta })$.

Next, we present the construction of $\widehat{\dot{\mathbf{\text{D}}}}$ for the Brier score and OMR separately. For the Brier score, ${\overline{D}}_1$, we take

$$\widehat{\dot{\mathbf{\text{D}}}}={\widehat{\dot{\mathbf{\text{D}}}}}_1=\frac{1}{N}\sum _{i=1}^N-2{\widehat{w}}_i\dot{g}\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\left\{y_i-g\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}{\mathbf{x}}_i.$$

For the the OMR, ${\overline{D}}_2$, recall that a simple estimator is given by the empirical average

$$\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i\left\{y_i+\left(1-2y_i\right)I\left(g\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)>c\right)\right\}.$$

Since $I\left(g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)>c\right)$ is not a differentiable function of θ, we first smooth each $I\left(g\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)>c\right)$ as ${\int }_c^{+\infty }{K}_h\left\{g\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)-u\right\}du$, where K(·) represents the Gaussian kernel function, and $K_h(a):=h^{-1}K(a/h)$ with some bandwidth h > 0. Then, $\dot{\mathbf{\text{D}}}(\overline{\mathit{\theta }})$ is estimated with

$$\widehat{\dot{\mathbf{D}}}={\widehat{\dot{\mathbf{D}}}}_2=\frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i\left(1-2y_i\right)\dot{g}\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)K_h\left\{g\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)-c\right\}{\mathbf{x}}_i.$$

With ${\tilde{\gamma }}_{\tilde{\mathit{\theta }}}^{(2)}$, we then solve

$$N^{-1}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\tilde{\gamma }}_{\tilde{\mathit{\theta }}}^{(2)\top }{\mathbf{\Psi }}_{{\tilde{\mathit{\theta }}}_i}\right)-g\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}=\mathbf{0}$$

to obtain ${\widehat{\mathit{\theta }}}_{\text{intri}}^D$ for estimation of $\overline{D}$ and employ the augmentation procedure in Section 3.2.

That is, we solve ${\tilde{\gamma }}_{{\widehat{\mathit{\theta }}}_{\text{intri}}^D}^{(2)}$ from

$$N^{-1}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{\text{Ψ}}}_{{\widehat{\mathit{\theta }}}_{\text{intri}}^D}\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\text{Ψ}}}_{{\widehat{\mathit{\theta }}}_{\text{intri}}^{D}i}\right)\right\}-{\lambda }_n^{(2)}\gamma =\mathbf{0},$$

and estimate $\overline{D}$ by ${\widehat{D}}_{\text{intri}}={\widehat{D}}_{\text{intri}}\left({\widehat{\mathit{\theta }}}_{\text{intri}}^D\right)$ where ${\widehat{D}}_{\text{intri}}(\mathit{\theta })=N^{-1}{\sum }_{i=1}^{N}d\left\{g\left({\tilde{\gamma }}_{\mathit{\theta }}^{(2)}{\mathbf{\Psi }}_{\mathit{\theta }i}\right),\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_i\right)\right\}$.

To present the asymptotic properties of ${\widehat{D}}_{\text{intri}}$, we define

$$\begin{array}{c}{\overline{\gamma }}_{\overline{\mathit{\theta }}}^{(2)}=\underset{\gamma }{\text{argmin}}E\left[R{\left\{1-2\overline{\mathcal{Y}}+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}\mathbf{x}\right\}}^2{\left\{y-g\left({\gamma }_{\overline{\mathit{\theta }}}^{\top }\overline{\mathbf{\text{Ψ}}}\right)\right\}}^2\right],\\ \text{s.t.}\text{E}{\left[{\mathbf{x}}^{\top },\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\right]}^{\top }\left\{y-g\left({\gamma }_{\overline{\mathit{\theta }}}^{\top }\overline{\mathbf{\text{Ψ}}}\right)\right\}=0.\end{array}$$

Theorem A1 provides the asymptotic expansion of ${\widehat{D}}_{\text{intri}}$ and its proof, together with the proof of Theorem 3 from the main text, is detailed in Appendix G.2.

Theorem A1.

Under Conditions 1, Conditions A1 and A2 from Appendix G.2, and with the bandwidth $h\asymp n^{-\frac{1}{4}}$, $n^{\frac{1}{2}}\left({\widehat{D}}_{\mathit{\text{intri}}}-\overline{D}\right)$ weakly converges to a Gaussian distribution with mean zero, and is asymptotically equivalent to $\widehat{\mathcal{T}}\left({\overline{\gamma }}_{\overline{\mathit{\theta }}}^{(2)}\right)$ where

$$\widehat{\mathcal{T}}(\gamma )=n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^N{V}_{i}I\left({\mathcal{S}}_i=s\right)\left\{1-2{\overline{\mathcal{Y}}}_i+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\right\}\left\{y_i-g\left({\gamma }^{\top }{\overline{\mathbf{\Psi }}}_i\right)\right\}\right].$$

This implies that (i) ${\widehat{D}}_{intri}$ is asymptotically equivalent to ${\widehat{D}}_{SSL}$ when the imputation model $\text{P}(y=1\mid \mathbf{u})=g\left({\gamma }^{\top }\overline{\mathbf{\text{Ψ}}}\right)$ is correctly specified and (ii) the asymptotic variance of $n^{\frac{1}{2}}\left({\widehat{D}}_{intri}-\overline{D}\right)$ is minimized among $\left\{\widehat{\mathcal{T}}(\gamma ):\text{E}{\left[{\mathbf{x}}^{\top },\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\right]}^{\top }\left\{y-g\left({\gamma }^{\top }\overline{\mathbf{\text{Ψ}}}\right)\right\}=\mathbf{0}\right\}$. Consequently, the asymptotic variance of the intrinsic efficient estimator is always less than or equal to the asymptotic variance of $n^{\frac{1}{2}}\left({\widehat{D}}_{SSL}-\overline{D}\right)$ and $n^{\frac{1}{2}}\left({\widehat{D}}_{SL}-\overline{D}\right)$.

G.2. Asymptotic Properties of ${\widehat{\theta }}_{\text{intri}}$ and ${\widehat{D}}_{\text{intri}}$

We first introduce the smoothness condition on the link function g(·), which is stronger than Condition 2, but still holds for the most commonly used link functions such as the logit and probit functions.

Condition A1.

The link function $g(\cdot )\in (0,1)$ is continuously twice differentiable with derivative $\dot{g}(\cdot )$ and the second order derivative $\ddot{g}(\cdot )$.

Given Condition A1, we let ${\overline{\gamma }}^{(2)}={\overline{\gamma }}_{\overline{\mathit{\theta }}}^{\left(2\right)}$ and define

$${\mathbf{A}}_1=\text{E}\left[R{\left({\mathit{e}}^{\top }{\mathbf{A}}^{-1}\mathbf{x}\right)}^2{\mathbf{\Phi }}^{\otimes 2}\left\{{\dot{g}}^2\left({\overline{\gamma }}^{(1)T}\mathbf{\Phi }\right)+\ddot{g}\left({\overline{\gamma }}^{(1)\top }\mathbf{\Phi }\right)\left[y-g\left({\overline{\gamma }}^{(1)\top }\mathbf{\Phi }\right)\right]\right\}\right],$$

$${\mathbf{A}}_2=\text{E}\left[R{\left\{1-2\overline{\mathcal{Y}}+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}\mathbf{x}\right\}}^2{\overline{\mathbf{\Psi }}}^{\otimes 2}\left\{{\dot{g}}^2\left({\overline{\gamma }}^{(2)\top }\overline{\mathbf{\Psi }}\right)+\ddot{g}\left({\overline{\gamma }}^{(2)\top }\overline{\mathbf{\text{Ψ}}}\right)\left[y-g\left({\overline{\gamma }}^{(2)\text{T}}\overline{\mathbf{\Psi }}\right)\right]\right\}\right],$$

${\mathbf{\text{B}}}_1=\text{E}\left[\mathbf{\text{Φ}}{\mathbf{x}}^{\top }\dot{g}\left({\overline{\gamma }}^{(1)\top }\mathbf{\text{Φ}}\right)\right]$ and ${\mathbf{B}}_2=\text{E}\left[\overline{\mathbf{\text{Ψ}}}\left\{{\mathbf{x}}^{\top },\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\right\}\dot{g}\left({\overline{\gamma }}^{(2)\top }\overline{\mathbf{\Psi }}\right)\right]$. We next present the regularity condition on the covariates and regression coefficients required by Theorem 3.

Condition A2.

There exists ${\mathbf{\text{Θ}}}^{\prime }=\left\{\mathit{\theta }:\parallel \mathit{\theta }-\overline{\mathit{\theta }}{\parallel }_2<{\delta }^{\prime }\right\}$ for some δ^′ > 0, such that for any $\mathit{\theta }\in {\mathbf{\text{Θ}}}^{\prime }$, there is no γ such that $P\left({\gamma }^{\top }{\mathbf{\text{Φ}}}_1>{\gamma }^{\top }{\mathbf{\text{Φ}}}_2\mid y_1>y_2\right)=1$ or $P({\gamma }^{\top }{\mathbf{\text{Ψ}}}_{\mathit{\theta }1}>{\gamma }^{\top }{\mathbf{\text{Ψ}}}_{\mathit{\theta }2}\mid y_1>y_2)=1$. It is also the case that $\mathbf{\text{A}}\succ 0,{\mathbf{\text{A}}}_1\succ 0,{\mathbf{\text{A}}}_2\succ 0,{\mathbf{\text{B}}}_1^{\top }{\mathbf{\text{A}}}_1^{-1}{\mathbf{\text{B}}}_1\succ \mathbf{0}$ and ${\mathbf{B}}_2^{\top }{\mathbf{A}}_2^{-1}{\mathbf{B}}_2\succ \mathbf{0}$.

Remark A1.

Condition A2 is analog to Condition 3. It assumes there is no linear combination of $\mathbf{\Phi }$ or $\overline{\mathbf{\Psi }}$ perfectly separating the samples based on y, and the Hessian matrices of the constrained least square problems for ${\overline{\gamma }}^{(1)}$ and ${\overline{\gamma }}^{(2)}$ are positive definite. Again, these assumptions are mild and common in the M-estimation literature [Van der Vaart, 2000].

Under these regularity conditions, we present the proofs of Theorem 3 and A1. In our development, we take $\tilde{\mathit{\theta }}$ as the SSL estimator for θ introduced in Section 3.1, but note that proof remains basically unchanged when taking $\tilde{\mathit{\theta }}$ as the SL estimator. We first derive the consistency (error rates) of ${\widehat{\dot{\mathbf{\text{D}}}}}_{\text{1}}$ and ${\widehat{\dot{\mathbf{\text{D}}}}}_{\text{2}}$. For the Brier score, letr ${\overline{\dot{\mathbf{\text{D}}}}}_{\text{1}}$ denote the derivative the of $D_1(\mathit{\theta })$ evaluated at $\overline{\mathit{\theta }}$. We then use ${\widehat{\rho }}_s\stackrel{p}{\to }{\rho }_s$, Theorem 1, Conditions 1 and A1, and the classical Central Limit Theorem to derive that

$$\begin{array}{l}{\widehat{\dot{\mathbf{D}}}}_1-{\overline{\dot{\mathbf{D}}}}_1=\frac{1}{N}\sum _{i=1}^N-2{\widehat{w}}_i\dot{g}\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\left\{y_i-g\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}{\mathbf{x}}_i-\frac{1}{N}\sum _{i=1}^N-2w_i\dot{g}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\left\{y_i-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}{\mathbf{x}}_i\\ +\frac{1}{N}\sum _{i=1}^N-2w_i\dot{g}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\left\{y_i-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}{\mathbf{x}}_i-\text{E}\left[2\dot{g}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\left\{y-g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\right\}\mathbf{x}\right]\\ =O_p\left(\parallel \tilde{\mathit{\theta }}-\overline{\mathit{\theta }}{\parallel }_2\right)+O_p\left(n^{-\frac{1}{2}}\right)=O_p\left(n^{-\frac{1}{2}}\right).\end{array}$$

For the estimator of the derivative for the OMR, let ${\overline{\dot{\mathbf{\text{D}}}}}_2$ be the limiting value of ${\widehat{\dot{\mathbf{\text{D}}}}}_{\text{2}}$. We then have

$$\begin{gathered} {\widehat{\dot{\mathbf{D}}}}_2-{\overline{\dot{\mathbf{D}}}}_2=\frac{1}{N}\sum _{i=1}^N\left(1-2y_i\right)\left[{\widehat{w}}_i\dot{g}\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)K_h\left\{g\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)-c\right\}-w_i\dot{g}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)K_h\left\{g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)-c\right\}\right]{\mathbf{x}}_i \\ +\frac{1}{N}\sum _{i=1}^N{w}_i\left(1-2y_i\right)\dot{g}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)K_h\left\{g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)-c\right\}{\mathbf{x}}_i-\text{E}\left[(1-2y)\dot{g}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\mathbf{x}\mid g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)=c\right]f_g(c) \\ =:{\mathbf{\text{Δ}}}_1+{\mathbf{\text{Δ}}}_2 \end{gathered}$$

where $f_g(c)$ represent the density function of $g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)$ evaluated at c. This follows from the fact that ${\overline{\dot{\mathbf{D}}}}_2=\text{E}\left[(1-2y)\dot{g}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\mathbf{x}\mid g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)=c\right]f_g(c)$. Since the Gaussian kernel K(·) is continuously differentiable and by Theorem 1, Conditions 1 and A1, we have

$${\Vert {\mathbf{\Delta }}_1\Vert }_2=h^{-1}{O}_p\left(\parallel \tilde{\mathit{\theta }}-\overline{\mathit{\theta }}{\parallel }_2\right)=O_p\left(n^{-\frac{1}{2}}{h}^{-1}\right).$$

For Δ₂, Condition 1 and the classical Central Limit Theorem imply that

$$\frac{1}{N}\sum _{i=1}^N{w}_i\left(1-2y_i\right)\dot{g}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)K_h\left\{g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)-c\right\}{\mathbf{x}}_i-\text{E}(1-2y)\dot{g}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)K_h\left\{g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)-c\right\}\mathbf{x}=O_p\left\{{(nh)}^{-\frac{1}{2}}\right\},$$

and from Condition 1,

$$\begin{gathered} \text{E}(1-2y)\dot{g}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)K_h\left\{g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)-c\right\}\mathbf{x}-\text{E}\left[(1-2y)\dot{g}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\mathbf{x}\mid g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)=c\right]f_g(c) \\ ={\int }_0^1\left\{\mathit{r}(u)f_g(u)K_h(u-c)-\mathit{r}(c)f_g(c)\right\}du \\ ={\int }_{-c/h}^{(1-c)/h}\left\{\mathit{r}(c+hv)f_g(c+hv)K(v)-\mathit{r}(c)f_g(c)\right\}dv=O(h), \end{gathered}$$

where $r(u)=\underset{\left\{\mathbf{x}:g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)=u\right\}}{\int }\{1-2\text{P}(y=1\mid \mathbf{x})\}\dot{g}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\mathbf{x}{f}_{x\mid g}(\mathbf{x}\mid u)d\mathbf{x}$, and $f_{x\mid g}(\cdot \mid u)$ represent the density of x given that $g\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)=u$. By Condition 1, there exists C > 0 such that $\parallel \mathit{r}(a)-\mathit{r}(b){\parallel }_2⩽C|a-b|$ for any $a,b\in ℝ$. Thus, we have ${\Vert {\mathbf{\Delta }}_2\Vert }_2=O_p\left\{{(nh)}^{-\frac{1}{2}}+h\right\}$ and with $h\asymp n^{-\frac{1}{4}}$, we obtain ${\Vert {\widehat{\dot{\mathbf{D}}}}_2-{\overline{\dot{\mathbf{D}}}}_2\Vert }_2=O_p\left(n^{-\frac{1}{4}}\right)$. It then follows that for both Brier score and OMR, $\parallel \widehat{\dot{\mathbf{\text{D}}}}-\overline{\dot{\mathbf{\text{D}}}}{\parallel }_2=O_p\left(n^{-\frac{1}{4}}\right)=o_p(1)$.

Leveraging these results, we establish the asymptotic normality of ${\tilde{\gamma }}^{(1)}$ and ${\tilde{\gamma }}_{\tilde{\mathit{\theta }}}^{(2)}$. Similar to Appendices C and E, we apply the ULLN [Pollard, 1990], together with Conditions 1, A1, and A2, and the facts that ${\widehat{\rho }}_s\stackrel{p}{\to }{\rho }_s,{\widehat{\rho }}_{1s}\stackrel{p}{\to }{\rho }_{1s}$, and that $\tilde{\mathit{\theta }},{\widehat{\mathbf{A}}}^{-1}$ and $\widehat{\dot{\mathbf{\text{D}}}}$ are consistent for their respective limits, to obtain

$$\begin{gathered} \underset{\gamma \in {\text{Γ}}^{(1)}}{\sup }\left|\frac{1}{n}\sum _{i=1}^n{\widehat{\zeta }}_i{\left({\mathit{e}}^{\top }{\widehat{\mathbf{A}}}^{-1}{\mathbf{x}}_i\right)}^2{\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}}^2-\text{E}\left[R{\left(e^{\top }{\mathbf{A}}^{-1}\mathbf{x}\right)}^2{\left\{y-g\left({\gamma }^{\top }\mathbf{\text{Φ}}\right)\right\}}^2\right]\right|=o_p(1); \\ \underset{\gamma \in {\text{Γ}}^{(1)}}{\sup }{\Vert \frac{1}{N}\sum _{i=1}^N{\widehat{w}}_i{\mathbf{x}}_i\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\text{Φ}}}_i\right)\right\}-\text{E}\left[\mathbf{x}\left\{y-g\left({\gamma }^{\top }\mathbf{\text{Φ}}\right)\right\}\right]\Vert }_2=o_p(1); \end{gathered}$$

$$\begin{gathered} \underset{\gamma \in {\text{Γ}}^{(2)}}{\sup }\mid \frac{1}{n}\sum _{i=1}^n{\widehat{\zeta }}_i{\left\{1-2\mathcal{Y}\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)+{\widehat{\dot{\mathbf{D}}}}^{\top }{\widehat{\mathbf{A}}}^{-1}{\mathbf{x}}_i\right\}}^2{\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\Psi }}_{{\tilde{\mathit{\theta }}}_i}\right)\right\}}^2 \\ -\text{E}\left[R{\left\{1-2\overline{\mathcal{Y}}+\dot{\mathbf{D}}{(\overline{\mathit{\theta }})}^{\top }{\mathbf{A}}^{-1}\mathbf{x}\right\}}^2{\left\{y-g\left({\gamma }^{\top }\overline{\mathbf{\text{Ψ}}}\right)\right\}}^2\right]\mid =o_p(1); \end{gathered}$$

$$\underset{\gamma \in {\text{Γ}}^{(2)}}{\sup }\frac{1}{N}{\Vert \sum _{i=1}^N{\widehat{w}}_i{\left[{\mathbf{x}}_i^{\top },\mathcal{Y}\left({\tilde{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right]}^{\top }\left\{y_i-g\left({\gamma }^{\top }{\mathbf{\Psi }}_{\tilde{\mathit{\theta }}i}\right)\right\}-\text{E}{\left[{\mathbf{x}}^{\top },\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\right]}^{\top }\left\{y-g\left({\gamma }^{\top }\overline{\mathbf{\text{Ψ}}}\right)\right\}\Vert }_2=o_p(1),$$

where ${\text{Γ}}^{(1)}$ and ${\text{Γ}}^{(2)}$ are two compact sets containing ${\overline{\gamma }}^{(1)}$ and ${\overline{\gamma }}^{(2)}$, respectively. This implies that ${\Vert {\tilde{\gamma }}^{(1)}-{\overline{\gamma }}^{(1)}\Vert }_2=o_p(1)$ and ${\Vert {\tilde{\gamma }}_{\tilde{\mathit{\theta }}}^{(2)}-{\overline{\gamma }}^{(2)}\Vert }_2=O_p(1)$. We then expand (9) and (G.2) to derive that

$$\begin{gathered} {\tilde{\gamma }}^{(1)}=\underset{\gamma }{\text{argmin}}{\left(\gamma -{\overline{\gamma }}^{(1)}\right)}^{\top }\left[{\mathbf{A}}_1\left(\gamma -{\overline{\gamma }}^{(1)}\right)+2\left\{1+o_p(1)\right\}{\text{Ξ}}_{11}+o_p\left({\Vert {\tilde{\gamma }}^{(1)}-{\overline{\gamma }}^{(1)}\Vert }_2+n^{-\frac{1}{2}}\right)\right], \\ \text{s.t.}{\mathbf{B}}_1^{\top }\left(\gamma -{\overline{\gamma }}^{(1)}\right)-\left\{1+o_p(1)\right\}{\text{Ξ}}_{12}+o_p\left({\Vert {\tilde{\gamma }}^{(1)}-{\overline{\gamma }}^{(1)}\Vert }_2+n^{-\frac{1}{2}}\right)=\mathbf{0}; \end{gathered}$$

$$\begin{gathered} {\tilde{\gamma }}_{\tilde{\theta }}^{(2)}=\underset{\gamma }{\text{argmin}}{\left(\gamma -{\overline{\gamma }}^{(2)}\right)}^{\top }\left[{\mathbf{A}}_2\left(\gamma -{\overline{\gamma }}^{(2)}\right)+2\left\{1+o_p(1)\right\}{\text{Ξ}}_{21}+o_p\left({\Vert {\tilde{\gamma }}^{(2)}-{\overline{\gamma }}^{(2)}\Vert }_2+n^{-\frac{1}{2}}\right)\right], \\ \text{s.t.}{\mathbf{B}}_2^{\top }\left(\gamma -{\overline{\gamma }}^{(2)}\right)-\left\{1+o_p(1)\right\}{\text{Ξ}}_{22}+o_p\left({\Vert {\tilde{\gamma }}^{(2)}-{\overline{\gamma }}^{(2)}\Vert }_2+n^{-\frac{1}{2}}\right)\text{,} \end{gathered}$$

$$\text{where}{\text{Ξ}}_{11}=\frac{1}{n}\sum _{i=1}^n{\zeta }_i{\left(e^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\right)}^2\dot{g}\left({\overline{\gamma }}^{(1)\top }{\mathbf{\text{Φ}}}_i\right){\mathbf{\text{Φ}}}_i\left\{y_i-g\left({\overline{\gamma }}^{(1)\top }{\mathbf{\text{Φ}}}_i\right)\right\};$$

$${\text{Ξ}}_{12}=\frac{1}{N}\sum _{i=1}^N{w}_i{\mathbf{x}}_i\left\{y_i-g\left({\overline{\gamma }}^{(1)\top }{\mathbf{\text{Φ}}}_i\right)\right\};$$

$${\mathbf{\Xi }}_{21}=\frac{1}{n}\sum _{i=1}^n{\zeta }_i{\left\{1-2\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)+{\dot{\mathbf{D}}}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\right\}}^2\dot{g}\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\text{Ψ}}}}_i\right){\overline{\mathbf{\text{Ψ}}}}_i\left\{y_i-g\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\Psi }}}_i\right)\right\}+{\dot{\text{Ξ}}}_{\mathit{\theta },21}^{\top }(\tilde{\theta }-\overline{\theta });$$

$${\text{Ξ}}_{22}=\frac{1}{N}\sum _{i=1}^N{w}_i{\left[{\mathbf{x}}_i^{\top },\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right]}^{\top }\left\{y_i-g\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\text{Ψ}}}}_i\right)\right\}+{\dot{\text{Ξ}}}_{\mathit{\theta },22}^{\top }(\tilde{\mathit{\theta }}-\overline{\mathit{\theta }}),$$

and ${\dot{\text{Ξ}}}_{\mathit{\theta },21}$, ${\dot{\text{Ξ}}}_{\mathit{\theta },22}$ are two fixed loading matrices of the order O(1). By Condition 1 and the classical Central Limit Theorem, $n^{\frac{1}{2}}{\left({\mathbf{\Xi }}_{11}^{\top },{\mathbf{\Xi }}_{12}^{\top },{\mathbf{\Xi }}_{21}^{\top },{\mathbf{\Xi }}_{22}^{\top }\right)}^{\top }$ converges to a Gaussian distribution with mean 0. By Theorem $n^{\frac{1}{2}}(\tilde{\mathit{\theta }}-\overline{\mathit{\theta }})$ also converges to a mean-zero Gaussian distribution. Analogous to the proof of Theorem 5.21 of Van der Vaart [2000], we then obtain

$${\tilde{\gamma }}^{(1)}-{\overline{\gamma }}^{(1)}=\left[{\mathbf{A}}_1^{-1}-{\mathbf{A}}_1^{-1}{\mathbf{B}}_1{\left({\mathbf{B}}_1^{\top }{\mathbf{A}}_1^{-1}{\mathbf{B}}_1\right)}^{-1}{\mathbf{B}}_1^{\top }{\mathbf{A}}_1^{-1}\right]{\mathbf{\Xi }}_{11}+{\mathbf{A}}_1^{-1}{\mathbf{B}}_1{\left({\mathbf{B}}_1^{\top }{\mathbf{A}}_1^{-1}{\mathbf{B}}_1\right)}^{-1}{\mathbf{\Xi }}_{12}=O_p\left(n^{-\frac{1}{2}}\right);$$

$${\tilde{\gamma }}_{\tilde{\theta }}^{(2)}-{\overline{\gamma }}^{(2)}=\left[{\mathbf{A}}_2^{-1}-{\mathbf{A}}_2^{-1}{\mathbf{B}}_2{\left({\mathbf{B}}_2^{\top }{\mathbf{A}}_2^{-1}{\mathbf{B}}_2\right)}^{-1}{\mathbf{B}}_2^{\top }{\mathbf{A}}_2^{-1}\right]{\mathbf{\Xi }}_{21}+{\mathbf{A}}_2^{-1}{\mathbf{B}}_2{\left({\mathbf{B}}_2^{\top }{\mathbf{A}}_2^{-1}{\mathbf{B}}_2\right)}^{-1}{\mathbf{\Xi }}_{22}=O_p\left(n^{-\frac{1}{2}}\right).$$ (G.3)

By Conditions 1, A1 and A2, the consistency of ${\widehat{\rho }}_1$ for its limit, and the asymptotic expansion of ${\tilde{\gamma }}^{(1)}-{\overline{\gamma }}^{(1)}$ derived above, we can use the argument of Appendix E to show that ${\widehat{\mathit{\theta }}}_{\text{intri}}\stackrel{p}{\to }\overline{\mathit{\theta }}$ and obtain the expansion

$$\begin{gathered} n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{\text{intri}}-\overline{\mathit{\theta }}\right)=n^{\frac{1}{2}}{\mathbf{A}}^{-1}\left[N^{-1}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\overline{\gamma }}^{(1)\top }{\mathbf{\Phi }}_i\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}+{\mathbf{B}}_1^{\top }\left({\tilde{\gamma }}^{(1)}-{\overline{\gamma }}^{(1)}\right)\right]+o_p(1), \\ =n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right){\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\gamma }}^{(1)\top }{\text{Φ}}_i\right)\right\}\right]+o_p(1)=\widehat{\mathcal{W}}\left({\overline{\gamma }}^{(1)}\right)+o_p(1). \end{gathered}$$

The second equality follows from the fact that

$${\mathbf{\text{B}}}_1^{\top }\left({\tilde{\gamma }}^{(1)}-{\overline{\gamma }}^{(1)}\right)=0+{\text{Ξ}}_{12}.$$

Thus, the asymptotic variance of $n^{\frac{1}{2}}\left({\mathit{e}}^{\top }{\widehat{\mathit{\theta }}}_{\text{intri}}-{\mathit{e}}^{\top }\overline{\mathit{\theta }}\right)\text{is E}\left[R{\left({\mathit{e}}^{\top }{\mathbf{A}}^{-1}\mathbf{x}\right)}^2{\left\{y-g\left({\overline{\gamma }}^{(1)\top }\mathbf{\text{Φ}}\right)\right\}}^2\right]$, which is minimized among those of $\left\{{\mathit{e}}^{\top }\widehat{\mathcal{W}}(\gamma ):\text{E}\left[\mathbf{x}\left\{y-g\left({\gamma }^{\top }\mathbf{\text{Φ}}\right)\right\}\right]=\mathbf{0}\right\}$. From Theorem $n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}-\overline{\mathit{\theta }}\right)$ is asymptotically equivalent with $\mathcal{W}(\overline{\gamma })$. Therefore, when the imputation model is correctly specified, that is, there exists γ₀ such that $\text{P}(y=1\mid \mathbf{u})=g\left({\mathbf{\text{Φ}}}^{\top }{\gamma }_0\right),\overline{\gamma }={\overline{\gamma }}^{(1)}={\gamma }_0$, it follows that $n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{\text{intri}}-\overline{\mathit{\theta }}\right)$ is asymptotically equivalent to $n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{\text{SSL}}-\overline{\mathit{\theta }}\right)$. This completes the proof of Theorem 3.

Using our previous arguments, we next establish Theorem A1. Similar to (G.3), we expand

$$n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{\text{intri}}^D-\overline{\mathit{\theta }}\right)\text{as}$$

$$\begin{gathered} n^{\frac{1}{2}}\left({\widehat{\mathit{\theta }}}_{\text{intri}}^D-\overline{\mathit{\theta }}\right)=n^{\frac{1}{2}}{\mathbf{A}}^{-1}\left[N^{-1}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\overline{\gamma }}^{(2)\top }{\mathbf{\text{Ψ}}}_{{\tilde{\mathit{\theta }}}_i}\right)-g\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right\}+{\mathbf{B}}_1^{\top }\left({\tilde{\gamma }}_{\tilde{\mathit{\theta }}}^{(2)}-{\overline{\gamma }}^{(2)}\right)\right]+o_p(1), \\ =n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right){\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\text{Ψ}}}}_i\right)\right\}\right] \\ +n^{\frac{1}{2}}{\mathbf{A}}^{-1}\left[N^{-1}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\overline{\gamma }}^{(2)\top }{\text{Ψ}}_{\tilde{\mathit{\theta }}i}\right)-g\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\Psi }}}_i\right)\right\}+{\dot{\mathbf{\Xi }}}_{\mathit{\theta },22}^{\top }(\tilde{\mathit{\theta }}-\overline{\mathit{\theta }})\right]+o_p(1) \\ =n^{\frac{1}{2}}\sum _{s=1}^S{\rho }_s\left[n_s^{-1}\sum _{i=1}^{n}I\left({\mathcal{S}}_i=s\right){\mathbf{A}}^{-1}{\mathbf{x}}_i\left\{y_i-g\left({\overline{\gamma }}^{(2)\text{T}}{\overline{\mathbf{\text{Ψ}}}}_i\right)\right\}\right]+o_p(1), \end{gathered}$$

The third equality follows from the fact that $n^{\frac{1}{2}}(\tilde{\mathit{\theta }}-\overline{\mathit{\theta }})=O_p(1)$ and

$$\partial \left(N^{-1}\sum _{i=1}^N{\mathbf{x}}_i\left\{g\left({\overline{\gamma }}^{(2)\top }{\text{Ψ}}_{{\tilde{\mathit{\theta }}}_i}\right)-g\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\text{Ψ}}}}_i\right)\right\}\right)/\partial \mathit{\theta }+{\dot{\text{Ξ}}}_{\mathit{\theta },22}=o_p(1).$$

Using this result, and applying similar arguments as those used for ${\tilde{\gamma }}_{\tilde{\mathit{\theta }}}^{(2)}$, we have that

$$\begin{gathered} {\tilde{\gamma }}_{{\widehat{\mathit{\theta }}}_{\text{intri}}^D}^{(2)}-{\overline{\gamma }}^{(2)}=\left[{\mathbf{A}}_2^{-1}-{\mathbf{A}}_2^{-1}{\mathbf{B}}_2{\left({\mathbf{B}}_2^{\top }{\mathbf{A}}_2^{-1}{\mathbf{B}}_2\right)}^{-1}{\mathbf{B}}_2^{\top }{\mathbf{A}}_2^{-1}\right]{\mathbf{\Xi }}_{21}^{\prime }+{\mathbf{A}}_2^{-1}{\mathbf{B}}_2{\left({\mathbf{B}}_2^{\top }{\mathbf{A}}_2^{-1}{\mathbf{B}}_2\right)}^{-1}{\mathbf{\Xi }}_{22}^{\prime }=O_p\left(n^{-\frac{1}{2}}\right), \\ \text{where}{\text{Ξ}}_{21}^{\prime }=\frac{1}{n}\sum _{i=1}^n{\zeta }_i{\left\{1-2\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)+{\dot{\mathbf{D}}}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\right\}}^2\dot{g}\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\text{Ψ}}}}_i\right){\overline{\mathbf{\text{Ψ}}}}_i\left\{y_i-g\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\text{Ψ}}}}_i\right)\right\}+{\dot{\text{Ξ}}}_{\mathit{\theta },21}^{\top }\left({\widehat{\mathit{\theta }}}_{\text{intri}}^D-\overline{\mathit{\theta }}\right); \\ {\mathbf{\Xi }}_{22}^{\prime }=\frac{1}{N}\sum _{i=1}^N{w}_i{\left[{\mathbf{x}}_i^{\top },\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }{\mathbf{x}}_i\right)\right]}^{\top }\left\{y_i-g\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\text{Ψ}}}}_i\right)\right\}+{\dot{\text{Ξ}}}_{\mathit{\theta },22}^{\top }\left({\widehat{\mathit{\theta }}}_{\text{intri}}^D-\overline{\mathit{\theta }}\right). \end{gathered}$$

We then follow the same procedure as in Appendix F (specifically, noting that ${\tilde{\gamma }}_{{\widehat{\mathit{\theta }}}_{\text{intri}}^D}^{(2)}$ corresponds to the ${\widehat{\mathit{\theta }}}_{\text{intri}}^D$ plugged into ${\widehat{D}}_{\text{intri}}(\mathit{\theta })$, the derivation for the augmentation approach in Section 3.2 can be used directly) to derive that ${\widehat{D}}_{\text{intri}}\stackrel{p}{\to }\overline{D}$ and

$$n^{\frac{1}{2}}\left({\widehat{D}}_{\text{intri}}-\overline{D}\right)=n^{\frac{1}{2}}\left[N^{-1}\sum _{i=1}^N\left\{1-2{\overline{\mathcal{Y}}}_i+{\overline{\mathbf{D}}}^{\top }{\mathbf{A}}^{-1}{\mathbf{x}}_i\right\}\left\{y_i-g\left({\overline{\gamma }}^{(2)\top }{\overline{\mathbf{\text{Ψ}}}}_i\right)\right\}\right]+o_p(1)=\widehat{\mathcal{T}}\left({\overline{\gamma }}_{\overline{\theta }}^{(2)}\right)+o_p(1).$$

By the definition of ${\overline{\gamma }}_{\overline{\mathit{\theta }}}^{(2)}$, the asymptotic variance of $\widehat{\mathcal{T}}\left({\overline{\gamma }}_{\overline{\mathit{\theta }}}^{(2)}\right)$ is minimized among those of $\left\{\widehat{\mathcal{T}}(\gamma ):\text{E}{\left[{\mathbf{x}}^{\top },\mathcal{Y}\left({\overline{\mathit{\theta }}}^{\top }\mathbf{x}\right)\right]}^{\top }\left\{y-g\left({\gamma }^{\top }\overline{\mathbf{\Psi }}\right)\right\}=\mathbf{0}\right\}$. Additionally, we may use a similar procedure as that in Appendix F to derive that

$$n^{\frac{1}{2}}\left({\widehat{D}}_{\text{SSL}}-\overline{D}\right)=\widehat{\mathcal{T}}\left({\overline{\gamma }}_{\overline{\mathit{\theta }}}\right)+o_p(1).$$

Thus, when the imputation model for estimating D, i.e. $\text{P}(y=1\mid \mathbf{u})=g\left({\gamma }^{\top }\overline{\mathbf{\Psi }}\right)$ is correct, we have ${\overline{\gamma }}_{\overline{\mathit{\theta }}}={\overline{\gamma }}_{\overline{\mathit{\theta }}}^{(2)}$ and that $n^{\frac{1}{2}}\left({\widehat{D}}_{\text{intri}}-\overline{D}\right)$ is asymptotically equivalent to $n^{\frac{1}{2}}\left({\widehat{D}}_{\text{SSL}}-\overline{D}\right)$. These arguments establish Theorem A1.

H. Justification for Weighted CV Procedure

To provide a heuristic justification for the weights for our ensemble CV method, consider an arbitrary smooth loss function d(·,·) and let $\mathcal{D}(\mathit{\theta })=\text{E}\left[d\left\{y_0,\mathcal{Y}\left({\mathit{\theta }}^{\top }{\mathbf{x}}_0\right)\right\}\right]$. Let $\widehat{\mathcal{D}}(\mathit{\theta })$ denote the empirical unbiased estimate of $\mathcal{D}(\mathit{\theta })$ and suppose that $\widehat{\mathit{\theta }}$ minimizes $\widehat{\mathcal{D}}(\mathit{\theta })$ (i.e. $\dot{\widehat{\mathcal{D}}}(\widehat{\mathit{\theta }})=\mathbf{0}$). Suppose that $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-\overline{\mathit{\theta }})\to N(0,\mathbf{\Sigma })$ in distribution. Then by a Taylor series expansion of $\widehat{\mathcal{D}}(\overline{\mathit{\theta }})\text{at}\widehat{\mathit{\theta }}$,

$$\widehat{\mathcal{D}}(\widehat{\mathit{\theta }})=\widehat{\mathcal{D}}(\overline{\mathit{\theta }})-\frac{1}{2}{(\widehat{\mathit{\theta }}-\overline{\mathit{\theta }})}^{\top }\ddot{\widehat{\mathcal{D}}}(\widehat{\mathit{\theta }})(\widehat{\mathit{\theta }}-\overline{\mathit{\theta }})+o_p\left(n^{-1}\right)\text{and}$$

$$\text{E}\{\widehat{\mathcal{D}}(\widehat{\mathit{\theta }})\}=\mathcal{D}(\overline{\mathit{\theta }})-\frac{1}{2}{n}^{-1}\text{Tr}\{\ddot{\mathcal{D}}(\overline{\mathit{\theta }})\mathbf{\Sigma }\}+o_p\left(n^{-1}\right)$$

where $\ddot{\mathcal{D}}(\overline{\mathit{\theta }})=\partial \mathcal{D}(\mathit{\theta })/\partial \mathit{\theta }\partial {\mathit{\theta }}^{\top }$. For the K-fold CV estimator, ${\widehat{\mathcal{D}}}_{cv}=K^{-1}{\sum }_{k=1}^K{\widehat{\mathcal{D}}}_k\left({\widehat{\mathit{\theta }}}_{(-k)}\right)$, we note that since ${\widehat{\mathcal{D}}}_k(\mathit{\theta })$ is independent of ${\widehat{\mathit{\theta }}}_{(-k)}$

$$\begin{gathered} \text{E}\left({\widehat{\mathcal{D}}}_{cv}\right)=\mathcal{D}(\overline{\mathit{\theta }})+K^{-1}\sum _{k=1}^K\text{E}\left\{{\dot{\widehat{\mathcal{D}}}}_k(\overline{\mathit{\theta }})\right\}\text{E}\left({\widehat{\mathit{\theta }}}_{(-k)}-\overline{\mathit{\theta }}\right)+\frac{1}{2}\frac{K}{K-1}{n}^{-1}\text{Tr}\{\ddot{\mathbf{D}}(\overline{\mathit{\theta }})\text{Σ}\}+o_p\left(n^{-1}\right) \\ =\mathcal{D}(\overline{\mathit{\theta }})+\frac{1}{2}\frac{K}{K-1}{n}^{-1}\text{Tr}\{\ddot{\mathbf{D}}(\overline{\mathit{\theta }})\mathbf{\Sigma }\}+o_p\left(n^{-1}\right), \end{gathered}$$

where the second equality follows from the fact that $\text{E}\left\{{\dot{\widehat{\mathcal{D}}}}_k(\overline{\mathit{\theta }})\right\}=\dot{\mathcal{D}}(\overline{\mathit{\theta }})=\mathbf{0}$ when $\overline{\mathit{\theta }}$ minimizes $\mathcal{D}(\mathit{\theta })$. Letting ${\widehat{\mathcal{D}}}_{\omega }=\omega \widehat{\mathcal{D}}(\widehat{\mathit{\theta }})+(1-\omega ){\widehat{\mathcal{D}}}_{cv}$ with $\omega =K/(2K-1)$, it follows that $\omega n^{-1}-(1-\omega )K{n}^{-1}/(K-1)=0$ and thus

$$\text{E}\left({\widehat{\mathcal{D}}}_{\omega }\right)=\mathcal{D}(\overline{\mathit{\theta }})+o_p\left(n^{-1}\right).$$