Handling high-dimensional data with missing values by modern machine learning techniques

Abstract

High-dimensional data have been regarded as one of the most important types of big data in practice. It happens frequently in practice including genetic study, financial study, and geographical study. Missing data in high dimensional data analysis should be handled properly to reduce nonresponse bias. We discuss some modern machine learning techniques including penalized regression approaches, tree-based approaches, and deep learning (DL) for handling missing data with high dimensionality. Specifically, our proposed methods can be used for estimating general parameters of interest including population means and percentiles with imputation-based estimators, propensity score estimators, and doubly robust estimators. We compare those methods through some limited simulation studies and a real application. Both simulation studies and real application show the benefits of DL and XGboost approaches compared with other methods in terms of balancing bias and variance.

1. Introduction

Missing data are a critical problem in practical research including sample surveys, epidemiology, economics, and social science. Simply ignoring missing data in statistical analysis may lead to biased results, see Refs. [34,41]. There are two types of missingness in practice: item nonresponse and unit nonresponse. Item nonresponse is often handled by imputation approaches including hot-deck imputation [1,48], nearest neighbor imputation [8,9,63], predictive mean matching (PMM) imputation [28,40,51,64], multiple imputation (MI) [52–54], fractional imputation [31,32,62], among others. For a comprehensive review in dealing with item nonresponse, see Ref. [12]. Unit nonresponse is often handled by using inverse probability weighting techniques, see Refs. [27,29,33] among others. The validity of the above methods depend on the underlying outcome regression model and nonresponse model assumptions. Doubly robust approaches [2,30,50] and multiply robust approaches [10,11,25] have been proposed to improve the robustness to model misspecification.

Big data are becoming one of the hottest topics in current research of computer science, data mining, engineering, and applied mathematics. High-dimensional data have been regarded as one of the most important types of big data, see Ref. [23]. With the rapid development of information technology, high-dimensional data are common in modern research areas including genetic study [18,56,57], financial study [42,49], and health care study [37,44]. Currently, there are many supervised learning approaches for handling high-dimensional data. They can be classified mainly into three categories: (i) penalized regression methods such as least absolute shrinkage and selection operator (LASSO) [59], ridge regression (RR) [55], and smoothly clipped absolute deviation (SCAD) [20]. (ii) Tree-based approaches including decision or regression tree (RT) [38,39], random forests (RFs) [46,61], and boosting methods [15,16,43]. (iii) Deep learning (DL) approaches [24,36]. Unsupervised learning including clustering methods have also been developed in the context of high-dimensional data, see Refs. [21,35] among others.

Even though there are numerous articles on missing data analysis with low dimensional data and machine learning approaches with high-dimensional data, the research for machine learning-based approaches for handling high dimensional data with missing values is relatively new. Penalized regression methods have been used for imputation, see Refs. [45,65,66]. Tree-based imputation approaches have been discussed in Refs. [7,58,67]. DL-based methods for missing data imputation have been shown to perform well for high-dimensional medical [3,17] and genetic data [47]. In the previous literature, a systematic discussion and an empirical comparison of handling missing data with high dimensionality by using machine learning approaches are lacking. For example, there is the lack of the comparison among statistical learning approaches, tree-based approaches, traditional imputation methods, and DL approaches. To fill this research gap, we compare these approaches through an extensive simulation study as well as a real application. In addition, we also consider inverse probability weighted estimators and doubly robust estimators. We discuss DL-based approaches for handling missing data in the paper. For simplicity, we only focus on estimating the population mean for the scenario where only the outcome variable is missing. Our proposed methods can be naturally extended for estimating other parameters such as population percentiles, regression coefficients, and distribution functions, where both the outcome variable and the covariate variables are prone to missing values.

The paper is organized as follows. Section 2 introduces the mathematical notation and the problem. Penalized approaches are discussed in Section 3. Tree-based approaches including decision tree, RF, and boosting methods are considered in Section 4. Section 5 contains DL-based approaches. A simulation study is presented in Section 6. A real application is given in Section 7. Section 8 contains some discussions.

2. Basic setups

Suppose we have independently identically distributed ( i.i.d.) copies of $({\mathbf{X}}_i,Y_i,R_i),i=1,2,\dots ,n,$ where ${\mathbf{X}}_i=(X_{1i},X_{2i},\dots ,X_{di})$ is a d dimensional covariate vector, $Y_i$ is the study variable subject to missingness, and $R_i$ is the response indicator such that $R_i=1$ if $Y_i$ is observed and 0 otherwise. Denote by S the sample, $S_r$ the set of respondents, and $S_m$ the set of non-respondents. The vector ${\mathbf{X}}_i$ is assumed to be fully observed and the dimension d is not fixed and increases as n increases. We assume that the outcome regression model is

$$Y_i=m({\mathbf{X}}_i;{\beta }_0)+{ϵ}_i,i=1,2,\dots ,n,$$ (1)

where $m({\mathbf{X}}_i;\beta )$ is assumed to be an unknown function with unknown parameter ${\beta }_0=({\beta }_{0,1},{\beta }_{0,2},\dots ,{\beta }_{0,a_1}{)}^{\mathrm{\top }}$ and the ${ϵ}_i$'s are assumed to be independent errors with $E({ϵ}_i|{\mathbf{X}}_i)=0$ and $V({ϵ}_i|{\mathbf{X}}_i)={\sigma }^2$. Our proposed method in the next section can be naturally extended to the heteroskedasticity scenario. Suppose there are only $q_1$ non-zero parameters ${\beta }_0^{(1)}=({\beta }_{0,k_1},{\beta }_{0,k_2},\dots ,{\beta }_{0,k_{q_1}})$ out of ${\beta }_0$, where $(k_1,k_2,\dots ,k_{q_1})$ is a subset of $(1,2,\dots ,a_1)$ with $q_1$ being a fixed number. Such condition is the so-called sparsity condition. The missing mechanism is assumed to be missing at random and the response indicators are assumed to have a Bernoulli distribution, $R_i\sim \mathrm{Bern}({\pi }_i)$, with

$$\eta ({\pi }_i)=Pr(R_i=1|{\mathbf{X}}_i)=\varphi ({\mathbf{X}}_i;{\alpha }_0),$$ (2)

where η is the link function and $\varphi ({\mathbf{X}}_i;{\alpha }_0)$ is an unknown function with unknown parameter ${\alpha }_0=({\alpha }_{0,1},{\alpha }_{0,2},\dots ,{\alpha }_{0,a_2}).$ Suppose there are only $q_2$ non-zero parameters ${\alpha }_0^{(1)}=({\alpha }_{0,l_1},{\alpha }_{0,l_2},\dots ,{\alpha }_{0,l_{q_2}})$ out of ${\alpha }_0$, where $(l_1,l_2,\dots ,l_{q_2})$ is a subset of $(1,2,\dots ,a_2)$ with $q_2$ as a fixed number. For simplicity, suppose the parameters of interest ${\theta }_0$ can be defined as the solution of the estimating equation $E\{U(\mathbf{X},Y;{\theta }_0)\}=0$, where $U(\mathbf{X},Y;\theta )$ is assumed to be known and can be either a smooth or non-smooth function of $\theta$. Without loss of generality, we assume the solution of the previous estimating equation is unique and that the function $U(\mathbf{X},Y;\theta )$ satisfies the regularity conditions described in Ref. [14]. Special cases of ${\theta }_0$ include population means with $U(\mathbf{X},Y;{\theta }_0)=Y-{\theta }_0$, regression coefficients with $U(\mathbf{X},Y;{\theta }_0)=\mathbf{X}(Y-{\mathbf{X}}^{\mathrm{\top }}{\theta }_0)$, and population τ-th percentile with $U(\mathbf{X},Y;{\theta }_0)=I(Y\le {\theta }_0)-\tau$. For more information for estimating parameters defined as the solution of an estimating equation, see Ref. [60].

3. Penalized approaches

Under the working model assumption that $E(Y|X)=\tilde{m}(\mathbf{X};\beta )$, where $\beta =({\beta }_1,{\beta }_2,\dots ,{\beta }_{a_1}{)}^{\mathrm{\top }}$, the estimator $\hat{\beta }$ of ${\beta }_0$ can be obtained by minimizing the following penalized distance function:

$$Q_1(\beta )=\sum _{i\in S_r}{\{Y_i-\tilde{m}({\mathbf{X}}_i;\beta )\}}^2+p_{\lambda }(\beta ),$$ (3)

where $p_{\lambda }(\beta )$ is the penalty term. Popular examples include:

• RR: $p_{\lambda }(\beta )=\lambda \sum _{j=1}^{a_1}{\beta }_j^2$.

• Hard thresholding: $p_{\lambda }(\beta )=\sum _{j=1}^{a_1}\{{\lambda }^2-(|{\beta }_j|-\lambda {)}^{2}I(|{\beta }_j|>\lambda )\}$.

• LASSO (soft thresholding): $p_{\lambda }(\beta )=\lambda \sum _{j=1}^{a_1}|{\beta }_j|$.

• SCAD: $p_{\lambda }^{\prime }(\beta )=\sum _{j=1}^{a_1}\lambda \{I(|{\beta }_j|\le \lambda )+\frac{(a\lambda -|{\beta }_j|{)}_{+}}{(a-1)\lambda }I(|{\beta }_j|>\lambda )\}$.

Then, the penalized deterministic imputed estimator for the population mean ${\theta }_0=E(Y)$ can be written as

$${\hat{\theta }}_{PDI}=\frac{1}{n}\sum _{i=1}^n\{R_i{Y}_i+(1-R_i)\tilde{m}({\mathbf{X}}_i;\hat{\beta })\}.$$ (4)

To estimate general parameters ${\theta }_0$ defined as the solution of an estimating equation $E\{U(\mathbf{X},Y;{\theta }_0)\}=0$, we propose using the following stochastic imputation approach. One can first create a pool of residuals ${\mathcal{V}}_R=\{{\widehat{ϵ}}_i,i\in S_R\}$, where ${\widehat{ϵ}}_i=Y_i-\tilde{m}({\mathbf{X}}_i;\hat{\beta }).$ Then, generate the imputed values $Y_i^{\ast }=\tilde{m}({\mathbf{X}}_i;\hat{\beta })+{ϵ}_i^{\ast }$ for $i\in S_M$ with ${ϵ}_i^{\ast }$ selected from ${\mathcal{V}}_R$ randomly with equal selection probabilities. The estimator ${\hat{\theta }}_{PSI}$ of ${\theta }_0$ can be obtained by solving the following estimating equation:

$$\frac{1}{n}\sum _{i=1}^n\{R_{i}U({\mathbf{X}}_i,Y_i;\theta )+(1-R_i)U({\mathbf{X}}_i,Y_i^{\ast };\theta )\}=0.$$ (5)

The performance of the penalized imputation approach depends on the assumption of the working model $E(Y|\mathbf{X})=\tilde{m}(\mathbf{X};\beta )$. If the assumption violated, the estimators may be biased.

Alternatively, one can use a penalized approach for estimating the propensity score model $\pi ({\mathbf{X}}_i;{\alpha }_0)$. Specifically, suppose that the working model for the propensity score is $\tilde{\pi }({\mathbf{X}}_i;{\alpha }_0)$. Then, the estimator ${\hat{\alpha }}_p$ can be obtained by minimizing the following penalized distance function:

$$Q_2(\alpha )=-\sum _{i=1}^n[R_i\log \{\tilde{\pi }({\mathbf{X}}_i;\alpha )\}+(1-R_i)\log \{1-\tilde{\pi }({\mathbf{X}}_i;\alpha )\}]+q_{\lambda }(\alpha ),$$ (6)

where $\tilde{\pi }({\mathbf{X}}_i;\alpha )={\tilde{\eta }}^{-1}\{\tilde{\varphi }({\mathbf{X}}_i;\alpha )\}$ and $q_{\lambda }(\alpha )$ is the penalty term. The options of penalty terms have been discussed above. Then the propensity score-based estimator ${\hat{\theta }}_P$ can be obtained by solving the following estimating equation:

$$\sum _{i=1}^n\frac{R_i}{\tilde{\pi }({\mathbf{X}}_i;{\hat{\alpha }}_p)}U({\mathbf{X}}_i,Y_i;\theta )=0.$$ (7)

The performance of the propensity score estimator ${\hat{\theta }}_P$ depends on the assumption that $E(R|\mathbf{X})=\tilde{\pi }(\mathbf{X};{\alpha }_0)$. To improve the robustness, we consider a random hot-deck imputation procedure within classes [4,13] that can be described as follows:

1. Obtain the estimators $\tilde{m}(\mathbf{X};\hat{\beta })$ and $\tilde{\pi }(\mathbf{X};\hat{\alpha })$ by using the penalized methods described previously.

2. The sample is partitioned into C imputation classes using an equal quantile method that consists of ordering the $\tilde{m}({\mathbf{X}}_i;\hat{\beta })$-values, $i\in S,$ from the lowest to the largest and then forming C classes of (approximately) equal size.

3. A missing value in a given class is replaced by the value of a respondent belonging to the same class and chosen at random with probability proportional to ${\psi }_i={\tilde{\pi }}^{-1}({\mathbf{X}}_i;\hat{\alpha })-1$. Denote the imputed value as $y_i^{\ast }$ for $i\in S_m$, then the resulting doubly robust estimator ${\hat{\theta }}_{DR}$ of ${\theta }_0$ can be obtained by solving the following estimating equation:

   $$\sum _{i=1}^n\{R_{i}U({\mathbf{X}}_i,Y_i;\theta )+(1-R_i)U({\mathbf{X}}_i,Y_i^{\ast };\theta )\}=0.$$ (8)

The estimator ${\hat{\theta }}_{DR}$ is doubly robust in the sense that it is consistent if one of the two working models $\tilde{m}(\mathbf{X};{\beta }_0)$ and $\tilde{\pi }(\mathbf{X};{\alpha }_0)$ is correctly specified.

Remark 3.1

For estimating population mean ${\theta }_0=E(Y)$, one can use the following deterministic doubly robust estimator as well:

$${\hat{\theta }}_{DDR}=\frac{1}{n}\sum _{i=1}^n\frac{R_i}{\tilde{\pi }({\mathbf{X}}_i;{\hat{\alpha }}_p)}{Y}_i+\frac{1}{n}\sum _{i=1}^n\{1-\frac{R_i}{\tilde{\pi }({\mathbf{X}}_i;{\hat{\alpha }}_p)}\}\tilde{m}({\mathbf{X}}_i;\hat{\beta }).$$ (9)

4. Tree-based approaches

4.1. RT

For simplicity, we only consider the mid-point as the split point for tree construction in this section. Suppose that we have a partition of the support of $\mathbf{X}=(X_1,X_2,\dots ,X_d)$ into L regions, $D_1,D_2,\dots ,D_L$, then the RT-based imputed value can be written as:

$$Y^{\ast }(\mathbf{X};{\mathcal{W}}_L)=\sum _{l=1}^L{\hat{c}}_{l}I(\mathbf{X}\in D_l)=\sum _{l=1}^L{\hat{c}}_l{B}_l(\mathbf{X}),$$ (10)

where ${\hat{c}}_l$ is the sample mean of $Y_i$ for the respondents with $i\in D_l$ and $B_l(\mathbf{X})=I(\mathbf{X}\in D_l)$. Denote ${\mathcal{W}}_L=\{B_1(\mathbf{X}),B_2(\mathbf{X}),\dots ,B_L(\mathbf{X})\}$ as the corresponding dictionary (set of basis functions). In order to find the best partition, one can apply the following algorithm:

• (Step 1).

  Let ${\mathcal{W}}_t$ be the current dictionary (set of basis functions) with ${\mathcal{W}}_t=\{B_1(\mathbf{X}),B_2(\mathbf{X}),\dots ,B_t(\mathbf{X})\}$ where $B_t(\mathbf{X})=I(\mathbf{X}\in D_t)$ is the tth basis function, and $\{D_1,D_2,\dots ,D_t\}$ is the current partition of the support of $\mathbf{X}$. Denote ${\mathbf{W}}_0=\{1\}$.

• (Step 2).

  Expand the basis function by multiplying an additional split using $X_k$ to each of the basis functions. Thus, we have $t\times d$ possible candidates: ${\mathcal{W}}_{t+1}^{(i,k)}=\{B_1(\mathbf{X}),B_2(\mathbf{X}),\dots ,B_i^{(k)}(\mathbf{X}),\dots ,B_t(\mathbf{X}),B_{t+1}^{(i,k)}(\mathbf{X})\}$, where $B_i^{(k)}(\mathbf{X})$ and $B_{t+1}^{(i,k)}(\mathbf{X})$ are obtained by applying additional split on $B_i(\mathbf{X})$ using $X_k$.

• (Step 3).

  Among the $t\times d$ possible dictionaries, choose the best one ${\mathcal{W}}_{t+1}$ by finding the minimizer of the loss function: $\sum _{i\in S_r}\{Y_i-Y^{\ast }({\mathbf{X}}_i;{\mathcal{W}}_{t+1}^{(i,k)}){\}}^2$.

• (Step 4).

  If the Bayesian information criterion (BIC) by using ${\mathcal{W}}_t$ is smaller than the BIC by using ${\mathcal{W}}_{t+1}$, then stop splitting and the final model is based on ${\mathcal{W}}_t$.

After the above tree selection algorithm, the imputed estimator for ${\theta }_0$ can be obtained by solving (5) with $Y_i^{\ast }=\tilde{m}({\mathbf{X}}_i)+{ϵ}_i^{\ast }$ for $i\in S_M$ with ${ϵ}_i^{\ast }$ selected from ${\mathcal{V}}_R$ randomly with equal selection probabilities, where $\tilde{m}({\mathbf{X}}_i)=\sum _{t=1}^T{\hat{c}}_t{B}_t({\mathbf{X}}_i)$ with ${\mathcal{W}}_T=\{B_1(\mathbf{X}),B_2(\mathbf{X}),\dots ,B_T(\mathbf{X})\}$ denotes the final selected tree. ${\mathcal{V}}_R=\{{\widehat{ϵ}}_i,i\in S_R\}$, where ${\widehat{ϵ}}_i=Y_i-\tilde{m}({\mathbf{X}}_i).$ Similarly, the propensity score model can also be estimated by using RT method. The propensity score-based estimator and doubly robust estimator can be obtained by using similar techniques in Section 3.

4.2. RF

To improve the prediction precision, Breiman [6] proposed using RF with multiple trees. The algorithm for imputation can be summarized as follows:

• (Step 1).

  Draw B bootstrap samples $S_r^{(b)},b=1,2,\dots ,B$, from the original respondents $S_r$ by using simple random sampling design with replacement.

• (Step 2).

  For each bootstrap sample of respondents $S_r^{(b)},b=1,2,\dots ,B$, select $d^{\ast }$ predictors from d predictors $X_1,X_2,\dots ,X_d$ randomly for modeling.

• (Step 3).

  For each bootstrap sample $S_r^{(b)},b=1,2,\dots ,B$, obtain the tree model ${\mathcal{W}}_{T^{(b)}}^{(b)}=\{B_1^{(b)}({\mathbf{X}}^{(b)}),B_2^{(b)}({\mathbf{X}}^{(b)}),\dots ,B_{T^{(b)}}^{(b)}({\mathbf{X}}^{(b)})\}$ by using the algorithm described in Section 4.1.

• (Step 4).

  Obtain the final estimator of $m(\mathbf{X};{\beta }_0)$ by using $\hat{m}(\mathbf{X})=B^{-1}\sum _{b=1}^B{\tilde{m}}^{(b)}({\mathbf{X}}^{(b)})=B^{-1}\sum _{b=1}^B\sum _{t=1}^{T^{(b)}}{\hat{c}}_t^{(b)}{B}_t^{(b)}({\mathbf{X}}^{(b)})$.

Similarly, the propensity score model can also be estimated by using the RF method. The corresponding imputed estimator, propensity score estimator, and doubly robust estimator of ${\theta }_0$ can be obtained by using similar techniques as that in Section 3.

4.3. Boosting

Boosting [22] is another procedure which can be used to any statistical learning approaches to improve the precision of model predictions and has been used frequently in tree-based methods. Boosting is an iterative method that starts with a weak learner and improves the prediction at each sequential step by predicting the residuals of prior models and combining them together to make the final prediction. The idea for boosting is very similar to that of additive modeling, see Refs. [22,26]. Suppose that the prediction model has the following additive form:

$$m(\mathbf{X})=\sum _{q=1}^{Q}T(\mathbf{X};{\eta }_q),$$ (11)

where $T(\mathbf{X};{\eta }_q),q=1,2,\dots ,Q$, are Q different tree models with parameters ${\eta }_q$. They will be determined iteratively by using a forward stagewise procedure. For the quadratic loss function, the algorithm is as follows:

• (Step 1).

  Initialize ${\hat{m}}^{(0)}(\mathbf{X})$ with an offset value. The default value is ${\hat{m}}^{(0)}(\mathbf{X})={\overline{Y}}_r$, where ${\overline{Y}}_r=n_r^{-1}\sum _{i\in S_r}{Y}_i$.

• (Step 2).

  Increase q by 1 and compute the residuals ${\widehat{ϵ}}_i^{(q-1)}=Y_i-{\hat{m}}^{(q-1)}({\mathbf{X}}_i)$ for $i\in S_r$.

• (Step 3).

  Fit the residual vector ${\widehat{ϵ}}_i^{(q-1)}$ to ${\mathbf{X}}_i$ for $i\in S_r$ by using the RT method described in Section 4.1. Denote the fitted value as ${\hat{g}}^{(q)}({\mathbf{X}}_i)$.

• (Step 4).

  Obtain the updated fitted value ${\hat{m}}^{(q)}({\mathbf{X}}_i)={\hat{m}}^{(q-1)}({\mathbf{X}}_i)+\tau {\hat{g}}^{(q)}({\mathbf{X}}_i)$, with $0<\tau \le 1$ as a step-length factor.

• (Step 5).

  Iterate (step 2) to ( step 4) until some stopping criterias have been reached. For instance, one can consider using cross-validation or some information criterion.

Similarly, the propensity score model can also be estimated through the boosting method. The corresponding imputed estimator, propensity score estimator, and doubly robust estimator of ${\theta }_0$ can be obtained by using similar techniques than those described in Section 3.

5. DL approaches

DL has been regarded as the optimal strategy for handling hierarchical non-linear data structure. Suppose we want to use the following deep neural networks working model

$${\tilde{m}}^{(L)}(\mathbf{X})=g^{(L)}\circ g^{(L-1)}\circ \dots \circ g^{(1)}(\mathbf{X}),$$ (12)

where denotes the composition of two functions and L is the number of hidden layers, and is usually called depth of a neural networks model. Define $m^{(0)}=\mathbf{X}$, and we consider the multilayer perceptrons with the following choice of $g^{(l)}$ for $l=1,2,\dots ,L$,

$${\tilde{m}}^{(l)}=g^{(l)}({\tilde{m}}^{(l-1)})={\sigma }^{(l)}({\mathbf{W}}^{(l)}{\tilde{m}}^{(l-1)}+{\mathbf{b}}^{(l)}),$$

where ${\mathbf{W}}^{(l)}$ and ${\mathbf{b}}^{(l)}$ are the weight matrix and the bias/intercept, respectively, associated with the lth layer, and ${\sigma }^{(l)}$ is the so-called activation function for the lth layer. Popular choices of activation function include the rectified linear unit (ReLU) function: $\sigma \mathbf{(}\mathbf{z}\mathbf{)}=max\{\mathbf{z},0\}$ and scaled exponential linear unit (SELU) function $\lambda z$ if z>0, or $\lambda \alpha (e^x-1)$ uf $z\le 0$. Define $\mathbf{W}=({\mathbf{W}}^{(1)},{\mathbf{W}}^{(2)},\dots ,{\mathbf{W}}^{(L)})$, $\mathbf{b}=({\mathbf{b}}^{(1)},{\mathbf{b}}^{(2)},\dots ,{\mathbf{b}}^{(L)})$, and $\tilde{m}(\mathbf{X};\mathbf{W},\mathbf{b})={\tilde{m}}^{(L)}(\mathbf{X})$. Then the estimators $\hat{\mathbf{W}}$ and $\hat{\mathbf{b}}$ of the parameters $\mathbf{W}$ and $\mathbf{b}$ can be obtained by minimizing the following loss function:

$$\mathrm{L}(\mathbf{W},\mathbf{b})=\frac{1}{n_r}\sum _{i\in S_r}\mathrm{loss}(\tilde{m}({\mathbf{X}}_i;\mathbf{W},\mathbf{b}),Y_i),$$ (13)

where $\mathrm{loss}(\mathbf{x},y)$ is a given continuous function such as $L_p$ norm $\mathrm{loss}(\mathbf{x},y)=||\mathbf{x}-y|{|}^p$ for p = 1, 2. The optimization can be generally done numerically via stochastic gradient descent [5] and backpropagation [36], whilst guaranteeing reliable performance on unseen data (ensuring generalization). Then the DL-based imputed estimator of ${\theta }_0=E(Y)$ can be written as

$${\hat{\theta }}_{DI}=\frac{1}{n}\sum _{i=1}^n\{R_i{Y}_i+(1-R_i\tilde{m}({\mathbf{X}}_i;\hat{\mathbf{W}},\hat{\mathbf{b}}))\}.$$ (14)

By using the logit function as the final activation function in (12) and the logistic regression entropy function as the loss function in (13), one can obtain the propensity score-based estimator as follows:

$${\hat{\theta }}_{DP}=\frac{1}{n}\sum _{i=1}^n\frac{R_i}{\tilde{\pi }({\mathbf{X}}_i;\hat{\mathbf{M}},\hat{\mathbf{a}})}{Y}_i,$$ (15)

where $\hat{\mathbf{M}}$ and $\hat{\mathbf{a}}$ are the estimators of $\mathbf{M}$ and $\mathbf{a}$ which are corresponding weight matrix and the bias/intercept for predicting the nonresponse probability. The estimation of other parameters and doubly robust estimator can be developed by using similar techniques as those described in Section 3.

6. Simulation study

For each iteration, we generated a data set of n = 600 i.i.d. copies $\{{\mathit{X}}_i,Y_i,R_i\},i=1,2,\dots ,n$, where ${\mathit{X}}_i=(X_{i1},X_{i2},\dots ,X_{ip}{)}^{\mathrm{\top }}$ is the set of the p predictor variables, $Y_i$ is the outcome variable of interest, $R_i\sim \mathrm{Bernoulli}({\pi }_i)$ is the response indicator for the ith subject, and $X_{is}$ is the sth predictor value for the ith individual, which were generated from a uniform distribution $X_{is}\sim \mathrm{Unif}(-1,1)$, $s=1,2,\dots ,p$. We assumed two conditions of correlation matrix $\mathbf{\Sigma }$ of ${\mathit{X}}_i$: if $j\le k$, ${\mathrm{\Sigma }}_{jk}={\rho }^{k-j}$ if $k\in \{j,j+1,\dots ,j+5\}$ and 0 for all other $j\le k$. If j>k, then ${\mathrm{\Sigma }}_{jk}={\mathrm{\Sigma }}_{kj}$, where ${\mathrm{\Sigma }}_{jk}$ is the jkth element of $\mathbf{\Sigma }$ and $\rho \in \{0,0.4\}$. The vector ${\mathit{X}}_i$ with specific correlation matrix was generated by using R package ‘MultiRNG’ and the method discussed in Ref. [19]. The random variables $X_{is}$ are mutually independent when $\rho =0$ and have auto-regressive structure when $\rho =0.4$. The outcome regression model and nonresponse model are as follows:

• (S1).
  $$\begin{array}{rl}{Y}_i& =1-0.8Z_{i1}+2.6Z_{i2}+1.8Z_{i3}+3Z_{i1}^2-0.4Z_{i2}^2-Z_{i3}^2-0.5Z_{i1}{Z}_{i2}\\ & +1.4Z_{i1}{Z}_{i3}-0.2Z_{i2}{Z}_{i3}+{ϵ}_i\end{array}$$

  and

  $$\begin{array}{rl}\log \left(\frac{{\pi }_i}{1+{\pi }_i}\right)& ={\alpha }_0+1.2Z_{i1}-0.8Z_{i2}+2Z_{i3}+4Z_{i1}^2-0.3Z_{i2}^2-0.2Z_{i3}^2\\ & -0.5Z_{i1}{Z}_{i2}+3Z_{i1}{Z}_{i3}+0.4Z_{i2}{Z}_{i3};\end{array}$$
• (S2).
  $$Y_i=1-0.8Z_{i1}+1.2Z_{i2}+0.5Z_{i3}+5Z_{i1}^2-0.2Z_{i2}^2+2Z_{i1}{Z}_{i2}+1.5Z_{i3}^3+{ϵ}_i$$

  and

  $$\log \left(\frac{{\pi }_i}{1+{\pi }_i}\right)={\alpha }_0+1.2Z_{i1}-0.8Z_{i2}+2Z_{i3}+2Z_{i1}^2-Z_{i2}^2-0.5Z_{i1}{Z}_{i2}-0.2Z_{i3}^3;$$

where $Z_{i1}=\sin (180X_{i1})$, $Z_{i2}=\exp (X_{i2})$, and $Z_{i3}=\log (\mathrm{cdf}(X_{i3}))$, with cdf() as the cumulative standard normal distribution function. We set the ${\epsilon }_i\sim N(0,{\sigma }^2)$ and $\sigma =$ 1.5 and 1 for S1 and S2, respectively, which resulted in variance of Y explained by X to be around 65%. We set ${\alpha }_0=$ 2.3 and 3.5 for S1 and S2, respectively, to have an overall average response rate of 60%.

In addition, we considered a scenario of mixed types of distribution of X and error term as follows:

• (S3).
  $$Y_i=1+0.5X_{i1}+0.25X_{i2}^3-0.8X_{i3}^4+1.75X_{i2}{X}_{i3}+0.5|X_{i1}{|}^{7/3}-1.2X_{i1}{X}_{i2}{X}_{i3}+{ϵ}_i$$

  and

  $$\log \left(\frac{{\pi }_i}{1+{\pi }_i}\right)={\alpha }_0+X_{i1}+1.5X_{i2}^3-0.8X_{i3}^4+1.2X_{i2}{X}_{i3}+0.5|X_{i1}{|}^{7/3}-X_{i1}{X}_{i2}{X}_{i3};$$

where $X_{i1}\sim \mathrm{Gamma}(0.5,1)$, $X_{i2}\sim \mathrm{Binom}(2,0.25)$, $X_{i3}\sim N(0.5,1)$, ${\epsilon }_i\sim N(0,3)$, and ${\alpha }_0=-0.4$. Similarly, we controlled the variance of Y explained by X to be around 65% and the overall average response rate to be 60%.

We compare the following approaches in the simulation study:

• (M1).Parametric approaches by using linear model (LM). The model will be built by stepwise model selection based on BIC.
• (M2).MI approaches with five imputed values. We will use unconditional mean imputation (Mean), predictive mean matching (PMM), and Bayesian linear regression (Norm) to impute the missing value. The three methods we selected represent commonly used naive (Mean), semi-parametric (PMM), and parametric (Norm) models in MI. The models were implemented using R package ‘mice’.
• (M3).Regularized approaches by using LASSO, RR, and SCAD. The R package glmnet with default setting was used. The tuning parameter will be selected using 10-fold cross-validation. For LASSO and SCAD, we used a de-biased estimator by refitting LM based on selected variables.
• (M4).RT, RF, and XGboosting approach (XGB). The hyper-parameters were tuned through grid search using caret or MLR package R in Table 1. Five times repeated five-fold cross-validation were used to evaluate the model performance during the tuning process. RT and RF were tuned for each iteration. Considering the complexity of tuning XGB, we randomly picked 1–3 simulated datasets to tune the parameters and applied them to all datasets.
• (M5).DL imputation approaches. We consider totally four DL architectures with four or five fully connected layers: DL-1r and DL-2r using ReLU and linear activation function (Figure 1), and DL-1s and DL-2s using SELU and linear activation function presented in Figure 2; DL-1r and DL-1s have number of nodes reduced in deeper layers, while DL-2r and DL-2s have same number of nodes in all layers. In order to control overfitting, we dropout nodes in some layers with rate of 0.25–0.4. Also, the training validation split is 60%:40%. Because of the high computation demand for DL simulation, we randomly pick 10 generated data sets to tune the learning rate from 0.1, 0.01, 0.001, and 0.0001. The selected learning rate will be applied to all data sets.

Table 1.

List of tuning parameters involved in M4 approaches.

Approach	Tuning package	Parameter	Range
RT	mlr	maxdepth	$2:min(p,15)$
	mlr	complexity parameter (cp)	$2^{(-9:-3)}$
RF	caret	mtry	$2:min(p,15)$
XGB	mlr	$max$_depth	2:10
	mlr	nrounds	20:200
	mlr	eta	0:0.35
	mlr	alpha	0:10
	mlr	lambda	0:2
	mlr	lambda_bias	0:2
	mlr	booster	(‘gblinear’, ‘gbtree’)

Note: p is the number of predictors in the data.

Figure 1.

Structures for DL-1r and DL-2r.

Figure 2.

Structures for DL-1s and DL-2s.

We consider imputation, propensity score, and doubly robust approaches in LASSO and SCAD. Suppose we are interested in estimating the population mean ${\theta }_0=E(Y)$. We implement the DL approaches in Python and all other approaches in R.

We performed 1000 iterations to compare the approaches described previously in terms of relative bias, relative standard deviation (SD), and relative root mean squared error (RMSE). The results for scenario (S1) to scenario (S3) are presented in Tables 2– 10. Under our simulation settings, the boosting and DL approaches outperformed the other approaches in terms of balancing the relative bias and relative SD in most of the scenarios. Boosting method was top in terms of least relative SD. Boosting method and DL methods have comparable performance in terms of RMSE. With proper structures, DL methods can produce lease relative bias and RMSE. MI methods have similar performance as LM, LASSO, SCAD, RR, RT, and RF when the dimension is low (p = 10). However, they have worse performance when the dimension becomes high. We cannot even obtain the estimates when the dimension is extremely high (p = 1000 ). The LM, LASSO-DB, LASSO-DR, RR, and RF have similar performance in all scenarios. LASSO-PS, SCAND-PS, and RT show comparable or better performance than LM, LASSO-DB, LASSO-DR, RR, and RF. The correlation in predictors did not affect the rank of the approaches a lot.

Table 2.

Relative bias (%) of different approaches in scenario S1.

	p = 10		p = 400		p = 1000
Model	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$
LM	46.49	45.96	47.08	46.87	46.65	47.02
MI-PMM	46.36	45.84	41.23	41.32	NA	NA
MI-Norm	46.42	45.90	−88.45	−85.09	NA	NA
MI-Mean	39.68	40.17	39.99	39.76	NA	NA
LASSO-DB	47.88	47.05	49.79	49.50	49.85	49.75
LASSO-PS	37.99	38.15	37.54	37.27	36.90	36.81
LASSO-DR	47.79	47.02	48.84	48.58	48.72	48.68
SCAD-DB	46.47	45.94	47.54	47.26	47.43	47.53
SCAD-PS	51.04	50.25	45.30	45.17	42.38	42.58
SCAD-DR	49.24	48.37	49.09	48.86	48.61	48.72
RR	45.87	45.27	40.88	40.88	40.35	40.34
RT	49.93	49.05	50.48	50.20	50.08	49.98
RF	47.87	46.88	41.56	41.51	40.58	40.47
XGB	−7.08	−7.65	−7.11	−7.22	−7.31	−7.32
DL-1r	−5.86	−6.03	−1.75	−1.91	−2.04	−2.25
DL-2r	1.18	0.45	6.46	5.82	6.49	6.39
DL-1s	−7.77	−8.52	−9.32	−10.01	−1.73	−2.46
DL-2s	−10.38	−11.33	−10.73	−11.25	3.49	2.52

Table 10.

Relative RMSE (%) of different approaches in scenario S3.

Model	p = 10	p = 400	p = 1000
LM	36.45	36.85	39.28
MI-PMM	53.84	258.41	NA
MI-Norm	35.80	2507.57	NA
MI-Mean	87.87	88.13	NA
LASSO-DB	60.29	66.12	67.72
LASSO-PS	56.01	56.74	57.31
LASSO-DR	54.63	56.26	57.30
SCAD-DB	35.22	37.44	39.53
SCAD-PS	65.15	58.51	57.72
SCAD-DR	75.82	58.90	57.16
RR	39.53	82.43	84.67
RT	57.15	62.22	63.02
RF	48.28	87.45	89.82
XGB	27.95	28.28	28.34
DL-1r	24.53	26.41	27.64
DL-2r	25.83	26.21	27.23
DL-1s	23.12	29.23	29.04
DL-2s	23.12	26.61	27.01

Table 3.

Relative SD (%) of different approaches in scenario S1.

	p = 10		p = 400		p = 1000
Model	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$
LM	6.13	6.10	6.11	6.00	8.01	7.35
MI-PMM	7.24	7.10	11.39	11.17	NA	NA
MI-Norm	6.93	6.97	3446.20	4016.47	NA	NA
MI-Mean	5.80	5.65	5.54	5.60	NA	NA
LASSO-DB	6.89	6.73	6.89	6.95	7.06	6.91
LASSO-PS	5.81	5.62	5.48	5.48	5.77	5.51
LASSO-DR	6.25	6.19	6.11	6.14	6.39	6.22
SCAD-DB	6.10	6.09	6.25	6.12	6.60	6.32
SCAD-PS	7.10	6.74	6.46	6.40	6.67	6.21
SCAD-DR	6.03	6.03	5.94	5.82	6.36	6.05
RR	6.00	5.95	5.51	5.55	5.78	5.54
RT	6.44	6.72	6.16	6.06	6.48	6.21
RF	6.05	6.13	5.53	5.54	5.75	5.51
XGB	4.23	4.15	4.14	4.15	4.17	4.11
DL-1r	6.60	6.61	9.51	9.19	9.53	9.63
DL-2r	6.14	5.97	6.37	6.39	6.79	6.44
DL-1s	5.43	5.50	4.79	4.79	6.25	6.12
DL-2s	4.94	4.66	4.52	4.52	6.66	7.10

Table 5.

Relative bias (%) of different approaches in scenario S2.

	p = 10		p = 400		p = 1000
Model	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$
LM	19.98	19.40	19.73	19.72	19.51	19.74
MI-PMM	20.51	19.20	−0.73	−0.56	NA	NA
MI-Norm	20.35	19.64	7.92	12.87	NA	NA
MI-Mean	16.20	11.44	16.27	16.29	NA	NA
LASSO-DB	17.12	17.28	13.81	13.80	13.54	13.64
LASSO-PS	9.24	7.71	9.78	9.72	10.01	10.33
LASSO-DR	19.39	19.13	17.23	17.20	16.97	17.17
SCAD-DB	20.05	19.58	18.15	18.16	17.39	17.74
SCAD-PS	6.44	4.88	8.44	8.46	8.90	9.15
SCAD-DR	21.34	21.02	20.26	20.27	19.54	19.90
RR	20.39	19.89	17.23	17.25	17.11	17.29
RT	12.49	12.89	9.41	10.03	8.47	8.70
RF	21.02	20.04	15.58	15.56	15.45	15.61
XGB	−7.62	−9.34	−8.40	−8.20	−8.58	−8.29
DL-1r	11.01	13.02	5.16	3.68	−0.86	−2.14
DL-2r	8.00	7.51	3.78	2.72	0.00	0.96
DL-1s	5.84	6.07	−2.32	−12.54	0.41	−20.24
DL-2s	3.49	8.71	−1.65	−7.96	−6.12	−17.30

Table 6.

Relative SD (%) of different approaches in scenario S2.

	p = 10		p = 400		p = 1000
Model	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$
LM	9.10	8.95	9.29	9.40	11.50	10.47
MI-PMM	9.59	9.65	16.09	16.67	NA	NA
MI-Norm	10.06	9.79	1510.95	925.54	NA	NA
MI-Mean	7.05	9.27	7.05	7.10	NA	NA
LASSO-DB	11.56	10.80	12.14	12.00	11.75	11.90
LASSO-PS	6.91	7.14	6.80	6.85	6.52	6.62
LASSO-DR	10.29	9.74	10.52	10.42	10.17	10.24
SCAD-DB	8.94	8.74	10.11	10.24	9.95	9.88
SCAD-PS	8.42	8.58	8.62	8.51	8.30	8.56
SCAD-DR	9.40	9.27	9.67	9.74	9.31	9.25
RR	8.28	8.08	7.04	7.13	6.88	6.92
RT	11.18	11.20	11.02	11.30	9.83	10.39
RF	9.50	9.00	7.13	7.14	6.87	6.95
XGB	5.89	6.23	5.77	5.79	5.49	5.65
DL-1r	8.56	8.74	7.47	7.61	7.28	7.57
DL-2r	7.73	7.91	6.71	6.56	6.14	6.22
DL-1s	8.26	8.03	7.75	7.01	9.26	6.26
DL-2s	7.72	7.93	6.77	7.34	7.57	7.05

Table 7.

Relative RMSE (%) of different approaches in scenario S2.

	p = 10		p = 400		p = 1000
Model	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$
LM	21.96	21.36	21.81	21.84	22.65	22.34
MI-PMM	22.64	21.49	16.11	16.68	NA	NA
MI-Norm	22.70	21.95	1510.97	925.63	NA	NA
MI-Mean	17.67	14.73	17.73	17.77	NA	NA
LASSO-DB	20.66	20.38	18.39	18.29	17.93	18.11
LASSO-PS	11.53	10.51	11.91	11.90	11.95	12.27
LASSO-DR	21.95	21.47	20.19	20.11	19.78	20.00
SCAD-DB	21.95	21.44	20.78	20.85	20.04	20.31
SCAD-PS	10.60	9.87	12.06	12.00	12.17	12.53
SCAD-DR	23.32	22.97	22.45	22.49	21.64	21.95
RR	22.01	21.47	18.61	18.67	18.44	18.62
RT	16.76	17.07	14.49	15.11	12.98	13.55
RF	23.07	21.97	17.13	17.12	16.91	17.09
XGB	9.63	11.22	10.19	10.04	10.19	10.03
DL-1r	13.95	15.68	9.08	8.45	7.33	7.87
DL-2r	11.13	10.91	7.70	7.10	6.14	6.29
DL-1s	10.12	10.06	8.09	14.36	9.27	21.18
DL-2s	8.47	11.78	6.97	10.83	9.73	18.68

Table 8.

Relative bias (%) of different approaches in scenario S3.

Model	p = 10	p = 400	p = 1000
LM	33.29	33.77	33.95
MI-PMM	51.15	228.38	NA
MI-Norm	30.35	247.39	NA
MI-Mean	86.17	86.28	NA
LASSO-DB	54.69	61.18	62.78
LASSO-PS	54.14	54.93	55.56
LASSO-DR	52.27	53.81	54.72
SCAD-DB	31.83	34.61	36.88
SCAD-PS	60.78	55.62	54.46
SCAD-DR	70.20	55.81	53.40
RR	36.65	80.64	82.96
RT	55.08	60.46	61.32
RF	45.57	85.35	87.81
XGB	25.65	25.96	26.05
DL-1r	18.78	18.72	20.21
DL-2r	21.26	21.69	22.37
DL-1s	17.08	23.50	23.57
DL-2s	17.84	21.89	22.21

Table 9.

Relative SD (%) of different approaches in scenario S3.

Model	p = 10	p = 400	p = 1000
LM	14.83	14.76	19.77
MI-PMM	16.80	120.90	NA
MI-Norm	19.00	2495.34	NA
MI-Mean	17.21	17.99	NA
LASSO-DB	25.37	25.06	25.37
LASSO-PS	14.37	14.23	14.04
LASSO-DR	15.89	16.41	17.00
SCAD-DB	15.09	14.29	14.22
SCAD-PS	23.45	18.15	19.11
SCAD-DR	28.65	18.84	20.41
RR	14.82	17.11	16.96
RT	15.23	14.68	14.54
RF	15.97	19.04	18.89
XGB	11.12	11.22	11.17
DL-1r	15.78	18.63	18.85
DL-2r	14.66	14.70	15.52
DL-1s	15.58	17.39	16.97
DL-2s	14.70	15.12	15.37

7. Real data application

We conducted a real data analysis using a study data set from gene expression omnibus. The study found a total of 24 miRNAs to be significantly associated with metabolic syndrome traits in a population of 200 men. We downloaded the expression profile of 1918 miRNAs in 200 subjects and their body mass index (BMI) as our variable of interest. We removed 883 non-informative miRNAs having variance of expression less than or equal to 0.1, which resulted in 1035 miRNAs in 200 subjects for the real data analysis. In the 200 subjects, BMI likely follows a normal distribution (Shapiro–Wilk normality test p-value 0.164) with mean  $\pm$ SD: 26.62 $\pm$3.32, median of 26.40 (Figure 3). A simple linear regression about the association between BMI and miRNA showed that there were 2 and 25 miRNAs were significant with Benjamini-Hochberg multiple testing correction and without correction at significance level of 0.05. The average pair-wise correlation between miRNAs was 0.26. We generated the response indicators of the subjects using a similar procedure of our simulation. Let ${\mathbf{X}}_i$ represent the miRNA expression, and we randomly picked three miRNAs $X_1$, $X_2$, and $X_3$ to generate the response indicator. First, we made the non-linear transformation $Z_1=\sin (180X_1)$, $Z_2=\exp (X_2)$, and $Z_3=\log (\mathrm{cdf}(X_3))$. Then the non-response model was generated by

$$\log \left(\frac{{\pi }_i}{1+{\pi }_i}\right)={\alpha }_0+Z_{i1}+Z_{i2}+Z_{i3}+Z_{i1}^2+Z_{i2}^2+Z_{i3}^2+Z_{i1}{Z}_{i2}+Z_{i1}{Z}_{i3}+Z_{i2}{Z}_{i3}.$$

We set ${\alpha }_0=-1.9$ to have an overall average response rate of $50\mathrm{\%}$.

Figure 3.

Distribution of BMI in real data.

In this high-dimensional real data analysis, we tuned the parameters for machine learning methods again following the procedures in the simulation. The result is presented in Table 11. DL and boosting were top two methods having the least relative bias. The propensity score estimate using SCAD had the highest bias. The LM resulted in a small relative bias, but greatest RMSE about the individual BMI. DL ranked second after RF in RMSE.

Table 4.

Relative RMSE (%) of different approaches in scenario S1.

	p = 10		p = 400		p = 1000
Model	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$	$\rho =0$	$\rho =0.4$
LM	46.90	46.36	47.48	47.25	47.34	47.59
MI-PMM	46.92	46.39	42.78	42.81	NA	NA
MI-Norm	46.93	46.42	3447.34	4017.37	NA	NA
MI-Mean	40.10	40.56	40.37	40.15	NA	NA
LASSO-DB	48.37	47.53	50.26	49.99	50.35	50.23
LASSO-PS	38.43	38.57	37.94	37.67	37.35	37.22
LASSO-DR	48.20	47.43	49.22	48.97	49.14	49.07
SCAD-DB	46.87	46.34	47.95	47.65	47.89	47.95
SCAD-PS	51.53	50.70	45.75	45.62	42.90	43.03
SCAD-DR	49.61	48.75	49.44	49.20	49.03	49.10
RR	46.26	45.66	41.25	41.26	40.76	40.72
RT	50.34	49.51	50.86	50.56	50.50	50.36
RF	48.25	47.28	41.92	41.88	40.99	40.84
XGB	8.24	8.70	8.23	8.33	8.42	8.40
DL-1r	8.82	8.94	9.67	9.38	9.75	9.89
DL-2r	6.25	5.99	9.08	8.65	9.40	9.07
DL-1s	9.48	10.14	10.48	11.09	6.49	6.60
DL-2s	11.49	12.25	11.64	12.13	7.51	7.53

Table 11.

Performance summary of different approaches in real data analysis.

	Bias	Relative bias	RMSE	Relative RMSE
LM	−0.191	−0.72%	5.17	19.43%
LASSO_DB	−0.260	−0.98%	3.24	12.18%
LASSO_PS	−0.487	−1.83%	NA	NA
LASSO_DR	−0.272	−1.02%	NA	NA
SCAD_DB	−0.395	−1.48%	3.30	12.39%
SCAD_PS	−0.919	−3.45%	NA	NA
SCAD_DR	−0.358	−1.34%	NA	NA
Ridge	−0.275	−1.03%	3.23	12.12%
RT	−0.442	−1.66%	4.17	15.65%
RF	−0.255	−0.96%	3.15	11.82%
XGB	−0.130	−0.49%	3.60	13.53%
DL	0.072	0.27%	3.21	12.06%

8. Discussion

In this paper, we presented imputation, propensity score, and doubly robust approaches based on several machine learning approaches for handling high dimensional data with missing values. Our proposed approaches are very general and can be used for estimating general parameters of interest including population means and quantiles. We compared several missing data approaches for handling high-dimensional data by using both simulation study and real application. XGboosting and DL approaches outperform other methods with high-dimensional non-linear model structure. The key consideration of using DL and other machine learning methods is to avoid overfitting. Statistical inference with machine learning-based approaches is a very challenging research problem. We will pursue the research in our future research.

Funding Statement

Dr Sixia Chen is partly supported by the National Institute on Minority Health and Health Disparities at National Institutes of Health (1R21MD014658-01A1) and the Oklahoma Shared Clinical and Translational Resources (U54GM104938) with an Institutional Development Award (IDeA) from National Institute of General Medical Sciences.

Disclosure statement

No potential conflict of interest was reported by the author(s).