Model Checking for Logistic Models with Study of Telehealth During the COVID-19 Pandemic Among PWH in DC

Abstract

We propose a projection-based test to check logistic regression models and apply the test to study telehealth utilization during the COVID-19 pandemic among patients with HIV. The test is shown to be consistent and can detect root-$n$ local alternatives. The use of the proposed test to investigate a COVID-19 dataset reveals that the probability of telehealth utilization depends on the following variables: overweight, education, and age and the interaction between age and ethnicity. Specifically, the probability for the Hispanic group decreases with older age, whereas there is no trend between the probability with the age for the group of non-Hispanic. This interaction may be ignored when we apply other goodness-of-fit methods. The simulation studies also show the performance of the proposed method is remarkably attractive compared to its competitors.

1. Introduction

Telehealth has previously been used to provide HIV care and prevention services in rural areas and settings with limited healthcare access and has slowly expanded to urban patients who deferred face-to-face meetings [1]. However, during the COVID-19 pandemic, telehealth was rapidly deployed for HIV ambulatory care [2]. Data for this analysis come from COVID-19 survey, a sub study of the DC Cohort study which is, a longitudinal study of persons living with HIV and receiving care in the Washington, DC area [3]. Study participants included those who completed a cross-sectional electronic COVID-19 survey that was sent to DC Cohort participants from Oct 30, 2020 to Auguest 19, 2022 (n = 1673). Among 1551 PWH with complete data, 68% reported at least one telehealth visit during the study.

Applying a logistic regression model to fit the logit of telehealth against the covariates (CHD, hypertension, age, ethnicity, overweight, education level, and number of comorbidities) and using the AIC [4] criterion to select the important covariates, we find only three covariates: ethnicity, overweight and education. Yet, a further exploration (see more details in Sect. 3) reveals that age acts as an important effect modifier of ethnicity. We are interested in whether the model without an interaction term is enough to investigate the relationship between telehealth and the covariates. This is an essential goodness-of-fit problem.

The Hosmer–Lemeshow test [5], the most commonly applied approach in the literature, could be applied to address such a goodness-of-fit concern. However, this test has some limitations and generally has very low power [6]. Two alternatives have been developed to remedy limitations of the Hosmer–Lemeshow test; i.e., the Osius–Rojek test [7] and the Stukel test [8]. Simulation studies suggest that the Stukel test is slightly less powerful at detecting quadratic departures from linearity but more powerful at detecting interactions than the Hosmer–Lemeshow test. Furthermore, the Stukel test seems considerably more powerful than the Hosmer–Lemeshow test at detecting departures from the logit link function [6, 9]. The Stukel test has been advocated to use for goodness-of-fit checking. Unfortunately, the test works less satisfactorily for the current dataset (see more details in Sect. 3).

To address the problems, we extended the projection-based goodness-of-fit test for parametric regression [10] to the current setting by integrating a model-based bootstrap (MBB) procedure [11, 12]. The intuitive purpose is to avoid studying the regression, i.e., $\epsilon (X,\mathit{\beta })$ against $X$ [see the definition of $\epsilon (X,\mathit{\beta })$ in Sect. 2], because this regression is a multivariate nonparametric function and leads to curse-of-dimensionality. Instead, we consider the regression $\mathrm{E}\left\{\epsilon (X,\mathit{\beta })\mid X^{\mathrm{\top }}W\right\}$ after projecting $X$ to a direction $W$. Allowing the direction $W$ to vary, the new regression on a scalar does not lose any information yet avoids the curse-of-dimensionality problem. The proposed test will be shown to be free from the data generation process and can detect the alternative hypothesis distinguishing from the null hypothesis at the rate up to $n^{-1/2}$, the best rate, and has outstanding power performance under the alternatives (see more details in Sect. 4).

In Sect. 2, we present the projection-based test statistic and the asymptotic properties of the test statistic under null and alternative hypotheses, and suggest a bootstrap method to calculate the critical value. We report the results for the telehealth study in Sect. 3, and the simulation studies in Sect. 4. A brief proof of the main result is presented in the “Appendix”.

2. The Test Statistics

Let $Y$ be a binary response variable and $X={\left(x_1,\dots ,x_p\right)}^{\mathrm{\top }}$ be the vector of covariates. Their relationship follows the logistic regression model

$$\text{logit}\left\{P\left(Y=1∣X\right)\right\}=X^{\mathrm{T}}\mathit{\beta },$$

where $\mathit{\beta }={\left({\beta }_1,\dots ,{\beta }_p\right)}^{\mathrm{\top }}$ is a vector of unknown parameters. Let $g\left(v\right)=\{1+\text{exp}(-v){\}}^{-1}$ be the logistic distribution function. Write $\epsilon (X,\mathit{\beta })=Y-g\left(X^{\mathrm{\top }}\mathit{\beta }\right)$ and consider

$${\mathscr{H}}_0:\mathrm{E}\left\{\epsilon \right(X,\mathit{\beta })\mid X\}=0\text{for}\text{some}\mathit{\beta }$$ (1)

against the alternative hypothesis:

$${\mathscr{H}}_1:\mathrm{E}\left\{\epsilon \left(X,\mathit{\beta }\right)∣X\right\}\ne 0\text{for}\text{any}\mathit{\beta }\in {\mathbb{R}}^p.$$

So the alternative hypothesis is very general to cover various situations such as quadratic or interaction or nonlinear departures from linearity.

Note that $\mathrm{E}\left\{\epsilon \right(X,\mathit{\beta })\mid X\}=0$ if and only if $\mathrm{E}\left\{\epsilon (X,\mathit{\beta })\mid X^{\mathrm{\top }}W\right\}=0$ almost sure for any unit vector $W$ [13]. Instead of considering $\mathrm{E}\left\{\epsilon \right(X,\mathit{\beta })\mid X\}$, we focus on $\mathrm{E}\left\{\epsilon (X,\mathit{\beta })\mid X^{\mathrm{\top }}W\right\}$. Study of this projection brings a substantial convenience because the former involves a regression against a vector $X$, which may raise curse-of-dimensionality problems, while the latter takes expectation on a projection $X^{\mathrm{\top }}W$, a scalar no matter what the dimension of $X$ is. Yet, this projection does not lose any information.

Let $\left\{\left(Y_i,X_i\right),i=1,\dots ,n\right\}$ be a sample from $(Y,X)$. Then the log-likelihood function is

$$ℒ\left(\mathit{\beta }\right)=\sum _{i=1}^n\left[Y_i\text{log}\left\{g\left(X_i^{\mathrm{\top }}\mathit{\beta }\right)\right\}+\left(1-Y_i\right)\text{log}\left\{1-g\left(X_i^{\mathrm{\top }}\mathit{\beta }\right)\right\}\right].$$

The maximum likelihood estimator (MLE) ${\widehat{\mathit{\beta }}}_n$ of $\mathit{\beta }$ is the solution to

$$L_n\left(\mathit{\beta }\right)=\frac{\partial ℒ\left(\mathit{\beta }\right)}{\partial \mathit{\beta }}=\sum _{i=1}^n{X}_i\left\{Y_i-g\left(X_i^{\mathrm{\top }}\mathit{\beta }\right)\right\}=\mathbf{0}.$$

Write $\widehat{\epsilon }\left(X_i,{\widehat{\mathit{\beta }}}_n\right)=Y_i-g\left(X_i^{\mathrm{\top }}{\widehat{\mathit{\beta }}}_n\right)$. Define

$$M_n\left(u,W\right)=\frac{1}{\sqrt{n}}\sum _{i=1}^n\widehat{\epsilon }\left(X_i,{\widehat{\mathit{\beta }}}_n\right)I\left(X_i^{\mathrm{\top }}W\le u\right)\text{for}u\in {\mathbb{R}}^1,$$

where $W$ is uniformly distributed on the unit sphere in ${\mathbb{R}}^p:{\mathbb{S}}^p=\left\{w\in {\mathbb{R}}^p:\Vert w\Vert =1\right\}$.

The proposed test statistic is

$$T_n={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{{\mathbb{S}}^p}{\left\{M_n\left(u,w\right)\right\}}^{2}dw{F}_{nw}\left(du\right),$$

where $F_{nw}\left(u\right)=1/n{\sum }_{i=1}^{n}I\left(X_i^{\mathrm{\top }}w\le u\right)$. We reject the null hypothesis for large values of $T_n$. The empirical process $M_n(u,W)$ is a weighted cumulative sum of the residuals, and $T_n$ is a Cramér–von Mises type test statistic.

A direct calculation yields

$$T_n=\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n\sum _{l=1}^n\widehat{\epsilon }\left(X_i,{\widehat{\mathit{\beta }}}_n\right)\widehat{\epsilon }\left(X_j,{\widehat{\mathit{\beta }}}_n\right)A_{ijl},$$

where $A_{\mathit{\text{ijl}}}={\int }_{{\mathbb{S}}^p}I\left(X_i^{\mathrm{\top }}w\le X_l^{\mathrm{\top }}w\right)I\left(X_j^{\mathrm{\top }}w\le X_l^{\mathrm{\top }}w\right)dw$, which can further be expressed as [10]

$$A_{\mathit{\text{ijl}}}=C_p\left|\pi -\text{arccos}\left\{\frac{{\left(X_i-X_l\right)}^{\mathrm{\top }}\left(X_j-X_l\right)}{\left|X_i-X_l\right|\left|X_j-X_l\right|}\right\}\right|$$

with $C_p={\pi }^{(p/2)-1}/\mathrm{\Gamma }(p/2+1)$, where $\mathrm{\Gamma }(\cdot )$ is the gamma function. Write $\mathbf{A}={\left({\sum }_{l=1}^n{A}_{\mathit{\text{ijl}}}\right)}_{n\times n}$. Immediately, we have $T_n=1/n^2{\widehat{\mathit{\epsilon }}}^{\mathrm{\top }}\mathbf{A}\widehat{\mathit{\epsilon }}$, where $\widehat{\mathit{\epsilon }}={\left(\widehat{\epsilon }\left(X_1,{\widehat{\mathit{\beta }}}_n\right),\dots ,\widehat{\epsilon }\left(X_n,{\widehat{\mathit{\beta }}}_n\right)\right)}^{\mathrm{\top }}$. Thus, the multiple-integration calculation of the statistic can be transformed to a calculation of a summation, which makes the calculation of $T_n$ straightforward.

Below we briefly examine asymptotic properties for the proposed statistic under the null and alternative hypotheses. Let $H\left(v\right)$ be the derivative of $g\left(v\right)$; i.e., $g\left(v\right)\{1-g(v\left)\right\}$. For any function $\eta$, denote ${\eta }_i\left(\mathit{\beta }\right)=\eta \left(X_i^{\mathrm{\top }}\mathit{\beta }\right)$ and ${\eta }_i=\eta \left(X_i^{\mathrm{\top }}{\mathit{\beta }}_0\right)$ with ${\mathit{\beta }}_0$ being the true value, for instance, $H_i\left(\mathit{\beta }\right)=H\left(X_i^{\mathrm{\top }}\mathit{\beta }\right),g_i=g_i\left({\mathit{\beta }}_0\right)$. Let $G_n\left(\mathit{\beta }\right)={\sum }_{i=1}^n{X}_i{H}_i\left(\mathit{\beta }\right)X_i^{\mathrm{\top }}$ and $S_n={\sum }_{i=1}^n{X}_i{X}_i^{\mathrm{\top }}$. Let $\mathbf{H}=\text{diag}\left(H_1,\dots ,H_n\right)$.

In what follows, ${\lambda }_{\text{max}}\left(A\right)$ and ${\lambda }_{\text{min}}\left(A\right)$ denote the maximum and minimum eigenvalues for a symmetric matrix $A$ respectively. $A_1\ge A_2$ means $A_1-A_2$ is semi-positive definite for two matrices $A_1$ and $A_2$.

Assumption

• (A1) There exist two positive constants $c_{\text{min}}$ and $C_{\text{max}}$ such that $c_{\text{min}}n\le {\lambda }_{\text{min}}\left(S_n\right)\le {\lambda }_{\text{max}}\left(S_n\right)\le C_{\text{max}}n$.

• (A2) There is a positive constant $c_{00}$ such that $\underset{1\le i\le n}{\text{min}}g\left(X_i^{\mathrm{\top }}{\mathit{\beta }}_0\right)\left\{1-g\left(X_i^{\mathrm{\top }}{\mathit{\beta }}_0\right)\right\}\ge c_{00}$. The condition (A2) bounds the eigenvalues of $S_n$ in probability. This is a basic assumption to ensure $S_n/n$ is not ill-conditioned. This assumption is needed for establishing asymptotic properties of ${\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_n$ [16].

It can be shown that the estimated empirical process $M_n(u,W)$, $-\mathrm{\infty }<u<\mathrm{\infty }$, converges to a centered Gaussian process, and $T_n$ converges to an integrated squared Gaussian process. Let ${\mathrm{\Sigma }}_{\beta }=\mathrm{E}\left(X\mathbf{H}{X}^{\mathrm{\top }}\right),\mathrm{\Gamma }\left(u\right)=\mathrm{E}\left\{g^{\prime }\left(X^{\mathrm{\top }}{\mathit{\beta }}_0\right)X^{\mathrm{\top }}{\mathrm{\Sigma }}_{\beta }^{-1}I\left(X^{\mathrm{\top }}W\le u\right)\right\}$, and ${\mathrm{\Psi }}_u(X,\epsilon )=\left\{I\left(X^{\mathrm{\top }}W\le u\right)-\mathrm{\Gamma }\left(u\right)X\right\}\epsilon$. We have the following result. A sketch proof is deferred to the Appendix.

Theorem 1 Suppose that Conditions (A1)and (A2) hold. Under the null hypothesis (1), the estimated empirical process $M_n(u,W)$ converges in distribution to $M\left(u\right)$, $-\mathrm{\infty }<u<\mathrm{\infty }$, in the Skorohod space $S[-\mathrm{\infty },\mathrm{\infty }]$, where $M\left(u\right)$ is a centered Gaussian process with covariance functions

$$\text{cov}\left\{M\left(u_1\right),M\left(u_2\right)\right\}=\mathrm{E}\left\{{\mathrm{\Psi }}_{u_1}(X,\epsilon ){\mathrm{\Psi }}_{u_2}(X,\epsilon )\right\}.$$

For the test statistic $T_n$, we have

$$T_n\stackrel{L}{\to }\int \{M\left(u\right){\}}^{2}F\left(du\right),$$

where $F$ is the distribution of $X^{\mathrm{\top }}W$.

In the following, we investigate the power behavior of the statistic under local and global alternatives. We consider the local alternative with a deviation of a nonlinear measurable function of $X$ from the null hypothesis; i.e.,

$${\mathscr{H}}_{1n}:\text{Pr}\left\{E(Y\mid X)=g\left(X^{\mathrm{\top }}{\mathit{\beta }}_0+r_{n}D\left(X\right)\right)\right\}=1,$$ (2)

where $D\left(X\right)$ cannot take the form of $X^{\mathrm{\top }}\mathit{\beta }$ for any $\mathit{\beta }$ and is a measurable function of $X$ satisfying with $0<\mathrm{E}\left\{D^2\left(X\right)\right\}<\mathrm{\infty },r_n$ is a constant depending on $n$. The expression (2) covers a variety of alternatives, rather than a specific parametric alternative, so the proposed test is more applicable due to possibly broad alternatives.

Checking the derivation for (9) in the “Appendix”, we can validate that, under the alternatives (2),

$$M_n(u,W)=\frac{1}{\sqrt{n}}\sum _{i=1}^n\left\{I\left(X_i^{\mathrm{\top }}W\le u\right)-\mathrm{\Gamma }\left(u\right)X_i\right\}{\epsilon }_i+r_n{n}^{1/2}E\left[g\left(X^{\mathrm{\top }}{\mathit{\beta }}_0\right)\left\{1-g\left(X^{\mathrm{\top }}{\mathit{\beta }}_0\right)\right\}D\left(X\right)I\left(X^{\mathrm{\top }}W\le u\right)\right]+o_p\left(1\right).$$

As a result, we have the following proposition on the behavior of asymptotic power of the proposed statistic.

Proposition 1 Suppose that Conditions (A1) and (A2) hold. Under the alternatives (2), $T_n\stackrel{\mathrm{P}}{\to }\mathrm{\infty }$ if $r_n{n}^{1/2}\to \mathrm{\infty }$, and is different from $\int \left\{M\right(u\left){\}}^{2}F\right(du)$ if $r_n{n}^{1/2}$ approaches to a positive but finite constant.

Therefore, the proposed statistic can detect a local alternative which approaches the null model at the parametric rate, $n^{-1/2}$. The statistic $T_n$ diverges to infinity when the global alternative hypothesis (2) converges to the null at a slower than the parametric rate, and in this case the test has asymptotic power 1 and is consistent.

Theorem 1 gives the asymptotic distribution of the statistic $T_n$ under the null hypothesis. But the asymptotic distribution contains several expectations that depend on the distribution of $X$; i.e., the distribution of $T_n$ may be case-dependent. Such a dependence complicates calculation of the critical value. To avoid the potential complexities, we calculate the critical value using the bootstrap resampling, which was generally adapted in the relevant literature. Specifically, we adopt the resampling strategy (i.e., MBB procedure) proposed by [17] and [12] as follows.

• Step 1. Generate a sample $\left\{Y_i^{*},i=1,\dots ,n\right\}$ of independent Bernoulli random variables, where $Y_i^{*}$ has the probability of success given by $g\left(X_i^{\mathrm{\top }}{\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_n\right)$.

• Step 2. Calculate the bootstrap test statistic, denoted by $T_n^{*}$, based on the bootstrap sample $\left\{\left(Y_i^{*},X_i\right),i=1,\dots ,n\right\}$.

• Step 3. Repeat Steps 2 and 3 $B$ times and obtain $T_{n1}^{*},\dots ,T_{nB}^{*}$. Then calculate the $1-\alpha$ empirical quantile of the bootstrap test statistics $\left\{T_{n1}^{*},\dots ,T_{nB}^{*}\right\}$ as the $\alpha$-level critical value.

This model-based bootstrap strategy integrates the model assumption and is therefore expected to be efficient in contrast to the wild bootstrap resampling method. Note that for the bootstrap procedure, it is not necessary to calculate or estimate any quantities such as the distribution or moments of $X$. Also, the testing procedure is data-driven; given only the sample, $\left\{\left(Y_1,X_1\right),\dots ,\left(Y_n,X_n\right)\right\}$, the proposed testing procedure using the bootstrap-generated critical value can determine whether the logistic regression model fits the data adequately without any other information on the data generation process. For the bootstrap testing statistic $T_n^{*}$, we can verify the following proposition by using the similar arguments to those in the proof of the main results of [12].

Proposition 2 Under the null hypothesis (1) or alternative hypothesis (2), if Conditions (A1) and (A2) are satisfied, the conditional distribution of $T_n^{*}$ converges in distribution to the limiting null distribution of $T_n$ given the sample $\left\{\left(Y_1,X_1\right),\dots ,\left(Y_n,X_n\right)\right\}$.

Proposition 2 indicates that the bootstrap version of the test statistic $T_n$ has the same asymptotic distribution as that of the proposed test. By generating i.i.d. random variables ${\left\{Y_i^{*}\right\}}_{i=1}^n$ repeatedly, we can get a series of bootstrap test statistics, which can be regarded as a sample from the population $T_n$. Then we can obtain the empirical quantile of the distribution of $T_n$. The critical value determined by this method approximates the theoretical one regardless of whether the data are from the null hypothetical model (1) or the alternative hypothetical model (2).

Reference [14] proposed a similar test except that they used $I\left(X^{\mathrm{\top }}{\tilde{\mathit{\beta }}}_n\le u\right)$ instead of $I\left(X^{\mathrm{\top }}W\le u\right)$ in $M_n(u,W)$, where ${\tilde{\mathit{\beta }}}_n$ is an estimate, say maximum likelihood estimate, of $\mathit{\beta }$ based on the data. So they considered one fixed direction ${\tilde{\mathit{\beta }}}_n$, and their test may be inconsistent except under specific situations, and also less powerful. Along with but different from [14], in which several nonparametric regressions were required to calculate the power, Reference [15] suggested adopting the MBB procedure (see Steps 1–3 on p. 7) to calculate the critical value.

Reference [8] introduced a generalized logistic model of form

$$P(Y=1\mid X)={\mu }_{\alpha }\left(\eta \right)=\text{exp}\left(h_{\alpha }\left(\eta \right)\right)/\left(1+\text{exp}\left(h_{\alpha }\left(\eta \right)\right)\right),$$

where $h_{\alpha }\left(\eta \right)$ is a strictly increasing nonlinear function of $\eta$ indexed by two shape parameters, ${\alpha }_1$ and ${\alpha }_2$, and

$$h_{\alpha }=\left\{\begin{array}{ll}\left(\text{exp}\left({\alpha }_1\left|\eta \right|\right)-1\right)/{\alpha }_1,& {\alpha }_1>0\\ \eta ,& {\alpha }_1=0\\ -\text{log}\left(1-{\alpha }_1\left|\eta \right|\right)/{\alpha }_1,& {\alpha }_1<0\end{array}\right\}\text{for}\eta ⩾0,$$

and

$$h_{\alpha }=\left\{\begin{array}{ll}-\left(\text{exp}\left({\alpha }_2\left|\eta \right|\right)-1\right)/{\alpha }_2,& {\alpha }_2>0\\ \eta ,& {\alpha }_2=0\\ \text{log}\left(1-{\alpha }_2\left|\eta \right|\right)/{\alpha }_2,& {\alpha }_2<0\end{array}\right\}\text{for}\eta \le 0,$$

with $\eta =X^{\mathrm{\top }}\beta$ being the linear predictor.

The additional parameters $\left({\alpha }_1,{\alpha }_2\right)$ allow the tails of the logistic regression model to be heavier/longer or lighter/shorter than the standard logistic regression model. Because this generalization is still a parametric model, a score or log-likelihood test can be used to test whether the parameters of a generalized logistic model are equal to zero; i.e., the generalized logistic model reduces to the standard logistic model when ${\alpha }_1={\alpha }_2=0$, which is equivalent to testing whether ${\alpha }_1={\alpha }_2=0$ in the new logistic model: $\text{logit}={\mathbf{X}}^{\mathrm{\top }}\mathit{\beta }+{\alpha }_1{z}_1+{\alpha }_2{z}_2$ [8], where $z_1={\left({\mathbf{X}}^{\mathrm{\top }}\widehat{\mathit{\beta }}\right)}^2\times \mathit{I}(\hat{\pi }\ge 0.5)/2$ and $z_2=-{\left({\mathbf{X}}^{\mathrm{\top }}\stackrel{\mathit{ˆ}}{\mathit{\beta }}\right)}^2\times \mathit{I}(\hat{\pi }<0.5)/2$, and $\hat{\pi }$’s are the fitted values from the model fitting.

3. Analysis of Association Between Telehealth and Covariates

In this section, we present the detailed analysis for the telehealth study. Table 1 gives the descriptive statistics of the variables.

Table 1.

Descriptive statistics by telehealth

	0	1	Combined	Test statistic
CHD	3% (15)	4% (41)	4% (56)	${\chi }_1^2=0.74,\mathrm{P}=0.39$ 1
Hypertension	40% (191)	42% (433)	41% (624)	${\chi }_1^2=1.02,\mathrm{P}=0.313$ 1
Overweight	18% (88)	22% (225)	21% (313)	${\chi }_1^2=2.81,\mathrm{P}=0.094$ 1
Comorbid0	36% (176)	31% (315)	33% (491)	${\chi }_4^2=6.58,\mathrm{P}=0.16$ 1
Comorbid1	31% (150)	31% (322)	31% (472)
Comorbid2	19% (93)	21% (215)	20% (308)
Comorbid3	9% (43)	11% (110)	10% (153)
Comorbid4	4% (21)	6% (62)	6% (83)
Education1	8% (41)	11% (108)	10% (149)	${\chi }_3^2=9.19,\mathrm{P}=0.027$ 1
Education2	26% (127)	27% (279)	27% (406)
Education3	26% (126)	31% (313)	29% (439)
Education4	39% (189)	32% (324)	34% (513)
Age	44.0 54.0 62.0 (52.9 ±12.2)	43.0 54.0 62.0 (52.5 ±12.5)	43.0 54.0 62.0 (52.7 ±12.4)	$F_{\mathrm{1,1505}}=0.06,\mathrm{P}=0.799$ 2
Ethnicity	94% (452)	89% (908)	90% (1360)	${\chi }_1^2=8.99,\mathrm{P}=0.003$ 1

abc represent the lower quartile $a$, the median $b$, and the upper quartile $c$ for continuous variables. $x\pm s$ represents $\bar{X}\pm 1\text{SD}$. Numbers after percents are frequencies.

Tests used: ¹ Pearson test;

²Wilcoxon test

We first present the results in Table 2 based on the logistic model (3) that fits the data with all covariates.

$$\begin{gathered} \text{logit}\left\{\mathrm{p}\right(\text{telehealth}=1\mid \text{covariates}\left)\right\}={\beta }_0+{\beta }_1\text{CHD}+{\beta }_2\text{hypertension} \\ +{\beta }_3\text{overweight}+{\beta }_4\text{comorbid}+{\beta }_5\text{education} \\ +{\beta }_6\text{age}+{\beta }_7\text{ethnicity}. \end{gathered}$$ (3)

In the full model, only ethnicity is significant. Therefore, we apply the AIC criterion to select the important covariates and obtain ethnicity, overweight and education. We then further observe that there is an interesting pattern between the probability of telehealth and age, and display the pattern in Fig. 1. Specifically, the probability for the group of Hispanic decreases with older age, whereas there is no a trend between the probability and the age for the group of Non-Hispanic. We now apply the procedure to analyze the telehealth dataset.

Table 2.

The estimated coefficients and associated standard errors based on model (3)

	Estimate	Standard error	z value	$\mathrm{P}\mathrm{r}(>|\mathrm{z}\left|\right)$
(Intercept)	0.83	0.32	2.58	0.01
CHD	0.13	0.33	0.40	0.69
Hypertension	−0.08	0.16	−0.50	0.61
Overweight	0.06	0.17	0.33	0.74
Comorbid1	0.28	0.16	1.75	0.08
Comorbid2	0.38	0.21	1.79	0.07
Comorbid3	0.49	0.29	1.71	0.09
Comorbid4	0.59	0.36	1.63	0.10
Education2	−0.07	0.22	−0.31	0.76
Education3	0.03	0.22	0.15	0.88
Education4	−0.32	0.21	−1.51	0.13
Age	−0.00	0.00	−0.94	0.35
Ethnicity	0.66	0.22	3.01	0.00

Fig. 1.

Conditional plot: separate fitted curves for two ethnicity groups: left for Hispanic; right for Non-Hispanic

Based on these observations, we model the relationship between the logit of telehealth = 1 and the four potential covariates: overweight, education, age, and ethnicity using the following logistic model.

$$\begin{gathered} \text{logit}\left\{\mathrm{p}\right(\text{telehealth}=1\mid \text{covariates}\left)\right\}={\beta }_0+{\beta }_1\text{overweight}+{\beta }_2\text{education}+{\beta }_3\text{age} \\ +{\beta }_4\text{ethnicity}. \end{gathered}$$ (4)

We applied the proposed test $T_n$ and the Stukel test [8] $T_n^{\text{ST}}$ to check the goodness-of-fit of model (4). After conducting 500 Bootstrap replications, we obtained the p-values based on $T_n$ and $T_n^{\text{ST}}$ 0.004 and 0.618, respectively. Therefore, the proposed test suggests a favor to reject the null hypothesis, while the Stukel test suggests that we cannot reject the null hypothesis.

We then introduced the interaction term of age and ethnicity to model (4). That is, we fit (5) and repeated two tests. The resultant p-values are 0.054 and 0.99. So model (5) with the interaction term is more appropriate than model (4) and reasonably fits the dataset.

$$\begin{gathered} \text{logit}\left\{\mathrm{p}\right(\text{telehealth}=1\mid \text{covariates}\left)\right\}={\beta }_0+{\beta }_1\text{overweight}+{\beta }_2\text{education}+{\beta }_3\text{age} \\ +{\beta }_4\text{ethnicity}+{\beta }_5\text{age}*\text{ethnicity.} \end{gathered}$$ (5)

For a further exploration of this dataset, we present the estimated results based on model (5) in Table 3. The variables age, ethnicity and their interaction show a significance. The interaction effect between age and ethnicity is also supported by the left panel of Fig. 1, where the probability of telehealth is indifferent in the group of Non-Hispanic, but shows a decreasing trend with the age in the Hispanic group. Given the dominated sample size in the group of Non-Hispanic, the decreasing trend in the group of Hispanic may be concealed if we ignore the interaction between age and ethnicity.

Table 3.

The estimated coefficients and associated standard errors based on model (5)

	Estimate	Standard error	z value	$\mathrm{P}\mathrm{r}(>|\mathrm{z}\left|\right)$
(Intercept)	3.028	0.842	3.597	0.000
Overweight	0.261	0.141	1.846	0.065
Education2	−0.096	0.217	−0.441	0.659
Education3	−0.001	0.216	−0.006	0.995
Education4	−0.361	0.210	−1.723	0.085
Age	−0.034	0.016	−2.100	0.036
Ethnicity (Non-Hispanic)	−2.353	0.860	−2.736	0.006
Age:ethnicity (Non-Hispanic)	0.036	0.017	2.158	0.031

4. Simulation Studies

In this section, we present simulation results to evaluate the finite sample performance of the proposed method and compare with Stukel test $\left(T_n^{\text{ST}}\right)$ and the proposed test $\left(T_n\right)$. In the following, all simulation results were based on 1000 replications. For each replication, the bootstrap process was repeated 1000 times. The nominal level was set to be 0.05 and 0.1.

Example 1 In this example, we generated data based on the real data example we analyzed in the previous section. That is, we simulated data based on the model:

$$\begin{gathered} \text{logit}\left\{p\right(y=1\left)\right\}={\beta }_1\text{overweight}+{\beta }_2\text{education}+{\beta }_3\text{age} \\ +{\beta }_4\text{ethnicity}+C{\beta }_5\text{age}*\text{ethnicity}, \end{gathered}$$

where the covariates are the observations in the telehealth study and ${\beta }_j$’s are set to the estimated values. $C$ varies from 0 to 1.5 with an increment of 0.25. $C=0$ corresponds to no interaction, while $C=1$ somewhat represents the situation in analysis of the telehealth study, but an interaction between age and ethnicity truly exists there. We repeated 1000 times for each $C$ case and counted the rejection proportion among these 1000 replications. The proportion rates based on the two test against $C$ are presented in Fig. 2. The Stukel test cannot maintain the type I errors, and the power (for $C\ne 0$) is very low. In contrast, the rejection proportion of the proposed test $T_n$ remains close to the nominal level, and the power increases with larger $C$. The advantage of the proposed test is evident.

Fig. 2.

Results for Example 1. Rejection proportions of the two tests against different $C$ values:

■ (the proposed test) and

● (the Stukel test) with the nominal levels 0.05 (left) and 0.1 (right)

Example 2 We generate the data from the following logistic models:

$$P\left(Y=1∣X\right)=\frac{1}{1+\text{exp}\left\{-X^{\mathrm{\top }}\mathit{\beta }+C\cdot D\left(X\right)\right\}},$$

where $D\left(X\right)={\left(X_1{X}_3\right)}^2+{\left\{X_2\left(1-X_3\right)\right\}}^2,X_1~\text{Uniform}[-\mathrm{1,1}]$, $X_2~\text{Normal}(0,1)$, and $X_3~\mathit{\text{Bin}}\left(0.5\right)$. $C$ ranges from 0 to 5 with increment 1.

We calculated the proportions of times the null hypothesis rejected among the 1000 replicates. They are the empirical sizes under the null hypothesis (i.e., $C=0$) and the empirical powers under the alternative hypothesis (i.e., $C\ne 0$). We show the rejection proportions of the tests in Fig. 3. Again, in this example, the performance of the proposed test $T_n$ is very promising in the sense that the empirical sizes of $T_n$ are closes to the nominal levels and the empirical powers increase with the sample size and distance away from the null hypothesis. On the other hand, the empirical powers of $T_n^{\text{ST}}$ are almost consistently smaller than those of $T_n$ for all $C$ values, even though the test controls the empirical sizes well. However, $T_n^{\text{ST}}$’s performance becomes better with larger sample sizes and close to that of the proposed test.

Fig. 3.

Results for Example 2. Rejection proportions of the two tests against different $C$ values:

■ (the proposed test) and

● (the Stukel test) with the nominal levels 0.05 (left) and 0.1 (right) for two sample sizes 100, 150 (downwards)

5. Discussion

The proposed method is applicable for checking goodness-of-fit of logistic regression in analysis of any biomedical data with binary response. Given its broad framework, weak assumptions, and stable numerical performance, we expect that the proposed procedure should be very useful and helpful in detecting potential lack-of-fitting.

The test is not limited to the logistic models. It would be applicable to generalized linear models with slightly extra effort to establish associated theoretical results. But numerical performance may depend on the distributions.

Theorem 2.1 establishes an asymptotic distribution for the proposed statistic under the null hypothesis. However, the asymptotic distribution contains several terms that depend on the data distribution and need to be estimated. This brings challenges and difficulties in calculating the critical value. We, therefore, adopt a MBB resampling strategy to avoid the potential difficulties. This strategy is generally needed/suggested to calculate critical values for nonparametric tests [18, 19]. However, this resampling strategy brings the concern of computational burden (especially for simulation studies). But the cost is endurable. For instance, it spends about 2.5 h to finish the simulation study for a setting. Like all other tests, the projection-based test may have low power to detect the omission of the main effect for a continuous variable. Further efforts are needed to explore the causes and to develop effective statistics. It is also worth pointing out that the proposed projection test may not be robust to outliers if the estimator of $\mathit{\beta }$ is not robust because its deficiency will be inherited. Integrating projection and robustness to develop a robust based projection statistic is interesting and warrants a further investigation, but it is beyond the scope of this paper.

Our procedure was developed for a fixed dimension of covariates. Reference [20] studied the same topic when the dimension is divergent. But the MBB procedure does not work well for their settings; instead, they suggested an alternative resampling algorithm; that is, they generated a sample $\left\{Y_i^{*},i=1,\dots ,n\right\}$ of independent Bernoulli random variables with the probability of success $p_i^{*}$, which was generated from $g\left(X_i^{\mathrm{\top }}{\hat{\beta }}_n\right)$. However, the performance of their algorithm is worse than that of the MBB procedure for the current settings. We will investigate why their algorithm is not applicable here.

The potential interaction between age and ethnicity discovered in this study may not be the universal feature, and different patterns may be observed in different studies. For instance, a telehealth study of DC Cohort participants using the same COVID survey found differing significant findings, including that females were more likely to utilize telehealth compared to males, which can be explained by use of an alternate study end date and additional variables in the model [21]. Given the complexities of the studies, it is also unclear if the uncovered pattern will stay similar in future work.

Appendix: A Sketch Proof of Theorem 1

By applying the injection theorem, we can establish the following representative [16]:

$${\widehat{\mathit{\beta }}}_n-{\mathit{\beta }}_0={\left\{{\tilde{Q}}_n\left({\widehat{\mathit{\beta }}}_n\right)\right\}}^{-1}{L}_n\left({\mathit{\beta }}_0\right),$$ (6)

where ${\tilde{Q}}_n\left(\mathit{\beta }\right)={\int }_0^1{Q}_n\left({\mathit{\beta }}_0+s\left(\mathit{\beta }-{\mathit{\beta }}_0\right)\right)ds$ and $Q_n\left(\mathit{\beta }\right)=\partial L_n\left(\mathit{\beta }\right)/\partial {\mathit{\beta }}^{\mathrm{\top }}$.

A direct manipulation based on the definition of $M_n(u,W)$ yields

$$\begin{gathered} M_n(u,W)=\frac{1}{\sqrt{n}}\sum _{i=1}^n\left\{Y_i-g\left(X_i^{\mathrm{\top }}{\widehat{\mathit{\beta }}}_n\right)\right\}I\left(X_i^{\mathrm{\top }}W\le u\right) \\ =\frac{1}{\sqrt{n}}\sum _{i=1}^n{\epsilon }_{i}I\left(X_i^{\mathrm{\top }}W\le u\right)-\frac{1}{\sqrt{n}}\sum _{i=1}^n\left\{g\left(X_i^{\mathrm{\top }}{\widehat{\mathit{\beta }}}_n\right)-g\left(X_i^{\mathrm{\top }}{\mathit{\beta }}_0\right)\right\}I\left(X_i^{\mathrm{\top }}W\le u\right) \\ ≔B_{n1}(u,W)-B_{n2}(u,W). \end{gathered}$$ (7)

The second term can be expressed as

$$B_{n2}\left(u,W\right)=\frac{1}{\sqrt{n}}\sum _{i=1}^n{X}_i^{\mathrm{\top }}\left({\widehat{\mathit{\beta }}}_n-{\mathit{\beta }}_0\right)I\left(X_i^{\mathrm{\top }}W\le u\right){\int }_0^1{g}^{\prime }\left(X_i^{\mathrm{\top }}{\mathit{\beta }}_0+s\left(X_i^{\mathrm{\top }}\left({\widehat{\mathit{\beta }}}_n-{\mathit{\beta }}_0\right)\right)\right)ds.$$

This term can be re-expressed as follows by substituting ${\widehat{\mathit{\beta }}}_n-{\mathit{\beta }}_0$ given in (6) and applying the mean-value theorem.

$$\begin{gathered} B_{n2}(u,W)=\frac{1}{\sqrt{n}}\sum _{i=1}^n{X}_i^{\mathrm{\top }}{\left\{{\tilde{Q}}_n\left({\widehat{\mathit{\beta }}}_n\right)\right\}}^{-1}\sum _{j=1}^n{X}_j{\epsilon }_{j}I\left(X_i^{\mathrm{\top }}W\le u\right)g^{\prime }\left(X_i^{\mathrm{\top }}{\tilde{\mathit{\beta }}}_n\right) \\ =\frac{1}{\sqrt{n}}\sum _{j=1}^n\left[\sum _{i=1}^n{X}_i^{\mathrm{\top }}{\left\{{\tilde{Q}}_n\left({\widehat{\mathit{\beta }}}_n\right)\right\}}^{-1}I\left(X_i^{\mathrm{\top }}W\le u\right)g^{\prime }\left(X_i^{\mathrm{\top }}{\tilde{\mathit{\beta }}}_n\right)\right]X_j{\epsilon }_j, \end{gathered}$$

where ${\tilde{\mathit{\beta }}}_n$ is a vector such that $X_i^{\mathrm{\top }}{\tilde{\mathit{\beta }}}_n$ lies between $X_i^{\mathrm{\top }}{\widehat{\mathit{\beta }}}_n$ and $X_i^{\mathrm{\top }}{\mathit{\beta }}_0$. A direct simplifications yields

$$\begin{gathered} \underset{j}{\text{max}}\left|\frac{1}{n}\sum _{i=1}^n\left[X_i^{\mathrm{\top }}{\left\{{\tilde{Q}}_n\left({\widehat{\mathit{\beta }}}_n\right)\right\}}^{-1}I\left(X_i^{\mathrm{\top }}W\le u\right)g^{\prime }\left(X_i^{\mathrm{\top }}{\tilde{\mathit{\beta }}}_n\right)\right.\right. \\ \left.-E\left\{X^{\mathrm{\top }}{\mathrm{\Sigma }}_{\beta }^{-1}I\left(X^{\mathrm{\top }}W\le u\right)g^{\prime }\left(X^{\mathrm{\top }}{\mathit{\beta }}_0\right)\right\}\right]X_j\mid \to 0. \end{gathered}$$

Thus,

$$B_{n2}\left(u,W\right)=\frac{1}{\sqrt{n}}\mathrm{\Gamma }\left(u\right)\sum _{i=1}^n{\epsilon }_i{X}_i+o_p\left(1\right).$$ (8)

Combining (7) and (8) yields.

$$M_n(u,W)=\frac{1}{\sqrt{n}}\sum _{i=1}^n\left\{I\left(X_i^{\mathrm{\top }}W\le u\right)-\mathrm{\Gamma }\left(u\right)X_i\right\}{\epsilon }_i+o_p\left(1\right).$$ (9)

$I\left(X^{\mathrm{\top }}W\le u\right)$ is monotonic with respect to $u$. By Lemma 9.10 of [22], the function class $\left\{I\left(X^{\mathrm{\top }}W\le u\right):u\in {\mathbb{R}}^1\right\}$ is a VC-class. Analogously the function class $\left\{\mathrm{\Gamma }\left(u\right):u\in {\mathbb{R}}^1\right\}$ is a VC-class as well. By Lemma 2.6.8 of [23], the function classes $\left\{\epsilon I\left(X^{\mathrm{\top }}W\le u\right):u\in {\mathbb{R}}^1\right\}$ and the class $\left\{\mathrm{\Gamma }\left(u\right)\epsilon X:u\in {\mathbb{R}}^1\right\}$ are all VC-class. Therefore, the function class $\left\{{\mathrm{\Psi }}_u(u,y,\epsilon ,w):u\in {\mathbb{R}}^1\right\}$ is a VC-class by Lemma 9.17 of [22]. We can take the envelope function as $\left|\epsilon \right|+\mathrm{E}(\Vert X\Vert )\left|\epsilon \right|{\mathrm{\Sigma }}_{\beta }^{-1}\left|X\right|$. By Theorem 2.6.7, Theorem 2.5.2 of [23], we can prove that for given $W=w$, the estimated empirical process $M_n(u,W)$ converges weakly to $M\left(u\right)$ in the Skorohod space $S[-\mathrm{\infty },\mathrm{\infty }]$. By the continuous mapping theorem, we can prove the result for $T_n$.