Deletion Diagnostics for Alternating Logistic Regressions

Abstract

Deletion diagnostics are introduced for the regression analysis of clustered binary outcomes estimated with alternating logistic regressions, an implementation of generalized estimating equations (GEE) that estimates regression coefficients in a marginal mean model and in a model for the intracluster association given by the log odds ratio. The diagnostics are developed within an estimating equations framework that recasts the estimating functions for association parameters based upon conditional residuals into equivalent functions based upon marginal residuals. Extensions of earlier work on GEE diagnostics follow directly, including computational formulae for one-step deletion diagnostics that measure the influence of a cluster of observations on the estimated regression parameters and on the overall marginal mean or association model fit. The diagnostic formulae are evaluated with simulations studies and with an application concerning an assessment of factors associated with health maintenance visits in primary care medical practices. The application and the simulations demonstrate that the proposed cluster-deletion diagnostics for alternating logistic regressions are good approximations of their exact fully iterated counterparts.

1. Introduction

Many questions in medical research involving association structure among correlated binary responses are suitably addressed with marginal regression models. Consider, for example, cross-sectional observational medical practice data where patient outcomes are nested within physicians that are nested within practices. An analysis of such data described in Preisser and Qaqish (1996) used generalized estimating equations (GEE) (Liang and Zeger, 1986) to estimate the effects of explanatory variables on the population-averaged probability of whether or not a patient made a health maintenance visit in the prior year. An additional question that may be posed involves modeling the marginal within-practice association structure of the response, and, in particular, characterizing the degree of association within physicians and within practices. To address both questions, an approach that simultaneously fits models to the marginal probability and the within-cluster association is needed.

For correlated binary data, pairwise odds ratios (Lipsitz et. al., 1991) are a natural choice for modeling within-cluster association. When cluster sizes (eg., the number of patients sampled per practice) are small, second-order generalized estimating equations (GEE2) provide estimates of pairwise odds ratios with good efficiency (Zhao and Prentice, 1990; Liang et. al., 1992). However, GEE2 is not computationally feasible when cluster sizes are large as in the medical practice data where they range from 19 to 197. Alternating logistic regressions (ALR), an implementation of GEE for the regression analysis of clustered binary data (Carey et. al., 1993) provides more efficient estimation of association parameters than first-order GEE but less efficiency than GEE2, with the degree of efficiency loss depending upon the underlying multivariate distribution (Qaqish et. al., 2011). In situations with small clusters, estimation by ALR is nearly as efficient as GEE2 (Carey et al., 1993; Lipsitz SR and Fitzmaurice, 1996).

Alternating logistic regressions have been applied in diverse settings including epidemiological investigations and randomized community trials (see Reboussin et al., 2011 and references therein). A practical limitation, however, is that there currently do not exist influence diagnostics for ALR. In the medical practice data example, it is natural to assess whether data from individual practices have a large influence on the pairwise odds ratio estimates of within-physician and within-practice clustering. In addition to summarizing the magnitude of clustering, pairwise odds ratio estimates directly influence design effects and are useful in sample size estimation for planning future studies to be analyzed with ALR (Katz et al., 1993; Reboussin et al., 2011). Identifying influential practices can guide the range of values of pairwise odds ratios to be used in power analyses, such as in the planning of a randomized cluster trial to change medical practice routines with the aim of increasing health maintenance visits. Diagnostics can also identify clusters that are subsequently discovered to be different from others with respect to a population of interest, which may justify their omission in data analysis.

This article proposes formulae and algorithms for estimating the effect of the deletion of a cluster of observations on the regression parameters in marginal models for means and intracluster associations. The ‘one-step’ formulae, which are equivalent to taking the difference after performing an extra iteration of the model estimation procedure following cluster-deletion, can provide dramatic reductions in computation time compared to their fully iterated ‘exact’ counterparts. Section 2 describes the estimating equations procedure and introduces the formulae to estimate the change in parameter estimates upon deletion of a cluster. The diagnostics are developed within an estimating function framework that recasts the ALR estimating functions for association parameters based upon conditional residuals (Carey et al., 1993) into equivalent functions based upon marginal residuals (Qaqish et al., 2011). Extensions of first order GEE diagnostics (Preisser and Qaqish, 1996) follow directly, including computational formulae for one-step deletion diagnostics that measure the influence of a cluster of observations on the estimated regression parameters and on the overall marginal mean or association model fit. Section 3 presents two simulation studies to evaluate the performance of the diagnostics. In section 4, the formulae are applied to the medical practice data. Section 5 contains concluding remarks.

2. Statistical methods

2.1. Alternating logistics regressions based upon marginal residuals

The development of deletion diagnostics in this paper is based upon a new representation of the ALR method through marginal residuals proposed by Qaqish et al. (2011). Let Y_i be the response vector for the ith cluster where Y_i = (Y_i1, …, Y_ini)′ is the vector of responses from the n_i observations in the ith cluster, i = 1, …, K. Let μ_i be the vector of population marginal means, E[Y_i] = μ_i. A generalized linear model is $g({\mu }_{ij})={\mathbf{X}}_{ij}^{\prime }\mathit{\beta }$ where g(·) is the link function and X_ij = (x_0ij, x_1ij, …, x_p−1,ij)′ is the p × 1 vector of covariates for the j–th observation in the i–th cluster. Estimation by the ALR procedure is performed by iteratively solving two estimation equations, one for the marginal mean model parameters β, and the other for the marginal bivariate association parameters α. The estimating equations for β are:

$$\sum _{i=1}^K{\mathbf{D}}_i^{\prime }{\mathbf{V}}_i^{-1}({\mathbf{Y}}_i-{\mathit{\mu }}_i)=0$$ (1)

where D_i = ∂μ_i/∂β, ${\mathbf{V}}_i(\mathit{\beta },\mathit{\alpha })=\mathbf{\text{diag}}({\sigma }_{ijj}^{1/2}){\mathbf{R}}_i(\mathit{\alpha })\mathbf{\text{diag}}({\sigma }_{ijj}^{1/2})$, σ_ijj = μ_ij(1 − μ_ij),and R_i(α) is a working correlation matrix.

The second set of estimating equations correspond to α. Let m_i = n_i(n_i − 1)/2 and j and k index observations within a cluster. Let W_ijk = Y_ijY_ik and define μ_ijk = E[W_ijk] = pr(Y_ij = Y_ik = 1). The dependence or association between Y_ij and Y_ik can be represented by the pairwise odds ratio (Lipsitz et. al., 1991; Liang et. al., 1992; Carey et. al., 1993)

$${\psi }_{ijk}=\frac{{\mu }_{ijk}(1-{\mu }_{ij}-{\mu }_{ik}+{\mu }_{ijk})}{({\mu }_{ij}-{\mu }_{ijk})({\mu }_{ik}-{\mu }_{ijk})}.$$

A log pairwise odds ratio model is specified for the association,

$$\text{log}[{\psi }_{ijk}({\mu }_{ij},{\mu }_{ik},{\mu }_{ijk})]={\mathbf{Z}}_{ijk}^{\prime }\mathit{\alpha }$$

where Z_ijk = (z_0ijk, z_1ijk, …, z_q−1,ijk)′ is a covariate q-vector associated with the pair (Y_ij, Y_ik), and α is a vector of association parameters. The method of Qaqish et al. (2011) uses an m_i-vector T_i with elements T_ijk obtained as the residuals from the linear regression of W_ijk on Y_ij and Y_ik. Specifically,

$$T_{ijk}=W_{ijk}-\{{\mu }_{ijk}+b_{ijk:j}(Y_{ij}-{\mu }_{ij})+b_{ijk:k}(Y_{ik}-{\mu }_{ik})\},$$

where

$$\begin{array}{c}\hfill b_{ijk:j}={\mu }_{ijk}(1-{\mu }_{ik})({\mu }_{ik}-{\mu }_{ijk})/d_{ijk},\hfill \\ \hfill b_{ijk:k}={\mu }_{ijk}(1-{\mu }_{ij})({\mu }_{ij}-{\mu }_{ijk})/d_{ijk},\hfill \end{array}$$

$d_{ijk}={\sigma }_{ijj}{\sigma }_{ikk}-{\sigma }_{ijk}^2$, and σ_ijk = cov(Y_ij, Y_ik) = μ_ijk − μ_ijμ_ik. The reformulated ALR estimating equations based upon marginal residuals are defined by

$$U_{\mathit{\alpha }}=\sum _{i=1}^{K}E\left[\frac{-\partial {\mathbf{T}}_i\prime }{\partial \mathit{\alpha }}\right]{\mathbf{P}}_i^{-1}{\mathbf{T}}_i=\sum _{i=1}^K{\mathbf{C}}_i^{\prime }{\mathbf{P}}_i^{-1}{\mathbf{T}}_i$$ (2)

where P_i = diag{υ_ijk} and

$${\upsilon }_{ijk}≔\text{var}(T_{ijk})=\frac{{\mu }_{ijk}({\mu }_{ij}-{\mu }_{ijk})({\mu }_{ik}-{\mu }_{ijk})(1-{\mu }_{ij}-{\mu }_{ik}+{\mu }_{ijk})}{{\mu }_{ij}{\mu }_{ik}(1-{\mu }_{ij}-{\mu }_{ik}+2{\mu }_{ijk})-{\mu }_{ijk}^2}.$$

Expression (2) is equivalent to the ALR estimating equations based upon conditional residuals given by (7) of Carey et. al. (1993). Qaqish et al. (2011) allow P_i to be non-diagonal in a procedure they call orthogonalized residuals, thereby generalizing the ALR procedure and increasing efficiency. They show, following standard arguments (Liang and Zeger, 1986; Prentice, 1988), the asymptotic distribution of K^1/2 (θ̂ − θ), where θ = (β, α)′, is multivariate Gaussian with mean zero. A property of (2) is that, unlike the case based upon conditional residuals, the empirical variance estimator of the asymptotic covariance matrix corresponding to the association model is invariant to permutations of the i-th subject’s response vector Y_i (current versions of ALR software, e.g., SAS PROC GENMOD v. 9.2, average the variance estimates based on a given ordering and its reversal). Kuk (2004) proposed a modified symmetrized version of the ALR equations based upon conditional residuals that have permutation invariance for the standard errors. By, Qaqish, and Preisser (2008) provide an R package for ALR and orthogonalized residuals, which includes the regression diagnostics proposed in the next section. A SAS macro is available at http://www.bios.unc.edu/~qaqish/software.htm.

2.2. Cluster–deletion diagnostics

The regression diagnostics proposed in this section are computationally fast formulae because all matrix components of the diagnostics are available at convergence of the iteratively reweighted least squares algorithm. They are one-step deletion diagnostics because the computational formulae are equivalent to deleting the cluster and computing one more iteration of (1) and (2).

Let β̂_[i] denote the estimate with the i-cluster deleted. The one-step GEE deletion diagnostic to approximate β̂ − β̂_[i] is

$${\mathrm{D}\mathrm{B}\mathrm{E}\mathrm{T}\mathrm{A}\mathrm{C}}_i={(\mathbf{D}\prime {\mathbf{V}}^{-1}\mathbf{D})}^{-1}{\mathbf{D}}_i^{\prime }{\mathbf{V}}_i^{-1}{({\mathbf{I}}_{n_i}-{\mathbf{H}}_{1i})}^{-1}({\mathbf{Y}}_i-{\mathit{\mu }}_i)$$ (3)

where $\mathbf{D}=({\mathbf{D}}_1^{\prime },\dots ,{\mathbf{D}}_K^{\prime })\prime$, V = blockdiag(V1, …, V_K), I_d represents the identity matrix of dimension d and

$${\mathbf{H}}_{1i}={\mathbf{D}}_i{(\mathbf{D}\prime {\mathbf{V}}^{-1}\mathbf{D})}^{-1}{\mathbf{D}}_i^{\prime }{\mathbf{V}}_i^{-1}.$$ (4)

Note that ${(\mathbf{D}\prime {\mathbf{V}}^{-1}\mathbf{D})}^{-1}={({\sum }_{i=1}^K{\mathbf{D}}_i^{\prime }{\mathbf{V}}_i^{-1}{\mathbf{D}}_i)}^{-1}$ is the expression used to compute the model-based variance estimator for β̂. Preisser and Qaqish (1996) gave a proof for a formula that is equivalent to (3). Others (e.g., Ziegler and Arminger, 1996, and Vens and Ziegler, 2012) have also discussed cluster-deletion for assessing the influence of clusters on the GEE estimator β̂. Hammill and Preisser (2006) proposed (3) in light of its connections, on a matrix component basis, to (1) and they showed its algebraic equivalency to expression (5) of Preisser and Qaqish (1996). Expression (4) is the leverage matrix (corresponding to β) for cluster i (Mancl and DeRouen, 2001). The leverage of a cluster may be defined as the trace of H_1i, that is the sum of the diagonal elements which may individually be viewed as leverages of observations. Preisser and Qaqish (1996) gave a slightly different formula for the cluster leverage matrix.

Using arguments similar to their derivation of DBETAC_i, the one-step formula to approximate α̂ − α̂_[i] is

$${\mathrm{D}\mathrm{A}\mathrm{L}\mathrm{P}\mathrm{H}\mathrm{A}\mathrm{C}}_i={(\mathbf{C}\prime {\mathbf{P}}^{-1}\mathbf{C})}^{-1}{\mathbf{C}}_i^{\prime }{\mathbf{P}}_i^{-1}{({\mathbf{I}}_{m_i}-{\mathbf{H}}_{2i})}^{-1}{\mathbf{T}}_i$$ (5)

where $\mathbf{C}=({\mathbf{C}}_1^{\prime },\dots {\mathbf{C}}_K^{\prime })\prime$, P = blockdiag(P₁, …, P_K), and

$${\mathbf{H}}_{2i}={\mathbf{C}}_i{(\mathbf{C}\prime {\mathbf{P}}^{-1}\mathbf{C})}^{-1}{\mathbf{C}}_i^{\prime }{\mathbf{P}}_i^{-1}$$ (6)

is the cluster leverage matrix corresponding to α. Following Preisser and Qaqish (1996), standardized versions of the diagnostics denoted DBETACS_i and DALPHACS_i may be obtained by dividing the components of DBETAC_i and DALPHAC_i by their respective standard errors. All computations are carried out be inserting estimates β̂ and α̂ into the formulae above.

It is worth mention that computation of (5) is non-trivial for large cluster sizes. For the medical practice data analyzed in section 4, max(n_i) = 197, and, thus, max(m_i) = 19, 306. Fortunately, due to its special structure, the matrix in (5) of this dimension involving H_2i can be easily inverted using an algorithm based upon the Sherman-Morrison-Woodbury formula (Sherman and Morrison, 1950). Details of the computational approach are provided by Preisser, Qaqish and Perin (2008).

The assessment of the influence of a cluster of observations on the overall model fit may be carried out with diagnostic measures that are extensions of Cook’s distance for linear regression (Cook and Weisberg, 1982). Cluster level Cook’s Distance for β is defined as for GEE1 (e.g., Hammill and Preisser, 2006):

$${\mathrm{D}\mathrm{C}\mathrm{L}\mathrm{S}}_{\mathit{\beta },i}(p)=(\mathit{\beta ̂}-{\mathit{\beta ̂}}_{[i]})\prime {\widehat{\text{var}}}^{-1}(\mathit{\beta ̂})(\mathit{\beta ̂}-{\mathit{\beta ̂}}_{[i]})/p.$$ (7)

Analogously, Cook’s Distance describing the influence of the i-th cluster on the overall fit of the model for α is defined as

$${\mathrm{D}\mathrm{C}\mathrm{L}\mathrm{S}}_{\mathit{\alpha },i}(q)=(\mathit{\alpha ̂}-{\mathit{\alpha ̂}}_{[i]})\prime {\widehat{\text{var}}}^{-1}(\mathit{\alpha ̂})(\mathit{\alpha ̂}-{\mathit{\alpha ̂}}_{[i]})/q.$$ (8)

Following Ziegler et. al. (1998), we use empirical variance estimators in equations (7) and (8). Our measures are similar to the measures implemented by Ziegler et. al. (1998) in the context of modeling within-cluster correlations using the GEE approach of Prentice (1988). In that context, Preisser and Perin (2007) provided computationally fast formula for the influence of the i-th cluster on the overall fit of the correlation model. A similar computationally fast formula for the influence of the i-th cluster on the overall fit of the within-cluster log odds ratio model, estimated with alternating logistic regressions, may be obtained by substituting DALPHAC_i for α̂ − α̂_[i].

Interpretations for cluster diagnostics are not straightforward when cluster sizes vary. Generally, one might expect that larger clusters tend to have larger influence, so plots of cluster diagnostics against cluster size are recommended to assess their influence.

3. Simulation studies

3.1. The performance of the one-step approximation

The first of two simulation studies was conducted to determine the extent to which the clusters with the most extreme exact cluster Cook’s distance are identified by the one-step cluster Cook’s distance. As in a study on the performance of Cook’s Distance in the generalized linear mixed model (Xiang et al. 2002), the simulation study assessed the diagnostics ability to identify the clusters with the largest and second largest exact Cook’s distances. We have two a priori expectations. First, we expect the one-step approximation, to a large extent, will identify the same clusters as those identified by the exact cluster Cook’s distance. Second, we expect the probability of identifying the same clusters to increase as the value of the exact cluster Cook’s distance increases. The simulation experiment was based upon 500 data sets generated from the following model using the algorithm of Qaqish (2003) described in the appendix:

$$\text{logit}({\mu }_{ij})={\beta }_0+{\beta }_1{x}_{1ij}+{\beta }_2{x}_{2ij}$$ (9)

$$\text{log}{\psi }_{ijk}={\alpha }_0+{\alpha }_1{z}_{1ijk}+{\alpha }_2{z}_{2ijk}$$ (10)

where

$$\mathit{\beta }=\left[\begin{array}{c}\hfill {\beta }_0\hfill \\ \hfill {\beta }_1\hfill \\ \hfill {\beta }_2\hfill \end{array}\right]=\left[\begin{array}{c}\hfill -0.8\hfill \\ \hfill 0.27\hfill \\ \hfill 0.20\hfill \end{array}\right]\text{and}\mathit{\alpha }=\left[\begin{array}{c}\hfill {\alpha }_0\hfill \\ \hfill {\alpha }_1\hfill \\ \hfill {\alpha }_2\hfill \end{array}\right]=\left[\begin{array}{c}\hfill 1.05\hfill \\ \hfill 0.35\hfill \\ \hfill -0.35\hfill \end{array}\right].$$

and x_1ij, x_2ij, z_1ijk, and z_2ijk are continuous covariates: x_1ij = [2(i − 1)/(K − 1)] − 1 is a cluster level covariate taking equally spaced values in the interval [−1,1], i = 1, …, K; x_2ij = [2(j − 1)/(n − 1)] − 1 is an observation level covariate taking equally spaced values in the interval [−1,1] where n denotes the number of observations in each cluster, n = n_i for all i; j = 1, …, n; z_1ijk = x_1ij; and z_2ijk = |x_2ij − x_2ik|. These parameter values were chosen so as to induce response vectors with positive within-cluster association that decreased over time, akin to autoregressive correlation in longitudinal data settings. For each replication, we simultaneously fit models (9) and (10) with the ALR estimating procedure given by equations (1) and (2). Within a replication, for each cluster, we computed both exact cluster Cook’s distance and one-step approximated cluster Cook’s distance for both β and α. Note that the computation of exact cluster Cook’s distance for all clusters required an additional K applications of ALR per replication to obtain fully iterated parameter estimates after deletion of a single cluster. Due to the computational intensity of the experiment, only the combination of (K = 100, n = 20) was considered, requiring a total of 500(K+1) =50,500 applications of ALR (K runs for single cluster-deletion datasets to compute exact cluster Cook’s distances and a single run on the full data of K clusters to compute the one-step cluster Cook’s distances).

The results were as follows: 88% of the time, the one-step formula (8) correctly identified the most influential cluster on α; 82% of the time, diagnostic (8) correctly identified the top two most influential clusters on β (possibly in the reversed order); 60% of the time, the one-step formula (7) correctly identified the most influential cluster on β; 31% of the time, the diagnostic (7) correctly identified the top two most influential clusters on β. While the latter result was not very good, we note that DCLS_β,i was able to identify at least one of the two most influential clusters 88% of the time.

To determine if the probability of correctly identifying the most influential clusters with DCLS_β,i increased as the influence of those clusters increased, as measured by the magnitude of the exact fully-iterated Cook’s distance, two logistic regressions were carried out. First, the binary indicator for whether the cluster with largest exact cluster Cook’s distance matched with the cluster with the largest DCLS_β,i was regressed on the exact cluster Cook’s distance. A significant monotonically increasing relationship (figure 1a) shows that for a value of Cook’s distance at the third quantile (with respect to the 500 simulated largest exact cluster Cook’s distance values), the probability of detecting the most influential cluster is approximately 75%. Next, the binary indicator for whether the two clusters with largest exact cluster Cook’s distance were the same as the two clusters with largest DCLS_β,i was regressed on the 2nd largest exact cluster Cook’s distance. A significant monotonically increasing relationship (figure 1b) shows that for values of the second largest Cook’s distance at the 75th, 90th, and 95th percentiles, respectively, the estimated probabilities of detecting the two most influential clusters are approximately 35%, 42%, and 47%, respectively. A more extensive report of the simulations is available as a technical report (Preisser, By, and Qaqish, 2008).

Figure 1.

Detecting the most influential clusters with respect to β. 1(a)–probability of detecting the cluster with the largest exact Cook’s distance. 1(b)–probability of detecting the two clusters with largest exact Cook’s distance. Q₁, Q₂, and Q₃ represent the appropriate quantiles of exact cluster Cook’s distance from the 500 simulations with K=100 and n=20. P₉₀ and P₉₅ denote the 90-th and 95-th percentiles respectively.

3.2. Responsiveness of diagnostics to contamination models

The aim of the second simulation study is to consider the distribution of extreme cluster Cook’s distance for β and α under binary response contaminated data models. The simulation study investigates whether the Cook’s distance measures are responsive to contamination of the response data? Second, if it is responsive to contamination, does it behave in some predictable way relative to the Cook’s distance for the uncontaminated data? We expect to see a shift in the distribution Cook’s distance under contamination relative to the uncontaminated data. Furthermore, we expect this shift to grow (to a point) as the level of contamination increases.

Data are generated from the models (9) and (10) with constant cluster size n using the same values of β and α from the previous section. Contamination is considered under two scenarios: (1) random contamination (RC) and (2) cluster concentrated (CC) contamination. Under random contamination, each of the Kn observations is contaminated with probability p_c. Letting Y_ij,c be the contaminated observation and Y_ij be the original observation, the contamination is done as follows:

$$Y_{ij,c}=\{\begin{array}{cc}\hfill 1-Y_{ij}& \text{with probability}{p}_c\hfill \\ \hfill Y_{ij}& \text{with probability}1-p_c\hfill \end{array}$$

Under cluster contamination, each cluster is first chosen with probability 2p_c. Once the cluster is chosen, each of the n observations within the chosen cluster is contaminated with probability 0.50. The simulation experiment was conducted to investigate 63 scenarios: K = {50, 100, 200}, n = {5, 20, 50}, p_c = {0, 0.02, 0.05, 0.10} and cc = {RC,CC} (when p_c > 0).

To address whether the one-step cluster Cook’s distance diagnostics for β and α are sensitive to contamination, the empirical distributions of their largest order statistics under models of contamination were examined relative to empirical distributions of the largest Cook’s statistic when their was no contamination. This was accomplished visually with QQ plots for each combination of K and n. Figure 2 compares the QQ plots of the largest order statistic for cluster Cook’s distance for α under contaminated data relative to uncontaminated. For cluster size 5 (lower panels of Figure 2), there is not any observable difference between the empirical distribution of the cluster Cook’s distance under contamination relative to no contamination. However, for cluster size n = 50 (as well as for n = 20, not shown), the spacing (or clear separation) in the loess fits, and the fact that the curves generally lie above the 45 degree line, indicates that the distibution of Cook’s distance under contamination is shifted to the right, and that their values increase monotonically with the level of contamination. Plots for K = 200 (not shown) are similar to plots for K = 100 for a given value of n. Thus, at least for α, cluster Cook’s distance under random contamination behaves in the manner that is expected by shifting to the right as the level of contamination increases. The second largest, third largest, and fourth largest cluster Cook’s distance for α also exhibit this behavior (plots not shown).

Figure 2.

QQ plot of cluster Cook’s distance for α. Vertical axis denotes cluster Coook’s distance under random contamination. C2, C5 and C10 denotes 2, 5, and 10 percent contamination respectively. The horizontal axis denotes cluster Cook’s distance under no contamination.

This behavior is not consistent in general. In fact, for every other situation, the QQ plots either show that the distribution of the contaminated Cook’s distance is no different than the uncontaminated or that if a shift is present, it is shifting to the left as the contamination increases. For example, for β under random contamination and small cluster sizes (n = 5), the distribution has a tendency to shift further and further to the left as the level of contamination increases, opposite as was expected (plots not shown). For the other cluster sizes (n = 20 and n = 50) as they pertain to β under random contamination, there is no apparent difference between contaminated and uncontaminated distributions. A report of the entire simulation experiment, including exhaustive displays of QQ plots for both random and cluster-concentrated contamination cases, is available (By, Preisser and Qaqish, 2008).

4. Application of Diagnostics to Medical Practice Data

The proposed cluster deletion diagnostics are illustrated with medical practice data. In 1990–1991, chart review data were collected from a random sample of 3889 medical charts in 57 medical practices (clusters). The cluster sizes (number of patients per practice) ranged from 19 to 197 with a mean of 68. A logistic regression model was specified for the probability that the j-th patient in the i-th practice made at least one maintenance visit during the years 1990 and 1991. Preisser and Qaqish (1996) introduced and applied to the medical practice data computationally efficient formulae for cluster deletion diagnostics to identify practices with the largest influences on regression coefficients in the model for the marginal mean. ALR may be used to fit the same logistic regression model for the marginal mean while specifying an additional model for the within-practice association. The pairwise odds ratio model has the form of (10) where z_1ijk = 1 if patients j and k in practice i have the same physician (and z_1ijk = 0, otherwise); and z_2ijk = (n_i − 68)/50. Note α₀ is the log pairwise odds ratio of health maintenence visit for two patients from the same medical practice who saw different doctors for a practice with cluster size n_i = 68; α₁ is the change in the log pairwise odds ratio between two patients who saw the same doctor, relative to the association between two patients who saw different doctors; and α₂ is the change in the within-practice log pairwise odds ratio for two patients comparing two practices that differ in cluster size by 50 patients.

The first column of results in Table 1 shows the ALR parameter estimates applied to the full data set where all three association model covariates are statistically significant at the 0.05 level. Recall that the standard errors in Table 1, based upon the reformulated ALR of Qaqish et al. (2011) given in equation (2), are invariant to the ordering of a subjects’ responses. For a practice of mean cluster size, the estimated between-physician within-practice odds ratio is 1.71, and the within-physician odds ratio is 2.29. These associations decrease with increasing cluster size. It is natural to inquire whether certain practices have an undue influence on these estimates. The remaining columns of Table 1 show the fully iterated results obtained upon deleting selected clusters, suggesting that some clusters have a moderate influence on estimates of within-practice and within-physician clustering.

Table 1.

Model parameter estimates (est.) with empirical standard errors (se)^* for the logistic model of the marginal probability a patient made a health maintenance visit during the years 1990 and 1991 (β), and for the log odds ratio model of within-practice association (α). Data are from the North Carolina Early Cancer Detection Program at the Lineberger Comprehensive Cancer Center. Results are presented for the full data, and based upon selected cluster deletions.

	Full Data		Without #15		Without #19		Without #34
Parameter	est.	(se)	est.	(se)	est.	(se)	est.	(se)
β‡
INTERCEPT	−0.106	(0.173)	−0.070	(0.182)	−0.178	(0.170)	−0.155	(0.167)
NBRMDS	−0.034	(0.039)	−0.013	(0.043)	−0.018	(0.053)	−0.030	(0.039)
M3	0.249	(0.170)	0.340	(0.161)	0.244	(0.171)	0.228	(0.173)
SPECLTY	−0.078	(0.249)	−0.178	(0.263)	0.056	(0.250)	−0.075	(0.239)
MDAGE	−0.264	(0.064)	−0.297	(0.059)	−0.268	(0.066)	−0.227	(0.065)
MDSEX	0.424	(0.262)	0.511	(0.266)	0.476	(0.283)	0.472	(0.255)
MDFLU	−0.072	(0.099)	−0.123	(0.087)	−0.092	(0.104)	−0.073	(0.102)
PATAGE	−0.097	(0.034)	−0.103	(0.035)	−0.101	(0.035)	−0.103	(0.035)
BLACKPAT	−0.395	(0.123)	−0.413	(0.122)	−0.401	(0.124)	−0.426	(0.124)
MALEPAT	−0.411	(0.065)	−0.422	(0.067)	−0.431	(0.066)	−0.421	(0.068)
NOINSUR	−0.416	(0.119)	−0.422	(0.122)	−0.439	(0.123)	−0.393	(0.120)
ᇇ
INTERCEPT	0.538	(0.173)	0.613	(0.172)	0.524	(0.156)	0.410	(0.147)
SAMEMD	0.290	(0.112)	0.207	(0.098)	0.256	(0.107)	0.347	(0.113)
CLSIZE	−0.179	(0.062)	−0.178	(0.069)	−0.147	(0.072)	−0.139	(0.052)

^*standard errors are from the reformulated ALR based upon marginal residuals.

^‡NBRMDS = number of doctors in practice minus one; M3 = (The number of patients over 50 years old seen per day minus 15)/10. SPECLTY = doctor’s specialty: 0 if family or general practice, 1 if internal medicine; MDAGE = (doctor’s age in years minus 45)/10; MDSEX=1 if female, 0 if male; MDFLU = Doctor’s flu vaccination: 0 in the last two years, 1 if 3 to 5 years ago, 2 if never; PATAGE = (patient’s age in years - 65)/10; BLACKPAT = 1 if black, and 0 if white; MALEPAT = 1 if patient is male, and 0 if female; and NOINSUR = 1 if patient is not insured, and 0 if insured.

^‡‡SAMEMD = 1 if two patients have the same doctor, and 0 otherwise; CLSIZE is the size of the cluster centered at 68 scaled by 50.

Figure 3 presents cluster-deletion diagnostic statistics for medical practices for the ALR procedure given by (1) and (2). Plots (a), (b) and (c) depict the difference, given by the vertical distance between a point and the 45 degree line, between the fully-iterated standardized parameter estimate after removal of a medical practice (’exact’) and the approximate change given by DALPHACS_i, for α₀, α₁, and α₂, respectively. These plots show that the proposed one-step diagnostics provide good approximations of the exact change, as nearly all the points fall close to the 45 degree line. The most infuential clusters are identified in the figure, practice # 34 for α̂₀ and α̂₂ (Figures 3(a) and 3(c), respectively), and practice # 15 for α̂₁ (Figure 3(b)). Figure 3(d) shows the three clusters with the most influence on the overall fit of the within-practice association model; practice # 34 has the second greatest influence by this measure. Table 2 gives the actual values of the standardized cluster-deletion diagnostic statistics for selected practices. It is interesting that Preisser and Qaqish (1996) using DCLS_β,i (see their expression (9)) identified cluster # 5 as having the greatest influence among clusters on the overall fit of the marginal mean model. This was also true for the ALR analysis of the influence of clusters on β (not shown). However, as shown in Table 2, cluster # 5 was not particularly notable for its overall influence on the fit of model (10) despite its having the second largest influence on α₁. A file provided as supplementary material provides the SAS code used to obtain the “full data” results in Table 1 and one-step cluster-deletion diagnostics reported in Table 2 and Figure 3.

Figure 3.

Cluster deletion diagnostics for within-cluster association model. DALPHA’s and the exact deletion diagnostic for α in plots 3(a), 3(b), and 3(c) are standardized by the appropriate empirical standard errors.

Table 2.

One-step approximated cluster-deletion diagnostic values for selected clusters. The values are based upon the “full data” analysis in Table 1 using data from the North Carolina Early Cancer Detection Program. Ranks are shown in parentheses.

					DALPHACS*
Cluster #	ni		DCLSα‡		INTERCEPT		SAMEMD		CLSIZE
5	191	(2)	0.059	(7)	−0.121	(13)	0.367	(2)	−0.123	(17)
15	158	(3)	0.064	(5)	−0.197	(8)	0.417	(1)	−0.050	(43)
19	197	(1)	0.126	(3)	−0.039	(32)	0.284	(6)	−0.452	(2)
34	100	(12)	0.143	(2)	0.629	(1)	−0.359	(3)	−0.521	(1)
50	120	(9)	0.061	(6)	0.356	(3)	−0.150	(12)	−0.080	(27)
52	140	(6)	0.146	(1)	0.358	(2)	−0.333	(4)	0.235	(6)

^*DALPHACS consists of the components of DALPHAC divided by their corresponding empirical standard errors based on the full data analysis results as reported in Table 1. Rankings are based on the absolute value of DALPHAC.

^‡Cluster level Cook’s distance for α.

5. Discussion

Application to the medical practice data, as well as results from the first simulation study (section 3.1), demonstrate that the proposed cluster-deletion diagnostics for alternating logistic regressions are good approximations of their exact counterparts. On the other hand, results from the second simulation failed to provide convincing evidence that the extreme Cook’s Distance diagnostics respond in a consistent manner to contaminated binary response models. Besides their relevance to alternating logistic regressions, these are the first published results of their kind concerning the behavior in practical situations of DCLS_β,i (Preisser and Qaqish, 1996) for generalized estimating equations (Liang and Zeger, 1986).

The proposed diagnostic formulae are computationally fast. Computation of the cluster diagnostics for the medical practice data of section 4 took 13 minutes on dual 700 MHz SPARC processors, compared to over 12 hours for computation of their fully iterated ’exact’ counterparts. Qualitatively similar computational savings using formulae similar to (5)–(8), in the context of modelling intracluster correlations using the estimating equations approach of Prentice (1988), have been reported by Preisser and Perin (2007), for four data sets from medicine and public health. One-step formulae that have structure similar to those presented here could be developed for other estimating equation procedures for correlated binary data (Kuk and Nott, 2000; le Cessie and van Houwelingen, 1994; Lipsitz and Fitzmaurice, 1996). Furthermore, it should be possible to extend the deletion diagnostics of Preisser and Qaqish (1996) to GEE2.

In general, the question of an appropriate cut-off value for Cook’s distance is a difficult one, even in the case of multiple linear regression (Muller and Mok 1997; Diaz-Garcia and Gonzalez-Farias, 2004). Vens and Ziegler (2012) suggest that for GEE a cluster could be called influential (with respect to β) if DCLS_β,i(p) exceeds some threshold, e.g.,${\chi }_p^2(1-\alpha )$. They apply this criterion to longitudinal data from a clinical trial where every cluster (patient) has the same number of visits. In such cases of equal cluster sizes, it seems intuitively reasonable to apply a common cut-off value. However, in the medical practice data where cluster sizes range from 19 to 197, we would expect that larger clusters tend to have more influence, as Figure 3(d) shows. Therefore our practice (e.g., Preisser and Qaqish, 1996) has been to visually examine the graph of Cook’s D versus cluster size to judge if and which clusters appear to deviate from the overall pattern.

Although cluster-deletion diagnostics seem to be the most useful, diagnostic formulae for other kinds of subset deletion could be developed. Preisser and Qaqish (1996) proposed a one-step approximation to β̂ − β̂_[m] for GEE where m denotes an arbitrary subset of observations to be deleted. A similar formula for α̂ − α̂_[m] could be easily developed with derivations similar to those found in their appendix. Besides cluster-deletion diagnostics, we have developed and implemented in SAS/IML software observation-deletion diagnostics for ALR that approximate the change in regression coefficients when a single observation (eg., patient) is deleted. The formulae, not presented here, are similar in form to observation-deletion diagnostics of Preisser and Perin (2007). When applied to the medical practice data, however, they reveal that no single patient was found to have large influence (results not shown).

A referee has cited a discussion in the last paragraph of Kuk (2004) asking whether invariance of the sandwich estimator of the variance of the within-cluster association parameter estimates (α̂) to the ordering of observations within a cluster is an important feature for longitudinal data. We agree with Kuk (2004) that “unlike the case of clustered or multivariate data, invariance to permutations is not a natural requirement for estimators based on longitudinal data, which are not indexed arbitrarily, but according to time.” Nonetheless, the invariant variance estimators for the ALR estimator based on (2), and specifically described in Qaqish et. al. (2011), will generally give different values in practice than the variance estimators of Carey et al. (1993) or Kuk (2004), both which are based on conditional residuals. As a practical matter for software developer and user, a general method applicable to both clustered and longitudinal data will increase the appeal of ALR, so the complete permutation-invariance results of Kuk (2004) provide an important step. The recently introduced marginal residuals formulation of ALR (Qaqish et al., 2011) has potentially even greater appeal because its development based on classical estimating function theory (e.g., Godambe, 1960), in addition to enabling regression diagnostics to be developed in this article in a straightforward way, has led to the development of the orthogonalized residuals method with a non-diagonal P_i in equation (2) providing for possible efficiency gains.

APPENDIX

In this article, correlated binary data with marginal mean and within-cluster association structures described by equations (9) and (10), respectively, were randomly generated using an algorithm based on the conditional linear family (CLF) of multivariate binary distributions (Qaqish, 2003). Suppressing subscript “i” in order to simplify the description of data generation for a single cluster, this algorithm simulates Y, a n-vector of Bernoulli variates with mean vector μ and covariance matrix, $\mathbf{V}=\text{diag}({\sigma }_{jj}^{1/2})\mathbf{R}\text{diag}({\sigma }_{jj}^{1/2})$ where R contains the marginal correlations corr(Y_j, Y_k) = ρ_jk. A preliminary step involves determining the elements of R from the marginal means μ and pairwise odds ratios {ψ_jk}. Specifically, we map (β,α) into (μ, {ψ_jk}), thence into (μ, {ρ_jk}), and finally into (μ,V). The key formula (Dale, 1986; Liang, Zeger, and Qaqish, 1992) to determine ρ_jk(μ_j, μ_k, ψ_jk) is:

$$\mathbf{E}(Y_j{Y}_k)=\frac{[1-({\mu }_j+{\mu }_k)(1-{\psi }_{jk})-{\{{[1-({\mu }_j+{\mu }_k)(1-{\psi }_{jk})]}^2-4({\psi }_{jk}-1){\psi }_{jk}{\mu }_j{\mu }_k\}}^{1/2}]}{2({\psi }_{jk}-1)}$$

if ψ_jk ≠ 1; otherwise E(Y_jY_k) = Pr(Y_j = 1, Y_k = 1) = μ_jμ_k if ψ_jk = 1. Then, pairwise correlations are given by ${\rho }_{jk}=(\mathbf{E}(Y_j{Y}_k)-{\mu }_j{\mu }_k)/\sqrt{{\sigma }_{jj}{\sigma }_{kk}}$.

Next, we describe the procedure for simulating Y from a multivariate binary distribution belonging to the CLF with (μ,V). For j = 2, …, n, define Z_j = (Y₁, …, Y_j−1)^⊤, θ_j = E(Z_j), G_j = cov(Z_j), and s_j = cov(Z_j, Y_j). Note that G_j and s_j are determined from V. For a given (μ, V), a (j − 1)-vector b_j is defined as $b_j=G_j^{-1}{s}_j$ (j = 2, …, n). The CLF is defined by

$$P(Y_j=1|Z_j=z_j)={\mu }_j+b_j^{\top }(z_j-{\theta }_j)={\mu }_j+\sum _{k=1}^{n-1}{b}_{jk}(y_k-{\mu }_k)(j=2,\dots ,n).$$

The simulation algorithm proceeds as follows. First, simulate Y₁ as Bernoulli with mean μ₁, then for j = 2, …, n, simulate Y_j as Bernoulli with conditional mean given above. It then follows that E(Y) = μ and for 1 < j ≤ n, $\text{cov}(Z_j,Y_j)=\text{cov}(Z_j,b_j^{\top }{Z}_j)=G_j{b}_j=s_j$. The vector Y thus obtained has the required mean, μ, and covariance, V. There are some restrictions on allowable μ and V as discussed by Qaqish (2003). Note that the full joint distribution of Y, whose explicit specification is not required, can be computed as products of the conditional means. For any valid (μ, V) that is reproducible by the CLF, there is a corresponding unique value of the 2_n × 1 vector of joint probabilities. SAS programs to implement the CLF procedures are provided at http://www.bios.unc.edu/distrib/gee/clf/. There is also an R package called binarySIMCLF (By and Qaqish, 2009). The file provided as supplementary material provides an example of the SAS programs applied to generate correlated binary outcomes according to the CLF for models in equations (9) and (10).