Ensuring Positiveness of the Scaled Difference Chi-square Test Statistic

Abstract

A scaled difference test statistic ${\tilde{T}}_d$ that can be computed from standard software of structural equation models (SEM) by hand calculations was proposed in Satorra and Bentler (2001). The statistic ${\tilde{T}}_d$ is asymptotically equivalent to the scaled difference test statistic T̄_d introduced in Satorra (2000), which requires more involved computations beyond standard output of SEM software. The test statistic ${\tilde{T}}_d$ has been widely used in practice, but in some applications it is negative due to negativity of its associated scaling correction. Using the implicit function theorem, this note develops an improved scaling correction leading to a new scaled difference statistic T̄_d that avoids negative chi-square values.

Introduction

Moment structure analysis is widely used in behavioural, social and economic studies to analyze structural relations between variables, some of which may be latent see, e.g., Bollen and Curran (2006), Grace (2006), Yuan and Bentler (2007), and references therein. In such analyses it frequently happens that two nested models M₀ and M₁ are compared using estimation methods that are non-optimal (asymptotically) given the distribution of the data; e.g. ML estimation is used when the data are not multivariate normal. In those circumstances, the usual chi-square difference test T_d = T₀ - T₁, based on the separate models’ goodness of fit test statistics, is not χ² distributed. A correction to T_d by a scaling factor was proposed by Satorra (2000) and Satorra and Bentler (2001) to improve the chi-square approximation. The latter is the focus of this paper.

Satorra and Bentler’s (2001) correction provided a simple procedure to obtain an approximate scaled chi-square statistic using hand calculations on regular output of SEM analysis; it has the drawback, however, that a positive value for the scaling correction is not assured. The present paper develops a simple procedure by which a researcher can compute the exact SB difference test statistic based only on output from standard SEM programs.

Setup and Notation

Throughout we adhere to the notation and results of Satorra and Bentler (2001). Let σ and s be p-dimensional vectors of population and sample moments respectively, where s tends in probability to σ as sample size n → +∞. Let $\sqrt{n}(s-\sigma )$ be asymptotically normally distributed with a finite asymptotic variance matrix Γ (p × p). Consider the model M₀ : σ = σ(θ) for the moment vector σ, where σ(.) is a twice-continuously differentiable vector-valued function of θ, a q-dimensional parameter vector. Consider a discrepancy function F = F (s, σ) in the sense of Browne (1984), and the estimator θ̂ based on F or on an (asymptotically equivalent) weighted least squares (WLS) analysis with weight matrix $V=\frac{1}{2}{\partial }^{2}F(s,\sigma )∕\partial \sigma \partial {\sigma }^{\prime }$ evaluated at σ = s.

Let M₀ : σ = σ⋆(δ), a(δ) = 0, and M₁ : σ = σ⋆(δ) be two nested models for σ. Here δ is a (q+m)-dimensional vector of parameters, and σ⋆(.) and a(.) (an m-valued function) are twice-continuously differentiable vector-valued functions of δ ∈ ϴ₁, a compact subset of R^q+m. Our interest is in the test of a null hypothesis H₀ : a(δ) = 0 against the alternative H₁ : a(δ) ≠ 0.

For the developments that follow, we require the Jacobian matrices Π(p× (q + m)) := ∂σ⋆(δ)/∂δ’ and A(m × (q + m)) := ∂a(δ)/∂δ’, which we assume to be regular at the true value of δ, say δ₀. We also assume that A is of full row rank. By using the implicit function theorem, associated to M₀ (more specifically, to the restrictions a(δ) = 0), there exists (locally in a neighborhood of δ₀) a one-to-one function δ = δ(θ) defined in an open and compact subset S of R^q, and a θ₀ in the interior of S such that δ(θ₀) = δ₀ and σ(δ(θ)) satisfies the model M₀. Let H = ∂δ(θ)/∂δ’ ((q + m) × q) be the corresponding Jacobian matrix evaluated at θ₀. Hence, by the chain rule of differentiation, Δ= ∂σ/∂θ’ = (∂σ(δ)/∂δ’)(∂ δ(θ)/∂θ’)= ΠH. Since a(δ(θ)) = 0, it holds that with A evaluated at δ₀, AH = 0 with r(A)+r(H) = q+m, and r(.) denoting the rank of a matrix. Thus, H’ is an orthogonal complement of A. Let P ((q + m) × (q + m)) := Π’VΠ. Associated to M₁, the less restricted model σ = σ⋆(δ), the goodness-of-fit test statistic is $T_1=nF(s,\tilde{\sigma })$, where $\tilde{\sigma }$ is the fitted moment vector in model M₁ with associated degrees of freedom r₁ = r₀ - m and scaling factor c₁ given by

$$c_1≔\frac{1}{r_1}tr\left\{U_1\Gamma \right\}=\frac{1}{r_1}tr\left\{V\Gamma \right\}-\frac{1}{r_1}tr\left\{P^{-1}{\Pi }^{\prime }V\Gamma V\Pi \right\},$$ (1)

where

$$U_1≔V-V\Pi P^{-1}{\Pi }^{\prime }V.$$ (2)

We refer to Satorra and Bentler (2001) for further details.

Scaling Correction for the Difference Test

When both models M₀ and M₁ are fitted, for example by ML, then we can test the restriction a(δ) = 0, assuming M₁ holds, using the chi-square difference test statistic T_d := T₀ - T₁. Under the null hypothesis, we would like T_d to have a χ² distribution with degrees of freedom m = r₀ - r₁. This is the restricted test of M₀ within M₁. For general distributions of the data, the asymptotic chi-square approximation may not hold. To improve on the chi-square approximation, Satorra (2000) gave explicit formulae that extends the scaling corrections proposed by Satorra and Bentler (1994) to the case of difference, Wald, and score type of test statistics. General expressions for those corrections were also put forward in Satorra (1989, p.146). Specifically, for the test statistic T_d we are considering, Satorra (2000, p. 241) proposed the following scaled test statistic:

$${\stackrel{‒}{T}}_d≔T_d∕{\widehat{c}}_d,\text{where}{c}_d≔\frac{1}{m}tr\left\{U_d\Gamma \right\}$$ (3)

with

$$U_d=V\Pi P^{-1}{A}^{\prime }{\left(A{P}^{-1}{A}^{\prime }\right)}^{-1}A{P}^{-1}{\Pi }^{\prime }V.$$ (4)

Here, ĉ_d denotes c_d after substituting consistent estimates of V and Γ, and evaluating the Jacobians A and at the estimate δ̂₀ when fitting M₀ (or M₁). Since tr {U_dΓ} can be expressed as the trace of the product of two positive definite matrices, tr {U_dΓ} > 0, and thus c_d > 0; the same for ĉ_d > 0. Consequently, T̄_d is ensured to be a non-negative number.

A practical problem with the statistic T̄_d is that it requires computations that are outside the standard output of current structural equation modeling programs. Furthermore, difference tests are usually hand computed from different modeling runs. Satorra and Bentler (2001) proposed a procedure to combine the estimates of the scaling corrections c₀ and c₁ associated to the chi-square goodness-of-fit test for the two fitted models M₀ and M₁ in order to compute a consistent estimate of the scaling correction c_d for the difference test statistic. A modified (easy to compute) scaled test statistic ${\tilde{T}}_d$ with the same asymptotic distribution as T̄_d was proposed. Both statistics were shown to be asymptotically equivalent under a sequence of local alternatives (so they have the same asymptotic local power). Their procedure to compute ${\tilde{T}}_d$ is as follows (see Satorra and Bentler, 2001, p. 511).

1. Obtain the unscaled and scaled goodness-of-fit tests when fitting M₀ and M₁ respectively; that is, T₀ and T̄₀ when fitting M₀, and T₁ and T̄₁ when fitting M₁;

2. Compute the scaling corrections ĉ₀ = T₀/T̄₀, ĉ₁ = T₁/T̄₁, and the unscaled chi-square difference T_d = T₀ - T₁ and its degrees of freedom m = r₀ - r₁;

3. Compute the scaled difference test statistic as

   $${\tilde{T}}_d≔T_d∕{\tilde{c}}_d\text{with}{\tilde{c}}_d=(r_0{\widehat{c}}_0-r_1{\widehat{c}}_1)∕m.$$

Here r₀ and r₁ are the respective degrees of freedom of the models M₀ and M₁.

The basis for computing the scaling correction for the difference test statistic is the following alternative expression for U_d of (4) (see Satorra and Bentler, 2001, p. 510)

$$U_d=U_0-U_1,$$ (5)

where U₁ is given in (2) and U₀ := V - VΠH(H’Π’VΠH)^-1H’Π’V. Since (5) implies

$$m{c}_d=tr\left\{U_d\Gamma \right\}=tr\left\{(U_0-U_1)\Gamma \right\}=r_0{c}_0-r_1{c}_1,$$ (6)

it follows that c_d = (r₀c₀ - r₁c₁)/m. This is the theoretical basis for Satorra and Bentler’s (2001) proposal as given in steps 1-3 above.

Problem with the Current Scaled Difference Test

For an arbitrary matrix V > 0, (5) and (6) are exact equalities when the matrices U_d, U₀ and U₁ are evaluated at a common value δ (as, e.g., the fitted value under model M₀ or under M₁); however, they are just asymptotic equalities when U_d, U₀ and U₁ are evaluated at different estimates that converge to the same true value under the null hypothesis. Under Satorra and Bentler’s (2001) proposal, ${\tilde{c}}_d$ evaluates U₀ and U₁ at the estimates δ̂₀ and δ̂₁ respectively. Since δ̂₁ will in general not satisfy the null model M₀ (i.e., it will not be of the form δ = δ(θ) for the function implied by the implicit function theorem), when it deviates highly from M₀, the estimated difference ${\tilde{c}}_d=(r_0{\widehat{c}}_0-r_1{\widehat{c}}_1)∕m$ may turn out to be negative. This may happen in small samples, or when M₀ is highly incorrect; a result can be an improper value for ${\tilde{T}}_d$. Satorra and Bentler (2001, p. 511) warned on the possibility that an improper value of ${\tilde{T}}_d$ could arise.

In order to be sure to avoid a negative value for ${\tilde{c}}_d$ and hence ${\tilde{T}}_d$, currently one would need to resort to computing T̄_d using the (3). Unfortunately this is impractical or impossible for most applied researchers who only have access to standard SEM software.

Fortunately, as we show next, the exact value of T̄_d can also be obtained from the standard output of SEM software, using a new hand computation.

A New Scaled Test Statistic T̄_d

Denote by M₁₀ the fit of model M₁ to a model setup with starting values taken as the final estimates obtained from model M₀, and with number of iterations set to 0. Consider ĉ₁⁽¹⁰⁾ := T₁⁽¹⁰⁾/T̄₁⁽¹⁰⁾, where T₁⁽¹⁰⁾ and T̄₁⁽¹⁰⁾ are the standard unscaled and scaled test statistic of this additional run. Note that the estimate ĉ₁⁽¹⁰⁾ uses model M₁ but the matrices and A are now evaluated at δ̂₀ := δ(θ̂), where θ̂ is the estimate under M₀. Since now all the matrices involved in (5) are evaluated at δ^₀, the equality (6) holds exactly, and not only asymptotically, as when U₀ is evaluated at δ^₀ and U₁ at δ^₁. The scaling correction that is now computed is

$${\widehat{c}}_d^{\left(10\right)}≔(r_0{\widehat{c}}_0-r_1{\widehat{c}}_1^{\left(10\right)})∕m,$$ (7)

which, in view of (6), is the scaling correction of (3) when U_d is evaluated at the estimate δ^₀ when fitting M₀, and thus it is a positive number. The new scaled difference statistic is thus defined as

$${\stackrel{‒}{T}}_d^{\left(10\right)}≔(T_0-T_1)∕{\widehat{c}}_d^{\left(10\right)},$$ (8)

Clearly, ${\stackrel{‒}{T}}_d^{\left(10\right)}={\stackrel{‒}{T}}_d$ that is, ${\stackrel{‒}{T}}_d^{\left(10\right)}$ coincides numerically with the scaled statistic (3) proposed in Satorra (2000).

An Illustration

We use this data just for illustrative purposes, and because it provides an example where the standard scaling correction fails to be positive. We use a latent variable model discussed for this data by Bentler, Satorra and Yuan (2009). The Bonett-Woodward-Randall (2002) test shows that these data have significant excess kurtosis indicative of non-normality at a one-tail .05 level, so test statistics derived from ML estimation may not be appropriate and we do the scaling corrections.

The model considered is a structured means model, with the mean cigarette sales indirectly affecting the mean rates of the various cancers. The specified model is

$$V_j={\lambda }_{j}F+E_j,j=2,\dots ,5,F=\beta V_1+D_1,V_1=\mu +E_1,$$

where the V_j’s denote observed variables; F, D₁, and E_j are the common, residual common, and unique factors respectively; λ_j denotes a factor loading parameter, β is the effect of cigarette smoking on the cancer factor, and μ is the mean parameter for rates of smoking. The units of measurement for the factor were tied to V₂, with λ₂ = 1. The following values for the ML and Satorra-Bentler (1994) (SB) scaled chi-square statistics are obtained

$$T_1=107.398,{\stackrel{‒}{T}}_1=65.3524,r_1=9,{\widehat{c}}_1=1.6434,$$

along with the degrees of freedom r₁ and the scaling correction ĉ₁. The model does not fit, though for the sake of the illustration we are aiming for, this is not of concern to us.

Restricted Model, M₀

The same model is now fitted with the added restriction that the error variances of the kidney and leukemia cancers, E₄ and E₅, are equal. This model gives the following statistics

$$T_0=139.495,{\stackrel{‒}{T}}_0=97.4034,r_0=10,{\widehat{c}}_0=1.4322.$$

Difference Test

Our main interest lies in testing the difference between M₀ and M₁, which we do with the chi-square difference test. The ML difference statistic is

$$T_d=139.495-107.398=32.097,$$

which, with 1 degree of freedom (m), rejects the null hypothesis that the error variances for E4 and E5 are equal. Since the data is not normal, we compute the SB (2001) scaled difference statistic. This requires computing the scaling factor ${\tilde{c}}_d=(r_0{\widehat{c}}_0-r_1{\widehat{c}}_1)∕m$ given by

$$[10\left(1.4322\right)-9\left(1.6434\right)]∕1=14.322-14.7906=-.4686.$$

The scaling factor ${\tilde{c}}_d$ is negative, so the SB difference test cannot be carried out; or, if carried out, it results in an improper negative chi-square value.

New Scaled Difference Test

As described above, to compute the scaled statistic T̄_d we implement (7) and (8). The output that is missing in the prior runs is the value of the SB statistic obtained at the final parameter estimates for model M₀ when model M₁ is evaluated. This can be obtained by creating a model setup M₁₀ that contains the parameterization of M₁ with start values taken from the output of model M₀. Model M₁₀ is run with zero iterations, so that the parameter values do not change before output including test statistics is produced.¹ The new result gives

$$T^{\left(10\right)}=139.495,{\stackrel{‒}{T}}^{\left(10\right)}=94.9551,r_1=9,{\widehat{c}}^{\left(10\right)}=1.4691,$$

where as expected, T¹⁰ = T₀ as reported above (i.e., the ML statistics are identical), and the value ĉ¹⁰ is hand-computed. As a result, we can compute

$${\widehat{c}}_d^{\left(10\right)}=(r_0{\widehat{c}}_0-r_1{\widehat{c}}_1^{\left(10\right)})∕m=[\left(10\right)\left(1.4322\right)-\left(9\right)\left(1.4691\right)]=1.10,$$

which, in contrast to the SB (2001) computations, is positive. Finally, we can compute the proposed new SB corrected chi-square statistic as

$${\stackrel{‒}{T}}_d={\stackrel{‒}{T}}_d^{\left(10\right)}=(T_0-T_1)∕{\widehat{c}}_d^{\left(10\right)}=(139.495-107.398)∕1.10=29.179,$$

which can be referred to a ${\chi }_1^2$ variate for evaluation.

Discussion

The implicit function theorem was used to provide a theoretical basis for the development of a practical version of the computationally more difficult scaled difference statistic proposed by Satorra (2000).² The proposed method is only marginally more difficult to compute than that of Satorra and Bentler (2001) and solves the problem of an uninterpretable negative χ² difference test that applied researchers have complained about for some time.

Like the method it is replacing, the proposed procedure applies to a general modeling setting. The vector of parameters σ to be modeled may contain various types of moments: means, product-moments, frequencies (proportions), and so forth. Thus, this scaled difference test applies to methods such as factor analysis, simultaneous equations for continuous variables, log-linear multinomial parametric models, etc.. It can easily be seen that the procedure applies also in the case where the matrix Γ is singular, when the data is composed of various samples, as in multi-sample analysis, and to other estimation methods. It applies also to the case where the estimate of Γ reflects the fact that we have intraclass correlation among observations, as in complex samples. Hence this new statistic should be useful in a variety of applied modeling contexts. Simulation work will be needed to understand its virtues and limitations, relative to other alternatives, in such contexts.