The Houdini Transformation: True, but Illusory

Abstract

Molenaar (2003, 2011) showed that a common factor model could be transformed into an equivalent model without factors, involving only observed variables and residual errors. He called this invertible transformation the Houdini transformation. His derivation involved concepts from time series and state space theory. This paper verifies the Houdini transformation on a general latent variable model using algebraic methods. The results show that the Houdini transformation is illusory, in the sense that the Houdini transformed model remains a latent variable model. Contrary to common knowledge, a model that is a path model with only observed variables and residual errors may, in fact, be a latent variable model.

It is usually thought that structural equation models with latent variables and path models with only observed variables are completely different model types and hence that they cannot be mapped into each other. We show in this paper that models that contain only observed and residual variables may in fact be equivalent to latent variable models. To motivate the development, consider the interdependent system

$$x_1=W_{12}{x}_2+{\delta }_1$$

$$x_2=W_{21}{x}_1+{\delta }_2$$

where the residuals are further structured via δ₁ = ε₁ − W₁₂ε₂ and δ₂ = ε₂ − W₂₁ε₁. For concreteness, let x₁ be (2×1) and x₂ be (6×1), with $W_{12}=\left[\begin{array}{cccccc}\hfill .12\hfill & \hfill .16\hfill & \hfill .20\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill .2013\hfill & \hfill .2349\hfill & \hfill .2685\hfill \end{array}\right]$ and $W_{21}^{\prime }=\left[\begin{array}{cccccc}\hfill 1.5\hfill & \hfill 2\hfill & \hfill 2.5\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1.2\hfill & \hfill 1.4\hfill & \hfill 1.6\hfill \end{array}\right]$, ${\Psi }_{11}=E\left({\epsilon }_1{\epsilon }_1^{\prime }\right)=diag(.96,.75)$, ${\Psi }_{12}=E\left({\epsilon }_1{\epsilon }_2^{\prime }\right)=0$, and ${\Psi }_{22}=E\left({\epsilon }_2{\epsilon }_2^{\prime }\right)=diag(.91,.84,.75,.64,.51,.36)$. There is no hint in that this system might be equivalent to a latent variable model, although it does have an odd error structure. However, the Houdini transformation that is developed below can be used to show that this model is equivalent to the standard orthogonal factor model x = Λζ + ε, where $x={(x_1^{\prime },x_2^{\prime })}^{\prime }$, $\Lambda ={({\Lambda }_1^{\prime },{\Lambda }_2^{\prime })}^{\prime }$, E(ζζ′) = I, $\epsilon ={({\epsilon }_1^{\prime },{\epsilon }_2^{\prime })}^{\prime }$ and ${\Lambda }^{\prime }=\left[\begin{array}{cccccccc}\hfill .2\hfill & \hfill 0\hfill & \hfill .3\hfill & \hfill .4\hfill & \hfill .5\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill .5\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill .6\hfill & \hfill .7\hfill & \hfill .8\hfill \end{array}\right]$.

It is instructive to focus on the distinction between latent versus observed variable models using the most well-known latent variable model, namely, factor analysis. A continuing theme in its 100+ year history (e.g., Cudeck & MacCallum, 2007) has been the key distinction made between this method and principal components analysis. There is consensus that factor analysis is a true latent variable model while components analysis, although in some ways appearing similar to the factor model, simply involves transformations of observed variables. Although it has recently proven possible to relate these methods in a precise way (Bentler & de Leeuw, 2011), and although limit theorems exist on when they become identical (e.g., Bentler & Kano, 1990), in any given empirical application with a relatively small number of variables they are not interchangeable. It is then surprising to discover, as Molenaar (2003, 2011) has done, that any common factor model can be transformed in a 1:1 way so that the resulting model contains no common factors, only observed variables and residuals. Molenaar's proof was lengthy and invoked time series theorems of Granger and Morris (1976). Here we provide a simple direct proof based on manipulations of standard structural equations. We obtain the desired result, but, extending Molenaar's (2003, 2011) conclusions, we show that the resulting model remains a latent variable model.

We need to have some clarity on what a latent variable model is, and is not. While there are several defining characteristics of latent variable models as reviewed by Bollen (2002), we use the definition provided by Bentler (1982) based on the Bentler-Weeks (1980) model. In this approach, all variables in a model are either dependent or independent. A variable is a dependent variable if is expressed in a model as a function of one or more other variables (i.e., it appears on the left side of an equation), while all other variables are considered independent variables (i.e., they never appear on the left side of any equation). Then a model for p observed variables is a latent variable model if, and only if, the dimensionality of the independent variables is greater than the dimensionality of the observed models. Specifically, with p observed variables, the covariance matrix of independent variables must be rank p+k for some k>0. Factor analytic models with p variables and k factors meet this requirement, while principal components with k components do not. The model resulting from a Houdini transformation involves equations in observed variables and errors, as is typical of path models with residual errors. In general, such models are not latent variable models, implying that the Houdini transformed model is not a latent variable model. But we will see that the observed variable model with residual errors obtained from a Houdini transformation remains a latent variable model even though the common factors have disappeared.

MODEL SETUP

For some model generality, let us consider a standard factor analytic measurement model for p variables

$$x=\mu +\wedge \xi +\epsilon$$

with latent factors ξ and residual errors or unique variates ε, and latent variable regressions

$$\xi =B\xi +\zeta .$$

Thus this is a factor analytic simultaneous equation model. As usual, we assume that ζ, ε are mutually uncorrelated, and that (I − B) is full rank, so that ξ = (I − B)⁻¹ ζ. It follows that x = μ + Λ(I − B)⁻¹ ζ + ε, where now the ζ play the role of common factors.

With the covariance matrix of the ζ given as Φ, and that of ε given as ψ (taken here as diagonal), both assumed full rank, the covariance structure of the model is given by

$$\Sigma =\wedge {(I-B)}^{-1}\Phi {(I-B)}^{-1}\prime {\wedge }^{\prime }+\Psi .$$ (1)

In this model, the independent variables ζ and ε are of order k and p, respectively. With p observed variables we have p<(p+k), and hence this is a latent variable model.

HOUDINI MANIPULATIONS

To simplify model (1) prior to our manipulations, we rewrite $\stackrel{‒}{\Lambda }=\Lambda {(I-B)}^{-1}{\Phi }^{.5}$, where Φ⁵Φ^5′ = Φ. Then it follows that model (1) can be written in a version that looks like an exploratory factor analysis model, namely,

$$\Sigma =\stackrel{‒}{\wedge }\stackrel{‒}{\wedge }\prime +\Psi .$$ (2)

Without loss of generality, if necessary, we permute the rows and columns of Σ so that when we write

$$\stackrel{‒}{\wedge }=\left[\frac{{\wedge }_1}{{\wedge }_2}\right]$$ (3)

the kxk part Λ₁ has full rank. Thus the structural equations can be written as

$$x=\left[\frac{x_1}{x_2}\right]=\left[\frac{{\wedge }_1}{{\wedge }_2}\right]\zeta +\left[\frac{{\epsilon }_1}{{\epsilon }_2}\right].$$ (4)

It follows that ${\Lambda }_1^{-1}{x}_1=\zeta +{\Lambda }_1^{-1}{\epsilon }_1$, and hence

$$\zeta ={\wedge }_1^{-1}(x_1-{\epsilon }_1).$$ (5)

Since (5) provides a way of expressing the factors in terms of observed and residual error variables, it can be used to eliminate the factors completely from the model. To accomplish this, we substitute (5) into the second set of equations in (4), obtaining

$$x_2={\wedge }_2{\wedge }_1^{-1}(x_1-{\epsilon }_1)+{\epsilon }_2.$$ (6)

We also do some manipulation of (6) to obtain an expression for x₁. This is

$$x_1={\wedge }_1{\left({{\wedge }_2}^{\prime }{\wedge }_2\right)}^{-1}{{\wedge }_2}^{\prime }(x_2-{\epsilon }_2)+{\epsilon }_1.$$ (7)

These two equations, (6) and (7), are key results. They express the observed variables as an interdependent system relating observed variables and residual errors only. The common factors are gone. This is the Houdini transformation, considered in a more general SEM context as compared to Molenaar (2003, 2011).

THE ILLUSORY ABSENT FACTORS

The fundamental question in this paper is whether the Houdini transformation has resulted in a model (equations (6) and (7)) that is a latent variable model or not. To answer this question, it should be noted that each of the above steps is reversible. That is, starting out with a model of the form (6) and (7) with no common factors, a model with common factors can be obtained. Hence it would seem that if one model is a latent variable model, so must be the other one.

We look at this by investigating the dimensionality of the independent variables. Clearly, the dependent variables in the model are x₁ and x₂, and the independent variables are ε₁ and ε₂. By the original assumptions, they are uncorrelated, and hence they span a p-dimensional space. At first glance, it seems that we do not have a latent variable model. But what about the variables (x₁ − ε₁) and (x₂ − ε₂) ? They are not dependent variables, but are they independent variables? To investigate this, from (4) we note that (x₁ − ε₁) = Λ₁ζ and (x₂ − ε₂) = Λ₂ζ are linear transformations of the k common factors ζ and hence both of these variables are k-dimensional latent variables. Furthermore, since they are two linear transformations of a single vector variate ζ that is independent of the unique variates ε₁ and ε₂, the dimensionality of the space of (6) and (7) is p + k. Hence the Houdini transformation did not reduce the space spanned by the independent variables.

We complete this section by relating equations (6) and (7) to our opening example. The abstract algebra can be mapped into the numbers of the example with the calculations $W_{21}={\Lambda }_2{\Lambda }_1^{-1}$ and $W_{12}={\Lambda }_1{\left({\Lambda }_2^{\prime }{\Lambda }_2\right)}^{-1}{\Lambda }_2^{\prime }$.

DISCUSSION

A universally accepted distinction in structural modeling is between observed variable path models, or simultaneous equation models, and latent variable models. What the Houdini transformation shows is that a model that contains only observed variables and residual errors actually can be a latent variable model. To our knowledge, this observation is new.

While the results in this paper provide one view on the Houdini transformation, we want to recognize that we have dealt with a specialized approach to eliminating latent factors from a model. Molenaar actually provided a broader view of this transformation which emphasizes the following additional features. The Houdini transformation involves a principled approach to derive a family of equivalent structural equation models which have not been considered before in the published literature. The equivalence relationship concerned is reminiscent of the relationship between state space models with latent state processes and transfer function models directly linking input and output processes (cf. Heij et al., 2007). The way in which the set of equivalent models is obtained implies that all models with latent factors are nested. This has important consequences for model selection. Moreover, the Houdini transformation of a model with q latent factors is obtained in a sequence of steps, where in each step the dimension of the

latent space is reduced. This raises new questions about the proper definition of the dimension of spaces spanned by common latent variables.