Exploration of model misspecification in latent class methods for longitudinal data: correlation structure matters

SUMMARY

Modeling longitudinal trajectories and identifying latent classes of trajectories is of great interest in biomedical research, and software to identify latent classes of such is readily available for latent class trajectory analysis (LCTA), growth mixture modeling (GMM) and covariance pattern mixture models (CPMM). In biomedical applications, the level of within-person correlation is often non-negligible, which can impact the model choice and interpretation. LCTA does not incorporate this correlation. GMM does so through random effects, while CPMM specifies a model for within-class marginal covariance matrix. Previous work has investigated the impact of constraining covariance structures, both within and across classes, in GMMs – an approach often used to solve convergence problems. Using simulation, we focused specifically on how misspecification of the temporal correlation structure and strength, but correct variances, impacts class enumeration and parameter estimation under LCTA and CPMM. We found (1) even in the presence of weak correlation, LCTA often does not reproduce original classes, (2) CPMM performs well in class enumeration when the correct correlation structure is selected, and (3) regardless of misspecification of the correlation structure, both LCTA and CPMM give unbiased estimates of the class trajectory parameters when the within-individual correlation is weak and the number of classes is correctly specified. However, the bias increases markedly when the correlation is moderate for LCTA and when the incorrect correlation structure is used for CPMM. This work highlights the importance of correlation alone in obtaining appropriate model interpretations and provides insight into model choice.

1. INTRODUCTION

Modeling longitudinal change is common in biomedical research, often with a goal to determine if different subpopulations have distinct trajectories. When observable covariates define the subpopulations, conventional methods (e.g., generalized estimating equations (GEE)^1,2 and generalized linear or linear mixed models (GLMM)^3,4,5) can be used to estimate group-specific trajectories with either a marginal or a subject-specific interpretation of trajectory coefficients. When subpopulation membership cannot be directly observed, growth mixture models (GMM)^6,7 latent class trajectory analysis (LCTA)^8–10 and covariance pattern mixture models (CPMM)¹¹ are often employed, using maximum likelihood estimation.

In contrast to GMM and CPMM, LCTA is an approach that builds the likelihood function assuming that within a class, observations within the response vector for each sampling unit are uncorrelated. That is, in a typical setting where repeated observations are taken over time on each randomly-sampled individual, observations within individual are assumed uncorrelated over time after accounting for latent class membership. Identifying (latent) classes of individuals with different trajectories is of primary interest.¹⁰ Nagin proposed that LCTA’s conditional independence assumption can be plausible, given the intended definition of ‘class’ in this model and that explicit modeling of latent classes may account for any intra-individual correlation at the population level; plus, it reduces model complexity.^10,12,13 Thus, in the presence of temporal correlation, the goal of LCTA is not to model the source of heterogeneity among trajectories, but rather to identify groups (classes) of trajectories that look similar. High and low trajectories could be identified as different classes, even if both are generated from one distribution. While LCTA is often used when GMMs will not converge, the meaning of “class” is different in the presence of temporal correlation, and information criteria enumeration statistics (e.g., BIC) are not based on the mixture of distributions underlying the data. The modeling approach taken by CPMM and GMM is intended to not only explain population heterogeneity through a mixture of $k$ multivariate distributions ($k$ classes), but also to model the natural within-class heterogeneity of the trajectories. We sought to determine how much impact correlation misspecification had, separate from variance, on class identification and trajectory parameter estimation in a relatively simple CPMM setting and how divergent the concept of “class” was if LCTA were used instead.

Both LCTA and GMM build on the generalized linear model of Nelder and Wedderburn¹⁴, in which univariate distributions within the exponential family can be modeled and maximum likelihood estimates of model parameters can be found. Thus, those methods have the flexibility to accommodate many discrete-valued and skewed distributions. The drawback in the longitudinal setting is that the responses for each independent sampling unit is vector-valued, and maximum likelihood estimation is not possible unless a proper multivariate likelihood (e.g., multivariate Normal or multivariate Binomial) can be fully specified. In GMM, this is constructed by first specifying the univariate response distribution at each time point, then including individual-level random effects to capture heterogeneity among individuals and within-individual correlation. However, if there is a mean-variance relationship in the response distribution, it must be assumed that conditional on the random effects, the deviations from an individual’s true trajectory are independent over time – known as the conditional independence assumption – in order to properly specify a multivariate likelihood for maximum likelihood estimation. The CPMM is only applicable to multivariate normal response vectors, which have no mean-variance relationship, so the multivariate likelihood can be well-defined without the conditional independence assumption.

The impact of covariance model misspecification when implementing LCTA, CPMM and GMM, including lack of convergence issues, has been investigated by a number of authors.^15–24 The focus has been on simplifying the structure of the variance model through constraints; that is, constraining the covariance structure. In the case of the general GMM, this could be a constraint on the covariance matrix of random effects, and in the case of Gaussian responses, constraints on the within-individual random error covariance structure, with or without constraints on the random effects covariance matrix. Examples include constraining a covariance matrix to be the same across classes, diagonal, or in the case of Gaussian responses, to have equal variances at each time point. Simplifying the covariance model structure to reduce the number of parameters can improve model convergence. However, these authors found through simulation studies using normally-distributed outcomes that that such constraints, when trying to re-identify the structure of the generated data, would negatively impact class enumeration and lead to bias in the parameter estimates. However, the role of correlation, separate from variance, in this setting is not well-understood.

To address this gap in the literature, we investigated how (1) the degree of within-individual correlation in the data generating process and (2) misspecification of the correlation structure impacted class enumeration and trajectory parameter estimation/interpretation when modeling the trajectory of normally distributed outcomes with CPMM and LCTA. We chose not to focus on GMMs for normally-distributed outcomes, as the covariance structure, defined in part by random effects, is often more restrictive than CPMM. We note that the use of random effects impact both correlations and variances simultaneously, so the impact of correlation alone cannot be studied in the presence of random effects. Also, the use of the identity link with the normal distribution will result in the same expected value of the trajectory parameters whether viewed as a marginal mean trajectory (CPMM) or the trajectory of an individual whose random effects are all 0 (GMM). Use of CPMM rather than GMM was strongly advocated by McNeish and colleagues, citing improved ability to fit models while maintaining usefulness of parameter interpretations.^20–23 In the one class setting under both GEE and GLMM and proper mean model specification, estimates of trajectory parameters are quite robust to misspecification of the covariance structure, but their standard errors are not. We set out to determine if the same holds for latent class methods in longitudinal data when only within-individual correlation is misspecified.

In this paper, we first demonstrate the utility of this work by applying LCTA and CPMM in a real data example where the two methods lead to discrepant results. This is followed by brief reviews of the LCTA, GMM, and CPMM methods and pertinent literature, and finally, a comparison of LCTA and CPMM performance using a simulation study to investigate the impact of covariance model misspecification. We conclude with a discussion of the work’s major findings and insights from this work.

2. MOTIVATING DATA EXAMPLE

Our motivating example is based on an analysis of the Duke Established Populations for Epidemiologic Study of the Elderly (EPESE)²⁵, a community sample of persons 65 years of age and older at measured at baseline, 3-, 6- and 10-year follow-up year points. We analyzed systolic blood pressure (SBP) of 1,354 10-year survivors to identify latent class trajectories.

Initial modeling, using a linear mixed model, indicated that within-individual correlation of about 0.36 was present and that an exchangeable correlation structure over time was appropriate. For both CPMM and LCTA, we fit a linear model for change in SPB over time. For CPMM, we specified the same exchangeable covariance structure, with one free correlation parameter, to be estimated for each class. The number of latent classes present was identified using the Bayesian Information Criterion (BIC)²⁶, and subjects were assigned to a latent class based on their highest estimated membership probability.

CPMM identified 3 latent classes (Figure 1b) as the best fit; one with a stable low trajectory over time, one with an increasing trajectory over time, and one with decreasing trajectory over time. Most subjects were assigned to the stable low trajectory (89.8%) with 3.4% and 6.8% being assigned to the increasing and decreasing trajectories, respectively. LCTA identified 6 latent classes (Figure 1c); two with stable trajectories, two with increasing trajectories, and two with decreasing trajectories. Like the CPMM solution, most subjects were assigned to one of the stable trajectories (74.8%) with 16.7% and 8.5% assigned to increasing and decreasing trajectories, respectively. Thus, LCTA assigned more subjects to trajectories that changed over time (25.2% vs. 10.2%) than did CPMM. Interestingly, it appears as if each of the three classes in the CPMM solution might have been split into two classes in the LCTA solution, or that the CPMM solution failed to recognize six distinct classes by collapsing them down to three.

Figure 1.

Results of the applying the LCTA and CPMM methods to repeated measures of systolic blood pressure data from the EPESE Program¹¹. (a) LCTA 3-class model fit. (b) CPMM 3-class model fit; selected CPMM solution. (c) LCTA 6-class model fit; selected LCTA solution. (d) CPMM 6-class model fit.

To further investigate these differences, we fit a 3-class LCTA model to assess how it compared to the 3-class CPMM solution. The results were markedly different (Figure 1a vs. 1b). Using LCTA, all 3 groups had stable trajectories, and participants were more evenly distributed between the two classes with lower intercepts when compared to the 3-class CPMM solution (35.6% and 56.7% using LCTA vs. 89.8% and 3.4% in CPMM). Similarly, we fit a 6-class CPMM model to see how it compared to the 6-class LCTA solution, and again the results were markedly different (Figure 1c vs. 1d). The 6-class CPMM model fit essentially overlapped that of 3-class CPMM solution.

The discrepant results produced by LCTA and CPMM on the same data underscores the potential role of within-individual correlation when implementing and interpreting results of these methods. These findings may also suggest LTCA and CPMM class enumeration, mean parameter estimates, and associated interpretations are not insensitive to the correlation model. Given the prevalent use of LCTA, GMM, and CPMM in biomedical research where within-individual correlation is often present and the use of LCTA when GMMs or CPMMs will not converge, we decided to assess performance of these methods on data with different underlying correlation structure and strength.

3. BACKGROUND

Latent class growth methods, known by various names depending on model structure and assumptions, start with the presumption that there are latent classes into which subgroups of the population fall. In the longitudinal data setting, classes are distinguished by different mean trajectories over time. These methods are used to identify the latent classes and, subsequently, infer how trajectory parameters, class membership probabilities, and trajectory shapes are related to observable covariates. In this discussion, we use the term ‘individual’ to mean the primary sampling unit of correlated observations over time, and assume individuals represent a random sample from population of interest.

Our work is focused only on the LCTA and CPMM modeling frameworks, which are special cases of structural equation models, with similar, but not identical constraints. The difference between LCTA and CPMM models lies in the assumptions for defining the likelihood, from which trajectory parameters and standard errors are estimated and consequently, the meaning of the term ‘class’. Often, the likelihood is used in metrics to determine the number of latent classes (a process called class enumeration). In LCTA, repeated measurements over time on the same individual are assumed uncorrelated within class and the mean trajectory is the same for everyone with the same covariate pattern in that class. Thus, the class-specific likelihood can be formed by the product of univariate distributions across time, for any distribution in the exponential family, weighted by the probability of being in that class. If the vector of responses over time is assumed to be multivariate normal, the multivariate likelihood can be easily defined – either with a completely general within-class covariance structure or a more parsimonious constrained structure. This is the approach taken by CPMM and is often considered a ‘marginal model’. In contrast, GMM is a ‘subject-specific’ approach appropriate for both normal and non-normal response data. The approach defines the trajectories over time as a function of subject-level random effects. Condition on the random effects, the distribution of response variable at each time point within a class is specified (e.g., normal, Binomial, Poisson) and builds multivariate likelihood by forming the product of that response distribution and the distribution of the random effects, as is done in generalized linear mixed models.²⁷ The conditional independence assumption is essential for forming that product when the responses are not normally-distributed.

3.1. LCTA, CPMM and GMM Likelihood Functions

We assume a parametric model, $f\left({\mathit{y}}_i,\lambda \right)$, is appropriate for modeling the phenomenon under study, where ${\mathit{y}}_i={\left(y_{i1},y_{i2},\dots ,y_{i{r}_i}\right)}^{\prime }$ is the vector of $r_i$ measurements taken over time on the $i^{th}$ individual $(i=1,2,\dots ,N)$ and $\mathit{\lambda }$ is the corresponding parameter vector of the distribution $f\left(\right)$, that is, $\mathit{\lambda }=(\mathit{\mu },\mathbf{\Sigma })$ for the multivariate distribution. The spacing and number of responses over time per person can vary, but for convenience of notation, we will assume $r_1=r_2=\cdots =r_N=r$. The latent subgroup distributions are assumed to differ in their $\mathit{\lambda }$ parameter values, but follow the same distribution $\left(\right)$. Because the latent classes are unobservable, the analyst sees a random sample of response vectors drawn from a finite mixture of $K$ distinct classes, where ${\mathit{y}}_{\mathit{i}}$ is drawn from the $k^{th}$ class, with probability ${\pi }_k$ and follows distribution $f\left({\mathit{y}}_i,{\mathit{\lambda }}_k\mid C_i=k\right)$. Here, $C_i$ is a random variable denoting class membership of the $i^{th}$ individual, which is assumed to follow a multinomial distribution with class probabilities (also called mixing probabilities) $\mathit{\pi }={\left({\pi }_1,\dots {\pi }_K\right)}^{\prime },{\sum }_{k=1}^K {\pi }_k=1$. The marginal (multivariate) density of ${\mathit{y}}_i$ is the sum of the product of the probability of class membership in the $k^{th}$ class and the distribution of ${\mathit{y}}_i$, given it was drawn from the $k^{th}$ class (equation (3.1)):

$$f\left({\mathit{y}}_{\mathit{i}}\right)=\sum _{k=1}^K\mathrm{Pr}\left(C_i=k\right)f\left({\mathit{y}}_i,{\mathit{\lambda }}_k\mid C_i=k\right)=\sum _{k=1}^K{\pi }_{ik}f\left({\mathit{y}}_i,{\mathit{\lambda }}_k\mid C_i=k\right),$$ (3.1)

It is assumed that the response vectors ${\mathit{y}}_1,{\mathit{y}}_2,\dots ,{\mathit{y}}_N$ are stochastically independent, the number of measurements on the $i^{th}$ person is $r_i$. For convenience of notation, we assumed $r_i=r$ for all $i=1,2,\dots ,N$, although that assumption is not aways necessary. It depends on whether the spacing of time points is regular over time across participants and how the covariance structure of ${\mathit{y}}_i$ is modeled. The elements in ${\mathit{y}}_i$ are indexed by $j=1,2,\dots ,r$, and there are $K$ latent classes (indexed by $k=1,2,\dots ,K$). The above model can be extended to allow time-stable covariates (vector ${\mathit{Z}}_i$, of dimension $q$) to model the probability of belonging to class $k$. The time-stationary $\mathit{Z}$ covariates are sometimes called risk factors, in the sense that they are included in the model to help characterize the “risks” of belonging to different classes. In this case, the mixing probabilities become depending on those covariates – i.e., ${\mathit{\pi }}_i={\left({\pi }_{i1},{\pi }_{i2},\dots {\pi }_{iK}\right)}^{\prime }$. Timevarying covariates (denoted by the $r\times p$ matrix ${\mathit{W}}_i$) can be included to model mean trajectories over time for those with the same covariate pattern as individual $i$.

It follows that the conditional distribution of the observed data for individual $i$, with values for time-dependent covariates in ${\mathbf{W}}_i={\left(w_{i1},\dots ,w_{ir}\right)}^{\prime }$ and/or risk factors ${\mathit{z}}_i={\left(z_{i1},z_{i2},\dots ,z_{iq}\right)}^{\prime }$, can be expressed as given in equation (3.2), with the probability of group membership modeled using a generalized logit function,

$$\begin{array}{l}f\left({\mathit{y}}_i\right)=f\left({\mathit{y}}_i\mid \mathit{\lambda },{\mathit{z}}_i,{\mathbf{W}}_i\right)=\sum _{k=1}^K\text{Pr}\left(C_i=k\mid {\mathit{z}}_i\right)f\left({\mathit{y}}_i,{\mathit{\lambda }}_k\mid C_i=k,{\mathit{W}}_i\right)\hfill \\ =\sum _{k=1}^K\frac{\text{exp}\left({\theta }_k+{\mathit{\beta }}_k^{\prime }{\mathit{z}}_i\right)}{{\sum }_{i=1}^K\text{exp}\left({\theta }_i+{\mathit{\beta }}_i^{\prime }{\mathit{z}}_i\right)}f\left({\mathit{y}}_i,{\mathit{\lambda }}_k\mid C_i=k,{\mathit{W}}_i\right).\hfill \end{array}$$ (3.2)

In LCTA, it is assumed that all observations, both among individuals and within individuals are uncorrelated, given that class membership is known. That implies that the multivariate distribution $f\left({\mathit{y}}_i,{\mathit{\lambda }}_k\mid C_i,{\mathit{W}}_i\right)$ in equation (3.2) can be written as the product of univariate distributions of each $y_{ij}$, $j=1,2,\dots ,r$ and associated parameters from $\lambda$ (equation (3.3)).

$$f\left({\mathit{y}}_i\right)=f\left({\mathit{y}}_i\mid \mathit{\lambda },{\mathit{z}}_i,{\mathbf{W}}_i\right)=\sum _{k=1}^K\text{Pr}\left(C_i=k\mid {\mathit{z}}_i\right){\prod }_{j=1}^{r}f\left(y_{ij},{\mathit{\lambda }}_k\mid C_i,{\mathit{W}}_i\right)$$ (3.3)

For example, if in each class, $f\left({\mathit{y}}_i,\mathit{\lambda }\right)$ is multivariate normal, with $\mathit{\lambda }$ being the set of parameters in the mean vector $\mathit{\mu }={\left({\mu }_1,\dots ,{\mu }_r\right)}^{\prime }$ and variance/covariance matrix $\mathbf{\Sigma },$, then $f\left(y_{ij},\mathit{\lambda }\right)$ is univariate normal with mean ${\mu }_j$, and variance ${\sigma }_j^2$, where ${\sigma }_j^2$ is the $j^{th}$ diagonal of $\mathbf{\Sigma }$. This implicitly assumes that the off-diagonal elements of $\mathbf{\Sigma }$ are 0 (uncorrelated). Thus, any within-person correlation in $\mathbf{\Sigma }$ is ignored in the likelihood estimation; only the diagonal variances are estimated.

It follows that in LCTA, the likelihood function that is optimized to estimate number of classes and class-specific trajectories can be written as in equation (3.4).

$$\begin{gathered} ℒ\left(\pi ,\mathit{\lambda };{\mathit{y}}_1,\dots ,{\mathit{y}}_N\right)=\prod _{i=1}^{N}f\left({\mathit{y}}_i\right) \\ =\prod _{i=1}^N\left\{{\sum }_{k=1}^K\mathrm{Pr}\left(C_i=k\mid {\mathit{z}}_i\right)f\left({\mathit{y}}_i,{\mathit{\lambda }}_k\mid C_i,{\mathit{W}}_i\right)\right\} \\ =\prod _{i=1}^N\left[{\sum }_{k=1}^K\left\{\mathrm{Pr}\left(C_i=k\mid {\mathit{z}}_i\right)\prod _{j=1}^{r}f\left(y_{ij},{\mathit{\lambda }}_{kj}\mid C_i,{\mathit{w}}_{ij}\right)\right\}\right] \end{gathered}$$ (3.4)

In the CPMM framework, the assumption of intra-individual independence over time can be relaxed to accommodate longitudinal within-individual correlation by allowing ${\mathit{\Sigma }}_{\mathbf{k}}$ to be any valid covariance matrix of the multivariate normal. It is the difference between using the product of $r$ univariate normal distributions (LCTA) versus using an $r$-dimensional multivariate normal distribution in the likelihood function.

In comparison, the GMM model defines the class-specific multivariate distribution $f\left({\mathit{y}}_i,{\mathit{\lambda }}_k\mid C_i=k,{\mathit{W}}_i\right)$ in equation (3.1) as the product of univariate distributions at each time point, conditional on a $q\times 1$ vector of random effects for the $i^{th}$ individual $\left({\gamma }_i\right)$ and the distribution of ${\mathit{\gamma }}_i$ (equation (3.5)). In most settings, $f\left({\mathit{\gamma }}_i\right)$ in class $k$ is assumed to follow a $q$-dimensional multivariate normal with mean vector 0 and covariance matrix, ${\mathit{G}}_k$. The product in (3.5) is a result of the conditional independence assumption, where ${\mathbf{\Sigma }}_k$, captured in ${\lambda }_k$, is a diagonal matrix.

$$f\left({\mathit{y}}_i,{\mathit{\lambda }}_k,{\mathit{G}}_k\mid C_i=k,{\mathit{W}}_i\right)=\prod _{j=1}^r f\left(y_{ij},{\lambda }_k\mid C_i=k,{\mathit{W}}_i,{\mathit{\gamma }}_i\right)f\left({\gamma }_i\mid {\mathit{G}}_k\right)$$ (3.5)

To eliminate the dependence of the full likelihood on unobservable random effects, we marginalize the component in (3.5) of the full likelihood by integrating over the distribution of ${\gamma }_i$, whereby it becomes a function of both the parameters in ${\lambda }_k$ and in ${\mathit{G}}_k$.

Both the LCTA and GMM methodology and associated software packages, can support a number of univariate response variable distributions for the conditional distribution in (3.5). Those most commonly found in the software are members of the exponential family (e.g., normal/Gaussian, Poisson, Bernoulli/Binomial, and exponential distributions), but others (e.g., log-normal, truncated normal) may be allowed as specified in the documentation. When observations are independent with the same mean trajectory within class (e.g., LCTA), then the likelihood in equation (3.4) would hold for any univariate distribution $f\left(\right)$. However, if within-individual correlation is allowed, after conditioning on class membership, then the likelihood function requires the full multivariate distribution $f\left(\right)$ for the vector ${\mathit{y}}_i$ as described in (3.1). In the case of normally-distributed data (i.e., CPMM), this is well-defined and easy. However, for other distributions, expressing the full multivariate likelihood can be more challenging (e.g., Bahadur’s representation of an $r$-dimensional Bernoulli distribution²⁸) and is not available in current software. Rather, the approach in (3.5) is used to form the multivariate likelihood in GMM - that is, using univariate distributions conditional on random effects and class membership, multiplied by the marginal distribution of random effects, which is usually assumed to be the same across classes. Maximum likelihood estimation in used in LCTA, GMM and CPMM, and capability for fitting CPMMs is usually included in GMM software.

3.2. Estimation

The marginal distribution $f\left({\mathit{y}}_i\right)$ depends on class membership through the ${\pi }_{ik}$, which is a function of the parameters and covariates $\left({\mathit{z}}_i\right)$ in the logistic model component, and the mean and variance parameters in ${\mathit{\lambda }}_k$ needed to define the conditional distribution -- $f\left({\mathit{y}}_i{\mathit{\lambda }}_k\mid C_i=k,{\mathit{\gamma }}_{\mathit{i}}\right)$ (equation (3.2)) for CPMM and LCTA and $f\left({\mathit{y}}_i,{\mathit{\lambda }}_k,{\mathit{G}}_k\mid C_i=k\right)$ for GMM after integrating out the random effects. The EM algorithm is often used to maximize the likelihood where the latent (unobserved) variables corresponding to class membership $\left(C_i\right)$ are treated as missing data.

3.3. Class Enumeration and Other Considerations

A major consideration in the practical application of LCTA, GMM and CPMM is class enumeration; that is, determining the correct number of latent classes $K$. This is based on trajectory parameters and variability seen within each class, with a goal of separating individual trajectories into groups (classes) which are more similar within the class than they are to other classes. Current practice is to rely on one or more measures of fit to make that decision, given which measures are available in the analyst’s software. Common measures used are information criterion (IC) indices, an entropy measure, or statistical tests based on the likelihood comparisons of nested models. Since the number of classes must be specified before parameters can be estimated, a typical approach is to fit a $k$-class model and compare it to a $k-1$ class model, to determine if the additional class provides a substantially better fit. The process stops when additional classes and associated additional within-class parameters are no longer improving the fit to the data. Common IC metrics include Akaike’s Information Criterion (AIC),^29,30 consistent AIC (CAIC),³¹ bias-corrected AIC (AICC),^32,33 Bayesian Informatic Criterion (BIC),²⁶ and Sclove’s sample size adjusted BIC (ABIC).³⁴ Entropy is based on estimation of the overlap of the distributions via the mixing probabilities.³⁵ It ranges from 0 (effectively random assignment to classes) to 1 (perfect assignment).

When using the likelihood ratio test for comparing a $k$-class model to a ($k-1$)-class model, one component of that test is the hypothesis that ${\pi }_k=0$, which is testing a parameter on the boundary of its parameter space. In these situations, the usual LRT asymptotic null distribution is no longer appropriate.³⁶ A non-standard likelihood ratio test (LRT), known as the Lo-Mendell-Rubin test (LMR)³⁷ or the bootstrap LRT (BLRT) have been used in this setting. However, there are cautions, as the ($k-1$)-class model may not be truly nested within the $k$ class model.³⁸ Another approach to comparing two models is the Bayes factor (denoted $B_{10}$), which is the posterior odds for one model against the other, assuming neither is favored a priori.³⁹ A more empirical rule was suggested by Jones et al.⁴⁰, based on the work of Kass and Raftery⁴¹, to approximate the natural log of $B_{10}$ with the difference in $\mathrm{B}\mathrm{I}\mathrm{C}$, as presented in equation (3.7). Here, $\mathrm{\Delta }\mathrm{B}\mathrm{I}\mathrm{C}$ is the BIC of the more complex model minus the BIC of the simpler model.

$$2{\mathit{log}}_e\left(B_{10}\right)\approx 2\left(\mathrm{\Delta }\mathrm{B}\mathrm{I}\mathrm{C}\right).$$ (3.7)

No one index or approach to class enumeration has been shown to consistently outperform the others, as factors such as class sample sizes, separation of class trajectories, imbalance in mixing proportions across classes, model misspecification, and use of covariates in estimating mixing proportions impact these measures.^7,38,42

3.4. Impact of Model Misspecification on LCTA, GMM and CPMM

Our work is focused on the impact of misspecification of the temporal correlation structure, separate from variance, in the CPMM/LCTA setting. Several authors have touched on this by focusing on the impact of model misspecification in the GMM/LCTA setting, and by removing individual-level random effects, in the CPMM setting. All work discussed here stayed within the framework of Gaussian responses with the identity link, where temporal correlation structure is dependent on the covariance matrix of random effects and the covariance matrix of the additive residual error term. Because convergence issues often arise when using very general GMM models, much of the work is focused on the impact of simplifying such models to obtain convergence, such as imposing homogeneous variance structures within- or across classes measures,^{16–19,33,43} and/or imposing temporal independence assumptions.^16–19,43 The considered impacts of these constraints in their simulation studies vary across the authors, but typical ones include ability to recover the class structure of the generated data,^16–19,43 bias and precision in model parameters,^18,19,43 convergence issues,¹⁶ accuracy in class assignment,^16–18,43 and which enumeration measures were most insensitive to misspecified models.¹⁸ Besides constraints on covariance parameters, other factors affecting these measures of performance consideration were: relative magnitude of variances across classes,^17,18 class separation,^17–19 sample size in each class,^18,19 mixing probabilities,^18,19 and number of time points.¹⁹ A couple of authors varied the level of correlation in the random error term distribution, including 0 correlation,^15,17,19 but the others assumed that matrix to be diagonal throughout their simulations. Allowing additional correlation in the additive random error vector in this setting can sometimes create an overparameterized covariance structure, if there are also random effects in the model, which leads to non-convergence. Thus, it is common to restrict it to be diagonal, which is consistent with the conditional independence assumption needed for non-normally-distributed responses, if maximum likelihood estimation is to be used.

Heggeseth and Jewell¹⁵ presented asymptotic calculations of expected parameter bias in a simple two-class mixture of two multivariate Gaussian distribution with different means but common covariance. They considered an exchangeable and exponentially-declining correlation structures ($\rho =0,.5,.99$). different mixing probabilities, with same or different covariance structure in the two classes. They reported that larger separation reduced the magnitude of bias under model misspecification (especially when correlation was high), but bias was larger when true correlation was far from modeled correlation.

In general, these authors found that misspecification of the covariance structure led to bias in the trajectory and variance parameters, and reduced the likelihood that classes extracted would be consistent with those underlying the data generation. Both bias and precision worsened as the ratio of smallest to largest residual variance across classes became larger, as the number of classes with different residual variances increased, and mixing probabilities became more disparate. The greater the separation of classes and smaller the within-class variability, the better the method was at recovering the underlying class structure and reducing parameter bias, especially with high temporal correlation. However constraining correlation to be 0 (i.e., LCTA) when present disrupted the ability to recover the underlying class structure, both in number of classes extracted and matching the estimated trajectories of the extracted classes to those generated. Overparameterization of the covariance model (e.g., assuming temporal correlation when there was none), would lead to convergence problems, while the more the variance constraint(s) diverged from the truth of the underlying data, the poorer the modeled performed on class recovery, parameter bias and class accuracy.

McNeish and colleagues look at the modeling and convergence issues from a different perspective.²² They argue that random effect modeling within GMMs is intended to focus on person-to-person variability, but that most using that model and reporting results from such are not as interested in the interpretation of a ‘typical’ person in a class (i.e., a person whose random effect vector is the zero vector), but rather on the mean trajectory in the class. When assuming the responses to be normally-distributed and using the identity link function in the GMM, the difference comes down to how the within-class variability is modeled marginally, since the marginal mean parameter vector will be the same as that of the “typical” person. They justify this perspective by a review of the literature using such models and the lack of information presented in those papers that reflects an interest in the person-to-person variability. By removing the random effects from the model (within the Gaussian framework), the distribution of the remaining residual vectors can be more flexible in allowing different correlation patterns and are much less likely to have convergence issues, even when variances are not constrained within or across classes. Thus, they suggest that CPMMs are better suited to answer the question about an “average” trajectory in a class. In their simulation studies (ref), data was generated under different GMM variance constraints, including the case of no random effects (CPMM). They found that as conditions were less idea (i.e., lower sample sizes, less separation among classes, more attrition over time), the most general parameterization of the GMM had increasing difficulty converging. Constraining variances to be equal in those GMMs improved convergence greatly, but if class separation was low, the constrained GMM and LCTA models did not recover the generated classes very well. The CPMM models fit assumed a compound symmetric structure for ${\mathbf{\Omega }}_k$, which did not match the data generation (which had a decaying correlation structure over time), but is a common choice. Overall, the CPMM approach performed better across a number of scenarios, while performance of LCTA and GMM was more dependent on how well their assumptions matched the underlying data.

The studies cited above were focused on recovering the mixture of distributions with different trajectories, which is a common goal when using these methods and maintains the same definition of “class” throughout. However, it can be argued that this is not always the goal, such as when using LCTA in the presence of temporal correlation, because the meaning of ‘class’ is not directly related to how the data were generated. Instead, ‘class’ in LCTA implies a collection of trajectories that are similar to each other and different from those in other classes.

To our knowledge, no work has been published studying the impact of misspecifying the correlation structure, separate from the variance, on LCTA and CPMM and if that impact changes with the strength of the correlation. This work aims to fill that gap in the literature.

4. SIMULATION STUDIES

Because our goal was to study the impact of correlation structure misspecification on latent class methods for longitudinal data, we varied the underlying correlation structure and the magnitude of within-individual correlation in our simulation study. We also varied the sample size and number of repeated measures per individual. All other relevant modeling parameters were held constant across simulation scenarios.

4.1. Simulation Design

Data was randomly generated under the following the model, conditional on the ${\mathit{y}}_i$ being a member of the $k^{\text{th}}$ class:

$$y_{ij.k}={\beta }_{0.k}+{\beta }_{1.k}{t}_{ij}+{\epsilon }_{ij}$$

where ${\mathit{y}}_{i.k}={\left(y_{i1.k},y_{i2.k},\dots y_{ir.k}\right)}^{\prime }$, ${\mathit{t}}_i={\left(t_{i1},t_{i2},\dots t_{ir}\right)}^{\prime }$, ${\epsilon }_i={\left({\epsilon }_{i1},{\epsilon }_{i2},\dots {\epsilon }_{ir}\right)}^{\prime }$, ${\mathit{\epsilon }}_i\sim MV{N}_r(0,\mathrm{\Sigma })$, and ${\mathit{\epsilon }}_i$ independent of ${\mathit{\epsilon }}_{i^{\prime }}$, $i\ne i^{\prime }$ for $i,i^{\prime }=1,\dots ,n_k$ (the number of individuals in class $c$) and $j=1,\dots ,r$ (the number of repeated measurements on each individual) for $k=1,\dots ,K$ (the number of latent classes). We assumed common measurement time points, the class-specific trajectories were linear in time and that the covariance structure was homogeneous across classes.

We set $K=3$ (three latent classes), $r=5$ with $t_i={\left(\begin{array}{lllll}0& 1& 2& 3& 4\end{array}\right)}^{\prime }$ (five equally-spaced longitudinal measures with no missing data), and $n_k=100$ (three equal-sized classes of 100 individuals for a total sample size of 300) for the primary simulation study. In secondary studies, we set $n_k=250$ (total sample of 750) and $n_k=500$ (total sample of 1,000) for $r=5$ with ${\left.t_i=(\begin{array}{llll}0& 1& 2& 3\end{array}4\right)}^{\prime }$, and we set $r=7$ with $t_i={\left(\begin{array}{lllll}0& 1& 2& \dots 5& 6\end{array}\right)}^{\prime }$ and $r=10$ with $t_i={\left(\begin{array}{lllll}0& 1& 2& \dots 8& 9\end{array}\right)}^{\prime }$ for $n_k=100$ (total sample size of 300). The value of each class’s intercept and slope parameters were set to ensure complete separation of the population-mean class trajectories. We set ${\beta }_{0.1}=0.5$ and ${\beta }_{1.1}=0$ (the trajectory parameters of Class 1). The trajectory parameters of Classes 2 and 3 were obtained by increasing the intercept and slope parameters of the previous class by one standard error of the underlying regression parameters. That is, we set $\left({\beta }_{0.2},{\beta }_{1.2}\right)=(1.14,0.32)$ for Class 2 and $\left({\beta }_{0.3},{\beta }_{1.3}\right)=(1.78,0.68)$, for Class 3.

Only $\mathrm{\Sigma }$ was varied across different simulation settings. We generated data under three covariance structures: independent, autoregressive of order 1 (AR(1)), and compound symmetric (CS). For the latter two, we varied the strength of the within-individual correlation coefficient. Specifically, we set $\rho$ to be one of following: (0.05, 0.1, 0.25, 0.5). It should be noted that the independent covariance structure is equivalent to the AR(1) and CS structures with $\rho =0$. We chose to refer to this structure as independent to emphasize the lack of within-individual correlation. For all covariance structures, the diagonal variance terms were set to 1. This resulted in 9 simulation settings.

For each simulation setting, 500 replicate datasets were randomly generated. Figure 2 displays the observed individual-level trajectories of 300 subjects for one simulated dataset under a compound symmetric correlation structure with $\rho =0.50$. From Figure 2a, we see that true group membership is not easily discerned when examining the observed data values in totality. From Figure 2b, we see that even with complete separation of the population-mean class trajectories, the variability about these mean trajectories can cause overlap in the observed individual-level trajectories of subjects from different classes.

Figure 2.

Observed individual-level trajectories of 300 subjects for one simulated dataset under a compound symmetric (rho = 0.50) correlation structure with (a) assumed population-mean trajectories overlaid and (b) individual-level trajectories colored by true class membership.

For each replicate dataset, 3 models were fit: (1) LCTA, (2) CPMM modeled with a one parameter AR(1) correlation structure (referred to here as CPMM-AR1), and (3) CPMM modeled with a one-parameter CS correlation structure (referred to here as CPMM-CS). Recall that LCTA assumes an independent structure, so fitting it to data generated under an AR(1) or CS correlation structure with ρ > 0 indicates a misspecified correlation structure. For CPMM, the correlation structure was only misspecified if the correlation structure used to generate the data was different from the correlation used to model the data. By varying the value of the correlation parameter, $\rho$, we were able study how the impact of the correlation structure misspecification may change depending on the strength of the within-individual correlation, as well as the impact of within-individual correlation strength even when the correlation structure is correctly specified.

LCTA, CPMM-AR1, and CPMM-CS models were fit assuming 1, 2, 3 (truth), 4, and 5 classes and the observed value of the following fit indices captured: Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and sample size adjusted BIC (ABIC). Estimates of the class-specific trajectories parameters were also captured for each model fit. The performance of each model was evaluated based on two criteria: (1) class enumeration, that is, could the model identify the true number of latent classes present in the data and (2) parameter estimation, that is, could the model accurately and precisely estimate the population-mean class trajectories when the true number of classes is used in the model fit? Results are reported for each combination of simulation setting, fit index, and model.

To evaluate class enumeration, the optimum model based on a given fit index was identified among 1, 2, 3, 4, or 5 classes and the proportion of the 500 replicate datasets selecting a given number of classes was calculated. To evaluate parameter estimation, the class-specific intercept and slope estimates from the (correct) 3-class model were captured for each of the 500 replicate datasets, and the bias and the standard deviation of the bias estimated by finding the mean and standard deviation, respectively, of the deviation of model estimates from the true parameter values (${\beta }^{\prime }\mathrm{s}$) listed above. To understand the role of class mixing in parameter estimation, the median (25^th, 75^th percentile) number of subjects from true Class 1, 2, or 3 who were assigned to each class from the 3-class model fit were calculated. Ideally, assuming the correct model specification, the models (in combination with a fit criterion) would choose the true number of classes in 100% of the replicated datasets and all individuals would be classified into their correct class, leading to unbiased and minimum variance trajectory parameters.

LCTA was performed with the SAS software using the PROC TRAJ add-on procedure which maximized the likelihood using a general quasi-Newton algorithm^40,44,45. CPMM was performed with the Mplus software, which uses the E-M algorithm for maximizing the likelihood in mixture models^46,47. The R package MplusAutomation⁴⁸ was used the implement the simulation in Mplus. Results tables and figures were generated using the R software⁴⁹, version 3.6.1.

4.2. Class Enumeration Results

In Table 1, we report the proportion of 500 replicate datasets selecting 2, 3, and 4 or more (4+) classes for the LCTA, CPMM-AR1, and CPMM-CS models for each simulation setting and fit index. When data were generated with no correlation ($\rho =0$), LCTA performed class enumeration well, selecting 3 classes in 84%, 100%, and 85% of the replicated datasets using AIC, BIC, and ABIC, respectively. Similar results were observed for CPMM, with CPMM performing slightly worse than LCTA when the AIC selection criterion was used to choose the optimum fit (71% for CPMM-AR1 and CPMM-CS vs. 84% for LCTA).

Table 1.

Class enumeration results: Proportion of 500 randomly generated datasets that selected K classes for data generated with 3 classes and 100 subjects per class by estimation method, selection criterion, and correlation structure and level assumed when generating the data.

LCTA Method
Generated as		AIC			BIC			Sample Size Adjusted BIC
True ∑	True ρ	K = 2	K = 3	K = 4+	K = 2	K = 3	K = 4+	K = 2	K = 3	K = 4+
I = AR(1) = CS	0.00	0.00	0.84	0.16	0.00	1.00	0.00	0.00	0.85	0.15
AR(1)	0.05	0.00	0.71	0.29	0.00	0.99	0.01	0.00	0.74	0.26
AR(1)	0.10	0.00	0.39	0.61	0.00	0.98	0.02	0.00	0.46	0.54
AR(1)	0.25	0.00	0.01	0.99	0.00	0.45	0.55	0.00	0.00	1.00
AR(1)	0.50	0.00	0.00	1.00	0.00	0.00	1.00	0.00	0.00	1.00
CS	0.05	0.00	0.60	0.40	0.00	0.98	0.02	0.00	0.67	0.33
CS	0.10	0.00	0.19	0.81	0.00	0.83	0.17	0.00	0.23	0.77
CS	0.25	0.00	0.00	1.00	0.00	0.03	0.97	0.00	0.00	1.00
CS	0.50	0.00	0.00	1.00	0.00	0.00	1.00	0.00	0.00	1.00
CPMM Method Modeled as AR(1)
Generated as		AIC			BIC			Sample Size Adjusted BIC
True ∑	True ρ	K = 2	K = 3	K = 4+	K = 2	K = 3	K = 4+	K = 2	K = 3	K = 4+
I = AR(1) = CS	0.00	0.00	0.71	0.29	0.00	1.00	0.00	0.00	0.84	0.16
AR(1)	0.05	0.00	0.76	0.24	0.00	1.00	0.00	0.00	0.86	0.14
AR(1)	0.10	0.00	0.75	0.25	0.00	1.00	0.00	0.00	0.89	0.11
AR(1)	0.25	0.00	0.74	0.26	0.16	0.84	0.00	0.00	0.87	0.13
AR(1)	0.50	0.11	0.68	0.20	0.75	0.25	0.00	0.20	0.69	0.11
CS	0.05	0.00	0.59	0.41	0.00	1.00	0.00	0.00	0.76	0.24
CS	0.10	0.00	0.30	0.70	0.00	0.95	0.05	0.00	0.46	0.54
CS	0.25	0.00	0.01	0.99	0.00	0.35	0.65	0.00	0.02	0.98
CS	0.50	0.00	0.00	1.00	0.00	0.01	0.99	0.00	0.00	1.00
CPMM Method Modeled as CS
Generated as		AIC			BIC			Sample Size Adjusted BIC
True ∑	True ρ	K = 2	K = 3	K = 4+	K = 2	K = 3	K = 4+	K = 2	K = 3	K = 4+
I = AR(1) = CS	0.00	0.00	0.71	0.29	0.03	0.97	0.00	0.00	0.83	0.17
AR(1)	0.05	0.00	0.62	0.38	0.10	0.89	0.00	0.00	0.79	0.21
AR(1)	0.10	0.00	0.50	0.50	0.19	0.81	0.01	0.01	0.67	0.32
AR(1)	0.25	0.00	0.09	0.91	0.43	0.45	0.12	0.01	0.19	0.80
AR(1)	0.50	0.00	0.01	0.99	0.03	0.57	0.40	0.00	0.05	0.95
CS	0.05	0.00	0.79	0.21	0.15	0.85	0.00	0.01	0.88	0.11
CS	0.10	0.01	0.74	0.25	0.28	0.72	0.00	0.03	0.86	0.11
CS	0.25	0.05	0.72	0.23	0.59	0.41	0.00	0.09	0.79	0.12
CS	0.50	0.04	0.77	0.19	0.53	0.47	0.00	0.07	0.85	0.08

(1) I = Independent; AR(1) = auto-regressive lag 1; CS = compound symmetric; LCTA method always modeled as Independent

When data were generated with correlation (ρ > 0), LCTA’s class enumeration performance using AIC or ABIC quickly deteriorated, even for weak or moderate correlations and across all indices for stronger correlation. For example, when $\rho =0.10$, LCTA selected 3 classes in 39%, 98%, and 46% of the replicated datasets using AIC, BIC, and ABIC, respectively, when data were generated under an AR(1) correlation structure and 19%, 83%, and 23% when data were generated under a CS correlation structure. The deterioration was more pronounced for the CS structure compared to the AR(1) structure. While LCTA’s performance was better preserved when using BIC, the performance did deteriorate when the correlation level was moderate to strong. For example when $\rho =0.25$, LCTA selected 3 classes in 1%, 45%, and 0% of the replicated datasets using AIC, BIC, and ABIC, respectively, when data were generated under an AR(1) correlation structure and 0%, 3%, and 0% when data were generated under a CS correlation structure. When LCTA did not select the correct number of classes, LCTA invariably over-estimated the number of classes, selecting 4 or more classes for all simulation settings studied.

In contrast, CPMM’s class enumeration performance did not deteriorate, if at all, until strong correlations were present, if the correct correlation structure was specified in the model. When the data were generated under AR(1) structure with $\rho =0.25$, CPMM-AR1 selected 3 classes in 74%, 84%, and 87% of the replicated datasets using AIC, BIC, and ABIC, respectively. However, when $\rho =0.50$, the corresponding percentages of correct classification dropped to 68%, 25%, and 69%. When data were generated under a CS correlation structure, CPMM-CS’s performance was preserved for all correlation levels for the fit indices AIC and ABIC (72%-88%) and BIC with no or low correlation (85%, 72%) but not with higher correlations (41%, 47%). When an incorrect correlation structure was used in CPMM, that is, CPMM-AR1 fit to data generated under a CS correlation structure or CPMM-CS fit to data generated under an AR(1) correlation structure, CPMM’s class enumeration performance was similar to that of LCTA, deteriorating quickly at weak or moderate levels of correlation.

Unlike LCTA, when CPMM did not select the correct number of classes, there was heterogeneity in whether the number of classes was under- or over-estimated. If the correct correlation structure was used in the model, using the BIC index resulted in under-estimating the number of classes more often than over-estimating it. However, using the AIC or ABIC index resulted in both under- and over-estimating the number of classes, with over-estimation being more likely and the frequency of under-estimation increasing with the level of correlation. If an incorrect correlation structure was specified in the model, CPMM generally over-estimated the number of classes when 3 classes was not selected. The only exception was CPMM-CS fit to data generated under an AR(1) correlation structure using the BIC index; in this setting, CPMM more frequently under-estimated the number classes for weak to moderate correlation levels and over-estimated for strong correlation levels.

When the sample size increased, class enumeration results improved for BIC and ABIC and remained the same for AIC when the correct correlation structure was specified in the model fit (i.e., LCTA fit to data generated an independent correlation structure, CPMM-AR1 fit to data generated under an AR(1) correlation structure, and CPMM-CS fit to data generated under a CS correlation structure). Notably, CPMM selected 3 classes in at least 98% of the replicated data sets for all correlation levels when the correct correlation structure was used in the model fit. Similar improvements in class enumeration occurred when the number of repeated measures per individual was increased. However, when the correlation structure was misspecified, class enumeration results deteriorated with increasing sample size for all 3 models. Specifically, the number of classes was over-estimated more frequently and for lower levels of correlations. The same deterioration was not observed when increasing the number of repeated measures per individual; the results remained more consistent. The only exception was fitting the LCTA and CPMM-AR1 models to data generated under a CS correlation structure where the number of classes was more frequently over-estimated as the number of repeated measures increased. Class enumeration results are reported in Table S1 and S4 for increasing sample sizes in Table S7 and S10 for increasing numbers of repeated measures.

4.3. Parameter Estimation Results

In Table 2, we report the estimated mean bias and its standard deviation for the class-specific trajectory parameters from a 3-class model fit, for the LCTA, CPMM-AR1, and CPMM-CS models in each simulation setting across the 500 replicates.

Table 2.

Parameter estimation results: Estimated mean bias (standard deviation) of trajectory parameter estimators from 3-class model of 500 randomly generated datasets with 3 classes and 100 subjects per class by estimation method, assigned class, and correlation structure and level assumed when generating the data.

LCTA Method
		Class 1		Class 2		Class 3
True ∑	True ρ	Intercept	Slope	Intercept	Slope	Intercept	Slope
I = AR(1) = CS	0.00	0.000 (0.082)	−0.002 (0.034)	−0.000 (0.094)	−0.007 (0.040)	0.001 (0.082)	−0.048 (0.035)
AR(1)	0.05	−0.017 (0.086)	0.002 (0.038)	0.006 (0.092)	−0.006 (0.041)	0.009 (0.088)	−0.046 (0.037)
AR(1)	0.10	−0.013 (0.095)	−0.003 (0.037)	−0.000 (0.101)	−0.003 (0.042)	0.024 (0.093)	−0.051 (0.039)
AR(1)	0.25	−0.057 (0.108)	0.001 (0.044)	0.007 (0.112)	−0.005 (0.051)	0.058 (0.104)	−0.047 (0.039)
AR(1)	0.50	−0.174 (0.129)	0.009 (0.043)	0.004 (0.118)	−0.002 (0.057)	0.178 (0.125)	−0.054 (0.044)
CS	0.05	−0.028 (0.095)	0.002 (0.035)	−0.003 (0.093)	−0.003 (0.041)	0.030 (0.085)	−0.049 (0.035)
CS	0.10	−0.061 (0.091)	0.007 (0.034)	0.002 (0.093)	−0.001 (0.044)	0.066 (0.089)	−0.054 (0.032)
CS	0.25	−0.170 (0.107)	0.021 (0.035)	0.000 (0.100)	−0.005 (0.049)	0.161 (0.111)	−0.065 (0.033)
CS	0.50	−0.316 (0.135)	0.038 (0.031)	0.013 (0.110)	0.001 (0.049)	0.344 (0.136)	−0.085 (0.029)
CPMM Method Modeled as AR(1)
		Class 1		Class 2		Class 3
True ∑	True ρ	Intercept	Slope	Intercept	Slope	Intercept	Slope
I = AR(1) = CS	0.00	0.000 (0.082)	−0.002 (0.034)	−0.000 (0.094)	−0.007 (0.040)	0.001 (0.082)	−0.048 (0.035)
AR(1)	0.05	0.006 (0.083)	−0.003 (0.034)	−0.003 (0.098)	−0.001 (0.042)	0.002 (0.088)	−0.048 (0.036)
AR(1)	0.10	0.002 (0.095)	−0.003 (0.037)	0.000 (0.104)	−0.003 (0.042)	0.009 (0.093)	−0.051 (0.039)
AR(1)	0.25	−0.007 (0.112)	0.003 (0.053)	0.008 (0.130)	−0.005 (0.062)	0.008 (0.104)	−0.048 (0.041)
AR(1)	0.50	−0.035 (0.213)	0.007 (0.110)	0.008 (0.174)	−0.001 (0.102)	0.027 (0.167)	−0.047 (0.063)
CS	0.05	−0.024 (0.095)	0.003 (0.035)	−0.003 (0.093)	−0.003 (0.041)	0.026 (0.086)	−0.049 (0.035)
CS	0.10	−0.054 (0.091)	0.007 (0.034)	0.002 (0.094)	−0.001 (0.044)	0.058 (0.089)	−0.054 (0.032)
CS	0.25	−0.153 (0.108)	0.020 (0.035)	−0.001 (0.103)	−0.006 (0.051)	0.143 (0.111)	−0.064 (0.033)
CS	0.50	−0.294 (0.151)	0.032 (0.033)	0.006 (0.126)	−0.002 (0.059)	0.311 (0.151)	−0.081 (0.030)
CPMM Method Modeled as CS
		Class 1		Class 2		Class 3
True ∑	True ρ	Intercept	Slope	Intercept	Slope	Intercept	Slope
I = AR(1) = CS	0.00	0.001 (0.083)	−0.002 (0.034)	−0.000 (0.094)	−0.007 (0.041)	0.000 (0.084)	−0.048 (0.035)
AR(1)	0.05	0.016 (0.089)	−0.005 (0.037)	−0.002 (0.105)	−0.002 (0.057)	−0.006 (0.094)	−0.048 (0.057)
AR(1)	0.10	0.025 (0.107)	−0.005 (0.052)	0.004 (0.122)	−0.004 (0.097)	−0.013 (0.109)	−0.051 (0.081)
AR(1)	0.25	0.116 (0.210)	0.092 (0.262)	0.002 (0.194)	−0.014 (0.365)	−0.110 (0.214)	−0.131 (0.244)
AR(1)	0.50	0.421 (0.166)	0.299 (0.425)	0.006 (0.131)	−0.007 (0.453)	−0.423 (0.163)	−0.345 (0.415)
CS	0.05	−0.002 (0.113)	0.002 (0.088)	−0.004 (0.103)	−0.003 (0.045)	0.005 (0.102)	−0.048 (0.078)
CS	0.10	−0.001 (0.098)	0.000 (0.036)	0.004 (0.114)	−0.001 (0.054)	0.008 (0.103)	−0.048 (0.070)
CS	0.25	−0.011 (0.115)	−0.001 (0.048)	−0.002 (0.153)	−0.004 (0.100)	0.001 (0.122)	−0.044 (0.046)
CS	0.50	−0.016 (0.139)	−0.007 (0.051)	−0.002 (0.174)	−0.010 (0.088)	0.020 (0.134)	−0.046 (0.042)

(1) The class output of CPMM and LCTA was aligned to the true classes in the simulation model based on estimated trajectory parameters.

(2) I = Independent; AR(1) = auto-regressive lag 1; CS = compound symmetric; LCTA method always modeled as Independent.

For LCTA, there was little to no bias for the intercept and slope parameters across all 3 classes when data were generated with no correlation ($\rho =0$). When data were generated with correlation (ρ > 0), the intercept bias increased as the level of correlation increased for Classes 1 and 3, but remained low for Class 2. For example, when the data were generated under an AR(1) correlation structure, the estimated intercept bias was 0.000, 0.000, and 0.001 for Class 1, 2, and 3, respectively, when $\rho =0$, was −0.013, 0.000, and 0.006 when $\rho =0.10$, and was −0.174, 0.004, and 0.178 when $\rho =0.50$. A similar pattern was observed for data generated under a CS correlation structure; however, the magnitude of the intercept bias was larger. Unlike the intercept bias, the slope bias remained stable as the correlation level increased when data were generated under an AR(1) correlation structure and increased slightly for strong correlation ($\rho =0.50$) when data were generated under a CS correlation structure.

Similar results were observed for CPMM-AR1 and CPMM-CS when the incorrect correlation structure was used in CPMM. When the correct correlation structure was used in the model fit, both the intercept and slope bias were near zero for all correlation levels, with a small increase in bias for Class 1 and 3 when the correlation was strong. For example, when $\rho =0.05$ vs. $\rho =0.50$, the Class 1 intercept bias was 0.006 vs. −0.035 for CPMM-AR1 fit to data generated under an AR(1) correlation structure and was −0.002 vs. −0.016 for CPMM-CS fit to data generated under an CS correlation structure.

Interestingly, for most simulation settings, the intercept bias in Class 1 was approximately equal in magnitude to the intercept bias in Class 3, but in the opposite direction. The intercept bias was generally negative for Class 1 and positive for Class 3, with the exception of fitting CPMM-CS to data generated under an AR(1) correlation structure where the pattern was reversed. Supplemental Figures 1 - 6 demonstrate these phenomena across the simulation settings. We posit the larger bias observed for Class 1 and Class 3 intercept parameters in some scenarios can be explained by class mixing. That is, the classes generated by the model fits were mixtures of the true classes.

In Table 3, we report the median (25^th percentile, 75^th percentile) number of subjects across 500 replicate datasets from true Class 1, 2, or 3 who were assigned to each estimated class from a 3-class fit for the LCTA, CPMM-AR1, and CPMM-CS models in each simulation setting. Class mixing was observed for each simulation setting for all 3 models. The general mixing pattern was as follows: most subjects in each assigned class were truly from that class. A small portion of the subjects assigned to Class 1 were truly from Class 2, and, similarly, a small portion of the subjects assigned to Class 2 were truly from Class 1. Similar mixing was observed between Class 2 and 3.

Table 3.

Class assignment results: Median (25^th, 75^th percentile) number of subjects in assigned class versus true class from 3-class model for 500 randomly generated datasets with 3 classes and 100 subjects per class by estimation method and correlation structure and level assumed when generating the data.

LCTA Method
Generated as		Assigned Class 1			Assigned Class 2			Assigned Class 3
True ∑	True ρ	True C1	True C2	True C3	True C1	True C2	True C3	True C1	True C2	True C3
I = AR(1) = CS	0.00	93 (91,95)	6 (4,9)	0 (0,0)	7(5,9)	87 (84,90)	6 (4,9)	0 (0,0)	6 (4,9)	94 (91,96)
AR(1)	0.05	93 (90,95)	7(5,10)	0 (0,0)	7(5,10)	86 (82,89)	7(5,10)	0 (0,0)	7(5,10)	93 (90,95)
AR(1)	0.10	91 (88,94)	8 (5,10)	0 (0,0)	9 (6,12)	84 (81,88)	8 (6,12)	0 (0,0)	7(5,10)	92 (88,94)
AR(1)	0.25	88 (84,92)	10 (6,13)	0 (0,0)	12 (8,16)	81 (76,84)	12 (8,16)	0 (0,0)	9 (6,13)	88 (84,92)
AR(1)	0.50	82 (75,87)	12 (8,16)	0 (0,0)	18 (13,24)	76 (71,79)	19 (14,25)	0 (0,0)	11 (8,16)	81 (75,86)
CS	0.05	91 (88,94)	7(5,10)	0 (0,0)	9 (6,12)	84 (81,88)	8 (6,11)	0 (0,0)	7(5,10)	92 (89,94)
CS	0.10	90 (86,93)	9 (6,12)	0 (0,0)	10 (7,14)	82 (78,86)	11 (7,15)	0 (0,0)	8 (5,12)	89 (85,93)
CS	0.25	84 (78,89)	11 (7,15)	0 (0,0)	16 (11,21)	77 (73,81)	16 (11,22)	0 (0,0)	10 (7,15)	84 (78,89)
CS	0.50	78 (71,84)	14 (10,19)	0 (0,0)	22 (16,29)	71 (67,75)	25 (18,31)	0 (0,0)	13 (9,18)	75 (69,82)
CPMM Method Modeled as AR(1)
Generated as		Assigned Class 1			Assigned Class 2			Assigned Class 3
True ∑	True ρ	True C1	True C2	True C3	True C1	True C2	True C3	True C1	True C2	True C3
I = AR(1) = CS	0.00	93 (91,95)	6 (4,9)	0 (0,0)	7(5,9)	87 (84,90)	6 (4,9)	0 (0,0)	6 (4,9)	94 (91,96)
AR(1)	0.05	93 (90,95)	7(4,10)	0 (0,0)	7(5,10)	86 (82,88)	7(5,10)	0 (0,0)	7(5,10)	93 (90,95)
AR(1)	0.10	92 (89,95)	8 (5,11)	0 (0,0)	8 (5,11)	84 (80,87)	8 (6,11)	0 (0,0)	8 (5,11)	92 (89,94)
AR(1)	0.25	90 (84,93)	10 (6,15)	0 (0,0)	10 (7,16)	79 (73,83)	10 (7,15)	0 (0,0)	10 (6,14)	90 (85,93)
AR(1)	0.50	86 (78,91)	13 (8,20)	0 (0,0)	14 (9,22)	73 (65,78)	15 (8,24)	0 (0,0)	13 (7,20)	84 (76,92)
CS	0.05	91 (88,94)	7(5,10)	0 (0,0)	9 (6,12)	84 (81,88)	8 (6,11)	0 (0,0)	7(5,10)	92 (89,94)
CS	0.10	90 (87,93)	9 (6,12)	0 (0,0)	10 (7,13)	82 (78,85)	11 (7,15)	0 (0,0)	8 (5,12)	89 (85,93)
CS	0.25	85 (78,89)	11 (7,16)	0 (0,0)	15 (11,22)	77 (73,81)	16 (11,22)	0 (0,0)	11 (7,15)	84 (78,89)
CS	0.50	78 (69,85)	13 (8,18)	0 (0,0)	22 (15,30)	73 (68,77)	23 (16,32)	0 (0,0)	13 (8,18)	76 (68,84)
CPMM Method Modeled as CS
Generated as		Assigned Class 1			Assigned Class 2			Assigned Class 3
True ∑	True ρ	True C1	True C2	True C3	True C1	True C2	True C3	True C1	True C2	True C3
I = AR(1) = CS	0.00	93 (91,96)	6 (4,9)	0 (0,0)	7 (4,9)	87 (84,90)	6 (4,9)	0 (0,0)	6 (4,9)	94 (91,96)
AR(1)	0.05	93 (90,95)	7(5,10)	0 (0,0)	7 (5,10)	85 (82,88)	7(5,10)	0 (0,0)	7(5,10)	93 (90,95)
AR(1)	0.10	92 (89,95)	8 (5,11)	0 (0,0)	8 (5,11)	83 (79,87)	8 (6,11)	0 (0,0)	8 (5,11)	92 (89,94)
AR(1)	0.25	87 (71,94)	18 (9,44)	0 (0,1)	12 (3,29)	60 (8,79)	11 (2,26)	0 (0,1)	18 (9,42)	88 (72,95)
AR(1)	0.50	57 (4,73)	27 (11,63)	14 (2,30)	32 (1,56)	22 (10,39)	32 (2,57)	15 (2,31)	28 (11,63)	57 (4,72)
CS	0.05	92 (88,94)	8 (5,11)	0 (0,0)	8 (6,12)	84 (80,88)	8 (5,11)	0 (0,0)	8 (5,11)	92 (89,95)
CS	0.10	92 (88,94)	10 (6,14)	0 (0,0)	8 (6,12)	81 (76,85)	10 (6,14)	0 (0,0)	9 (5,13)	90 (86,94)
CS	0.25	87 (81,92)	12 (7,17)	0 (0,0)	13 (8,19)	75 (69,80)	13 (8,18)	0 (0,0)	12 (8,18)	87 (82,92)
CS	0.50	85 (76,91)	12 (7,19)	0 (0,0)	15 (9,24)	73 (66,78)	14 (8,22)	0 (0,0)	13 (7,20)	86 (78,92)

(1) The class output of CPMM and LCTA was aligned to the true classes in the simulation model based on estimated trajectory parameters.

(2) I = Independent; AR(1) = auto-regressive lag 1; CS = compound symmetric; LCTA method always modeled as Independent; C# = Class #.

When the data were generated with no correlation ($\rho =0$), the mixing between Class 1 and 2 and between Class 2 and 3 occurred in roughly the same amounts in both directions for all 3 models. For example, for LCTA, the median number of subjects assigned to Class 1 from Class 2 was 6, to Class 2 from Class 1 was 7; and both to and from Class 2 and Class 3 was 6. The subjects truly in Class 2 assigned to Class 1 were not a random sample of subjects from Class 2; instead they were those whose subject-specific trajectories were in the lower tail of the distribution in Class 2. Similarly, the subjects truly in Class 1 assigned to Class 2 were those in the upper tail of the Class 1 distribution of subject-specific trajectories. Figure 3 demonstrates this phenomenon for the same replicate dataset described in Figure 2. Thus, the intercept bias was negative for Class 1 because subjects assigned to Class 1 in the model fits resulted in an observed distribution of subject-specific trajectories with a lower mean than the true distribution in Class 1. Conversely, the intercept bias was positive for Class 3 because subjects assigned to Class 3 in the model fits resulted in an observed distribution of subject-specific trajectories with a higher mean than the true distribution in Class 3. The intercept bias was near zero for Class 2, not only because the number of subjects truly in Class 1 or 3 assigned to Class 2 were roughly equal, but also because the Class 2 intercept parameter is the average of the Class 1 and 3 intercept parameters based on our simulation design. If either condition did not hold, the bias would have been non-zero.

Figure. 3.

Observed individual-level trajectories of 300 subjects for one simulated dataset under a compound symmetric (rho = 0.50) correlation structure grouped by true class membership for assigned class from the 3-class LCTA model fit. (a) Subjects assigned to Class 1. (b) Subjects assigned Class 2. (c) Subjects assigned to Class 3.

When data were generated with correlation ($\rho >0$), the level of class mixing increased as the correlated strength increased for all 3 models. For example, for LCTA, the median number of subjects assigned to Class 2 from Class 1 increased from 9 to 22 and from 8 to 25 for subjects assigned to Class 2 from Class 3 when the data were generated under a CS correlation structure. The imbalance of the mixing increased as the correlation strength increased when the incorrect correlation structure was used in the model fit. For example, for LCTA, the median number of subjects assigned to Class 1 from Class 2 was 7 and the median assigned to Class 2 from Class 1 was 9 when $\rho =0.05$ and was 14 and 22, respectively, when $\rho =0.50$ when the data were generated under a CS correlation structure. Similar results were observed when LCTA and CPMM-CS were fit to data generated under an AR(1) structure and when CPMM-AR1 was fit to data generated under a CS correlation structure. For CPMM, balance was maintained as the correlation strength increased when the correct correlation structure was used in the model fit. Thus, the class mixing and imbalance findings mirrored the bias findings; as one went up, so did the other.

Increasing the sample size had minimal impact on trajectory parameter estimation and class assignment from a 3-class fit for all 3 models. There was a slight improvement in the mean estimated bias and its standard deviation as the sample increased and in class assignments. Larger improvements were observed when increasing the number of repeated measures per individual. Notably, when 10 repeated measurements were generated per individual instead of 5, all subjects were correctly classified for most of the replicated data sets for all correlation levels for all 3 models. Parameter estimation results are reported in Table S2 and S5 for increasing sample sizes in Table S8 and S10 for increasing numbers of repeated measures. Class assignment results are reported in Table S3 and S6 for increasing sample sizes in Table S9 and S12 for increasing numbers of repeated measures.

5. SUMMARY

Identifying latent classes of longitudinal trajectories is of great interest in biomedical research. Recent findings have been published in fields including addiction,^50,51 internal medicine,^52,53,54 and mental health.^55,56 Two commonly used methods for performing these analyses are LCTA, which assumes within-individual observations are independent over time conditional on class membership, and CPMM, which extends the LCTA methodology to allow for such correlations. Given that within-individual correlation over time is often nonnegligible in biomedical settings, we sought to understand (1) how the magnitude of within-individual correlation and (2) how misspecification of the correlation structure impact class enumeration and parameter estimation under LCTA and CPMM, in the setting of continuous normally distributed outcomes. Unlike GEE and GLMM, where estimates of trajectory parameters are quite robust to misspecification of the correlation structure, we found that the same is not true for LCTA and CPMM. Failure to account for the correlation properly in any of these models also has impact on the standard errors of those trajectory parameters, and thus affects confidence intervals or hypothesis tests used to make inferences on them.

First, we found that even in the presence of weak correlation, LCTA performed poorly in class enumeration, resulting in incorrect estimation of the number of trajectory classes, using AIC, BIC, and ABIC. Second, we found that misspecification of the correct correlation structure in CPMM led to similar results. We note that fitting LCTA to data where within-individual correlation is present is an implicit misspecification of the correlation structure. We only considered AR(1) and compound symmetric (CS) structures, neither of which is nested in the other. While not yet explored, we would anticipate that over-specification of the correlation structure would have less impact on class enumeration in CPMM; for example, specifying a structure in which the true structure is nested, such as specifying a Toeplitz structure when either AR(1) or CS was the underlying truth. Third, we found even when the number of classes was correctly specified, both LCTA and CPMM may yield biased estimates for some class trajectory parameters when the correlation structure was misspecified. We note that different maximization algorithms can impact enumerations indices, leading to potential differences in the number of classes chosen across software. However, misspecification of the correlation structure prevails irrespective of the algorithm employed. Poor model performance in this setting may be anticipated given latent classes can be viewed as missing data and correlation structure misspecification leads to a misspecified likelihood. What was not anticipated was the magnitude of the performance deterioration, even with low or moderate correlation.

When interpreting the class enumeration results of our simulation study, it is important to note the definition of “class” is different for LCTA vs. CPMM. LCTA classes are intended to identify groups of trajectories that look similar, whereas CPMM classes are intended to identify groups of trajectories that come from a single distribution in a mixture of distributions that describe the population of trajectories.¹² As such, LCTA may extract more classes to describe the underlying data structure than CPMM (or GMM). This phenomenon is observed in our simulation study and is exacerbated by misspecification of the correlation structure and magnitude of the within-individual correlation. Thus, if the goal of the research is to determine how many classes are present in the population, LCTA may return an overestimate. However, if the goal is to describe the growth trajectories in the population, the overestimation may still provide a reasonable characterization of the population. Post-hoc analyses are often performed to correlate individual characteristics with the identified classes, and clinical/scientific meaning is often assigned to each identified class. In this setting, using a latent class method that is more likely to return the correct number of classes is beneficial.

What does this mean for analysts applying these methods in practice? Failure to specify the correct correlation structure can result in selecting the wrong number of latent classes. This finding is critical because if the class number estimate is wrong, the estimated trajectories in each class will likely fail to adequately represent the underlying structure of the data and provide meaningful interpretations. Even if the number of classes is correctly identified, the probabilities of class memberships may not adequately distinguish which trajectories are members of which class (low entropy), resulting in more weight to data from the wrong class and thus biasing the trajectory parameter values. Our results demonstrate that careful examination and identification of an appropriate correlation structure, based on prior knowledge and empirical evidence, is necessary to choose whether the independence assumption of LCTA can be met, and if not, to incorporate the correct correlation structure when using CPMM methodology, when possible. One could look at pairwise correlations over time to get a sense of the strength and type of longitudinal correlation. A better approach would be to fit a flexible 1-class longitudinal model with a saturated or full mean model and examine the correlation/covariance structure over time. While not yet explored in this setting, we anticipate that use of an unstructured correlation based on the observed data would often risk consequences from overfitting and/or convergence issues, due to inadequate sample sizes in each class to estimate all parameters, attrition, and/or lack of regularity in timing of data capture. Overfitting the correlation structure can lead to negative bias in the variances and lack of generalizability.

In this work, we purposely kept the simulation design simple to highlight the impact of correlation structure misspecification, separate from variance, on the performance of LCTA and CPMM and how divergent the concept of ‘class’ is between the two methods. We observed with increasing sample size, the performance of these methods improved under some of the simulation scenarios we considered. The differences between classes were relatively large. Were the class trajectories more similar or the error variances larger, the performance of class assignment would deteriorate even when the correct correlation structure was assumed. We compared correlation structures needing only one correlation parameter, did not vary the level of correlation or error variability across classes, and did not consider non-normal outcomes. Repeated measures were generated at fixed, regularly-spaced follow-up times. Irregular follow-up times can easily be incorporated in the GMM framework because they rely only on random effects to model correlation. The same is true for the LCTA framework because of the conditional independence assumption; conditional on class, all observations are considered to be independent, so timing of the measurements is not a factor. It is more challenging to account for irregular follow-up times in the CPMM framework. It can be done, but an exchangeable or spatial correlation structures would likely have to be used. All of these factors may impact the performance of these methods and are interesting avenues of future research. This work fills an important gap in the literature by demonstrating the importance of correlation alone when identifying latent classes of trajectories. Specifically, the importance of using a method that allows one to specify the correct correlation structure when fitting the model.

DATA AVAILABILITY STATEMENT

Code that can be used to replicate these findings can be found here: https://github.com/megan-neely/LCTA_CPMM_Corr_Misspec_Sim.git. Collaboration is welcomed and sharing of additional material related to this study can be agreed upon by contacting the corresponding author, Megan Neely (megan.neely@duke.edu).