Evaluating the Quality of Classification in Mixture Model Simulations

Abstract

The purpose of this study was to evaluate the degree of classification quality in the basic latent class model when covariates are either included or are not included in the model. To accomplish this task, Monte Carlo simulations were conducted in which the results of models with and without a covariate were compared. Based on these simulations, it was determined that models without a covariate better predicted the number of classes. These findings in general supported the use of the popular three-step approach; with its quality of classification determined to be more than 70% under various conditions of covariate effect, sample size, and quality of indicators. In light of these findings, the practical utility of evaluating classification quality is discussed relative to issues that applied researchers need to carefully consider when applying latent class models.

Introduction

Mixture modeling (McLachlan & Peel, 2004) is widely used as an analytic tool within the behavioral, educational, and social sciences (e.g., DiStefano & Kamphaus, 2006; Eid et al., 2003; Klonsky & Olino, 2008; Pinquart & Schindler, 2007; Van Gaalen & Dykstra, 2006; Yang et al., 2005). An important difference between mixture modeling and most conventional analysis tools is that mixture modeling tries to explain and describe potential unobserved population heterogeneity. In contrast, conventional analysis tools (such as simple regression) assume that the studied population is instead comprised of a homogeneous distribution. Mixture modeling can also be considered a more person-oriented approach that emphasizes the patterns of individual characteristics and not a variable-oriented approach that explores the relations between variables (Bergman et al., 2003; Bergman & Magnusson, 1997).

A basic form of mixture modeling, latent class analysis (LCA) is used for cross-sectional categorical indicator variables based on a latent variable that represents a set of latent classes (Collins & Lanza, 2013; Goodman, 1974a, 1974b; Lazarsfeld & Henry, 1968). LCA has an advantage in that researchers can classify individuals and identify their latent classes according to their overall response patterns to the items rather than just by the sum or the mean score of their responses.

Although many studies have only used measured indicators when conducting LCA (J. J. Li & Lee, 2010; Monga et al., 2007; Sullivan et al., 2002), there has been an increase in the number of studies that have looked to include covariates to improve classification performance and goodness-of-fit indices (Carlson et al., 2005; Sysko et al., 2011). Although the use of covariates can improve model fit indices, covariates can potentially change the structure and interpretation of latent classes. Therefore, applied researchers need to decide whether to use a model with or without covariates to most effectively address their research questions, which means clear guidelines for the use of covariates in mixture modeling are required.

To establish these guidelines, several simulation studies have been conducted, but they have led to different conclusions. Some researchers have recommended the inclusion of covariates in the modeling process, in what is referred to as the one-step approach (Lubke & Muthén, 2007; Wurpts & Geiser, 2014). However, some have argued that covariates should be only partially included (Clark & Muthén, 2009), while others have asserted that models containing covariates should be compared with an exclusion model before choosing one or the other (L. Li & Hser, 2011). It has also been suggested that covariates should be excluded completely from the class classification process (Nylund-Gibson & Masyn, 2016).

Models without covariates, which employ a three-step approach (Asparouhov & Muthén, 2014; Kim et al., 2016; Vermunt, 2010), have recently become more widely used. The three-step approach classifies the latent classes first and then estimates the relationship between the latent classes and the covariates. Unlike the one-step approach, which analyzes the effect of covariates and latent classes simultaneously, this approach excludes the effects of covariates when identifying the classes.

Although the three-step approach has been shown to perform well in many simulation studies, the classification quality of this method has rarely been investigated or compared with other approaches. Many previous simulation studies have focused on assessing the estimated coefficients with coverage and the mean square error (Clark & Muthén, 2009) or evaluating models using model fit indices. Some studies that have attempted to evaluate the classification quality have only examined the bias in the probability or proportions of the classes. Thus, previous research on this topic has not taken full advantage of simulation-based analysis in which researchers have access to information on the true classes within the generated data. With this information, the classification quality can be evaluated by quantifying the proportion of subjects correctly classified into their true latent classes rather than merely comparing the estimated class proportions with the true proportions from the data.

The recent study by Cassiday et al. (2021) did attempt to quantify the proportion of subjects that were correctly classified, but it employed a growth mixture model (GMM), which deals with longitudinal data. Therefore, examining the classification quality of LCA models can be helpful for cross-sectional data. In addition, previous studies focusing on LCA with covariates have focused primarily on only two latent classes and have avoided determining the actual number of these classes. In real data, it is rare for only two latent classes to be identified and this has been pointed out as a limitation of previous studies (Lubke & Muthén, 2007).

To fill this research gap, this study aimed to evaluate the classification performance of a basic LCA model with and without the inclusion of covariates using Monte Carlo simulations. To overcome the limitations of previous simulation studies, this study focused on three latent classes, while the model fit indices widely used in previous LCA studies and classification quality were measured to determine the performance of the tested models.

Theoretical Background

Basic LCA Model

LCA seeks patterns that respond to multiple variables. These response patterns can be divided into several latent classes and the attributes of these classes are determined by the characteristics of the response patterns (Collins & Lanza, 2013). Figure 1 presents a simplified version of an LCA model, with the squares representing indicator variables and the circle representing a latent class.

Figure 1.

Basic Latent Class Analysis Model.

The probability that a specific response pattern $\overrightarrow{y}$ will appear is given by Equation (1). Suppose there are J latent class indicators where $R_j$ is the observed response to an item and C is the number of latent classes:

$$P(\overrightarrow{Y}=\overrightarrow{y})=\sum _{c=1}^C{\gamma }_c\underset{j=1}{\overset{J}{\Pi }}\underset{r_j}{\overset{R_j}{\Pi }}{\rho }_{j,r_j|c}^{I(y_j=r_j)},$$ (1)

where ${\gamma }_c$ represents the class proportions and ${\mathrm{\rho }}_{j,r_j|c}$ is the probability of an item response for each class that will answer $r_j$ to the j th item. $I(y_j=r_j)$ is a function that outputs 1 as the response to item $j=r_j$ and 0 otherwise. The probability of a particular response pattern appearing for a specific latent class c is as follows:

$$P(\overrightarrow{Y}=\overrightarrow{y}|L=c)={\mathrm{\gamma }}_c\underset{j=1}{\overset{J}{\Pi }}\underset{r_j}{\overset{R_j}{\Pi }}{\rho }_{j,r_j|c}^{I(y_j=r_j)}.$$ (2)

In Equation (2), because the classes cannot be observed directly, they are estimated using a likelihood function.

LCA determines the probability that a pattern belongs to the c th latent class. This is called the posterior probability and it is used to assign the latent class. It can be expressed as in Equation (3):

$$P(L=c|\overrightarrow{Y}=\overrightarrow{y}).$$ (3)

Using the posterior probability formula $P(A|B)=(P(B|A)P(A))/P(B)$ , the posterior probability is as follows:

$$P(L=c|\overrightarrow{Y}=\overrightarrow{y})=\frac{\left(\underset{j=1}{\overset{J}{\Pi }}\underset{r_j}{\overset{R_j}{\Pi }}{\rho }_{j,r_j|c}^{I(y_j=r_j)}\right){\gamma }_c}{\sum _{c=1}^C{\gamma }_c\underset{j=1}{\overset{J}{\Pi }}\underset{r_j}{\overset{R_j}{\Pi }}{\rho }_{j,r_j|c}^{I(y_j=r_j)}}.$$ (4)

The response pattern is then allocated to the specific latent class with the highest probability. The class-specific indicator probability can be expressed as the estimated values using a logit form. Generally, the class-specific indicator probability is represented by a single threshold value on an inverse logit scale when the latent class indicators have a binary response:

$${\rho }_{j,1|c}=\frac{1}{1+\exp \left({\tau }_{\mathrm{jc}}\right)}.$$ (5)

The class proportions are parameterized as intercepts on the inverse multinomial logit scale with ${\alpha }_{0C}=0$ for identification:

$${\gamma }_c=\frac{\exp \left({\alpha }_{\mathrm{oc}}\right)}{\sum _{s=1}^C\exp \left({\alpha }_{\mathrm{os}}\right)}.$$ (6)

LCA With Covariates

Covariates can affect indicators either indirectly through the latent class variable or directly (Nylund-Gibson & Masyn, 2016). Figures 2 and 3 illustrate the two cases, respectively, with $x_i$ representing the covariate.

Figure 2.

LCA Model With the Indirect Effect of a Covariate.

Note. LCA = latent class analysis.

Figure 3.

LCA Model With the Direct Effect of a Covariate.

Note. LCA = latent class analysis.

For an indirect effect model, the class proportion with covariate x_i is illustrated in Equation (7), where ${\mathrm{\alpha }}_{0C}={\mathrm{\alpha }}_{1C}=0$ for identification:

$${\gamma }_c=\frac{\exp ({\alpha }_{\mathrm{oc}}+{\alpha }_{1c}{x}_i)}{\sum _{s=1}^C\exp ({\alpha }_{\mathrm{os}}+{\alpha }_{1s}{x}_i)}.$$ (7)

This represents the class-specific probability when the direct effect does not differentiate between classes:

$${\rho }_{j,1|c}=\frac{1}{1+\exp ({\tau }_{\mathrm{jc}}-{\beta }_j{x}_i)}.$$ (8)

Based on Equations (7) and (8), the coefficients can be calculated and used in the data-generating process.

Previous Simulation Studies

Several studies have been conducted on the effect of covariates in various models (Table 1). Lubke and Muthén (2007) and Wurpts and Geiser (2014) have suggested that analyses should include covariates, whereas Clark and Muthén (2009) asserted that the inclusion of covariates should be determined according to criteria such as the entropy, coverage, proportions of subjects, convergence rates, and coefficient bias.

Table 1.

Summary of Previous Simulation Studies.

Simulation studies
Authors	Generation model				Analysis model		Criteria	Recommendation
	Model	Number of classes (proportion)	Type of covariate (#)	Type of indicator (#)	Model	Number of classes
Lubke & Muthén (2007)	FMA	2 (0.5:0.5)	Continuous (1)	Continuous (8)	Same as the generation model	Same as the generation model	Coverage, proportion of subjects, entropy, convergence rates	Inclusion
Clark & Muthén (2009)	LCA	2 (0.5:0.5)	Continuous (1)	Binary (10)	Five different regression approaches	Same as the generation model	Mean square error, coverage	Partial inclusion
L. Li & Hser (2011)	GMM	2 (0.5:0.5)	Binary (1) Continuous (1)	Continuous (7)	With or without a covariate	1, 2, 3, 4, 5	Best number of classes by AIC, BIC, ABIC, LMR, ALMR, and BLRT	Need to compare with and without the covariate
Wurpts & Geiser (2014)	LCA	2 (0.67:0.33) 3 (0.4:0.4:0.2)	Continuous (1)	Binary (4~12)	Same as the generation model	Same as the generation model	Class proportion bias, mean conditional response probability bias, covariate effect bias	Inclusion
Nylund-Gibson & Masyn (2016)	LCA	2 (balanced 0.5:0.5, unbalanced 0.8:0.2)	Continuous (1)	Binary (5)	Three different models	2, 3, 4	Best number of classes by BIC, BLRT	Exclusion

Note. # denotes the number of covariates or indicators; FMA = factor mixture model analysis; LCA = latent class analysis; GMM = growth mixture model; AIC = Akaike information criterion; BIC = Bayesian information criterion; BLRT = bootstrap likelihood ratio test. ABIC = adjusted BIC; LMR = Lo-Mendell-Rubin likelihood ratio test; ALMR = adjusted LMR likelihood ratio test.

These studies, however, have ignored the two steps in the process of analyzing actual data in applied research: (a) determining the appropriate model for the data based on theory and accepted practice and (b) identifying the number of classes using model fit indices. This is a concern because it is especially important for simulation-based studies to identify the number of classes using model fit indices in their analysis. Unlike applied research dealing with actual data, simulations have access to information on the generated data and are able to compare different analysis models. The results of these analyses can help applied researchers to make more informed decisions regarding their study model.

Although Li and Hser (2011) and Nylund-Gibson and Masyn (2016) sought to determine the number of latent classes in their analyses, the results did not agree. These two studies also had some important limitations. First, they identified only two latent classes in their analyses, although real-life cases with only two identified latent classes are very rare. In addition, they did not measure the classification quality of their models. Even when the proportion of classes derived from the analysis model is the same as that in the generated data, it is still possible for subjects from a particular class in the generated data to be misclassified in the analysis model. As mentioned earlier, LCA is an exploratory analysis process that uses model fit indices to determine the number of classes. Therefore, the present simulation study replicates this exploratory process and then examines the classification quality.

Simulation Study

Design and Manipulated Conditions

Three studies were designed to test three data-generating models that differed according to the effect of the covariates: Study 1 with no covariate effect, Study 2 with an indirect covariate effect, and Study 3 with a direct covariate effect.

Study 1

The purpose of Study 1 was to compare models with and without a covariate when the generated model did not include a covariate. When the data were generated, the covariate effect was restricted to 0 because a covariate model cannot be analyzed if a covariate is not present.

The analysis models for Study 1 are presented in Figure 4. As the correct model (noneffect covariate generation model), Model 1 does not have a covariate, whereas Model 2 is a misspecified model that includes a covariate and Model 3 is a saturated model that estimates all possible parameters. Although saturated models may fail in the estimation of a complicated model because they estimate all of the possible coefficients, they can produce better estimates for simple models. Thus, Model 3 is also tested because its performance may be better than that of the generation model. It was expected that there would be little difference between Models 1 and 2 in identifying the number of classes and in their classification quality because Model 2 was designed to estimate the effect of the covariate as 0.

Figure 4.

Simulation Design for Study 1.

Note. In the generation model, the dashed line indicates that the coefficient is fixed at 0.

Study 2

Study 2 compared the performance of models with and without a covariate when the generated model included a covariate. In the generation model, the covariate had an indirect effect on the indicators and a direct effect on the latent class variable. Study 2 had three conditions for the effect of the covariate on the latent class variable. This effect is related to Equation (7) and is employed as the odds ratio for the class proportion. An odds ratio of 1.5 has a small effect size, 2.5 has a moderate effect size, and 4.0 has a large effect size (Rosenthal, 1996). When converted, the parameters were 0.41, 0.92, and 1.39, respectively.

The analysis models for Study 2 are presented in Figure 5. Model 1 was a misspecification model with no covariate, whereas Model 2 was the correct model with an indirect covariate effect and Model 3 was a saturated model. If the performance of the model without a covariate is unsatisfactory, there will be many differences in restoring the number of classes and in the classification quality. In contrast, if the difference between the two models is small, selecting the model without a covariate will improve the parsimony.

Figure 5.

Simulation Design for Study 2.

Study 3

The goal of Study 3 was to compare models with and without a covariate when the generated model had a covariate with a direct effect on a single indicator. As shown in Equation (8), the covariate for the indicator affects the threshold value via the odds ratio for the class-specific probability. Study 3 employed the same three conditions as Study 2 for the effect of the covariate using the same standard values.

The four analysis models for Study 3 are shown in Figure 6. Both Models 1 and 2 were misspecified, with the former having no covariate and the latter an indirect covariate. Model 3 was a saturated model that estimates all of the possible parameters. Model 4 was the correctly specified model for a covariate with a direct effect. In Study 3, because both Models 1 and 2 are misspecified, which makes it difficult to predict the results, the better-performing of the two would be selected in applied research.

Figure 6.

Simulation Design for Study 3.

Common Simulation Conditions for All Three Studies

Each of the three studies shared a number of the same simulation settings. There were 10 indicators, each of which was a binary variable with two response categories. There were three latent classes in the generation model present in equal proportions (i.e., 1:1:1). The covariate was a continuous variable with an average of 0 and a variance of 1. The sample sizes were set at 300, 500, and 1,000 with 500 replications.

The quality of an indicator is important because it affects the separation of the latent classes. The higher the indicator quality, the higher the likelihood that the subjects in one class will have the same response patterns. An indicator can be considered high quality when the probability that the members of a group will respond to one outcome of a binary indicator is 0.9 (and 0.1 for the other outcome), while a probability of 0.8 and 0.7 is considered moderate and low quality, respectively (Collins & Wugalter, 1992). This class-specific probability is illustrated by Equation (8), and each threshold can be obtained by calculation. The calculated threshold parameters for the present study were 2.20, 1.39, and 0.85, respectively. The threshold conditions for each class are the same as those presented in Table 2.

Table 2.

Threshold Values for Each Class.

	Quality of indicators
	High			Moderate			Low
Indicator	C#1	C#2	C#3	C#1	C#2	C#3	C#1	C#2	C#3
1	−2.2	−2.2	2.2	−1.39	−1.39	1.39	−0.85	−0.85	0.85
2	−2.2	−2.2	2.2	−1.39	−1.39	1.39	−0.85	−0.85	0.85
3	−2.2	−2.2	2.2	−1.39	−1.39	1.39	−0.85	−0.85	0.85
4	−2.2	−2.2	2.2	−1.39	−1.39	1.39	−0.85	−0.85	0.85
5	−2.2	−2.2	2.2	−1.39	−1.39	1.39	−0.85	−0.85	0.85
6	−2.2	2.2	2.2	−1.39	1.39	1.39	−0.85	0.85	0.85
7	−2.2	2.2	2.2	−1.39	1.39	1.39	−0.85	0.85	0.85
8	−2.2	2.2	2.2	−1.39	1.39	1.39	−0.85	0.85	0.85
9	−2.2	2.2	2.2	−1.39	1.39	1.39	−0.85	0.85	0.85
10	−2.2	2.2	2.2	−1.39	1.39	1.39	−0.85	0.85	0.85

Study 1 investigated a total of nine conditions (3 [sample size] × 3 [quality of the indicators]), whereas both Studies 2 and 3 had 27 conditions each (3 [effect of the covariate] × 3 [sample size] × 3 [quality of the indicators]). Thus, a total of 63 conditions were tested, with 500 replications generated for each condition.

In addition to this, the data were analyzed 3 times for each analysis model with two, three, or four classes. Thus, there were 81 analysis conditions for Study 1 (9 [generated conditions] × 3 [analysis model] × 3 [number of classes for each analysis model]), 243 analysis conditions for Study 2 (27 [generated condition] × 3 [analysis model] × 3 [number of classes for each analysis model]), and 324 analysis conditions for Study 3 (27 [generated condition] × 4 [analysis model] × 3 [number of classes for each analysis model]). Overall, a total of 648 analysis conditions were tested with 500 replications. All models were conducted using the one-step approach.

Model Performance Criteria

Model Fit Indices

Because applied research typically attempts to determine the number of latent classes by considering various model fit indices, this study also followed the same process. The selected model fit indices were the Akaike information criterion (AIC; Akaike, 1974), Bayesian information criterion (BIC; Schwarz, 1978), Lo-Mendell-Rubin adjusted likelihood ratio test (LMRLRT; Lo et al., 2001), and bootstrap likelihood ratio test (BLRT; McLachlan & Peel, 2004). In the present study, the AIC and BIC were compared to determine the number of latent classes, while the statistical significance was measured using the p values for the LMRLRT and BLRT.

For the AIC and BIC, a lower value represents a better model fit. In this study, when analyzing the 500 replications for each condition, the AIC and BIC values for the models with two, three, or four classes were compared, and that with the lowest AIC and BIC value was selected as the number of classes for each replication.

The LMRLRT and BLRT are typically employed to compare a model with k latent classes with a model with k− 1 latent classes. In this case, k is selected as the number of classes if the p value is significant, whereas k− 1 is selected if it is insignificant. In this study, this method was used to compare the models with two, three, or four classes and determine the number of classes for each replication.

Classification Quality

After determining the number of latent classes for each condition, the quality of the classification was assessed. Classification quality is measured as the proportion of subjects within a particular generated latent class that are correctly identified as a member of that class by the model. Although monitoring the classification quality is an important factor in the overall evaluation of a model, many previous studies have not considered this. Only recently has this attracted more attention, including the study by Cassiday et al. (2021), who focused on the classification quality of a GMM. In the present study, classification quality was quantified by recording which latent class each subject was assigned to when generating the data (i.e., their generated class) and which class they were assigned to using the posterior probability from the model analysis (i.e., their predicted class; Table 3).

Table 3.

Example of the Data Recorded to Determine Classification Quality Using the Generated and Predicted Class Assignment of the Subjects.

Condition 1 _ Replication 1
ID	Generated class	Predicted class
1	1	3
2	2	2
3	3	1
4	3	1
⋮	⋮	⋮

To measure classification quality, a 3 × 3 matrix was constructed for each replication based on the generated class and predicted class of each subject (Table 4). Individual subjects assigned to the first class when the data were generated could later be placed in a second class by the model because label switching can occur during the analysis process (Collins & Lanza, 2013).

Table 4.

Example of Successful Classification.

		Predicted class
Condition 1 _ Replication 1		1	2	3
Generated class	1	1	98	1
	2	1	2	97
	3	96	2	2

The classification quality was determined based on whether the subjects in a specific generated class were also assigned to the same class using the model. This is because label switching in Bayesian mixture modeling can occur (Sperrin et al., 2010) if the proportion of labeled classes for the generated model is simply compared with that of the analysis model.

Table 4 presents an example of successful classification. The subjects that were assigned to Class 1 in the generated data were predicted to belong to Class 2 by the estimation model. This also occurred for Class 2 (members were predicted to be in Class 3) and Class 3 (members were predicted to be in Class 1), indicating that label switching had occurred. However, because the members of each generated class mostly remained together after the analysis, this classification was considered high quality.

In contrast, an example of poor classification is presented in Table 5. Around half of the members of generated Class 1 and half of those of generated Class 2 were predicted to be Class 2 during the analysis, which represents incorrect classification. In this case, the reason for determining the classification quality becomes clear. As in many previous simulation studies that have only measured the class proportions, the ratio of the latent classes was similar before and after analysis; thus, the performance was considered satisfactory. However, this is not the case because, in the example, the subjects sharing the attributes of generated Class 1 are predicted to be members of either Class 2 or 3. In other words, subjects from different classes are classified as being part of the same class in the analysis, which represents poor classification quality despite the calculation of good model fit indices.

Table 5.

Example of Unsuccessful Classification.

		Predicted class
Condition 1 _ Replication 2		1	2	3
Generated class	1	1	45	44
	2	1	45	44
	3	96	2	2

This study investigated the accuracy of the classification from 500 replications and the proportion of well-classified subjects according to the total sample size. For example, in Table 4, the following is calculated:

$$97\mathrm{\%}=\frac{(98+97+96)}{300}\times 100.$$ (9)

Although there is no absolute threshold for determining good classification performance, model comparisons are possible based on this classification accuracy.

Programs

In this study, Mplus 7.2 was used to generate and analyze the data (Muthén & Muthén, 2012). The MplusAutomation package in R (Hallquist & Wiley, 2011) was employed to run Mplus repeatedly. MATLAB (MathWorks, 2012) was used to organize the data, compare the model fit indices, and calculate the ratios and proportion of well-classified separation and subjects.

Results

It was found that the overall convergence rate for the 500 replications was more than 96.8% for each condition, that is, more than 96.8% of the replications succeeded in finding a solution. The models fully converged when the analysis included two or three classes; however, there was a low level of nonconvergence when a model included a four-class solution. Specifically, the saturated four-class model solutions did not converge in some cases. When analyzing the two-, three-, and four-class models with one replication, if the four-class model did not converge, the model fit indices of the two- and three-class models were compared to determine the appropriate number of classes.

Study 1

Model Fit Indices

Table 6 shows the percentage of replications that selected the two-, three-, and four-class models as the optimal solution based on the AIC, BIC, LMRLRT, and BLRT. The class number with the highest proportion of replications for the two-, three-, and four-class models is shaded.

Table 6.

Number of Replications Selecting the Optimal Number of Latent Classes in Study 1.

			Analysis models
Generation model			Model 1 (correct)			Model 2 (misspecification)			Model 3 (saturated)
Sample size	Quality of indicators	Model fit index	No. of classes			No. of classes			No. of classes
			2	3	4	2	3	4	2	3	4
300	High	AIC	0	500	0	0	500	0	0	500	0
		BIC	0	500	0	0	500	0	0	500	0
		LMRLRT	0	500	0	1	499	0	0	500	0
		BLRT	0	500	0	0	500	0	0	500	0
	Moderate	AIC	0	390	110	0	393	107	0	495	5
		BIC	0	500	0	0	500	0	0	500	0
		LMRLRT	2	489	9	2	492	6	5	493	2
		BLRT	0	485	15	0	486	14	0	500	0
	Low	AIC	13	354	133	15	343	142	12	317	171
		BIC	491	9	0	497	3	0	497	3	0
		LMRLRT	345	96	59	364	74	62	359	59	82
		BLRT	102	369	29	141	336	23	156	309	35
500	High	AIC	0	500	0	0	500	0	0	500	0
		BIC	0	500	0	0	500	0	0	500	0
		LMRLRT	0	500	0	0	500	0	0	500	0
		BLRT	0	500	0	0	500	0	0	500	0
	Moderate	AIC	0	379	121	0	383	117	0	500	0
		BIC	0	500	0	0	500	0	0	500	0
		LMRLRT	0	488	12	0	492	8	0	500	0
		BLRT	0	488	12	0	492	8	0	500	0
	Low	AIC	0	347	153	0	329	171	1	318	181
		BIC	425	75	0	458	42	0	457	43	0
		LMRLRT	214	271	15	245	240	15	299	172	29
		BLRT	7	474	19	14	459	27	16	455	29
1,000	High	AIC	0	500	0	0	500	0	0	500	0
		BIC	0	500	0	0	500	0	0	500	0
		LMRLRT	0	500	0	0	500	0	0	500	0
		BLRT	0	500	0	0	500	0	0	500	0
	Moderate	AIC	0	402	98	0	397	103	0	500	0
		BIC	0	500	0	0	500	0	0	500	0
		LMRLRT	0	493	7	0	494	6	0	500	0
		BLRT	0	489	11	0	493	7	0	500	0
	Low	AIC	0	330	170	0	309	191	0	293	207
		BIC	68	432	0	101	399	0	111	389	0
		LMRLRT	13	480	7	16	478	6	39	455	6
		BLRT	0	479	21	0	477	23	0	475	25

Note. This table shows the number of replications that selected the two-, three-, and four-class models as the optimal solution based on the AIC, BIC, LMRLRT, and BLRT in Study 1. AIC = Akaike information criterion; BIC = Bayesian information criterion; LMRLRT = Lo-Mendell-Rubin adjusted likelihood ratio test; BLRT = bootstrap likelihood ratio test.

Overall, when the sample size was large, the models provided a good estimate of the latent class number; however, as the sample size decreased, the number of classes became increasingly affected by the quality of the indicators. When the quality of the indicators was high, the classification was perfect for all models. There was little difference between the correct and incorrect models, and they had similar estimation patterns. This was to be expected because the covariate effect in Model 2 was predicted to be 0.

Classification Quality

Based on the 500 replications for each condition using the three latent class models, the number of well-classified replications and the average proportion of well-classified subjects are presented in Table 7.

Table 7.

Classification Quality for 500 Replications in Study 1.

		Analysis models
Generation model		Model 1 (correct)		Model 2 (misspecification)		Model 3 (saturated)
Sample size	Quality of indicators	W-C replications a	W-C subjects b (%)	W-C replications	W-C subjects (%)	W-C replications	W-C subjects (%)
300	High	100.00	98.77	100.00	98.77	100.00	98.71
	Moderate	100.00	90.94	100.00	90.88	100.00	90.54
	Low	69.00	71.20	65.80	71.01	65.60	69.88
500	High	100.00	98.84	100.00	98.84	100.00	98.83
	Moderate	100.00	91.52	100.00	91.49	100.00	91.35
	Low	89.20	72.86	86.20	72.79	85.80	71.80
1,000	High	100.00	98.82	100.00	98.82	100.00	98.82
	Moderate	100.00	91.89	100.00	91.88	100.00	91.83
	Low	98.40	74.41	98.40	74.34	98.00	73.87

Note. This table shows the proportion of well-classified replications and subjects for 500 replications.

^aW-C replications (well-classified replications) are the proportion of replications in which the subjects for each generated class were classified correctly.

^bW-C subjects (well-classified subjects) are the average proportion of well-classified subjects.

When the quality of indicators was high or moderate, the replications were accurately classified with all three models, with more than 90% of the population belonging to the well-classified classes. However, when the quality of the indicators was low, the correct model had only slightly better or similar classification results to the other models. The average proportion of well-classified subjects was about 70%.

Study 2

Model Fit Indices

In Study 2, effect of the covariate was included, so results were 3 times larger than in Table 6 in Study 1. Because three-class models were well estimated when the indicator quality was moderate or high, the results were tabulated only for a low indicator quality. Table 8 presents the proportion of replications that selected the two-, three-, and four-class models as the optimal solution based on the AIC, BIC, LMRLRT, and BLRT for sample sizes of 300, 500, and 1,000. The class number with the largest proportion of replications for the two-, three-, and four-class models is shaded.

Table 8.

Number of Replications Choosing the Optimal Number of Latent Classes in Study 2 (for Indicators With Low Quality Only).

			Analysis models
Generation model			Model 1 (misspecification)			Model 2 (correct)			Model 3 (saturated)
Sample size	Effect of the Covariate	Model fit index	No. of classes			No. of classes			No. of classes
			2	3	4	2	3	4	2	3	4
300	0.41	AIC	17	347	136	8	327	165	13	335	152
		BIC	492	8	0	495	5	0	497	3	0
		LMRLRT	352	99	49	367	73	60	381	48	71
		BLRT	107	357	36	124	352	24	165	312	23
	0.92	AIC	17	352	131	7	341	152	15	313	172
		BIC	492	8	0	490	10	0	497	3	0
		LMRLRT	371	91	38	372	96	32	345	53	102
		BLRT	113	359	28	84	382	34	163	310	27
	1.39	AIC	22	351	127	8	353	139	17	316	167
		BIC	492	8	0	490	10	0	499	1	0
		LMRLRT	372	89	39	386	100	14	334	43	123
		BLRT	123	348	29	94	374	32	155	306	39
500	0.41	AIC	2	350	148	2	325	173	1	320	179
		BIC	428	72	0	454	46	0	462	38	0
		LMRLRT	224	266	10	238	255	7	291	194	15
		BLRT	12	466	22	11	464	25	22	457	21
	0.92	AIC	0	365	135	0	340	160	0	341	159
		BIC	437	63	0	415	85	0	462	38	0
		LMRLRT	239	255	6	206	288	6	283	187	30
		BLRT	13	463	24	6	475	19	24	448	28
	1.39	AIC	1	357	142	0	335	165	0	317	183
		BIC	449	51	0	401	99	0	459	41	0
		LMRLRT	267	226	7	191	306	3	276	198	26
		BLRT	19	464	17	7	470	23	25	450	25
1,000	0.41	AIC	0	340	160	0	321	179	0	317	183
		BIC	75	425	0	99	401	0	129	371	0
		LMRLRT	13	479	8	16	478	6	39	451	10
		BLRT	0	479	21	0	480	20	0	476	24
	0.92	AIC	0	347	153	0	344	156	0	330	170
		BIC	75	425	0	48	452	0	105	395	0
		LMRLRT	15	480	5	8	485	7	35	456	9
		BLRT	0	481	19	0	482	18	0	478	22
	1.39	AIC	0	357	143	0	342	158	0	323	177
		BIC	90	410	0	39	461	0	94	406	0
		LMRLRT	26	464	10	6	488	6	28	460	12
		BLRT	0	473	27	0	476	24	0	474	26

Note. This table shows the number of replications that selected the two-, three-, and four-class models as the optimal solution based on the AIC, BIC, LMRLRT, and BLRT in Study 2. AIC = Akaike information criterion; BIC = Bayesian information criterion; LMRLRT = Lo-Mendell-Rubin adjusted likelihood ratio test; BLRT = bootstrap likelihood ratio test.

When the sample size was large (i.e., 1,000), the models accurately estimated the original number of latent classes using all model fit indices. On the contrary, when the sample size was 300 or 500, the analysis models were influenced by the quality of the indicators. For the indicators with low quality, it had little effect on the effect of a covariate.

The estimation patterns using Models 1 and 2 for the number of classes were similar. Although Model 1 was misspecified, in some cases, it was more accurate in determining the original number of latent classes. Thus, Model 1 produced a similar latent class number estimation performance even though Model 2 was correct.

Classification Quality

Table 9 presents the number of replications for which subjects were accurately assigned to their actual generated class during the analysis and the average proportion of well-classified subjects. For each model, 500 replications were analyzed with three latent classes. The classification was affected by the quality of indicators. If the quality of the indicators was high or moderate, all 500 replications were successfully classified; however, this fell to about 70% for low-quality indicators. In addition, when the indicator quality was low, the proportion of correctly classified subjects was about 70% regardless of sample size. In the classification, the strength of the covariate effect did not influence the results.

Table 9.

Classification Quality for 500 Replications in Study 2.

			Analysis models
Generation model			Model 1 (misspecification)		Model 2 (correct)		Model 3 (saturated)
Sample size	Quality of indicators	Effect of the covariate	W-C replications a	W-C subjects b (%)	W-C replications	W-C subjects (%)	W-C replications	W-C subjects (%)
300	High	0.41	100.00	98.77	100.00	98.76	100.00	98.70
		0.92	100.00	98.79	100.00	98.76	100.00	98.71
		1.39	100.00	98.80	100.00	98.78	100.00	98.72
	Moderate	0.41	100.00	90.96	100.00	90.95	100.00	90.62
		0.92	100.00	90.87	100.00	91.17	100.00	90.85
		1.39	100.00	90.97	100.00	91.70	100.00	91.45
	Low	0.41	69.40	71.34	68.60	71.82	67.20	70.20
		0.92	71.20	71.41	73.20	73.52	67.34	71.97
		1.39	67.40	71.86	75.60	75.57	68.80	73.58
500	High	0.41	100.00	98.85	100.00	98.85	100.00	98.83
		0.92	100.00	98.85	100.00	98.84	100.00	98.83
		1.39	100.00	98.86	100.00	98.86	100.00	98.85
	Moderate	0.41	100.00	91.60	100.00	91.53	100.00	91.41
		0.92	100.00	91.61	100.00	91.77	100.00	91.63
		1.39	100.00	91.67	100.00	92.22	100.00	92.08
	Low	0.41	88.60	72.79	88.40	73.05	87.20	72.13
		0.92	87.00	72.97	89.80	74.81	86.60	73.77
		1.39	85.40	72.97	89.40	76.57	86.40	75.68
1,000	High	0.41	100.00	98.83	100.00	98.83	100.00	98.83
		0.92	100.00	98.85	100.00	98.85	100.00	98.85
		1.39	100.00	98.86	100.00	98.88	100.00	98.88
	Moderate	0.41	100.00	91.89	100.00	91.90	100.00	91.84
		0.92	100.00	91.94	100.00	92.10	100.00	92.04
		1.39	100.00	92.00	100.00	92.52	100.00	92.47
	Low	0.41	97.00	74.47	97.20	74.79	97.80	74.35
		0.92	97.00	74.72	97.40	76.41	97.60	76.04
		1.39	96.00	74.79	98.80	78.05	98.20	77.68

Note. This table shows the percentage of well-classified replications and subjects for 500 replications.

^aW-C replications (well-classified replications) are the proportion of replications for which the subjects of each generated class were classified correctly.

^bW-C subjects (well-classified subjects) are the average proportion of well-classified subjects.

Study 3

Model Fit Indices

As in Study 2, because the three-class models were accurately estimated, the results for Study 3 are presented only for low-quality indicators in Table 10. The proportion of replications that selected the two-, three-, and four-class models as the optimal solution based on the sample size is presented, with the class number for which the model fit was the best is shaded.

Table 10.

Number of Replications Choosing the Optimal Number of Latent Classes in Study 3 (for Indicators With Low Quality Only).

			Analysis models
Generation model			Model 1 (misspecification)			Model 2 (misspecification)			Model 3 (saturated)			Model 4 (correct)
Sample size	Effect of the Covariate	Model fit index	No. of classes			No. of classes			No. of classes			No. of classes
			2	3	4	2	3	4	2	3	4	2	3	4
300	0.41	AIC	12	348	140	10	257	233	9	321	170	12	340	148
		BIC	493	7	0	495	5	0	496	4	0	492	8	0
		LMRLRT	361	81	58	356	71	73	370	54	76	362	84	54
		BLRT	99	376	25	112	333	55	141	327	32	104	375	21
	0.92	AIC	15	343	142	0	136	364	15	316	169	18	338	144
		BIC	494	6	0	465	35	0	497	3	0	493	7	0
		LMRLRT	358	81	61	286	122	92	363	50	87	351	88	61
		BLRT	119	358	23	35	285	180	156	317	27	107	364	29
	1.39	AIC	19	353	128	0	133	367	19	308	173	18	342	140
		BIC	496	4	0	237	259	4	497	3	0	495	5	0
		LMRLRT	354	75	71	219	192	89	366	43	91	355	79	66
		BLRT	133	345	22	8	252	240	173	306	21	123	354	23
500	0.41	AIC	0	366	134	0	196	304	0	312	188	0	361	139
		BIC	435	65	0	448	52	0	453	47	0	430	70	0
		LMRLRT	228	263	9	278	201	21	302	185	13	227	261	12
		BLRT	5	477	18	7	387	106	20	458	22	6	475	19
	0.92	AIC	1	353	146	0	34	466	1	320	179	0	347	153
		BIC	444	56	0	268	228	4	464	36	0	433	67	0
		LMRLRT	241	254	5	209	247	44	303	174	23	233	261	6
		BLRT	11	467	22	2	113	385	16	459	25	7	472	21
	1.39	AIC	1	343	156	0	78	422	2	323	175	0	348	152
		BIC	453	47	0	20	438	42	462	38	0	443	57	0
		LMRLRT	255	240	5	145	278	77	314	158	28	252	240	8
		BLRT	16	462	22	0	118	382	20	453	27	13	465	22
1,000	0.41	AIC	0	332	168	0	66	434	0	326	174	0	335	165
		BIC	71	429	0	107	393	0	113	387	0	71	429	0
		LMRLRT	14	475	11	56	403	41	46	448	6	15	479	6
		BLRT	0	475	25	0	245	255	0	476	24	0	474	26
	0.92	AIC	0	341	159	0	8	492	0	320	180	0	331	169
		BIC	99	401	0	5	361	134	135	365	0	86	414	0
		LMRLRT	23	469	8	103	290	107	47	446	7	20	475	5
		BLRT	0	478	22	0	17	483	0	481	19	0	480	20
	1.39	AIC	0	327	173	0	49	451	0	323	177	0	331	169
		BIC	131	369	0	0	262	238	155	345	0	105	395	0
		LMRLRT	32	459	9	52	263	185	65	430	5	20	471	9
		BLRT	0	481	19	0	49	451	0	481	19	0	479	21

Note. This table shows the number of replications that selected the two-, three-, and four-class models as the optimal solution based on the AIC, BIC, LMRLRT, and BLRT in Study 3. AIC = Akaike information criterion; BIC = Bayesian information criterion; LMRLRT = Lo-Mendell-Rubin adjusted likelihood ratio test; BLRT = bootstrap likelihood ratio test.

In Studies 1 and 2, there was no difference in the patterns between Models 1 and 2. In Study 3, however, their patterns differed because they were both misspecified. The most obvious difference was that Model 2 tended to overestimate the number of latent classes.

In Studies 1 and 2, for a sample size of 300 or 500, if the quality of the indicators was low, the proportion of replications that correctly estimated the original class number was low. In Study 3, however, Model 2 tended to overestimate the class number, even when the quality of the indicators was moderate. This was more apparent when the effect of the covariate was large, that is, 0.92 and 1.39 rather than 0.41. Therefore, the model fit indices indicated that Model 1 was more suitable.

Classification Quality

Based on the 500 replications for each condition using the three latent class models, the number of well-classified replications and the average proportion of well-classified subjects are presented in Table 11. In terms of classification quality, Models 1 and 3 exhibited levels that were similar to Studies 1 and 2, but Model 2 did not. In the two previous studies, when the quality of the indicators was low, the proportion of well-classified subjects was about 70%, whereas in Study 3, this percentage fell to about 60% under certain conditions. This suggests that a model that estimates the effect of a covariate, such as Model 2, may be misleading if it is misspecified.

Table 11.

The Classification Quality of 500 Replications in Study 3.

			Analysis models
Generation model			Model 1 (misspecification)		Model 2 (misspecification)		Model 3 (saturated)		Model 4 (correct)
Sample size	Quality of indicators	Effect of the covariate	W-C replications a	W-C subjects b (%)	W-C replications	W-C subjects (%)	W-C replications	W-C subjects (%)	W-C replications	W-C subjects (%)
300	High	0.41	100.00	98.75	100.00	98.75	100.00	98.68	100.00	98.74
		0.92	100.00	98.65	100.00	98.65	100.00	98.58	100.00	98.63
		1.39	100.00	98.54	100.00	98.52	100.00	98.46	100.00	98.54
	Moderate	0.41	100.00	90.80	100.00	90.69	100.00	90.43	100.00	90.80
		0.92	100.00	90.43	100.00	90.12	100.00	90.18	100.00	90.52
		1.39	100.00	90.02	100.00	89.45	100.00	89.90	100.00	90.24
	Low	0.41	70.00	71.21	67.00	70.31	63.13	69.90	68.60	71.28
		0.92	68.20	70.83	58.20	65.64	67.13	69.62	67.80	71.12
		1.39	64.40	70.46	52.20	60.65	61.60	69.30	64.20	70.85
500	High	0.41	100.00	98.81	100.00	98.81	100.00	98.79	100.00	98.81
		0.92	100.00	98.73	100.00	98.73	100.00	98.70	100.00	98.73
		1.39	100.00	98.63	100.00	98.62	100.00	98.59	100.00	98.62
	Moderate	0.41	100.00	91.42	100.00	91.35	100.00	91.24	100.00	91.42
		0.92	100.00	91.11	100.00	90.94	100.00	90.91	100.00	91.10
		1.39	100.00	90.71	100.00	90.40	100.00	90.61	100.00	90.76
	Low	0.41	89.60	72.63	86.00	71.74	86.40	71.62	89.00	72.68
		0.92	85.00	72.37	73.00	65.68	84.20	71.46	87.60	72.58
		1.39	84.40	71.81	66.40	61.00	82.80	71.08	85.40	72.19
1,000	High	0.41	100.00	98.80	100.00	98.80	100.00	98.80	100.00	98.80
		0.92	100.00	98.72	100.00	98.72	100.00	98.72	100.00	98.73
		1.39	100.00	98.63	100.00	98.63	100.00	98.62	100.00	98.62
	Moderate	0.41	100.00	91.79	100.00	91.77	100.00	91.74	100.00	91.79
		0.92	100.00	91.50	100.00	91.46	100.00	91.45	100.00	91.52
		1.39	100.00	91.17	100.00	91.10	100.00	91.10	100.00	91.17
	Low	0.41	98.40	74.26	98.20	73.67	98.40	73.70	98.40	74.36
		0.92	98.00	73.83	86.00	65.47	97.80	73.42	98.40	74.07
		1.39	98.20	73.34	75.80	61.57	98.20	73.15	98.80	73.76

Note. This table shows the percentage of well-classified replications and subjects for 500 replications.

^aW-C replications (well-classified replications) are the proportion of replications in which the subjects of each generated class were classified correctly.

^bW-C subjects (well-classified subjects) are the average proportion of well-classified subjects.

On the contrary, the classification performance of Model 1 did not differ from that of the correct model. When the correct model does not restore the original class number well, Model 1 performed no better or worse. As can be seen in the tables, when the indicators are low-quality, all of the models generated inaccurate estimations. While larger sample sizes increased the classification performance to more than 70% for Models 1 and 3, Model 2 remained at 60%. Overall, the size of the covariate effect had no clear influence on the estimation results.

Because researchers dealing with real-life data do not have any information on the true model, they are more likely to choose this misspecified model. This misspecified model without a covariate is a simpler model to estimate than the correct model. Thus, for researchers pursuing parsimony and an explanation for the model, Model 1 is the better choice. However, when conditions are poor, it should be remembered that the classification quality for Model 1 was also about 70%.

Conclusion and Discussion

This study investigated the classification accuracy of the basic latent class model in the presence or absence of a covariate. To achieve this, data were generated under various conditions and analyzed using Monte Carlo simulations. For comparison, four model fit indices were used to compare the results of the models and the classification quality was also quantified.

Three individual studies were conducted in this research. First, for a generation model that included a covariate that did not have any effect, estimation models with and without a covariate and a saturated model were compared. In the generation model, the effect of the covariate on the latent classes was set to 0. It was found that the model fit indices and the classification quality of the models with and without a covariate did not differ.

The second study compared the same three analysis models as in Study 1 for a generation model that included a covariate with an indirect effect. The analysis was carried out under a greater variety of conditions, but the obtained results were similar to those from Study 1. It was confirmed that choosing the model without a covariate may be better in terms of the parsimony because the two models produced a similar performance and the model without a covariate is more straightforward.

The third study assumed that both of the two mainly compares models in this study were misspecified. Therefore, the performance of the analysis models, including the saturated model and the correct model (the model with a direct covariate effect), was higher than for the previous two studies. In Study 3, the model without a covariate produced a similar performance to that of the correct model and there were many errors in estimating the number of latent classes due to the effect of the covariate when the model with a covariate was misspecified. Thus, the use of a model without a covariate for class identification is recommended. These results support the conclusions reported by Nylund-Gibson and Masyn (2016).

The model without a covariate generated the best results in terms of parsimony and performance in the three simulated trials. This result supports the use of the three-step approach, which classifies latent classes while controlling for the influence of covariates (Asparouhov & Muthén, 2014; Kim et al., 2016; Vermunt, 2010).

Indicator quality was the most influential factor overall. In particular, when the quality was low, the models performed poorly. This is in accordance with Wurpts and Geiser (2014), who suggested the use of indicators that are of as high a quality as possible. An indicator’s quality is considered low when there are two response categories for a variable and the probability of responding to one side is 0.7 while that to the other side is 0.3. This suggests that researchers should be careful when determining the number of latent classes in actual analysis when indicator quality is this low for binary variables or when the threshold value for the analysis results is −0.85 or 0.85.

The present results provide some novel insights for LCA classification. Classification quality can only be determined using simulation analysis, but many previous studies have only focused on class proportions when analyzing the performance of target models. Even if these proportions appear to be accurate, the results may be unreliable if the classification quality is poor. The present study confirmed that classification quality needs to be considered when judging the performance of LCA models and to ensure reliable results.

The proportion of subjects that were correctly classified in the present study averaged more than 70%, although it varied depending on the model specifications. When the sample size, indicator quality, and effect of the covariate were high, the proportion of well-classified subjects regularly reached 100%. However, under poor conditions, future research is required to develop strategies to improve classification quality.

One of the strengths of this study is that it examined the classification quality of a basic LCA model, which is an analytic tool that is widely used in behavioral science. In addition, this study demonstrated that analysis models performed better when covariates were included under various conditions. These results can be a guideline for researchers employing LCA models for the analysis of cross-sectional data. Similar to Cassiday et al. (2021), who investigated the classification quality of a GMM for longitudinal data, the classification of other mixture models should be studied in the future.

Despite these contributions, this study has a number of limitations. First, the research conditions in the present study were relatively narrow. Only a single covariate was included, and only binary variables were used as indicators. Adding multiple covariates and multiple-response variables to the research conditions would add more clarity to this topic in the future. In addition, this study dealt only with balanced data; with unbalanced data in which the class proportions are not equal, the analysis may be more complicated if label switching occurs. It is also important to note that parameter estimates were not analyzed or compared because investigating the bias of the parameters was not the objective of this study.

Nevertheless, this study is meaningful in that it presents guidelines for simulation researchers because classification quality can act as a reference point for comparing models. In addition, applied researchers who conduct LCA using models with a covariate can use the results of this study as a reference for the analysis process.