Criterion validity, construct validity, and factor analysis: An introductory overview

Abstract

This is the fourth and last article in the series on translation, adaptation, or development of a rating scale and its psychometric testing. The focus of this article is on criterion validity, construct validity, and factor analysis. Validity refers to whether the tool measures “what it purports to measure.” Content validity, criterion validity, and construct validity are the different types of validity. Content validity was previously discussed in this series. Criterion validity assesses how a new scale correlates with a criterion or “gold standard.” Depending on the time of administration of the “gold standard,” this can be classified as concurrent or predictive validity. Pearson’s correlation coefficient is the measure used to establish criterion validity for continuous variables, while phi coefficient is used for dichotomous ones. Construct validity assesses whether the new tool performs consistently with the theoretical concepts. This can be of two types: convergent and divergent validity, and is estimated using Pearson’s correlation coefficient. Factor analysis (FA) is a multivariate technique that evaluates whether several variables are linearly related to a set of factors. It is also a method to assess construct validity. There are two methods of FA: exploratory and confirmatory FA; the steps of exploratory FA are discussed in detail here.

Earlier articles in this series have covered the principles of translating and adapting a rating scale,[1] reliability testing,[2] item generation,[3] and face and content validity checks of a newly developed rating scale.[3] The validity of a measurement tool refers to whether the tool “measures what it purports to measure.”[4] Conventionally, according to the “trinitarian doctrine,” validity is divided into the “three Cs” – content, criterion, and construct validity.[5] This article covers the concepts and practical aspects of criterion validity and construct validity, including the nuances of factor analysis (FA).

CRITERION VALIDITY

Traditionally, criterion validity is defined as the correlation of a scale with another measure of the phenomenon under study, preferably an accepted “gold standard.”[5] To establish criterion validity, the scores on the new scale are correlated with those of another measure that has already been demonstrated to be valid and reliable for the same construct—a behavior, disorder, or any other outcome of interest. In other words, the new instrument is tested against a criterion—that is, a benchmark or gold standard.[6] Depending on whether the criterion or the gold standard was administered at the same time or sometime after the administration of the tool for which validity is to be estimated, criterion validity can be classified as concurrent or predictive validity, respectively. The distinction between these two types of validity can also be based on the objectives of testing.[6]

Concurrent validity

Concurrent validity is assessed for tools that diagnose the existing clinical condition. For this, the new tool and the criterion (gold standard) measure are administered simultaneously or within a short span of time, and it is observed whether the results are consistent with each other.[5,6,7] For example, a new diagnostic interview for depression is administered simultaneously with the Structured Clinical Interview for DSM-5 (SCID-5)[8] to estimate its concurrent validity. Similarly, a new tool to assess quality of life can be administered along with a standard scale, such as the WHO Quality of Life (WHOQOL) scale,[9] to test the concurrent validity of the new scale.

Predictive validity

This is used for tools that are intended to predict future outcomes. Predictive validity assesses whether the tool accurately predicts a behavior, based on a criterion that is administered in the future, ranging from a few days to even a few years later.[5,6,7] For instance, an aptitude test is said to have good predictive validity if it accurately predicts a candidate’s chances of passing or failing an examination.[5] Likewise, if a social support scale for exclusive breastfeeding can predict the maternal behavior of exclusive breastfeeding after receiving social support, it is said to have good predictive validity.[10]

Measures of criterion validity

Depending upon whether the variables are dichotomous or continuous, the indices of criterion validity differ. For continuous variables, a bivariate correlation, such as the Pearson correlation coefficient, will be applicable [Table 1].[5] If we have two dichotomous measures, this will result in a 2 × 2 table [Table 2], and measures such as sensitivity, specificity, or a phi coefficient (φ) will be appropriate. In the scenario where a test tool scored on a continuous scale (e.g., an instrument to measure symptom severity of depression) is validated against a dichotomous “criterion” outcome, such as the diagnosis of depression (yes/no), the sensitivity and specificity values will be calculated for different scores of the test instrument. The score with the optimum sensitivity and specificity will be taken as the cut-off to make a diagnosis of depression. This is the basis of receiver operating characteristic (ROC) curves, and the area under the curve (AUC) in the ROC curve is a measure of validity.[5] A detailed discussion on the ROC curve is beyond the scope of this paper, and readers are referred elsewhere.[11]

Table 1.

Summary of the procedures to establish criterion and construct validity

Type of validity	Purpose	Measurement	Statistical method
Criterion validity
Concurrent validity	To determine the relationship between the index instrument and a criterion administered simultaneously	Correlation or association between the scores of the scale and the “gold standard”	If continuous variables: Pearson’s correlation coefficient If dichotomous variables: Sensitivity and specificity Phi coefficient (ϕ) ROC curve and AUC
Predictive validity	To examine whether scores of the scale predict future outcomes	Correlation and association between the scores of the scale and the criterion scores administered sometime later	If continuous variables: Pearson’s correlation coefficient If dichotomous variables: Sensitivity and specificity Phi coefficient (ϕ) ROC curve and AUC
Construct validity
Convergent validity	To examine whether the same construct measured using different scales (used to assess the same or related constructs) yields similar results	Correlation between the scores of the scale administered by two different methods (A high correlation indicates higher convergent validity)	Pearson’s correlation coefficient Multi-trait multi-method matrix
Discriminant validity	To examine whether the construct measured using a tool is different from another unrelated concept measured using a different tool	Correlation between the scale scores and the measure of another unrelated construct measured by two different methods (A lower correlation reflects higher discriminant validity)	Pearson’s correlation coefficient Multi-trait multi-method matrix
Factorial Validity	Data reduction Construct validation	To determine the optimum number of factors or domains that fit a set of items	EFA: by Principal Component Analysis or Principal Axis Factoring CFA: by Maximum Likelihood or model evaluation using fit indices

AUC – area under the curve, CFA – Confirmatory factor Analysis, EFA – Exploratory Factor Analysis, ROC - receiver operating characteristic

Table 2.

2 × 2 table to assess criterion validity for scale with dichotomous outcome

	Gold standard
	Diagnosis +	Diagnosis −
New tool
Diagnosis +	A	B
Diagnosis −	C	D

The major challenge in assessing criterion validity is the difficulty in obtaining a satisfactory criterion or gold standard measure. If the external criterion itself is invalid, and a weak correlation is obtained with the test measure, it would not be possible to conclude that the test measure is flawed. Construct validation becomes more critical in this scenario.[7]

CONSTRUCT VALIDITY

It refers to the extent to which an instrument measures the concept or theoretical construct that it intends to measure.[12] Cronbach and Meehl,[13] who introduced this concept, proposed that the construct validity of a tool should be estimated in the absence of a definite, accepted criterion that is adequate to define the construct being measured. It assesses whether the performance of the new tool is consistent with the predictions made based on a theory.[7]

There are two components of construct validity: convergent and discriminant validity.

Convergent validity

It assesses the correlation of the new scale with other measures of the same or related constructs. For instance, the correlation of a new scale intended to measure anxiety is estimated against the scores of a tool measuring autonomic awareness.[5]

Discriminant or divergent validity

This assesses the extent to that a new scale is related, rather not related, to a dissimilar or an unrelated construct.[5] For instance, a new scale to measure perceived stress is administered along with a standard tool, such as the WHOQOL scale,[9] which measures quality of life and is expected to decrease with increasing stress; the correlation is tested for divergent validity.

Measures of construct validity

The level of correlation between the test tool and another related tool is assessed using Pearson’s correlation coefficient. The value of the coefficient can vary from −1 to +1, representing a perfect negative or positive correlation, respectively, with 0 indicating no correlation.[14] See Table 1 for details of the methods to assess construct validity.

A technique that evaluates convergent and discriminant validity simultaneously is the multitrait-multimethod matrix (MTMM) analysis, described by Campbell and Fiske in 1959.[15] In this, two or more unrelated traits are measured simultaneously using two or more methods. As shown in Table 3, if two different traits (A, B) are measured using two different methods (C, D), the matrix would consist of ten correlations (2 traits × 2 methods) related to validity. Each cell in the matrix represents the correlation between two measures. It could be that of the same trait and different methods (homotrait-heteromethod), different traits using the same method (heterotrait-homomethod), or different traits using different methods (heterotrait-heteromethod). Homotrait-homomethod correlations reflect reliability. They are represented by the cells along the diagonal. High homotrait-heteromethod correlations reflect convergent validity, whereas low heterotrait-heteromethod correlations indicate discriminant validity. Moreover, if heterotrait-homomethod correlations are as high as or higher than heterotrait-heteromethod ones, it suggests that the method of measurement was more significant than the construct being measured. This is not desirable. By comparing these correlations, it can be assessed whether the methods used to measure different traits can capture the construct distinctly.[5,6] Advanced statistical methods, such as structural equation modelling, can be applied to MTMM designs for a more accurate evaluation of validity.[16] Another method to evaluate construct validity is FA.[17]

Table 3.

Demonstration of Multi-trait- Multimethod matrix (MTMM) analysis

	Trait A		Trait B
	Method C	Method D	Method C	Method D
Trait A
Method C	(ACxAC)*
Method D	ADxAC†	(ADxAD)*
Trait B
Method C	BCxAC⁑	BCxAD‡	(BCxBC)*
Method D	BDxAC‡	BDxAD⁑	BDxBC†	(BDxBD)*

*Homotrait-homomethod correlations. ^†homotrait-heteromethod correlations, ^⁑heterotrait-homomethod correlation, ^‡heterotrait-heteromethod correlations

FACTOR ANALYSIS

Developed in the early 20^th century by Spearman, FA has been widely used in the validation of measurement tools. FA refers to a variety of multivariate statistical techniques used to assess whether several variables are linearly related to a smaller number of factors. It helps in data reduction by regrouping variables that approximately measure the same concept into a limited number of clusters. In the development and validation of a questionnaire, FA is necessary as it provides evidence for the construct validity of the tool.[18,19] The correlation coefficient determines the relationship between two variables and is the basic statistic used in FA. Generally, a correlation matrix is used to perform FA.

There are two methods of FA: exploratory and confirmatory. Exploratory factor analysis (EFA) helps to generate a new theory or hypothesis when the relationship between variables is unknown. Confirmatory factor analysis (CFA) is used to test an existing theory by testing a hypothesis when there is some idea about that items belong to each factor.[5] In the initial phases of tool development, EFA is used to identify the factors or dimensions. Once previous EFA studies are available for the tool, CFA is done to confirm or reject the factor structure extracted in previous research. Thus, CFA may be undertaken for a different language version of the tool or in a different cultural setting.[18] In this article, we are focusing on EFA.

Exploratory factor analysis

EFA is used in the early stages of instrument development. It uses a data correlation matrix to cluster similar variables into the same factor. The main steps involved are assessing the suitability of the data for FA, extracting the factors, factor rotation, and interpretation.[19]

1. Assessment of the suitability of data for FA

To assess the suitability of data for FA, the adequacy of the sample size and the strength of the relationship between the variables must be evaluated. As a rule of thumb, the respondents-to-variable ratio should be at least 10:1 (preferably more); never less than 5:1. The sample size recommended for FA varies widely. It is generally suggested that a sample size of 300 provides much more reliable results. A sample size of less than 150 may be sufficient, provided the factor loading scores—a measure of how much each variable contributes to the factor – are high (>0.8).[17,18] The Kaiser-Mayer-Olkin (KMO) test is a measure of sampling adequacy, and it indicates whether the variables are correlated adequately for FA. It is an indicator of common variance in a data set, suggesting that latent factors may be present. KMO values can vary from 0 to 1. A higher KMO value suggests that the variables correlate well and are suitable for FA. If the value is 0.90–1.00, it is marvelous, 0.80–0.90 is meritorious, 0.70–0.80 is middling, 0.60–0.70 is mediocre, 0.50–0.60 is inadequate/miserable, and <0.50 is not acceptable for FA.[19,20]

Bartlett’s test of sphericity assesses the strength of the relationship between the variables. It tests the null hypothesis that the variables in the correlation matrix are orthogonal or unrelated. A P < 0.05 rejects the null hypothesis, suggesting that the variables are correlated enough and that the data is factorizable.[19] The correlation between variables in the matrix must be ≥0.3, as any value below that would suggest a poor relationship.[21]

2. Extraction of the factors

Factor extraction is used to identify a parsimonious model with the least number of factors that can best represent the interrelationship between the variables studied.[19] Each factor is a weighted combination of the variables.[5] Various methods are used for factor extraction. These include Principal Component Analysis (PCA), Principal Axis Factoring, Maximum Likelihood, or Parallel Analysis. These procedures analyze the interrelationships between the instrument’s items (variables) and domains to reveal the unknown or latent factorial structure (dimensions) of the construct being studied.[22] PCA is used for factor reduction, while the other methods are used to identify the latent construct and establish construct validity.[18] In this article, we will focus on PCA, the most popular method of factor extraction.

Principal component analysis

It reduces many variables or items to a smaller number of components or factors by extracting the maximum variance from the data for each element. It is a technique used for data reduction. A principal component or factor refers to a linear combination of the observed variables that is independent of other components.[21,22] PCA provides a factor matrix, which is a correlation matrix that shows the correlation between each item and each factor. There are as many factors as there are items or variables in this correlation matrix, from which the significant ones are extracted.[5]

Factor loading refers to the correlation of a variable or item with a factor; the higher the factor loading, the better the fit of the factor with the data. All items load on all factors, but the loading score varies with each factor and can vary from −1 to +1. A factor loading of >0.3 indicates a moderate correlation between that item and the factor. The square of the factor loading of an item refers to the percentage of variance of that item explained by that factor. The mean value of the sum of the squares of the factor loadings of all the items under a factor gives the proportion of variance explained by that factor.[18]

For each item, the sum of the squares of its loadings on the factor matrix is referred to as its communality. This indicates the proportion of variance for that item that is explained by the various factors extracted. The higher the communality, the greater the variance of the item that is explained by the extracted factors; in other words, this indicates a better fit of the factors to the data.[18]

Number of factors to be extracted:

Two methods can be used to determine the number of factors that can be extracted or retained. They are the Kaiser’s criterion and the Scree test.

Kaiser’s (Eigenvalue) criterion: The eigenvalue of a factor is given by the sum of the squared item loadings of that factor; it refers to the total variance in the data explained by that factor.[14,19] According to Kaiser’s criterion, all factors with an eigenvalue greater than one can be retained. This can lead to over-extraction of factors.[19,21,22]

The scree test is a graphical method used to determine the number of factors to be extracted. A scree plot is a graph in which eigenvalues are plotted along the y-axis and the factors along the x-axis (see Supplementary FIles). Factor extraction is stopped at the point where there is an ‘elbow’ or levelling/tailing off the plot.[19,21,22]

The first factor or principal component accounts for the maximum variance in the data and has the highest eigenvalue. The second factor explains the maximum amount of remaining variance in the data, and is uncorrelated to the first factor, and so on. The eigenvalues decrease progressively.[5,21,22] The total variance in the data explained by all the factors retained is called the cumulative variance. A cumulative variance of 75%–90% is desirable, but even 50% is considered acceptable.[22]

3. Factor rotation and interpretation

It is often difficult to interpret the factors that are retained after initial extraction. Typically, most items load on the first factor or exhibit considerable cross-loading, that is, high loading (>0.3 or 0.4) on two or more factors. As these factors can affect the interpretation, factor rotation is undertaken. Factor rotation involves rotating the geometric axes of the factors to modify the factor loadings; it does not change the overall variance explained by the extracted factors. After the initial extraction of the factors using Kaiser’s criterion or scree plot (as explained in the previous section), all the other factors are discarded. All the items are factored again, forcing them to load onto the specified retained factors. This solution is then rotated. This process is called factor rotation.[22] It maximizes the variance of the loadings of the items under each factor. That is, it increases the chances of an item having high loadings on one factor and low loadings on other factors; this simplifies the interpretation of the factors.[23]

There are two methods of factor rotation: orthogonal and oblique. Orthogonal rotation is used when the hypothesis involves uncorrelated factors, while oblique rotation is used when factor correlations are allowed.[16,18] In a study assessing student performance, if two unrelated variables like the skills in math and language are studied, orthogonal rotation can be used for factor rotation. Oblique rotation can be tried if the variables studied correlate well, like math and analytical thinking. The former involves various types of rotation: varimax, quartimax, and equamax, of which the varimax method is generally preferred. In the varimax method, the variance of the loadings is maximized; each variable will have a high loading on one factor and small loadings on other factors.[23] Oblique rotation includes direct oblimin, ProMax, orthoblique, and Procrustes; the choice is based on the software used.[19,22] Orthogonal rotation is the most commonly used method, probably due to its ease of interpretation.

In the rotated component matrix, the spread of factor loadings is maximized – those that are high become higher, and those that are low become lower. Some items may not be “clearly loaded” on any factor; that is, they may not have high loading on any factor, and they may be removed. Variables or items that represent a factor are retained under it. If they represent multiple factors or no factors, they may be removed. A satisfactory variable will have a “good” factor loading of at least 0.4 with one factor, and show poor loading (<0.3) with other factors. If there is high cross-loading in this matrix, the difference between the highest and second-highest factor loading is determined. If this loading difference is sufficiently large (≥0.2), the item may be retained under the highest loading factor. If the discrepancy is less than that, the cross-loaded items may be deleted, and FA may be repeated.[19,20,24] Before removing an item from the dataset, the conceptual significance of the item should be considered. The conceptual significance is more relevant than the statistical one. Once the poorly loaded (<0.3) or cross-loaded (loading difference >0.2) items are removed, FA is repeated to obtain a better solution.[22]

For accepting a factor, it should have an eigenvalue >1, and each item under it should have a loading of at least 0.4. A factor should have at least three variables or items to be labeled as stable and solid, but this depends on the study design. In the rotated component matrix, if ≤2 variables load on a factor, it can be considered reliable only if the variables are highly correlated with each other (correlation coefficient, r > 0.7) and poorly correlated with the other variables.[21,22] As there is some subjectivity involved in determining the number of factors and their interpretation, it has to be ensured that the factors and the items constituting them are theoretically robust and conceptually meaningful to represent a given construct or its dimensions; if not, such factors may be dropped or not interpreted.

The steps for FA in IBM SPSS Statistics are given in Box 1. See Supplementary File 1 ^{(1.6MB, tif)} for sequential screenshots explaining how to perform FA using IBM SPSS Statistics, and Supplementary File 2 (a-d) for an illustrative example of the key steps of FA using the data of a scale.[25]

Box 1.

Steps to do Factor Analysis in IBM SPSS Statistics

Factor analysis in SPSS Once the data is opened in SPSS, click Analyze → Dimension Reduction → Factor… Then select and move all the variables or items to the box “Variables” Select Descriptives to open that window and click Univariate descriptives and Initial solution under Statistics, and Coefficients, Significance levels and KMO and Bartlett’s test of sphericity under Correlation Matrix. Then click Continue. Select Extraction to open that window. Ensure that “Principal components” is shown in the Methods section. Click Correlation matrix under the Analyze section, and check Unrotated factor solution and Scree plot under Display. In the Extract section, click Based on Eigen value, see whether Eigen values greater than value is “1” and click Continue. Select Rotation, click Varimax under the Method section, and Rotated solution under Display. Then click OK to see the results in the Output.

FA is an iterative process, where the analysis is repeated, testing different factors, and finally accepting the factor structure that provides the maximum cumulative variance. Solutions are repeatedly refined and compared till the most meaningful solution is reached.[22] It helps to interpret the theoretical connections between various dimensions of the construct being measured. Using this iterative process, the factor structure of the construct can be tested and refined, keeping the theoretical perspectives in mind.

This is the final article in the series on scale development, validation, and psychometric testing. The series sequentially described the process of development and validation of the scale, establishing reliability, and three types of validity—content, criterion, and construct. It could be argued that the distinctions between the validity categories are arbitrary. In 1989, Messick construed all these different types of validity as a part of construct validity.[7] Therefore, the process of scale development and validation should be examined under one validity lens, specifically construct validity.[26] Accordingly, the entire process of establishing construct validity, along with the corresponding procedures, is described in this series of articles. We hope that this series of articles will help researchers in the development and validation of mental health measurement tools that adequately represent their study variables, and spur research that is theoretically sound, culturally appropriate, and scientifically valid.

Statement on generative artificial intelligence technology

The authors attest that there was no use of the generative AI technology in the generation of text, figures, or other informational content of this manuscript.

Conflicts of interest

There are no conflicts of interest.

SUPPLEMENTARY FILES

Supplementary File 1

Steps of Factor Analysis in IBM SPSS Statistics

Supplementary File 2.

Steps of Factor Analysis prepared based on the data of the Everyday Abilities Scale for India[25]

a). Assessment of the suitability of the data KMO and Bartlett’s Test
Kaiser-Meyer-Olkin Measure of Sampling Adequacy									0.715
Bartlett’s Test of Sphericity
Approx. Chi-square									263.760
df									66
Sig.									0.000
b). Extraction of factors – Principal Component Analysis
Component	Initial Eigen values			Extraction Sums of Squared Loadings			Rotation Sums of Squared Loadings
	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %
1	6.807	56.723	56.723	6.807	56.723	56.723	5.913	49.277	49.277
2	1.612	13.437	70.159	1.612	13.437	70.159	2.506	20.883	70.159
3	0.917	7.639	77.799
4	0.858	7.154	84.953
5	0.493	4.110	89.063
6	0.405	3.378	92.440
7	0.344	2.867	95.307
8	0.212	1.763	97.070
9	0.181	1.507	98.577
10	0.112	0.933	99.510
11	0.036	0.300	99.810
12	0.023	0.190	100.000

Extraction Method: Principal Component Analysis.

c). Scree plot

d). Factor rotation

Funding Statement

Nil.