A perspective on using partial least squares structural equation modelling in data articles

Abstract

This perspective article on using partial least squares structural equation modelling (PLS-SEM) is intended as a guide for authors who wish to publish datasets that can be analysed with this method as stand-alone data articles. Stand-alone data articles are different from supporting data articles in that they are not linked to a full research article published in another journal. Nevertheless, authors of stand-alone data articles will be required to clearly demonstrate and justify the usefulness of their dataset. This perspective article offers actionable recommendations regarding the conceptualisation phase, the types of data suitable for PLS-SEM and quality criteria to report, which are generally applicable to studies using PLS-SEM. We also present adjusted versions of the HTMT metric for discriminant validity testing that broaden its applicability. Further, we highlight the benefit of linking data articles to already published research papers that employ the PLS-SEM method.

1. Background

This perspective article on using partial least squares (PLS) for estimating relationships among observed and latent variables [1,2] – also referred to as PLS path modelling (e.g., [3,4]), the PLS approach to structural equation modelling [e.g., [5]], or PLS structural equation modelling (PLS-SEM; e.g., [6,7]) – in data articles has at least two objectives. First, it is intended as a guide for authors who wish to publish datasets that can be analysed with PLS-SEM as a stand-alone data article. Stand-alone data articles are not linked to an original research article published in another journal. These datasets are frequently generated through some form of replication of previous studies and thus may not meet the criterion of novel theoretical contribution as defined by many traditional journals in the broader field of business and management. However, it is important to emphasise that empirical contributions are no less important for the advancement of science than theoretical contributions [8]. Therefore, a stand-alone data article can be seen as valuable as long as it makes a meaningful empirical contribution. Köhler and Cortina [8] offer a typology of the various forms of replication and examine the contribution of each type. They distinguish between literal replication, quasi random replication, constructive replication, confounded replication, and regressive replication. The degree of the empirical contribution varies across the different forms. Specifically, the value of confounded replications and regressive replications are unclear as they contain shortcomings vis-à-vis the original article [8].

Second, this perspective article aims to highlight the benefits of publishing data articles alongside original research articles in other journals. Linked data articles can be regarded as an important enabler on the road towards ‘gold standard reproducibility’ [9]. Research articles that only describe the data but do not make the data publicly available are only partially reproducible and may not be fully replicable. This is partly due to the limited space available in research articles that can be dedicated to the methods section. Some journals have strict word count limits. Although it is possible to host supplementary files with research articles, the reader may not be able to easily understand them without additional explanation. Therefore, reproducibility requires publication of the research article that provides the theoretical underpinning and interpretation of the results, a detailed description of the method and research design, as well as the publication of the data set [9].

A linked data article allows authors to focus more on the theoretical framing and interpretation of results in the main research article and to offer a more detailed description of all the decisions that have been made in the research design and data analysis aspect of the process in the data article. Some decisions that hold relevance for future replication studies may not be reported in the original research article because of space constraints. Further, in linked data articles, authors can provide details of all the items in their dataset beyond the items analysed in the main research article.

Both types of data articles support the advent of open science. Open science is fundamentally changing the landscape in which research, education and innovation are undertaken, archived, curated and disseminated around the world [10]. Open science is about ensuring research findings are accessible to all rather than keeping them behind a paywall. It encompasses four principles or pillars of openness, namely, open data, open code, open papers and open reviews [9,11]. The case for open science has become a particularly strong one as the world grapples with global challenges of resource scarcity, climate change, poverty, ill-health, pollution, rapid urbanisation and food insecurity [12]. However, although open science practices are increasingly permeating the natural and medical sciences, other fields, including the social sciences, business and management, and economics, are lagging behind in adopting these practices [13], [14], [15]. As studies in business and management increasingly use the PLS-SEM method (e.g., [16]), this perspective article also aims to encourage researchers to keep abreast of reporting standards and data curating practices in other disciplinary areas. We therefore revisit PLS-SEM's key characteristics and reporting standards, pointing to contradicting streams in the methodological literature that authors need to be aware of. In our discussion of model evaluation metrics, we also offer adjusted versions of the HTMT metric for discriminant validity testing that broaden its applicability.

2. Recommendations

2.1. The conceptualisation phase: ensuring the model is meaningful and highlights interesting relationships

A key characteristic of appropriately anchoring any quantitative research paper is the alignment of concepts with the operational part of a study [17]. While data articles do not feature extensive theoretical or conceptual sections, it is nevertheless critically important to demonstrate the underpinning conceptual or theoretical ‘story’ explored in the data and its origins. This step includes a justification why specific concepts were selected and how they are related. The model specification should be described or represented by a graphical model illustration. In addition to journal articles, construct definitions and their measurements (i.e., sets of items or ‘scales’) can also be found in scale handbooks. Scale handbooks are available for sub-fields and functional areas of business and management, such as marketing [18,19] entrepreneurship [20], work organisation [21], and organisational behaviour or human resource management [22]. In addition, journal articles routinely document item wordings in the main text or an appendix. In reviewing existing scales, researchers should closely consider their psychometric properties with regard to prior reliability and validity assessments [23]. Regarding the underpinning of the measurement scales’ quality, it is usually a signal of good quality if the constructs on which PLS path models are built originate from high level and reputed journal publications rather than outlets that are not widely known. Journal lists [24], [25], [26] offer guidance regarding the reputation and quality levels of published outputs, which improves the ‘face validity’ of the scales and items reported.

Advice to authors: Describe the concepts and their relationships and report the measurement scales, including underpinning literature. If the scales have been adapted, report the changes made to the scales (highlight the changes to the wording).

If the authors’ path model has already been published, they will still need to demonstrate a high standard of care with the reporting and referencing of source materials. In addition, authors should strengthen the trustworthiness and cogency of the data that they seek to make publicly available. Authors will also need to explain how the data article adds value to the information provided in the original research article. This is important because a linked data article should not be a simple replication of the methods section of the original article.

In stand-alone submissions to a data journal, where conceptual and theoretical reference work that develops the reported PLS path model does not yet exist, authors need to build their case carefully to demonstrate their model's usefulness. It is not sufficient to show that the data used to deploy the PLS-SEM method satisfies the common model evaluation criteria as documented in, for example, Hair et al. (2022). Careful, transparent, and practically engaging writing is required to provide a convincing chain of arguments demonstrating why the model offers meaning. Since space is limited, this may involve references to a conceptual debate elsewhere that could be enriched through the dataset. A multi-stage pattern-matching process can be a useful way to ascertain that the conceptual model is meaningful and relevant in a specific context [27].

Our advice to authors: Demonstrate that the model is meaningful and relevant.

2.2. Model estimation: Why PLS-SEM?

As much as the conceptual or theoretical level and measurement level cannot be divorced [28], the write-up of a conceptualisation stage in a data article and the model estimation stage are intertwined and cannot be separated [29]. Therefore, a clear statement of the objectives of the analysis is required in addition to an explanation of how the PLS-SEM results support the accomplishment of the stated objectives. With respect to the associated empirical aspects, this relates to, for example, high model complexity, the estimation of formative measurement models, and the use of small samples (e.g., [7]). Most notably, researchers have emphasized that PLS-SEM is particularly well suited for estimating models from an explanation-prediction perspective (e.g., [30,31,32]), which implies an understanding of the relationships assumed in the model as well as its ability to predict theoretical concepts under consideration [33].

Our advice to authors: Discuss how the use of PLS-SEM contributes to the overall aim of the analysis.

Researchers have developed variants of the standard PLS-SEM algorithm to deal with different data or model types. For instance, to adjust the original parameter estimates to accommodate common factor models, authors may revert to consistent PLS (PLSc), “which was first mathematically developed by Theo K. Dijkstra” [114] see also p. 299, for instance, [34,35], its PLSe1/PLS2e extensions [36,37], or Yuan et al.’s [38] Cronbach α based approach. Compared to the standard algorithm, PLSc builds on different assumptions regarding the nature of measurement, most notably, how theoretical concepts are represented in statistical models [29]. While different perspectives are tenable [39], authors need to be aware of the underlying assumptions and potential consequences if there is a misalignment between the model type (i.e., composite vs. common factor models) and estimator (e.g., standard PLS-SEM vs. PLSc-SEM) [40,41].

Our advice to authors: Consider the assumptions, characteristics, and different outcomes of a composite estimation of constructs compared to a common factor estimation.

2.3. Types of data suitable for PLS-SEM and quality criteria to report

The PLS-SEM algorithm relies on a series of linear regressions coupled with linear combinations to estimate the model parameters [2], Chapter 2 [42]. The indicator variables in a construct's measurement model should have data on a metric scale (ratio or interval measurement), such as age or income [43], Chapter 1. However, researchers also use survey data with ordinal scales (e.g., Likert-type scales). These data are useful for PLS-SEM when the researchers can justify equidistant data points (i.e., measurement on a quasi-metric scale). Alternatively, researchers may consider approaches for nonmetric partial least squares [44] and ordinal partial least squares (e.g., [45,46]). Also, Lohmöller [2], Chapter 4) presents a PLS-SEM approach that only uses binary variables (see also [47]), which can also be used for categorical and ordinal variables, or a mixture of both. However, binary, and categorical variables should usually not be mixed with metric or quasi-metric indictors in a construct's measurement model as this may complicate the interpretation of the results. PLS-SEM users can relatively easily include binary-coded data as control variables, moderators or grouping variables (i.e., when conducting a PLS-SEM multigroup analysis) in their analyses [for further details see [48]].

Our advice to authors: Report the scale of the data.

The data used in the analysis can come in the form of primary or secondary data (for further details on the suitability and use of secondary data in PLS-SEM, see [43], Chapter 1). In using such data, researchers typically draw on samples, drawn from a larger population. This step, however, requires (1) clearly defining the population about which inferences will be made, and (2) statements regarding the sample's (specific) representativeness [49]. The latter step entails safeguarding that the sample matches the population with regard to relevant characteristics (e.g., age, income, or gender). Researchers sometimes use samples generated by market research companies (e.g., [50]). These data usually include a weighting variable to ensure the sample's representativeness with regard to relevant characteristics. The weighted PLS-SEM algorithm [51] – see also [52] – allows aligning the sample and population with regard to such characteristics by weighting observations differently. Further, the use of digital sources of data such as Amazon's Mechanical Turk (MTurk) is on the rise. Authors are advised to familiarize themselves with the discussions around the pitfalls of such data sources [53].

Our advice to authors: Discuss the population and the sample's structure. Apply sampling weights in case of a mismatch between sample and population with regard to key characteristics.

As with any multivariate analysis method, the use of large samples is usually advantageous. However, in some situations, the population of interest is extremely small. For example, in business-to-business and family business research, populations are often restricted in size (e.g., [54,55]). In such situations, researchers must ensure that the PLS-SEM method meets the minimum sample size requirements (e.g., [5,43], Chapter 1). Researchers using PLS-SEM should therefore run power analyses or rely on heuristics such as Kock and Hadaya [56] inverse square root method to determine the sample size needed for achieving a certain level of statistical power (typically 80%; [57]). Researchers in epidemiology have proposed alternative means for minimum sample size requirements that do not rely on the concept of statistical power, but specify maximum allowable margins of error in confidence intervals for population effect sizes based on sample data [58]. Such an approach may be advantageous as there is no reliance on traditional inference testing, which has come under scrutiny in some research areas (e.g., [59]).

Our advice for authors: Ensure that the analysis yields sufficient levels of statistical power.

Datasets typically have missing values, which need to be treated. As missing value treatment is, in a sense, data manipulation, it should remain within reasonable limits. For instance, researchers have suggested that variables with more than 15% missing values be removed from the dataset [43], Chapter 1. Typical missing value treatment options are mean replacement, expectation-maximisation (EM), hot and cold deck, maximum likelihood, nearest neighbour, and regression imputation [60], Chapter 2. For example, Wang, Lu and Liu [61] suggested an EM algorithm for missing data imputation in PLS-SEM. Alternatively, researchers can delete observations with missing values, a process also known as casewise or listwise deletion. For studies using PLS-SEM, Hair, Hult, Ringle and Sarstedt [43], Chapter 1 suggest applying mean replacement when less than 5% of the values per variable are missing and to consider the use of casewise deletion otherwise. However, researchers need to ensure that the deletion of observations does not occur systematically and that sufficient observations remain for analysis.

Our advice for authors: Report the percentage of missing values for the entire dataset and the percentage of missing values per variable. Additionally, report the missing value treatment option used.

Besides treating missing values, authors should inspect their dataset for suspicious response patterns, such as straight lining or alternating extreme pole responses. An analysis of descriptive statistics (e.g., mean, variance and distribution of the answers per respondent) and/or graphical representation of the data enables the identification of suspicious response patterns (for further details, see [43], Chapter 1). Researchers should also rule out possibility that participants respond at random to survey items or tasks. Prior research has shown that rates of random responding can be substantial, offering practical suggestions for detecting corresponding response patterns [113].

Our advice to authors: Report the analyses carried out to identify and treat suspicious response patterns, including random responding.

Similarly, researchers typically address outliers, which can come in the form of extreme responses or extreme values. Statistical software packages support the identification of outliers through univariate, bivariate or multivariate statistics and charts, for example, box plots and stem-and-leaf plots [62], Chapter 5. Boxplots can help identify responses that are three times the interquartile range below the first quartile or above the third quartile. Such responses could be extreme outliers due to erroneous data in which case they need to be eliminated. If an explanation is available for exceptionally high or low values, outliers are typically retained; see Sarstedt and Mooi [62], Chapter 5, for more details about outliers and their treatment.

Our advice for authors: Report the outlier detection approach, provide a justification for deleting outliers, and report the number of outliers deleted.

Finally, researchers should provide the covariance matrix of their data as well as key descriptive statistics. The latter should include, for example, the minimum and maximum value, the mean value and median, the skewness and kurtosis, and the result of a nonnormality test statistic (e.g., the Anderson-Darling test or Cramér-von Mises test). PLS-SEM is a nonparametric method, which can accommodate nonnormal data. However, nonnormal data may lead to skewed bootstrap distributions, which negatively affect the bootstrap confidence intervals for significance testing. In such a situation, researchers may prefer bias-corrected and accelerated bootstrap confidence intervals over percentile-based confidence intervals – the standard variant in PLS-SEM studies [63].

Our advice for authors: Provide the covariance matrix and key descriptive statistics, including results of nonnormality tests.

2.4. Everything PLS-SEM, once the model stands

Model estimation using PLS-SEM requires reporting and citing the software used as well as any algorithm settings deviating from the defaults; not only for the PLS-SEM algorithm, but all computations including, for instance, bootstrapping and PLS_predict. Besides commercial software packages such as SmartPLS [64] and WarpPLS [65], authors can also revert to the open source statistical software R [66] and its cSEM [67] and SEMinR [68] packages for PLS-SEM.

After the results computation, authors should report the criteria for reflective and formative measurement model assessment and the structural model. Several articles and textbooks provide overviews of which PLS-SEM results and criteria should be reported and how [69], [70], [71]. Table 1 provides a summary of key criteria as discussed in, for example, Hair, Hult, Ringle and Sarstedt [43]. Additional components may include endogeneity assessment (e.g., [72]), model comparisons (e.g., [31]), or extended visualizations of PLS-SEM results such as importance-performance map analysis (e.g., [73]).

Table 1.

Results reporting.

Reflective measurement models
Evaluation purpose	Criterion
Internal consistency reliability	ρA ≥ 0.7 and < 0.95; consider reporting Cronbach's α and ρC as lower and upper boundaries, respectively (note that these two additional metrics require that indicator correlations in a measurement model are either all positive or negative)
Indicator reliability	Loadings: ≥ 0.7 (or 0.708 to be precise)
Convergent validity	Average variance extracted (AVE): ≥ 0.5
Discriminant validity	HTMT+ (and HTMT2+): ≤ 0.85 or 0.9. Assess whether the selected threshold falls into the (one-sided) bootstrap confidence interval
Formative measurement models
Convergent validity	Redundancy analysis
Collinearity	Variance inflation factor (VIF): ≤ 5, ideally ≤ 3
Significance and relevance of indicators	Bootstrap-based significance testing of indicator weights (and loadings); size of the coefficients
Structural model
Collinearity	Variance inflation factor (VIF): ≤ 5, ideally ≤ 3
Significance and relevance of path coefficients	Bootstrap-based significance testing; (effect) size of the coefficients
Predictive power and model fit	Primary focus is on prediction: Focus on predictive power assessment using PLSpredict and the cross-validated predictive ability test (CVPAT) Primary focus is on explanation: Model fit metrics (e.g., SRMR) and bootstrap-based tests for model fit, but consider limitations related to their applicability Focus is on both prediction and explanation: Consider the trade-off between model fit and predictive power

In terms of measurement model assessment, particular care should be devoted to discriminant validity assessment, which ensures that each construct is empirically unique and captures a phenomenon not represented by other constructs in a statistical model [74]. Safeguarding discriminant validity is of crucial concern to ensure that implications drawn from the analysis of structural model relationships are not the result of statistical discrepancies [75]. The primary criterion for PLS-SEM-based discriminant validity assessment is Henseler et al.’s [76] HTMT criterion and its recent extension HTMT2 [77]. While both criteria will not differ much in common applications, their results may be affected by negative correlation patterns. We therefore propose adjusted versions of the criteria, which rely on absolute correlations in their computations. We refer to these criteria as HTMT+ and HTMT2+ to indicate that they only employ positive (absolute) correlation values (hence, the additional plus symbol). We document these criteria and showcase their usefulness in the Appendix. To assess discriminant validity, researchers are advised to use the adjustment of the HTMT criterion (i.e., HTMT+), which is available, for instance, in the widely adopted SmartPLS software [64] for PLS-SEM analyses. In some extreme data and model constellations (especially when the number of indicators is low and their heterogeneity of loadings is particularly high), researchers may also consider the adjustment of the HTMT2 criterion (i.e., HTMT2+).

In terms of structural model assessment, authors need to assess their model's predictive power, which is often neglected in regression-based studies [32]. Authors are therefore advised to routinely run Shmueli et al.’s [78] PLS_predict procedure and follow extant guidelines for results reporting [79]. While the PLS_predict focuses on the model's predictive power on an item-level, authors are also advised to compare their model's predictive power on a construct level – either on the grounds of all endogenous constructs or one specific key target construct. Sharma et al.’s [80] extension of Liengaard et al.’s [31] cross-validated predictive ability test (CVPAT) facilitates such analyses.

In following the guidelines in Table 1, authors need to be aware of two streams in the PLS-SEM literature that suggest different procedures for model evaluation. While Schuberth, Rademaker and Henseler [81] emphasize the role of model fit for results assessment, using indices such as the GFI, NFI, and SRMR or bootstrap-based tests for model fit, Hair, Hult, Ringle and Sarstedt [43], Chapters 1 and 6 focus on predictive power assessment, emphasizing the method's causal-predictive nature (see also [82]). Both procedures are tenable, but authors need to be aware of the two streams and make an informed assessment which route they follow. In addition, researchers need to be aware of potential trade-offs between the two perspectives. Focusing on model fit may sacrifice predictive power [32] – a practice which has been intensively discussed in different methodological contexts [83,84]. Finally, researchers can conduct various robustness checks to safeguard the validity of the results. For instance, Hair et al. [85], Sarstedt et al. [69], and Sarstedt et al. [86] provide an overview of these analyses and the reporting of their results.

Our advice for authors: Comply with the most recent guidelines for PLS-SEM use, while considering different streams in the literature. Run robustness checks to safeguard the validity of the results.

2.5. How to communicate the value to user communities

Authors of data articles are expected to articulate why and how the data article generates value. [8]. Some generic examples to aid the communication of the value statement are provided here:

• Reference material: A dataset can be useful as a reference for understanding a particular phenomenon in a particular context and can illuminate the hands-on application of specific methodologies that were used to uncover the phenomenon. This can be of particular relevance for single-shot (cross-sectional) primary and survey-based data, which are usually not available via public databases or research archives of funding bodies.

• Replication: If the underlying dataset was produced through replication, authors need to be aware which form of replication they applied [8]. This will aid the communication of the dataset's contribution. With the provision of openly accessible data, it is possible to establish whether previously published empirical findings can be repeated and whether similar findings can be obtained under comparable conditions. When using the same data, this refers to the stability of results through verification or reanalysis and thus points to the findings’ robustness. As soon as additional and different data are introduced, further value can be provisioned through direct replication and extension [14], Table 9.1 on p. 159. Such replications are crucial in light of increasing concerns regarding the stability of even seminal behavioural effects like nudging [87]. Authors have the opportunity to use the data article to highlight the research avenues that the dataset feeds into as well as delineate other potential uses for the dataset. Some of this may stem from ‘future research’ sections of previous research articles or relate to new possibilities that the new data provide.

• Data pooling: Pooling data from previously existing studies can be useful to increase the sample size and the statistical power of the analysis. In international contexts, the ability to pool data can support the development of cross-country, comparative models and thus help establish an understanding of key differences. Conceptual issues of equivalence must be considered, which include research subject, time and context, to render such data pooling efforts useful [88]. Empirical considerations and words of caution regarding equivalence issues should be discussed in the model evaluation section. In the context of PLS-SEM, this involves running Henseler et al.’s [89] MICOM procedure.

• Temporal value: The provision of open access data may contribute to an understanding of value changes regarding the underpinning constructs over time. This will be of special value to studies in which sociological or psychological constructs are involved. While different sampling and equivalence concerns may render direct comparison difficult, tendential assessment can produce value for research communities.

Frequently, authors become overly excited when referring to their ‘unique’ datasets, even when these datasets would not have been possible without the help of intermediary parties and grant-funding bodies and may have been the result of significant collaborative efforts involving academics at various stages of their careers. Researchers in the social sciences, humanities, and business and management are still considerably reluctant to share their data [90]. Christensen, Freese and Miguel [14] report the case of a mysterious psychologist who claimed a loss of data in response to a query about suspicious results, which “exemplifies one reason researchers might not want to share their data: maybe they have something to hide” [14], p. 176. Null results in relationships not openly documented in the original manuscript may be such a reason. In support of open science, data journals, encourage the inclusion of null results in data articles. Another, less uncomfortable explanation for the reluctance of researchers to share datasets is the fear of losing further publication opportunities to extend the model and analysis, if other research teams manage to exploit these opportunities more quickly. Yet, with reproducibility and transparency requests increasing [91] and social science, humanities and business and management becoming eager to close the open data gap to the natural and medical sciences [92,93], it will also become attractive for curators of such datasets to receive appropriate recognition and credit. New citation standards regarding referencing of datasets, such as those included in the seventh edition of the Publication Manual of the American Psychological Association [94] and modern ‘how to’ guides that include datasets and data articles [95], will help increase the value of open data further.

3. Key points

• •Ensure model is meaningful: Report the model, measures, including any changes made and underpinning literature
• •Demonstrate that the model and PLS-SEM results are meaningful and relevant
• •Check whether the data is appropriate for analysis and report quality criteria
• •Report scale levels of data, representativeness of sample, minimum sample size requirements, percentage of missing values and missing data option used, analyses carried out to identify suspicious response patters and deleted observations, covariance matrix and descriptive statistics, results reporting including criteria for reflective and formative measurement models and the structural model.

CRediT authorship contribution statement

Christian M. Ringle: Conceptualization, Methodology, Resources, Software, Writing – original draft, Writing – review & editing, Visualization. Marko Sarstedt: Conceptualization, Methodology, Resources, Writing – review & editing, Visualization. Noemi Sinkovics: Conceptualization, Methodology, Resources, Writing – original draft, Writing – review & editing, Supervision, Project administration. Rudolf R. Sinkovics: Conceptualization, Resources, Writing – original draft, Writing – review & editing.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Although this research does not use the statistical software SmartPLS (https://www.smartpls.com), Christian M. Ringle acknowledges a financial interest in SmartPLS.

A. Appendix

A.1. Extending the standard HTMT metrics for discriminant validity testing

For any two constructs ${\xi }_i$ and ${\xi }_j$ measured by K_i and K_j indicators, Henseler, Ringle and Sarstedt [76] defined the HTMT as the arithmetic mean of the heterotrait-heteromethod correlations r_ig,jh relative to the geometric mean of the average monotrait-heteromethod correlations of ${\xi }_i$ (i.e., r_ig,ih) and the average monotrait-heteromethod correlation of ${\xi }_j$ (i.e., r_jg,jh):

$$\text{HTM}{\mathrm{T}}_{ij}=\frac{\frac{1}{K_i{K}_j}{\sum }_{g=1}^{K_i}{\sum }_{h=1}^{K_j}{r}_{ig,jh}}{\sqrt{\frac{2}{K_i\left(K_i-1\right)}{\sum }_{g=1}^{K_i-1}{\sum }_{h=g+1}^{K_i}{r}_{ig,ih}·\frac{2}{K_j\left(K_j-1\right)}{\sum }_{g=1}^{K_j-1}{\sum }_{h=g+1}^{K_j}{r}_{jg,jh}}}.$$

Based on prior research and simulation study results, Henseler, Ringle and Sarstedt [76] suggest a threshold value of 0.90 if constructs are conceptually very similar and 0.85 if the constructs are conceptually more distinct. That is, an HTMT value for two constructs that exceeds the corresponding threshold indicates a lack of discriminant validity. In addition, these authors suggested testing whether HTMT is significantly different from unity, which is a very liberal way of testing for discriminant validity testing. Rather than relying on cutoff values lower than unity, Franke and Sarstedt [74] suggest testing whether an HTMT value differs significantly from a selected threshold different from unity (e.g., 0.90). To do so, researchers can compute the HTMT statistic's bootstrap-based confidence interval and assess whether the corresponding cutoff value falls into the interval.

The HTMT criterion rests on the assumption that the indicators’ population loadings are equal, a requirement also referred to as tau-equivalence [96]. However, this assumption is unlikely to hold in many empirical settings [97]. Addressing this concern, Roemer, Schuberth and Henseler [77] introduced an extension of the original criterion, which considers the geometric mean instead of the arithmetic mean for calculating the average indicator correlations [77], Eq. 31:

$$\text{HTMT}{2}_{ij}=\frac{\sqrt[K_i·K_j]{{\prod }_{g=1}^{K_i}{\prod }_{h=1}^{K_j}{r}_{ig,jh}}}{\sqrt{\sqrt[\frac{K_i^2-K_i}{2}]{{\prod }_{g=1}^{K_i-1}{\prod }_{h=g+1}^{K_i}{r}_{ig,ih}}·\sqrt[\frac{K_j^2-K_j}{2}]{{\prod }_{g=1}^{K_j-1}{\prod }_{h=g+1}^{K_j}{r}_{jg,jh}}}}$$

Roemer et al.’s [77] simulation study shows that the HTMT2 slightly outperforms the original metric when the indicator loading patterns are extremely heterogeneous (e.g., some indicator loadings are around 0.5, while others are close to 1), particularly when ${\xi }_i$ and ${\xi }_j$ are highly correlated (see the HTMT2 discussion in [69]). While the HTMT2 addresses conceptual concerns regarding the original metric's tau-equivalence assumption, both HTMT variants require the average monotrait-heteromethod correlations to be positive: “Computation of the HTMT/HTMT2 assumes that all intra-block and inter-block correlations between indicators are either all-positive or all-negative” [67], p. 17. Similarly, Roemer et al. [77], p. 2647 note that the HTMT “is not defined for cases, in which indicator correlations are negative.” However, this is not necessarily the case in empirical applications of PLS-SEM and SEM in general.

Consider the following example taken from a model on customer satisfaction, where the construct ${\xi }_1$, represents the respondents’ trust [98] and is measured with indicators x₁₁, x₁₂, x₁₃, and x₁₄. The construct ${\xi }_2$ represents their perceived switching costs [99] and is measured with indicators x₂₁, x₂₂, x₂₃, and x₂₄. A professional market research firm collected responses from 5,368 different consumers in three parallel study waves. Each study wave drew random samples from the German population, but with slight variations in the indicator scaling. Specifically, construct measurement in wave #1 (n=1,812) relies on the original indicator wording; in wave #2 (n=1,767), x₁₁ was reverse-scaled; in wave #3 (n=1,789), x₁₁ and x₂₁ were reverse-scaled.

Table A1 documents the correlation patterns between the indicators of ${\xi }_1$ and ${\xi }_2$ for wave #1. The dark-shaded elements denote the heterotrait-heteromethod correlations, while the remaining off-diagonal elements correspond to the monotrait-heteromethod correlations of ${\xi }_1$ and ${\xi }_2$ respectively.

Table A1.

Indicator correlations (wave #1).

		${\xi }_1$				${\xi }_2$
		x11	x12	x13	x14	x21	x22	x23	x24
${\xi }_1$	x11	1.000
	x12	0.504	1.000
	x13	0.456	0.632	1.000
	x14	0.463	0.485	0.510	1.000
${\xi }_2$	x21	0.311	0.477	0.380	0.289	1.000
	x22	0.396	0.468	0.405	0.591	0.380	1.000
	x23	0.362	0.435	0.382	0.341	0.445	0.396	1.000
	x24	0.343	0.455	0.367	0.314	0.556	0.400	0.382	1.000

Note: The dark-shaded elements denote the heterotrait-heteromethod correlations, while the remaining off-diagonal elements correspond to the monotrait-heteromethod correlations.

All indicator correlations are positive, producing the following values for HTMT and HTMT2:

$$\text{HTMT}=\frac{0.395}{\sqrt{0.508·0.427}}=0.848$$

$$\text{HTMT}2=\frac{0.388}{\sqrt{0.505·0.423}}=0.840$$

Assuming a cutoff value of 0.90 or higher in light of the constructs’ conceptual similarity, both HTMT metrics would indicate discriminant validity as both values are lower than this threshold. As expected in light of the measurement models’ quality, HTMT and HTMT2 are in close correspondence [69]. Researchers could also run bootstrapping and construct confidence intervals to test whether the two HTMT values are significantly lower than this threshold.

However, a different picture emerges when computing the HTMT metrics based on the wave#2 correlation matrix. Using the indicator correlations documented in Table A2 as input produces an HTMT value far above 1, which is surprising as the reverse-coding of a single indicator (x₁₁) should not alter whether or not ${\xi }_1$ and ${\xi }_2$ are empirically distinct. More severely, HTMT2 is not defined because of the negative first terms that result in the formulas’ denominator.

$$\text{HTMT}=\frac{0.242}{\sqrt{0.045·0.452}}=1.703$$

$$\text{HTMT}2=\frac{0.416}{\sqrt{\sqrt[6]{-0.016}·\sqrt[6]{0.008}}}=notdefined$$

Table A2.

Indicator correlations (wave #2).

		${\xi }_1$				${\xi }_2$
		x11	x12	x13	x14	x21	x22	x23	x24
${\xi }_1$	x11	1.000
	x12	-0.520	1.000
	x13	-0.446	0.659	1.000
	x14	-0.426	0.499	0.501	1.000
${\xi }_2$	x21	-0.317	0.526	0.420	0.317	1.000
	x22	-0.413	0.506	0.419	0.621	0.397	1.000
	x23	-0.369	0.442	0.358	0.339	0.438	0.419	1.000
	x24	-0.362	0.549	0.441	0.388	0.579	0.460	0.419	1.000

Note: The dark-shaded elements denote the heterotrait-heteromethod correlations, while the remaining off-diagonal elements correspond to the monotrait-heteromethod correlations.

A similar problem occurs when using the indicator correlations from wave #3, where one indicator in each of the two constructs is reverse-coded (x₁₁ and x₂₁). Using the indicator correlations documented in Table A3 as input produces negative values in the formulas’ denominators, rendering the computation of HTMT and HTMT2 impossible:

$$\text{HTMT}=\frac{0.119}{\sqrt{0.064·\left(-0.030\right)}}=notdefined$$

$$\text{HTMT}2=\frac{0.375}{\sqrt{\sqrt[6]{-0.010}·\sqrt[6]{-0.005}}}=notdefined$$

Table A3.

Indicator correlations (wave #3).

		${\xi }_1$				${\xi }_2$
		x11	x12	x13	x14	x21	x22	x23	x24
${\xi }_1$	x11	1.000
	x12	-0.449	1.000
	x13	-0.373	0.615	1.000
	x14	-0.404	0.487	0.510	1.000
${\xi }_2$	x21	0.254	-0.469	-0.375	-0.309	1.000
	x22	-0.343	0.437	0.382	0.627	-0.350	1.000
	x23	-0.276	0.422	0.331	0.343	-0.448	0.373	1.000
	x24	-0.350	0.488	0.390	0.355	-0.546	0.384	0.407	1.000

Note: The dark-shaded elements denote the heterotrait-heteromethod correlations, while the remaining off-diagonal elements correspond to the monotrait-heteromethod correlations.

These and similar problems (e.g., negative and positive correlations cancelling each other out in the computation of the mean) have been recognized in the context of the original HTMT metric and considered in its implementation in SmartPLS, the most frequently used software for PLS-SEM analyses in various fields of business research (e.g., [100,101]) and other disciplines (e.g., [102,103]). Specifically, since the HTMT computation update in version 3.2.1 of SmartPLS 3 [104]2 and continuing with SmartPLS 4 [64], the software considers the absolute correlation values in the computation of the HTMT.3 Since the correlations are only intended to determine the indicators’ empirical overlap, their signs are not decisive. Hence, both the HTMT and HTMT2 metrics may draw on absolute correlations to avoid problems that occur when encountering correlation patterns as in the wave #2 and wave #3 examples.

To distinguish these adjusted versions from their previous implementations, we refer to these metrics as HTMT+ and HTMT2+:

$$\text{HTMT}{+}_{ij}=\frac{\frac{1}{K_i{K}_j}{\sum }_{g=1}^{K_i}{\sum }_{h=1}^{K_j}\left|r_{ig,jh}\right|}{\sqrt{\frac{2}{K_i\left(K_i-1\right)}{\sum }_{g=1}^{K_i-1}{\sum }_{h=g+1}^{K_i}\left|r_{ig,ih}\right|·\frac{2}{K_j\left(K_j-1\right)}{\sum }_{g=1}^{K_j-1}{\sum }_{h=g+1}^{K_j}\left|r_{jg,jh}\right|}}$$

$$\text{HTMT}2{+}_{ij}=\frac{\sqrt[K_i·K_j]{{\prod }_{g=1}^{K_i}{\prod }_{h=1}^{K_j}\left|r_{ig,jh}\right|}}{\sqrt{\sqrt[\frac{K_i^2-K_i}{2}]{{\prod }_{g=1}^{K_i-1}{\prod }_{h=g+1}^{K_i}\left|r_{ig,ih}\right|}·\sqrt[\frac{K_j^2-K_j}{2}]{{\prod }_{g=1}^{K_j-1}{\prod }_{h=g+1}^{K_j}\left|r_{jg,jh}\right|}}}$$

Since all correlations in wave #1 are positive, the results of HTMT+ and HTMT2+ are identical to those of HTMT and HTMT2. Computing the HTMT+ and HTMT2+ values for waves #2 and #3 as outlined above produces the following results, which are very close to the values from wave #1. This result is expected considering that the samples have been drawn from the same population.

Wave #2:

$$\text{HTMT}+=\frac{0.424}{\sqrt{0.509·0.452}}=0.885$$

$$\text{HTMT}2+=\frac{0.416}{\sqrt{0.503·0.448}}=0.876$$

Wave #3:

$$\text{HTMT}+=\frac{0.384}{\sqrt{0.473·0.418}}=0.865$$

$$\text{HTMT}2+=\frac{0.375}{\sqrt{0.467·0.413}}=0.854$$

This adjustment resolves the problem that a simple recoding of indicators or lack thereof may alter the results of a discriminant validity analysis.

In case of deliberately reverse-coded items, researchers may, of course recode the corresponding item(s) and apply the standard HTMT metrics [77]. However, negative correlation patterns may occur systematically, but without any prior expectations, causing the standard HTMT metrics to fail. For example, negative correlation patterns among indicators routinely occur in applications of personality scales and measurements of emotions (e.g., PANAS; [105]), clinical symptoms (e.g., [106,107]), response styles (e.g., [108,109]), and response times (e.g., [110])—to name a few. In such situations, reverse-coding might be difficult to defend from a conceptual standpoint, calling for metrics that are universally applicable to situations where negative correlation patterns occur. Note that the above examples are different from unsystematic variations in indicator loadings, for example, due to sampling variance. We will discuss this aspect in the following section.

A.2. Additional insights from simulations

Supplementary simulation studies will allow us to gain additional insights into how HTMT in its different variants can be used for discriminant validity testing. More specifically, one of the reviewers of this article pointed out that negative correlations may also occur unsystematically, despite positive population values due to sampling variation.4 To show that such negative correlations can have adverse consequences for the computation of the HTMT+ metric, the reviewer presented the results of a small simulation study assuming a two-construct model with a 0.3 correlation, each measured by three indicators with loadings of 0.5. When estimating this model with n = 100 and 100,000 replications, the HTMT+ produced a larger than the expected value of 0.3, while the regular HTMT was not biased upwards.

While the metric's behaviour is important to understand, this simulation design constitutes a borderline situation with limited practical relevance due to (1) very low indicators loadings, (2) low construct correlations, and (3) a small sample size. We expect that the issue described above will vanish with higher indicator loadings, larger correlations between the constructs, and higher sample sizes. Taking these limitations into account, we first replicated the simulation with two constructs correlated at 0.3 and each measured by four indicators, which aligns with our empirical example presented in the previous section and corresponds to the measurement model complexity commonly encountered in applications of PLS-SEM (e.g., [69]). Our simulation study additionally considers the HTMT2 and HTMT2+ metrics as well as a series of further simulation factors. First, we considered a broader range of loadings. Specifically, loadings of 0.5 would raise a red flag in any empirical application (Table 1) as they would produce unbearably low internal consistency reliability (and convergent validity) values, rendering the discriminant validity assessment meaningless, because “a necessary (but not sufficient) condition for validity of measures is that they are reliable” [111], p. 6. We therefore also considered homogeneous loadings of 0.7 and a pattern of heterogeneous loadings with values of 0.9, 0.8, 0.7, and 0.6 for both constructs, ensuring that the lowest loading still meets minimum standards in terms of indicator reliability (Table 1). We reiterated these patterns by assuming a negative loading of equal size for one of the two constructs (e.g., -0.7 for the first indicator of the first constructs, all other loadings 0.7). Second, we not only considered a sample size of 100, but also 200 and 300, because prior reviews of PLS-SEM report similar average sample sizes in empirical applications of the method (e.g., [69]). We additionally considered a sample size of 10,000 to shed light on the metrics’ asymptotic behaviour. Finally, as 10,000 replications produced highly similar results as 100,000 replications in our initial analysis, we considered this lower number to save computational time.5 We used the statistical software R [66] with its foreach package [112] for these simulations. Table A.4 shows the results of our analysis for measurement models with positive loadings only.

Table A4.

Simulation results (construct correlation 0.3, positive loadings only).

Expected HTMT value: 0.3; all loadings 0.5
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	0.305	0	0.420	0	0.329	5173	0.330	0
200	0.299	0	0.345	0	0.277	4715	0.267	0
300	0.301	0	0.323	0	0.271	4308	0.256	0
10,000	0.300	0	0.300	0	0.298	0	0.298	0
Expected HTMT value: 0.3; all loadings 0.7
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	0.300	0	0.313	0	0.295	2959	0.266	0
200	0.298	0	0.300	0	0.288	1341	0.273	0
300	0.299	0	0.300	0	0.289	519	0.282	0
10,000	0.300	0	0.300	0	0.299	0	0.299	0
Expected HTMT value: 0.3; loadings (per construct): 0.9 / 0.8 / 0.7 / 0.6
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	0.302	0	0.311	0	0.299	2629	0.273	0
200	0.300	0	0.302	0	0.292	1264	0.278	0
300	0.301	0	0.302	0	0.291	567	0.285	0
10,000	0.302	0	0.302	0	0.299	0	0.299	0

For loadings of 0.5, we find that the HTMT2 produces a great share of inadmissible solutions of up to 52% at sample sizes of 100, 200, and 300, while this is not the case for the other metrics. Looking at the estimates, we find that only HTMT returns (almost) unbiased average results for sample sizes of 100, 200, and 300. The other metrics improve in performance as sample sizes increases. Considering the scenario with 0.7 loadings for all indicators slightly changes the picture. HTMT2 still produces inadmissible solutions at small sample sizes, ranging between approximately 5% and 30%, which is less than in the previous scenario with lower loadings. In terms of parameter bias, HTMT shows practically no bias, whereas the HTMT+ metric is slightly upwards biased only for sample size of 100. HTMT2 and HTMT2+ are downwards biased at small sample sizes of 200 and 300, but less so compared to the scenario with 0.5 loadings. These metrics’ bias diminishes as sample size increases. Finally, in the case of heterogeneous loadings, the HTMT2’s share of inadmissible solutions is still high. The values of HTMT, HTMT+, and HTMT2 are highly similar, whereas HTMT2+ has a more pronounced underestimation tendency.

Table A.5 shows the results for the scenario where one loading is negative. As expected, HTMT and HTMT2 produce a significant share of inadmissible solutions. In the few cases where these metrics can be computed, the results are substantially biased. On the contrary, the HTMT+ and HTMT2+ produce the same results as when assuming only positive loadings (Table A.4).

Table A5.

Simulation results (construct correlation 0.3, negative and positive loadings).

Expected HTMT value: 0.3; first loading of the first construct -0.5, all other loadings 0.5
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	0.689	5083	0.420	0	0.430	9854	0.330	0
200	0.800	5015	0.345	0	0.326	9998	0.267	0
300	0.893	5060	0.323	0	n/a	10,000	0.256	0
10,000	2.252	4966	0.300	0	n/a	10,000	0.298	0
Expected HTMT value: 0.3; first loading of the first construct -0.7, all other loadings 0.7
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	1.095	5095	0.313	0	n/a	10,000	0.266	0
200	1.312	5079	0.300	0	n/a	10,000	0.273	0
300	1.401	5070	0.300	0	n/a	10,000	0.282	0
10,000	3.339	5105	0.300	0	n/a	10,000	0.299	0
Expected HTMT value: 0.3; loadings: -0.9 / 0.8 / 0.7 / 0.6 (first construct), 0.9 / 0.8 / 0.7 / 0.6 (second construct)
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	n/a	10,000	0.311	0	n/a	10,000	0.273	0
200	n/a	10,000	0.302	0	n/a	10,000	0.278	0
300	n/a	10,000	0.302	0	n/a	10,000	0.285	0
10,000	n/a	10,000	0.302	0	n/a	10,000	0.299	0

The previous settings assumed a relatively low construct correlation of 0.3. Understanding the metrics’ performance in such a scenario is important as low construct correlations favour the occurrence of negative indicator correlations, triggering a substantial number of inadmissible solutions in the HTMT2 metric. At the same time, however, discriminant validity issues are less likely emerge when construct correlations are low. This is also why Roemer et al. [77] explicitly disregarded such correlations and have not identified HTMT2’s tendency to produce inadmissible solutions. We therefore replicated the above analyses, but—in view of empirical applications—assumed a very high construct correlation of 0.8. Table A6 shows the results of our analysis for measurement models with positive loadings only.

Table A6.

Simulation results (construct correlation 0.8, positive loadings only).

Expected HTMT value: 0.8; all loadings 0.5
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	0.815	0	0.821	0	0.788	2457	0.774	0
200	0.804	0	0.804	0	0.783	362	0.778	0
300	0.804	0	0.804	0	0.789	34	0.788	0
10,000	0.800	0	0.800	0	0.800	0	0.800	0
Expected HTMT value: 0.8; all loadings 0.7
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	0.801	0	0.801	0	0.792	3	0.792	0
200	0.799	0	0.799	0	0.795	0	0.795	0
300	0.800	0	0.800	0	0.798	0	0.798	0
10,000	0.800	0	0.800	0	0.800	0	0.800	0
Expected HTMT value: 0.8; loadings (per construct): 0.9 / 0.8 / 0.7 / 0.6
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	0.806	0	0.806	0	0.793	28	0.793	0
200	0.805	0	0.805	0	0.795	0	0.795	0
300	0.806	0	0.806	0	0.798	0	0.798	0
10,000	0.806	0	0.806	0	0.800	0	0.800	0

As expected, we find that the number of inadmissible solutions in HTMT2 is much smaller than in the previous analyses, occurring mostly in situations where n = 100. Considering the parameter bias, we find that all metrics are generally in sync when the measurement models meet common quality standards. In these situations, HTMT2 has a marginal underestimation tendency, while HTMT has a marginal overestimation tendency. This also holds for the case of heterogeneous loadings, where one would expect the HTMT2 to outperform the HTMT metric. The HTMT+ and HTMT2+ metrics generally correspond to their standard versions, except for the borderline case with loadings of 0.5 and n = 100, where the differences are somewhat more pronounced.

A more detailed analysis of the results shows that HTMT and HTMT+ values are slightly too high (0.805 or 0.806 instead of 0.800) when loadings are heterogeneous, which parallels Roemer et al.’s [77] findings. However, in situation with equal loadings (in line with HTMT's assumption of tau-equivalent measurement models; [76,96]), this slight bias vanishes. On the contrary, HTMT2 and HTMT2+ are slightly too low at sample sizes of 100, 200, and 300, regardless of whether loadings are homogeneous or heterogeneous. This downward bias diminishes as sample sizes increase, which is in line with Roemer et al.’s [77] findings. Hence, HTMT and HTMT+ can be considered slightly more conservative, whereas HTMT2 and HTMT2+ can be considered more liberal metrics for discriminant validity assessment.

Finally, Table A7 shows the results for combinations of negative and positive loadings when the constructs are 0.8 correlated. The findings parallel those obtained in the setting with 0.3 construct correlations. HTMT and HTMT2 produce substantial amounts of inadmissible solutions, while their absolute variants (HTMT+ and HTMT2+) yield almost the same values as in the situation where all loadings are positive.6

Table A7.

Simulation results (construct correlation 0.8, negative and positive loadings).

Expected HTMT value: 0.8; first loading of the first construct -0.5, all other loadings 0.5
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	1.860	5054	0.821	0	0.930	9809	0.775	0
200	2.178	5034	0.804	0	0.973	9997	0.779	0
300	2.331	4967	0.804	0	n/a	10,000	0.789	0
10,000	5.869	5067	0.800	0	n/a	10,000	0.800	0
Expected HTMT value: 0.8; first loading of the first construct -0.7, all other loadings 0.7
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	2.969	5139	0.801	0	n/a	10,000	0.793	0
200	3.417	5104	0.799	0	n/a	10,000	0.795	0
300	4.031	5071	0.800	0	n/a	10,000	0.797	0
10,000	9.013	5019	0.800	0	n/a	10,000	0.800	0
Expected HTMT value: 0.8; loadings: -0.9 / 0.8 / 0.7 / 0.6 (first construct), 0.9 / 0.8 / 0.7 / 0.6 (second construct)
Sample size	HTMT	Number of inadmissible solutions	HTMT+	Number of inadmissible solutions	HTMT2	Number of inadmissible solutions	HTMT2+	Number of inadmissible solutions
100	n/a	10,000	0.806	0	n/a	10,000	0.793	0
200	n/a	10,000	0.805	0	n/a	10,000	0.795	0
300	n/a	10,000	0.806	0	n/a	10,000	0.798	0
10,000	n/a	10,000	0.806	0	n/a	10,000	0.800	0

To summarize, we find that particularly HTMT2 suffers from inadmissible solutions, not only in scenarios where negative loadings occur systematically (whether expected or not), but also in situations where construct correlations are low. While using HTMT2+ resolves this issue, the metric's performance doesn't substantially improve on the HTMT+ metric, even when loadings are heterogeneous (i.e., in this situation HTMT+ has a slight upwards bias while HTMT2+ has a downwards bias for smaller sample sizes). When loadings are homogeneous, HTMT+ actually shows a slightly better performance. In light of the direction of the bias, HTMT2+ can be considered more liberal as the metric shows a slight underestimation tendency. This behaviour can be problematic when comparing the metric against a firm cutoff value. Considering the occurrence of inadmissible solutions and the magnitude and size of the biases produced by the metrics, our results therefore suggest that researchers should primarily rely on the HTMT+ in their discriminant validity assessment tasks. However, they may also consider HTMT2+, especially when the loadings pattern is quite heterogenous (e.g., 0.95/0.8/0.65), and the sample is not too small (e.g., >300).

Due to the nature of this article, we focused on a limited number of simulation factors, which, however (1) directly relate to the reviewer's concern, and (2) extend the initial design to accommodate more realistic settings. Future research may add to these findings by researching the impact of further design factors on the metrics’ behavior, such as the number of indicators, which partly dictates the degree of possible heterogeneity in indicator loadings—assuming a certain quality of measurement models. Finally, future research could also extend on the concept of discriminant validity testing as implied by HTMT and consider measurement congruence, as recently discussed by [115].

Data Availability

• Empty reference data (filler link) (Original data) (https://www.sciencedirect.com/journal/data-in-brief).