Selecting Between Whole-Sample and Split-Sample Strategies for Exploratory and Confirmatory Factor Analyses: Guidance for Psychometric Scale Development

Abstract

Development of robust psychometric scales demands careful consideration of both sample size and analytic strategy. This simulation compared whole-sample, bootstrap, and split-sample approaches for exploratory and confirmatory factor analysis (EFA/CFA) across eight realistic sample sizes. Rather than asking whether small samples are problematic, a question that is already well established, we examined when splitting an available sample compromises factor-pattern recovery and CFA stability. Using synthetic data and resampling, we assessed effects on model fit, reliability, and factor structure reproducibility. Results showed that split-sample validation became most defensible when each half independently met minimum adequacy conditions; otherwise, whole-sample analysis supplemented by resampling-based stability checks provided more consistent evidence. Based on these findings, we propose a pragmatic decision framework for applied scale development. As lower-bound guidance, our results support a minimum of 200 cases for EFA and CFA and a case-to-item ratio of at least 10:1 to ensure replicability. When these conditions are not met, whole-sample or bootstrap-based approaches are likely to provide more stable solutions. Researchers should also consider model- and design-specific requirements, which may necessitate substantially larger samples than these minimum recommendations. In conclusion, these empirically grounded recommendations provide researchers with actionable guidance for optimising sample size and analytic approach, enhancing the validity and generalisability of psychometric instruments in psychoeducational measurement.

Introduction

The development of psychometric scales is a critical and multifaceted process in human and social sciences research. These instruments are designed to quantify latent constructs, such as attitudes, beliefs, and behaviour, that cannot be directly observed (Carpenter, 2018; Morgado et al., 2018). These scales typically comprise multiple items intended to represent theoretical variables, and their development requires theoretical insight, rigorous methodology, and robust statistical analysis (Carpenter, 2018; Morgado et al., 2018). The process generally follows three key stages: item generation, theoretical refinement, and psychometric evaluation (Carpenter, 2018; Scandura & Ford, 2023). High-quality measures are critical for drawing valid, reliable, and generalisable conclusions (Anthoine et al., 2014; Carpenter, 2018; Hawkins et al., 2020; Kline, 2016; Morgado et al., 2018).

One of the most important methodological considerations in scale development is determining an adequate sample size (Kyriazos, 2018). Insufficient samples compromise statistical power, the precision of parameter estimates, and the stability of latent structures. This issue is particularly relevant for exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), central techniques in scale construction whose accuracy depends on stable correlation and covariance estimates (Kline, 2016; Kyriazos, 2018; Williams & Hanna, 2023). Small samples increase the risk of unstable factor solutions, estimation failures, and misleading fit statistics (Kline, 2016; Kyriazos, 2018). Additionally, confidence intervals for model fit indices, such as the chi-square test and root mean square error of approximation (RMSEA), are heavily influenced by sample size, with small samples producing unacceptably wide intervals (Kyriazos, 2018). However, determining an appropriate sample size is not straightforward as it depends on factors such as model complexity, number of items, strength of factor loadings, reliability, and patterns of missing data (Kyriazos, 2018).

Another important consideration in scale development is the sampling strategy employed for EFA and CFA (Fokkema & Greiff, 2017; Wu et al., 2022; Zhou, 2022). In practice, the two methods are often used together: EFA provides post hoc interpretation of factor structures, while CFA tests a priori relationships among constructs or observed variables. As CFA is theory-driven, it is typically conducted as a subsequent step in the construct validation process, used to confirm structures suggested by EFA (Scandura & Ford, 2023). However, debate continues over whether both analyses should be performed on the same dataset (whole-sample strategy) or on separate subsamples (split-sample strategy) (Zhou, 2022). Critics of the whole-sample approach argue that it risks overfitting and circular reasoning when CFA is used strictly as a confirmatory tool (Flora & Flake, 2017; Fokkema & Greiff, 2017; Williams & Hanna, 2023). To mitigate this, they advocate cross-validation through splitting the sample, with EFA conducted on one half and CFA on the other (De Rooij & Weeda, 2020; Kline, 2016; Lorenzo-Seva, 2021). Others, however, caution that the split-sample approach is only feasible with very large samples and may reduce precision, increase standard errors, raise the likelihood of non-convergence, and yield conflicting EFA and CFA results that are difficult to interpret (Zhou, 2022).

These challenges are increasingly relevant given the growing reliance on online data collection, where lower response rates often constrain achievable sample sizes (Wu et al., 2022). Against this backdrop, two related but distinct problems must be separated. The first is absolute sample-size adequacy: whether a dataset contains enough information to recover stable factor structures and estimate CFA parameters with acceptable precision. The second is sample partitioning: whether dividing an otherwise available dataset into exploratory and confirmatory halves reduces each analytic sample below the level needed for stable estimation. The present study focuses on this second question, conditional on the first: in realistic scale-development contexts, when does splitting a sample provide meaningful cross-validation, and when does it simply create underpowered EFA and CFA subsamples?

Therefore, the contribution of this study is not to show that small samples are unstable, which is well established, but to quantify how whole-sample, bootstrap, and repeated split-sample strategies perform across a range of sample-size conditions and to translate these findings into practical guidance on when split-sample validation is, and is not, warranted.

Methods

Study Rationale and Design Overview

To evaluate the impact of sample size and analytic strategy on factor analysis outcomes, we employed simulation and resampling techniques. These approaches allow systematic testing of methodological scenarios under controlled conditions, offering insights into the stability, replicability, and accuracy of EFA and CFA results across varying sample sizes (Beaujean, 2018; De Bin et al., 2016).

Resampling methods, such as m-out-of-n bootstrapping, enable assessment of the precision and variability of model parameter estimates and fit indices by repeatedly sampling with replacement from the original dataset (Chernick, 2008a; De Bin et al., 2016; Mammen & Nandi, 2012). This provides insight into the robustness and convergence of factor solutions. In addition, repeated random subsampling, where the dataset is partitioned into mutually exclusive halves without replacement, mimics cross-validation and facilitates evaluation of generalisability across split-sample designs (Chernick, 2008a).

Based on the above rationale, to test methodological outcomes, we generated eight different original datasets with sample sizes of 150, 220, 300, 450, 600, 750, 1,000, and 1,500. These sample sizes were selected to provide sufficient variability for subsequent analyses, including resampling and factor model evaluation. For each original dataset, we applied both resampling strategies. The bootstrap procedure involved generating 300 replicates per sample size to assess the stability of model fit statistics. Concurrently, we created 300 pairs of randomly split-half samples per dataset to independently conduct EFA and CFA. An overview of the study procedure is presented in Figure 1. Together, these steps enabled a rigorous comparison of whole-sample and split-sample strategies across realistic indicative sample sizes, supporting empirically grounded recommendations for psychometric scale development.

Figure 1.

Workflow for data simulation, dataset creation, and factor analysis procedures. The diagram outlines: Step 1: simulation of synthetic datasets with three correlated latent factors and 24 observed items and generation of whole, bootstrap, and split-sample datasets across eight sample sizes; Step 2: Exploratory factor analysis (EFA) with specified retention and loading criteria, and extraction of factor solutions; and Step 3: Confirmatory factor analysis (CFA) with model fit evaluation using multiple indices (CFI, TLI, RMSEA, SRMR, etc.). *Factors 2 and 3 are directly correlated with Factor 1. In addition, Factor 2 is indirectly correlated with Factor 3 through their shared association with Factor 1

Synthetic Data Generation

Dataset Structure and Initialisation

Using IBM SPSS (Version 29.0.2.0), we simulated eight datasets comprising n cases (n = 150, 220, 300, 450, 600, 750, 1,000, and 1,500) to represent three latent factors and their observed indicators (8 items per factor). The simulation followed a stepwise process: (1) defining population parameters, (2) generating latent variables, and (3) producing observed variables with controlled statistical properties. Each case was assigned a unique identifier ( $I{D}_i=i$ for $i=1,\dots ,\mathrm{n}$ ), and a fixed random seed was used to ensure reproducibility. The same procedure was repeated identically for all the sample sizes.

Each latent factor, Factor 1, Factor 2, and Factor 3, was represented by exactly eight observed items, totalling 24 observed variables per dataset. The item generation procedure was identical across all factors and items, systematically applied to simulate realistic measurement variability.

The latent factors were correlated such that Factors 2 and 3 each correlated with Factor 1. These correlations, denoted $\rho$ , were uniformly drawn between 0.1 and 0.9, inducing an indirect correlation between Factors 2 and 3 through their mutual association with Factor 1. This structure reflects plausible correlated factor models common in psychometric constructs.

Detailed description of factor generation is provided under “Synthetic Data Generation”.

Generation of Factor 1 and Indicators

For each original datasets with a sample size of n (150, 220, 300, 450, 600, 750, 1,000, and 1,500), population parameters for Factor 1 were defined as follows (Stefancova et al., 2024):

• • The population mean, ${\mu }_1$ , was drawn from a uniform distribution:

$${\mu }_1\sim \text{Uniform}\hspace{0.17em}\left(7.5,12.5\right)$$ (1)

• • The population standard deviation, ${\sigma }_1$ , was drawn from a uniform distribution scaled by ${\mu }_1$ :

$${\sigma }_1\sim \text{Uniform}\left(\frac{{\mu }_1}{10},\frac{{\mu }_1}{5}\right)$$ (2)

• • Individual latent scores for Factor 1, denoted $X_{1i}$ (for $i=1,\dots ,\mathrm{n}$ ), were generated as independent draws from a normal distribution with a mean of ${\mu }_1$ and a variance of ${\sigma }_1^2$ :

$$X_{1i}\sim \mathcal{N}\left({\mu }_1,{\sigma }_1^2\right)$$ (3)

• • To facilitate the construction of correlated variables, Factor 1 scores were standardised:

$$Z_{1i}=\frac{X_{1i}-{\mu }_1}{{\sigma }_1}$$ (4)

where $Z_{1i}$ has a mean of 0 and a standard deviation of 1.

• • Observed items representing Factor 1 were generated in eight iterations per case. For each item j = 1, …, 8, indicator-specific means and standard deviations μ₁^(j), σ₁^(j) were drawn analogous to the latent factor parameters. The corresponding item scores (observed variables $X_{1i}^{\left(j\right)}$ ) were constructed by rescaling the standardised latent scores:

$$X_{1i}^{\left(j\right)}={\mu }_{1i}^{\left(j\right)}+{\sigma }_{1i}^{\left(j\right)}\hspace{0.17em}·\hspace{0.17em}{Z}_{1i}^{\left(j\right)}$$ (5)

This produced eight observed variables representing Factor 1 with controlled mean and standard deviation.

Generation of Factor 2 With Controlled Correlation

Population parameters for Factor 2 were similarly defined, with the population mean ${\mu }_2$ drawn using formula (1), and the population standard deviation ${\sigma }_2$ drawn using formula (2).

The correlation coefficient $\rho$ between Factor 1 and Factor 2 was randomly selected from a uniform distribution, constrained between 0.1 and 0.9:

$$\rho \sim \text{Uniform}\left(0.1,0.9\right)$$ (6)

For each case, latent Factor 2 standardised scores $Z_{2i}$ were generated to have the specified correlation $\rho$ with the standardised Factor 1 scores $Z_{1i}$ using the formula for correlated normal variables:

$$Z_{2i}=\rho Z_{1i}+\sqrt{1-{\rho }^2}·{\epsilon }_i,{\hspace{0.5em}\epsilon }_i\sim \mathcal{N}\left(0,1\right)$$ (7)

This ensures that $Z_2$ shares variance with $Z_1$ according to $\rho$ , while the residual term ${\epsilon }_i$ adds independent noise. The latent Factor 2 scores were then rescaled to the original metric:

$$X_{2i}={\mu }_2+{\sigma }_2·Z_{2i}$$ (8)

Observed items for Factor 2 were generated independently for eight indicators ( $X_{2i}^{\left(j\right)}$ ):

$$Z_{2i}^{\left(j\right)}={\rho }^{\left(j\right)}{Z}_{1i}+\sqrt{1-{\left({\rho }^{\left(j\right)}\right)}^2}·{\epsilon }_i^{\left(j\right)},{\hspace{1em}\epsilon }_i^{\left(j\right)}\sim \mathcal{N}\left(0,1\right)$$ (9)

$$X_{2i}^{\left(j\right)}={\mu }_2^{\left(j\right)}+{\sigma }_2^{\left(j\right)}·Z_{2i}^{\left(j\right)},j=1,\dots ,8$$ (10)

with indicator-specific parameters ${\mu }_2^{\left(j\right)},{\sigma }_2^{\left(j\right)},{\rho }^{\left(j\right)}$ drawn independently to simulate item heterogeneity.

Generation of Factor 3 and Indicators

Latent Factor 3 was generated independently by repeating the same procedure described for Factor 2, producing its own set of eight observed indicators with controlled correlations to Factor 1.

Datasets

For each synthetic dataset size (n = 150, 220, 300, 450, 600, 750, 1,000, and 1,500), data generation and latent factor simulation were conducted individually, following the principles described under “Synthetic Data Generation”. Additionally, m-out-of-n bootstrapping and repeated random sampling were each performed 300 times for each of synthetic datasets. Ultimately, the following datasets were created:

• • Whole datasets (DATAn):

Each DATAn dataset contained exactly n observations, generated according to the procedure detailed previously, with similar latent factor structures.

• • Bootstrap datasets (DATAn_bootstrap1 to DATAn_bootstrap300):

For each whole dataset, 300 bootstrap subsamples were generated. Each bootstrap dataset contained approximately 63% of the original sample size (i.e., m = 0.632n) and was resampled with replacement to simulate sampling variability.

• • Split-sample datasets:

For each whole dataset, 300 split subsample pairs were generated by randomly partitioning the original dataset into two mutually exclusive halves without replacement to mimic validation approaches:

• - The exploratory half labelled DATAn_split_E1 to DATAn_split_E300

• - The confirmatory half labelled DATAn_split_C1 to DATAn_split_C300

Each half contained approximately 50% of the observations in the original dataset.

All these datasets were collated in a master directory to streamline subsequent EFA and CFA analyses.

Exploratory Factor Analysis

Data Preparation and Screening

For each dataset, analyses focused on items linked to three pre-specified theoretical factors, comprising eight items each (F1.1–F1.8, F2.1–F2.8, F3.1–F3.8). Complete cases were identified to ensure valid input for factor extraction. Sampling adequacy was assessed by computing the Kaiser-Meyer-Olkin (KMO) measure and conducting Bartlett’s test of sphericity, confirming suitability for factor analysis.

Factor Extraction and Retention Criteria

Following principles provided by Hair et al. (2019b) and Williams and Hanna (2023), using R (Version 4.6.1; R Core Team, 2026), factors were extracted using maximum likelihood estimation to permit inferential tests and support model fit evaluation. Oblique (oblimin) rotation was applied to allow for correlations between factors. After examining scree plots and initial factor structures across different commonly applied eigenvalue thresholds (1.0, 1.2, and 1.5), we selected an eigenvalue cutoff of ≥1.5 to determine the number of retained factors for each dataset. While this threshold is more conservative than the Kaiser criterion (i.e., eigenvalue of 1.0) (Hair et al., 2019b), it was chosen to reduce the occurrence of factors composed of a single item, which were prevalent at lower eigenvalue cutoffs. Specifically, of the 4,808 datasets generated for EFA, 4,250 retained at least one single-item factor at ≥1.0, 2,234 did so at ≥1.2, and 473 did so at ≥1.5. Selecting a threshold of ≥1.5 therefore improved factor interpretability and model stability.

Items were retained on a factor if their absolute loading was ≥0.4. To ensure factor purity, items exhibiting cross-loadings were excluded if the difference between their highest and second-highest loadings was less than 0.2, or if an item loaded significantly (factor loading ≥0.4) on multiple factors.

Furthermore, factors composed of fewer than two items were flagged, as at least two indicators per factor are required for identification in CFA models with two or more factors (Kline, 2016). Consequently, any dataset where EFA produced factors with only one item was excluded from subsequent analyses.

Factor Relabelling and Reliability Assessment

Retained factors were systematically relabelled based on item composition: factors comprising only items F1.1 to F1.8 were designated as Factor A; those with items F2.1 to F2.8 as Factor B; and those with items F3.1 to F3.8 as Factor C. Factors containing items from multiple theoretical groups were labelled Factor Mix (see R syntax in Supplemental Material, for detailed information). Internal consistency (Cronbach’s alpha) was calculated for factors without single-item retention. Summary results across all datasets were aggregated and exported as a single output file.

Extraction of Shared Patterns

To identify recurring factor structures across datasets, first we excluded datasets where EFA produced any single-item factors to ensure factor stability. For each remaining dataset, the retained items within each relabelled factor were sorted alphabetically and concatenated to generate a unique pattern signature summarising each factor’s composition. All factor signatures for a dataset were combined into an overall signature representing its pattern of retained factors (see R syntax in Supplemental Material, for detailed information).

Datasets sharing identical number of factors and overall pattern signatures were grouped accordingly. These groups represent datasets exhibiting matching latent structures as revealed by EFA, regardless of sample size or resampling approach.

Confirmatory Factor Analysis

Model Specification and Estimation

Using R (Version 4.6.1; R Core Team, 2026), CFA was conducted to evaluate the factor structures identified in the preceding EFA across all datasets and their bootstrap and split-sample replications. For each dataset, CFA models were specified using the lavaan R package (Version 0.6–19), with factor-item relationships derived directly from prior EFA results. Factors containing fewer than two items were excluded from model specification to maintain model stability (Kline, 2016). Models were estimated using robust maximum likelihood with an oblique factor structure (see R syntax in Supplemental Material).

Method-Specific CFA Model Strategies

• Whole-Sample and Bootstrap CFA

For the whole-sample strategy, CFA model specifications for the original datasets (DATAn) were based on the factor solutions derived exclusively from the original dataset’s EFA (see Figure 1). Bootstrapped datasets (DATAn_bootstrap1 to DATAn_bootstrap300) were subjected to CFA using the same factor-item structure as their respective original dataset.

• Split-Sample CFA

For the split-sample strategy, each original dataset was partitioned into two halves: an exploratory sample (“Split E”) and a confirmatory sample (“Split C”) (see “Datasets” section and Figure 1). CFA models were constructed using EFA results from the “Split E” datasets and then tested exclusively on the paired “Split C” datasets. This approach enabled cross-validation of factor structures identified in one data partition by testing them in an independent partition, corresponding to real-world split-method strategy (De Rooij & Weeda, 2020).

Model Fit Evaluation and Confidence Interval Estimation

Model fit was assessed using the following indices: scaled chi-square statistic (CMIN), degrees of freedom (DF), comparative fit index (CFI), Tucker-Lewis index (TLI), root mean square error of approximation (RMSEA, with 90% confidence interval and p-value for close fit), and standardised root mean square residual (SRMR). For each main dataset, the stability of model fit was empirically estimated by calculating the mean and 95% confidence intervals for each index across bootstrap and split-sample replications. Confidence intervals were computed using the t-distribution:

$$C{I}_{95\%}=\overline{x}\pm t_{\frac{\mathrm{\alpha }}{2},n-1}\times \frac{s}{\sqrt{n}}$$ (11)

where $\overline{x}$ is the mean, s is the standard deviation, and n is the number of bootstrap or split replications retained for the analysis. This procedure provided an empirical estimate of the sampling variability in model fit statistics, facilitating robust interpretation of CFA results across resampled data subsets.

Visualisation of CFA Results

Model fit indices from CFAs were summarised and visualised to assess trends across sample sizes and strategies. CFA results from all datasets, including original whole-samples, bootstrap replications, and split-validation samples, were plotted on the same axes. For each fit index (CMIN, TLI, CFI, RMSEA, SRMR), values were mapped against the original sample size (n) for each method, enabling continuous assessment of fit stability as sample size increased.

Computation Methods

All computations for dataset generation, including generating the original datasets, bootstrap datasets, and split datasets, were performed using IBM SPSS (Version 29.0.2.0) with a fixed random seed to ensure replicability. The process involved generating uniform and normal random variables, standardising latent variables, and constructing correlated variables through linear combinations of standardised scores and independent normal noise.

EFA and CFA were conducted in R (Version 4.6.1; R Core Team, 2026). Sampling adequacy was assessed using the KMO() and cortest.bartlett() functions from the psych package (version 2.6.5; Revelle, 2026), which was also used for EFA and internal consistency testing. Specifically, EFA was performed using the fa() function with maximum likelihood estimation and an oblique (oblimin) rotation. Internal consistency of the retained multi-item factors was subsequently evaluated using the alpha() function. CFA was conducted using the cfa() function from the lavaan package (version 0.6–20; Rosseel et al., 2025), with robust maximum likelihood estimation and an oblique factor structure (see R syntax in Supplemental Material). In the development of the R syntax, authors were assisted by Perplexity AI (2025; free-tier subscription) (Jiang et al., 2024). The first author refined and fully verified all code for accuracy and reproducibility.

Results

Overview of Datasets and EFA Patterns

A total of 4,808 datasets were generated for this study. These comprised eight original datasets (one per sample size), and for each sample size, 300 bootstrap datasets (each of size 0.632n) and 300 split-half datasets. For the split-half condition, each of the 300 datasets was partitioned into two halves of size 0.5n for analysis; however, each split pair was counted once towards the total (see Figure 1).

EFA was conducted on each one of the 4,808 datasets: eight original datasets, 2,400 bootstrap datasets, and 2,400 split-E datasets. Across all datasets, EFA retained at least one single-item factor for 473 datasets, indicating unstable factor solutions. Among the 4,335 EFA solutions retained after excluding single-item-factor solutions, 331 distinct factor-pattern groups were observed; 177 of these were singleton patterns each occurring in only one dataset. The rest of the datasets exhibited factor patterns matching at least one other dataset.

Factor-pattern recovery varied substantially across sample sizes and strategies. Notably, for the dataset with the largest original sample size (DATA1500), the EFA factor pattern showed exceptional stability: nearly all corresponding bootstrap datasets (296 out of 300) and split exploratory halves (283 out of 300) exhibited patterns identical to the original (Figure 2). This pattern stability generally diminished as sample size decreased, although not in a strictly linear manner (Figure 2). The next most consistent factor patterns were observed for DATA600 in both bootstrap and split-E datasets (Figure 2). It is noteworthy that no two original whole datasets (DATAn) from different sample sizes shared identical factor patterns.

Figure 2.

Frequency of bootstrap and split datasets exhibiting identical factor patterns to their original datasets

CFA Results

Using the factor structures derived from EFA on the whole datasets, CFAs were conducted on the corresponding original datasets under the whole-sample strategy. All fit indices met commonly accepted criteria for good model fit (CMIN/DF < 3, CFI and TLI >0.9, SRMR <0.08, and RMSEA <0.08, with no 90% CI crossing 0) across all sample sizes (see Table 1 and Table S1 in Supplemental Material). However, some variation in fit indices was observed across the original datasets with differing sample sizes. In particular, SRMR showed a decreasing trend from the smallest sample size (n = 150) to the largest sample (n = 1,500), indicating improved model fit with larger samples under the whole-method strategy (see Table 1 and Figure 3).

Table 1.

SRMR values across sampling strategies and sample sizes

Dataset	Whole-sample strategy	Bootstrap strategy a				Split-sample strategy
		Mean b	SD	95% CI		Mean b	SD	95% CI
				Lower	Upper			Lower	Upper
DATA150	0.053	0.0663	0.0045	0.0658	0.0668	0.0764	0.0082	0.0748	0.078
DATA220	0.044	0.0536	0.0044	0.0531	0.0542	0.0595	0.0083	0.0583	0.0606
DATA300	0.041	0.0488	0.0043	0.0483	0.0493	0.0539	0.0066	0.0531	0.0547
DATA450	0.033	0.0403	0.0031	0.0399	0.0406	0.0451	0.0046	0.0446	0.0456
DATA600	0.026	0.0319	0.0023	0.0316	0.0321	0.0362	0.0034	0.0358	0.0366
DATA750	0.024	0.0289	0.0024	0.0286	0.0292	0.0339	0.0031	0.0335	0.0342
DATA1000	0.02	0.0243	0.0022	0.024	0.0246	0.0271	0.0031	0.0268	0.0275
DATA1500	0.016	0.0202	0.0014	0.0201	0.0204	0.0232	0.0018	0.023	0.0234

^aIn alignment with m-out-of-n bootstrapping, each bootstrap dataset contained approximately 63% of the original sample size (m = 0.632n).

^bEach split dataset contained 50% of the original sample.

CI, confidence interval; SD, standard deviation; SRMR, standardised root mean square residual.

Figure 3.

Trends of standardised root mean square residual (SRMR) across sample sizes and sample strategies

In line with the study aims, comparisons between the single whole-sample analyses and replicated bootstrap datasets were conducted by applying the model structure derived from each of the eight original datasets to its corresponding bootstrap datasets. As the EFAs on the original datasets (using eigenvalue ≥1.5) did not produce any single-item factors, CFAs were conducted on all bootstrap datasets (300 per sample size). Similar trends in fit indices to those observed for the original datasets were evident across bootstrap replications (see Figure 3 and Table S2 in Supplemental Material).

For the split-sample strategy, factor structures derived from EFA on the Split E datasets were applied to their corresponding Split C datasets (see “Method-Specific CFA Model Strategies”). CFAs were conducted only for split datasets whose EFA solutions did not contain single-item factors (see Table S2 in Supplemental Material for detailed counts). As with the whole-sample strategy, fit indices consistently met acceptable thresholds for model fit. However, these indices typically suggested relatively reduced fit compared to the whole-sample and bootstrap strategies (see Table 1, Figure 3, and Figures S1–S4 and Tables S2 in Supplemental Material).

Discussion

This simulation study systematically evaluated how sample size and sampling strategy affect the consistency, model fit, and reliability of psychometric scales. Our results indicate that, when developing a new psychometric scale without an a priori established model and under real-world sample size constraints, applying both EFA and CFA to the whole data set (i.e., whole-sample strategy) is generally preferable, unless minimum sample size guidelines are clearly met for a split-sample approach (these will be introduced later, under the heading “When Is Split-Sample Validation Useful in Factor Analysis for Scale Development?”). In these cases, CFA should serve as a complementary step for model evaluation, rather than a strictly confirmatory procedure.

Considering the EFA results, an important observation is that recovery of the original factor structure was markedly compromised when smaller samples were halved (see Figure 2). By contrast, the bootstrap datasets (each approximately 63% of the corresponding original sample) consistently showed higher recovery rates than the split datasets. This finding aligns with previous evidence (Jung, 2013; McNeish & Wolf, 2023) suggesting that split-sample strategies can be precarious for small datasets. In these cases, sampling idiosyncrasies may lead factor structures to diverge across subsamples, such that CFA fit indices computed on one half provide limited evidence about the EFA structure derived from the other. This is particularly consequential during early scale development, and underscores the importance of evaluating factor structure replicability alongside global fit indices.

Based on the CFA results, unsurprisingly, we observed that smaller sample sizes produced diminished model fit and reduced solution stability. Notably, the split-sample strategy consistently resulted in relatively lower-quality fit indices compared to the whole-sample strategy, with these discrepancies being most pronounced at the lower range of tested sample sizes. This aligns with existing literature, which recognises that factor patterns derived from larger samples are generally more stable than those from smaller samples (DeVellis & Thorpe, 2022). Our simulations confirmed this, where both the original datasets and their resampled counterparts (including bootstraps and split samples) exhibited a decline in model fit as sample size decreased, although fit indices remained within accepted thresholds (see Figure 3). Despite this general agreement on the importance of sample size, there is no universal consensus on what constitutes an “adequate” sample (Costello & Osborne, 2005; DeVellis & Thorpe, 2022; Hair et al., 2019b; Rouquette & Falissard, 2011).

Multiple rules of thumb have been proposed to guide minimum sample size determination in factor analysis. Some researchers recommend a minimum of 50–100 observations (Hair et al., 2019b), while others advocate 300 participants as a general rule, regardless of the number of items in a scale (Rouquette & Falissard, 2011). Another widely cited heuristic is the ratio of participants to items (N:q rule) (Jackson, 2003), often set at 5:1 to 10:1 (DeVellis & Thorpe, 2022; Hair et al., 2019b). However, some academics caution against relying on this exclusively, especially for short scales (Rouquette & Falissard, 2011). Statistical power and anticipated factor loadings also offer practical guidance: achieving 80% power at α = 0.05 requires at least 350 cases for a loading of 0.3, or 200 for a loading of 0.4 (Hair et al., 2019b).

Within our data, all sample sizes yielded fit indices within conventional thresholds (including SRMR <0.08), but important differences persisted. Specifically, SRMR showed a marked improvement between 150 and 220 cases (see Figure 3), approximately aligning with a 10:1 case-to-item threshold (i.e., 240 observations for 24 items in our simulated data) and supporting a 200-observation requirement for a factor loading of 0.4 to be significant.

Beyond sample size, the choice of sampling strategy for EFA and CFA is equally crucial. Focusing on the issue of overfitting, some statisticians discourage using CFA to confirm the factor structure derived from the same sample by EFA (Flora & Flake, 2017; Fokkema & Greiff, 2017; Williams & Hanna, 2023), recommending instead the split-sample strategy (De Rooij & Weeda, 2020; Kline, 2016; Lorenzo-Seva, 2021). However, this recommendation often overlooks the impact of reduced sample size on statistical power, the risk of Type II error, and the common practical difficulties of acquiring larger samples. Our findings directly illustrate this trade-off: splitting the data typically led to less stable fit, particularly below n = 450 (see Table 1 and Tables S1–S2 in Supplemental Material). At n = 150, for example, split-sample fits approached cutoff thresholds (TLI = 0.971 for split vs. 0.986 for bootstrap; SRMR = 0.076 vs. 0.066), signifying less robust solutions. Only as sample size increased (n ≥ 450) did the differences between split-sample, whole-sample, and bootstrap analyses diminish, and model stability substantially improved. This suggests that the split-sample approach becomes practical only when each half meets or exceeds established minimum size recommendations.

Importantly, across most sample sizes, fewer than 50% of resampled datasets (split or bootstrapped) recovered the same factor structure as the original (see Figure 2). This finding underscores that random subsamples, whether derived through splitting or m-out-of-n bootstrapping, are often not fully representative of the original data, thereby undermining the generalisability and stability of the recovered factors (Zhou, 2022).

When Is Split-Sample Validation Useful in Factor Analysis for Scale Development?

The decision to utilise a split-sample validation strategy requires weighing the theoretical benefit of independent cross-validation against the severe statistical costs of data partitioning (De Rooij & Weeda, 2020; Fokkema & Greiff, 2017; Maier et al., 2023; Zhou, 2022). In principle, splitting a dataset provides a test of model generalisability; it ensures that a factor structure discovered in an exploratory half can independently hold in a confirmatory half without capitalising on sample-specific noise (De Rooij & Weeda, 2020; Faraway, 2016). However, the main lesson from the present study is that split-sample validation should not be regarded as inherently superior to a whole-sample strategy. Its value depends on whether each resulting subsample remains large enough to recover a stable exploratory structure and support reliable confirmatory estimation. Applying a single random split to a moderately sized dataset imposes a severe 50% penalty on the available information. This brutal shrinkage artificially inflates standard errors and drastically elevates the risk of Type II errors, wherein true structural relationships or significant factor loadings go undetected simply due to a lack of power (Baldwin et al., 2022). Furthermore, relying on a single random split introduces an element of arbitrariness; the resulting structural stability becomes heavily dependent on the specific data partition rather than the underlying construct (Lorenzo-Seva, 2021). Under these conditions, poor split-sample performance reflects the combined effects of insufficient sample size and information loss due to partitioning, rather than the act of splitting itself or the underlying construct (Kline, 2016; Kyriazos, 2018; Williams & Hanna, 2023).

In the current simulation, split-E datasets were less likely than bootstrap datasets to recover the original factor pattern at smaller sample sizes (Figure 2), and split-C CFA fit was generally weaker and more variable than fit obtained from whole-sample or bootstrap-based strategies (Figure 3). These differences diminished as sample size increased. This indicates that split-sample validation is most interpretable when the exploratory half can reproduce a stable factor structure. Thus, while split-sample validation may be regarded as the preferred approach for EFA and CFA on independent subsamples, it should not come at the cost of compromising the robustness of either analysis. As such, the practical recommendation is that the split-sample validation should be reserved for situations where both the exploratory and confirmatory halves independently satisfy minimum adequacy conditions.

For new scale development with constrained sample sizes, a more defensible strategy may be to use the whole sample for model development and supplement this with bootstrap or repeated-resampling checks of factor-pattern stability. By repeatedly resampling from the full dataset, bootstrapping circumvents the devastating power reduction of a split, allowing researchers to evaluate the consistency of the factor structure without artificially inducing Type II errors (De Bin et al., 2016; Mammen & Nandi, 2012). When larger samples are available, however, split-sample validation is more appropriate because the confirmatory half can provide a stronger test of generalisability without rendering either analysis underpowered.

Practical Recommendations

Based on these findings and supporting literature, we propose a hierarchical set of guidelines for determining sample size and analytic strategy (whole-sample vs. split-sample strategies) in EFA and CFA for scale development (see Figure 4):

Figure 4.

Decision tree for sample size and validation strategy in factor analysis. This decision tree summarises recommended steps for determining sample size and selecting validation strategies for EFA and CFA. The hierarchy begins with ensuring a bare minimum of 200 observations^* for both EFA and CFA, followed by applying the N:q rule (at least 10:1 cases-to-items, ideally 20:1). If split-sample validation is considered, each half must independently meet these thresholds; otherwise, whole-sample or bootstrap approaches are preferred. n represents the sample size, m represents the size of the split sample (50% of n), and q represents the number of items included in the scale. CFA, confirmatory factor analysis; EFA, exploratory factor analysis; FL, factor loading. *Based on guidelines provided by Hair et al. (2019b), to achieve 80% power at a .05 significance level, the required sample size is approximately 200 for an FL of 0.40, 250 for an FL of 0.35, and 350 for an FL of 0.30

Decision 1: Bare Minimum

For newly developed scales, at least 200 observations should be included in both EFA and CFA, regardless of the number of items. This aligns with the traditional recommendations of Comrey (1988).

Decision 2: The N:q Rule

Sample size should also conform to a case-to-item ratio; our data endorse a 10:1 minimum (Tinsley & Tinsley, 1987). For scales with more than 20 items, this means raising the sample above 200. Ratios of 20:1 are preferable for greater stability and reproducibility (Costello & Osborne, 2005; Jackson, 2003).

Decision 3: Whole-Sample vs. Split-Sample Strategies

Split-sample validation (i.e., conducting EFA and CFA on separate halves) should only be used when each subsample independently meets minimum recommended sizes. Otherwise, splitting creates underpowered datasets, impairing model fit, precision, and generalisability (Chernick, 2008b; Faraway, 2016; Kyriazos, 2018; Zhou, 2022).

In practice, our simulations showed all strategies yielded at least acceptable fit, but whole-sample and bootstrap methods consistently outperformed split-sample analysis when the original sample was under 450 (see Figure 3 and Table S2 in Supplemental Material), where split halves failed to meet thresholds.

These empirically derived recommendations are especially relevant for new psychometric scale development, where an a priori model is absent and CFA is best viewed as an exploratory supplement to EFA (Williams & Hanna, 2023). However, whenever revisions are made to a model on a given dataset, subsequent confirmation should use new data; any model refined on the original sample will necessarily “fit” it well (Williams & Hanna, 2023). It is also important to emphasise that our suggested thresholds and decision-making framework should be regarded as the minimum necessary to achieve stable factor structures. Wherever possible, researchers should supplement these general thresholds with model- and design-specific approaches to sample size planning. Precision-based frameworks, including methods that estimate the sample size required to accurately recover factor loadings, factor scores, or structural parameters, provide a more principled basis for planning than fixed-N or N:q heuristics alone (Ferrando & Lorenzo-Seva, 2018; Lorenzo-Seva & Ferrando, 2024).

Beyond these strategies, our results show that bootstrapped or split-sample datasets often failed to recreate the original factor structure. For assessing model stability, we therefore recommend alternative strategies. One such approach is multiple group invariance analysis (e.g. multisample CFA, also known as MCFA), which evaluates measurement stability across real groups (such as gender, age, or cultural background) rather than relying on random data splits (Hair et al., 2019a). This type of analysis is particularly valuable for ensuring that observed differences between groups reflect genuine variations in the underlying construct rather than artefacts of the measurement tool (Byrne, 2016; Hair et al., 2019a). Unlike cross-validation through random sample splitting, multiple group invariance analysis is grounded in existing population groups, making it inherently more representative (Byrne, 2016; Hair et al., 2019a). This approach, while still requiring adequate and balanced sample sizes per group, avoids the substantial loss of power and precision associated with splitting a single dataset (Zhou, 2022).

Researchers should also recognise that factor-pattern stability depends not only on sample size and sampling strategy, but also on the number of items and factors and the magnitude of factor loadings (Guadagnoli & Velicer, 1988; Williams & Hanna, 2023). Consistent with this literature, our results show that stricter factor-retention decisions can materially improve solution stability: using an eigenvalue cutoff of ≥1.0 yielded single-item factors in 4,250 datasets, whereas a cutoff of ≥1.5 reduced this to 473, yielding more interpretable and stable solutions. More generally, weaker primary loadings are likely to require larger samples to achieve comparable stability, consistent with the recommendations in Figure 4. In CFA, sample-size requirements are additionally shaped by model complexity, indicator quality (e.g. reliability and loading magnitude) and power to detect misfit (Gagne & Hancock, 2006). These considerations should be weighed alongside sampling strategy when planning EFA–CFA validation studies.

A further consideration, particularly in applied psychological and educational research, is that missing data and non-normality can reduce usable information and, in turn, weaken the practical N:q ratio implied by Figure 4. Where missingness is present, listwise deletion should generally be avoided because it can substantially reduce the effective sample size. Instead, researchers are encouraged to consider modern approaches such as Full Information Maximum Likelihood (FIML) or Multiple Imputation (MI), which typically preserve power under Missing at Random assumptions and help maintain the intended N:q ratio (Hair et al., 2019b). For non-normal or ordinal item responses, researchers may also need to increase sample size beyond the nominal N:q targets and/or use robust estimators (e.g. maximum likelihood with robust standard errors and scaled test statistics). These adjustments help mitigate inflated fit statistics and improve the stability of factor solutions in applied settings (Hair et al., 2019b; Williams & Hanna, 2023).

Strengths and Limitations

Our study contains some of the largest simulations of their kind and therefore provides some of the most empirically grounded guidance for addressing key methodological decisions associated with psychometric scale development. By applying bootstrapping and multiple random sampling to generate numerous datasets from each original sample, rather than relying on a single split, we were able to make more robust comparisons between whole-sample and split-sample strategies. Furthermore, examining analyses across eight different sample sizes enhanced the generalisability of our interpretations.

Nonetheless, the study has limitations inherent to simulation designs (Chernick, 2008b). Specifically, we employed a relatively conservative eigenvalue threshold of 1.5 (rather than the more common 1.0) to avoid spurious factors comprised of single items (Hair et al., 2019b). We also adopted a stricter cutoff of 0.4 for factor loadings (instead of 0.3), which likely contributed to the consistently strong fit indices observed across conditions. While these decisions enhanced the clarity and interpretability of retained factors, they may not capture the full range of scenarios encountered in empirical research.

Moreover, although simulated data were patterned as closely as possible on real-world scenarios, no simulation can fully replicate the complexities of empirical data or account for all types of construct variance. In particular, many psychological constructs involve higher-order structures or bifactor models, which may require more complex model specifications and can be associated with weaker primary loadings than the three-factor structure simulated here.

It should also be noted that the present simulation uses continuous indicators and linear factor analysis models. Many applied psychological and educational scales, however, use ordinal response categories. In such settings, polychoric correlations and categorical estimators might be used instead of Pearson correlations for factor analysis. These methods are generally more sensitive to sample size and often require substantially larger samples to achieve stable solutions (Holgado–Tello et al., 2010).

Considering these points, our recommended sample sizes, analytic strategies, and the decision framework should be regarded as guiding principles rather than rigid rules, with decisions ultimately informed by theory, measurement context, and empirical conditions.

Conclusion

Our findings highlight the need for caution when using split-sample strategies, particularly in new scale development, in the absence of a well-established measurement model, or when working with moderate sample sizes. Based on our simulations, we recommend a minimum of 200 participants, with at least a 10:1, ideally 20:1, case-to-item ratio to maximise replicability. Split-sample cross-validation should only be considered for datasets where each subsample independently meets these minimum requirements; otherwise, whole-sample or bootstrap approaches are preferable. Where possible, model stability is best evaluated through alternative strategies such as multigroup invariance analysis or external validation with new samples.

Taken together, these recommendations provide empirically grounded guidance for researchers navigating methodological decisions in factor analysis and psychometric scale development. Adherence to these empirically supported principles will enhance both the credibility and generalisability of new measurement instruments in psychoeducational assessment.

ORCID iDs

Arash Arianpoor https://orcid.org/0000-0001-5646-2333

Craig S. Webster https://orcid.org/0000-0002-6997-4263

Silas C. R. Taylor https://orcid.org/0000-0003-1992-8485

Boaz Shulruf https://orcid.org/0000-0003-3644-727X

Pin-Hsiang Huang https://orcid.org/0000-0002-6101-0080

Ethical Considerations

This article does not contain any studies with human or animal participants, and informed consent is not required.

Data Availability Statement

The datasets generated during and analysed during the current study are available from the corresponding author on reasonable request. The R syntax used during this study is included in this published article and its supplementary information files.*