Do we really measure what we think we are measuring?

Summary

Tests used in the empirical sciences are often (implicitly) assumed to be representative of a given research question in the sense that similar tests should lead to similar results. Here, we show that this assumption is not always valid. We illustrate our argument with the example of resting-state electroencephalogram (EEG). We used multiple analysis methods, contrary to typical EEG studies where one analysis method is used. We found, first, that many EEG features correlated significantly with cognitive tasks. However, these EEG features correlated weakly with each other. Similarly, in a second analysis, we found that many EEG features were significantly different in older compared to younger participants. When we compared these EEG features pairwise, we did not find strong correlations. In addition, EEG features predicted cognitive tasks poorly as shown by cross-validated regression analysis. We discuss several explanations of these results.

Graphical abstract

Highlights

• •Many tests in science are assumed to measure the same phenomena
• •EEG features showing significant effects do not strongly correlate with each other
• •Cognitive tasks are only poorly predicted by EEG features
• •A significant result may tell less of a research question than believed

Cognitive neuroscience; Techniques in neuroscience.

Introduction

Representative paradigms with elaborated tests are crucial in all empirical sciences. In the brain sciences, neuroimaging methods are used to investigate mechanisms underlying cognition and perception. Typically, a link between a neuroimaging feature (e.g., brain volume, connectivity) and a cognitive function of interest is declared if the chosen test delivers a significant result (after accounting for confounding variables). There is often the implicit assumption that the significant neuroimaging feature is representative of the neural mechanism under investigation. Here, we explicitly tested this assumption with the example of resting-state electroencephalogram (EEG).

In resting-state studies, EEG is recorded for around 5 min during which participants do nothing else than rest quietly. Signal processing methods are applied to quantify spatial and/or temporal characteristics of spontaneous brain activity. The outcomes of the analysis methods, i.e., EEG features, are interpreted to reflect brain processes linked to certain aspects of perception and cognition. For example, activity in the alpha and theta bands has been linked to memory and executive functions,^1,2,3 alpha-band activity to visual perception,^4,5 temporal autocorrelations of alpha-band oscillations and EEG microstates dynamics to reaction times,^6,7 connectivity features and alpha activity to intellectual abilities,^8,9,10 just to give a few examples. Similarly, EEG features reveal abnormalities in patients with schizophrenia,^{11,12,13,14,15} depression,^16,17,18,19 and healthy older adults,^20,21,22,23 among others.

Each of these findings indicates a significant link between a given EEG feature and an aspect of cognition or a disease. In this respect, this approach has been very successful. Yet, these results provoke the questions of how the different EEG features relate to each other and how representative they are of the underlying mechanisms. For example, one might expect different EEG features, recorded from the same patients, to correlate with each other if they are supposed to point to the same aspect of the disease.

Here, we analyzed data from resting-state EEG recordings and a battery of cognitive tests. To obtain a comprehensive set of neurophysiological features, we applied widely used analysis methods to the same EEG data, including time-domain, frequency-domain, connectivity, and nonlinear dynamical analysis methods both in the electrode and source spaces. We extracted 175 EEG features. From the battery of cognitive tests, we obtained 12 cognitive variables describing several cognitive aspects. We correlated each EEG feature with each cognitive variable using methods that permitted us to examine linear and nonlinear relationships. Next, we correlated the features revealing significant correlations with the same cognitive variable using univariate and multivariate correlation methods. This comparison allowed us to investigate whether the features, showing a significant correlation with one of the cognitive variables, point to a common mechanism. In a second project, we conducted group comparisons between younger and older adults using each EEG feature. A significant group difference would indicate that the EEG features tap into important age-related changes in brain processing. To test whether the features showing group differences target common age-related aspects, we correlated the EEG features revealing significant group differences. Furthermore, we used principal component analysis to assess the latent dimensions of multiple EEG features showing significant correlations to a cognitive variable and significant group differences between younger and older adults. As a complementary analysis, we evaluated cross-validated regression models using each EEG feature to predict the cognitive variables. Importantly, we did not want to elaborate on any particular relationship between an EEG feature and cognitive ability or an EEG feature and aging. We were interested in how significant results from single analyses relate to each other.

Results

We analyzed data from the publicly available LEMON database.²⁴ This database includes resting-state EEG recordings and a battery of cognitive tasks. The sample used for the present study consisted of 201 participants, 138 younger adults (mean age = 25.43, SD = 3.39, 42 females) and 63 older adults (mean age = 67.66, SD = 4.79, 31 females). Using multiple analysis methods, we obtained 175 EEG features from the resting-state EEG recordings. The EEG features can be composed either of 61, 80, or 4 variables, corresponding to the number of electrodes, brain regions, or microstate parameters, respectively. From the battery of 6 cognitive tests, we obtained 12 cognitive variables. Details are shown in experimental model and subject details.

Correlations between EEG features and cognitive variables

We computed Spearman and distance correlations between each EEG feature and cognitive variable. Thus, for each age group and correlation type, we performed 2100 (175∗12) analyses. With this evaluation, we sought to identify the EEG features reflective of a neural process linked to each cognitive ability. Next, the EEG features showing a significant correlation to a cognitive variable were pairwise correlated either using Spearman or distance correlations and multivariate distance correlations. Strong correlations between EEG features would suggest that the features point to the same mechanism representative of the cognitive aspect under study. In younger adults, 109 analyses were significant using Spearman correlations and 121 using distance correlations (after correction for multiple comparisons for each pair of EEG feature and cognitive variable). For most cognitive variables, we found more than one EEG feature showing a significant correlation (Figure 1A). The correlations between these EEG features were weak in most of the cases. Similar results were found using multivariate distance correlations ($\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$), which permitted us to correlate EEG features considering all the variables, i.e., electrodes, brain regions, or microstate parameters (Figure 1B). Results for older adults were similar and are presented in Figure S1.

Figure 1.

Result of the correlation analysis in younger adults

(A) EEG features with significant correlations to cognitive variables. On the right side of the panel, we indicate the range (min-max) of the magnitudes of the significant correlations (see correlations between EEG features and cognitive variables in quantification and statistical analysis).

(B) Median (confidence interval: 25^th and 75^th percentiles) Spearman and distance correlations between the EEG features showing a significant correlation with the same cognitive variable. On the right side of the panel, we indicate the 25^th, 50^th, and 75^th percentiles of the multivariate distance correlations ($\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$; ranging from 0 to 1) between the EEG features (with all its variables) showing a significant correlation with the same cognitive variables.

For instance, for younger adults, both the life time statistics of the amplitude envelopes in the theta band (life time theta) and the node strength of delta connectivity measured in the electrode space using phase locking value (node str e-plv delta) correlated significantly with the working memory variable obtained from the test of attentional performance (Tap working memory; ρmax = −0.28 and ρmax = 0.29, respectively). However, life time theta and node str e-plv delta did not correlate strongly with each other (ρ = −0.11; $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}=$ 0.07; Figure 2). Similarly, using distance correlations, both the occurrence of microstate class C (microstate C) and the betweenness centrality of gamma connectivity measured in the electrode space using weighted phase lag index (betw cen e-wpli gamma) showed significant correlations with the module A variable of the trail making test (Tmt-A; ${\mathcal{R}}_{\mathcal{n}}$ max = 0.25 and ${\mathcal{R}}_{\mathcal{n}}$ max = 0.36, respectively). However, the two EEG features only weakly correlate with each other (${\mathcal{R}}_{\mathcal{n}}$ = 0.15; $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}=$ 0.01; Figure 3). ρmax and ${\mathcal{R}}_{\mathcal{n}}$ max denote the maximum significant Spearman or distance correlation (among all the electrodes, brain regions, or microstate parameters) of the EEG feature with the cognitive variable.

Figure 2.

Spearman correlations between the EEG features that correlated significantly with the Tap working memory variable in younger adults

The main diagonal has the ρmax (maximum Spearman rho) of the electrode or brain region showing the largest significant correlation to the cognitive variable. On the right side, we show the pairwise multivariate distance correlations between the EEG features (with all its variables).

Figure 3.

Distance correlations between the EEG features that correlated significantly with the Tmt-A variable in younger adults

The main diagonal contains the ${\mathcal{R}}_{\mathcal{n}}$max (maximum distance correlation) of the electrode, brain region, or microstate parameter showing the largest significant correlation to the cognitive variable. On the right side, we show the pairwise multivariate distance correlations between the EEG features.

While the correlations between EEG features were generally low (Figure 1B), some EEG features obtained from different analysis methods were highly correlated with each other. For example, in younger adults, both the Hjorth activity parameter (hjorth activity) and the standard deviation of the amplitude envelopes in the beta band (std ampl beta) showed significant correlations with the attention span module of the Cvlt (Cvlt attention span; ρmax = 0.28 and ρmax = 0.30, respectively). The two EEG features correlated strongly with each other (ρ = 0.84; $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}=$ 0.77). Similarly, using distance correlations, the animal categories variable of the Rwt test (Rwt animal categories) correlated significantly with the occurrence of microstate class B (microstate B; ${\mathcal{R}}_{\mathcal{n}}$ max = 0.27) and with the clustering coefficient of alpha connectivity measured in the electrode space using weighted phase lag index (clust coef e-wpli alpha; ${\mathcal{R}}_{\mathcal{n}}$ max = 0.28). The EEG features microstate B and clust coef e-wpli alpha exhibit a moderate correlation with each other (${\mathcal{R}}_{\mathcal{n}}$ = 0.41; $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}=$ 0.35). All pairs of EEG feature and cognitive variable that showed significant results are presented in Figure 4.

Figure 4.

Prediction of cognitive variables using EEG features

Cross-validated R² is shown (median R² across 50 iterations). A ridge regression model was built for each pair of EEG feature and cognitive variable. Abbreviations: A = Cvlt attention span, B = Cvlt delayed memory, C = Pts-2 subtest 3, D = Rwt animal categories, E = Rwt s words, F = Tap alertness, G = Tap simon congruent, H = Tap simon incongruent, I = Tap working memory, J = Tmt-A, K = Tmt-B, L = Vocabulary test. Green and orange squares indicate that Spearman and distance correlation analyses were significant, respectively. Purple squares indicate that both Spearman and distance correlations were significant for the same EEG and cognitive variable pair. Colormap limits are set between 0 and 0.2. Negative R² values are shown as zero.

Next, we used a principal component analysis (PCA) to examine whether EEG features, showing a significant correlation with a cognitive variable, can be grouped into a set of latent variables. Then, we used the EEG latent variables in a multiple regression model to predict the cognitive scores (see dimensionality reduction and multiple regression in quantification and statistical analysis). We found that a small number of EEG latent variables tended to explain a considerable amount of the variance of the EEG features that had a significant correlation (Spearman or distance correlation) with a cognitive variable (see Figures S5–S10). For instance, in younger adults, we applied PCA on the 18 EEG features that showed a significant Spearman correlation to the Tap working memory scores in younger adults (Figure 2). The first principal component explained 31.99% of the variance of the 18 EEG features. The first three principal components explained 57.61% of the variance of the 18 EEG features (Figure S6). Results are similar across cognitive variables. The proportion of variance explained by the first principal components of the EEG features showing a Spearman correlation to a cognitive variable ranged from 29.32% (Rwt animal categories; 7 EEG features) to 62.83% (Rwt s words; 8 EEG features; median across cognitive variables: 44.15%). For EEG features showing a significant distance correlation to a cognitive variable, the variance explained by the first principal components ranged from 27.81% (Rwt animal categories; 11 EEG features) to 58.64% (Rwt s words; 9 EEG features; median across cognitive variables: 33.90%).

Finally, we investigated whether a combination of EEG latent variables explains the cognitive variables better than a single latent variable. Thus, we asked whether various uncorrelated EEG features carry complementary information of the cognitive variables. To this end, we computed a multiple regression model. First, we used only the first PC as the predictor variable and then added, one by one, more PCs to the model (up to the third PC, i.e., three predictors). To compare models with different numbers of predictor variables, we used adjusted-$R^2$ to measure the goodness of fit. The analysis was performed for younger and older adults separately. For the EEG features showing a significant Spearman correlation with a cognitive variable in younger adults, the adjusted-$R^2$ values ranged from 0.00 (Pts 2 subtest 3) to 0.22 (Tmt-B) using only the first principal component in the regression model (median adjusted-$R^2$ across cognitive variables: 0.15), and from 0.11 (Cvlt attention span) to 0.32 (Cvlt delayed memory) using the first three principal components (median adjusted-$R^2$ across cognitive variables: 0.21; Table S2). For EEG features showing a significant distance correlation, in younger adults, the adjusted-$R^2$ values ranged from 0.07 (Tap alertness) to 0.20 (Tmt-B) using only the first principal component in the regression model (median adjusted-$R^2$ across cognitive variables: 0.12), and from 0.09 (Tap alertness) to 0.22 (Rwt animal categories) using the first three principal components (median adjusted-$R^2$ across cognitive variables: 0.14; Table S4). Results for older adults are presented in Tables S3 and S5. Importantly, the estimates of predictive performance were not obtained using cross-validation. As such, results should be taken with caution.

Interim conclusions: There are significant correlations between cognitive variables and EEG features obtained with different analysis methods, including connectivity, spectral power, and microstate methods. Classically, studies in the field investigate the relationship between one EEG feature and one cognitive variable in great detail with the tacit assumption that the EEG feature is representative of a proposed brain mechanism. However, we found that even though various EEG features show a significant correlation with a cognitive variable, these EEG features usually do not strongly correlate with each other. We found that a set of latent dimensions composed of multiple EEG features may explain some cognitive variables better than a single latent dimension. Yet, this was not the case for most cognitive variables. Hence, one cannot take it for granted that an EEG feature is representative of the research question at hand just because there is a significant correlation between the feature and a cognitive or another variable. We are not claiming that studies based on single correlations cannot provide meaningful information about brain mechanisms. We are just pointing out that a significant result does not guarantee it. We examine this notion further in the general discussion.

Prediction of cognitive variables using EEG features

The features studied in EEG research are hypothesized to reflect neurophysiological processes involved in cognitive function. Therefore, we would expect that EEG features predict cognitive scores adequately. In this section, to test for predictive ability, we go beyond correlations and used cross-validated machine learning (ML) models. Thus, we tested the ability of single EEG features to predict each cognitive variable in an out-of-sample manner. This approach has several advantages. First, ML models handle multivariate predictors very well. Neuroimaging data are often composed of information from several brain regions or electrodes, and thus, multivariate methods provide a compact way to use all the information. Second, models can be evaluated using cross-validation, where different parts of the data are used to train and test the models, providing a rigorous test for the generalizability of results. We tested two models, namely ridge models, which are sensitive to linear relationships between predictors and predicted variables, and random forest models, which detect nonlinear relationships.

In total, 2100 (175∗12) models were built using one EEG feature and one cognitive variable for each ML model (ridge or random forest) and age group (younger and older). Predictive performance was estimated using the coefficient of determination (R²). Models were trained using cross-validation on 67% of the data and tested on the left-out 33%. We repeated the entire procedure 50 times, with different allocations of participants in the train and test sets, and obtained the median predictive performance (see cross-validated prediction of cognitive variables using EEG features in quantification and statistical analysis). Note that R² calculations (using the sums of squares formula and not the squared correlation) can result in negative values when the model prediction on data not used in model training is less accurate than it would be by just predicting the mean value of the data.^25,26 For younger adults, the 25^th, 50^th, and 75^th percentiles of the 2100 (175∗12) R² values obtained using ridge regression were 0.00, 0.00, and 0.03 for the training data and −0.04, −0.03, and −0.02 for the testing data (Figure 4). For the random forest regression models, the 25^th, 50^th, and 75^th percentiles of the R² values were 0.65, 0.74, and 0.78 for the training data, and −0.14, −0.10, and −0.06 for the testing data. For older adults, the 25^th, 50^th, and 75^th percentiles of the R² values obtained using ridge regression were 0.00, 0.00, and 0.06 for the training data and −0.09, −0.06, and −0.04 for the testing data. For random forest regression models, the 25^th, 50^th, and 75^th percentiles of the R² values were 0.80, 0.82, and 0.83 for the training data, and −0.24, −0.16, and −0.10 for the testing data. See Data S2 to S5 for detailed results.

Interim conclusions: Predictions play a crucial role in science. With this analysis, we set out to assess the ability of EEG features to predict cognitive variables. We used cross-validated prediction models. There is a hypothesized relationship between neurophysiological features at rest and cognitive performance. Thus, if EEG features truly reflect core aspects of cognitive functioning, one might expect EEG to predict cognitive performance well. Surprisingly, we found generally weak predictive performance using two different regression models. Hence, there is the possibility that we might need to rethink to what extent neurophysiological features obtained from resting-state recordings truly have a clear-cut link to behavioral measures. Another option is that the relationships might be less strong than often implicitly thought. We like to stress that our results provide only a general overview of brain-behavior predictive success and are only related to resting-state EEG features.

Group comparisons of the EEG features between younger and older adults

Classically, case-control studies using EEG are set out to identify neurophysiological processes differing in two groups (e.g., patients and controls, younger and older adults). The tacit assumption is that a significant result shows that an EEG feature under study points, for example, to a cause of a disease. In this analysis, we examined differences between older and younger adults. Each of the 175 EEG features was subjected to group comparisons between older and younger adults. 108 out of the 175 EEG features (61.71%) contained at least one variable showing significant group differences between older and younger adults, indicating that important age-related effects are detected. The absolute effect sizes (r; ranging from 0 to 1) of the representative variables ranged between 0.18 (microstate E) and 0.58 (spectral entropy beta), corresponding to small to large effect sizes.²⁷ The 25^th, 50^th, and 75^th percentiles of the absolute significant effect sizes (one value per significant EEG feature) were 0.26, 0.31, and 0.42, respectively. For 56 out of the 108 EEG features showing significant group differences, the effects were positive, namely, older adults showed higher values than younger adults and the opposite was the case for the remaining 52 EEG features (Figure 5).

Figure 5.

Effect size and confidence intervals of the group differences between younger and older adults for each of the 175 EEG features

Negative effect sizes indicate that older adults had significantly reduced values compared to younger adults. Black dotted horizontal lines serve as a guide to the labels of the EEG features showing significant group differences.

Older adults showed significantly decreased node strength in theta connectivity measured in the source space using lagged phase synchronization (node str s-lps theta; r = -0.31), increased long-range temporal correlations in the delta band (dfa exponent delta; r = 0.29), as well as longer mean duration of the microstate class A (microstate A; r = 0.50), to name a few. Group differences were also observed in EEG features in the different frequency bands, for instance, older adults showed reduced spectral entropy in the delta band (spectral entropy delta; r = -0.41), reduced spectral amplitudes in the theta band in the source space (source ampl theta; r = -0.40), reduced node strength in alpha connectivity measured in the source space using lagged phase synchronization (node str s-lps alpha; r = -0.25), increased waiting time statistics of the amplitude envelopes in the beta band (waiting time beta; r = 0.33), and increased node strength in gamma connectivity measured in the electrode space using phase locking value (node str e-plv gamma; r = 0.26).

Interim conclusions: We identified various EEG features that showed group differences in older compared to younger participants. The effect sizes of the group differences ranged from small to large, with a median significant effect size of r = 0.31. Hence, these features point to age-related changes in brain processing. There is the question of whether the EEG features, showing clear-cut group differences, point to the same neurophysiological mechanism differing in older participants.

Correlations between EEG features showing age-related differences

In the previous analysis, we found 108 EEG features showing differences in older compared to younger participants. In this section, by pairwise correlating these EEG features, we ask whether the targeted brain processes point to a common mechanism underlying age-related differences. We calculated Spearman correlations between the representative variables (i.e., showing the largest group effect) of the 108 EEG features showing group differences (Figure 6). We found that 41.74% of the correlation values were significant for younger adults and 33.77% for older adults (without correction for multiple comparisons). Since significance depends on the sample size, we focus on the magnitudes of the pairwise correlations. The 25^th, 50^th, and 75^th percentiles of the magnitudes of the 5778 (108∗107/2) correlation values were 0.06, 0.13, and 0.29, for younger adults and 0.08, 0.17, and 0.31, for older adults.

Figure 6.

Spearman correlations between the 108 EEG features that showed a significant group difference between younger and older adults

The ρ correlations belonging to younger and older adults are presented in the upper and lower triangular parts of the matrix, respectively. To calculate the correlations, from each EEG feature, we selected the electrode, brain region, or microstate parameter showing the biggest effect size in the group comparisons between older and younger adults. See Figure S2 for the results using multivariate distance correlations.

Importantly, since the choice of EEG reference may influence the results,²⁸ we obtained zero-referenced EEG features and compared them to those with average reference and current source density (CSD; for connectivity features in the electrode space), which were the ones used in the previous analyses. We found a very good agreement between average/CSD and zero-referenced EEG features, as quantified by intraclass correlations (ICC) and distance correlations (see comparison between EEG reference choices in quantification and statistical analysis). For younger adults, the 25^th, 50^th, and 75^th percentiles of the ICC values between average/CSD and zero-referenced EEG features were 0.57, 0.92, and 0.97 (ICC ranges from 0 to 1), while for older adults, the percentile values were 0.65, 0.93, and 0.97, suggesting that the choice of reference does not affect the results. For younger adults, the 140 distance correlation values between average/CSD and zero-referenced EEG features were 0.79, 0.98, and 0.99, while for older adults, the distance correlations were 0.81, 0.99, and 1, for the 25^th, 50^th, and 75^th percentiles. While for most EEG features the ICC and distance correlation values were high, network EEG features, in particular, betweenness centrality features showed rather low ICCs (see Data S6). However, this did not change the relationship between EEG features (see: Figures S3 and S4). For zero-referenced EEG features, the pairwise correlations between the 140 EEG features (see Data S6) in younger adults were 0.09, 0.19, and 0.36, whereas in older adults the pairwise correlations were 0.11, 0.17, and 0.33, for the 25^th, 50^th, and 75^th percentiles. For average/CSD EEG features, the correlations were 0.10, 0.20, and 0.38 for younger adults, and 0.12, 0.20, and 0.35 for older adults (25^th, 50^th, and 75^th percentiles of the multivariate distance correlation).

EEG features may be more adequately summarized considering the whole set of electrodes, brain regions, or microstate parameters. Hence, in addition to the previous univariate correlation assessment, we calculated multivariate distance correlations, which allowed us to compare EEG features using all their variables (Figure S2). The results are similar. For younger adults, the magnitudes of the multivariate distance correlations ($\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$) were 0.12, 0.23, and 0.41, whereas for older adults the magnitudes were 0.12, 0.21, and 0.38, for the 25^th, 50^th, and 75^th percentiles. In younger adults, 58.54% of the multivariate distance correlations were significant, whereas 53.01% were significant for older adults (without correction for multiple comparisons).

Furthermore, to investigate whether the EEG features showing a group difference between older and younger participants can be grouped into a set of latent variables, we used PCA on the representative variables (i.e., the variables showing the largest group difference between older and younger participants) of the 108 EEG features. For older adults, the first PC explained 24.01% of the variance of the EEG features. The second and third PCs explained 13.47% and 7.54%, respectively (Figure S11). The first PC consists essentially of EEG features obtained from nonlinear dynamical and entropy analysis methods (e.g., rqa trapping time, lzc primitive, and permutation entropy). The second PC is mainly composed of beta and delta band features (e.g., mean ampl beta, node str e-plv delta, and relative ampl delta) and also nonlinear dynamical features (e.g., katz fractal dim and lyapunov exponent). The third PC contains mainly temporal and connectivity EEG features (e.g., microstate D, node str e-plv alpha, and life time delta). These latent dimensions could be interesting for future investigations of age-related differences in neurophysiology.

Most of the strong correlations were found between EEG features obtained from very similar methods. For example, the node strength and the clustering coefficient of delta connectivity estimated in the electrode space using the imaginary part of coherence (node str e-icoh delta and clust coef e-icoh delta, respectively) showed strong correlations with each other (ρ = 0.97, $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$ = 0.99 and ρ = 0.97, $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$ = 1 in younger and older adults, respectively). Similarly, EEG features obtained using recurrence quantification analysis, including determinism (rqa determinism) and laminarity (rqa laminarity), were strongly correlated (ρ = 0.99, $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$ = 0.98 and ρ = 0.99, $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$ = 1 in younger and older adults, respectively). The life and waiting time statistics of the amplitude envelopes in the gamma band (life time gamma and waiting time gamma) showed also strong positive correlations (ρ = 0.91, $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$ = 0.97 and ρ = 0.83, $\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$ = 0.98 in younger and older adults, respectively), to name a few examples.

Interim conclusions: Identifying brain mechanisms underlying cognition or perception is crucial in the brain sciences. Our analysis showed that various EEG features, all showing group differences in older compared to younger adults, mainly correlate weakly with each other. While there are strong correlations between similar methods, for example, between entropy and nonlinear measures, these features did not correlate, for instance, with EEG microstates, connectivity, or autocorrelation features. Hence, while the EEG features point to meaningful brain processes showing clear-cut group differences, they do not point to a general neurophysiological deficit in older compared to younger adults. The main conclusion from this analysis, as well as the previous ones, is that statistically significant effects might explain much less of the research question than it is often implicitly assumed.

Discussion

Science relies on tests targeting the crucial aspects of a research field. Classically, these tests show a significant group difference between an intervention and a control condition, between a case and a control group, or strong correlations to real-world or experimental outcomes. Then, in-depth studies are carried out to describe the test in great detail in order to understand the causes of the observed effects. It usually is assumed that these tests are representative of the interrogated mechanisms in the sense that other tests supposed to target the same mechanism should strongly correlate with each other. Here, we have shown that this rationale might not always hold. To exemplify our argument, we analyzed a publicly available database containing resting-state EEG recordings and performance scores from a battery of cognitive tests of older and younger adults.²⁴

We extracted 175 EEG features from the EEG recordings and 12 cognitive variables from the battery of cognitive tests. To identify the associations between EEG features and cognitive variables, we correlated each EEG feature with each cognitive variable using Spearman and distance correlations. For younger adults, Spearman correlations were significant for 109 analyses, while 121 analyses were significant using distance correlations. For older adults, Spearman correlations were significant for 57 analyses and distance correlations for 60 (Figure S1). Then, we correlated the EEG features that showed a significant correlation with a cognitive variable (Figure 1). Surprisingly, the correlations were weak in most cases, suggesting that not all the EEG features are representative of the investigated cognitive variable. Using PCA, we found that a set of latent dimensions composed of multiple EEG features may explain some cognitive variables better than a single latent dimension. Next, we found 108 EEG features revealing a significant group difference between older and younger participants. These features also did not show strong correlations, even though they showed clear-cut group effects. Using cross-validated regression analysis, we found very weak evidence that EEG features are adequate predictors of the cognitive variables suggesting that it may be possible that the link between a cognitive aspect and an EEG feature is less informative than believed.

How can these results be explained? There are at least five possibilities. First, EEG features might have low test-retest reliability. Even though there are significant correlations with the cognitive tasks, the low test-retest reliability may have led to weak correlations between EEG features (cf.²⁹). While here we do not have a measure of test-retest, certain EEG features have shown adequate reliabilities in previous studies.^{30,31,32,33,34,35} Second, there might be clear-cut group differences but the variance in the groups is low. In this case, one would not expect high correlations, the so-called reliability paradox.³⁶ However, variance, for example, was rather high in the older participants in our study. Third, the EEG features do not reflect the intended brain aspects well. For instance, the EEG features may point to nonlinear brain mechanisms. Yet, some of the EEG analysis methods are linear and as such, they might miss nonlinear patterns. Hence, the correlations between EEG features might be misled by the linear methods. Fourth, the EEG features target a common mechanism and, to a substantial amount, also target-unspecific, i.e., idiosyncratic, aspects, which have little to do with the targeted mechanism but contribute to the large inter-individual variability. For example, some EEG features might be sensitive to fatigue,³⁷ whereas others are not. Hence, if substantial parts of the variance correspond to such target-unspecific aspects and different EEG features tap into different target-unspecific aspects, the correlations may be low. Fifth, each EEG feature targets a different aspect of a highly heterogeneous, multifactorial mechanism. Previous research indicates that this scenario might be true in certain instances. For example, by combining different resting-state or evoked EEG features, previous studies have improved classification between groups or experimental conditions.^{38,39,40,41,42,43} Similarly, previous studies have reported that a combination of EEG features allowed a better characterization of certain brain processes.^44,45,46 Our results using PCA also show evidence that, for some cognitive variables, latent dimensions of EEG features might be more informative than single features. All these studies suggest that combining measures and features from the same paradigms might offer new insights into complex processes.

In our analysis, significant results came with small to medium effect sizes in the range from 0.20 to 0.37 (|ρ|) in the correlation analysis, and with small to large effect sizes in the range of 0.18–0.58 (|r|) in the case-control analyses according to Cohen²⁷ and from typical (0.2–0.3) to relatively large (>0.3) according to Gignac and Szodorai.⁴⁷ However, even for “large” effect sizes, there is a large proportion of unexplained variance. For example, for r = 0.5, the unexplained variance is 75%, and a Cohen’s d of 0.8 corresponds to a discriminability of 65% only (for the optimal decision criterion, i.e., hits = correct rejections). A good discriminability of 90% corresponds to a d = 2.5. Hence, the question is where does all the dominant noise come from? In an optimistic scenario, it comes from measurement noise. For example, EEG is a relatively noisy technique (e.g., electrode misplacement and volume conduction). Thus, the true effects might be larger. The pessimistic scenario is that noise is low and inter-individual variability is high, i.e., there are multiple factors and each paradigm taps into one, or a nonlinear combination of all of them plus a large amount of target-unspecific variance. This large amount of unexplained variance could account for why one may obtain both significant group differences and low correlations. Hence, even if clear-cut effects are found, this does not guarantee that a paradigm represents the intended aspects well. Therefore, when a test leads to a significant result, one needs to ask how representative the test is for the research question at hand. Particularly in complex systems where everything is correlated with everything to some degree, there can be many tests, which show significant but negligible effects.

Our impression is that the above considerations have been overlooked and hold true in many other research areas. As an example, in schizophrenia research, several studies have found atypical patterns in several resting-state EEG features, which are thereon studied in detail and linked to the crucial aspects of the disease. Here, again, the tacit assumption is that the EEG feature under study taps into the common and representative aspects of the disease, and for that reason, they should correlate with similar features. However, we have shown that this is not always the case. In a previous EEG study, we extracted 194 EEG features from the resting-state recordings of 121 patients with schizophrenia and 75 healthy controls. We found that 69 out of the 194 EEG features showed a significant group difference between patients with schizophrenia and healthy controls. However, the features showed mainly weak correlations with each other, questioning to what extent a single EEG feature is representative of the disease.⁴⁸ In another example, in vision research, weak correlations have been found between performances in various visual tasks in older and younger adults.^49,50,51,52 Visual illusions also correlate weakly,^53,54 suggesting that the underlying visual functions cannot be explained using only one visual paradigm. Similarly, four visual tasks assumed to capture visual magnocellular stream function showed weak relationships with each other.⁵⁵ The authors concluded that none of these tasks is a general measure of magnocellular function, as was assumed. Eisenberg et al.⁵⁶ showed that several questionnaires and cognitive tasks thought to point to the same psychological construct correlate weakly with each other and predict real-world outcomes poorly. The authors concluded that the construct lacks coherence.

Most natural sciences face severe crises. The brain sciences are among the fields hard-hit. First, many studies are underpowered and/or subject to questionable research practices. False positives are the consequence.^57,58,59 Second, even if the very same data are used, different analysis tools can lead to different results. This problem becomes more severe with complex analysis pipelines and more degrees of freedom.^60,61,62 Third, the uncontrolled use of open data for hypothesis testing can increase false positives.^63,64,65 Fourth, here, we have shown that there is one additional problem. Studies may have been conducted perfectly with clear-cut, significant results. Still, the studies may not target the mechanism assumed or they are less representative of the research question than believed. Overall, our results show that single measurements, even with “large” effect sizes, may be less meaningful than thought. To what extent the above-mentioned scenarios hold must be shown for each study individually.

Limitations of the study

One of the main limitations of our study is that resting-state EEG and behavior were not measured simultaneously. Hence, whether there are causal links between EEG features and cognitive variables is not clear-cut. The evoked EEG features show a more direct link to the temporal aspects of cognitive processing, and thus stronger brain-behavior relationships might be expected. As such, our results and interpretations of the weak correlations between EEG and cognition do not concern evoked EEG paradigms. Furthermore, there are at least two limitations that should be considered when interpreting the results from our prediction analysis. First, our sample size is small for a reliable assessment of predictive ability.⁶⁶ Second, we do not have an independent dataset to test our predictive models. We tried to account for these limitations using a repeated train-test split procedure, which produces unbiased estimates of predictive ability for small sample sizes. Nonetheless, larger sample sizes and independent datasets are needed. Finally, we do not have a measure of the test-retest reliability of the EEG features. Hence, EEG features with poor reliabilities may mislead the correlations. While our results show that similar EEG features correlate strongly with each other (resembling test-retest), measuring and accounting for reliability, e.g., using disattenuated correlations, will tell about the “true” correlations between EEG features.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Biological samples
EEG and cognitive data	Leipzig Study for Mind-Body-Emotion Interactions	https://doi.org/10.1038/sdata.2018.308
Software and algorithms
MATLAB	https://matlab.mathworks.com/	Version R2020b
Python	https://www.python.org/	Version 3.6
R Studio	https://www.r-project.org/	Version 4.0.1
Cartool	https://sites.google.com/site/cartoolcommunity	Version 3.8
LORETA	https://www.uzh.ch/keyinst/loreta	Version v20200414
Other
Scripts for data processing	Github repository	https://github.com/dgl59311/stats_eegfeatures

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact: Dario Gordillo.

Materials availability

The study did not generate new materials.

Experimental model and subject details

Data from 227 participants were collected in Leipzig, Germany, as part of the Leipzig Study for Mind-Brain-Body Interactions (LEMON²⁴). The sample comprises data from two age groups, 153 younger adults (between 20 and 35 years old) and 74 older adults (between 59 and 77 years old). Participants underwent a physiological and psychological screening at the Day Clinic for Cognitive Neurology of the University Clinic Leipzig and the Max Planck Institute for Human Cognitive and Brain Sciences. Written informed consent was provided by all the participants before data collection. Study protocols were in accordance with the Declaration of Helsinki and were approved by the ethics committee of the University of Leipzig (reference number 154/13-ff).

The data were made publicly available. In the present study, only data from participants that had resting-state EEG recordings were analyzed. Preprocessed resting-state EEG recordings were available for 203 participants (https://ftp.gwdg.de/pub/misc/MPI-Leipzig_Mind-Brain-Body-LEMON/EEG_MPILMBB_LEMON/). We excluded two participants (sub-010276, and sub-010277) due to differences in the sampling rate. The final sample used for the present study consisted of 201 participants, 138 younger adults (mean age = 25.43, SD = 3.39, 42 females), and 63 older adults (mean age = 67.66, SD = 4.79, 31 females).

Method details

EEG collection and preprocessing

EEG was recorded using a 62-channel active electrode ActiCAP system (Brain Products GmbH, Germany) placed according to a 10-10 system arrangement with the FCz electrode as the reference. The ground was placed on the sternum. Impedances of the electrodes were kept below 5 kΩ. EEG signals were band-pass filtered online between 0.015 Hz and 1 kHz. Data were digitized with a sampling rate of 2500 Hz. During recording, participants alternated between eyes-closed and eyes-open conditions, after 1 min. A 16-min recording was obtained for each participant. Following data acquisition, signals were band-pass filtered between 1 and 45 Hz (eighth order, Butterworth filter). Data were downsampled to 250 Hz.

Further offline preprocessing consisted of the rejection of artifactual channels and segments following a visual inspection. The dimensionality of the data was reduced using principal component analysis (PCA). The PCs (N ≥ 30) allowing to explain 95% of the total variance were retained. Other physiological artifacts, such as eye movements, blinks, or heartbeats, were identified using independent component analysis and removed. Finally, the retained components were back-projected to the electrode space. Further details on data acquisition and preprocessing are available in.²⁴

In the present study, we used 8-min blocks corresponding to the eyes-closed condition segments. Missing electrodes were interpolated using spherical spline interpolation in EEGLAB⁶⁷ to fit the same 61-channel montage for all participants. Then, the recordings were re-referenced to the average and down-sampled to 125 Hz.

Cognitive assessment

In total, participants performed six cognitive tests. The data were made publicly available by the LEMON study (https://ftp.gwdg.de/pub/misc/MPI-Leipzig_Mind-Brain-Body-LEMON/Behavioural_Data_MPILMBB_LEMON/Cognitive_Test_Battery_LEMON/). From the six cognitive tests, we extracted 12 variables. The tests and the extracted variables are described below. More details can be found in the documentation of the database.

California verbal learning task

The California Verbal Learning Task (CVLT) measures memory processes and verbal learning capacity.⁶⁸ Participants listened to a 16-word list (list A) over five trials. The words belonged to four different semantic categories. After each trial, participants were asked to recall as many words from list A as they could. Then, another 16-word list (list B) was presented as an interference list, which had to be recalled right after its presentation, and had to be followed by a recall of list A. After a delay of 20 min, participants were asked to recall the words from list A, with or without semantic category cues. Based on Donders,⁶⁹ and Mahjoory and colleagues,⁷⁰ we extracted two scores: attention and delayed-memory scores. The attention score was calculated by adding up the number of words that were correctly recalled after hearing list A for the first time and the number of correctly recalled words from list B. The delayed memory score was calculated by adding the number of correctly recalled words from list A after listening to list B, and the number of correctly recalled words from list A after the 20-min delay, with and without cues.

Test of attentional performance

The Test of Attentional Performance (TAP) consists of three modules that assess different aspects of attention.⁷¹ In the first module, participants had to press a button as soon as a cross appeared on the screen. Two conditions were tested: with and without a pre-stimulus audio signal. The alertness score was estimated from this module as the reaction time averaged across the two conditions. The second module corresponded to the Simon task. In the Simon task, participants had to press a left or right button to indicate the direction of an arrow appearing on the left or the right side of the screen. Congruent (i.e., the direction of the arrow matched its location) and incongruent (e.g., left-pointing arrow on the right side of the screen) trials were presented. The average reaction times (RT) and the percentage correct (PC) were recorded and combined into a rate correct score (i.e., RT/PC.⁷² We extracted two scores from this module, given by the rate correct score for the congruent and incongruent trials of the Simon task. In the third module, participants were presented (serially) with numbers from 1 to 9, and they had to press a button whenever the current number was the same as the second to last number (2-back task). We extracted a working memory score from this module, given by the percentage of correct matches.

Trail making test

The Trail Making Test (TMT) measures cognitive flexibility.⁷³ In module A, participants had to connect digits from 1 to 25 in ascending order. In module B, 13 numbers and 12 letters had to be alternately connected in their numerical and alphabetical order (e.g., 1-A-2-B-3-C- …). We extracted two scores from the TMT given by the inverse efficiency score (i.e., Task completion time/PC) for modules A and B.

Vocabulary test

The Vocabulary Test (VT) measures verbal intelligence and language comprehension.⁷⁴ Participants had to identify a target word among five distracting words. There were 42 trials. We extracted one score from the VT, given by the number of correctly identified target words over all the trials (VT-score).

Performance testing system-2 Subtest 3

The Subtest 3 of the Performance Testing System-2 (Pts-2) assesses logical deductive thinking.⁷⁵ For 3 min, 40 rows of eight symbols were presented to the participants. For each row, participants had to identify the symbol that did not follow the logical rule. We extracted one score from the Pts-2, given by the total number of correctly identified symbols.

Regensburger word fluency test

The Regensburger Word Fluency Test (Rwt) measures verbal fluency.⁷⁶ The Rwt test consisted of two modules. In the first module, for 2 min, participants had to list as many words starting with the letter “S” as they could. In the second module, participants had to list words representing animals, for 2 min. We extracted two scores from the Rwt, given the total number of correct words in each module.

EEG features extraction

Using time-domain, frequency-domain, nonlinear, and connectivity analysis methods, we extracted 175 features from the resting-state EEG signals. For some analysis methods, we filtered the EEG signals into five frequency bands: delta (1–4 Hz), theta (4–8 Hz), alpha (8–13 Hz), beta (13–30 Hz), and gamma (30–45 Hz). The dimensionality of the analysis outcomes (i.e., EEG features) depended on the analysis method. For instance, for each participant, we obtained either 61 or 80 variables if the analyses were conducted in the electrode or source space, respectively, or 4 variables for EEG features extracted using microstates analysis. Hence, each EEG feature is always composed of more than one variable. We described the analysis methods below. The list of all the EEG features extracted is available in Data S1.

Statistics of amplitude envelopes

For each frequency band, we calculated five statistical descriptors of the distribution of the signal. Amplitude envelopes were extracted using Hilbert transform. The descriptors were: mean and standard deviation of the amplitude envelopes, kurtosis, skewness, and total power of the signals. First, EEG signals were divided into non-overlapping 4-s segments and filtered into five frequency bands. Then, the statistical descriptors were calculated for the amplitude envelope values of each electrode, at each time segment. The average across time segments was used for further analyses. The analyses were conducted in the electrode space. We obtained 25 EEG features from these analyses (Features: 1–5, 68–72, 81–85, 151–155, 166–170; in Data S1).

Spectral amplitudes

To estimate spectral amplitudes, first, EEG signals were divided into non-overlapping 4-s segments. For each electrode time series, at each time segment, we used Fourier analysis to obtain the frequency amplitudes. Relative spectral amplitudes were defined for each frequency band, as the ratio between the sum of the squared Fourier coefficients within the bounds of the frequency band of interest (e.g., within 1–4 Hz, for the delta band), and the squared Fourier coefficients of the full-band signal. The average across time segments was used for further analyses. This analysis was conducted in the electrode space. We obtained five EEG features from this analysis (Features: 137–141; in Data S1).

In addition, we estimated the current source amplitudes for each frequency band using the software LORETA.⁷⁷ We defined 80 brain regions of interest (80; 40 per hemisphere) according to the AAL atlas (see Table S1). ROIs included gray matter voxels within a 10-mm radius of the seed. This analysis was conducted in the source space. We obtained five features from this analysis (Features: 156–160; in Data S1).

Temporal correlations

For each frequency band we calculated long-range and short-range temporal correlations of EEG oscillations. Long-range (>1 s) temporal correlations (LRTC) were calculated using detrended fluctuation analysis (DFA⁷⁸). Short (∼< 1 s) temporal correlations were calculated using life – and waiting-time statistics.⁷⁹ For DFA, first, we extracted the amplitude envelopes from the EEG time series using Hilbert transform. The amplitude envelopes were integrated. Then, we defined 30 window sizes, varying from 3 to 50 s, distributed evenly on a logarithmic scale. The integrated signal was divided into 50% overlapping segments for each previously defined window-size. At each of these segments, the integrated signal was detrended, and the fluctuation function (i.e., variance) was obtained. The average fluctuation function across segments of each window size was calculated. The average fluctuation functions were plotted in logarithmic axes, and a line was fit. The slope of the line indicated the scaling exponent and this value was used for further analyses. To estimate short-range temporal correlations, we extracted the amplitude envelopes of the EEG time series using Hilbert transform. The median amplitude envelope was used as a threshold, which defined the onset and end of the oscillation bursts. The distributions for life and waiting times were built using the durations of all the burst events that occurred above or below the threshold, respectively. The 95^th percentile of the distributions of life and waiting times were used for further analyses. The analyses were performed in the electrode space. We obtained 15 EEG features from these analyses (Features: 58–62, 73–77, 171–175; in Data S1).

Network and connectivity measures

First, EEG functional connectivity was calculated using five connectivity algorithms. Three of these algorithms (i.e., phase locking value, imaginary part of coherence, and weighted phase lag index) were defined in the electrode space, and two (i.e., lagged coherence and lagged phase synchronization) in the source space. EEG Functional connectivity was calculated at each of the five frequency bands. Electrode connectivity analyses were performed using FieldTrip.⁸⁰ Before estimating electrode connectivity, scalp current densities were obtained from the EEG time series using the FieldTrip function ft_scalpcurrentdensity with the spline method. For source connectivity analyses, first, cortical activity was estimated with the exact low-resolution electromagnetic tomography (eLORETA) algorithm using the software LORETA. We defined 80 brain regions (40 per hemisphere) according to the AAL atlas (see Table S1). ROIs included gray matter voxels within a 10-mm radius of the seed. Then, using the Brain Connectivity Toolbox (BCT ⁸¹), three network statistics (i.e., betweenness centrality, clustering coefficient, and node strength) were calculated from each of the connectivity matrices. The BCT functions employed were betweenness_wei, clustering_coef_wu, and strength_und. The extracted EEG features consisted of the network analysis outcomes. We obtained 45 features in the electrode space (Features: 7–21, 32–46, 101–115, in Data S1) and 30 features in the source space (Features: 22–31, 47–56, 116–125, in Data S1).

Microstates

First, for a given participant, the voltage maps at the peaks of the global field power (GFP) signal were extracted. Maps at GFP peaks have been indicated to have a higher signal-to-noise ratio, providing a more stable representation of the EEG topographies.⁸² Then, a k-means clustering procedure was performed on these maps with k (i.e., the number of cluster centroids) equal to 5. GFP peak maps were then assigned to the cluster centroid to which they showed the highest spatial correlation, as long as the correlation value was above 0.5, otherwise, maps were left unassigned.

Second, a k-means clustering procedure was performed on the concatenated cluster centroids from all the participants, obtained in the previous step. The algorithm was initialized 200 times for each value of k, with k varying from 1 to 15. The optimal number of subject-level cluster centroids was selected according to a metacriterion based on seven independent indicators for optimal cluster solution.⁸³ Out of the 200 initializations, the cluster solution showing the highest fraction of explained variance was the one retained. The polarity of the voltage maps was always ignored. Maps were only assigned to a given microstate class if they showed a spatial correlation larger than 0.5. Finally, the cluster centroids obtained from the subject-level analysis were assigned to the EEG data of each participant, this time, not only considering the GFP peaks but all the data. Voltage maps showing a correlation below 0.5 to any of the cluster centroids were left unassigned. Temporal smoothing (Besag factor = 10 and window half size = 2) was applied to avoid the interruption of quasi-stable segments.⁸⁴ Segments equal to or smaller than three samples were rejected. For each microstate class, we extracted four temporal statistics namely the global explained variance, mean duration, time coverage, and frequency of occurrence. Microstates analysis was performed using the software Cartool version 3.8.⁸⁵ We obtained five EEG features from this analysis (Features: 86–90, in Data S1).

Entropy and complexity measures

We quantified the complexity of EEG signals using five different methods: approximate entropy, sample entropy, spectral entropy, permutation entropy, and Lempel-Ziv complexity. First, we divided the EEG signals into non-overlapping 4-s segments. Approximate,⁸⁶ permutation,⁸⁷ and sample⁸⁸ entropies were calculated for the full-band EEG signals, using an embedding dimension value of three. Approximate entropy was computed using the function approximateEntropy from the Predictive Maintenance Toolbox for MATLAB. Permutation entropy was calculated based on Unakafova.⁸⁹ Sample entropy was computed using the code provided by Martínez-Cagigal.⁹⁰ The time delay to estimate approximate and permutation entropies was set to one. Lempel-Ziv complexity was calculated from the full-band EEG time series,⁹¹ using the code provided by Thai.⁹² Spectral entropy was calculated for each frequency band, and it was defined as the Shannon’s entropy of the ratio between the normalized power spectral density (PSD) within the frequency band bounds (e.g., within 1–4 Hz, for the delta band), and the full-band EEG signal. All calculations were performed at each time segment, and the average across segments was used for further analyses. The analyses were conducted in the electrode space. We obtained 10 EEG features from these analyses (Features: 6, 79, 80, 126, 150, 161–165, in Data S1).

Nonlinear dynamical measures

Before obtaining nonlinear dynamical features, the EEG signals were divided into non-overlapping 4-s segments. To obtain recurrence quantification analysis (RQA) features, first, for each electrode time series at each segment, we built recurrence plots and extracted eight RQA features using the CRP toolbox for MATLAB.⁹³ The RQA features were: determinism, entropy, laminarity, maximal diagonal line length, maximal vertical line length, mean diagonal line length, recurrence times entropy, and trapping time. Recurrence plots were constructed using a fixed radius allowing a 10% recurrence rate. The Lyapunov exponent⁹⁴ and correlation dimension⁹⁵ were obtained using the functions lyapunovExponent, and correlationDimension, available in the Predictive Maintenance Toolbox for MATLAB. We also calculated the Higuchi’s,⁹⁶ and Katz’s⁹⁷ fractal dimensions using the code provided by Monge-Álvarez.⁹⁸ The kmax parameter in Higuchi’s fractal dimension calculation was set to 25. All nonlinear dynamical features were obtained from the full-band EEG signals. For RQA measures, correlation dimension, and Lyapunov exponent, the embedding parameters (time delay and embedding dimension) were calculated using the function phaseSpaceReconstruction from the Predictive Maintenance Toolbox in MATLAB. The analyses were conducted in the electrode space. We obtained 12 EEG features from these analyses (Features: 57, 63, 67, 78, 142–149, in Data S1)

Phase-amplitude coupling

EEG features describing cross-frequency (CF) interactions via phase-amplitude coupling were obtained using the modulation index.⁹⁹ First, EEG time series were divided into non-overlapping 4-s segments and were filtered into the five frequency bands. For each band-pass filtered electrode time series, at each segment, the amplitude envelope and the instantaneous phase were extracted using the Hilbert transform. Then, we defined ten different CF interactions, these were between delta phase-theta amplitude, delta phase-alpha amplitude, delta phase-beta amplitude, delta phase-gamma amplitude, theta phase-alpha amplitude, theta phase-beta amplitude, theta phase-gamma amplitude, alpha phase-beta amplitude, alpha phase-gamma amplitude, and beta phase-gamma amplitude. For each CF interaction, we obtained the corresponding phase and amplitude time series (e.g., phase time series in the theta band and amplitude time series in the gamma band for theta phase-gamma amplitude). Then, the phase values were binned into 18 values (from −180 to 180°) and the mean amplitude value (of the modulated frequency) over each bin was calculated. Hence, we obtained the mean amplitude value of the modulated frequency, for each phase value of the phase-modulating frequency. The Kullback-Leibler (KL) divergence indicated whether the amplitude values are uniformly distributed according to the phase values (i.e., no phase-amplitude coupling). The KL divergence was calculated for each of the ten cross-frequency interactions, and the average value across time segments was used for further analyses. The analysis was conducted in the electrode space. We obtained ten EEG features from this analysis (Features: 91–100, in Data S1).

Time-domain amplitude features

Peak-to-peak amplitude asymmetry and coefficient of variation were calculated for each frequency band using range-EEG analysis.¹⁰⁰ Range features were obtained using the code provided by Toole and Boylan.¹⁰¹ We also obtained the Hjorth parameters activity, mobility, and complexity from the full band EEG time series.¹⁰² First, we divided the EEG time series into non-overlapping 4-s segments. Then, we calculated the time-domain amplitude features for each time segment and electrode signal. The average across segments was used for further analyses. The analyses were conducted in the electrode space. We obtained 13 EEG features from these analyses (Features: 64–66, 127–136, in Data S1).

Quantification and statistical analysis

Correlations between EEG features and cognitive variables

This section describes the analysis behind the results presented in the correlations between EEG features and cognitive variables subsection in results. To investigate the associations between the 175 EEG features and the 12 cognitive variables, we calculated Spearman correlations and distance correlations. Hence, we conducted 2100 correlation analyses (175∗12; for each correlation metric) for older and younger adults separately, because of the large age differences between the samples (see experimental model and subject details). Four older adults had missing values in some cognitive variables (sub-010044 in Tap alertness, sub-010047 in Tmt-A, sub-010050 in Tap working memory and Tap simon incongruent, and sub-010099 in Rwt s words and Rwt animal categories). These participants were only excluded in the analyses where the EEG features were correlated with the variables in which they had missing values. These participants were included for the rest of the correlation analyses.

For each pair of EEG feature (with all its electrodes, brain regions, or microstate parameters) and cognitive variable (e.g., approximate entropy and Tmt-A), we calculated Spearman correlation coefficients and distance correlations. EEG features can have a different number of variables, depending on whether the analysis method was conducted in the electrode space, source space, or using microstates analysis (see method details). Thus, for EEG features obtained in the electrode space (e.g., approximate entropy) we obtained 61 correlations to one cognitive variable (each electrode was correlated with the cognitive variable). For EEG features in the source space (e.g., source spectral amplitude in the alpha band, source ampl alpha) we obtained 80 correlations and 4 correlations for each microstate class. The p values of the correlations were corrected using False Discovery Rate (FDR¹⁰³) with an error rate of 5%. The p values of the distance correlations were obtained using 1000 repetitions of a bootstrapping procedure and corrected with FDR. We used the distance_corr function implemented in the pingouin 0.5.1 package for Python.¹⁰⁴

Next, to evaluate whether EEG features showing significant correlations with the same cognitive variable relate to each other, we calculated Spearman or distance correlations between these EEG features, depending on the method used in the previous step (i.e., either Spearman or distance correlation). From each EEG feature showing a significant correlation to a cognitive variable, we took the electrode, brain region, or microstate parameter showing the highest correlation to be the representative variable for that EEG feature in the analysis. The EEG features revealing a significant correlation to the cognitive variables were also compared using multivariate distance correlations,¹⁰⁵ which considered all the variables of the EEG features (see correlations between EEG features showing age-related differences section below).

Dimensionality reduction and multiple regression

This section describes the analysis behind the results presented in the correlations between EEG features and cognitive variables subsection in results. We analyzed the EEG features that showed a significant correlation to a cognitive variable using principal component analysis (PCA). Importantly, EEG and cognitive variables were log-transformed to improve normality using the function PowerTransformer from scikit-learn. For example, for the Tap working memory variable, we found 18 EEG features showing a significant correlation. We took one variable from each of these 18 EEG features (the one showing the largest correlation to the Tap working memory variable) and used a PCA. We calculated the proportion of explained variance. This analysis was performed for each cognitive variable and age group separately.

Then, to understand whether the set of latent variables obtained using PCA can explain the cognitive variables, we used multiple regression. We used the function OLS from the statsmodels 0.13.2 package for Python.¹⁰⁶ We obtained the latent variables from the EEG features that showed a significant correlation to a cognitive variable. Each EEG feature contributed with one variable (i.e., electrode, brain region, or microstate parameter), which was the one showing the highest correlation to the corresponding cognitive variable. We generated a regression model, first, using the first PC of the EEG data, and then, we added PCs one by one, up to the third PC (i.e., model 1: PC1, model 2: PC1 and PC2, model 3: PC1, PC2, and PC3). To quantify predictive performance, we used adjusted-R.² In this analysis, we did not use cross-validation to calculate adjusted-R² values.

Cross-validated prediction of cognitive variables using EEG features

This section describes the analysis behind the results presented in the prediction of cognitive variables using EEG features subsection in results. We investigated how well the EEG features predicted the cognitive variables using ridge regression and nonlinear random forest regression. We used the Scikit-learn 1.0.2 package for Python.²⁶ For each pair of EEG feature (with all its electrodes, brain regions, or microstate parameters) and cognitive variable, prediction performance was calculated using 50 repetitions of a train-test split procedure. The predictive performance was calculated using the coefficient of determination R². First, 33% of the data were left out for validation (testing set) and the remaining 67% of the data (training set) were used for model optimization. For linear ridge models, before training the model we applied a power transform to the data to improve normality using the PowerTransformer function. The amount of penalization λ (100 values from ${10}^{-3}$ to ${10}^5$ on an evenly spaced logarithmic scale) was selected using cross-validation with the efficient leave-one-out method implemented in the function RidgeCV. For random forest models, we used the function RandomForestRegressor. The parameters of the random forest models were selected using a grid search procedure with 3-fold cross-validation as implemented in the GridSearchCV function. We used 100 estimators and adjusted the tree-depth considering the values 4, 6, 8, or no constraint, and also adjusted the maximum number of features using log2, sqrt, or auto as options. The model with the parameters giving the best performance in the training set was applied to the testing set. The prediction performance R² was calculated using the function r2_score. The R² values were aggregated for each of the 50 repetitions of the procedure and the median predictive performance was reported. The analysis was conducted for the sample of older and younger adults separately.

Group comparisons of EEG features between older and younger adults

This section describes the analysis behind the results presented in the Group comparisons of the EEG features between younger and older adults subsection in results. We conducted group comparisons between older and younger adults using each of the 175 EEG features. For each variable (61 electrodes, 80 brain regions, or 4 microstate parameters) of a given EEG feature, we conducted a Mann-Whitney test using the function wilcox_test from R Studio version 4.0.1 ¹⁰⁷ and the package coin 1.4_2.¹⁰⁸ The p values and effect size r-values (with bootstrap confidence intervals) were obtained using the R Studio package rstatix 0.7.0.¹⁰⁹ The p values were corrected for multiple comparisons using FDR with an error rate of 5%, within each EEG feature.

Correlations between EEG features showing age-related differences

This section describes the analysis behind the results presented in the correlations between EEG features showing age-related differences subsection in results. First, for each EEG feature that contained at least one variable showing a significant difference between younger and older adults (after correcting for multiple comparisons), we selected the variable (i.e., the electrode, brain region, or microstate parameter) with the largest effect size to be the representative variable for that feature for the correlation analysis. For each age group separately, we computed pairwise Spearman correlations between these variables.

Second, to consider not only one but all variables of the EEG features revealing significant group differences, we used an unbiased multivariate distance correlation test for independence in high dimensions.¹⁰⁵ In high dimensions, the original distance correlation statistic (used in the correlation between EEG features and cognitive variables section) increases even under independence. The absolute unbiased estimate accounts for the bias in high dimensions and thus can provide an effect size of the relationship between two EEG features with all its variables (electrodes, brain regions, or microstate parameters) ranging from 0 to 1. We reported the square root of the absolute unbiased distance correlation ($\sqrt{\left|{\mathcal{R}}_{\mathcal{n}}^{\ast }\right|}$) because the unbiased distance correlation approximates the population squared distance correlation. The effect sizes and p values were obtained using the function dcorT.test from the R Studio package energy 1.7_10.¹¹⁰

Comparison between EEG reference choices

To investigate the effect of the EEG reference choice on our results, we re-analyzed the EEG data using zero-reference as implemented in the REST toolbox.¹¹¹ We obtained 140 EEG features (not considering source space EEG features) with this reference choice. These zero-referenced EEG features were compared to average or current source density CSD (for electrode connectivity features) referenced EEG features, which were the ones used in the main analyses, using intraclass correlations and distance correlations. For the comparison, we used the intraclass correlation with an absolute agreement (ICC2; to account for mean differences across references) as implemented in the function intraclass_corr from the pingouin 0.5.1 package for Python. Since EEG features have different numbers of variables (i.e., electrodes or microstate parameters), we calculated the ICC2 values for each variable of each EEG feature. For example, for the EEG feature microstate A, we obtained 4 ICC2 values (for global explained variance, mean duration, time coverage, and frequency of occurrence).

Furthermore, to examine whether the choice of reference affects the correlations between EEG features, we pairwise correlated the 140 EEG features obtained with zero reference using multivariate distance correlations, for younger and older adults separately.

Published: January 20, 2023

Data and code availability

• •This paper analyzes existing, publicly available data. These accession numbers for the datasets are listed in the key resources table.
• •All original code has been deposited at Github and is publicly available as of the date of publication. DOIs are listed in the key resources table.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.