The Psychometric Upgrade Psychophysiology Needs

Abstract

Although biological measurements are constrained by the same fundamental psychometric principles as self-report measurements, these essential principles are often neglected in most fields of neuroscience, including psychophysiology. Potential reasons for this neglect could include a lack of understanding of appropriate measurement theory or a lack of accessible software for psychometric analysis. Generalizability theory is a flexible and multifaceted measurement theory that is well suited to handling the nuances of psychophysiological data, such as the often unbalanced number of trials and intraindividual variability of scores of event-related brain potential (ERP) data. The ERP Reliability Analysis Toolbox (ERA Toolbox) was designed for psychophysiologists and is tractable software that can support the routine evaluation of psychometrics using generalizability theory. Psychometrics can guide task refinement, data-processing decisions, and selection of candidate biomarkers for clinical trials. The present review provides an extensive treatment of additional psychometric characteristics relevant to studies of psychophysiology, including validity and validation, standardization, dimensionality, and measurement invariance. Although the review focuses on ERPs, the discussion applies broadly to psychophysiological measures and beyond. The tools needed to rigorously assess psychometric reliability and validate psychophysiological measures are now readily available. With the profound implications that psychophysiological research can have on understanding brain-behavior relationships and the identification of biomarkers, there is simply too much at stake to ignore the crucial processes of evaluating psychometric reliability and validity.

1. Introduction

Success in the quantitative study of individual differences hinges on sound psychometrics. For example, the use of measurements with high psychometric reliability is crucial in any study of individual differences, including studies of personality characteristics, psychiatric symptoms, developmental changes, or any other quantitatively characterized relationship. The use of valid measures ensures that variations in measurements are caused by changes in the intended attribute or process of interest. Biological measurements are constrained by the same fundamental psychometric principles as self-report measurements, but these essential psychometric principles receive little to no attention in most fields of neuroscience, including psychophysiology. Grounding psychophysiological research in psychometrics will foster a better understanding of psychological and biological phenomena across a wide spectrum of disciplines, particularly those that pursue an individual differences approach, including clinical research and practice.

Psychometricians and theoretical psychologists have long been rigorously developing and debating approaches for evaluating psychometric reliability and concepts of validity (e.g., Brown, 1910; Cronbach & Meehl, 1955; Loevinger, 1957; Novick & Lewis, 1967; Spearman, 1910). Their focus was typically on elaborating and evaluating psychometric methods for self-report, achievement, or aptitude measures. Although robust methods are available to evaluate the psychometrics of self-report methods, only recently have some of these methods been tailored for use with psychophysiological data. These recent developments are important because commonplace approaches to assessing psychometric reliability are poorly equipped for the nuances of psychophysiological data. The purpose of this review article is to highlight some appropriate methods and solutions for evaluating the psychometric reliability and validation of psychophysiological data. The emphasis of the present review is on studies of event-related brain potentials (ERPs), but the principles described below are widely applicable to all psychophysiological modalities and beyond.

2. ERPs Are Operationalizations

An understanding of the basic nature of ERP recording and analysis is important in applying psychometric principles. The collection and analysis of ERP data are computationally intensive and methodologically complex, and many of the decisions in the data-processing pipeline leading up to and including the scoring of ERP waveforms are decided by the researcher (Clayson et al., 2019; Luck & Gaspelin, 2017). An ERP waveform represents biological signals that are spectrally and temporally complex. An ERP component refers to a distinct voltage change in the scalp-recorded ERP waveform related to neural events, but an observed ERP peak in that waveform can represent a contribution of a single or multiple ERP components (Donchin & Heffley, 1978; Kappenman & Luck, 2012). Varying the measurement of ERP peaks in temporal or spatial dimensions can capture distinct contributions of ERP components.

Considering the decisions required to distill raw continuous data into a single score for statistical analysis, the amalgamation of these decisions represents an operationalization of an ERP component. ERPs typically represent an operationalization of a construct because an ERP component is only one potential measure of any given construct (e.g., activity of anterior cingulate cortex [ACC]). Additionally, any particular approach to processing and scoring an ERP represents additional layers of that operationalization. ERP scores cannot be reified as they are no more than a descriptive summary of scores at a time and location.

The error-related negativity (ERN) component provides a ready example of an operationalization of error-related performance monitoring. An ERN is a negative-going peak that occurs within 100 ms following error responses and represents a manifestation of cognitive, affective, motivational, and motor processes (Gehring et al., 2012). Although the functional significance of ERN is debated (Clayson et al., 2021e), ERN¹ is generally considered an index of error detection. Interpretation of ERN findings across studies is complicated by inconsistent operationalizations, including the use of different tasks for recording (e.g., flanker, Stroop, go/nogo), different data-processing pipelines (e.g., filtering, ocular artifact correction/rejection, trial inclusion criteria), and different scoring (e.g., time windows, electrode sites, difference waves, scoring approaches [peak amplitude, time-window mean amplitude, etc.]).

Any single operationalization of ERN makes assumptions about the neural activity of interest, and those assumptions ultimately impact interpretations. Meta-analytic work supports the possibility that variations in how ERN is recorded, analyzed, and scored alter interpretations by showing that ERN findings can be moderated by task (Lutz et al., 2021; Martin et al., 2018; Mathews et al., 2012; Pasion & Barbosa, 2019; Riesel, 2019), data-processing pipeline (Clayson, 2020), and scoring procedures (Boen et al., 2022; Pasion & Barbosa, 2019). Further evidence comes from a multiverse analysis of different data-processing pipelines and scoring procedures (3,456 total ERN scores per person) that indicated that different methodological decisions changed the final ERN average amplitudes and variability of scores (Clayson et al., 2021b). Therefore, ERN recorded during different tasks or analyzed using different pipelines or scoring procedures could reflect different neural activity—possibly reflecting differences in the strength of contributions of cognitive, affective, motivational, and motor processes—and could have different psychometric properties.

The present review is agnostic about the optimal way to record, process, and score ERPs. That approach will probably differ depending on the research question and the intended purpose of measurement. Nonetheless, psychometrically informed methods provide the techniques and tools to evaluate important characteristics of any operationalization of an ERP. Routine psychometric analysis of ERP operationalizations will ensure that the neural event of interest is being recorded, processed, and analyzed rigorously.

3. Psychometric Reliability

Generalizability theory (G theory), developed over sixty years ago (Brennan, 2001; Cronbach et al., 1972; Cronbach et al., 1963; Shavelson & Webb, 1991; Vispoel et al., 2018a, 2018b), is a measurement theory with a flexible and multifaceted framework that is well suited to handling the data structures typical of ERP scores. The application of G theory to ERP research has been extensively described elsewhere² (Baldwin et al., 2015; Carbine et al., 2021; Clayson et al., 2021a; Clayson et al., 2021c; Clayson et al., 2021d; Clayson & Miller, 2017a, 2017b). Considering that classical test theory (CT theory) remains popular among psychophysiologists, CT theory will be briefly described, then a brief overview of important aspects of G theory is provided, and the two measurement theories³ are contrasted with an eye toward application to studies of psychophysiology.

3.1. Classical Test Theory

CT theory posits that a person’s observed score comprises two contributions: the true score and an error score (Lord & Novick, 1968; Novick, 1965; Novick & Lewis, 1967). The true score represents the intended characteristic(s) being measured. The error score represents unsystematic measurement error and captures all random inaccuracies—measurement or otherwise (e.g., variations in electrode impedances, physiological or environmental interference, mood variations, fatigue, ambiguity in questions). An assumption of CT theory is that the measurement error is random and uncorrelated with the true score. To estimate the psychometric reliability⁴ of scores, parallel forms of a measure are constructed, and it is assumed that the true scores are the same on both measures, the variability in scores is similar, and error variance is unsystematic (see Vispoel et al., 2018a). If all these conditions hold, then the correlation between parallel forms will represent reliability. Under this assumption of classically parallel forms, the variance of observed scores equals the combination of true-score variance and error variance, and reliability can be represented as the proportion of true-score variance to total observed-score variance (i.e., true-score variance + error variance; see Vispoel et al., 2018a).

3.2. Generalizability Theory

G theory represents an extension of CT theory and provides a comprehensive framework for decomposing the undifferentiated error term used in CT theory. The partitioning of true-score variance and error variance in CT theory is similar to a one-factor design in an analysis of variance (ANOVA). G theory can include this one-factor consideration or expand the estimation of reliability—simultaneously partitioning variances into universe-score variance and multiple sources of error, similar to a multiple factorial design in ANOVA (see Shavelson et al., 1989; Vispoel et al., 2018a). As a result, the emphasis of G theory is on estimating the universe score, and how the universe score is conceptualized provides the framework for estimating psychometric reliability. The use of the term ‘universe score’ in G theory is analogous to classical test theory’s ‘true’ score and emphasizes that any observed score and its reliability depend on the domain(s) (i.e., universe(s)) to which a researcher wants to generalize (Cronbach et al., 1972). Each potential universe of generalization (e.g., items, occasions, raters, tasks) is represented as a facet or a potential source of measurement error within the targeted G-theory design that is analogous to a factor or variable within an ANOVA model. Different items, occasions, raters, tasks, and so on would represent conditions within a facet and would parallel levels within a factor of an ANOVA. Failing to consider all relevant facets and conditions can result in an overestimation of the overall reliability of scores that properly account for all relevant sources of measurement error (Vispoel et al., 2018a). People are considered the object of measurement, and universe and population are logically interchangeable terms in G theory.

There are many potential facets relevant to studies of ERPs. Examples include the task or hardware used to record the ERP, event type, occasion, number of trials retained for averaging, and sensors used for scoring. Levels of the task facet could include a flanker or Stroop task, levels of event type could include congruent or incongruent trials, and sensors used for scoring could include Cz or FCz. The G-theory framework is inherently flexible by design, and the researcher can include any number of facets and levels of each facet. This powerful flexibility facilitates the identification of how much each facet contributes to the variability in observed scores.

Different types of reliability can be estimated based on whether a researcher is interested in characterizing the relative standing of persons (i.e., generalizability coefficient, which uses a relative error term) or the absolute differences in scores (i.e., dependability coefficient, which uses an absolute error term). Generalizability and dependability coefficients represent different ways of conceptualizing error (see Figure 1), with dependability coefficients having a broader definition than generalizability coefficients that encompasses mean differences in scores. Generalizability coefficients capture the ranking of individuals, such as whether the same participants score high at Time 1 and at Time 2. Generalizability coefficients can be useful for ERP researchers when systematic changes are expected between Time 1 and Time 2 (e.g., due to habituation or fatigue), and the occasion variance can be ignored. Dependability coefficients consider not only the ranking of individuals but also any change in the values of scores, such as whether the same participants score high at Time 1 and Time 2 and show similar values at Time 1 and Time 2.

Figure 1.

Venn diagrams for a one-facet design (person × item) are shown. Total observed score variance is shown on the top, and score variance is represented by the circles with each circle representing a facet. The area of overlap between circles represents interactions between facets. These Venn diagrams illustrate the difference between generalizability coefficients ($E{\rho }^2$), which use the relative error variance, and dependability coefficients ($\varphi$), which use the absolute error variance. Generalizability coefficients only consider variances in the error term that impact the relative standing or ranking of participants in the estimation of reliability (i.e., relative error), because the focus is on the suitability of the measurements for comparing participant scores. Therefore, only variances that impact the ‘Person’ circle will be included in the estimation of reliability. For this one-facet design, only the interaction between person and item (${\sigma }_{pi,e}^2$) overlaps with the ‘Person’ circle, shown on the left (see Table 1 for formulas for comparison). However, dependability coefficients consider all sources of variance that impact observed scores in the estimation of reliability (absolute error variance). For the one-facet design, that includes item-score variance (${\sigma }_i^2$) and the interaction between person and item (${\sigma }_{pi,e}^2$; shown on the right).

Two types of reliability coefficients relevant to ERP studies are coefficients of equivalence and coefficients of stability (see Table 1 and Figures 1 & 2). Coefficients of equivalence include measures of internal consistency for items considered randomly equivalent, which refer to how well peoples’ average scores distinguish differences between people (i.e., between-person variability) relative to the scores that contribute to a person’s average (i.e., within-person variability). Thus, coefficients of equivalence represent the extent to which results can be generalized across items. Coefficients of stability capture the stability of measurements over time for items considered randomly equivalent (i.e., temporal stability of measurements) and represent the extent to which results can be generalized across occasions. Coefficients of equivalence can use data from multiple sessions to estimate internal consistency. An advantage of this approach is that occasion-specific variance in scores is partitioned out of internal consistency estimates. Similarly, coefficients of stability can partition the variance that is relevant to internal consistency from the variance related to test-retest reliability.

Table 1.

Generalizability Theory Estimates of Internal Consistency and Test-Retest Reliability

	Generalizability Coefficient ($\mathit{E}{\mathit{\rho }}^2$)	Dependability Coefficient ($\mathbf{\varphi }$)
Single Session
CE: Internal Consistency	$\frac{{\sigma }_p^2}{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi,e}^2$}\!\left/ \!\raisebox{-1ex}{$n_i$}\right.}$	$\frac{{\sigma }_p^2}{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi,e}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_i^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.}$
Multiple Sessions
CE: Internal Consistency	$\frac{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{po}^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.}{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{po}^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{pio,e}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.}$	$\frac{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{po}^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.}{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{po}^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{pio,e}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_i^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{io}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.}$
CS: Test-Retest Reliability	$\frac{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.}{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{po}^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{pio,e}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.}$	$\frac{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi}^2$}\!\left/ \!\raisebox{-1ex}{$n_i$}\right.}{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{po}^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{pio,e}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_o^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{io}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.}$
CES	$\frac{{\sigma }_p^2}{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{po}^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{pio,e}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.}$	$\frac{{\sigma }_p^2}{{\sigma }_p^2+\raisebox{1ex}{${\sigma }_{pi}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{po}^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{pio,e}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_i^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_o^2$}\!\left/ \!\raisebox{-1ex}{$n_o^{\prime }$}\right.+\raisebox{1ex}{${\sigma }_{io}^2$}\!\left/ \!\raisebox{-1ex}{$n_i^{\prime }{n}_o^{\prime }$}\right.}$

Note: For single session data, internal consistency estimated using the ratio of universe score variance (${\sigma }_p^2$), item variance (${\sigma }_i^2$), error variance (${\sigma }_{pi,e}^2$), and the number of trials ($n_i^{\prime }$). Multiple session data considers the universe score variance (${\sigma }_p^2$), item variance (${\sigma }_i^2$), occasion variance (${\sigma }_o^2$), transient error variance (${\sigma }_{po}^2$), specific-facet trial score variance (${\sigma }_{pi}^2$), specific-facet occasion score variance (${\sigma }_{po}^2$), error variance (${\sigma }_{pio,e}^2$), the number of trials ($n_i^{\prime }$), and the number of occasions ($n_o^{\prime }$). These formulas and their derivations can be found elsewhere (see Clayson et al., 2021a; Clayson et al., 2021c; Clayson et al., 2021d; Clayson & Miller, 2017a; Vispoel et al., 2018a). CE = coefficient of equivalence; CS = coefficient of stability; CES = coefficient of equivalence and stability.

Figure 2.

Venn diagrams for a two-facet design (person × item × occasion) are shown. Total observed score variance is shown on the top and score variance is represented by the circles. Each circle represents a facet, and the overlap between circles represents interactions between facets. Generalizability coefficients ($E{\rho }^2$) only consider variance sources in the error term that impact the relative standing or ranking of participants (i.e., relative error) in the estimation of reliability. Therefore, only variances that overlap with the ‘Person’ circle will be included in the estimation of reliability. The contribution of error variance to the generalizability coefficient of equivalence and stability (CES; see Table 1) includes only those facets that interact with person scores: the interaction between person and item (${\sigma }_{pi}^2$), the interaction between person and occasion (${\sigma }_{po}^2$), and the three-way interaction (${\sigma }_{pio,e}^2$). The contribution of error variance to the dependability ($\varphi$) CES includes all facets and interactions between facets (absolute error variance).

3.2.1. Two-Facet G-Theory Analysis.

The following example illustrates how G theory can be used to identify the key sources of error variance that limit the reliability of scores. G theory was used to examine the internal consistency (coefficient of equivalence) and test-retest reliability (coefficient of stability) of N2 ERP component scores following food-related go and nogo stimuli (Carbine et al., 2021; Clayson et al., 2021d). The estimation of reliability used a two-facet model that included items and measurement occasion, allowing the partitioning of observed-score variance into person-related, item-related, and occasion-related variances (see bottom row of Figure 1 for Venn diagram). Estimates of internal consistency, both dependability coefficients and generalizability coefficients, were high ($\ge$ .97), suggesting that N2 scores to food-related stimuli were suitable for studying individual differences. Given the high level of internal consistency, these findings might point to reducing task length, if helpful for the purpose of the study. The relationship between the number of trials included in the ERP average and internal consistency for nogo N2 trials is shown in Figure 3.

Figure 3.

Generalizability coefficients of equivalence (internal consistency, solid line) and coefficients of stability (test-retest reliability, dashed line) for nogo N2 scores using the estimated standard deviation components from Figure 1 in Clayson et al. (2021d; see also Carbine et al., 2021). Generalizability coefficients are plotted as the number of trials included in the reliability estimates increase from 1 to 100 trials.

The generalizability and dependability estimates of test-retest reliability were low, ranging from .48 to .68. An analysis of the variance components clarified that the person × occasion variance (${\sigma }_{po}^2$) was large compared to other sources of variance, essentially placing a ceiling on the test-retest reliability estimates (see Figure 2 for relevant variance components). As shown in Table 1, the impact of person × occasion variance on overall test-retest reliability is not minimized by including more trials. Therefore, attempts to increase the internal consistency of N2 scores will have no beneficial impact on the test-retest reliability because there is no impact on the primary source of error variance: person × occasion variance. As shown in Figure 3, test-retest reliability appears to asymptote just below .70 due to the person × occasion variance not being impacted by the number of included trials. Therefore, this G-theory analysis pinpointed person × occasion variance as the necessary target for improving test-retest reliability of N2 scores. This indicates a need to determine the facets that impact individual differences between sessions, and in the context of the Carbine et al. (2021) study that could include controlling for levels of satiety, time of the testing session, and levels of fatigue across measurement occasions.

3.3. Features of ERP Data

The psychometric analysis of ERPs should reflect the characteristics of the underlying data. Best practices for psychometric analysis include choosing approaches that fit the researcher’s assumptions about the data, rather than forcing the data to fit the psychometric analysis approach. Therefore, it is helpful to characterize the features of a typical ERP score to identify the appropriate psychometric approaches for assessing reliability of ERP scores.

3.3.1. Normal Distribution.

The modal approach to scoring ERP components is to average the waveforms for all epochs together for a condition and a participant and to score the observed ERP component peak at one or more scalp electrode sites. The assumption of this averaging procedure is that any given trial includes the contribution of an ERP signal of interest and random noise (see Clayson et al., 2013). Therefore, epochs are averaged together to minimize the contribution of background noise to the ERP peaks of interest prior to scoring condition $\mathrm{X}$ participant averages. If random noise is constant across the task, then the ERP peaks will likely be normally distributed, assuming a constant ERP signal of interest over the course of the task. As such, ERP component scores are treated as normally distributed within participants⁵.

Time-window mean amplitude scores (e.g., average activity from 200 to 400 ms) are more likely to be normally distributed than peak amplitude or peak latency scores. The arithmetic average of single-trial, time-window mean amplitude scores is the same as the score extracted from the subject average waveforms, because the measurement window is not biased by single-trial noise. However, this is not the case for peak amplitude or latency scores because random noise during any trial waveform biases the selection of a peak for scoring (Clayson et al., 2013; Luck et al., 2021). Therefore, the standard analysis of item scores in G theory is most appropriate for time-window mean amplitude scores and those psychophysiological data that are normally distributed across trials within people (see Other Psychophysiological Data section for a discussion of the use of data splits for peak amplitude or latency scores).

3.3.2. Unbalanced Number of Trials.

During ERP data processing, epoch rejection procedures are commonly used to remove large artifacts from the ERP, such as gross motor artifact. As a result, even if the same number of stimuli is presented to each participant, the number of accepted trials may differ across conditions or participants. Those ERPs that depend on participant responses (e.g., correct or error trials in ERN analysis) may differ in number of trials per condition, because some high-performing participants might commit fewer errors than others.

3.3.3. Intraindividual Variability.

A feature of some ERPs, as with trial-wise data more broadly (e.g., response times; Williams et al., 2021), is that there are between-person differences in the intraindividual variability of ERP trial scores (Clayson et al., 2021c; Clayson et al., 2022c; Clayson et al., 2024; see also Volpert-Esmond, 2022). This intraindividual variability in scores means that some participants may be mischaracterized by group-level reliability estimates because some participants will shower higher or lower data quality than the average participant.

3.4. Assumptions of G Theory vs. CT Theory

The assumptions of G theory are less restrictive than those of CT theory, and CT theory provides a useful comparison as a well-known measurement theory. Comparisons of G theory and CT theory specifically for ERP researchers can also be found elsewhere (Baldwin et al., 2015; Clayson et al., 2021c; Clayson & Miller, 2017a, 2017b), along with other useful comparisons of measurement theories for broader audiences (Brennan, 2000, 2010; Vispoel et al., 2018a).

An assumption in CT theory is that any observed score is the sum of a true score and an error score and that these scores are independent of each other. Whereas CT theory assumes a single, undifferentiated source of random error, G theory partitions observed variance into multiple sources (i.e., facets), including error variance. Therefore, the impact of multiple sources of variance can be differentiated, which provides a more comprehensive and realistic analysis of psychometric reliability than CT theory. In ERP research there are numerous potential sources of error variance (e.g., see discussion of facets above), and the contribution of each source and combination of sources can be pinpointed and optimized for psychometric reliability. For example, a two-facet G-theory analysis (person × number of trials × occasion) identified that despite high internal consistency of ERP scores, individual differences in changes across occasions (person × occasion interaction) was placing an upper limit on test-retest reliability (see Two-Facet G-Theory Analysis section; see also Carbine et al., 2021; Clayson et al., 2021d).

The assumption of classically parallel measures is key in CT theory, and this refers to the notion that item splits (e.g., odd vs. even), forms (e.g., Stroop vs. flanker), or assessment occasions (e.g., time 1 vs. time 2) have equal means, standard deviations, error variances, and true scores. When these assumptions are not met, reliability coefficients can be misleading⁶ (Charter, 2001; Cho, 2016; Cronbach, 1951; Feldt & Charter, 2003; Warrens, 2015). G theory circumvents the assumption of equal variances by explicitly modeling the variance of each task split and has a less restrictive assumption of randomly parallel measures: all items come from the same universe of admissible observations with each item being considered exchangeable with any other (Cronbach et al., 1963; Vispoel et al., 2018a). In ERP studies, Spearman-Brown adjusted split-half reliability is widely used (Brown, 1910; Spearman, 1910), because it is straightforward to use with unbalanced data. However, the use of the coefficient likely leads to an overestimation of the internal consistency of ERP scores (Charter, 2001; Cho, 2016; Cronbach, 1951; Feldt & Charter, 2003; Warrens, 2015), as unequal variances between arbitrary splits are highly likely, particularly in instances when few trials are retained for averaging (see appendix of Clayson et al., 2021a).

Intraindividual variability appears endemic to ERP data (Clayson et al., 2022c; Clayson et al., 2024), and CT theory assumes homogeneity of variances of true and error scores across individuals. Even if one assesses the psychometric reliability of ERP scores from a group of participants and obtains one group-level coefficient that characterizes all participants, the danger is that group-level reliability estimates might mask the low reliability of some participant data. For example, Clayson et al. (2022c) observed that group-level internal consistency estimates mischaracterized subject-level reliability ERN scores (57% of correct-trial scores, 32% of error-trial scores), such that there was a sixfold increase from the smallest to largest internal consistency estimate. The flexible framework of G theory allows the incorporation of participant-specific error variances in the estimations of psychometric reliability and can provide participant-level reliability estimates to characterize each participant’s data (see Clayson et al., 2021c).

Some ERP research employs coefficient alpha (i.e., Cronbach’s alpha) to describe the internal consistency of scores. Cronbach has discouraged the continued use of coefficient alpha for most data⁷ and advocated the use of the standard error of measurement (Cronbach, 2004). However, coefficient alpha remains a well-known and popular estimate from CT theory and is akin to the average of all possible split halves (Cronbach, 1951). Coefficient alpha is poorly suited to ERP data. Coefficient alpha requires all participants to have the same number of trials, which is uncommon due to artifact rejection during data processing. Although ERP data from a group of participants could be truncated so that all participants have the same number of trials for the estimation of coefficient alpha, this practice is ill advised. Doing so would misrepresent the internal consistency of the entire dataset, which are the scores that are likely used for statistical analysis. Any changes over the course of the task or the full intraindividual variability of a person’s ERP scores will not be captured by truncating the ERP data.

3.5. Software for G Theory

The benefits of G theory’s comprehensive framework for understanding and estimating sources of variance come with the added cost of analytic complexity. G-theory equations are built around estimates of variance, but how variance is estimated is left up to the researcher. Some approaches use traditional ANOVA procedures (Brennan, 2001, 2003), structural equation models (SEMs; Vispoel et al., 2023; Vispoel et al., 2018a), or multilevel models (Brennan, 2010; Li, 2023). Variance components can be estimated with these procedures, and then these components can be used in equations for psychometric reliability estimates (see Table 1 for formulas). Although there are several software packages for estimating variance components and directly examining psychometric reliability using generalizability theory, few general-purpose software packages⁸ can handle the unbalanced and multifaceted nature of ERP data and estimate the typical psychometric reliability metrics of common interest to ERP researchers (see Table 1).

The ERP Reliability Analysis Toolbox (ERA Toolbox; https://peclayson.github.io/ERA_Toolbox/) was designed for psychophysiologists (Clayson et al., 2021a; Clayson et al., 2021c; Clayson et al., 2021d; Clayson & Miller, 2017a). The ERA Toolbox uses Bayesian multilevel models to estimate variance components of scores, and these models are well suited for the unbalanced trial-level data common in ERP studies by taking advantage of partial pooling. Participants from the same population are expected to be similar to each other, and partial pooling takes advantage of this by pulling extreme observations closer to the group mean, resulting in more efficient parameter estimation than the arithmetic solution (e.g., Gelman, 2006; Gelman et al., 2012). Therefore, participants with few trials do not strongly impact parameter estimation, and the variance of such scores is pulled toward the group mean.

The ERA Toolbox handles the estimation of variance components and uses these estimates to calculate psychometric reliability using generalizability theory. The toolbox provides traditional estimates of group-level internal consistency (Baldwin et al., 2015; Clayson & Miller, 2017a) and test-retest reliability (Carbine et al., 2021; Clayson et al., 2021d). The formulas for estimating the reliability coefficients are shown in Table 1. The toolbox can also provide the group-level internal consistency of difference scores (Clayson et al., 2021a), which are common in studies of ERPs. Additionally, participant-level data quality and internal consistency coefficients can be estimated by using participant-specific error variances from Bayesian multilevel models (Clayson et al., 2021c). Although the ERA Toolbox was designed with psychophysiology researchers in mind, the toolbox would work well with any trial-wise data, psychophysiological (e.g., fMRI, pupillography, facial electromyography) or otherwise (e.g., response times).

3.6. Applications of Psychometric Reliability

Psychometric reliability is a property of observed scores, but it is not a universal property of a measure (Thompson, 2003; Vacha-Haase, 1998). Psychometric reliability necessarily depends on many factors, such as the sample and population of interest, the hardware or task used for recording, the specific data-processing pipeline (e.g., sensors used for analysis), the contrast within a task (e.g., event-specific scores [error trials, unpleasant trials], difference scores [error minus correct, unpleasant minus pleasant]), or the scoring procedure (e.g., time-window mean amplitude, peak amplitude). A demonstration of adequate reliability in one narrow application (e.g., internal consistency of P300 in healthy undergraduates) cannot be assumed to generalize to other contexts (e.g., internal consistency of P300 in people diagnosed with schizophrenia). Poor psychometric reliability increases the likelihood both of finding non-replicable results and of missing true phenomena (Loken & Gelman, 2017) and even calls into question the validity of their interpretation (Meehl, 1986). These problems are particularly exacerbated in the small-sample studies common in psychopathology research, especially in clinical neuroscience (e.g., Szucs & Ioannidis, 2020) and in studies using ERPs (Clayson et al., 2019). Therefore, psychometric reliability should be routinely evaluated in any study of individual differences, and this practice is consistent with the author guidelines of Psychophysiology (Psychophysiology; Author Guidelines, 2023) and International Journal of Psychophysiology (International Journal of Psychophysiology; Guide for Authors, 2023).

Internal consistency estimates can guide decisions about how to process and analyze ERP scores for studies of individual differences. Different data-processing pipelines impact observed internal consistency, and the impact of some pipelines on the internal consistency of ERN scores, for example, has been examined. Key data-processing stages include event type (correct vs. error trial), filter cutoffs, reference schemes, eye-movement correction procedures, baseline adjustment windows, electrode sites, and amplitude measurement approaches (Clayson, 2020; Klawohn et al., 2020b; Sandre et al., 2020). Studies of ERN data quality also shed light on important methodological choices, because internal consistency is impacted by within-person variability (i.e., data quality). A multiverse analysis⁹ of 3,456 pipelines confirmed that each of the above data-processing stages had downstream effects on the data quality of ERN scores within the same sample of participants (Clayson et al., 2021b). The use of internal consistency estimates to guide data-processing decisions helps to identify optimal pipelines for studying individual differences in ERP scores.

Psychometric reliability estimates can guide the selection of candidate biological markers in clinical trials (Clayson et al., 2021d; Light et al., 2020; Light & Swerdlow, 2020). Ideally, therapeutic targets should show high internal consistency (an index of the capability of scores to distinguish participants) and test-retest reliability (stability of scores over a relevant time frame). For example, in studies of the frequency following response (FFR), there is no gold-standard measurement approach (Krizman & Kraus, 2019). Utilizing those measurement approaches that yield high internal consistency and test-retest reliability will identify the candidate markers most likely to be useful for intervention research. A psychometric evaluation of 18 different approaches to scoring FFR in schizophrenia patients (Clayson et al., 2022b) identified candidate measurements with high psychometric reliability for use in therapeutic interventions (Clayson et al., 2021f; see also Clayson et al., 2022a). Because unreliable data call into question the conclusions of a study, the verification of score reliability should be an early step in the selection of biological markers for use as therapeutic targets. A lack of attention to psychometric reliability has likely limited the widespread application of findings from studies of neural measures to psychopathology research (Lilienfeld & Strother, 2020).

3.7. Other Psychophysiological Data

There is an added challenge in evaluating the psychometric characteristics for psychophysiological data that are not normally distributed at the single-trial level. For example, time-frequency EEG poses such challenges because such data are intrinsically higher-dimensional in nature than ERP data and reliability estimates will require models that reflect that individual-trial scores show a Chi-square distribution. Although the psychometric reliability of time-frequency EEG data has been investigated (e.g., Rocha et al., 2020), the reliability of oscillatory phenomena has received almost no attention in the literature. Multilevel models tailored to the underlying data distributions could be used to estimate variance components for reliability estimation with the formulas in Table 1.

Another challenge is posed by data in which the score from the average of trials is not the same as the average scores of single trials. For example, when EEG trials are averaged together before calculating the average power in a frequency band (i.e., average power that primarily includes phase-locked information), different scores¹⁰ will be obtained than if power is first calculated on single trials and then single-trial estimates are averaged (i.e., total power that includes phase- and non-phase-locked information). Even more commonly, the average of single-trial ERP peaks is almost always higher than the peak of the average of single-trial ERP epochs, due to the variability in latency of the single-trial peaks. G theory has been developed to work on parallel item task splits, which could be appropriate for estimating reliability of average power or peak amplitude scores (see Vispoel et al., 2022). The use of task splits, instead of single-trial scores, would benefit from the mitigation of background EEG noise on power or peak amplitude scores afforded by averaging many trials together. When conducting a G-theory analysis of item splits, the scores derived from split data become the units of analysis. For example, average power from splits comprising 100 trials each could be extracted from data with 500 trials to estimate reliability and determine the optimal task length as a function of splits (i.e., 100 trials). The size of the splits is determined by the researcher and the appropriate length of the splits will depend on type of data.

4. Psychometric Validation

Validation research is ultimately about epistemology and concerns the methodological effort to evaluate the relationships among tests and the consequences of their use. The focus on the application of scores means that a measure can have different validities for different applications (Fahrenberg et al., 1986). For example, an ERP score might yield different validities for inferences from a sample with psychiatric illness than for inferences from a sample of healthy comparison participants.

4.1. Construct Validation

Construct validity refers to whether evidence supports the interpretation or meaning of scores for an intended construct (Cronbach & Meehl, 1955). Convergent and discriminate validity are two types of construct validity relevant to ERPs. Whereas convergent validity describes the extent to which two measures of the same construct produce similar inferences, discriminate validity concerns the extent to which two measures of different theoretically relevant constructs produce different inferences (Campbell & Fiske, 1959; Fiske, 1971). For studies of ERPs, convergent and discriminate validity are highly relevant for attempts to generalize ERP findings across paradigms—particularly for ERP components referred to via generic nomenclature. For example, different paradigms used to record N2, a negative-going, scalp-recorded ERP component that peaks between 200 and 350 ms after stimulus onset, may elicit different processes, such as attention, cognitive control, mismatch, novelty, or sequential matching (see Folstein & Van Petten, 2008).

Studies of convergent and discriminate validity can identify instances in which inferences might be based on jingle or jangle fallacies. A jingle fallacy occurs when two measures assess different constructs but are assumed to measure the same construct because the measures have the same or similar names, and a jangle fallacy occurs when two measures assess the same construct but are assumed to measure different constructs because the measures have different names (Kelley, 1927). An example of a jingle fallacy could be assuming that the N2 component findings from two different paradigms measure the same process, because they are both “N2s”. An example of a jangle fallacy could be to assume that N2 is distinct from ERN when recorded during the same paradigm because different component names are commonly used in some contexts.

A practical approach for examining the convergent and discriminate validity of ERPs is the use of a multitrait-multimethod matrix, wherein theoretically similar and dissimilar measurements are correlated (Campbell & Fiske, 1959). Bivariate correlations among scores that purportedly measure the same construct should be high (convergent validity), and correlations among scores that measure different constructs should be low (discriminate validity). This approach has been applied to ERPs in the form of an alternatively named multicomponent-multitask matrix (Riesel et al., 2013). Multicomponent refers to different ERP components, and multitask refers to different tasks used for recording ERPs.

A multicomponent-multitask matrix was used to evaluate the convergent and discriminate validity of ERN and error positivity (Pe) recorded during flanker, Stroop, and Go/Nogo tasks (Riesel et al., 2013). The error positivity component (Pe) is a slow, tonic waveform that follows ERN and is larger for error trials than for correct trials (Nieuwenhuis et al., 2001; Overbeek et al., 2005). Whereas ERN is considered to index early error detection, Pe is considered to reflect later error awareness (Steinhauser & Yeung, 2012; Ullsperger et al., 2010; Wessel, 2012; Wessel et al., 2012). Convergent validity of ERN and, separately, of Pe was supported by showing that each component correlated with itself across paradigms, and discriminate validity of the two components was supported by showing numerically smaller correlations between ERN and Pe within a task than between each component and itself across tasks (Riesel et al., 2013). A registered report provided a direct replication that supported these conclusions, and it conceptually replicated the findings by statistically comparing the correlations (Clayson et al., in press). Although the strength of the correlations was modest for ERN (.45 to .53) and Pe (.41 to .67)¹¹, statistical comparisons supported the convergent and discriminate validity of these two components. These findings suggest that the interpretations of ERN and Pe are not falling prey to jingle or jangle fallacies.

Clayson et al. (in press) extended the findings of Riesel et al. (2013) by examining N2. Comparisons of ERN and N2 did not show evidence of discriminate validity, and the correlations between ERN and N2 were similar in magnitude (.49 to .60) to correlations of convergent validity of ERN (.45 to .53). These findings suggest that studies that rely on distinct interpretations of the functional significance of ERN and N2 could fall prey to the jangle fallacy. In fact, ERN and N2 have both been conceptualized as indices of discrepancy detection, with ERN related to response conflict and N2 related to stimulus conflict (Larson et al., 2014). The findings of Clayson et al. (in press) could provide weak support for the convergent validity of the two ERPs.

4.2. Generalizability Theory for Validation

G theory provides a helpful framework for evaluating construct validity. The inclusion of different facets of measurement in G theory analysis permits the examination of score consistency across different conditions of measurement. Ideally, scores would be consistent between different items, methodological parameters, paradigms used for recording, or groups. For example, if ERP scores show different between-person variances for different items (correct vs. error; pleasant vs. unpleasant; rare vs. frequent), experimental paradigms, or groups, then these differences might indicate the component is not consistently measuring the same construct across levels of a facet.

G theory can pinpoint facet-related differences in error variances, highlighting the sources of variability. Some sources of measurement error (i.e., within-person variability) could represent systematic differences between people. For example, particular participant groups might show greater within-person error variances than other groups, or certain paradigms might tend to have more variability in an ERP response than other paradigms. Other sources of measurement error might include methodological differences (e.g., filtering, ocular artifact correction) or interactions between facets (e.g., patient vs. control differences in error variances larger for an easy task than a difficult one). By identifying the sources of error variance, steps can be taken to minimize their impact on scores and consequently increase the validity of the measurements. Thus, G theory can be used to examine how scores generalize across conditions and facets of measurement and guide decisions for refining measures.

5. A Note on Psychometric Validity

Traditionally, validity is defined in terms of how a test is used. Some familiar definitions of validity¹² include the extent to which a test measures what it purports to measure (Kelley, 1927), how well test scores correlate with scores on tests from other theoretically relevant constructs (Cronbach & Meehl, 1955), or how appropriate a test label is (Messick, 1989). These familiar definitions of validity emphasize the application of test scores (i.e., epistemology), descriptive theories of meaning, and correlations of a test with other tests or criterions. However, questions of psychometric validity are primarily theoretical ones and cannot be solely answered by methodological investigation.

A simple and intuitive definition of validity is that the attribute of interest exists and that changes in the attribute cause changes in measured observations (Borsboom, 2005; Borsboom & Mellenbergh, 2007; Borsboom et al., 2004). This alternative definition shifts validity theory away from epistemology, meaning, and correlation to ontology, reference, and causality (Borsboom et al., 2004). A theoretical attribute can include neural processes, such as hemodynamic or electromagnetic activity in the brain—the specific brain activity typically of interest is not directly observable, and a researcher requires inferential processes to judge differences in brain activity among people.

A measurement model describes how attributes cause changes in observations. For example, ERP waveforms represent a manifestation of activity from numerous postsynaptic potentials (see Luck, 2014). Psychophysiologists might assume that differences in observed ERP peaks are systematically related to differences in underlying brain activity (theoretical attribute) between people. In addition, a central aspect of psychophysiology involves studying psychological and biological phenomena together and determining individual differences in those phenomena. Biological phenomena (brain activity) have been advocated as measures of psychological phenomena, but clear measurement models are required to justify how psychological phenomena cause changes in brain activity or vice versa. Failing to do so (if it is even possible¹³) can be construed as a problem of test validity, which concerns any attempt to connect data to attributes.

The problem of validity is a fundamental and substantive issue for all fields that use measurements. Studies that evaluate psychometric reliability and validity can practically advance measurement practices, but such studies should be motivated by theories of the validity of attributes. The present review focuses on the evaluation of psychometrics, but the validation process cannot replace the theoretical validity problem.

6. Additional Psychometric Concepts for Clinical Application

6.1. Standardization

Many different projects are moving performance-monitoring ERP components—such as ERN—toward standardization (e.g., Clayson, 2020; Klawohn et al., 2020b), optimization (e.g., Clayson et al., 2021b; Sandre et al., 2020), and clinical application (e.g., Clayson et al., 2022c; Hajcak et al., 2019). For example, a large ERN data set was published as normative (Imburgio et al., 2020), despite incomplete understanding of the psychometric validity of ERN in the field (see commentary Clayson et al., 2021e). Such efforts to establish normative datasets are challenged by a lack of standardization of methods for recording ERN and a lack of an understanding of the methodological choices that impact ERN scores (Clayson et al., 2021e), which can be advanced by studies that evaluate psychometric reliability and validity.

On the path to clinical application and publication of normative databases, rigorous standardization of paradigms, administration, and data analysis is critical to achieving dependability (distinct from dependability in G theory), which is a fundamental concept in psychological testing. Dependability in neuropsychological assessment implies that the information from psychological measures is invariant across time and situations (Russell et al., 2005). For example, during neuropsychological evaluations, a client’s scores are compared to normative data on standardized assessment measures, and these comparisons are used for individualized diagnosis and treatment planning. Normative scores provide information about how an individual’s performance compares to a reference group.

A keystone of all neuropsychological evaluation is the standardization of test administration, scoring, and interpretation, because nuisance factors must be mitigated to make observations comparable across administrations. A failure to properly standardize measures leads to spurious inferences, because deviant scores from the normative sample could be extreme due to any number of factors in the administration or scoring of the measures, rather than differences in the construct of interest (Bigler & Dodrill, 1997). Therefore, strict standardization in recruitment (c.f. sample of convenience), assessment, and scoring should be followed when creating normative databases. Failing to standardize ERP assessment before providing normative databases has deleterious consequences for researchers and clinicians who may prematurely rely on them (Clayson et al., 2021e). After proper standardization and verification of psychometrics, G-theory approaches are well suited to assessing the reliability of optimal cut points (i.e., criterion scores) for distinguishing normal from abnormal function (e.g., Brennan, 2001; Vispoel et al., 2018a); for example, identifying an amplitude threshold to distinguish abnormally large ERN.

6.2. Measurement Dimensionality and Invariance

Psychometric validation includes the examination of measurement invariance, which is a demonstration of psychometric equivalence across relevant conditions, occasions, or groups that theoretically should not impact scores (Meredith, 1964; Struening & Cohen, 1963; Vandenberg & Lance, 2000). The demonstration of measurement invariance justifies the comparison of conditions, occasions, or groups and ensures that differences in scores are not primarily due to artifacts of the measurement process (e.g., involvement of different processes). Typically, multigroup confirmatory factor analysis is used to examine measurement invariance of questionnaire data, and the loading of each item onto a latent factor is examined (see Putnick & Bornstein, 2016; Steinmetz et al., 2009).

However, in the typical ERP study, univariate analyses—not summed or average item scores from self-report data—are the primary foci. Analyses could include examining the relationship between an ERP component and a measure of impulsivity, clinical group differences in ERP amplitude, or between-condition ERP differences within persons. Unlike items on a questionnaire, ERP scores are typically considered parallel in that there is no special meaning to the first vs. second vs. third trial¹⁴. For example, an ERP to an incongruent trial is considered exchangeable with any other incongruent trial in the task, and parallel items are an assumption of the averaging approach commonly used. A motivation of evaluating measurement invariance is to ensure that observations of condition, occasion, individual, or group differences are due to differences in the construct of interest rather than differences due to measurement issues.

Clinical studies of ERPs often focus on interpretations of how groups differ in ERP measures. The assumption is that any difference between ERPs is due to differences in the construct of interest. However, differences between groups could be due to differences in measurement error rather than differences in the construct of interest. High measurement error can cause magnitude (i.e., attenuated or exaggerated effects) and sign errors (i.e., change in pattern of effects, such as patients showing larger rather than smaller ERPs than controls) in between-group comparisons (Clayson et al., 2021c; Flegal et al., 2017; Gelman & Carlin, 2014) and in examinations of individual differences (Cooper et al., 2017; Hedge et al., 2017; Loken & Gelman, 2017; Rouder & Haaf, 2018; Seghier & Price, 2018). The observation of more error variance (i.e., within-person variability) in a clinical group than in a control group could be due to clinical factors related to the construct of interest or due to measurement issues (e.g., noisier data due to nuisance factors unrelated to the construct, such as movement artifact, attention, and fatigue). Directly examining between- and within-person variances can shed light on whether differences¹⁵ are due to a construct or due to measurement.

An extension of generalizability theory could include a straightforward statistical comparison using a multilevel location-scale model that examines between- and within-person variances. Traditional, location-only multilevel models include fixed and random effects to estimate conditional means but hold residual variance in the model constant across fixed/random effects (i.e., homoskedasticity). Multilevel location-scale models expand the multilevel structure to the scale portion of the model (i.e., variances of the data), permitting the simultaneous modeling of means (i.e., location) and variances (i.e., scale). Studies using location-scale models have been successfully applied in ERP studies (Clayson et al., 2021c; Clayson et al., 2022c; Clayson et al., 2024; for commentary on the use of these models in studies of ERPs, see Volpert-Esmond, 2022) and can directly examine condition-related differences (e.g., trial type, clinical group, occasion) differences in within-person variances and determine the impact of covariates on those variances.

6.3. Clinical Applications

The process of evaluating psychometric reliability and validity has implications for identifying biomarkers, which is an important focus of the NIMH Research Domain Criteria (RDoC) initiative. The RDoC initiative emphasizes examining the feasibility of psychological and neurophysiological measures of dimensional constructs with an eye toward clinical prediction and application (Cuthbert et al., 2023; Kozak & Cuthbert, 2016). The search for biomarkers of mental illness aims to improve precision medicine in psychiatry. This search will require explicit attention to the psychometrics of task measurements, psychophysiological or otherwise, applied to distinguish differences between people.

EEG and ERPs are often used in the search for tractable biomarkers because they are direct, non-invasive, and portable measures of neural activity. Some EEG/ERP features show promising clinical utility, such as ERN in anxiety (e.g., Hajcak et al., 2019), reward positivity in depression (e.g., Klawohn et al., 2020a), and auditory steady state responses in schizophrenia (Molina et al., 2020). However, their utility is limited by their psychometric properties and by the use of tasks that have not been optimized for clinical populations. As one example, standard practice employs tasks that show robust healthy vs. clinical group differences, but these tasks may not be well-suited to examining dimensional relationships (see Infantolino et al., 2018). A focus on psychometric reliability can inform which tasks to use for distinguishing people and can provide a basis for determining the length of tasks. Unfortunately, psychophysiological data rarely receive close psychometric scrutiny, which consequently leads to the reliability of many measures being virtually unknown for clinical or even healthy samples. Unless the evaluation of reliability becomes standard practice, it is hard to imagine any biomarker gaining meaningful clinical traction in precision medicine.

The G theory formulas provided in Table 1 and implemented in the ERA Toolbox (Clayson & Miller, 2017a) can be used to optimize tasks for clinical psychophysiology research. Because unreliable data call into question a study’s conclusions, estimation of score reliability should be done before extensive data collection is underway, and the verification of score reliability should be an early step in data analysis. The demonstration of psychometric reliability in RDoC-inspired research is a precondition for between-subjects measurement comparison—including matching the “right individuals” to the “right treatment”. Guiding biomarker evaluation through psychometrics will pave the way for better selection of biomarkers and task development, ultimately improving the clinical utility of these biomarkers in precision medicine.

7. Conclusion

Tools needed to rigorously assess the psychometric reliability of ERP scores and to validate ERP measures are now readily available for improving the psychometrics of ERPs. Generalizability theory is a multifaceted and flexible framework that is well suited to the psychometric analysis of ERPs and other trial-wise scores, and tractable software is already available to implement this measurement framework in studies of psychophysiological measurements (e.g., ERA Toolbox, https://peclayson.github.io/ERA_Toolbox/). The importance of routine evaluation of psychometrics in psychophysiology cannot be overstated; ERP work can only be as replicable as ERP measures are psychometrically reliable. As authors, journal reviewers, or grant reviewers, we have a responsibility to foster the integrity of the field by holding research to appropriately high standards concerning psychometrics. With the profound implications that psychophysiological research can have on theory and practice, there is simply too much at stake to ignore the crucial processes of evaluating psychometric reliability and validity.