Characterizing the brain’s dynamical response from scalp-level neural electrical signals: a review of methodology development

Abstract

The brain displays dynamical system behaviors at various levels that are functionally and cognitively relevant. Ample researches have examined how the dynamical properties of brain activity reflect the neural cognitive working mechanisms. A prevalent approach in this field is to extract the trial-averaged brain electrophysiological signals as a representation of the dynamical response of the complex neural system to external stimuli. However, the responses are intrinsically variable in latency from trial to trial. The variability compromises the accuracy of the detected dynamical response pattern based on trial-averaged approach, which may mislead subsequent modelling works. More accurate characterization of the brain’s dynamical response incorporating single trial variability information is of profound significance in deepening our understanding of neural cognitive dynamics and brain’s working principles. Various methods have been attempted to address the trial-to-trial asynchrony issue in order to achieve an improved representation of the dynamical response. We review the latest development of methodology in this area and the contribution of latency variability-based decomposition and reconstruction of dynamical response to neural cognitive researches.

Characterization of the brain’s dynamical response and its variability from instance to instance

Characterization of dynamical brain response to stimuli in cognitive tasks forms a cornerstone in neurocognitive research. To examine how this complex dynamical system behaves and how it is associated with function and cognition, neuroscientists usually give the brain a ‘kick’ (e.g., sensory input) and observe its neural response, like physicists examining the dynamics nature of a pendulum (Fig. 1b). The response is not merely an increase or decrease of activity strength in some kind, but displays a rich structure of dynamical system response pattern at the time scale of millisecond, which can be used to infer the underlying architecture and configuration of dynamical neural system at various levels (Deco et al. 2008; Graben et al. 2008; Kiebel et al. 2006, 2008). Electroencephalography (EEG) technology provides a non-invasive means to measure such dynamical neural responses with sufficiently high temporal resolution. Since EEG signal contains a large amount of spontaneous background activity, the pattern of the response activity to the ‘kick’, also known as event-related potential (ERP), becomes visible only after averaging multiple trials, which cancels out the strong spontaneous activity (Fig. 1c). ERP waveform (Fig. 1c), typically showing a delicate response pattern of dynamic oscillators, has engendered a large amount of research on brain-behavior relationships (dating back to the 1930s (Davis 1939)). At scalp level, the average ERP approach has hitherto remained the main approach to obtaining the dynamical response pattern with sufficient temporal resolution, and it has been demonstrated to be a powerful tool for investigating the neural dynamics-cognition relationships. Manipulations of cognitive processes (e.g., to perform fast or to perform accurate) or stimulus properties (e.g., luminance) can specifically alter an ERP peak or trough in a temporally fine-grained manner, showing a fascinating psychophysical phenomenon (Tobimatsu and Celesia 2006). Such a trial-averaged ERP approach has given birth to fruitful research outcomes with respect to the neural mechanisms of perception, emotion, memory, language, and various other cognitive processes.

Fig. 1.

EEG as a tool to characterize the brain’s dynamical response. a A typical EEG experiment paradigm in which discrete events are presented to the subject to elicit brain response while EEG signal is being recorded continuously. b Eliciting brain response by stimulus can be analogized to hitting a pendulum and observing its dynamic response. c The average ERP method assumes that a specific response activity is evoked by stimulus and is added to the spontaneous activity. By averaging a number of trials aligned to stimulus onsets the spontaneous activity will be cancelled out and the evoked response will remain. However, due to the trial-to-trial variability of brain response (represented by shifting blue peaks), the average ERP may end up showing a blurred version of the response pattern (bottom). d Real EEG data showing that there are different sub-components in the single trial ERPs with differential latency variabilities. The data are single trials ERP sorted by P3 latencies from electrode CPz of a single subject from a face recognition task (Rellecke et al. 2012)

This figure is taken from Ouyang (2020). Permission has been obtained

Trial-averaging appears to be a powerful approach to characterizing the dynamical responses to external stimuli. However, the averaged waveform is not an accurate representation of the dynamical responses due to brain response variability (Fig. 1c). Unlike a pendulum, the brain is an active dynamical system that responds variably to the same ‘kick’. This variability may stem from neural functional mechanisms (e.g., adaptation and learning (Brooks et al. 2015; Collins and Frank 2018; Dhawale et al. 2017), from dynamical nature of multilevel neural working (Mendonca et al. 2016), or simply from noise (Faisal et al. 2008). This renders the trial-averaged ERP inaccurate in describing brain response due to the blurring effect, as explained in Fig. 1c. In the worst case, the inaccuracy could lead to misleading conclusion in neurocognitive research (Ouyang et al. 2016; Stokes and Spaak 2016). From the dynamical system point of view, the distorted representation of system response will also render all the inferences of system architecture and configurations questionable (Kashyap et al. 2019).

Strictly speaking, the brain’s dynamical response pattern only genuinely unfolds in a single trial. It is thus important to characterize the response pattern at single trial level. However, this endeavor is greatly hindered by a fundamental challenge—how can a genuine response activity be differentiated from the overlapping, spontaneous, self-sustaining activity in a single trial? In fact, the spontaneous activity—the activity that reflects the sustained neural dynamics in functional operation—occupies the major power in a single trial brain EEG (Cole and Voytek 2017), even after the artifactual signals are removed. This overwhelmmingly dominant ongoing activity makes it difficult to identify the exact pattern of an externally elicited response. These inherent neural data features create a “single trial vs. average” dilemma in the characterization of dynamical brain responses using EEG technology in cognitive neuroscience research: In a single trial, the dynamical response pattern is genuinely preserved but is mixed with strong spontaneous activity; In the average ERP, the spontaneous activity are effectively canceled out but the dynamical response pattern are distorted by the trial-to-trial variability. Advanced signal processing technology and a theoretical framework for addressing this dilemma are therefore needed in view of the importance of knowing what the dynamical response really looks like in a single trial for a deeper understanding of various neural cognitive mechanisms.

No two brain responses are the same. The brain’s fundamental ability of adapting to the environment and thriving lies in its flexibility and malleability of its internal system and behaviors. Therefore, being variable is one of the defining features of the brain dynamical systems. It has been proposed by many dynamical system researchers that the brain system lies in a critical state that balances reliability and variability in which various functions are best achieved and maintained (Cocchi et al. 2017; Wang et al. 2016, 2019). As such, the variability information of brain dynamical response provides another key channel to investigating the core mechanisms, aside from the pattern of the dynamical response per se. Various neurophysiological factors can contribute to response variability. The key question is, to what extent is cognitive behavior variability reflected in single trials ERP?

A clear answer to this question is the first step that stimulates and illuminates further development of single trial ERP-based characterization of brain dynamical responses. Reliably obtaining the variability information of the brain dynamical response remained a challenging topic for a long time, again, due to the strong spontaneous EEG activity that hampers its reliable estimation. Nevertheless, statistically, the relationship between trial-to-trial variability of brain response (e.g., amplitude, latency, oscillatory power) and various external covariates (e.g., response speed, correctness, reward signal) has been extensively confirmed (Bridwell et al. 2018). In fact, numerous recent findings have revealed a strikingly close relationship between single trial ERP variability and complex real time cognitive processes ranging from memory/evidence-based decision making (Loughnane et al. 2016; Ratcliff et al. 2016) to real time dynamics of expectation, feedback processing, and cognitive control in the dynamic reinforcement learning process (Collins and Frank 2018; Frank et al. 2015). These concrete findings have firmly pointed out that the high temporal resolution, non-invasive technology of EEG is able to reveal rich information associated with complex cognitive variability from trial to trial. However, confirming the functional relevance of the trial-to-trial variability is still substantially different from precisely characterizing the variability pattern. The latter is more important for informing and validating dynamical modelling studies. For example, knowing the significant correlation between single trial neural response strength and reaction times does not mean knowing the distribution pattern of the neural response strength across trials (which can be Gaussian, ex-Gaussian, or Poisson) because the samples may be too few or amount of noise may be too strong to reliably infer the distribution patterns. Such distribution patterns are crucially important as they reflect the properties of the underlying model that generates the responses.

Characterizing the trial-to-trial variability of brain response as revealed by ERP has been an increasingly trendy research in recent years. The two major aspects of variability information are amplitude and latencies, which have both been shown to be highly variable (Ouyang et al. 2015), also see Fig. 1d). The variability of the morphology of the entire response pattern of spatiotemporal features across trials has been less attended, which we also believe to be an important aspect to look into. Regarding the more detailed feature of the variability pattern, it has been shown that the early ERP components, such as P1, N1, P2, that reflect the early stages of low level perception, appear to be less variable, whereas the late components, such as P3, N400, P600, that reflect high-level cognitive processes, appear to be highly variable (Ouyang et al. 2015; Wang et al. 2015). The variability is ubiquitous in all task paradigms (Ouyang et al. 2017). Ample evidence of cognitive relevance of the variability in single-trial ERPs has been reported in the ERP literature (Arazi et al. 2017; Loughnane et al. 2016; Pisauro et al. 2017; Stefanics et al. 2018). This further suggests a need to shift from the conventional trial-averaging approach to more advanced approaches of characterizing dynamical responses after explicitly addressing the latency asynchrony issue. The major reasons for this shift are: (1) an average pattern mis-represents the dynamical responses in single trials (Fig. 1). And such mis-representation of neural response patterns could mislead researchers’ understanding of neural working mechanisms (Stokes and Spaak 2016) and behavioral effects on neural system (Ouyang et al. 2016), and many other aspects such as precise timing, subtle effects, and intricate dynamics. (2) Rich information about the dynamics of neurocognitive and functional processes is only accessible in single trials.

Significance of addressing the issue of trial-to-trial variability in brain response

According to the issues related to trial-to-trial brain response variability that we elaborated above, we argue that addressing them will have profound benefits in many domains in neural cognitive research. (1) Obtaining a more accurate pattern of dynamical response by compensating the variability effect will provide important information for inferring the dynamical and functional mechanisms of neural systems. (2) Obtaining a more accurate response pattern is beneficial to better characterization of trial-to-trial variability, as the rectified pattern can serve as a better template. The trial-to-trial variability is also an important feature dimension of the dynamical system response. (3) The improved dynamical response pattern and trial-to-trial variability information provide new channels for studying the brain-cognition relationships, cross-sectional differences, and individual differences from the dynamical system’s perspective. Below we provide an overview of the development of methods that are oriented to study more precise representation of brain’s dynamical response pattern and its variability measured by the tool of ERP.

Current state of methodology

The pursuing of a more accurate characterization of brain’s dynamical response beyond simple averaging has a long history and is still advancing. Since a time marker-locked average ERP is a blurred version of the dynamical response, de-blurring is a major approach to restore the response pattern. The earliest relevant attempt in this line dates back to half a century ago (Woody 1967). Woody pioneered the method of identifying the single trial latencies of ERP components and re-synchronizing single trials according to the estimated latencies with the aim of obtaining a ‘rectified’ ERP, thus better representing the dynamical response pattern. Since then, various methods and approaches have been attempted. In the following, we summarized the developments in this area including the latest ones.

Averaging after resynchronization

Resynchronization is the core procedure for dealing with the asynchrony problem. A coarse approach is to identify the single trial latencies of ERP and re-synchronize single trials to the identified latencies instead of to stimulus onsets and obtain a new ERP (Patterson et al. 2000; Pomalazaraez and Mcgillem 1986; Woody 1967). This approach can adjust large ERP components such as P3 (Kutas et al. 1977; Patterson et al. 2000; Pomalazaraez and Mcgillem 1986; Spencer et al. 2000). However, resynchronizing single trials to the latencies of a late ERP component is as problematic as stimulus-locked averaging, because an ERP is not simply a homogeneous ensemble temporally locked to a single time event (Fig. 1c). For instance, a reaction time (RT)-locked average ERP will simply blur stimulus-locked portions (Berchicci et al. 2016). The multi-compositional nature of ERP makes simple resynchronization of single trials to one component’s latency an ineffective approach to improving the detection of dynamical response pattern. To address this issue, methods that decompose ERP into multiple components with differential trial-to-trial variability had been developed.

Time marker-based ERP decomposition

Neural cognitive processes are functionally modular—they can be divided into, for example, perception, central cognition, and response/execution (Hurley 2001). Neural activations associated with different sub-processes have different degrees of trial-to-trial variability. They may be locked to stimulus onset, response, or neither-nor ((Ribeiro et al. 2016; Schiff et al. 2014; Verleger et al. 2014), or refer to illustration in Fig. 2), meaning that resynchronization could be and should be done separately on different sub-components. One straightforward idea is to decompose these several component clusters with different latency variabilities and resynchronize them separately (Fig. 2). The earliest attempt at such decomposition was simply to separate an ERP into a stimulus-locked component cluster and a response-locked component cluster based on markers of stimulus onsets and reaction times, which can be done with mathematical derivation (Bardy et al. 2014; Dandekar et al. 2012; Hansen 1983; Smith and Kutas 2015a, b; Takeda et al. 2008; Yin et al. 2009; J. Zhang 1998). In essence, these time marker-based decomposition methods all share the same mathematical core: a general linear model (GLM) in which the time markers serve as the regressors (independent variables), the raw EEG data serve as the dependent variables, and the waveform associated with each regressor is the coefficient vector to be solved in the GLM framework. The mathematical derivation for the component decomposition can be summarized below.

Fig. 2.

Illustration of ERP decomposition and reconstruction. It is assumed that there are three component clusters in single trials ERPs: stimulus-locked cluster (blue), central cluster (red), and response time-locked cluster (green). To obtain a better representation of the brain response pattern at a single trial, the three clusters need to be decomposed (middle panel) and separately re-synchronized (right panel). (Color figure online).

Figure was adapted from Ouyang et al. (2016). Permission has been obtained

Assuming two activation components co-existing in every single trial, temporally locked to different events, the EEG trace can be described as:

$$EEG\left(t\right)=\sum _{\tau =1}^{T_1}{C}_1(\tau )X_1\left(t-\tau \right)+\sum _{\tau =1}^{T_2}{C}_2(\tau )X_2\left(t-\tau \right)+\epsilon .$$ 1

where C is the component waveform function and X is timing functions coding the time markers of the components with value one (e.g., stimulus onsets or response times) and others zero. T₁ and T₂ indicate the duration of components and ε is the noise term. In reality, more than two components locked to different time events can be modeled and (1) can be simply extended to such cases. Equation (1) can be written in a matrix form:

$$\text{EEG}=\text{X}·\text{C +}\epsilon .$$ 2

With the information of the time markers X, the least square error-based solution of the components can be expressed as Eq. (3) below (Dandekar et al. 2012), which serves as the solution for the decomposed components:

$$\text{C}={({\text{X}}^{\text{tX}})}^{-1}·{\text{X}}^{\text{t}}·\text{EEG}.$$ 3

One obvious limitation of this approach is that it requires RT to be included in the experiment, a requirement which many experiments with covert responses (e.g., internal counting) cannot fulfill. Even in the ERP data with both stimulus onsets and RT markers (or other markers), an issue may arise as to whether RT precisely represents the latency of the late latency-variable ERP components. The second limitation, which is much subtler and less widely known, is a noise amplification issue (Ouyang et al. 2015). Specifically, when two sets of markers (e.g., stimulus and RT) have very small inter-marker jitter across trials, the mathematical solution of the two marker-locked components are two complementary waveforms (with large amplitude) that are clearly not biologically plausible. This is due to the close-to-singularity of the covariance matrix of the two regressors (Ouyang et al. 2015). Similar issues exist in dipole source localization when different dipole sources have a high spatial correlation, in which case the source temporal activity will have complementary patterns resembling amplification of noise (Wolters et al. 1999). A solution to this issue that is both practically and theoretically sound has yet to be found. From practical perspective, a common solution is introducing regularization, the configuration of which is however usually dependent on specific circumstances.

ERP decomposition without time marker

Motivated by the potential issue that the external time markers may not precisely represent an underlying component’s latency, and are often not available, researchers have proposed several methods to decompose ERP without fully relying on time markers (Ouyang et al. 2015; Takeda et al. 2010; Truccolo et al. 2003; Wu et al. 2014). The basic approach is to estimate the latencies of the components whose single trial latencies are not or inaccurately represented by external time markers, thus creating ‘time markers’ that are to be fed into the time marker-based ERP decomposition methods. For instance, recognition of a word during sentence comprehension elicits functionally differentiable processes such as low-level visual, semantic and syntactic processes, which are presumably variable in latency that is difficult to measure externally. N400 and P600 are two neural activations associated with subprocesses of language processing. Wang et al. (2015) attempted to estimate single trial latencies of these two components and separate them (F. Wang et al. 2015) based on the assumption that the two components have differential trial-to-trial latency variability. The decomposition using the estimated latencies was similar to the marker-based decomposition whereby each type of marker is coupled with a specific component, but is based on a more robust iterative scheme (Wang et al. 2015). The work thus demonstrated that the precision of single trial latency estimation sufficed for the decomposition. However, these non-time marker-based methods still inherit the limitation of marker-based methods as described above (e.g., noise amplification), with an additional limitation resulting from the inaccuracy in the single trial latency estimation.

General issues and challenges

Although the distortion issue in using trial-averaged ERP to represent the dynamical response is widely known (Jung et al. 2001; Kutas et al. 1977; Makeig and Onton 2011; Ouyang 2020; Ouyang et al. 2017; Sassenhagen and Bornkessel-Schlesewsky 2015; Saville et al. 2015; Walhovd et al. 2008), there is still not a commonly accepted solution that has come into play in the community. Researchers are still mainly using trial-averaged ERP as a representation of the brain dynamical response. A sound framework for improving the representation of the dynamical response is strongly needed. Nevertheless, recent development has seen some promising methods emerged. In the following, we continue to summarize some recent methodological developments that have attempted to address these issues.

Reconstruction of a more accurate representation of the dynamical response

With the advancement of signal processing techniques and theoretical modelling, a substitute for trial-averaged ERPs should be sought to more accurately characterize the dynamical neural response at the single trial level. Obtaining this substitute certainly needs to (1) incorporate trial-to-trial variability information and (2) differentially treat the sub-components with different variability features. A new framework was recently proposed by Ouyang et al. (2016, 2020) that was designed to obtain such a substitute. The procedure, called ERP reconstruction, comprises the following steps: (1) decomposing an ERP into different subcomponents with different variability; (2) obtaining the latencies of each subcomponent at single trial level; (3) separately re-synchronizing each subcomponent according to their own single trial latencies (either estimated or prescribed) in the temporal axis with respect to stimulus onset (Fig. 2). The moving of each single trials is referred to the median latency of all trials, i.e., trials with latencies smaller than the median should move rightward, vice visa; and (4) summing up the re-synchronized subcomponents and obtaining an ERP that has been adjusted for the blurring effect. The procedure is illustrated in Fig. 2. In principle, the reconstructed ERP is a more precise reorientation of the dynamical response pattern occurs at single trial level. Comparatively, the stimulus-locked averaged version misrepresents the pattern, as the blurred portion (Figs. 1 and 2) does not actually occur in a single trial. The reconstruction effect in a real EEG dataset is shown in Fig. 3.

Fig. 3.

Comparison of a standard averaged ERP (left) and a reconstructed ERP (right)

Figure was adapted from Ouyang et al. (2015). Permission has been obtained

Recovering the internal dynamics of neural cognitive sub-processes

Although ERP reconstruction can, in principle, provide a more accurate representation of the dynamical response, it still possesses an inherent limitation that the reconstructed ERP still represents a mixture of different neural cognitive sub-processes, each one of which may have unique dynamical features and functional signatures. An illustration of this issue has been shown in several previous studies (Ouyang et al. 2013; Sturmer et al. 2013; Verleger et al. 2014). The brain response to an external input is functionally modular—it can be divided into different stages such as sensation, central evaluation and response action. Different stages possess a uniquely rich structure of dynamical activation pattern of their own (Ouyang et al. 2015). Moreover, different stages have different degree of variability, which accumulatively contribute to behavioral variability (Ouyang et al. 2015). Both standard ERP and reconstructed ERP are a summed representation of different subprocesses. Beside the fact that they cannot provide detailed dynamics pattern of sub-processes, the summed ERP may also provide ambiguous neural effects (Ouyang et al. 2013). How these different sub-processes are mapped to the ERP sub-components, how their dynamics and variability are reflected and to what extent they can be extracted from single trial data are important questions in cognitive neurodynamics research. This query requires further decomposition of ERP sub-components at a finer-grained level of cognitive sub-processes that generate neural activations overlapping with each other, which further requires tackling of several theoretical and methodological complexity in investigating separate components, such as (1) What should be the number of sub-components supported by a sound theoretical basis? (2) How to validate the decomposition? (3) What could be the additional complexity issue brought by the decomposition methods?

Oriented to addressing these frontier issues, the recent development of trial-to-trial variability-based ERP analysis methodology has demonstrated that the brain dynamical response pattern and its variability at the level of differential sub-processes can be reliably accessed with a sophisticated signal processing method. Figure 4 presents the results derived by a recent methodological framework Residue Iteration Decomposition (RIDE, (Ouyang et al. 2015)). RIDE assumes ERPs to be composed of different internal sub-components with differential latency variability. A unique aspect of this framework is that the decomposition and extraction of the sub-components are based on the information of single trial latency variability (Ouyang et al. 2015). Figure 4 shows that the RIDE framework decomposes the ERP into stimulus-locked component cluster S, central component cluster C and RT-locked component cluster R. The novel scenario provided by this framework is that each component cluster displays a distinct dynamical activation pattern and clearly differential single trial variability that are otherwise hidden and mixed in the conventional average ERP and reconstructed ERP. In order to deal with theoretical issues such as noise amplification, the RIDE methodology incorporated sophisticated signal processing procedures including L1-norm minimization-based iteration and strict time-window specifications, which bears a high degree of complexity in this framework (Ouyang et al. 2015). Nevertheless, the analysis scenario of dissociating overlapping ERP sub-components points to an appealing direction of characterizing brain dynamical responses at a finer-grained level of cognitive sub-processes. Since this decomposition framework is relatively new, much more systematic evaluation and validations need to be conducted to further establish its utility in neural cognitive research. One major aspect of evaluation is its sensitivity to parameters in EEG pre-processing and in the method itself. For example, choice of referencing method (Dong et al. 2017; Yao 2001), filtering method, and artifact rejection procedures can substantially affect the obtained ERP, which will in turn affect the decomposed or reconstructed ERP. Optimizing the parameters for the new methodologies to best investigate the internal dynamics and variability of neural sub-processes would be an important direction to develop.

Fig. 4.

Dynamical pattern and trial-to-trial variability of overlapping ERP sub-components. The RIDE method decomposes ERP data into different temporally overlapping sub-component clusters with different features of single trial variability. The framework reveals that some sub-components are stimulus-locked, some are RT-locked and some are in-between, and they can be separated. The separation reveals richer neural dynamical activity pattern that are otherwise mixed and hidden in the stimulus-locked average ERP. The consistency of the separation scenario has been demonstrated elsewhere (Ouyang et al. 2015). Data are from a single subject performing a face recognition task. The time zero indicates the presentation time of the facial stimulus. Vertical black line: stimulus onset. Black curve: RT. The single trials data were normalized in amplitude and filtered under 40 Hz

Contribution of latency variability-based ERP decomposition and reconstruction to neural cognitive research

Studying the ERP sub-components provides access to the structures of neural cognitive dynamics associated with different sensory and cognitive stages that are otherwise inter-mixed, distorted, or hidden in the average waveform. Therefore, tackling the sub-components in the average ERP waveform is crucial for the future development in neural cognitive researches, especially regarding the neural dynamical system. Recent applications of ERP decomposition and reconstruction methods have demonstrated the benefits in this regard. A brief summary is provided below.

The benefits of applying the above-mentioned methods for improving the brain dynamical response pattern and for extracting the internal dynamics and trial-to-trial variabilities of sub-processes can be categorized into the following three areas: (1) Restoring the true neural effects associated with external factors that are otherwise attenuated and covered in the standard ERP due to the latency variability-induced distortion; (2) Pin-pointing the neural effects in a specific sub-process from the entire cognitive process of perceiving, evaluating, and responding to a stimulus; (3) examining the functional signature of trial-to-trial variability of neural cognitive sub-processes.

A detailed demonstration of the application in the first area has been provided in Ouyang et al. 2016. Applications in this area, specifically, restoring true neural effects blurred by latency jitter, have covered many different cognitive constructs and topics including episodic memory effect (Murray et al. 2019), novelty processing effect (Warren et al. 2020), cognitive flexibility (Kopp et al. 2020), language processing (Fjaellingsdal et al. 2020), and reliability of ERP components (Martin-Loeches et al. 2017). These applications indirectly reflect the ubiquity of the latency variability across a broad range of cognitive paradigms. For the second application area, ample applications of RIDE decomposition algorithms in recent years have shown that the decomposition of ERP into different sub-components revealed richer internal structures and mechanisms regarding the modular cognitive stages. Selected examples include differentiating neural activities associated with early, direct perceptual response and with late, indirect, top-down controlled response in executive function tasks (Sturmer et al. 2013), investigating the cognitive transitioning and binding processes (Opitz et al. 2020; Takacs et al. 2020), pinpointing the specific neural cognitive processes that are affected by external manipulations (Peng et al. 2020; Steinemann et al. 2018), or by various brain disorders, aging, or different drugs (Bluschke et al. 2020; Giller and Beste 2019; Kleimaker et al. 2020; Muckschel et al. 2020; Wolff et al. 2019), differentiating neural activates of different cognitive stages in various other tasks (Valt et al. 2020). In addition to the research area, the benefits of improved brain response characterization by overcoming the latency variability issue are, in principle, applicable in clinical area, which has been explored as well (De Venuto et al. 2018). As for the third application area, the functional signature of trial-to-trial variability, one study has shown that the cross-trial variability of brain response in individuals estimated by RIDE was modulated by COMT genotype (Rostami et al. 2017). Furthermore, individual difference regarding the correlation between neural variables of and cognitive abilities has also been shown to be better revealed in specific sub-components of brain response extracted by RIDE (Meyer et al. 2019), which showed that the extracted neural characteristics are further associated with inter-subject variability. While a considerable number of applications in recent years have demonstrated the contribution of the new methodologies in neural cognitive research, many questions still remain open. In Fig. 5 we summarized the contributions as elaborated above and some outstanding remaining questions in this field.

Fig. 5.

Contribution of latency variability-based ERP decomposition and reconstruction to neural cognitive research and remaining questions to be addressed

Concluding remarks

In this review article, we have provided an overview of the long-standing latency asynchrony issue in brain research that has been based on trial-averaged ERPs as a tool for depicting the brain’s dynamical responses, and the latest developments in methodology in addressing the limitations of trial-averaging approach. It is worth to note that the latency asynchrony issue is by no means a negligible technical limitation compromising data fidelity. Instead, it distorts neural representations in terms of (but not confined to) timing (Miller et al. 2009), behavioral effect (Zhang et al. 2015), functional role (Bodmer et al. 2018), and anatomical feature (Yang et al. 2017). With the advancement in signal processing techniques and theoretical modelling, the limitations that latency asynchrony imposes on brain response characterization are being progressively addressed. A more detailed characterization of dynamical response concerning single trial variability and the dynamics of sub-components that are mixed in the compound of average ERPs has started to show advantages in neural cognitive research. We have presented the latest methodological development that can be used for either remedying the standard ERP with comparable simplicity or accessing the richer structure of ERP sub-components and single trial variabilities. These latest developments point to a future trend of exploring the rich patterns of complex neural dynamics associated with cognition.