Amplitude modulation of steady-state visual evoked potentials by event-related potentials in a working memory task

Abstract

Previous studies have shown that the amplitude and phase of the steady-state visual-evoked potential (SSVEP) can be influenced by a cognitive task, yet the mechanism of this influence has not been understood. As the event-related potential (ERP) is the direct neural electric response to a cognitive task, studying the relationship between the SSVEP and ERP would be meaningful in understanding this underlying mechanism. In this work, the traditional average method was applied to extract the ERP directly, following the stimulus of a working memory task, while a technique named steady-state probe topography was utilized to estimate the SSVEP under the simultaneous stimulus of an 8.3-Hz flicker and a working memory task; a comparison between the ERP and SSVEP was completed. The results show that the ERP can modulate the SSVEP amplitude, and for regions where both SSVEP and ERP are strong, the modulation depth is large.

Introduction

Stimulated by a flicker with a constant frequency, a continuous brain response can be observed from the scalp and is called the steady-state visual-evoked potential (SSVEP) [1]. SSVEP has been used more and more in the study of cognitive tasks including memory [2–6], decision selecting [7], emotional processing [8], and of various clinical disease [9–13]. In these studies, the amplitude and phase changes were thought of as indicators of the modulation of SSVEP by a cognitive task, but none had ever studied the mechanism of this modulation.

As we know, in the study of various cognitive problems, the event-related potential (ERP), obtained by a repetitive stimulation and then a stimulus time-locked average, is the most effectively and widely-utilized technique in current practice. It has been shown that there exist episodic spatial patterns of amplitude modulation (AM) on spontaneous electroencephalogram (EEG) in the gamma range by ERP [14–16].

Although the neural mechanisms of SSVEP and spontaneous EEG are not the same, it has been proved that both can be modulated by a cognitive task. Furthermore, it is probable that the neural network for generating SSVEP can provide an appreciable EEG component when not activated by the repetitive stimulus. So, referenced to the AM of the spontaneous EEG by ERP, we hypothesize that there are the similar AM of SSVEP by ERP in the situation of the simultaneous stimulus of a flicker and a cognitive task, and the modulation depth is highly relative to the activity strength of the task.

In this work, an 8.3-Hz flicker is used to evoke SSVEP and a working memory task to evoke ERP. The ERP is extracted by the traditional average method directly, while the SSVEP is estimated by the technique named steady-state probe topography (SSPT) [7]. By computing the correlation coefficient (CC) between the ERP and SSVEP, it is found that amplitude modulation of SSVEP by ERP does exist and is notable in the frontal and occipital areas, where the neural activities of SSVEP and ERP are both strong. This result suggests that modulation of SSVEP by a cognitive task can be accomplished by the ERP approach.

Materials and methods

Subjects

Ten healthy right-handed adults (males) with normal or corrected to normal vision served as paid volunteer subjects after giving informed consent. The mean age is 25 (range 25 ± 3) years old. All subjects are graduate students. The study is approved by the Human Research and Ethics Committee, University of Electronic Science and Technology of China.

Working memory task

The working memory task, similar to that of Silberstein [3], is divided into five stages. In the first stage, a cross surrounded by a circle, designed to attract the attention/fixation of the subject, is displayed at the center of the screen for 2 s; in the second stage, called the “intake” stage, one irregular polygon and three filled circles are displayed in the four quadrants of a cross for 2 s, and the subject is required to remember the shape and position of the irregular polygon; in the third stage, called the “hold” stage, only a cross is displayed for 4 s, and the subject is required to remember the information received in the second stage; in the fourth stage, an irregular polygon in one quadrant of a cross is displayed and the subject is required to judge whether the shape and position of the displayed object are the same as that presented in the second stage. If they are, the subject pushes the button “1” by using the right hand; if not, the button “3” is pushed using the same hand; in the fifth stage, just a black screen is displayed for 2 s and the subject does not do anything but relaxes in this phase.

Stimuli

One white high-luminance light-emitting diode (LED) flicker is stuck on the center of computer screen with a horizontal-right angle of 1.3° and a vertical-up angle of 3.5°. It is driven by an 8.3-Hz pulse in order to evoke the steady-state visual potential. The background luminance is 0.25 cd/m², and the maximum luminance of LED is 2.5 cd/m², so the modulation depth is 82%.

Subjects look at the screen while sitting in a comfortable chair in a shielded recording chamber. The screen is 60 cm far away and in front of the subject. In the experiment, subjects are required to avoid too much eye blinking and big movement. First, the steady-state visual-evoked potentials by the 8.3-Hz flicker are collected for a period of 2 min and used to confirm the distribution of SSVEP under the no task condition. Second, the working memory task is executed without the flicker, and the EEG collected in this period is used to extract ERP directly. Finally, after a 5-min rest, the same process is repeated with the 8.3-Hz flicker stimulus, and the SSVEP modulated by the task in this period is estimated by SSPT method. In this study, 160 trials are done continuously under each situation, and 12 s is the time cost spent for each trial.

EEG recordings

EEG is recorded from the 129 scalp electrodes mounted in an elastic cap (Geodesic Sensor Net: 128 measuring electrodes and one reference electrode Cz), and these electrodes include the international 10–20 recording sites [17]. The electrode impedances are kept below 10 kΩ, and salt water is dropped into the electrodes when necessary to keep the low impedance. EEG signals are digitized at 250 Hz, filtered by a band pass of 0.3–45 Hz, and stored on a disk for off-line analysis. Figure 1 shows the positions of recording electrodes, in which the “COM” is the ground electrode, and the “REF” is the reference electrode (Cz).

Fig. 1.

Illustration of the positions of the recording electrodes

Data analysis

Before analysis, the EEG data are preprocessed as below: if the original EEG amplitude in a trial exceeded 100 μV, this trial is thought invalid (artifact) and is replaced by the mean value of its three nearest neighboring recording sites. Upon completion of above preprocessing, all the 129 channel data are re-referenced to an average reference.

After the preprocessing, for the cases of ERP without flicker, the data in all trials at each electrode are averaged for each subject. The superposition cycle is the period of the memory task, i.e., 12 s, and the superposition times are divided into 160 epochs, which can produce 129 data segments on a 12-s time span for each subject. In our experiment, ten subjects are tested. The obtained data are averaged across all ten subjects to obtain the final average ERP. In order to check the validity of the average method in the long-time cognitive task, the data averaged by more than 120 times are compared with the final average ERP, and the CC between them is computed.

Before processing the data under the flicker stimulus, it is important to confirm whether the SSVEP has been evoked successfully. Each 12-s segment is processed by the fast Fourier transform (FFT), and this can produce a frequency accuracy of 0.08 Hz (1/12). The amplitude of 8.3 Hz and the average of the band (8.3–0.5 Hz to 8.3 + 0.5 Hz) are computed separately. If the former was more than twice of the latter, the segment is thought valid; otherwise, the segment is invalid and is not processed by the SSPT method.

The SSPT method is adopted for estimating SSVEP under the cognitive task. A sliding window with a 1.2-s span is first adopted on the SSVEP data, and the amplitude of 8.3 Hz in this window is obtained by FFT. Then, the window is shifted one stimulus cycle (1/8.3 s), and the same procedure as above is repeated. This process is continued until all the trials are analyzed. Finally, the amplitude of 8.3 Hz in all 12-s epochs is averaged to obtain an average signal at each electrode. For each subject, the average 8.3-Hz SSVEP amplitudes on all electrodes are further averaged, and the obtained value is taken as the normalization factor (NF). The average SSVEP amplitude on each electrode is divided by the NF value to obtain the normalized SSVEP amplitude, and then these normalized SSVEP amplitudes are averaged across all subjects. This averaged normalized SSVEP amplitude can be used to compare with the averaged ERP.

For the 2 min SSVEP without cognitive task, the same SSPT method is utilized to extract the 8.3-Hz signal to validate the distribution of SSVEP in this situation. The distribution of SSVEP under no task and the distribution of ERP are displayed topographically by using EEGLAB software separately.

In order to understand the AM of SSVEP by ERP under a cognitive task, the CC between the ERP and SSVEP is computed and shown topographically. Furthermore, N-way analysis of variance is performed by MATLAB software to verify the variability of the amplitude of different electrodes.

Results

Validity of the average method in study of a long-time cognitive task

Figure 2 shows the ERPs averaged from 120 epochs and 160 epochs at electrode P7 (Ch58 of the Geodesic Sensor Net), respectively, with a CC between them of 0.98. In fact, when the number of the superposition epochs is larger than 120, there is almost no change for the superposition results at all electrodes. This consistency indicates that the canonical average method usually used in a short-time cognitive task can be used also in studying the processing of a relatively long-time cognitive task.

Fig. 2.

The average ERPs superposed by 120 epochs (thin) and by 160 epochs (dotted) at electrode P7 (Ch58 of Geodesic Sensor Net). The CC between them is 0.98

Topographies of ERP and SSVEP

Figure 3 shows the topographies of the average SSVEP amplitude under no task and the absolute amplitude of ERP in different phases of the task. Under the no-task situation, the SSVEP activities concentrate at the frontal and occipital lobes. The SSVEP in the frontal lobe is similar to that in the occipital lobe (F(1, 18) = 1.1, p = 0.3079), and the SSVEP in the parietal area is similar to that in the temporal lobe (F(1, 18) = 0.47, p = 0.5038), too. However, the SSVEP in the occipital lobe is significantly greater than that in the temporal lobe (F(1, 18) = 10.02, p < 0.01).

Fig. 3.

Distribution of SSVEP and ERP. a The distribution of average SSVEP under no cognitive task. b The distribution of average ERP in “intake” phase of memory task. c The distribution of average ERP in “hold” phase of memory task. The SSVEP is normalized, and the ERP is shown in absolute amplitude (μV)

The main neural electric activities of the ERP are at the frontal and the occipital lobes. The absolute value of ERP in the frontal lobe is significantly greater than in the occipital lobe in the “intake” (F(1, 18) = 7.83, p < 0.01) and “hold” (F(1, 18) = 16.7, p < 0.01) phases. In contrast the absolute value of the ERP in the occipital lobe is significantly greater than in the temporal lobe in the “intake” (F(1, 18) = 3.54, p < 0.01) and “hold” (F(1, 18) = 4.6, p < 0.01) phases.

Comparison between ERP and SSVEP

It is demonstrated that the SSVEP amplitude varies with the ERP amplitude in an obvious relationship, which suggests that the amplitude modulation on SSVEP by ERP does exist and varies across regions. At some electrodes, the ERP wave is very similar to the SSVEP, but at some others, they can be simply the inverse of each other. Quantitatively, the absolute CC between these two signals could be larger than 0.7 for some brain areas, while it could be near to zero for other areas. On the other hand, it could be positive at some electrodes, yet negative for others. Figures 4 and 5 show ERP and SSVEP at two electrodes, Fz and Pz (Ch11 and Ch54 of Geodesic Sensor Net).

Fig. 4.

The ERP and SSVEP obtained at electrode Fz (Ch11 of Geodesic Sensor Net). a ERP averaged over 160 epochs. b The normalized SSVEP. The CC between them is −0.52

Fig. 5.

The ERP and SSVEP obtained at electrode Pz (Ch54 of Geodesic Sensor Net). a ERP averaged over 160 epochs. b The normalized SSVEP. The CC between them is 0.72

The distribution of the CC between the ERP and SSVEP

Figure 6 shows the topography of the average CC between the ERP and SSVEP. High positive or negative correlations are found at most parts of the frontal and occipital lobes, where the neural activity of SSVEP and ERP are both strong (see Fig. 3), and there exist certain large negative correlations at some parts of the left and right temporal lobe as well.

Fig. 6.

The topography of average correlation coefficient (CC) between ERP and SSVEP

Discussion

Frequently, a 13-Hz stimulus was adopted in previous studies [3]. In order to verify whether there is any significant difference between different frequency stimuli, we used 8.3 Hz as the stimulus and asked the tested subject to do the same memory work as in Silberstein’s work. The experimental results showed no significant difference [6]. This suggests that SSVEP in a wide band can be used to study the process of a cognitive task. So, in this study, we still select 8.3 Hz to evoke SSVEP and do not consider the frequency impact.

In this work, we are concerned in the relationship between the ERP and SSVEP shown in a working memory task. Since Silberstein and coworkers had already studied the SSVEP amplitude’s change when the subject performed a similar working memory task by using a 13-Hz flicker as stimulus [3], we, therefore, do not repeat the discussion on this process.

ERP for a long-time task

In this work, the superposition cycle is 12 s, which is much longer than that in a normal ERP experiment [18, 19]. In order to check whether the canonical average method is still valid for a long-time task, we compare the results averaged from 120 epochs or more and find that the difference between them is very small. This suggests that the traditional average method can be adopted for such a long-time task.

In general, although the weak ERP can be strengthened by using the average method in a long-time cognitive task experiment, other types of noise, such as a weak signal with a cycle period equal to an integer multiple of the superposition cycle, can be amplified too. Thus, the obtained average ERP may be a mixed one with the component induced by the cognitive task as the main part. This may explain why scientists do not use the ERP to study the process of a relatively long-time cognitive task directly [2, 5]. During a short-time cognitive task, other neural activities might be asynchronous and smaller compared with the ERP, yet the average method does not necessarily improve the amplitudes of these signals. The traditional average method is still a very efficient method in these studies.

SSVEP for cognitive task study

When using a flicker to evoke SSVEP when there is no cognitive task and using SSPT to analyze the data, various SSVEP amplitudes observed though the flicker are found to be stable. This fact suggests that, except for some determined events such as a cognitive event, there is something else contributing to the SSVEP based on the experimental results, which could include a certain type of potential mental activities or physiological noise. In other words, the SSVEP amplitude includes two parts: one induced by the background noise and the other induced by the repetitive stimulus, with the latter bigger than the former [20]. So, even if the SSVEP has been used to study the long-time cognitive task process in many studies [2, 4, 5], we still strongly suggest that the explanation about the variance of SSVEP amplitude and phase should be made with more care .

Combined with Section 4.1, either the average ERP or SSVEP cannot estimate the pure cognitive contribution, and this makes it difficult to appraise which method is better or which result is closer to reality. In a concrete task, it may be better if these two methods are combined.

The modulation on SSVEP by ERP in a cognitive task

Some studies have suggested that the SSVEPs in the alpha band are generated by the same network [1, 21, 22] and concentrate at the occipital and frontal lobes [22]. In fact, the 8.3-Hz SSVEP activities identified in this study clearly indicate that they concentrate at the occipital and frontal lobes as well. On the other hand, the neural activities revealed by the absolute amplitude of ERP in this work concentrate mainly at the frontal lobe and partially at the occipital lobe, and this is consistent with the conclusion that the activity in the frontal lobe was found to be distinctly stronger than that in other regions in a similar memory task [3, 5].

In Fig. 6, the absolute values of the CCs between the ERP and SSVEP are large in the frontal and occipital lobe, which suggests that, in these regions, the contributions by the working memory task to the SSVEP amplitude change and the ERP are similar. This phenomenon can be explained by amplitude modulation theory. The SSVEP with a sinusoidal structure can be seen as a carrier wave, while the ERP and part of the background activity can be seen as a modulation wave. When the SSVEP amplitude varies with the ERP accordingly, a positive CC can be obtained; when the SSVEP amplitude varies with the ERP inversely, a negative CC can be observed. In a region where the ERP is strong enough, background activity such as the alpha band may decrease, which is called the event-related desynchronization [23, 24]. Therefore, the contribution to the SSVEP amplitude change from the ERP is bigger than that from other background activities in this region. In a memory task, the activity concentrates at the frontal lobe and occipital lobe; the background activity in these areas is relatively small, so the AM in these areas is notable.

It should be mentioned that different cognitive tasks can activate different regions in the brain. So, in some other tasks, the modulation on SSVEP by ERP can be observed in distinct areas, and this is an interesting topic to explore.

It is very surprising that there exists positive or negative correlation between near electrodes in the frontal and occipital areas. This may be caused by the intrinsic properties of SSVEP. Some researchers have demonstrated that SSVEP can possess the property of a traveling wave [25], which suggests that the phase of SSVEP between various electrodes may be different. Sometimes, they may be inverse to each other, which results in a strongly negative correlation between SSVEP and ERP.

In certain areas in the temporal lobe, the CCs are large, as well. This can be explained as, due to the very small background activities in these regions when executing a memory task; the ERP and SSVEP in these regions are both weak, but they are still locally and relatively larger than the background activities. Thus, the interaction between ERP and SSVEP is relatively strong, which leads to a large CC.

Comparison between the AM of SSVEP and spontaneous EEG by the ERP

In an SSVEP experiment, the background activities always exist. However, compared with the component evoked by a repetitive stimulus, the contribution of the background activities to the final SSVEP is small. Therefore, a continuous AM on SSVEP by ERP is observed at the areas where both the SVVEP and ERP are strong. However, in other studies, only with the stimulus of a cognitive task [14–16], though the ERP in some areas is strong and stable, the carrier frequency in EEG is relatively weaker and more variable compared to the SSVEP, so only an episodic spatial pattern of AM by ERP is observed.

It is meaningful to note that there is some important difference between the AM in biological field and the AM in physics. In an AM system in physics, normally, the frequency of carrier is distinctly higher than that of the modulation signal. In the biological field, however, the difference between the two frequencies is not so distinct. For example, in this work, the frequency of carrier is 8.3 Hz, and in other works, it is 13 Hz [12, 13] or in the gamma band [15, 16], while the modulation frequency, i.e., the main component of ERP, is also within this range. From this point, the mechanism of interaction between the modulation signal and the carrier signal in biology may be different from that in a physical AM system. On the other hand, the CCs between the ERP and SSVEP are positive in some areas and negative in other areas. Although this can be explained by the traveling wave property of SSVEP to some extent, it may suggest that the AM mechanism in biology field is different from that in physics field. Though it is very difficult to clarify the modulation mechanisms in a biological system such as the brain, it is still a problem worth researching.

Conclusion

In this work, through a common working memory task, the SSVEP amplitude change and the ERP are collected, and the results show that the AM of SSVEP by ERP is clearly observed, and the AM depth is different in different brain areas, especially for the frontal and occipital areas where both SSVEP and ERP are strong. The correlation coefficient between them is strong too, which indicates a large modulation depth. In other types of cognitive tasks, the AM of SSVEP by ERP can be observed in other brain regions. The method introduced in this paper can be used to study them in such situations.