Skip to main content
NCBI home page
UKPMC Funders Author Manuscripts logoLink to UKPMC Funders Author Manuscripts
. Author manuscript; available in PMC: 2023 Sep 8.
Published in final edited form as: J Neurosci Methods. 2022 Oct 10;382:109726. doi: 10.1016/j.jneumeth.2022.109726

Application of rapid invisible frequency tagging for brain computer interfaces

Marion Brickwedde a,c,*, Yulia Bezsudnova a, Anna Kowalczyk a, Ole Jensen a, Alexander Zhigalov a,b
PMCID: PMC7615063  EMSID: EMS185163  PMID: 36228894

Abstract

Background

Brain-computer interfaces (BCI) based on steady-state visual evoked potentials (SSVEPs/SSVEFs) are among the most commonly used BCI systems. They require participants to covertly attend to visual objects flickering at specified frequencies. The attended location is decoded online by analysing the power of neuronal responses at the flicker frequency.

New method

We implemented a novel rapid invisible frequency-tagging technique, utilizing a state-of-the-art projector with refresh rates of up to 1440 Hz. We flickered the luminance of visual objects at 56 and 60 Hz, which was invisible to participants but produced strong neuronal responses measurable with magnetoenceph-alography (MEG). The direction of covert attention, decoded from frequency-tagging responses, was used to control an online BCI PONG game.

Results

Our results show that seven out of eight participants were able to play the pong game controlled by the frequency-tagging signal, with average accuracies exceeding 60 %. Importantly, participants were able to modulate the power of the frequency-tagging response within a 1-second interval, while only seven occipital sensors were required to reliably decode the neuronal response.

Comparison with existing methods

In contrast to existing SSVEP-based BCI systems, rapid frequency-tagging does not produce a visible flicker. This extends the time-period participants can use it without fatigue, by avoiding distracting visual input. Furthermore, higher frequencies increase the temporal resolution of decoding, resulting in higher communication rates.

Conclusion

Using rapid invisible frequency-tagging opens new avenues for fundamental research and practical applications. In combination with novel optically pumped magnetometers (OPMs), it could facilitate the development of high-speed and mobile next-generation BCI systems.

Keywords: Brain-computer interface, Rapid-invisible frequency tagging, Covert attention, Magnetoencephalography, Eye-tracking

1. Introduction

In recent years, brain-computer interfaces (BCIs) have developed from niche applications to wide implementations in clinical and rehabilitation settings as well as fundamental research paradigms (Bagherzadeh et al., 2020; Bergmann et al., 2019; Biasiucci et al., 2018; Brickwedde et al., 2019; Peles et al., 2020; Zrenner et al., 2018). One of the most common types of BCIs is based on steady-state visual evoked potentials or fields (SSVEPs/SSVEFs, e.g., Cheng et al., 2002; Kelly et al., 2005; Ma et al., 2022). These are characterised by rhythmic neuronal activity at the frequency of sensory stimulation. Specifically, flicker or frequency-tagging at a specific frequency will evoke neuronal responses echoing the pattern of stimulation (Brickwedde et al., 2020; Colon et al., 2012; Müller and Hillyard, 2000; Regan, 1982; Snyder, 1992; Stapells et al., 1984; Zhigalov et al., 2019). The magnitude of SSVEPs/SSVEFs is known to be attention-dependent (Müller et al., 1998; Müller and Hillyard, 2000; Toffanin et al., 2009; Zhigalov et al., 2019), which can be utilized to provide human participants with control over BCI systems.

However, most of the SSVEP/SSVEF BCI systems utilise stimulation at relatively low frequencies, usually within a range of 5–40 Hz, making the flickering clearly visible to participants (Landis, 1954; Seitz et al., 2005). The resulting visual flicker can be very distracting, straining to the eye and cause fatigue (Lin et al., 2012). Another important consideration pertains to epileptic seizures, which are most strongly provoked by lower frequency flicker in the range of 15 – 25 Hz (Fisher et al., 2005).

While the magnitude of SSVEPs/SSVEFs responses decreases with an increase in visual stimulation frequency (Regan, 1977), the temporal resolution increases greatly by increasing the number of queries and therefore neuronal responses in a given temporal interval. Thus, the use of higher frequencies for visual stimulation could greatly enhance the communication rate compared to existing BCI systems.

Intrinsic oscillatory activity in lower frequency ranges is dominant across all sensory cortices and it is believed to fulfil crucial functional roles for sensory processing and memory (Haegens and Zion Golumbic, 2018; Jensen and Mazaheri, 2010). As such, scalp-level measurements during low-frequency sensory stimulation can produce a signal mix of cortical oscillatory activity and SSVEPs/SSVEFs, which can become difficult to disentangle. At the same time, low-frequency SSVEP/SSVEF activity can affect and entrain intrinsic oscillatory activity (Keitel et al., 2014; Spaak et al., 2014), which can influence task performance and question the interpretability of fundamental research analyses concerning oscillatory activity.

State-of-the-art projectors are now able to produce refresh rates of up to 1440 Hz. This technological advancement has introduced the possibility to implement rapid invisible frequency tagging (RIFT), which is attention-dependent comparable to its low-frequency counterparts, but invisible to participants (Drijvers et al., 2021; Seijdel et al., 2022; Zhigalov et al., 2019; Zhigalov and Jensen, 2020). Typically, RIFT is applied at frequencies above 60 Hz. It does not entrain intrinsic oscillatory activity (Duecker et al., 2021) and elicits very localized responses (Zhigalov et al., 2019; Zhigalov and Jensen, 2020), diminishing potential confounds of task performance and interpretability of low-frequency activity, as well as introducing the possibility to work with a limited number of sensors.

Therefore, we implemented rapid-invisible frequency tagging into an online BCI system to provide proof-of-principle for the validity of this technique. Participants used covert attention while fixating a cross at the centre of the screen, to control a quickly paced pong game. We were interested in the overall performance of participants and the efficacy of our system to track visuo-spatial attention.

2. Materials and methods

2.1. Participants

Eight healthy participants with previous experience in MEG studies were recruited at the University of Birmingham (5 women, mean age: 32 ± 3). Participants had normal or corrected-to-normal vision and no history of neurological disorders. The study protocol was approved by the Ethics Committee of the School of Psychology at Birmingham University and in accordance with the Declaration of Helsinki. All participants provided written informed consent.

2.2. Setup

Participants were seated comfortably in the gantry of a 306-sensor TRIUX Elekta system with 204 orthogonal planar gradiometers and 102 magnetometers (Elekta, Finland), in a dimly lit magnetically shielded room. We used a PROPixx DLP LED projector (VPixx Technologies Inc., Canada) mounted outside of the shielded room. Importantly, the projector supported a refresh rate of 1440 Hz, which allowed the application of high-frequency flicker or tagging, which was invisible to participants. Entering through an aperture in the shielded room and being routed by a reflecting mirror, the video output was projected onto a 71 * 40 cm screen placed 1.5 m away from the participant. During the experiment, horizontal and vertical eye movements were tracked with EyeLink 1000 Plus, SR Research Ltd, Canada. MEG signals passed through an embedded analogue filter, 0.1–330 Hz, and sampled at 1000 Hz. Different types of data, including MEG, eye-tracking, and triggers, were combined into a single data-stream in the acquisition computer. The combined data were then split into 100 ms blocks and sent to the stimulus computer using the Fieldtrip buffer (Oostenveld et al., 2011). Finally, these data were used to train a classifier to decode the direction of covert attention based on the power of the frequency-tagging response.

2.3. BCI-pong training

Before the training session, the eye-tracker was calibrated by asking participants to sequentially fixate on different spatial locations across the screen. This information was then used to automatically discard trials with eye-blinks during the frequency-tagging phase of each trial. During training, participants were asked to fixate on the cross at the centre of the screen. Each trial started with 1 s baseline period. Afterwards, a grey ball was displayed for 2 s at a random location on either the right or the left side of the screen. At the same time, the left and the right sides of the screen (see, Fig. 1, dashed line) were frequency-tagged at 56 and 60 Hz, respectively. These frequencies were chosen based the trade-off between strength of SSVEP/SSVEF response and visibility of the stimulus, as higher frequencies are invisible to participants but also elicit smaller SSVEP/SSVEF responses (Herrmann, 2001; Herbst et al., 2013). In the experiment, we presented the tagging patches on a grainy texture which allowed us to make the visual stimulation at the lower range of the spectrum (i.e. 56 and 60 Hz) practically invisible. The edges of the tagged patches were 100 % transparent, slowly fading in to 100 % visibility over 10 % of the length/height of the patch. A static vertical bar of ~2 degrees of visual angle was displayed in the lower part of the screen (Fig. 1), to make the training session as close to the subsequent gaming session as possible.

Fig. 1. Experimental setup of the BCI-pong training and game.

Fig. 1

Open in a new tab

(A) In the training session, participants were asked to covertly attend either left or right side of the screen for 2 s. At the same time, invisible flickering patches were presented at the right and left bottom of the screen. Dashed areas indicate the location of flickering patches and corresponding tagging frequencies. (B) During the game, participants covertly attended the right or left side of the screen, to move a bar from one side of the screen to another. The goal was to always position the bar at the side of the screen the pong ball was falling towards. Inside of the bar, a dark line displayed the continuous classifier output as a quantitative measure of covert attention. Additionally, a small black bar in the middle was always present.

Participants were asked to keep fixating on the cross, while covertly attending to the side of the screen indicated by the location of the pong ball (see Fig. 1). Additionally, participants were asked to reduce blinking as much as possible and to limit blinking to the end of the trial. All participants performed 80 valid trials (40 to the left and right, randomly sequenced). The training was performed once at the beginning of the experiment.

MEG data acquired in the training session were used to train a classifier (see, Classification algorithm section).

2.4. BCI-pong game

In the game, participants controlled the position of the horizontal bar by changing the direction of their covert attention (Fig. 1). The goal of the game was to keep the pong ball afloat, by moving the bar to the right or the left side of the screen. During the game, online MEG data were stored in a 1-s buffer, which was updated every 100 ms. This way, we assessed the power spectral density of the frequency-tagging response using Fourier transform (1), as implemented in MATLAB

P(f)=|∑j=1nX(j)Wn(j−1)(f−1)|2 (1)

where

Wn=e(−2πi)/n

P is the power spectral density, X denotes single-sensor MEG signal, Wn is one of n roots of unity, n is the signal length.

The power of the tagged response at 56 and 60 Hz was used in the classifier, to infer the direction of covert attention. The processing time was negligibly small (less than 10 ms on average) compared to the data acquisition delay (100 ms) which ensured uninterrupted feedback for the user.

The game started with the pong ball appearing at a random position in the top quarter of the screen, moving towards the bottom. If the bar and the ball were at the same side of the screen and colliding, the ball was reflected back in a 90 degree angle and moved towards the wall. There, it bounced off until it was moving towards the bottom again. If the bar was on the opposite side when the ball reached collision height, the ball fell through the bottom and was reset to a random starting position at the top (see Fig. 1). The first second after a reset, the ball always moved with reduced speed to accommodate for an orientation period necessary to predict where the ball was moving towards. To make sure that blinks did not introduce systematic bias between conditions, we automatically discarded trials in which participants blinked in the last second before the collision in the game. Each participant played the 5-minute BCI-pong game four times. At the end of each game, a score indicating the number of hits and misses was presented on the screen.

2.5. Classification algorithm

The data acquired through the training session were used to train a support vector machine (SVM) classifier with a 4-fold cross-validation (Cawley and Talbot, 2002; Hastie et al., 2008) to decode the direction of covert attention. Considering that the frequency-tagging response is well localised within the primary visual cortex (Zhigalov and Jensen, 2020), we used only seven occipital gradiometers (Fig. 3A) to quantify the modulation of power by covert attention. The power of the frequency-tagging response was computed at 56 and 60 Hz for each gradiometer, resulting in fourteen features. These features along with labels indicating the direction of attention were used to train a linear SVM classifier, as implemented in MATLAB.

Fig. 3. Selected gradiometers and classifier weights at the group level.

Fig. 3

Open in a new tab

(A) Seven occipital sensors were used to train the classifier. The patches flickering at 56 and 60 Hz were presented in the left and right hemifields, respectively. (B, C) Classifier weights at the group-level. Bars indicate the standard error of the mean. The three coloured open and filled circles represent left-side and right-side sensors, respectively.

2.6. Data processing

Offline processing was performed in MATLAB R2020b using custom-made scripts as well as the Fieldtrip toolbox version 20210106 (Oostenveld et al., 2011). Recordings were kept as raw as possible to reflect the BCI purpose, which included not performing common artifact removal techniques or baseline corrections. Accordingly, the MEG data was divided into epochs and filtered between 1 and 80 Hz. Additionally, a 50 Hz notch filter was applied. Epochs constituted of a range from 1s before and after frequency-tagging for the training session and 3s pre to 1s post pong ball collision events during gaming.

Time-frequency decomposition was performed between 40 and 70 Hz using a Hanning tapered dynamic sliding window covering 20 cycles per frequency in steps of 25 ms. Relevant sensors were identified using an occipital-to-parietal region of interest and subtracting left-trial from right-trial data. For the game data, only successful trials were considered. Perceptually uniform and universally readable colormaps were applied to all visualisations (Crameri et al., 2020).

To calculate the signal-to-noise ratio, we calculated the power during the last second before collision in the game at the tagged frequencies. As such, we averaged over the seven sensors applied in the BCI using for each participant, using the following formula:

SNR=p(f1)0.5∗[p(f2)+p(f3)] (2)

Where p(f1) refers to the power at the tagged frequency and while p(f2) and p(f3) refer to p(f1 ± 2 Hz).

The information transfer rate was calculated with the following formula (Wolpaw et al., 2002):

ITR=(log2(N)+P∗log2(P)+(1−P)∗log2(1−PN))∗(ST) (3)

Where N is the number of targets, P denotes the classification accuracy, S is the number of trials and T is the time in minutes.

To exploratively estimate any trends between frequency-tagging responses and training as well as gaming accuracy, we performed correlation analyses using Spearman correlation. However, the outcome of the between-subject analysis was not assessed statistically as this would require larger sample sizes.

Lastly, power at 56 and 60 Hz as well as eye-movement data were averaged over a 1 s sliding window in steps of 1 ms to reflect the BCI setup used in the study. The data was then downsampled to 100 Hz and split into right and left trials. Again, for the game session, only successful trials were considered. For both training and game sessions, attention-left and attention-right trials were tested against each other using cluster permutation on dependent sample t-tests for each time point. Right- and left side conditions were shuffled for 1000 iterations to calculate the 5 % highest t-sums of randomly formed clusters. Only clusters whose t-sum exceeded 5 % of the previously found clusters with the highest t-sum were considered significant.

3. Results

In this study, we utilised a novel rapid, invisible frequency-tagging technique in a brain-computer interface paradigm. In a training session, participants were required to covertly attend either the right or the left side of the screen. Their SSVEF responses over seven occipital sensors were used to train a support vector machine (SVM) to distinguish between left and right trials. Afterwards, participants used convert attention to control the position of the bar in the BCI-pong game, testing the efficacy of the trained classifier.

3.1. Time-frequency analysis of the frequency-tagging response

We first performed a time-frequency analysis of power of MEG signals, investigating whether the short 1-second timeframe was sufficient to wilfully produce changes in neuronal activity.

During training, participants were instructed to focus their attention to the left or right side of the screen for 2 s while rapid-frequency tagging was applied. At the group level, the power at the gradiometer sensor with the strongest frequency-tagging response clearly increased during the tagging interval at both 56 and 60 Hz and dropped to a baseline after cessation of the tagging (see Fig. 2A). A similar increase in power was observed during the game, where continuous frequency tagging was applied (see Fig. 2B).

Fig. 2.

Fig. 2

Open in a new tab

Time-frequency representations and topography of the frequency-tagging response during the training (A, C) and game sessions (B, D). A,B, At the group-level, the gradiometers with the strongest frequency-tagging response showed a clear increase in power at 56 Hz during covert attention-left trials and at 60 Hz during covert attention-right trials. In the training session, time 0 and 2 s denote the on and offset of the frequency-tagging, respectively. In the gaming session, time 0 s denotes the moment of any hit or miss events, while frequency-tagging was uninterrupted. C,D, Topographies of 56 Hz power during attention-left trials and 60 Hz power during attention-right trials revealed spatially distinct and well-localised responses.

Signal-to-noise ratios for each participant were above one at both tagged frequencies (mean 56 Hz: 3.1; std 56 Hz 1.8; mean 60 Hz: 2,0; std 60 Hz: 0.8). This indicates sufficient signal-induced power in relation to noise.

3.2. Topography of the frequency tagging response

To assess the spatial distribution of the frequency-tagging response, we computed the spectral power at 56 Hz during covert attention-left trials and at 60 Hz during covert attention-right trials for each gradiometer. It should be noted that the spectral power was assessed during a 2s tagging interval in the training session, and during a 1s interval during the game (1s before the ball collided with the bar or not). At the group level, the response was mainly localised over occipital areas, and the topographies were strikingly similar in training and game sessions(Fig. 2). This suggests that covert attention modulates the neuronal response in the visual cortex locally.

Together, the results indicated that the covert attention robustly modulates the power of frequency-tagging responses within a 1-s interval, meeting the requirements for the BCI-pong game. 3.3. Classification and performance.

We implemented a linear SVM classifier to decode the direction of covert attention based on the spectral power of the frequency-tagging response. In line with the topography of the strongest responses (see, Fig. 2), only seven occipital gradiometers (Fig. 3A) were selected to train the classifier. The weights of the classifier for 56 and 60 Hz (Fig. 3B,C) showed a roughly inverse pattern (or inverse contribution) as expected for bilateral visual stimulation. Interestingly, the largest weights associated with two bilateral sensors (Fig. 3, blue circles) were highly consistent across participants, suggesting that only two sensors might be sufficient to provide a reasonable classification accuracy at the group level.

The cross-validation accuracy during the training session was above chance level (50 %) for seven out of eight participants, and on average it was slightly above 60 % (Fig. 4A). The spatially distinct and well-localised responses were again visible when looking at individual participants. Interestingly, the difference in spectral power at 56 Hz (as well as at 60 Hz) between attention-left and attention-right trials at the sensor with the strongest tagging response during training, did not show a clear linear relationship with classification accuracy (Fig. 4B), suggesting that the single gradiometer does not provide sufficient accuracy.

Fig. 4. BCI-pong performance.

Fig. 4

Open in a new tab

(A) Cross-validation accuracy of the classifier for individual participants and the group average. Topographies of spectral power at 56 Hz are displayed for three representative participants and for the group average. (B) Relationship between power at 56 Hz in general as well as attention left versus right power differences at 56 Hz and classifier accuracy. Each dot represents one participant. No clear connection is apparent. (C) Accuracy during the game for individual participants and the group average. Topographies of spectral power at 56 Hz are displayed for three representative participants and for the group average. (D) Performance in the game seems to be related to attention left versus right power differences at 56 Hz rather than to 56 Hz power in general during training. Each dot represents one participant.

In the online game session, 7 out of 8 participants performed above chance level (Fig. 4C). Notably, most participants showed higher accuracy during the game compared to the accuracy estimated based on the training data. As such, the average accuracy exceeded 60 %. And 4 out of 8 participants reached accuracies clearly over 70 %. Again, topographies showed the same distinct and local pattern as before for individual participants, with stronger power compared to the training session. On average, the information transfer rate of 3.9 bpm was achieved during the game, with a maximum value of 9.8 bpm.

Offline analysis of the MEG data showed that the topography of the frequency-tagging response did not change during the game, suggesting that covert attention modulated neuronal activity mainly in occipital-parietal areas. Interestingly, rather than power in general, the difference in spectral power at 56 Hz between attention-left and attention-right training trials at the sensor with strongest tagging response during training, seemed to be highly predictive for the success in the PONG game (r= 0.55, Pearson correlation; Fig. 4D). This might suggest that attentional modulation of spectral power becomes more spatially local during the game.

Overall, our results showed that most participants were able to control covert attention and hence, successfully play the BCI-pong game.

3.3. The influence of saccades on BCI performance and frequency-tagging response

To confirm that our BCI in fact relied on covert attention and did not simply reflect amplified frequency-tagging responses due to eye movements towards the flickering targets, we analysed the eye tracker data. During training, there was no apparent difference in saccadic behaviour between left and right trials (Fig. 5A). Yet the spectral power at 56 Hz was higher during attention-left compared to attention-right trials, and as expected, the power at 60 Hz was higher during attention-right trials compared to attention-left trials (see Fig. 5A). We conclude that this difference in the frequency tagging response is explained by covert attention rather than eye movements. The same was true for game sessions. Again, differences in eye movements between right and left trials were not observable (Fig. 5B). The differences in spectral power were significantly enhanced in the gaming session compared to the training session. As such, significant condition differences (p < 0.01, permutation test) between attention-left and attention-right trials were observable in both frequencies spanning from – 700 ms prior to and 100 ms after hit or miss events.

Fig. 5. Influence of saccades on BCI -performance during fixation.

Fig. 5

Open in a new tab

(A) Changes in eye position and spectral power at 56 and 60 Hz during training. No difference between attention-left and attention-right trials is apparent for eye positions, contrary to the spectral power. Red solid lines indicate attention-right trials, while black dashed lines indicate attention-left trials. Grey areas mark the time interval where frequency-tagging was applied (B) The game session shows no differences in eye positions for attention-left and attention-right trials; however, it reveals larger differences in spectral power for these trials. Significant condition effects are marked in bold. Grey areas mark the time-window used to estimate direction of covert attention.

Furthermore, the eye positions of all participants were relatively stable (Fig. 6A) during the critical period (i.e., the last second before hit or miss) for each trial of the game. The majority of horizontal eye movements remained within 1 degree of visual angle from the fixation cross (on average 97 % of eye positions over participants). Furthermore, barely any eye positions were recorded outside of 2 degrees left and right to the fixation cross (on average 1 % per participant). Within these 2 degrees, there is no clear pattern of an increase in spectral power to either direction. Aggregating eye positions into 40 bins and averaging the power of the frequency-tagging response does not show a clear pattern nor a difference between 56 Hz and 60 Hz (Fig. 6B).

Fig. 6.

Fig. 6

Open in a new tab

(A) Illustration of the eye positions of all participants across trials during the last second before hit or miss during the game. The black squares indicate 3, 2, and 1 degree of visual angle around the fixation cross, respectively. Especially for horizontal eye movements, most data points stay within 1 degree of visual angle. (B) Histograms illustrating the mean power at 56 Hz and 60 Hz covering 40 binned positions over horizontal eye movements within 2 degrees of visual angle around the fixation cross. No clear association between power of the response and eye-position is observable.

Our data shows that when eye-movement is restricted, it does not affect frequency-tagging responses.

4. Discussion

In this study, we have shown that rapid invisible frequency-tagging (RIFT) can be used to decode the direction of covert attention through power estimation of neuronal responses. Our results demonstrate that after a brief training, most participants were able to play the BCI-pong game successfully.

The novel RIFT technique provides several advantages compared to conventional SSVEP/SSVEF based BCI systems (Wolpaw et al., 2002). Here, we highlight three key advantages: invisibility of the flicker, faster detection of neuronal changes, and high signal-to-noise ratio.

Particularly, RIFT produces high-frequency visual flicker that is not perceived by the participant. Accordingly, there is less distraction and strain compared to low-frequency flicker techniques. This is important for both practical applications and fundamental research (Kaplan et al., 2013), as it allows for longer BCI-use by avoiding fatigue related to visual flicker. Additionally, RIFT evokes a robust neuronal response that can be decoded within a short period of time. In line with previous reports, the slower flicker frequency (56 Hz versus 60 Hz) elicited stronger neuronal responses (Regan et al., 1977; Herrmann, 2001). Our study shows that the high-frequency tagging response modulated by covert attention can be reliably detected within a less-than-one-second window. This property alone can significantly increase the information transfer rate (ITR) in the existing BCI systems. With an average and maximum ITR of 3.9 bpm and 9.8 bpm, respectively, our system produced higher information transfer rates compared to other state-of-the-art covert-attention BCI systems based on alpha activity and SSVEPs. For example, Horschig et al. (2015) reported an average and maximum ITR of 2 bpm and 8.9 bpm, while Kelly et al. (2005) reported a maximum ITR of 7.6 bpm. Although, in this study the analysis window was limited by one second, there is a potential to further reduce the window, while preserving the decodability of responses. Finally, high-frequency flicker evokes a neuronal response with high signal-to-noise ratio since the response is not affected by the pronounced endogenous rhythms at low frequencies. In contrast, when using low-frequency visual stimulation, there is a chance that the stimulation frequency may be in a close proximity to endogenous rhythms (Haegens and Zion Golumbic, 2018; Jensen and Mazaheri, 2010), which could lead to attenuation of and interference with the neuronal response (Keitel et al., 2014; Spaak et al., 2014).

Inter- and intraindividual variability in the ability to control BCIs are commonly observed. In fact, BCI illiteracy is often estimated to affect at around 20 % of all users (Benaroch et al., 2021; Lotte and Jeunet, 2015). As the individual response strength to frequency-tagging stimuli differs between participants, it could be assumed the explanation for interindividual differences. Interestingly, our data implies that more than the SSVEF amplitude, the average difference in SSVEF responses between right- and left covert attention trials was related to game performance. Both the neuronal responsiveness to covert attention shifts as well as the skill of the participant to allocate covert attention could be crucial factors for successful BCI control. Provided that the average difference between covert attention right- and left trials was distinct, participants were able to adapt their strategy and improve allocation of covert attention during the game, which was highly motivating. Consequently, gaming performance does not need be strongly related to training accuracy, where no feedback was provided, and outliers and bad trials could affect cross-validation.

Eye-movements during BCI control might raise concern as to whether saccades towards the frequency-tagging stimuli rather than covert attention modulated the neural responses. We showed that while attention left- and right-trials significantly differed in power, the saccadic eye movements were indistinguishable in these trails. This is important as it emphasises that covert attention could not simply be reduced to eye-movements towards the frequency tagged objects. On a smaller scale, directional biases in microsaccades are correlated with neural signatures of covert spatial attention (Lowet, 2018). However, a recent study revealed that this link was not causal in monkeys (Yu et al., 2022).

Considering a future application of RIFT BCI for highly mobile BCI applications, we specifically designed our BCI system using only a small number of MEG sensors and with minimal processing requirements. As we could show that the RIFT response is strongly localised, it can be captured by only few sensors. Although, in this study we used seven sensors to control the BCI, the group level analysis of classification weights suggested that already two bilateral sensors in the occipital region provide a reasonable classification accuracy. This shows that our BCI can be easily implemented in mobile and flexible systems for example with novel optically pumped magnetometers (OPMs), which eases application for children and clinical populations, while being less restrictive on movement compared to conventional measuring systems.

4.1. Limitations of the study

First, for this proof of principle, we recruited a limited number of participants (N = 8) and implemented short training sessions. This can be easily expanded to a larger cohort of participants enabling more rigorous statistical analysis.

Second, we used a simple decoding technique, linear SVM. More sophisticated machine learning approaches may provide a better classification accuracy and hence, improve the performance in the game. Additionally, a self-adapting classifier could be implemented online, to skip the training procedure, thereby enhancing the ease of continued application.

Third, we used a traditional MEG system to record the brain activity, which due to the rigid position of the sensors, may provide slightly inconsistent results related to the participant’s head movement during the experiment. Moreover, such MEG systems have poor mobility, and hence, the practical application may be limited. Recent development in OPM shows that magnetic fields of the brain can be recorded using highly flexible OPM sensors (Boto et al., 2018; Brooks et al., 2021), and thus, the problem of rigid sensors position, and poor mobility can easily be overcome.

Finally, our BCI was limited to two targets. Taking into account the limited range of possible frequencies between ~50 and 70 Hz that are invisible but still elicit strong SSVEP/SSVEF signals (Lin et al., 2012; Herbst et al., 2013), only very few targets could be added. This limitation can be easily overcome by application of broadband frequency-tagging, which creates varying signals between 50 and 70 Hz and can be decoded using coherence or phase-locking methods (e.g. Zhigalov et al., 2020).

4.2. Future work

The use of rapid invisible frequency-tagging has strong advantages compared to existing BCI systems. However, future work can further improve the application of this technique.

Our study shows that covert attention modulates frequency-tagging responses locally, and thus, a very limited number of sensors is necessary to detect the neuronal responses of RIFT. In this case, OPM sensors provide an excellent solution, since they do not constraint the participant movement, and importantly, such sensors can be placed flexibly and have a much lower cost compared to conventional MEG systems.

While the invisibility of the stimulation is a great advantage, it would still be interesting to systematically compare slower and faster stimulation frequencies with regards to BCI-performance and user fatigue. Additionally, it would be interesting to apply the same approach using EEG, which is more readily available than MEG, to test whether similar performances would be achieved.

5. Conclusion

This study provides novel insights into brain-computer interfaces based on covert attention. We applied a new rapid-frequency tagging technique that has two key advantages compared to the traditional methods. First, it does not produce visible flicker and hence less distractive to participants. Second, it allows decoding attentional modulation of the neuronal response within a short period of time and hence, enables high-speed communication through increased information transfer rates. Combining the RIFT technique with novel wearable OPM sensors will allow making this BCI system more mobile and move development of BCIs to the next level.

Acknowledgements

The work was supported by the following funding: a James S. McDonnell Foundation Understanding Human Cognition Collaborative Award, United States of America (grant number 220020448), the Wellcome Trust Investigator Award in Science, United Kingdom (grant number 207550), a BBSRC grant, United Kingdom (BB/R018723/1) as well as the Royal Society Wolfson Research Merit Award, United Kingdom.

Footnotes

Declarations of Competing Interest

There are no competing interests.

Contributor Information

Yulia Bezsudnova, Email: yxb968@student.bham.ac.uk.

Anna Kowalczyk, Email: a.u.kowalczyk@bham.ac.uk.

Ole Jensen, Email: o.jensen@bham.ac.uk.

Alexander Zhigalov, Email: a.zhigalov@bham.ac.uk.

Data availability

Data will be made available on request.

References

  1. Bagherzadeh Y, Baldauf D, Pantazis D, Desimone R. Alpha Synchrony And The Neurofeedback Control Of Spatial Attention. Neuron. 2020;105(3):577–587.:e5. doi: 10.1016/j.neuron.2019.11.001. [DOI] [PubMed] [Google Scholar]
  2. Benaroch C, Sadatnejad K, Roc A, Appriou A, Monseigne T, Pramij S, Mladenovic J, Pillette L, Jeunet C, Lotte F. Long-term BCI training of a tetraplegic user: adaptive riemannian classifiers and user training. Front Hum Neurosci. 2021;15 doi: 10.3389/fnhum.2021.635653. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bergmann TO, Lieb A, Zrenner C, Ziemann U. Pulsed facilitation of corticospinal excitability by the sensorimotor μ-alpha rhythm. J Neurosci. 2019;39(50):10034–10043. doi: 10.1523/JNEUROSCI.1730-19.2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Biasiucci A, Leeb R, Iturrate I, Perdikis S, Al-Khodairy A, Corbet T, Schnider A, Schmidlin T, Zhang H, Bassolino M, Viceic D, et al. Brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke. Nat Commun. 2018;9(1) doi: 10.1038/s41467-018-04673-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Boto E, Holmes N, Leggett J, Roberts G, Shah V, Meyer SS, Muñoz LD, Mullinger KJ, Tierney TM, Bestmann S, Barnes GR, et al. Moving magnetoencephalography towards real-world applications with a wearable system. Nature. 2018;555(7698):657–661. doi: 10.1038/nature26147. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Brickwedde M, Krüger MC, Dinse HR. Somatosensory alpha oscillations gate perceptual learning efficiency. Nat Commun. 2019 doi: 10.1038/s41467-018-08012-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Brickwedde M, Schmidt MD, Krüger MC, Dinse HR. 20 Hz steady-state response in somatosensory cortex during induction of tactile perceptual learning through LTP-like sensory stimulation. Front Hum Neurosci. 2020 doi: 10.3389/fnhum.2020.00257. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Brooks MJ, Boto E, Rea M, Shah V, Osborne J, Holmes N, Hill RM, Leggett J, Rhodes N, Bowtell R. Theoretical advantages of a triaxial optically pumped magnetometer magnetoencephalography system. Neuroimage. 2021 doi: 10.1016/j.neuroimage.2021.118025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Cawley GC, Talbot NLC. Improved sparse least-squares support vector machines. Neurocomputing. 2002;48 www.elsevier.com/locate/neucom . [Google Scholar]
  10. Cheng M, Gao X, Gao S, Xu D. Design and implementation of a braincomputer interface with high transfer rates. IEEE Trans Biomed Eng. 2002;49(10):1181–1186. doi: 10.1109/TBME.2002.803536. [DOI] [PubMed] [Google Scholar]
  11. Colon E, Legrain V, Mouraux A. Steady-state evoked potentials to study the processing of tactile and nociceptive somatosensory input in the human brain. Neurophysiol Clin/Clin Neurophysiol. 2012;42(5):315–323. doi: 10.1016/J.NEUCLI.2012.05.005. [DOI] [PubMed] [Google Scholar]
  12. Crameri F, Shephard GE, Heron PJ. The misuse of colour in science communication. Nat Commun. 2020;11(1) doi: 10.1038/s41467-020-19160-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Drijvers L, Jensen O, Spaak E. Rapid invisible frequency tagging reveals nonlinear integration of auditory and visual information. Human Brain Mapping. 2021 doi: 10.1002/hbm.25282. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Duecker K, Gutteling TP, Herrmann CS, Jensen O. No evidence for entrainment: endogenous gamma oscillations and rhythmic flicker responses coexist in visual cortex. J Neurosci. 2021;41(31):6684–6698. doi: 10.1523/JNEUROSCI.3134-20.2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Fisher RS, Harding G, Erba G, Barkley GL, Wilkins A. Photic- and pattern-induced seizures: a review for the Epilepsy Foundation of America Working Group (In) Epilepsia. 2005;46(9) doi: 10.1111/j.1528-1167.2005.31405.x. [DOI] [PubMed] [Google Scholar]
  16. Haegens S, Zion Golumbic E. Neuroscience and Biobehavioral Reviews. Vol. 86. Elsevier Ltd; 2018. Rhythmic facilitation of sensory processing: a critical review; pp. 150–165. [DOI] [PubMed] [Google Scholar]
  17. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. 2nd. Springer; 2008. [Google Scholar]
  18. Herbst SK, Javadi AH, van der Meer E, Busch NA. How long depends on how fast–perceived flicker dilates subjective duration. PLoS One. 2013;8(10):e76074. doi: 10.1371/journal.pone.0076074. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Herrmann CS. Human EEG responses to 1–100 Hz flicker: resonance phenomena in visual cortex and their potential correlation to cognitive phenomena. Experimental Brain Research. 2001;137:346–353. doi: 10.1007/s002210100682. [DOI] [PubMed] [Google Scholar]
  20. Horschig JM, Oosterheert W, Oostenveld R, Jensen O. Modulation of Posterior Alpha Activity by Spatial Attention Allows for Controlling a Continuous Brain-Computer Interface. Brain Topography. 2015;28(6):852–864. doi: 10.1007/s10548-014-0401-7. [DOI] [PubMed] [Google Scholar]
  21. Jensen O, Mazaheri A. Shaping functional architecture by oscillatory alpha activity: Gating by inhibition. Front Hum Neurosci. 2010;4 doi: 10.3389/fnhum.2010.00186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kaplan AY, Shishkin SL, Ganin LP, Basyul IA, Zhigalov A. Adapting the P300-Based Brain-Computer Interface for Gaming: A Review. IEEE Transactions on Computational Intelligence and AI in Games. 2013;5(2):141–149. doi: 10.1109/TCIAIG.2012.2237517. [DOI] [Google Scholar]
  23. Keitel C, Quigley C, Ruhnau P. Stimulus-driven brain oscillations in the alpha range: Entrainment of intrinsic rhythms or frequency-following response? J Neurosci. 2014;34(31):10137–10140. doi: 10.1523/JNEUROSCI.1904-14.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kelly SP, Lalor EC, Reilly RB, Foxe JJ. Visual spatial attention tracking using high-density SSVEP data for independent brain-computer communication. IEEE Trans Neural Syst Rehabil Eng. 2005;13(2):172–178. doi: 10.1109/TNSRE.2005.847369. [DOI] [PubMed] [Google Scholar]
  25. Landis C. Determinants of the critical flicker-fusion threshold. Physiol Rev. 1954;34(2):259–286. doi: 10.1152/physrev.1954.34.2.259. [DOI] [PubMed] [Google Scholar]
  26. Lin FC, Zao JK, Tu KC, Wang Y, Huang YP, Chuang CW, Kuo HY, Chien YY, Chou CC, Jung TP. SNR analysis of high-frequency steady-state visual evoked potentials from the foveal and extrafoveal regions of human retina; Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2012. pp. 1810–1814. [DOI] [PubMed] [Google Scholar]
  27. Lotte F, Jeunet C. Towards Improved BCI based on Human Learning Principles; The 3rd International Winter Conference on Brain-Computer Interface; Gangwon-Do. 2015. pp. 1–4. [DOI] [Google Scholar]
  28. Ma P, Dong C, Lin R, Ma S, Jia T, Chen X, Xiao Z, Qi Y. A classification algorithm of an SSVEP brain-Computer interface based on CCA fusion wavelet coefficients. J Neurosci Methods. 2022;371:109502. doi: 10.1016/j.jneumeth.2022.109502. [DOI] [PubMed] [Google Scholar]
  29. Müller MM, Hillyard S. Concurrent recording of steady-state and transient event-related potentials as indices of visual-spatial selective attention. Clin Neurophysiol. 2000;111(9):1544–1552. doi: 10.1016/S1388-2457(00)00371-0. [DOI] [PubMed] [Google Scholar]
  30. Müller MM, Picton TW, Valdes-Sosa P, Riera J, Teder-Sälejärvi WA, Hillyard SA. Effects of spatial selective attention on the steady-state visual evoked potential in the 20-28 Hz range. Cogn Brain Res. 1998;6(4):249–261. doi: 10.1016/S0926-6410(97)00036-0. [DOI] [PubMed] [Google Scholar]
  31. Oostenveld R, Fries P, Maris E, Schoffelen JM. FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intell Neurosci. 2011;2011 doi: 10.1155/2011/156869. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Peles O, Werner-Reiss U, Bergman H, Israel Z, Vaadia E. Phase-specific microstimulation differentially modulates beta oscillations and affects behavior. Cell Rep. 2020;30(8):2555–2566.:e3. doi: 10.1016/j.celrep.2020.02.005. [DOI] [PubMed] [Google Scholar]
  33. Regan D. Steady-state evoked potentials. J Opt Soc Am. 1977;67(11):1475. doi: 10.1364/JOSA.67.001475. [DOI] [PubMed] [Google Scholar]
  34. Regan D. Comparison of transient and steady-state methods. Ann N Y Acad Sci. 1982 doi: 10.1111/j.1749-6632.1982.tb50784.x. [DOI] [PubMed] [Google Scholar]
  35. Seijdel N, Marshall TR, Drijvers L. Rapid Invisible Frequency Tagging (RIFT): a promising technique to study neural and cognitive processing using naturalistic paradigms. Cerebral Cortex. 2022 doi: 10.1093/cercor/bhac160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Seitz AR, Nanez JE, Holloway SR, Watanabe T. Visual experience can substantially alter critical flicker fusion thresholds. Hum Psychopharmacol: Clin Exp. 2005;20(1):55–60. doi: 10.1002/hup.661. [DOI] [PubMed] [Google Scholar]
  37. Snyder A. Steady-state vibration evoked potentials: description of technique and characterization of responses. Electroencephalogr Clin Neurophysiol /Evoked Potentials Sect. 1992;84(3):257–268. doi: 10.1016/0168-5597(92)90007-X. [DOI] [PubMed] [Google Scholar]
  38. Spaak E, de Lange FP, Jensen O. Local entrainment of alpha oscillations by visual stimuli causes cyclic modulation of perception. J Neurosci. 2014;34(10):3536–3544. doi: 10.1523/JNEUROSCI.4385-13.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Stapells DR, Linden D, Braxton Suffield J, Hamel G, Picton TW, Wp MT. Electrophysiologic Techniques in Audiology and Otology Human Auditory Steady State Potentials. 1984;5(2) doi: 10.1097/00003446-198403000-00009. [DOI] [PubMed] [Google Scholar]
  40. Toffanin P, de Jong R, Johnson A, Martens S. Using frequency tagging to quantify attentional deployment in a visual divided attention task. Int J Psychophysiol. 2009;72(3):289–298. doi: 10.1016/J.IJPSYCHO.2009.01.006. [DOI] [PubMed] [Google Scholar]
  41. Wolpaw JR, Birbaumer N, McFarland Dj, Pfurtscheller G, Vaughan TM. Brain-computer interfaces for communication and control. Clin Neurophysiol. 2002;113:767–791. doi: 10.1016/S1388-2457(02)00057-3. [DOI] [PubMed] [Google Scholar]
  42. Yu G, Herrmann JP, Katz LN, Krauzlis RJ. Microsaccades as a marker not a cause for attention-related modulation. eLife. 2022;11:e74168. doi: 10.7554/eLife.74168. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Zhigalov A, Jensen O. Alpha oscillations do not implement gain control in early visual cortex but rather gating in parieto-occipital regions. Hum Brain Mapp. 2020;41(18):5176–5186. doi: 10.1002/hbm.25183. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Zhigalov A, Herring JD, Herpers J, Bergmann TO, Jensen O. Probing cortical excitability using rapid frequency tagging. NeuroImage. 2019;195:59–66. doi: 10.1016/J.NEUROIMAGE.2019.03.056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Zrenner C, Desideri D, Belardinelli P, Ziemann U. Real-time EEG-defined excitability states determine efficacy of TMS-induced plasticity in human motor cortex. Brain Stimul. 2018;11(2):374–389. doi: 10.1016/J.BRS.2017.11.016. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Data will be made available on request.

RESOURCES