Functional near-infrared spectroscopy-based affective neurofeedback: feedback effect, illiteracy phenomena, and whole-connectivity profiles

Abstract.

Background: Affective neurofeedback constitutes a suitable approach to control abnormal neural activities associated with psychiatric disorders and might consequently relief symptom severity. However, different aspects of neurofeedback remain unclear, such as its neural basis, the performance variation, the feedback effect, among others.

Aim: First, we aimed to propose a functional near-infrared spectroscopy (fNIRS)-based affective neurofeedback based on the self-regulation of frontal and occipital networks. Second, we evaluated three different feedback approaches on performance: real, fixed, and random feedback. Third, we investigated different demographic, psychological, and physiological predictors of performance.

Approach: Thirty-three healthy participants performed a task whereby an amorphous figure changed its shape according to the elicited affect (positive or neutral). During the task, the participants randomly received three different feedback approaches: real feedback, with no change of the classifier output; fixed feedback, keeping the feedback figure unmodified; and random feedback, where the classifier output was multiplied by an arbitrary value, causing a feedback different than expected by the subject. Then, we applied a multivariate comparison of the whole-connectivity profiles according to the affective states and feedback approaches, as well as during a pretask resting-state block, to predict performance.

Results: Participants were able to control this feedback system with $70.00\%\pm 24.43\%$ ($p<0.01$) of performance during the real feedback trials. No significant differences were found when comparing the average performances of the feedback approaches. However, the whole functional connectivity profiles presented significant Mahalanobis distances ($p\ll 0.001$) when comparing both affective states and all feedback approaches. Finally, task performance was positively correlated to the pretask resting-state whole functional connectivity ($r=0.512$, $p=0.009$).

Conclusions: Our results suggest that fNIRS might be a feasible tool to develop a neurofeedback system based on the self-regulation of affective networks. This finding enables future investigations using an fNIRS-based affective neurofeedback in psychiatric populations. Furthermore, functional connectivity profiles proved to be a good predictor of performance and suggested an increased effort to maintain task control in the presence of feedback distractors.

1. Introduction

We can describe neurofeedback and brain–computer interfaces (BCI) as a group of devices and protocols that use neurophysiological signals to detect mental states and take this information to promote more realistic interactions between humans and machines.^1,2 For this, the participant might reach the self-control of his neural response patterns directly and consciously, without the interference of external stimuli.¹ Specifically to the affective neurofeedback, this self-control focuses on areas or networks related to different affective states, such as basic emotions, or states of valence.^3,4 Generally, affective neurofeedback protocols are based on electrophysiological asymmetries in frontal areas using electroencephalography (EEG) or different hemodynamic patterns in specific cortical and/or subcortical areas using functional magnetic resonance imaging (fMRI).⁴ Among a wide range of applications, the use of affective neurofeedback constitutes a suitable approach to control abnormal neural activities associated with psychiatric disorders and might consequently relief symptom severity.⁵ Previous studies, including healthy subjects and patients with schizophrenia, major depressive disorder, personality disorders, addiction, obsessive-compulsive disorder, among others, show that the voluntary control of the neural activity in regions of interest is feasible.^6,7 Furthermore, in some cases, this control was associated with clinical improvement.^7,8

Recently, functional near-infrared spectroscopy (fNIRS) has been proposed as a source of neurophysiological information to neurofeedback systems,⁹ including affective neurofeedback applications.¹⁰ fNIRS uses low-energy light detectors and transmitters to indirectly measure the local neural activity based on changes of oxyhemoglobin (${\mathrm{O}}_2\mathrm{Hb}$) and deoxyhemoglobin (HHb) concentration in the cortical surface,¹¹ including the prefrontal cortex (PFC), a cortical area commonly related to affect induction and processing.^12,13 For neurofeedback applications, fNIRS has several advantages: (1) it has relatively easier data acquisition protocols, (2) reduces the discomfort and anxiety of patients during experimental preparation and data acquisition, (3) can be acquired in comfortable seating positions; and (4) can be expanded to naturalistic settings. Moreover, it bears the additional advantages of portability, simplicity, and computational economy of its features, opening doors for robust applications using neurofeedback.^9,12,14,15

In this context, we first aimed to evaluate the efficacy of an fNIRS-based affective neurofeedback system. In fact, previous studies reported neurofeedback experiments founded on the self-regulation of activity in the orbitofrontal cortex (OFC) and PFC using fMRI^16,17 and fNIRS¹⁰ but limited to a univariate approach. Additionally, recent meta-analyses demonstrated that the occipital area is a core region to affective elicitation^18,19 and might be related to the vividness and effectiveness of the strategy of self-regulation.^20,21 Thus, our system focuses on these frontal and occipital networks to induce self-regulation based on positive affect. Indeed, this approach was already demonstrated using frontal areas and an fMRI neurofeedback,¹⁷ however not yet reported using fNIRS or including the occipital network.

While the previous literature explores different topics of affective neurofeedback implementation, two important aspects should be carefully observed: the feedback approach and the illiteracy phenomena. The feedback approach refers to the information continuously presented to the participant about his/her performance.²² This immediate response is used to keep the user’s interest and attention, besides allowing the brain to develop fast and practical strategies to adjust and improve neural activity control.²³ To our knowledge, no other studies evaluated the feedback effect in affective neurofeedback experiments. However, motor imagery experiments suggest that the feedback can lead to distraction and reduced attention,²⁴ or generate frustration and stress.²⁵ Thus, the feedback might cause an essential impact on affective neurofeedback protocols since this would hinder the elicitation and maintenance of the targeted emotions or affective states.

Therefore, our second aim was to evaluate three different feedback approaches on performance: real, fixed, and random feedback. High control performance with the real feedback would be expected due to the possibility of reinforcement or corrections of the elicitation strategies.²² On the other hand, although the absence of feedback (fixed condition) does not allow corrections to the strategies in use, its reduction of distractors could lead to performances close to the real condition.²⁴ Finally, a random feedback would cause an initial encouragement²⁶ but also might lead to frustration and irritation²⁵ after a while. Thus, we would expect a low control performance in this condition.

The illiteracy phenomena state that even with long periods of training, clear instructions, and improvements in the experimental protocol, it is expected that some participants will present poor control performance.²⁷ These are examples of “nonperformers” or “illiterates,” which can compose up to 50% of potential neurofeedback users.^25,28 During the last years, there is an intense debate about possible predictors for this lack of ability to control different protocols of neurofeedback.²⁹ While some studies evaluated the influence of the mental strategy used to control the neurofeedback system,^30,31 others correlated psychological aspects with performance, such as the self-confidence,³² frustration,²⁵ and concentration.³³ Using neurophysiological data, the neurofeedback performance was predicted from functional^28,34–38 and structural³⁹ resting-state measures. Therefore, our third aim was to evaluate possible predictors to the affective neurofeedback performance. Based on previous findings, we might expect the pretask resting state as a promising predictor of performance.

However, instead of performing univariate analyses to understand the feedback effect and the illiteracy phenomena, we applied here a connectivity-based multivariate analysis. For this, we used the connectivity information from all the recorded areas simultaneously. This approach was inspired in the concept of functional connectivity fingerprints,^40,41 which is based on the idea that individuals have a functional connectivity profile that is both unique and reliable, similarly to a fingerprint.⁴¹ Hence, with this whole-connectivity approach, we were able to evaluate the contribution of the entire set of functional connections to the desired aspect (here, the performance in different affective states and feedback approaches).⁴⁰

2. Methods

2.1. Participants

Thirty-three healthy participants (17 females), aged between 20 and 35 years (mean age of $25.58\pm 3.26$ years) and all undergraduate or graduate students, were recruited. The subjects had no diagnosis of neurological (ICD-10: G00-G99) and/or psychiatric diseases (ICD-10: F00-F99) and had normal or corrected-to-normal vision.

Ethical approval was obtained from the local ethics committee and all participants provided written consent prior to participation.

2.2. Data Acquisition

The recording was performed using the NIRScout System (NIRx Medical Technologies, LLC., Los Angeles, California) with an array of optodes (12 light sources/emitters and 12 detectors) covering the orbitofrontal, prefrontal, temporal, and occipital areas. Optodes were arranged in an elastic band, with nine pairs of source–detectors positioned over the fronto-temporal regions and three pairs of source–detectors over the occipital region (Fig. 1). We selected these regions considering the core role of the PFC, OFC, and occipital cortex in the elicitation of affective states.^18,19 Four positions of the International 10–20 System were adopted as reference points during the setup: detectors 1 and 9 were positioned approximately over the T7 and T8 positions, respectively, whereas the Fpz and Oz were in the center of channels 5–5 and 11–11, respectively (Fig. 1). Source–receptor distance was 30 mm for contiguous optodes, and the used wavelengths were 760 and 850 nm. The differential pathlength factor (DPF) was set to 7.25 and 6.38, respectively, for both fronto-temporal and occipital regions.⁴² Signals obtained from these 32 channels were measured with a sampling rate of 5.2083 Hz (maximum sampling rate of the equipment—62.5 Hz—divided by the number of sources—12) using the NIRStar 14.0 software (NIRx Medical Technologies, LLC, Los Angeles, California).

Fig. 1.

(a) Schematic representations of channel configuration. Different colors represent the regions of interest of each NIRS channel: yellow for the lateral orbitofrontal cortex (lOFC), green for the medial orbitofrontal cortex (mOFC), blue for the medial prefrontal cortex (mPFC), pink for the lateral prefrontal cortex (lPFC), purple for the occipital/striate cortex, and orange for the occipital/primary visual cortex. Pictures of the probe used during the data acquisition are provided in (b) for the left lateral, (c) for the frontal, (d) for the right lateral, and (e) for the back views. In all subfigures, red squares represent sources, blue circles represent the detectors, dotted lines the NIRS channels.

2.3. Experimental Configuration

Each subject performed the neurofeedback task (see Sec. 2.4). In addition, they completed an 11-point Likert mood scale immediately before and after the session to quantify sleepiness, agitation, strength, confusion, agility, apathy, satisfaction, worry, perspicacity, stress, attention, capacity, happiness, hostility, interest, and retraction.⁴³

For the experiment execution, subjects were seated in a padded chair with armrests, positioned at a 1-m distance in front of the monitor. They were asked to remain relaxed, with hands within sight resting on the table or on the armrests of the chair. They were also asked to avoid eye movements, as well as any body movement. The recording room became completely dark and the subject used earplugs.

2.4. Neurofeedback Task

During the neurofeedback task, subjects were asked to use their mental states to transform an amorphous figure in a perfect circle. For this purpose, they were instructed to imagine/remember personal experiences (autobiographical memory) with positive affect context during “positive trials” or to remain relaxed (not thinking about particularly emotional contents, here called as neutral affect) during “neutral trials.” The trial label varied according to the color of the figure presented on the screen (blue and yellow, respectively). The session consisted of two blocks of 5 min of continuous resting state (before and after the neurofeedback test), two training blocks used to train the classifier and two test blocks with visual feedback about the subject performance [Fig. 2(a)].

Fig. 2.

(a) Block structure of the experimental protocol, (b) example of trials distribution and (c) screen events during a classifier training block, and (d) example of trials distribution and (e) screen events during a feedback test block.

Visual stimuli were created and presented using the Psychophysics Toolbox extensions.^44–46

2.4.1. Classifier training blocks (no feedback)

Each classifier training block consisted of 10 trials (5 for positive affect and 5 for neutral affect) presented in a random order [Fig. 2(b)]. For the first 5 s of each trial, a white cross is displayed in the center of a black screen. This interval is essential to restore the fNIRS baseline level after the previous trial. A blue (indicating a trial for positive affect elicitation) or yellow (indicating a neutral trial) amorphous figure appears in the center of the display, and the instruction was to perform the corresponding affective task. To replicate the length of the test block trials (see Sec. 2.4.2), the figure remains on screen for 32 s (2 s of initial instruction $+30\mathrm{s}$ of feedback) [Fig. 2(c)].

After the figure disappears, a self-evaluation screen is presented, and participants were instructed to blink and move in this period but not in the other phases. Due to this orientation, this screen had no preset duration to allow the participant to feel comfortable to proceed to the next trial. At this point, the user was asked if he/she was able to elicit the proper neutral or positive experience, assigning a score in a 1 to 9 scale.

For each trial, real-time signal processing of both ${\mathrm{O}}_2\mathrm{Hb}$ and HHb concentrations from all NIRS channels was carried out every 1 s. Data from the previous 2 s (beginning with the instruction period) were merged with the actual second to compose a 3-s moving window [Fig. 3(a)]. The signals of each NIRS channel were initially filtered using a simple moving averages filter, used as an online low-pass filter with a cutoff of 1 Hz.^47–49 After this, the variation of ${\mathrm{O}}_2\mathrm{Hb}$ and HHb concentrations was calculated by the Beer–Lambert law, using the pretest resting block as the reference of concentration. Finally, inspired by voxel normalization approaches for fMRI-based neurofeedback,^50,51,52 for each channel, the moving ${\mathrm{O}}_2\mathrm{Hb}$ and HHb concentration values were corrected by the averaged concentrations from the same channel during the previous neutral condition. This approach was used for both positive and neutral conditions (although for the neutral condition this procedure is only possible from the second trial since it needs the first one for normalization).

Fig. 3.

Schematic representation of (a) the real-time signal processing, during both block of classifier training and block of feedback test and (b) the feedback logic. (c) An illustrative example of the window movement according to the trial timeline.

At the end of each block of classifier training, the resulting $300\times 64$ matrix was then used to train a linear discriminant analysis (LDA) classifier to recognize the two classes (positive and neutral affect). In this matrix, lines correspond to 300 examples resulting from 30 moving windows for each trial (5 per class, 10 in total). Columns presented 64 features of mean concentration of ${\mathrm{O}}_2\mathrm{Hb}$ and HHb from a total of 32 channels each. The use of mean concentration as input was due to its simplicity and discriminative power in BCI experiments.¹⁴

LDA was used based on its extensive application in BCI and neurofeedback protocols, allowing further comparisons with other experiments.^53,54 Also, considering the most common classifiers, LDA seems to consider all discriminative information available, allowing the interpretation of all areas/connections evoked during the task.⁵⁵ Thus, although studies such as Ref. 15 present other classifiers as more accurate, we used the LDA algorithm based on its informative power. The LDA was implemented using the BCILAB toolbox⁵⁶ and applying the default settings.

2.4.2. Feedback test blocks

Each feedback test block consisted of 11 trials presented in a random order, totaling 22 trials at the end of the experiment (10 trials with real, 6 with fixed, and 6 with random feedback) [Fig. 2(d)].

Each test trial starts with a baseline period comprised of a white cross in the center of a black screen [see Fig. 2(e)]. After 5 s, the cross disappears and a blue or yellow amorphous figure appears, indicating the target task. This stays the same for 2 s, after which the shape of the figure begins to change according to the output of the classifier. For these trials, real-time signal processing of data follows the same steps as previously described in Sec. 2.4.1 and presented in Fig. 3(a). Each moving window was then real-time classified using the LDA model created in the training blocks and the output ranged from $-1$ for definite neutral affect classification to $+1$ for definite positive affect classification.

According to the feedback of interest, each output of the classifier could be multiplied by a different numeric value [Fig. 3(b)]. During trials with real feedback, the outputs of the classifier were always multiplied by one, with no change of the original value. During trials with fixed feedback, the outputs were always multiplied by zero, keeping the feedback figure unmodified (in other words, keeping the target figure as feedback figure). Finally, during trials with random feedback, each output was multiplied by a random value between $-1$ and $+1$, causing an output possibly different of the expected by the subject. To facilitate the understanding of the feedback logic, the Appendix presents examples of signal traces and the consequent feedback of a randomly chosen subject during each feedback condition.

2.5. Offline Analysis

2.5.1. Quantifying performance

To compute performance, we first quantified the number of trials correctly performed during each feedback. Here, we considered a trial as correctly performed if the sum of the classifier outputs resulting from 30 moving windows was in accordance with the trial label, that is, if $\sum _{n=1}^{30}{y}_n<0$ for neutral trials or $\sum _{n=1}^{30}{y}_n\ge 0$ for positive trials. Then, the number of trials correctly performed was divided by the total of trials with the respective feedback. For example, if 8 of 10 trials were successfully executed, a real feedback would result in a performance of 80%; if 2 of 6 trials were correctly performed, a fixed feedback would result in a performance of 33.33%; and if 4 of 6 trials were successfully completed, a random feedback would result in 66.67%.

The difference of performance between the three feedbacks was evaluated by a paired-samples $t$-test, with Bonferroni correction to three multiple comparisons. Possible relations between the accuracies of each feedback type were also evaluated using a Spearman’s correlation and the $p$-values were Bonferroni corrected for multiple comparisons (three pairs of feedback). Finally, to investigate the experiment efficacy as a neurofeedback system, we considered the real feedback performance as the general task performance. Then, the significance of task performance was evaluated by a one-sample $t$-test against chance level (50%).

2.5.2. fNIRS preprocessing

The offline preprocessing was performed using the MATLAB software (MathWorks, Massachusetts) with the nirsLAB v2014.12 toolbox (NIRx Medical Technologies, LLC., Los Angeles, California). Each participant’s raw data were digitally bandpass filtered by a linear-phase FIR filter (0.01 to 0.2 Hz) to filter noises due to the heartbeat (0.8 to 1.2 Hz), respiration (0.3 Hz), and Mayor waves ($\sim 0.1\mathrm{Hz}$).^57–59 Then, each wavelength was detrended by their respective whole length record (without segmentation), and the variation in concentration of ${\mathrm{O}}_2\mathrm{Hb}$ and HHb was calculated by the Beer–Lambert law (DPF set to 7.25 and 6.38, respectively).⁴²

2.5.3. Evaluating literacy

To evaluate possible factors related to the task performance, we exclusively considered performances achieved during the real feedback. Therefore, we first tested the gender effect on literacy by a two-sample $t$-test. Then, the age effect was evaluated using a Spearman’s correlation coefficient.

The whole-connectivity profiles extracted from the 5 min of resting state recorded before and after the experiment were also tested as a possible predictor of literacy. For this, the connectivity between a pair of regions was evaluated by the magnitude squared coherence of the two corresponding NIRS channels,⁶⁰ here calculated using 20-s Hamming windows with 50% of overlap. This procedure was repeated for all combinations of channels and for both ${\mathrm{O}}_2\mathrm{Hb}$ and HHb, generating two $32\times 32$ matrices for each block of resting state (pre- and posttask). Then, we averaged all connectivity values in each matrix, resulting in a single whole-connectivity score for each block and each chromophore. After this, these values were correlated to the performance by a Spearman’s correlation coefficient, and the respective $p$-values were Bonferroni corrected for four multiple comparisons ($2\text{resting-state blocks}\times 2\text{chromophores}$).

Finally, the influence of psychological factors was also explored. The difference between the mood scores after and before the neurofeedback test ($\mathrm{\Delta }\text{mood}$) was correlated with the task performance by a Spearman’s correlation and the $p$-values were Bonferroni corrected for multiple comparisons (16 $\mathrm{\Delta }\text{mood}$ scores).

2.5.4. Brain connectivity during different affective states and feedback approaches

The connectivity between a pair of regions was evaluated by the magnitude squared coherence of the two corresponding NIRS channels,⁶⁰ here calculated using 20-s Hamming windows with 50% of overlap. This procedure was repeated for both ${\mathrm{O}}_2\mathrm{Hb}$ and HHb, as well as all possible combinations of NIRS channels, generating two $32\times 32$ matrices for each affective state (neutral and positive) and feedback type (real, fixed, and random).

The difference between the whole-connectivity matrices during each pair of affective states or feedback approaches was performed using the Mahalanobis distance. The Mahalanobis distance uses the data from each matrix as a multidimensional dataset and compares it to another multidimensional dataset. Mahalanobis distance showed greater sensitivity compared to other distance measures, because it considers mean and variance differences in connectivity matrices.⁴⁰ Its output is a number representing how distant one dataset is from the other. The smaller the distance between the matrices, the greater the similarity between the connectivity patterns during the comparison.⁶¹

To obtain each Mahalanobis distance, first, for each subject, we take the connectivity matrices for each trial in condition A (e.g., positive affect during real feedback) and condition B (e.g., neutral affect during real feedback). Then, we get the Mahalanobis distance between the matrices of conditions A and B. After this, we randomly shuffle the labels of the trials and calculate the Mahalanobis distance between the permuted A and B. We repeated this step ${10}^3$ times to generate a permutation distribution. After that, we calculated the $z$-scores of the real distance between A and B relative to the distribution of the permutations (this was accomplished by assigning a $p$-value to the real distance relative to the permutation distribution, and then using the inverse normal distribution to transform the $p$-value into a $z$-score). Up until here, all the analysis was done at the subject level, and the next step was simply to calculate one-sample $t$-tests for each affect and feedback condition. The resulting $p$-values were Bonferroni corrected for 6 multiple comparisons (3 feedback × 2 chromophores) when comparing the affective task effect and for 12 multiple comparisons (2 affective states × 3 feedback combinations × 2 chromophores) when comparing the feedback effect.

3. Results

The general performance ($\text{median}\pm \text{standard-deviation}$), calculated considering only the real feedback, was $70.00\%\pm 24.43\%$, being significantly greater than chance level ($p<0.01$). This level of performance indeed demonstrates that the paradigm implemented here worked as an affective neurofeedback. As can be seen in Fig. 4(a), more than half of the participants reached performances above 50%. Moreover, for 19 participants, the performance was higher or equal to 70%, and 5 of them reached 100% performance.

Fig. 4.

In (a), each subject’s performance for real (blue), fixed (green), and random (red) feedback. The dotted line represents 50% level of performance and the continuous line the 70% level. In (b), the distribution of subjects according to their performances in positive ($y$-axis) and neutral trials ($x$-axis) during the real feedback. The diameter of the circle is proportional to the number of participants for which that level of performance was observed. The distribution of participants according to their accuracies and pretask resting-state connectivity scores is presented in (c) for HHb-based connectivity scores and in (d) for ${\mathrm{O}}_2\mathrm{Hb}$-based connectivity scores. In both cases, red lines represent the trend.

A more detailed exploration of these performances is obtained by considering the performance for the neutral and positive trials, independently. As can be seen in Fig. 4(b), 6 subjects obtained accuracies under 50% for both classes, as well as 16 subjects reached more than 50% for both trials.

In a second step, we compared the effect of three different feedback approaches in performance. Additionally to the real feedback, both fixed ($66.67\%\pm 20.57\%$, $p<0.001$) and random feedback conditions ($66.67\%\pm 13.06\%$, $p<0.001$) were significantly different from chance level. Although a positive correlation was found between real and fixed feedback performance curves ($r=0.745$, $p<0.001$), there were no significant differences when comparing the averages of each pair of feedback.

Additionally, we evaluated possible factors related to the illiteracy phenomena. No significant gender or age effects on performance were found. However, Fig. 4(c) shows that the pretask resting-state connectivity score calculated using HHb concentration was positively correlated to performance ($r=0.511$, $p=0.010$). In contrast, no significant correlation with the performance was found using the ${\mathrm{O}}_2\mathrm{Hb}$ concentration [Fig. 4(d)] or when evaluating posttask resting-state connectivity scores. Finally, no significant correlations were observed between $\mathrm{\Delta }\mathrm{mood}$ scores (VAMS questionnaire) and subjects’ performance.

To evaluate the relevance of each feature in our neurofeedback setup, we first normalized the weights assigned by the LDA for all subjects, during both training blocks. For each trained classifier (more details in Sec. 2.4.1), we first subtracted the minimum absolute weight from all features’ weights in that features set. Then, all resulting values were divided by the maximum absolute weight. Then, with weights now ranging from 0 to 1, we averaged these values among all participants. Figure 5 shows these averaged and normalized weights for features calculated using ${\mathrm{O}}_2\mathrm{Hb}$ and HHb concentrations. Although some of these channels present higher contribution to identifying one class, it is notable that the most relevant features that contribute for both classes are distributed around the lateral and medial portions of the OFC (with particular attention to the left mOFC), the medial parts of the PFC and occipital areas, and are based on HHb concentration.

Fig. 5.

Average weights assigned by the LDA classifier for each feature using HHb and ${\mathrm{O}}_2\mathrm{Hb}$ concentrations. In (a), the average of weights from all features in classes, and (b) the average of those weights relevant to classify (b) positive and (c) neutral trials. Hottest colors indicate higher relevance while cooler colors indicate lower relevance.

Lastly, we looked at the whole functional connectivity matrices related to positive and neutral conditions according to the delivered feedback. As can be seen in Figs. 6(a)–6(b) and 7(a)–7(b), HHb matrices presented the highest coherence values, with stronger connections among neighbors and homologous contralateral channels. Although the connectivity patterns remain remarkably similar for both positive and neutral trials, and for the three feedback conditions, all distance-based comparisons (task effect and feedback effect comparisons) presented statistical significance with $p\ll 0.001$, as shown in Fig. 8. To better visualize the differences in connectivity matrices due to the affective states, we plotted the difference matrices, which were calculated by subtracting the positive class from the neutral one [Figs. 6(c) and 7(c)]. Notably, the differences were generally positive, indicating that the neutral class has strong overall connectivity. Moreover, the real feedback presented the lower variation among all the feedback conditions.

Fig. 6.

Coherence matrices comparing all-to-all channels using ${\mathrm{O}}_2\mathrm{Hb}$ concentrations. Graphs from the (a) first and (b) second columns correspond to neutral and positive trials, respectively, and the (c) last column corresponds to the difference among them. Matrices from each line correspond to real, fixed, and random feedback, respectively. Hottest colors were indicating higher coherences (or differences) while cooler colors indicate lower coherences (or differences).

Fig. 7.

Coherence matrices comparing all-to-all channels using HHb concentrations. Graphs from the (a) first and (b) second columns correspond to neutral and positive trials, respectively, and the (c) last column corresponds to the difference among them. Matrices from each line correspond to real, fixed, and random feedback, respectively. Hottest colors were indicating higher coherences (or differences) while cooler colors indicate lower coherences (or differences).

Fig. 8.

Bar graph with mean and standard error for $z$-scored Mahalanobis distances between each comparison, during (a) task and (b) feedback effect evaluation. Asterisks represent significant difference from zero ($p<0.05$). Blue bars correspond to ${\mathrm{O}}_2\mathrm{Hb}$ data, whereas red bars to HHb data.

4. Discussion

Here, the three following objectives were set out: (i) to evaluate if it is possible to develop an fNIRS-based affective neurofeedback system using the self-control of networks activities, including the OFC, PFC, and occipital cortex, (ii) to further test the feedback effect in performance and in the subject’s multivariate functional connectivity, and (iii) to investigate possible demographic, psychological, or physiological predictors of performance.

4.1. Performance and Literacy

Exclusively considering the general performance (during real feedback), we found that our volunteers were able to self-control the accessed networks activity using neutral and positive affective states. Even more, the majority of our participants reached performances over the 70% threshold, which is suggested by the BCI/neurofeedback community as sufficient to perform device control and communication.⁶²

Subjects performing under the mentioned threshold would probably improve their performance after a few training sessions, as already observed in longitudinal experiments using different BCI and neurofeedback approaches.^63–66 However, it is expected that some of them keep the poor performances even after exhaustive training sessions or subject-specific improvements of the system (e.g., with specific features or different classifiers).²⁷ These are examples of BCI/neurofeedback illiterates, commonly described in the literature, whose best option to attain proficiency would be switching to another neurofeedback approach.^27,29

Here, we found the pretask resting-state connectivity as positively correlated to the task performance. It means that the greater the connectivity during the resting state, the greater might be the performance on an affective neurofeedback. This correlation is expected considering recent studies that report the default mode network (DMN, also known as the “resting-state network”) and the affective workspace network (AWN) being partly overlapped.^18,67–69 Also, considering that this connectivity score is based on the same regions used as input to the classifier, it is expected that highly interconnected cortices tend to produce precise and clear activation patterns. Furthermore, this result is in agreement with previous studies reporting the resting-state connectivity as a relevant predictor of performance in different BCI/neurofeedback tasks and recording modalities.^37,70,71

To the best of our knowledge, this is only the second work using an fNIRS-based real-time affective neurofeedback, and the first one to consider a multiregional approach. However, our results are in agreement with previous findings of hemodynamic-based neurofeedback, mainly with fMRI targeting the OFC^72,73 or the PFC.^10,16,17 As expected, relevant features were placed around the OFC, PFC, and the occipital cortex. These regions were recently listed as fundamentals to the affective induction and processing.^18,19 The mOFC, in special, was the most relevant feature among all. This is reasonable considering that this region guides internal responses to affective contexts,⁶⁸ which is a core aspect to achieve an affective-based self-regulation. Also, the occipital cortex plays an important function to improve the vividness and effectiveness of the autobiographical memory evocation.^20,21

Concerning the predominance of changes in HHb as relevant features, although ${\mathrm{O}}_2\mathrm{Hb}$ and HHb present similar results in comparative studies for LDA classification,¹⁴ the best features set might vary according to the application. For example, motor imagery studies have found ${\mathrm{O}}_2\mathrm{Hb}$ as the most robust features,^57,74 whereas in a similar affective experiment our group also reported a predominance of HHb features among the most relevant ones.^75,76 Previous studies describe different advantages for each measure. The ${\mathrm{O}}_2\mathrm{Hb}$ is reliable and shows higher retest reliability.⁷⁷ On the other hand, although presenting higher variability, HHb features are commonly related to the fMRI blood oxygen-level dependent signal.⁷⁸

Thus, these results open one more protocol option to be applied in clinical populations,^6,7 with the advantage that fNIRS is a portable and easy-to-setup equipment,^9,12 also enabling applications out of the lab,^79,80 and with a new multivariate approach.

4.2. Feedback Effect on Performance

The performance of the user of neurofeedback is usually variable during the learning process.⁸¹ Such variability tends to decrease when the user reaches complete control of the system. However, difficulties in properly controlling the neurofeedback system might cause frustration and disappointment, obviously impairing learning affective protocols. Moreover, for potential therapeutic applications, users will need to be able to self-regulate their networks activities during stressful situations, such as anxious or depressive states.⁶ In this context, it is important to simulate some distractors/stressing stimuli to evaluate its possible consequences in the neurofeedback tasks.

In the performance during fixed feedback, which might be considered as a mild task distractor, we found a high correlation with the performance in real feedback. No significant difference between these feedback conditions was observed. These results are in line with previous EEG findings.^24,25 We argue that this is an important result for the potential therapeutic application of an affective neurofeedback system. In a real situation, for instance, where a patient would need to apply the affective neurofeedback training to relief a given situational symptom, he will need to do this without any type of feedback. This scenario is indeed simulated by the fixed feedback condition.

Contrary to our expectations, the random feedback was not different from the other feedback approaches. However, this finding is consistent with previous results in other nonaffective tasks, such as motor imagery BCI in EEG.^26,82 Since this is the first experience of all participants with an affective neurofeedback system, a possible explanation is that a greater task engagement was caused by the random feedback, which worked as a strong task distractor.²⁶ Possibly, exposure to longer periods of random feedback would lead to frustration and demotivation.⁸²

4.3. Task and Feedback Effects on fNRIS Whole-Connectivity

Significant differences in distance in all the performed comparisons suggest that the neural networks accessed have different connectivity patterns for each condition (affective states and feedback).^40,61 Significant differences between positive and neutral connectivity profiles were already expected given that the classifier used in our neurofeedback was able to find different neural patterns during both affective states.⁵⁴

It is notable that the connectivity maps for both affective states are highly overlapped. This functional overlapping may be related to two neural networks that share some brain areas and connectivity pathways. The positive affect processing may be related to the AWN,^18,19,68 which involves subcortical areas classically listed as “emotional centers,” such as amygdala and ventral striatum, as well as cortical areas implicated in affective processing, such as the lOFC, the ventrolateral prefrontal cortex, the ventromedial prefrontal cortex, the dorsomedial prefrontal cortex (dmPFC), and the lateral portions of the right temporal/occipital cortex.^18,19,83 On the other hand, the neutral affective state might be reasonably related to increased activity in the DMN,^84,85 which include areas such as the mPFC, the posterior cingulate cortex, and the inferior parietal lobule.^86,87

Two of these overlapped areas play an important role in both tasks used during this affective neurofeedback experiment: the dmPFC and the occipital cortex. The dmPFC is highly associated and engaged during remembering of personal events (autobiographical memory),⁶⁹ as well as during the spontaneous thinking.⁸⁸ In addition, the occipital cortex is crucial to the quality of the autobiographical memory because it regulates subjective vividness during imagination.^20,21 Even more, fNIRS studies found high connectivity between contralateral occipital areas during resting state.^89,90 The higher connectivity found in neutral trials for all feedback approaches may also be explained by a DMN characteristic. Previous studies found that the brain activity decreases during different tasks when compared to passive mental states, which suggests the existence of a baseline neural activity in the absence of external attentional focus.^91–93

Finally, in both ${\mathrm{O}}_2\mathrm{Hb}$ and HHb maps (Figs. 6 and 7), the neutral to positive differences showed increased values according to the presented feedback (real feedback with lower differences and random feedback with higher differences). In accordance, an fNIRS study observed reduced prefrontal oxygenation when attentional distractors were presented during an affective task.⁹⁴ Negative distractors may lead to changes exclusively on the brain activity but not in the task performance. This may be caused by a compensatory brain effort to maintain similar performance levels even with competitive stimulus.⁹⁵

4.4. Innovation and Limitations

As previously mentioned, up to our knowledge, this is the first study to propose fNIRS-based affective neurofeedback based on the self-regulation of frontal and occipital networks. This is a relevant result considering that neurofeedback is a noninvasive and nonpharmacological approach for the treatment of psychiatric disorders.^6,7 Our protocol might be especially valuable for mood disorders, such as the major depressive disorder and the obsessive–compulsive disorder, considering the self-control of critical areas of its neurocircuitry.⁹⁶ In addition, we pioneered used a connectivity analysis to evaluate the feedback effect and the illiteracy phenomena related to affective neurofeedback protocols.

However, this study possesses some limitations of note. First, we adopted an unbalanced number of trials for real, fixed, and random feedback. This difference is a consequence of the protocol length, which might take up to 60 min considering the system setup and the task execution. Thereby, to avoid the influence of fatigue and stress related to the long duration,⁹⁷ we presented more trials with real feedback than with fixed and random feedback. This choice was based on the use of the real feedback to validate the neurofeedback protocol.

Additionally, this study has a limited sample size with controlled aspects, such as health history and educational level. Although it was enough to validate the neurofeedback protocol, as well as to allow an initial evaluation of the feedback effect and the illiteracy phenomena, this aspect should be improved in future studies.

Therefore, the next step of the research should consider the use of a balanced number of trials, as well as an increased sample size, for example, including psychiatric patients or participants with different educational levels. These aspects might result in more variability to validate our results and reinforce the better understanding of the self-control of neurofeedback protocols.^97–99 Also, future studies should use the self-evaluation scores to correlate it with the effective performance or connectivity patterns. This data will provide more information regarding the relationship between system and user from the participants’ perspective.

5. Conclusion

In conclusion, our results suggest that fNIRS might be a feasible tool to develop affective neurofeedback systems based on self-regulation of frontal and occipital networks. Additionally, it seems to be possible to predict performance using a short pretask resting-state period, suggesting that the general background connectivity rules the self-control capacity. Finally, although no significantly different performances were found using real, fixed, and random feedback conditions, offline functional connectivity profiles analyses suggest a neural basis for an increased effort to maintain task control in the presence of distractors.

Biography

Biographies for the authors are not available.

Appendix.

As previously described, each trial starts with a black screen with a cross during the first 5 s. This period is essential to both ${\mathrm{O}}_2\mathrm{Hb}$ and HHb concentrations to recover to its baseline level. Then, during the following 2 s, the orientation screen illustrates the trial class (a yellow figure for neutral trials or a blue figure for positive trials). The next second is used to compose the first moving window [for more details refer to Fig. 3(c)], meaning that the first feedback is provided 8 s after the beginning of the trial. Consequently, the 30th, and last, feedback screen appears after 37 s after the start of the trial.

The feedback presentation is ruled by the logic described in Sec. 2.4.2, where first, the classifier output is converted according to the feedback condition. Then, the figure format is shaped according to the trial class, as shown in Table 1. To facilitate the understanding of this logic, Figs. 9–11 show examples of signal traces and the consequent feedback of a randomly chosen subject during each feedback condition.

Table 1.

Simplified rules for screen updates according to the trial class.

Converted output	Neutral trial	Positive trial
Less than 0	More shaped figure	More deformed figure
Equal to 0	Same figure	Same figure
Greater than 0	More deformed figure	More shaped figure

Fig. 9.

Example of signal traces from subject 8 during a positive trial with real feedback, including the classifier output from each moving window and the converted output according to the respective feedback condition. Continuous green and dotted red lines represent HHb and ${\mathrm{O}}_2\mathrm{Hb}$ concentrations, respectively.

Fig. 10.

Example of signal traces from subject 8 during a neutral trial with fixed feedback, including the classifier output from each moving window and the converted output according to the respective feedback condition. Continuous green and dotted red lines represent HHb and ${\mathrm{O}}_2\mathrm{Hb}$ concentrations, respectively.

Fig. 11.

Example of signal traces from subject 8 during a neutral trial with random feedback, including the classifier output from each moving window and the converted output according to the respective feedback condition. Continuous green and dotted red lines represent HHb and ${\mathrm{O}}_2\mathrm{Hb}$ concentrations, respectively.

In Fig. 9, we can see an example of a positive trial with real feedback. In this case, the classifier output is the same as the converted output. It is possible since, during real feedback conditions, the classifier output is always multiplied by one. Then, the figure format will be reliable to the participant’s expectation. Also, following rules for positive trials as shown in Table 1, positive converted outputs lead to more shaped figures, whereas negative converted outputs lead to more deformed figures.

On the other hand, Fig. 10 shows an example of a neutral trial with fixed feedback. In this case, due to the fixed feedback condition, all results from feedback calculation are equal to zero. Therefore, following Table 1, the same figure is presented for all feedback screens.

Finally, Fig. 11 shows an example of a neutral trial with random feedback. In this case, feedback calculation outputs are predominantly different from the classifier output and, consequently, different from the self-regulation patterns. Then, following rules for neutral trials as shown in Table 1, converted outputs with negative results lead to more shaped figures, whereas positive results lead to more deformed figures.

Disclosures

All authors declare no conflicts of interest.