Co-localization of theta-band activity and hemodynamic responses during face perception: simultaneous electroencephalography and functional near-infrared spectroscopy recordings

Abstract.

Face-specific neural processes in the human brain have been localized to multiple anatomical structures and associated with diverse and dynamic social functions. The question of how various face-related systems and functions may be bound together remains an active area of investigation. We hypothesize that face processing may be associated with specific frequency band oscillations that serve to integrate distributed face processing systems. Using a multimodal imaging approach, including electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), simultaneous signals were acquired during face and object picture viewing. As expected for face processing, hemodynamic activity in the right occipital face area (OFA) increased during face viewing compared to object viewing, and in a subset of participants, the expected N170 EEG response was observed for faces. Based on recently reported associations between the theta band and visual processing, we hypothesized that increased hemodynamic activity in a face processing area would also be associated with greater theta-band activity originating in the same area. Consistent with our hypothesis, theta-band oscillations were also localized to the right OFA for faces, whereas alpha- and beta-band oscillations were not. Together, these findings suggest that theta-band oscillations originating in the OFA may be part of the distributed face-specific processing mechanism.

1. Introduction

The human face is a universally salient social stimulus that conveys uniquely valuable information to a viewer with the potential to influence the viewer’s actions, thoughts, and emotions. Many neuroimaging studies document that human neural systems differentiate faces from other visual stimuli. In particular, the fusiform face area (FFA) in the medial temporal lobe and the occipital face area (OFA) show increased hemodynamic responses to faces compared to objects and are thought to be part of a larger distributed network of face-specific areas.^1,2 There is evidence that these face-selective areas tend to be right-lateralized.^3–6 The OFA, while less well-studied than the FFA, is selective for faces and has been shown to be necessary for face perception.^7–9 For example, lesions to the OFA are associated with prosopagnosia even when the FFA is intact.¹⁰ There is also some evidence to support a hierarchical model of face perception, in which visual input travels from the primary visual cortex first to the OFA, and then on to the FFA and to the right superior temporal sulcus.^1,8,11 If true, this model suggests that the OFA plays an important role in the early stages of face processing.

Electroencephalography (EEG) studies also confirm the salience of face stimuli; for example, faces generate an N170 response, a negative amplitude event-related response that occurs $\sim 170\mathrm{ms}$ after the visual presentation of a face.¹² Although a negative peak can be generated by other visual stimuli, the N170 response occurs faster and the amplitude is greater in response to faces relative to other object stimuli. The N170 is generally right-lateralized in neurotypical individuals¹³ and correlates with measures of social function.¹⁴ Although many studies have provided evidence that the N170 is a specific neuronal response related to the visual processing of faces,^15–18 the effect of faces on oscillatory activity from groups of neurons has been less studied. We aimed to extend these well-documented findings of the face-specific N170 response by investigating frequency band oscillations that co-localize to face-specific areas during face perception.

In addition to event-related responses, EEG signals include specific frequency bands, which are thought to represent oscillatory activity of the neurons. In general, synchronization of different frequency bands has been shown to be related to attention-switching, memory, and other cognitive functions.^19–23 In visual tasks, theta oscillations (4 to 8 Hz) have been associated with stimulus perception²⁴ and stimuli with high emotional valence.²⁵ Theta-band activity is often associated with memory processes, and theta oscillations originating in the hippocampus may enable retention during memory tasks.^26,27 Additionally, theta power increases have been associated with memory in both visuospatial and verbal tasks.²⁸ Theta oscillations show power increases during passive visual tasks²⁹ and may play a role in error processing.³⁰ In visual perception, theta oscillations are associated with basic features of the visual cue.³¹

In addition to their role as salient visual stimuli, faces also trigger emotional and motivational systems. Theta-band responses to faces have been associated with these higher-order features. For example, in face perception tasks, increases in theta power in frontal electrodes were associated with viewing familiar faces.^32,33 Event-related desynchronization in the theta band in anterior electrodes has also been associated with emotional facial expressions, and this effect has been shown to be right-lateralized.³⁴ Theta-band synchronization in the right parietal cortex and insula increases during implicit perception of angry and happy faces, whereas it localizes to the temporal lobe and bilateral prefrontal cortices during explicit perception.³⁵ The same study found that emotional faces led to greater theta-band synchronization and individuals with higher sensitivity to emotional stimuli showed greater synchronization. Fearful faces have also been shown to increase theta-band power and increase the amplitude of the N170 effect.³⁶

Synchronizations in theta activity have been used as a measure of functional connectivity between frontal areas in oddball studies.²¹ Few studies have investigated the source localization of theta-band activity but task-dependent sources in the frontal, parietal, and temporal lobes have been reported. One study examining theta-band activity (6.5 to 8 Hz) during rest in patients with epilepsy localized activity to the medial frontal and parietal lobes, specifically the precuneus, anterior and posterior cingulate, and medial frontal gyrus.³⁷ The varied sources of theta-power increases have led Başar et al.²⁹ to propose that theta oscillations may drive functional connections between a widely distributed system of areas.

In contrast, alpha oscillations have been associated almost exclusively with attention mechanisms. Decreased alpha power during higher cognitive tasks and increased beta oscillations (13 to 30 Hz) during sensorimotor tasks have been shown in experiments using multimodal stimuli.³⁸ The low-frequency delta band (0.3 to 4 Hz) has been associated with salience detection of visual stimuli,³⁹ although evidence for its involvement specifically in face perception has been mixed.³⁸ Due to the association of the theta band with visual stimulation and the distributed, task-dependent nature of the anatomical sources of theta-band activity reported in the literature, we hypothesized that we would likely observe a spatially localized increase in theta-band oscillations in the OFA during a face perception task. We predicted that this localized increase in oscillatory activity would not be seen in the alpha and beta bands.

Simultaneous EEG measures of oscillatory activity and BOLD-related measures of face processing are not common practice due to the signal artifacts as a result of the rapid gradient switching in fMRI studies. However, functional near-infrared spectroscopy (fNIRS) acquires hemodynamic signals absent the high magnetic field. fNIRS is a neuroimaging method that uses absorption of near-infrared light to measure oxyhemoglobin (OxyHb) and deoxyhemoglobin (deOxyHb) concentrations in the cortex as a hemodynamic proxy for neural activity.^40–42 Until recently, fNIRS has mainly been used with infants, for whom fMRI studies are not an option. In the last decade, advances in signal processing, increased head coverage, and enhanced computational methods have made fNIRS a viable alternative to fMRI for investigating adult perception and cognition, particularly in natural conditions. Recent developments in the acquisition, processing, and 3-D representation of fNIRS signals allow for the presentation of hemodynamic signals that can be compared easily with those acquired using more traditional fMRI methods.^43,44 EEG-fNIRS experiments are well-suited to acquire electrocortical and hemodynamic signals related to faces.

One goal of acquiring simultaneous EEG and fNIRS measurements includes the ability to relate the fast electrical signals of EEG with the slow, hemodynamic signals of fNIRS. One such study accomplished this by correlating the raw signals and identifying electrodes and fNIRS channel locations, in which this correlation was high.⁴⁵ Other studies have correlated EEG signals with hemodynamic responses.^46–48 In epilepsy, aberrant EEG signals have been correlated with hemodynamic changes in areas associated with default mode activity.⁴⁹ Theta-band activity, in particular, has been correlated with increased hemodynamic activity in task-positive networks as well as decreased activity in the default mode network with increasing cognitive demand.^50,51 These findings bring us closer to a much-needed understanding of the relationship between the sources of these signals, and face perception serves as a case in which these two well-defined responses can be further related.

For this study, simultaneous EEG and fNIRS recordings were collected in order to compare the spatial localization of hemodynamic activity with band-specific electrocortical activity during perception of faces compared to objects. The area of interest for this study was the right OFA. This region is located posteriorly and laterally relative to the FFA, making activity from this area easily accessible using fNIRS. Source localizations associated with the alpha, beta, and theta bands of the EEG time course were also examined. Our aim is to relate spatial localizations of face processing based on both hemodynamic and electrocortical signals.

2. Materials and Methods

2.1. Participants

Twenty-two participants (mean age $27.6\pm 6.6$, 15 females, and 21 right-handed⁵²) took part in the experiment. The mean laterality quotient from the Edinburgh Handedness Inventory was 68.8. All participants were previously screened using a right-handed finger-thumb tapping localizer task (see below). By self-report, participants had no known neurological or psychiatric conditions and had normal or corrected to normal vision. Participants provided written informed consent in accordance with guidelines approved by the Yale University Human Investigation Committee (HIC #1501015178) and were compensated for their participation. All data were obtained at Yale School of Medicine, New Haven, Connecticut.

2.2. Screening Procedure

It has been well-documented that fNIRS signals are not obtainable from every individual, due to factors such as hair style, bone density, and skull thickness.^53–55 In order to ensure that we were able to obtain valid signals from each subject prior to their participation in the study, each participant underwent a screening using a finger-thumb tapping task. This ensured that any subsequent findings were based on viable signals. For this preliminary task, participants wore 80 optodes (40 emitters and 40 detectors) in the same 134-channel layout described below in Sec. 2.5. The numbers 1, 2, 3, or 4 appeared randomly on a screen and participants were asked to tap the corresponding finger depending on the number, with “1” for index finger, “2” for middle finger, “3” for ring finger, and “4” for pinky finger. The task was a block design, in which participants tapped 20 times in 15 s, followed by 15 s of rest, repeated 6 times for a total of 3 min. Hemodynamic data were analyzed using event-triggered averages over the six blocks, with each 30-s block constituting an event, and the amplitude of the response in each channel was used to generate a $p$ value for that channel. If at least one fNIRS channel in the left motor cortex had a $p$ value less than the threshold of $p=0.001$ for both oxyHb and deoxyHb signals, the participant was invited back at a later date to take part in the faces and object-viewing experiment described below. Based on a data set of 128 participants, we have found that $\sim 8\%$ of subjects are excluded using this procedure.⁵⁶

2.3. Stimuli

Visual stimuli consisted of pictures of faces and objects obtained from standardized image sets. Face stimuli were all forward-facing images from the Face Place stimulus set (courtesy of Tarr, Center for the Neural Basis of Cognition and Department of Psychology, Carnegie Mellon University, funding provided by NSF Award No. 0339122).⁵⁷ Object stimuli consisted of images of clothing, electronics, household items, and food acquired from the Bank of Standardized Stimuli set.⁵⁸

2.4. Paradigm

Each task block consisted of nine pictures of faces or objects, each of which was displayed on a monitor for 2 s (18 s total) (Fig. 1). The size of the presented images was the same for the faces and the objects. In each block, seven of the pictures were of one type, whereas two were of the other type (“oddball trials”). For example, the “face” blocks consisted of seven pictures of faces and two pictures of objects. The position of the two oddball trials in the block was random and was intended to reduce subject perceptions of a uniform block and possible adaptation effects. We have previously used this design in a Stroop task using gesture, in which task blocks included random foils of a different condition.⁵⁹ The 18-s task blocks were interspersed with 15-s of rest during which the participant fixated on a crosshair. Each run consisted of three face blocks and three object blocks, which were randomized throughout the run. Each participant underwent two runs, with a short break between runs when the experimenter came in the room to check that the subject was still comfortable and wished to continue. Participants were instructed to focus on the center of the screen (visual angle of 0 deg). Each picture was presented once over the course of two runs.

Fig. 1.

Experimental paradigm. One run consisted of six blocks. Each block contained two stimuli that were of a different category from the others in order to avoid the perception of a uniform stimulus block. Two runs were presented for a total of 54 faces and 54 objects. Individual face (F) and object (O) events were used for fNIRS and EEG ERP analyses, and blocks were used for frequency band source localization.

The experimental paradigm of blocks consisting of individual picture events was designed to optimize both the EEG and fNIRS analyses. Each face and object presentation was treated as an event in the event-related potential (ERP) EEG analysis and in the general linear model for the fNIRS analysis. Entire blocks were used in the frequency band source localization analyses, with “face blocks” consisting of mostly face pictures and “object blocks” consisting of mostly object pictures. Stimulus presentation, fNIRS signal acquisition, and EEG signal acquisition were synchronized using TTL triggers that were sent to all machines simultaneously.

2.5. Functional Near-Infrared Spectroscopy Acquisition

Functional NIRS and EEG signals were acquired simultaneously using custom-made caps with interspersed optode and electrode holders [Fig. 2(a)]. The distance between optodes was 3 or 2.75 cm depending on the cap used to fit the participant’s head. Caps were placed such that the foremost optode holder was $\sim 1\mathrm{cm}$ above nasion and the backmost optode holder was on or below inion (see previous work for details).⁶⁰ For a given participant, the cap that more closely fit these requirements was used. A Shimadzu LABNIRS system was used to collect fNIRS data at a sampling rate of 123 ms. Forty emitters and forty detectors (80 optodes total) were placed in the cap in a 134-channel layout covering frontal, parietal, temporal, and occipital lobes [Fig. 2(b)]. Each emitter transmitted three wavelengths of light, 780, 805, and 830 nm, and each detector measured the amount of light that was not absorbed. The amount of light absorbed by the blood was converted to concentrations of OxyHb and deOxyHb using a modified Beer–Lambert equation.⁶¹ A Patriot Polhemus digitizer (Polhemus, Colchester, Vermont) was used to record the position of EEG electrodes and fNIRS optodes as well as five anatomical locations (nasion, inion, Cz, left tragus, and right tragus) for each subject. See Supplemental Table S1 for median locations of each channel across the group with anatomical labels.

Fig. 2.

(a) A participant wearing 32 EEG electrodes and 80 fNIRS optodes (obtained with permission). (b) Channel layout for one representative participant. Green dots represent the 134 fNIRS channels and red dots represent the 32 EEG electrodes.

2.6. Electroencephalography Acquisition

A g.USBamp (g.tec medical engineering GmbH, Austria) system with two bioamplifiers and 32 electrodes was used to collect EEG data at a sampling rate of 256 Hz. Electrodes were arranged in a layout similar to the 10-10 system; however, exact positioning was limited by the location of the electrode holders, which were held rigid between the optode holders. Electrodes were placed as closely as possible to the following positions: Fp1, Fp2, AF3, AF4, F7, F3, Fz, F4, F8, PC5, PC1, PC2, PC6, T7, C3, Cz, C4, T8, CP5, CP1, CP2, CP6, P7, P3, Pz, P4, P8, PO3, PO4, O1, Oz, and O2. Conducting gel was applied to each electrode to reduce resistance by ensuring contact between the electrodes and the scalp. As gel was applied, data were visualized using a bandpass filter to allow frequencies between 0.5 and 60 Hz. An ear clip was used as the reference electrode, and the ground electrode was placed on the forehead between AF3 and AF4.

2.7. Data Analysis

2.7.1. fNIRS analysis

The analysis methods used here have been described previously.^59,62 Briefly, digitized optode locations were converted to Montreal Neurological Institute (MNI) coordinates using NIRS-SPM software,⁶³ and channels from each participant were registered to group median channel locations. Both OxyHb and deOxyHb signals were analyzed, though studies have shown that deOxyHb signals, compared to OxyHb signals, show greater similarity with the blood oxygen level-dependent signal of fMRI^40,61,64 and reduced susceptibility to systemic effects.^65–67 Signals were acquired every 0.123 s (8.13 Hz). Wavelet detrending was used to remove baseline drift using the algorithm provided by NIRS-SPM.⁶³ Channels in which the root mean square of the raw signal was more than 10 times greater than the average signal were automatically identified and removed from the analysis. This process resulted in the exclusion of 7 channels total across all channels in all participants, i.e., 7 out of 2948 total channels were removed. A principal component analysis spatial filter was used to remove global components caused by non-neural factors, such as blood pressure, heart rate, and respiration.^67,68 For the within-subject general linear model analysis, face and object neural events were convolved with the canonical hemodynamic response function and modeled to fit the data. Oddball stimuli were included in the relevant condition. For example, in the “face” condition, all face trials were modeled, including those that occurred during a predominantly “object” block. Linear interpolation was used to project the resulting beta values into MNI space based on each subject’s channel locations, and voxels that were >17 mm away from the surface of the brain were excluded. Second-level analysis was done using one-tailed $t$-tests in SPM8.

2.7.2. Event-related potentials

EEG signals were preprocessed using EEGLAB v13.5.4b⁶⁹ in MATLAB 2014a (Mathworks, Natick, Massachusetts), including high-pass filtering (1 Hz) and manual removal of channels, before being converted to a format compatible with SPM8 (Wellcome Trust, London, UK). Two types of channels exhibiting noise characteristic of poor contact with the scalp were rejected based on visual inspection: (1) signals whose amplitude exceeded $\pm 100\mu \mathrm{V}$ and (2) signals that were completely flat with low-frequency drift. With these criteria, an average of $\sim 3$ channels per person was removed and the signals from the surrounding channels were interpolated. Data were then divided into epochs from $-100$ to 700 ms relative to stimulus onset and averaged. As with the fNIRS analysis, oddball stimuli were also included in the average for the relevant condition. Event-triggered averages over the 10 parieto-occipital electrodes (P3, P4, P7, P8, PZ, PO3, PPO4, O1, O2, and Oz) for each participant were used to determine whether or not the data showed the N170 response.^12,70 Subjects were considered to have an N170 response if a negative peak between 100 and 200 ms was observed after the onset of a face and preceded the negative peak for objects.¹² Event-triggered average data from a single parietal electrode in a representative participant with an N170 response to faces is shown in Fig. 3 (inset) in contrast with event-triggered averaged hemodynamic responses from one channel in one representative participant for the face’s condition and for the object’s condition (Fig. 3).

Fig. 3.

Comparison of time courses for the event-triggered average signals for the hemodynamic and EEG responses. The different time scales of the signals are shown using an inset. Representative event-triggered average hemodynamic signals from one participant in one channel (channel 134) for faces (oxyHb, solid red; deoxyHb, solid magenta) and for objects (oxyHb, blue dashed; deoxyHb, cyan dashed). Black dotted line marks end of 18-s task period. Solid vertical line marks the 600 ms time period of the inset EEG signal. Inset: representative event-triggered EEG response from one participant showing N170 response in the Pz electrode for faces (red) with a smaller negative peak for objects (blue) shifted to a longer latency. Dotted lines show the 100- to 200-ms time range during which the response was expected to occur.

2.7.3. Frequency band source localization

Frequency band source localization was done for all 22 participants, whether or not they had shown an N170 ERP response to face stimuli. A “normal” mesh model for boundary element method (BEM) was generated using an MNI template, assuming 8196 current sources in the cortical layer. Co-registration to the template brain was based on the individually digitized coordinates for each of the 32 electrodes. A BEM was used to generate the forward model. The multiple sparse priors (MSP) algorithm in SPM8^71,72 was used in the inversion to identify the time course and spatial localization of the sources of the ERP response, without using source priors or restricting solutions. In the MSP algorithm, an empirical Bayes model is used to reconstruct the sources of the distributed electrical neural activity. This generated an image for each subject for faces and for objects. Group-level analysis was performed with one-tailed $t$-tests on the images generated by the difference between the face and object responses.

The source localization algorithm described above was used on the full time course, separated into task and rest blocks for faces and objects. The sources were localized for induced alpha band (8 to 13 Hz), beta band (13 to 30 Hz), and theta-band (4 to 8 Hz) oscillations during the 18-s task blocks and compared to the 15-s rest blocks. Frequency bands were obtained using wavelet decomposition function in SPM. Each block was labeled by block type based on the majority stimulus for that block, resulting in six events per run. For this analysis, oddball stimuli were not separated; i.e., if a block had mostly face stimuli, it was considered a face block despite having two object stimuli. For each frequency band, the analysis was run twice: once with the time range of the task block (0 to 18 s) and once with the time range of the rest blocks (18 to 33 s). This resulted in four images for each frequency band: the face task blocks (face block), object task blocks (object block), rest blocks following face blocks (face rest), and rest blocks following object blocks (object rest). For each subject, one-tailed $t$-tests were used to generate an image of the face block > face rest and object block > object rest contrasts. Group-level data were created from these images, and the contrast of face > object was generated using one-tailed $t$-tests in SPM8.

2.7.4. Small volume analysis

Localization of activity in the right OFA was confirmed using Neurosynth.⁷³ Neurosynth is an online meta-analysis tool that generates maps of activity based on previously published studies. An association image was generated using the search term “faces” from 864 studies in order to ensure that the image would encompass all face-specific areas. The cluster located in the right hemisphere and thresholded at $p<0.05$ was isolated. This cluster was then divided into two clusters: one in the fusiform gyrus consistent with the FFA [peak voxel at (42, $-50$, $-24$), $n$ of voxels = 1154] and one in the occipital cortex consistent with the OFA [peak voxel at (44, $-76$, $-14$), $n$ of voxels = 564]. A mask generated from the OFA cluster, located in lateral occipital cortex, was used as the volume of interest (VOI). The intersection of this OFA VOI and the whole-brain image of beta values generated from the first-level analysis was created for each subject. These individual subject images were combined using a second-level analysis in SPM, and the group result was rendered at a threshold of $p<0.05$. All images were rendered on a standardized brain using an MNI template.

3. Results

All fNIRS and EEG frequency band source localization results presented are from analysis of all 22 participants.

3.1. fNIRS Results

Event-triggered average plots are shown in Sec. 5, averaged across all participants for the faces > objects contrast, with OxyHb in red and deOxyHb in blue. For rendered images, deoxyHb signals are inverted so that overall hemodynamic activity is presented using the same color scheme in both oxyHb and deoxyHb figures. Yellow-red represents increases in hemodynamic activity, i.e., decreases in deOxyHb and increases in OxyHb concentrations. Using the deoxyHb signal, the small volume analysis revealed hemodynamic activity for the faces > objects contrast [$p<0.05$, Fig. 4(a)], with peak voxel at (32, $-88$, $-16$) ($t=2.60$, $n$ of voxels = 52). According to the atlas,⁷⁴ the label for this peak voxel was Broadmann Area 18, or visual association cortex (V2), consistent with the OFA. Using the oxyHb signal, the small volume analysis resulted in a cluster of activity with peak voxel at (32, $-88$, $-16$) ($t=2.31$, $n$ of voxels = 55) [Fig. 4(b)]. As with the cluster found using the deoxyHb signal, the oxyHb cluster peak voxel was also in V2.

Fig. 4.

Results of small volume analysis of activity localized to the right OFA (posterior view). Red represents increased activity for faces > objects contrast, $p<0.05$. Black circles indicate the right OFA VOI. Increased activity in this area is shown using the (a) deoxyHb activity, (b) oxyHb activity, and (c) theta-band source localization.

3.2. ERP N170 Response

Ten subjects showed a negative peak between 100 and 200 ms after the onset of face stimuli in parieto-occipital electrodes that preceded a negative peak for object images. The event-triggered average for these ten subjects in these electrodes is shown in Fig. 5.

Fig. 5.

Event-triggered average ERP responses in 10 posterior electrodes, averaged across the 10 participants exhibiting the N170 response. Graphs are organized in the rows that the electrodes were positioned on the head (left and right sides are labeled). Black dashed lines mark 100 and 200 ms after stimulus presentation. Red represents faces and blue represents objects. Shaded regions show standard deviation.

3.3. Frequency Band Source Localization

Source localization of the theta-band oscillations for the face > object contrast revealed activity in the right OFA VOI [$p<0.05$, Fig. 4(c)], with peak voxel at (36, $-70$, $-12$) ($t=2.84$, $n$ of voxels = 375). Localization of the alpha and beta band oscillations produced no positive activity in the VOI.

3.4. Neurosynth Mask Overlap

To further confirm that all areas of increased activity were localized to the OFA, we report the overlap between the posterior, OFA cluster obtained from the Neurosynth “faces” map and the face > object contrasts using both fNIRS and EEG. When the overlap of voxels between the group results and the mask was calculated, 41.7% of the deoxyHb activity, 9.96% of the oxyHb activity, and 48.0% of the theta-band source localization overlapped with the Neurosynth “faces” mask. The right-lateralized OFA VOI overlaid with the fNIRS activity (both oxyHb and deoxyHb signals) and theta-band source localization results (all rendered at $p<0.05$) are shown in Fig. 6. Axial slices are shown using an MNI template in MRIcroGL.⁷⁴

Fig. 6.

Axial slices showing face > objects results overlaid on the OFA VOI from the neurosynth “faces” association map (yellow). Red indicates oxyHb activity, blue indicates deoxyHb activity, and cyan indicates theta-band source localization for this contrast at $p<0.05$. Purple reflects overlap of red (oxyHb) and blue (deOxyHb) activity. Green reflects overlap of cyan (theta band) and yellow (VOI). Lighter shades indicate areas of overlap.

4. Discussion

Multimodal imaging approaches to simultaneously acquire fNIRS and EEG recordings have the potential to bridge the theoretical gap between fast neuronal responses and slower hemodynamic changes. Here, we apply the technical advantages of two integrated surface-mounted detectors, optodes for fNIRS signals and electrodes for EEG signals, to simultaneously acquire neural responses to faces across a time scale extending from milliseconds to seconds. In this study, a classical passive-viewing task was used to differentiate responses to faces and objects. EEG signals in the frequency of the theta band showed spatial localization to the right OFA during face presentation, whereas alpha and beta band activity was not localized to this area. Hemodynamic activity measured using fNIRS showed face-specific activity localized to the right OFA, consistent with prior fMRI findings. A subset of participants also demonstrated the classical N170 event-related EEG response to faces. This is the first study to our knowledge to use simultaneous electrocortical (EEG) and hemodynamic (fNIRS) signals to demonstrate the OFA as a common spatial source for hemodynamic activity and theta-band oscillations in face perception. Results are consistent with our hypothesis that theta oscillations localized to the OFA serve as a component of the face perception mechanism.

Prior studies using intracranial electrodes and fMRI have characterized the relationship between neuronal spike responses, local field potentials, and hemodynamic signals.⁷⁵ Other simultaneous EEG/fMRI studies have also served to relate these two types of measures, often by correlating the two signals.^45–48 Importantly, here we do not directly relate the electrocortical and hemodynamic signals. Rather, we use source localization during a task with a known spatial response to show that both theta-band oscillations and hemodynamic responses originate in the same part of the brain during face processing. The OFA response to faces is well-established, suggesting that the similar theta-band oscillation co-localization to this area may be due to a common neuronal source for theta-band oscillations and hemodynamic signals. However, further research is necessary to directly relate to the two signals to confirm whether theta-band oscillations are responsive to faces and whether there is a direct relationship between these oscillations and the hemodynamic responses.

Although our findings show that theta oscillations and hemodynamic activity generated in response to face stimuli were localized to a common source, this was not observed with other electrocortical frequency bands. What could this common source for theta oscillations mean? One possibility is that theta oscillations serve as a signal that is sent out by the neurons in the OFA in preparation to interact with other areas of the brain. This is consistent with the integrated theta system proposed by Başar et al.,²⁹ who suggested that the theta band originates in different parts of the brain in order to form functional connections between them during visual processing. Supporting this framework, a recent study showed that theta-band activity is correlated with the packaging of information about neuronal activity by layer II/III columnar cells, which provides short-range connectivity with nearby areas in the network.⁷⁶ Face processing is a complex task that readies mechanisms for emotion processing, working memory, and communication.^31,59,77 Although our study focused on the right OFA as a region of interest, face perception is known to involve a network of areas, including the FFA and the posterior superior temporal sulcus.¹ This could suggest a bottom-up mechanism through which visual presentation of faces initiates theta oscillations of neurons in the OFA in preparation to form functional connections with other parts of the face processing network. Due to the passive task of the study, we do not measure the relationship between functional connectivity and theta oscillations. Future simultaneous EEG-fNIRS studies using more challenging cognitive tasks have the potential to investigate whether the theta band co-localizes with areas that are functionally connected to the OFA during cognitive tasks involving face perception.

Prior studies have also observed spatial co-localization for increases in theta-band oscillations and increases in hemodynamic activity. One recent study using simultaneous EEG and fMRI found that theta-band activity was negatively correlated with BOLD responses in parts of the default mode network during a memory task.^50,78 As the cognitive demand of the task went up, BOLD responses in the default mode network decreased while the theta response band response localized to this area increased. In another study, increased cognitive demand in multiplication tasks resulted in greater fNIRS activity as well as greater theta-band power in frontal areas.⁵¹ Theta power has also been correlated with increased functional connectivity between parts of the default mode network during resting state,⁷⁹ consistent with the idea that oscillations in the theta-band range may form the basis for functional connections between hemodynamically active areas.

Our findings, showing theta-band source localization to the same area as hemodynamic responses, may have been driven by the emotional and motivational response to faces, rather than lower level features of the face as a salient visual stimulus. In this study, neutral faces were used as the stimuli. It is possible that using angry, happy, or fearful faces would have increased the observed theta-band power or that the theta band may have localized to areas other than the OFA in accordance with prior studies.^34–36 If the emotional features of faces contributed to the observed results, different effects by gender may be expected.⁸⁰ Due to limitations of sample size and the fact that the majority of our participants were female (15 out of 22 participants), we did not conduct an analysis of the effect of gender on the response. However, this is an interesting area for future research.

In our study, theta oscillations localized to the OFA and not the FFA, as shown by the group cluster of activity in Fig. 6. Haxby et al.¹ have proposed that the OFA may serve as an early node in a hierarchical face-processing network and that information is relayed from the OFA to the FFA and the right superior temporal sulcus during visual face processing.⁸ Studies of the OFA using intracranial recordings and MEG source localization both provide support for this theory.^11,81 In this study, using fNIRS and EEG, we record two temporally distinct processes, both driven by face-specific neuronal activity. Theta oscillations arise from the fast neuronal oscillatory activity, so theta localization to the OFA could be due to this area’s early role in the face perception process. It may be that theta-band oscillations originating in the OFA spread to other parts of the face-processing network, in accordance with Haxby’s hierarchical model. Alternatively, it could be that the spatial localization of the theta band in this study was limited due to the use of only 32 electrodes and that the use of more electrodes would have resulted in source localization to the FFA as well as the OFA. We further describe this limitation and interpretation below.

In contrast to the theta band, alpha and beta oscillations were not localized to the OFA during the face-viewing epochs. These bands may be less specific to face processing and might, therefore, not be upregulated during passive viewing. Alpha-band power is increased during rest and suppressed during visual perception and cognitive tasks.⁸² In vision studies, the anticipatory suppression of alpha oscillations is associated with attention and inhibitory control.⁸³ Beta oscillations are associated with sensorimotor activity, and beta power may be driven by the salience of the stimuli.³⁸ Although beta activity has been evoked in occipital electrodes during passive visual tasks, beta power is increased during integration of multimodal stimuli.⁸⁴ Both alpha and beta desynchronization have been observed in response to emotional face stimuli; this is not detected for neutral stimuli, so information contained in these frequency bands may be related to emotion processing rather than neutral face perception.³⁸ An alternative explanation may be that alpha and beta bands are more affected by high-frequency noise in the signal, preventing a localizable signal.

Although fusiform activity during face perception tasks is well-documented using fMRI, our results show that fNIRS can also be used to measure face-related activity from the OFA. Although this region does not overlap the well-described face-specific FFA, it also demonstrates preferential activity for faces.^1,2,7,8 This is the first study to use simultaneous fNIRS and EEG in a face perception task, extending previous investigations of the hemodynamic and electrocortical effects of seeing a human face. The discovery that theta-band activity is co-localized to the same area as hemodynamic signals stimulated by faces suggests that theta-band oscillations may also be involved in face processing. Because the sources are task-specific and common to these temporally distinct processes, we speculate that the neural processes are linked. More generally, our findings generate hypotheses for future research about the possible relationship between theta-band source localization and hemodynamic activity during the processing of a face. We hypothesize that theta oscillations that originate in the OFA are transmitted to other face-specific areas and that this binding mechanism may lead to the recruitment of oxygenated blood to active areas in the face-processing network. The unmet need to understand the relationships between the two signals underlies the recent increased interest in creating viable EEG-fNIRS systems to collect electrocortical and hemodynamic data simultaneously. Future studies using simultaneous EEG-fNIRS are necessary to test the hypothesis that theta oscillations may relate to functional connectivity in the face processing network during cognitive tasks that involve face perception.

4.1. Limitations

The penetration of functional NIRS is limited to the superficial gray matter, which restricts the cortical regions from which signals are acquired. This limits the brain regions sensitive to this technique, and findings of this study are restricted to fNIRS-compatible brain areas. Because of this, the area of interest for this study was the OFA, rather than the FFA.

Other limitations of the study arise from the simultaneous acquisition of both fNIRS and EEG signals. Custom-made caps with alternating electrode and optode holders in a grid pattern were used. Some electrodes were lifted away from the scalp when the optodes were placed. Although gel was used to bridge the space between the electrodes and the scalp, in some areas, this connection was not stable and the signals from some electrodes were unusable. Even electrodes from which data were included may have shown weaker signals than if the electrodes were flush against the scalp. Another limitation arose from the design of the experiment. Trials were presented in a mostly blocked design in order to maximize the opportunity to detect hemodynamic signals using fNIRS. This resulted in only 54 face trials and 54 object trials, whereas typically at least 100 trials are recommended to elicit early perceptual responses.⁸⁵ The absence of interstimulus intervals (ISIs) between events may also have decreased sensitivity to evoked potentials. ISIs were not used in order in the design of the paradigm in order to maximize similarity to prior fMRI experiments. However, in the future, block designs incorporating ISIs may be the most efficient at eliciting both hemodynamic and event-related responses. These limitations, the suboptimal placement of electrodes, the relatively small number of picture presentations, and the constant interval between events, may explain our observation of an N170 response in only a subset of the participants. However, the main result reported here focused on the co-localization of the theta-band oscillations and the hemodynamic activity to the OFA specific to face perception. The lack of an observable N170 response was not considered an exclusion criterion for potential theta-band oscillations, therefore, all participants were included in the source localization analyses.

A final limitation of the study was the use of 32 electrodes for source localization analyses. It has been well-documented that a greater number of electrodes ensure greater spatial specificity of point-like sources.^86–88 One study has reported that the use of 32 or 64 electrodes results in the estimation of fewer sources in regions of interest (fewer true positives) as well as more sources outside the regions of interest (more false positives).⁸⁹ In this study, we were interested specifically in the response to faces in face-selective areas such as the OFA. Because the source localization result was located in an area that we a priori expected to show activity in response to faces, we have more confidence that this activity is not a random false positive. Due to the trade-off between ease of experimental setup and spatial specificity, 32 electrodes have been used successfully for source localization in many studies.^90–92 However, the results of this and other studies that use this relatively small number of electrodes should be interpreted with caution.

5. Appendix A: fNIRS Event-Triggered Average Plots

For visualization purposes, event-triggered average was performed in each channel on both the oxyhemoglobin and deoxyhemoglobin fNIRS data, for the Face-Objects contrast. Plots include activity averaged across all participants. Five channels that were within 3 cm of the peak voxel of the occipital face area volume-of-interest are shown circled in green.

Fig. 7.

Event-triggered average plot of fNIRS signals for face-objects contrast, averaged across all 22 subjects. Red represents OxyHb, blue represents deOxyHb. Front, back, left, and right aspects of the head are labeled. Green circles highlight the subset of channels within 3 cm of the peak voxel of the OFA VOI in the majority of subjects.

Biographies

Swethasri Dravida received her BS degree in mathematics and brain and cognitive sciences from MIT in 2013. She is a graduate student at Yale School of Medicine. Her current research interests include using functional near-infrared spectroscopy and EEG to study social interaction, especially in clinical contexts such as autism.

Yumie Ono received her PhD in electrical engineering from Waseda University in 2004. She is currently a professor at Meiji University in the Department of Electronics and Bioinformatics of School of Science and Technology. She is also the director of the Health Science and Medical Engineering Laboratory at Meiji University. Her research interests include functional brain imaging and development of biological measurement systems for medical applications.

J. Adam Noah received his PhD in biomedical sciences from Marshall University School of Medicine in 2003. He is an associate research scientist at Yale School of Medicine in the Department of Psychiatry and the Brain Function Laboratory. His research interests include functional near-infrared spectroscopy and integration of other multimodal and behavioral recording techniques for applications in communication and social interactions, neurofeedback, and cognitive neuroimaging.

Xian Zhang received his PhD in psychology and visual science from Columbia University, New York, USA, in 2003. He is an associate research scientist at the Brain Function Laboratory in the Department of Psychiatry of Yale School of Medicine. His research interests include computational neuroscience, signal processing, and neuroimaging technologies, such as EEG, fNIRS, and fMRI and their applications in psychiatry, vision science, social interactions, and decision making.

Joy Hirsch received her PhD in psychology and visual science from Columbia University and is now a professor of psychiatry and neurobiology at Yale School of Medicine, and a professor of neuroscience at the University College London. She is also the director of the Brain Function Laboratory at Yale University. Her research is focused on investigations of neural circuitry that underlies human social interactions using multimodal neuroimaging techniques including fNIRS, fMRI, EEG, eye-tracking, and behavioral measures. Prior to recruitment to Yale, she was a director of the fMRI research center at Columbia University.

Disclosures

The authors report no conflicts of interest.