Steady-state visual evoked potentials as a research tool in social affective neuroscience

Abstract

Like many other primates, humans place a high premium on social information transmission and processing. One important aspect of this information concerns the emotional state of other individuals, conveyed by distinct visual cues such as facial expressions, overt actions, or by cues extracted from the situational context. A rich body of theoretical and empirical work has demonstrated that these socio-emotional cues are processed by the human visual system in a prioritized fashion, in the service of optimizing social behavior. Furthermore, socio-emotional perception is highly dependent on situational contexts and previous experience. Here, we review current issues in this area of research and discuss the utility of the steady-state visual evoked potential (ssVEP) technique for addressing key empirical questions. Methodological advantages and caveats are discussed with particular regard to quantifying time-varying competition among multiple perceptual objects, trial-by-trial analysis of visual cortical activation, functional connectivity, and the control of low-level stimulus features. Studies on facial expression and emotional scene processing are summarized, with an emphasis on viewing faces and other social cues in emotional contexts, or when competing with each other. Further, because the ssVEP technique can be readily accommodated to studying the viewing of complex scenes with multiple elements, it enables researchers to advance theoretical models of socio-emotional perception, based on complex, quasi-naturalistic viewing situations.

Our sensory systems allow us to effortlessly interact with complex, changing environments, optimizing behavioral outcomes and minimizing exposure to threat and danger. Experimental work in the cognitive and neural sciences has demonstrated that this interaction of perception and action is crucial for how sparse and noisy visual information is used by the human brain to extract meaningful information (Grossberg, 2013; Summerfield & Egner, 2009). Together, this research highlights the fact that sensory systems are best understood as embodied in living organisms with motives and goals, interacting with the environment in a dynamic fashion, while constantly adapting to changes and maximizing outcomes (Buzsaki, Peyrache, & Kubie, 2015; Schroeder, Wilson, Radman, Scharfman, & Lakatos, 2010). Central motive states, goals, and predictions from memory likewise bias the selection, extraction, and interpretation of visual cues, and enable efficient computations of semantic content to be translated into adaptive behavior. Since humans, like other primates, are social animals, much of the adaptive value of a given visual cue is determined in the context of social interactions, where emotional states and motives of different agents play an important role. Accordingly, a rich body of theoretical and empirical work has addressed how socio-emotional cues are processed by the human visual system. One of the main premises of this line of research has been that the visual analysis of socio-emotional information is highly dependent on situational contexts and previous experience (for recent reviews, see Barrett, Mesquita, & Gendron, 2011; Wieser & Brosch, 2012), but stringent, quantitative tests of this broad hypothesis have been made difficult by methodological issues.

In the present review, we ask how current concepts and methods in human visual neuroscience can be harnessed to address research questions related to the perception of socio-emotional stimuli. It is intuitively plausible as well as empirically well supported that visual percepts do not typically arise de novo, in a space and time void of expectations, but instead are guided by predictions derived from mnemonic and contextual cues (Clark, 2013; Engel, Fries, & Singer, 2001). Innovative electrophysiological techniques are now available that can be used to examine issues of prediction and context-dependency. Here, we focus on the utility of the steady-state visual evoked potential (ssVEP) frequency tagging approach for addressing these issues in human observers. Using this approach, lower-tier visual cortical processes related to one specific element of a complex scene can be “tagged” by flickering or contrast-reversing this element at different rates than the background and other objects (e.g., Appelbaum, Wade, Vildavski, Pettet, & Norcia, 2006; Martens, Trujillo-Barreto, & Gruber, 2011; Miskovic & Keil, 2013a; Verghese, Kim, & Wade, 2012; Wang, Clementz, & Keil, 2007; Wieser & Keil, 2011; Wieser, McTeague, & Keil, 2011). This approach has been fruitful for studying different aspects of visual cognition, such as changes in feature-based or spatial visual selective attention (Andersen, Müller, & Hillyard, 2009; Müller et al., 2006), contrast gain or response gain (Song & Keil, 2013), and target detection under competition (Kim & Verghese, 2012). It has also been increasingly used for studying the interaction between bottom-up saliency and top-down processes such as temporal attention (Wieser & Keil, 2011) or expectancy (Moratti & Keil, 2009). Notably, the competitive or synergistic interactions between overlapping stimuli can be quantified and systematically studied in vivo at the level of cortical population activity using a technique referred to as “ssVEP frequency-tagging”. The present review discusses the potential uses of this method for addressing current research questions in social and affective neuroscience. We first present outstanding questions and challenges in these fields as they pertain to the perception of social cues that convey socio-emotional information.

1. Investigating the neural correlates of socio-emotional processing: Conceptual and methodological challenges

As discussed above, humans continuously gather social information in order to improve the prediction and interpretation of behavior shown by conspecifics, as well as to guide their own behavior. In this process, information on emotional states as communicated through facial expressions is a crucial element (Smith, Cottrell, Gosselin, & Schyns, 2005). While early categorical models postulated that both the facial expressions of emotion and their perception are unique, natural, automatic, and intrinsic phenomena (Ekman, 1992; Smith et al., 2005), more recent models emphasize the dependency of face perception on situational context and the perceiver’s expectations and predictions (Barrett et al., 2011; Wieser & Brosch, 2012). These notions converge with recent trends in vision research, emphasizing that perception in general is highly dependent on contextual cues (see Bar, 2004; Gilbert & Sigman, 2007 for extensive reviews), guided by previous experience and current states of the observer (Schroeder et al., 2010). Despite the fact that “context” may refer to different concepts in different areas of research, researchers have widely agreed that perception of a stimulus is modulated by properties of the surrounding scene it is embedded in (Gilbert, Ito, Kapadia, & Westheimer, 2000) as well as by semantic properties of the situation in which the stimulus appears, including memory of past events (Fuster, 2009). Contextual effects on visual perception have also been defined at the neural level, referring to interactions between higher-order and lower-tier regions in the traditional visual hierarchy (Roelfsema, 2006), or to the neural effects that change a neuron’s response to its receptive field input (Phillips, Clark, & Silverstein, 2015). These effects have been reliably demonstrated for sensory areas low in the traditional visual hierarchy. For example, a number of electrophysiological studies in awake non-human primates have found that the activity of V1 neurons is modulated by contextual information outside their receptive fields (e.g., Lamme, Rodriguez-Rodriguez, & Spekreijse, 1999; Zipser, Lamme, & Schiller, 1996). Many potential neurophysiological mechanisms for mediating the contextual facilitation of visual perception have been proposed, including fundamental neural processes such as divisive normalization (Reynolds & Heeger, 2009) or neural synchrony (Womelsdorf et al., 2007). In terms of large-scale interactions it has been posited that context-based predictions are mediated via top-down feedback across the visual hierarchy, which may resemble the back-propagation of prediction errors (Friston & Kiebel, 2009), or may involve modulatory prediction signals entering the visual areas from neuronal populations in the prefrontal cortex (Bar, 2004). Likewise, the reverse hierarchical visual processing model assumes that sparse visual information is forwarded and contrasted against a model or prediction in higher order visual areas, from where then feedback fine-tunes activity in earlier stages of visual processing (e.g., Hochstein & Ahissar, 2002; Moratti, Méndez-Bértolo, Del-Pozo, & Strange, 2014).

The visual processing of socio-emotional cues is to a large extent characterized by the principles outlined above: namely, natural vision is a massively parallel and iterative process in which bidirectional flow of information within the visual pathways dynamically and rapidly mediates the observer’s predictions and interpretations of motivationally relevant social signals (Yang, Rosenblau, Keifer, & Pelphrey, 2015). Adding to the complexity of social interactions, it is likely that individual social stimuli and their context overlap, either completely or in part. For example, faces or whole bodies are rarely seen de-contextualized but rather, as being embedded within a visual scene (e.g., Wieser and Keil, 2014). This raises a number of issues that have been traditionally difficult to address in the human neuroscience laboratory, by means of behavioral and traditional neuroimaging methods. How are contextual constraints and priors from memory encoded in (back-projected) signals that modulate visual processes lower in the visual hierarchy? What are the spatio-temporal neural dynamics underlying bi-directional communication? How do units at multiple levels (within and across columns, areas, different cortices) interact to generate context-dependent percepts of socio-emotional information? Are these interactions characterized by competition or synergy, i.e. does the processing of socially significant stimuli incur a cost at the expense of processing contextual stimuli and vice versa, or is social processing related to facilitated processing of its context? Ultimately, one of the core challenges preventing progress in addressing these questions is the need to quantify specific visual responses to many discrete cues simultaneously present in our social world while assessing the interactions between those brain signals. As we will demonstrate in the following paragraph, steady-state brain potentials together with frequency tagging may offer a suitable and feasible research avenue to address this category of questions.

2. Steady-state VEPs and frequency tagging as a tool for studying socio-emotional processing

Electrophysiological approaches for accommodating complex real-world stimuli in the laboratory are often hampered by the fact that the neural responses to the numerous elements and features occurring in complex environments overlap in space and time (Keysers & Perrett, 2002). This makes the analysis of interactive dynamics between discrete stimuli, objects, or features in a social scene difficult. Even gold-standard measures of overt attentional orienting such as eye gaze measurements are difficult to interpret when the stimuli of interest fully overlap in space. By contrast, frequency tagging of multiple stimuli using ssVEPs capitalizes on a robust temporal entrainment phenomenon observed in the recording of large-scale neuronal populations. Rhythmic modulation of stimulus contrast or luminance at a constant rate over a period of time evokes oscillatory field responses in the visual cortex at the same frequency as the modulation rate of the stimulus (in addition to a range of non-linear responses at different harmonic and inter-modulation frequencies). The large-scale frequency-following responses are referred to as either flicker-ssVEP (elicited by luminance modulation) or reversal-ssVEP (elicited by contrast reversal). The resulting response is best quantified by analyzing the evoked (trial-averaged) neuro-electrical response in the frequency or the time-frequency-domain. Traditional Fourier-based methods represent the most popular and most prevalent approach for analyzing the frequency characteristics of a signal (see Pivik et al., 1993 for a description of the basic principles). A limitation of classical Fourier approaches is that time information is lost. However, time-frequency (T-F) analysis methods have been developed to address this gap. A tremendous variety of T-F methods are available including spectrograms, complex demodulation, wavelet transforms, the Hilbert transform, and many others. What these varied methods share in common is that they allow researchers to study the changes of the signal spectrum over time, taking into account the power (or amplitude) of the signal at a given frequency as well as changes in its oscillatory phase. Today, open source software packages that include sophisticated ways of calculating and quantifying frequency-domain aspects of electrophysiological time series signals - the fieldtrip toolbox (http://fieldtrip.fcdonders.nl/start), EEGLAB (http://sccn.ucsd.edu/eeglab/), and EMEGS (http://www.emegs.org/) among others - are readily available to researchers worldwide (see Keil et al., 2014 for more details).

Here, we focus on key characteristics of ssVEPs that are important for its use as a research tool in social and affective neuroscience. For an extensive review on the fundamental principles and stimulating paradigms in vision research, the reader is directed to the review by Norcia and colleagues (Norcia, Appelbaum, Ales, Cotterau, & Rossion, 2015). The oscillatory ssVEP is precisely defined in the frequency domain as well as the time- frequency domain. Thus, it can be reliably separated from noise and quantified as the evoked spectral power in a narrow frequency range. The resulting spectrum typically contains a strong signal at the frequency of the driving stimulus as well as peaks at integer multiples of the fundamental (driving) frequency—the higher harmonics. These harmonic responses depend on the duty cycle, the stimulation method, and the complexity of the stimulus array (Kim & Verghese, 2012).

2.1 Generation of ssVEPs

Several studies have demonstrated that the flicker-evoked ssVEP is predominantly generated in the primary visual and to some extent in adjacent, higher order, cortices (Di Russo et al., 2007; Müller, Teder, & Hillyard, 1997; Wieser & Keil, 2011). Interestingly, a number of studies have suggested different neural sources for the fundamental frequency and the higher harmonics, although these findings await replication and further interpretation (Kim, Grabowecky, Paller, & Suzuki, 2011; Kim & Verghese, 2012). Generally, the ssVEP can easily be driven in lower-tier (retinotopic) visual cortices using high-contrast luminance modulation governed by a rapid square-wave (on-off) stimulation. Alternatively, the rapid modulation of specific stimulus dimensions other than luminance or contrast, while holding these lower-level properties constant (or varying them randomly) may evoke ssVEPs in brain areas sensitive to the particular feature or stimulus dimension of interest (Giabbiconi, Jurilj, Gruber, & Vocks, 2016; Rossion & Boremanse, 2011). For example, stimulation techniques have been used that isolate the ssVEP response to face identity generated in higher order visual areas such as the fusiform cortex (Rossion, Prieto, Boremanse, Kuefner, & Van Belle, 2012). These techniques are more extensively discussed in section 2.4.

A related question often discussed in the context of ssVEP frequency tagging is the preferred sensitivity of visual cortical networks to stimulation at certain, so-called “natural” frequencies (Herrmann, 2001). Often referred to as “resonance”, over-proportionally heightened evoked oscillatory responses have been reported for frequencies in the alpha frequency range (e.g., Schurmann, BasarEroglu, & Basar, 1997), and also for frequencies in the gamma range around 30 to 40 Hz (e.g., Lithari, Sanchez-Garcia, Ruhnau, & Weisz, 2016; Rager & Singer, 1998). Regan (1989) reported evidence for three separate ssVEP components, with unique temporal frequency tuning profiles, roughly corresponding to low, medium and high frequencies. Initial studies of frequency tagging reported that these putative resonance frequencies of the visual system may interact with experimental manipulations, leading to different outcomes of ssVEP amplitude changes depending on the tagging frequencies used (Ding, Sperling, & Srinivasan, 2006). More recent work, however, has found no evidence to support that notion (Keitel, Andersen, & Muller, 2010) and instead supported the hypothesis that stable ssVEP amplitude modulation by cognitive or affective manipulations is seen across a wide range of tagging frequencies (Baldauf & Desimone, 2014; Deweese, Muller, & Keil, 2016; Miskovic et al., 2015).

A final issue to be addressed in this brief overview of issues in ssVEP generation (see Norcia et al., 2015, for a comprehensive review) is the question to what extent ssVEPs can be regarded as a linear superposition of transient ERPs, or alternatively represent a non-linear response that possesses properties beyond the linear combination of individual brain responses. Initial studies in the field have argued that observing higher harmonics when using harmonically simple (e.g., sinusoidal) modulation of luminance or contrast is considered evidence of non-linearity used widely in information processing (Regan, 1972). Subsequent work has examined the extent to which ssVEPs can be explained by properties of transient ERPs, and has observed that especially with square wave (on-off) modulation, ssVEPs may be modeled by transient ERP features with satisfactory accuracy (Capilla, Pazo-Alvarez, Darriba, Campo, & Gross, 2011). The interpretation of these findings is not straightforward, because the ERP itself represents a combination of individual trials that vary greatly in latency (phase) and amplitude, limiting its use for unambiguously explaining the generating mechanism underlying ssVEPs. In line with this notion, other studies have reported that the ssVEP amplitude is poorly predicted by single-trial power changes at the driving frequency, but is best predicted by entrainment of the single-trial phase with the driving stimulus (Moratti, Clementz, Gao, Ortiz, & Keil, 2007). Together, the ssVEP is a signal that is readily quantified, often with clearly defined neuroanatomical origins, but similar to ERPs, the precise mechanism leading to its generation is still debated. Systematic studies documenting the sensitivity of the ssVEP to experimental manipulations with and without frequency tagging have however demonstrated its value, particularly for quantifying competition in visual cortical areas. The use of frequency tagging of ssVEPs to quantify population-level competition is discussed next.

2.2 Implementation of frequency tagging: Working with simultaneous and/or overlapping stimuli

Given the frequency-specificity of the ssVEP, "tagging" a discrete visual object or a discrete feature (e.g., a face, a scene element, a Gestalt consisting of multiple stimuli, etc.) is readily feasible by modulating it at a fixed temporal rate, different from the other elements and features in the field of view. In the simplest case, the ssVEP to a stimulus that is flickered or contrast-modulated in front of a simple, ‘empty’ (e.g., black) background allows the separation of the visual response related to the flickered stimulus from ongoing spontaneous brain activity. Considering a slightly more sophisticated design, the content of the static background can be manipulated by the experimenter and the neural response to the superimposed stimulus can then be quantified in the corresponding ssVEP analysis. With appropriate experimental control, this version of frequency tagging allows researchers to quantify the potential interactions (competitive or synergistic) between a foreground stimulus and varying backgrounds with respect to activity in the visual cortices engaged by the tagged stimulus. In one example study (Müller, Andersen, & Keil, 2008), this approach was used to identify competitive cost effects of task-irrelevant background pictures varying in emotional content, while observers performed a detection task that required attending to the tagged stimulus, consisting of a cloud of flickered dots exhibiting random movement (Brownian motion) that fully covered the background image (see Figure 1 A). Fully overlapping arrays are often used in studies where multiple scene elements are tagged with the goal of quantifying time-varying changes in their competitive versus synergistic interactions.

Figure 1.

Examples of stimulus displays used in ssVEP paradigms in social vision research. A) Foreground task (moving dots) superimposed upon color pictures as used in Müller et al. (2008) and Deweese et al. (2014). Flickering the dots and/or the background picture allows the quantification of trade-off between overlapping stimuli over time. Typically, randomly moving dots are the task stimulus with observers detecting episodes of coherent motion or directional change. Attentional capture by social or emotional stimuli can be measures indirectly by a drop in the dot-evoked ssVEP amplitude when socially relevant pictures are presented. B) Overlaid stimuli as used in Wieser & Keil (2011). Task-relevant Gabor patches are overlaid on faces to investigate the influence of face processing on task-evoked visual cortex activity. In this paradigm, Gabor patch-evoked ssVEPs can be separated from face-evoked ssVEPs providing direct measures of competition of overlapping stimuli. By contrast, spatial arrangements of tagged stimuli (C) allow researchers to study the time course and amount of differential engagement with stimuli at different locations. Note that ssVEPs allow a continuous quantification of how visuo-cortical engagement is shared between stimuli at any given moment in time. Such information is not available based on eye tracking, fMRI, or ERP paradigms.

In cases where researchers wish to capture trade-off or synergistic effects between multiple elements in the field of view (e.g., Wieser, Reicherts, Juravle, & Von Leupoldt, 2016), overlapping or adjacent, ssVEP frequency tagging represents the only known technique that – unlike eye-tracking for example- provides continuous information about each stimulus separately, at higher temporal resolution than hemodynamic measures such as BOLD-fMRI or fNIRS. It should be noted however that the temporal resolution of ssVEPs is below the resolution of the original EEG data. This limitation emerges as the result of the Fourier uncertainty principle: precisely measuring the neural response at a given tagging frequency implies a loss of temporal resolution since time is needed to define the exact frequency (see Keil, 2013 for examples and an expanded explanation). Typically, temporal uncertainty of neural event timing in ssVEP amplitude envelopes is in the range of hundreds of milliseconds, depending on the method used. Often however researchers are not primarily interested in the time course, but in the power at each frequency, reflective of the visual cortical engagement evoked by each of multiple competing stimuli. Irrespective of the analysis technique applied, the principle of frequency tagging follows a straightforward approach, as described above. Two or more elements of the visual scene are modulated at different rates and their respective ssVEPs are separated in the frequency domain (e.g., Appelbaum & Norcia, 2009).

In situations where fully overlapping stimuli are needed to address a given research question, transparency or sparseness of one or more of the overlapping arrays can be used to obtain strong tag-specific neural responses while retaining high visibility of the overlapping objects (see Figure 1b). In situations where adjacent stimuli are preferred, for instance to investigate spatial competition, the two (or more) stimuli to be compared should be of equal size, luminance, contrast and so forth to avoid differences in signal strength owing to physical, rather than semantic, properties (see Figure 1c). When investigating competitive spatio-temporal interactions in studies of attention, concerns about the physical proximity of tagged items become highly relevant. Receptive field sizes exhibit marked variation at different stages of visual processing – they are confined to relatively small regions of space in early striate cortex and they progressively increase in size (and complexity) at more anterior stages of the visuocortical stream (Kastner et al., 2001; Kastner & Ungerleider, 2000). Therefore, depending on the particular stimuli being utilized, competitive interactions may either dominate or decay contingent on the distance between tagged items. For example, when examining attentional competition using relatively simple cues, separation above ~5° of visual angle has been demonstrated to abolish within-hemifield suppressive effects in the ssVEP signal (Fuchs, Andersen, Gruber, & Müller, 2008).

Parameters such as the monitor retrace rate, the duration of the stimulus array, and the EEG sampling rate constrain the availability and usefulness of tagging frequencies in a given experiment. For instance, on a monitor with a retrace rate of 60 Hz, frequencies for tagging are available at the inverse of the product of the retrace interval (16.666 ms) with all integer numbers, multiplied by 1000. Sometimes researchers may prefer that the two parts of the active duty cycle in each stimulus train (e.g., on and off times of the stimulus in a flicker-ssVEP) are of equal duration, which further reduces the candidate tagging frequencies. Furthermore, when using multiple frequencies for as many visual objects simultaneously, it is mandatory to make sure that the tagging frequencies do not exhibit harmonic relations (such as when using 6 Hz and 12 Hz to tag two stimuli, for example), because this prevents the independent analysis of the two spectral responses (in the example, the second harmonic of the 6 Hz stimulus will be at the tagging frequency of the other stimulus, i.e. at 12 Hz). It is advisable to consider tagging frequencies that also do not share a common sub-harmonic, while keeping in mind other constraints such as the epoch duration. Epoch duration (the duration of the EEG data segment used for frequency analysis) determines the spectral resolution used to quantify the ssVEP at each frequency. When the epoch is too short in duration, frequency resolution may not suffice for discriminating between the two or more frequencies used for tagging. It is also advisable to use epoch durations that hold integer numbers of cycles for each frequency, resulting in a spectrum with bins at the exact stimulation frequency (for an extensive discussion of these points, see Bach & Meigen, 1999). This serves to minimize distortions related to so-called "spectral leaking" or smearing of oscillatory responses across two or more bins of the spectrum that are equally distant from the actual frequency. Such leaking may lead to misinterpretation of competition effects, as well as condition differences, especially when the mapping of tagging frequencies to stimuli and experimental condition is not counter balanced across the experiment. The interested reader is directed to reviews and guidelines regarding the technical aspects of ssVEP procedures (Bach & Meigen, 1999; Keil et al., 2014; Norcia et al., 2015; Vialatte, Maurice, Dauwels, & Cichocki, 2010).

Following trial averaging, the ssVEP spectral power at the tagging frequencies can then be used to independently capture neural population activity evoked by each stimulus (see Figure 2). In factorial research designs, changing competitive or synergistic dynamics between tagged objects are reflected in interactions between the experimental condition and tagged object. Linear indices of competition are likewise easily computed and used as dependent variables. It is also recommended to fully counter-balance the tagging frequencies across experimental conditions to avoid confounding signal strength with experimental condition (energy dissipation for higher frequencies is inevitable due to the 1/f shape of the spectrum of brain electric activity).

Figure 2.

Grand mean steady-state visual evoked potential (ssVEP) from OZ sensor averaged from 21 subjects in a faces superimposed on visual scenes paradigm similar to Wieser & Keil, 2014 (unpublished data). The depicted ssVEP contains a superposition of two driving frequencies (12 and 15 Hz), as shown by the frequency domain representation of the same signal (Fast Fourier Transformation of the ssVEP in a time segment between 200 and 3000 msec) in the upper panel. The mean scalp topographies of both driving frequencies show clear activity peaks over visual cortical areas.

Note: The regularity observed here is the result of additively combining two sinusoidal time series of different frequency. This pooled signal is characterized by a slower oscillation of its overall envelope at the frequency of the difference between the two original sinusoids. In this example, an envelope cycle at 15–12 = 3 Hz is visible.

2.3 Trial-by-trial analysis of visual cortical activation

Due to their excellent signal-to-noise ratio, ssVEPs also offer the opportunity to track the dynamics of visuocortical changes at the level of single trials. Trial-by-trial estimates can be calculated by means of a moving window analysis (Keil et al., 2008). Here, a moving average window containing a fixed number of cycles of the ssVEP (e.g., four) is shifted in steps of one ssVEP cycle across the trial epoch. The resulting ssVEP averages can be transformed into the frequency domain by discrete Fourier transformation, and normalized by the length of the segment (Figure 3). It should be noted that this approach (which is analogous to linear deconvolution) quantifies only the part of the electrophysiological response that is stable across the trial, and precisely time and phase-locked to the driving stimulus throughout its duration. Oscillatory events related to the driving stimulus that vary in phase, or that systematically alter the phase across the viewing epoch will lead to a distortion or alteration of the phase and amplitude metrics obtained from moving window analysis. This same principle underlies signal averaging of ERPs, and represents a special case of cross-correlation in which the signal is correlated with an impulse function representing stimulus onsets (Regan, 1972).

Figure 3.

A simple approach for ssVEP estimation across a single trial. Indices of power and phase-locking can be extracted from single trials of ssVEP time series (Keil et al., 2008), by applying a simple moving average technique. As shown in the schematic, a window of fixed length k, sample points, containing an integer number of ssVEP cycles (4 cycles at 10 Hz in the present example), is shifted along a single trial ssVEP signal in n steps, spaced at intervals corresponding to the duration of one ssVEP cycle (here: 100 ms, corresponding the 10 Hz ssVEP response). Equation 1. Point-wise averaging the contents of the n windows (w) yields a time-domain representation of the part of the signal that is locked to the ssVEP driving stimulus. Equation 2. Determining the complex phase φ at the ssVEP frequency (f_stim) for each window (w) by means of discrete Fourier transform or a similar method allows phase averaging: Complex Fourier components at f_stim for each w are unit normalized by dividing by the absolute value, and then averaged across all n windows. The modulus of the resulting complex number represents an intuitive measure of the stability of the phase at the ssVEP frequency across the duration of the trial. Note that application of the method requires that integer numbers of EEG sample points are available for the window length k, as well as for the step size between windows t₁(w+1)− t₁(w), achieved by following the recommendations by Bach and Meigen (1999). It should be noted that the resulting measure reflects the regular and stable alignment of the oscillatory waveform to the driving stimulus over time, but not the phase-consistency within the ssVEP signal.

These single-trial derived measures can then be examined across experimental conditions, for example in order to capture trial-by-trial changes in cortical facilitation that result within an experimental learning block (Wieser, Miskovic, Rausch, & Keil, 2014b), and provide reliable estimates of short-term experience-dependent changes in visual cortex. This allows for specifically following up the time course of aversive or appetitive learning, could inform us about the time course of extinction of previously learned associations in visual cortex, and may be therefore especially suited for clinical applications such as exposure therapy.

2.4 Control over the stimulus dimension of interest

An important research question often includes exploring the putative role of physical stimulus parameters for efficiently and accurately perceiving social cues. Specifically, these questions may pertain to the role of spatial frequency, color, brightness, contrast, and so on for conveying relevant aspects of social information. While these questions are relevant eo ipso, many physical dimensions engage specific neurophysiological sub-systems, thus providing researchers with potential avenues for functionally dissecting the vast array of parallel processes involved in social processing (see the following section). For instance, flicker-ssVEPs may be used to examine how the content of a given social stimulus affects luminance processing. Furthermore, reversing or modulating aspects of facial contrast, in specific regions of the face provides an index of visual processing of that particular aspect or feature, which can then be examined using factorial experimental designs and appropriate behavioral tasks. Recently, a similar approach termed Fast Periodic Visual Stimulation has been applied to studies of face processing (e.g., Ales, Farzin, Rossion, & Norcia, 2012; Alonso-Prieto, Van Belle, Liu-Shuang, Norcia, & Rossion, 2013; Boremanse, Norcia, & Rossion, 2014; Liu-Shuang, Norcia, & Rossion, 2014). This approach uses spectral analyses of ssVEP signals evoked by a rapid presentation stream that periodically and intermittently includes a stimulus dimension of interest. The brain response to this recurring event or stimulus can then be quantified in the spectral domain, at the frequency in which it is embedded in the overall stream. The Fast Periodic Visual Stimulation approach is another productive approach to manipulating a specific stimulus dimension at a certain temporal rate and using the spectral representation of the signal to quantify the feature-specific response. For example, this has been used to investigate face identity discrimination by alternating different face identities after adaptation to one facial identity was induced through repeated presentation of that identity (Retter & Rossion, 2016). With regard to research on socio-emotional processing this may help to investigate the importance of specific features such as the eye region in faces for emotional attention.

2.5 Functional connectivity of cortical regions involved in social vision

Since ssVEP designs provide full experimental control over the frequency of neuronal stimulation and possess high signal-to-noise ratios, they provide some advantages for guiding estimates of event-related functional connectivity. Algorithms for quantifying electrophysiological functional connectivity assume stationarity of the underlying oscillators. Empirical EEG data, however, often exhibit dynamic shifts in frequency structure as a function of task demands or endogenous brain activity and this non-stationarity can lead to distortions in functional connectivity estimates (Cohen, 2015). Since the ssVEP technique implies an external, periodic force applied to the visual system, it may represent one way of stabilizing oscillator frequency and minimizing distortions arising from non-stationarity.

A recent study has examined the internal reliability of ssVEP based estimates of functional connectivity and demonstrated good split-half reliability when averaging as few as 10 artifact-free ssVEP trials (Miskovic & Keil, 2015). Recent work in aversive conditioning used ssVEP-based functional connectivity to characterize the long-range neural synchrony between reconstructed sources in the visual and higher-order pre-motor/motor and frontal cortices specifically during the perception of “avoidable” threats (Khodam Hazrati, Miskovic, Príncipe, & Keil, 2015; McTeague, Gruss, & Keil, 2015; Miskovic & Keil, 2014). For social neuroscience research, metrics of functional connectivity may help to further elucidate the extra-visual pathways involved in social perception and their modulation by different types of experimental contexts. For example, these analyses might clarify how early visual areas and higher-order visual areas in the ventral stream (e.g., fusiform gyrus) communicate in social perception through long-range neural synchrony. Specifically, connectivity metrics based on sustained engagement of visual cortices allow researchers to test hypotheses on re-entrant bias signals, thought to modulate visual cortical activity based on information computed in higher-order cortices or deep brain structures. In addition, the combination of single-trial estimates as mentioned above and functional connectivity analyses may further strengthen our knowledge about the temporal dynamics of cortical cross-talk (e.g., Keil et al., 2008; Lachaux et al., 2002).

Despite their obvious benefits, ssVEP guided estimates of functional connectivity are not without limitations that need to be considered, both in terms of how they impact the interpretation of such metrics as well as the underlying methodological soundness of the analyses. One caveat is that periodic driving necessarily implies a common phase-reset mechanism, driven by the exogenous pacemaker-like visual stimulus that could inflate estimates of cross-regional synchrony (i.e., the third variable problem). However, it is possible in principle to control for these effects using partial estimates of neuronal coherence (Nunez & Srinivasan, 2006), using phase-lagged indices of functional connectivity, or examining induced functional connectivity after subtracting out the contribution of exogenously driven phase locking. Similarly, volume conduction poses considerable challenges for any measure of cross-regional synchronization that is derived from surface-recorded electro-cortical signals. A number of solutions have been proposed in the field as ways of dealing with or minimizing the contribution of volume conduction, including the application of spatial filters and/or source localization estimates as a pre-requisite to calculating functional connectivity as well as the use of algorithms that are sensitive to linear signal mixing. Moreover, volume conduction can reasonably be expected to be constant across experimental conditions, so that a balanced and well-controlled experimental design can effectively address many of the issues concerning shared versus condition-specific connectivity patterns. Another consideration in the application and interpretation of ssVEP based estimates of functional connectivity concerns the extent to which the rhythmic mode of stimulus delivery places strong constraints on signal propagation and the nature of cortical information transfer. As mentioned earlier, the ssVEP is believed to originate primarily from striate and early extra-striate regions, so that using it to study re-entrant signal flow from more anterior cortical regions might be problematic. There are several possible theories, each with some level of empirical support, concerning whether the ssVEP is strictly a local phenomenon and how it might propagate to more broadly distributed sources (Vialatte et al., 2010). The input frequency of the visual stimulus likely plays a significant role in shaping the extent of information transfer from visual to non-visual regions (Srinivasan, Fornari, Knyazeva, Meuli, & Maeder, 2007; Thorpe, Nunez, & Srinivasan, 2007). Taken together with evidence that the preferred oscillatory frequency varies across large-scale cortical regions (Rosanova et al., 2009), ssVEP paradigms may pose unique challenges for tracking cross-regional information flow. In addition to these frequency considerations, the spatial configuration of the visual stimuli (e.g., discrete checkerboard stimuli that activate small patches of visual cortex versus full-field flicker) also strongly influences the phase gradients of ssVEP signals that indicate whether the ssVEPs appear as traveling or standing waves (Burkitt, Silberstein, Cadusch, & Wood, 2000). Given these caveats, there is evidence that ssVEPs are not strictly local events and that they do engage broadly distributed functional networks under some contexts (Srinivasan et al., 2007), increasing the coupling between the visual cortex and non-visual structures (Lithari et al., 2016). Thus, the scalp recorded, driven response does not inform researchers regarding the micro- and mesoscopic neural interactions that are well established through animal model research and invasive recordings in humans.

3. ssVEP studies of socio-emotional cue perception

Although the use of ssVEPs in studies of social perception is relatively new, there is a larger literature using this methodology to probe the processing of more complex affective scenes. Across several studies conducted in multiple laboratories, researchers have reported that the flicker ssVEP is amplified when viewing emotionally engaging, compared to neutral, scenes. This effect has been described as particularly robust when using scenes containing threat or mutilation cues. Presenting pleasant, neutral, and unpleasant images from the IAPS at a rate of 10 Hz each for several seconds, Keil and collaborators (Keil et al., 2003) found higher ssVEP amplitude and accelerated phase (i.e. latency of the oscillatory cycle) for arousing, compared to calm pictures. These differences were most pronounced at central posterior as well as right parieto-temporal recording sites. The magneto-cortical counterpart of the ssVEP, the steady-state visual evoked magnetic field also varied with the emotional arousal ascribed to the picture content by the observers (Moratti, Keil, & Stolarova, 2004). In this latter study, source estimation procedures pointed to involvement of fronto-parietal attention networks in evoking and directing attentional resources towards the relevant stimuli. Generally, ssVEP-based paradigms are often characterized by the collection of robust data in short amounts of time, which highlights their appeal for translational and clinical research (e.g., Moratti, Rubio, Campo, Keil, & Ortiz, 2008; Moratti, Strange, & Rubio, 2015).

A research question that has prompted considerable debate is the extent to which endogenously cued shifts of attention and affective arousal modulate sensory processing in an additive or interactive manner. Crossing the factors of selective spatial attention and affective content, Keil and colleagues investigated the ssVEP amplitude and phase as a function of endogenous attention-emotion-interactions (Keil, Moratti, Sabatinelli, Bradley, & Lang, 2005). Participants silently counted random-dot targets embedded in a 10-Hz flicker of colored pictures presented to both hemifields. An increase of ssVEP amplitude was observed as an additive function of spatial attention and emotional content. Mapping the statistical parameters of this difference on the scalp surface, the authors found occipito-temporal and parietal activation contralateral to the attended visual hemifield. Differences were most pronounced during selection of the left visual hemifield, at right temporal electrodes. Results suggest that affective stimulus properties modulate the spatio-temporal process of information propagation along the ventral stream, being associated with amplitude amplification and timing changes in posterior and fronto-temporal cortex. Importantly, even non-attended affective pictures showed electrocortical facilitation throughout these sites, providing evidence in favor of the additive hypothesis.

3.1 Competition between natural scenes and task stimuli

To investigate competition of affective scenes for attentional resources in visual cortex, Müller et al. (Müller et al., 2008) recorded ssVEPs to continuously presented randomly moving dots, flickering at 7.5 Hz. These dots provided a challenging attention task, as they were continuously in motion throughout the trial, and each dot changed its position in a random direction at every frame refresh. Occasionally, the dots produced a 35% coherent motion (targets), lasting for only 2 successive cycles of 7.5 Hz (i.e., 267 ms), which the participants were instructed to detect and indicate. In two studies employing this paradigm, a reduction in the overall ssVEP amplitude evoked by flickering dots when the dots were superimposed on emotionally arousing, compared to neutral distractor pictures was found corroborating behavioral results with significant decreases in target detection rates when emotional compared to neutral pictures were concurrently presented in the background. This decline in task performance and electrocortical activity persisted for several hundred milliseconds (Hindi Attar, Andersen, & Müller, 2010; Müller et al., 2008). Such affective competition was even observed when the emotional pictures were presented for only 200 ms, which possibly provides evidence for a slow reentrant neural competition mechanism for emotional distractors, which continues after the offset of the emotional stimulus (Müller, Andersen, & Attar, 2011). In a further study, the attentional load of the task was manipulated such that participants either performed a detection (low load) or discrimination (high load) task at a centrally presented symbol stream that flickered at 8.6 Hz. Although behavioral data confirmed successful load manipulation with reduced hit rates and prolonged reaction times in high load compared to low load trials, ssVEP amplitudes elicited by the symbol stream did not differ with load, demonstrating that affective competition was not attenuated by attentional load (Hindi Attar & Müller, 2012). In a recent study with snake-fearful individuals, McGinnis et al. (2014) showed that this competition effect is significantly larger in high-fear participants, where snake distractors elicited a sustained attenuation of task evoked ssVEP amplitude, greater than the attenuation prompted by other unpleasant arousing content, indicating sustained hypervigilance to fear cues. Taken together, these studies demonstrate competition between affective and non-affective scene elements, and importantly, the feasibility of using the frequency-tagging method as a direct measure of sustained attentional resource allocation in early visual cortex.

3.2 Faces and facial expressions

In contrast to transient ERPs, ssVEPs elicited by face stimuli allow for a more fine grained examination of individual face discrimination, perceptual face detection (Rossion, 2014), sustained attention and facial expression discrimination (McTeague, Shumen, Wieser, Lang, & Keil, 2011), as well as the interaction of facial expressions and individual face identity (Gerlicher, van Loon, Scholte, Lamme, & van der Leij, 2013).

Recent studies have suggested that oscillatory stimulation of visual cortex with faces captures high-level visual processes, and provides robust indices of individual face discrimination without requiring any face-related behavioral task (Alonso-Prieto et al., 2013; Liu-Shuang et al., 2014; Rossion & Boremanse, 2011). Moreover, using ssVEPs, individual perception thresholds for face detection can be quantified. To accomplish this, Ales et al. (2012) presented phase-scrambled face stimuli alternating with a fully scrambled image at a rate of 3/s and measured the corresponding ssVEP using a novel sweep-based method, in which a stimulus is presented at a specific temporal frequency while parametrically varying (‘sweeping’) the detectability of the stimulus. They observed an abrupt response at the first harmonic (3 Hz) between 30% and 35% phase coherence for the face. This provides an objective perceptual face detection threshold, to be estimated in single participants from as few as 15 trials (Ales et al., 2012). Employing the same stimulation paradigm, it was recently demonstrated that inversion and contrast-reversal increase the threshold and modulate the supra-threshold response function of face detection (Liu-Shuang, Ales, Rossion, & Norcia, 2015). In a different stimulation paradigm (luminance-flicker), it was investigated if inversion-related ssVEP modulation depends on the stimulation rate at which upright and inverted faces were flickered (Gruss, Wieser, Schweinberger, & Keil, 2012). Results demonstrated that amplitude enhancement of the ssVEP for inverted faces was found solely at higher stimulation frequencies (15 and 20 Hz). By contrast, lower frequency ssVEPs did not show this inversion effect. These findings suggest that stimulation frequency affects the sensitivity of ssVEPs to face inversion. In addition, the excellent SNR and the ability of the visual cortex to remain in a state of entrainment for minutes makes this technology an excellent research tool for investigating adaptation in populations of visual neurons. In this vein, Gerlicher et al. (2013) investigated how facial expressions interact with facial identity by presenting the same face identity or different face identities flickering with a frequency of 3 Hz. The amplitude of the steady state visual evoked potential (ssVEP) was decreased for the same face identity as compared with different identities as in previous research (Rossion & Boremanse, 2011). Most interestingly however, this adaptation was attenuated in a linear fashion when faces contained happy, mixed or fearful expressions with the least degree of adaptation to fearful expressions. These results indicate that threat associated with fearful faces seems to prevent a reduction of driven neural activity over time, even if the actual face identity remains identical.

Facial expression processing was investigated in a ssVEP study that recorded responses to facial expressions that differed in valence (i.e., anger, fear, happy, neutral) in a group of individuals with low and high levels of social anxiety (McTeague et al., 2011). The authors found ssVEP amplitude enhancement for emotional (angry, fearful, happy) relative to neutral expressions only in high socially anxious individuals, and that this was maintained throughout the entire 3,500-ms viewing epoch. These data suggest that a temporally sustained, heightened visuocortical response bias toward affective facial cues is associated with generalized social anxiety. Interestingly, no affective modulation of face-evoked ssVEPs was evident in non-anxious individuals.

As previously established, ssVEPs also offer an excellent research tool to track the dynamics of visuocortical changes due to learning experiences (Keil, Miskovic, Gray, & Martinovic, 2013; Miskovic & Keil, 2013a; Miskovic & Keil, 2013b; Miskovic & Keil, 2014). In three recent studies, the time course and short-term cortical plasticity in response to affective learning of faces was investigated in classical conditioning paradigms (Ahrens, Mühlberger, Pauli, & Wieser, 2015; Wieser, Flaisch, & Pauli, 2014a; Wieser et al., 2014b). In one study, enhanced ssVEP amplitudes were observed for faces that were repeatedly paired with offensive nonverbal hand gestures (Wieser et al., 2014a). These results suggest that cortical engagement in response to faces aversively conditioned with nonverbal gestures is facilitated in order to establish persistent vigilance for social threat-related cues. This form of social conditioning involves the establishment of a predictive relationship between social stimuli and motivationally relevant outcomes. The same effect of cortical facilitation was found when faces were repeatedly paired with verbal insults, but interestingly visuocortical discrimination between negatively and positively conditioned faces was impaired in individuals with heightened levels of social anxiety (Ahrens et al., 2015). In another study it was demonstrated that direct-gaze face cues paired with aversive noise compared to averted-gaze face cues elicited larger ssVEP amplitudes during conditioning, whereas this differentiation was not observed when averted-gaze faces were paired with the aversive US (Wieser et al., 2014b). Importantly, a more fine-grained trial-by-trial analysis of visual cortical activation across the learning phase revealed a delayed build-up of cortical amplification for the averted-gaze face cues. This suggests that the temporal dynamics of visual cortex engagement with aversively conditioned faces vary as a function of the cue with gaze direction as an important modulator of the speed of the acquisition of the conditioned response.

Taken together, face-evoked ssVEPs provide insights into how the visual brain discriminates facial identities and expressions, how these processes are related to neural adaptation, and how configural processing of face entities may also take place in earlier visual cortices. Moreover, short-term functional response plasticity of the visual system due to social learning was demonstrated.

3.3 Competition between faces and/or task items

In addition to the aforementioned studies which predominantly tagged one visual stimulus at a time, ssVEP methodology allows for frequency-tagging as mentioned above, which provides a continuous measure of the visual resource allocation to a specific stimulus amid competing cues. In this vein, Wieser, McTeague, and Keil (2011) employed ssVEP methodology while participants viewed two facial expressions presented simultaneously to the visual hemifields at different tagging frequencies (14 vs. 17.5 Hz). Regardless of whether happy or neutral facial expressions served as the competing stimulus in the opposite hemifield, high socially anxious participants selectively attended to angry facial expressions during the entire viewing period. Most interestingly, this heightened attentional selection for angry faces among socially anxious individuals did not coincide with competition (cost) effects for the co-occurring happy or neutral expressions. Rather, perceptual resource availability appeared augmented within the socially anxious group in response to a cue connoting interpersonal threat. In a further study, competition between a change detection foreground task and task-irrelevant background face stimuli was investigated (Wieser, McTeague, & Keil, 2012). Task-irrelevant facial expressions (angry, neutral, happy) were spatially overlaid with a task-relevant Gabor patch stream, which required a response to rare phase reversals. Face and task stimuli flickered at different frequencies (15 vs. 20 Hz). A competition effect of threatening faces was observed solely among individuals high in social anxiety such that ssVEP amplitudes were enhanced at the tagging frequency of angry distractor faces, whereas at the same time the ssVEP evoked by the task-relevant Gabor grating was reliably diminished compared with conditions displaying neutral or happy distractor faces. Thus, threatening faces appear to capture and hold low-level perceptual resources in viewers symptomatic for social anxiety at the cost of a concurrent primary task.

Frequency-tagging not only serves as an avenue for investigating competition among a multitude of facial stimuli, but it also provides a feasible paradigm to investigate how the perception of faces is influenced by concurrent contextual scenes, and vice versa, how perception of faces changes our perception of its surroundings. To this end, Wieser and Keil (2014) investigated the mutual effects of facial expressions (fearful, neutral, happy) and affective visual context (pleasant, neutral, threat) by assigning two different flicker frequencies (12 vs. 15 Hz) to the face and the visual context scene. While processing of the centrally presented faces was unaffected, viewing fearful facial expressions amplified the ssVEP in response to threatening compared to other background contexts. These findings suggest that viewing a fearful face prompts heightened vigilance for potential threat in the surrounding visual periphery.

Frequency-tagging may also inform research into the fundamental brain processes mediating face perception, such as research that examines part-based and integrated, i.e. holistic/configural encoding of facial features. Boremanse & Rossion (2014) flickered left and right halves of faces to isolate the repetition suppression effects for each part of a whole-face stimulus, changing or not changing face identity at every stimulation cycle. Frequency tagging the left and right halves of a face led to part-based repetition suppression at medial occipital electrodes, whereas integrated effects were found over the right occipito-temporal cortex. These observations provide objective evidence for a dissociation between part-based and integrated (i.e. holistic/configural) responses to faces in the human brain, showing not only that a whole face is different from the sum of its parts, but suggesting that only integrated responses may reflect high-level, face-specific representations.

4. Outlook, future directions, and caveats

This article reviewed recent work exploiting the unique properties of ssVEP methodology together with frequency tagging to gain further insights into the visual processing of social information. This technique has already been used within social cognition research to explore competition for attentional resources when multiple stimuli are present in the visual field simultaneously as well as to track trial-by-trial changes in visuocortical activity as a resulting of learning.

As stated throughout the paper, several limitations of ssVEP based measures need to be considered in addition to the unique strengths offered by this method. For example, for the investigation of specific temporal dynamics in the millisecond range, ERP methodology tends to be preferable over ssVEP paradigms (e.g., Rossi & Pourtois, 2014). Another issue, subject to ongoing research in our own laboratories, is the influence of changes in pupil size on the ssVEP response. While there are no studies available directly investigating this topic, it has been recently demonstrated that pupil size can influence the initial feedforward response in human V1 as reflected by the C1 component of the EEG (Bombeke, Duthoo, Mueller, Hopf, & Boehler, 2016). Last but not least, researchers who investigate competition by presenting multiple stimulus displays under fixation instructions (and with appropriate control of fixation compliance) should be aware that in these experiments, only shifts of covert attention are measured. To investigate covert as well as overt shifts of attention, carefully designed studies using a combination of eye-tracking and ssVEP technology seem warranted. Moreover, eye-tracking would allow for the investigation of fixation-locked ssVEP responses.

Open research questions that can be profitably addressed in the future using this methodology include contextual modulations of socio-affective perception and vice versa. In this vein, a first study has used faces overlaid on visual scenes to investigate interactions of facial expressions and affective visual context (Wieser & Keil, 2014). In the future it will be of importance to use natural scenes to which the face actually belongs (contrary to being simply overlaid as in the latter study), to improve the ecological validity of scene-face interactions. Another relevant research question is how several features of faces interact and improve the perception of relevant social cues such as the expressions of the face. For instance, the separate analysis of visuocortical processing of the eyes and other regions (mouth, etc.) may help to clarify their respective relevance for social cognition. Recent studies using eye-tracking and fMRI point towards the relevance of the eye region for the perception of fearful faces as shown by reflexive orienting towards the eye region, which seems to be mediated by amygdala activity (Gamer & Buchel, 2009; Gamer, Schmitz, Tittgemeyer, & Schilbach, 2013). Frequency tagging approaches may contribute to clarifying how covert attentional mechanisms support the recognition of facial expressions by highlighting specific facial features. Also, single-trial analysis of electrocortical short-term response plasticity to investigate the development of social processing and impression formation, for instance in aversive as well as appetitive learning paradigms, represents another fruitful research avenue. This may not only help us to understand how individual differences in person perception develop, but also how person perception and social learning mechanisms are disrupted in psychiatric disorders (e.g., Ahrens et al., 2015). Other research directions involve exploring the competition of socio-affective stimuli, for instance competition of several faces (crowded attention) or competition of task-relevant and irrelevant social information. Finally, it is worth noting that steady-state entrainment can be elicited in other sensory modalities (e.g., auditory, somatosensory), creating opportunities for investigating multimodal stimulation and inter-modal competition of social information. For example, attentional prioritization of social information may be investigated when social cues are presented simultaneously via visual and somatosensory routes (e.g., faces and touch, faces and pain). It will be of interest to determine whether social information provided in one modality boosts the processing of social information in another modality or whether information presented in one modality can exert interfering effects on processing occurring in a second modality.