Use of eye tracking improves the detection of evoked responses to complex visual stimuli during EEG in infants

Highlights

• •Infant’s EEG responses to visual stimuli (ERVS) may elucidate higher level cortical processing.
• •The ERVS paradigm was enhanced by eye tracker integration and analysis protocol optimization.
• •This increased the reliability and usability of the method, making it applicable for clinical use.

Abstract

Objective

To improve the reliability of detecting EEG responses evoked by complex visual stimuli to the level required for clinical use by integrating an eye tracker to the EEG setup and optimizing the analysis protocol.

Methods

Infants were presented with continuous orientation reversal (OR), global form (GF), and global motion (GM) stimuli. Eye tracking was used to control stimulus presentation and exclude epochs with disoriented gaze. The spectral responses were estimated from 13 postcentral EEG channels using a circular variant of Hotelling’s T² test statistic.

Results

Among 39 healthy infants, statistically significant (p < 0.01) responses to OR/GF/GM stimuli were found from 92%/100%/95% recordings, respectively. The specificity test of the detection algorithm, using non-stimulated baseline EEG, did not yield any false-positive findings. Taken together, this yields 15% improvement on average in the detection performance compared to that in the current literature.

Conclusions

Changes to the test protocol and incorporation of the eye tracking information improves the detection of responses to complex visual stimuli in infants.

Significance

This work presents a test protocol suitable for use in a clinical environment at a level of reliability that allows individual diagnostics.

1. Introduction

Progress in neonatal care has improved survival and reduced the incidence of major neurological sequalae (Saigal and Doyle, 2008, Sellier et al., 2010); however, the mild effects of early adversities on later neurocognitive development remain a major challenge in developmental neurology (Johnson and Marlow, 2016). Early detection of developmental deviances is crucial for guiding individualized treatment and rehabilitation. Several standardized neurological assessment scales and neuroimaging and neurophysiological methods are used by clinicians for the early detection of compromised function (Merchant and Azzopardi, 2015).

Neurocognitive performance and complex visual processing in childhood are commonly affected by neonataladversities (Aarnoudse-Moens et al., 2009, Braddick and Atkinson, 2011, Johnson et al., 2016), and several lines of evidence suggest that assessment of the visual system will provide a proxy for assessing wider neurocognitive development or its risks. Detection of the change in orientation requires cortical neuronal activity (Braddick and Atkinson, 2011), and therefore, it can be used for studying the integrity of the visual pathway from the retina to the primary visual cortex, which is important in more severe neonatal adversities. Aspects of more complex visual processing may offer proxy to assess readiness for visual processing-related and reliant neurocognitive functions across multiple cortical areas beyond the primary visual cortex, e.g., the clustering of cortical areas referred to as dorsal and ventral streams (Braddick et al., 2000, Braddick et al., 2003, Braddick and Atkinson, 2011, Milner, 2012).

Dorsal stream processing, including temporoparietal areas, is a prerequisite for global motion (GM) perception and closely linked to visuomotor and visuospatial skills and attentional processes (Braddick et al., 2000, Braddick et al., 2003, Wattam-Bell et al., 2010, Milner, 2012). In addition to developmental disorders and dyslexia (Grinter et al., 2010, Braddick and Atkinson, 2011, Milner, 2012, Robertson et al., 2014), deficits in GM perception have been recently associated with mathematical learning difficulties in childhood (Braddick et al., 2016). In contrast, global form (GF) processing is believed to, at least partly, rely on distinct ventral stream functioning (Braddick et al., 2000) required for object recognition, including faces (Braddick and Atkinson, 2011). It has also been suggested that ventral stream processing activates brain mechanisms for perceptual processes that serve planning, recognition, and memory (Milner, 2012), and difficulties in GF processing have been associated, for instance, with dyspraxia (O’Brien et al., 2002, Grinter et al., 2010). Finally, in addition to the efficient functioning of both the streams, interaction between them is required for developing skills of flexible attention regulation and neurocognitive processing (Grinter et al., 2010, Milner, 2012, Binkofski and Buxbaum, 2013, Cloutman, 2013).

Studying complex visual processing typically relies on behavioral responses that may be difficult to achieve in preverbal infants, especially in neurologically compromised subjects. Intriguingly, recording cortical responses to complex visual stimuli may elucidate higher level cortical processing as early as few months of age (Braddick et al., 1986). This EEG response to visual stimulus (ERVS) method was introduced by Braddick et al., 1986, Braddick et al., 2000, Wattam-Bell et al., 2010 and is based on analyzing cortical steady-state responses to a periodic visual stimulus using orientation reversal (OR), GF, and GM paradigms. These paradigms are considered to reflect cortical visual processing at the level of the V1 area (OR) and the ventral (GF) and dorsal (GM) streams (Braddick et al., 2000, Braddick et al., 2005, Wattam-Bell et al., 2010). These visual evoked responses have been subsequently studied in several infant groups who were at risk of developmental compromise, including very or extremely preterm babies (Atkinson et al., 2002) and infants with perinatal brain injuries (Mercuri et al., 1996) or other neurodevelopmental disorders (Braddick and Atkinson, 2011, Lee et al., 2012).

While the test paradigm has been used in several studies by the original authors, its wider spread has been slow, mainly because of the methodological issues that make its clinical implementation a challenge. Most importantly, there have been inadequate means to control stimulation according to subject’s attention, the analysis scripts have not been openly available, and the stimulation system has relied on experienced and manual operation. Prior studies have also shown that the existing method of implementation may be unable to detect responses even in healthy infants. This suggests low sensitivity, which significantly challenges the ERVS method’s use in clinical diagnostics.

In the present work, we aimed to improve the ERVS method to enable reliable studies as part of the clinical routine in our pediatric neurophysiological unit. We believe that several problems with the ERVS paradigm may be solved. First, the reported low sensitivity could be due to poor coordination between the patient’s gaze and stimulus presentation, leading to analysis of signals when the infant is not observing the screen. This was addressed by implementing continuous gaze tracking to guide the stimulus presentation and analysis. Second, factors related to low electrode count (e.g., only one occipital channel as used by Atkinson et al., 2002, Braddick et al., 2005, Lee et al., 2012) or suboptimal computational paradigm could lead to false-negative results. In the present study, we thoroughly investigated the experimental variables to maximize the response rate in healthy infants.

2. Materials and Methods

2.1. Participants

We recruited a cohort (N = 39; 17 females) of normally developing infants (according to medical history and parent interview) that were born at full term (>37 weeks of gestation) at the Helsinki University Hospital. The results were recorded at mean age of 3.4 months (range 3.0–4.0 months). Two infants were re-tested at the age of 5.7 (subject 23) and 5.8 (subject 25) months because of technical difficulties in the first test session.

2.2. Devices and recordings

2.2.1. Eye tracking

The eye tracker system used in our study was Tobii T120 (Tobii Technology AB, Stockholm, Sweden) and is equipped with an integrated 17′’ thin-film transistor display (refresh rate: 60 Hz, response time: 4 ms). The system samples gaze data at 120 Hz, operates at 50–80 cm distance from the eyes, and can follow head movements within a window of 30 × 22 cm (at 70 cm from the screen).

The Tobii eye tracker system is based on pupil center corneal reflection, i.e., near-infrared illumination and its reflections from the cornea relative to the center of the pupil. The light reflections are captured by two cameras, and a general 3D model of the eye and the angles, distances, and other geometrical features of the reflections are used to calculate the positions of the eyes and the direction of gaze. This can then be projected onto the display to identify the pixel coordinates where the gaze is fixated.

2.2.2. EEG recording and re-referencing

The visual responses were collected by a multichannel EEG recording. We used the routine NicoletOne V44 EEG recorder (Cardinal Healthcare/Natus, U.S.A.) and ANT WaveGuard EEG caps (ANT-Neuro, Germany) with 20 Ag/AgCl electrodes positioned according to the international 10–20 electrode system. Because the analysis was focused on posterior–occipital regions, we also included Oz. We used 5 kΩ as the target impedance level at the electrode–skin interface before data acquisition. The sampling rate was 500 Hz, and Cz was used as the recording reference. After recording, the data were exported as an EDF file for further processing using custom-scripted MATLAB routines.

We compared multiple different offline montages. Because the evoked responses were expected in the postcentral regions, we performed the primary analysis using a regional frontal average reference computed from the five frontal EEG channels (F3, F4, F7, F8, and Fz). This choice was made to reduce the sensitivity to single electrode noise, which may readily confound monopolar recordings (as shown in Supplementary Fig. A.1). For instance, frontal muscle activity is unavoidably present in awake infants, but its lack of spatial coherence (Freeman et al., 2003) allows its easy removal by averaging. In addition, we also performed analyses using (i) simple Fz reference, (ii) the original Cz-referenced recording montage (comparable to Braddick et al., 2005), and (iii) the commonly used grand average montage (Wattam-Bell et al., 2010) including all channels (except Fp1 and Fp2) in the reference. Channels with a standard deviation of >300 μV over the whole recording were excluded from both average references.

2.2.3. Synchronization between stimuli, EEG, and eye tracking

The start and end times of the video files were time-stamped to the gaze data files to allow synchronization between stimuli and eye tracking. Synchronization between stimuli and EEG was achieved using a custom battery-powered light sensor (BPW34 photodiode; Vishay Semiconductors, U.S.A) attached on the corner of the stimulation screen. We incorporated black and white squares in this corner (visible in Fig.1A and B) to provide a frame-level timing of the visual stimuli. The light sensor transformed the binary patterns into pulses for a bipolar input of the EEG amplifier. In later studies, this custom-made light sensor was replaced by a commercially available, photodiode-based StimTracker (Cedrus, CA, U.S.A) that transfers trigger information directly to the EEG amplifier as transistor–transistor logic-compatible digital pulses.

Fig. 1.

Overview of the paradigm. A Infant sits in the lap of the caregiver and is shown visual stimuli continuously on the display of the eye tracker that is synchronized to a parallel EEG recording through a light sensor attached on the corner of the screen. The experimenter sits behind a light wall and controls the progression of the test according to the observations through a one-way transparent window and an interactive feedback mechanism of the presentation software. B In the orientation reversal (OR, top) stimulus, sine grating switches its orientation continuously between 45° and 135° angles. In the global form (GF, middle) and motion (GM, bottom) stimulation, coherent and non-coherent phases are presented alternately. The stimulus consists either of short white arc segments (GF) or small white dots (GM). C Above: The blue bars depict the evolution of gaze fixation quality during four successive 10-s-long GM stimulation sequences, each consisting of 20 reversal cycles (black dotted lines). The occasional gaze fixation quality drops indicate the participant’s temporary disorientation from the stimulus. The epoch selection algorithm identifies segments with good (green) and poor (yellow) gaze fixation for further analysis (fixed gaze threshold of 45% used here). Below: Power spectrum of good fixation epochs (green) reveals a distinct peak at a stimulus frequency of 2 Hz. The spectral peak is substantially decreased in rejected epochs with poor fixation (yellow) and completely missing from the spectrum calculated from the baseline epochs measured in between the stimulus sequences. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

2.3. Stimuli

2.3.1. Orientation reversal

The OR stimulus consists of high-contrast sinusoidal wave gratings with a spatial frequency of 0.45 cycles/degree at an average viewing distance of 60 cm. The grating pattern switches its orientation back and forth between oblique angles of 45° and 135° at a frequency of 4 Hz (Fig.1B top; Braddick et al., 1986, Braddick et al., 2005).

The ORs are accompanied by random phase shifts in the gratings recurring at 24 Hz. The purpose of these “jitters” is to control non-orientation-specific responses to local contrast change. Because of the random phase shifts, local luminance variations in the pattern are statistically similar in every jitter and OR and do not contribute to the component at the stimulus frequency (Braddick et al., 2005).

2.3.2. Global form

The GF stimulus consists of a set of short, white arc segments on a black background (N = 2000, length = 1.3°). The arrangement of the arcs switches between global coherent and non-coherent states at a frequency of 2 Hz. In the coherent state, the arcs are concentrically organized to create a global circular pattern, while in the non-coherent state, the arcs are randomly oriented without any perceivable global structure (Fig.1B middle; Wattam-Bell et al., 2010).

2.3.3. Global motion

The GM stimulus consists of a set of white dots (N = 2000, diameter = 0.3°) that move on a black background. Similar to GF, the stimulus alternates between states of global coherence and non-coherence at a frequency of 2 Hz. In the coherent state, the dots move along short tangential trajectories creating the perception of a global rotational movement, and in the non-coherent state, they move independently in random directions (Fig.1B bottom; Wattam-Bell et al., 2010).

2.3.4. Generation of stimuli as video files

All stimuli were in digital video format, first generated frame-by-frame in MATLAB using the routines included in Psychophysics toolbox (version 3; Brainard, 1997). The frames were then encoded into MPEG-4 video files of 10, 20, and 30 s lengths. The frame rates of the video clips were 24 Hz for OR and 60 Hz for GF and GM stimuli. The resolution was 1024 × 768 pixels for OR and 800 × 600 for GF and GM because of their higher frame rate. Low-resolution versions of the videos are shown in the Supplementary Material (Appendix B.1). The video files with original resolution are available on request from the author.

2.4. Test session

During the recording session (see Fig.1A), the infant was connected to an EEG recorder and placed in the sitting position in a baby carrier attached on their parent’s chest in front of the eye tracker display. The ambient light was kept dim to reduce the interference of diffuse light reflections with the eye tracker, and the infant-parent dyad was separated from the experimenter with a light wall to improve the baby’s concentration. The parents were informed not to look at the screen to avoid confusing the eye tracker.

Sessions always began with a routine calibration procedure (Tobii Studio: version 2.2.8). Then, for each of the three visual stimuli, we aimed to record at least 100 s of EEG from periods where gaze quality was of sufficiently high quality (accumulation monitored by the eye tracker). In some infants where this was not achievable because of poor cooperation, we prioritized the GF and GM stimuli over OR.

2.4.1. Fixation-based stimulus presentation

The stimuli were presented using the E-Prime software (version 2.0.8.22, Psychology Software Tools, PA, U.S.A) and E-Prime Extensions for Tobii (version 2.0.1.5) interfacing with the eye tracker hardware.

The presentation of visual stimuli was controlled by the infant’s gaze. We used an attractor, a simple video animation presented in the center of the screen, and the playback of stimulation video started only after the eye tracker had detected a sufficiently stable (600-ms-long segment of continuous fixation on the target) fixation on it. The operator could also see a real-time presentation of the infant’s eye positions, which could be used to manually launch the stimulus presentation with the pressing of a key.

2.5. Response analysis

The rationale of our data analysis was that continuous visual stimulation elicits a steady-state response in the visual cortex that may be distinguished as a significant increase in spectral energy at the stimulus frequency compared to EEG where no stimulus is applied (Braddick et al., 2005). Notably, this effect is much milder than the typical flash-evoked photic driving; moreover, the recording technician can recognize the response by online visual inspection of the EEG signal, although only occasionally (example in Supplementary Fig. A.1). Detection of the spectral peak may be accentuated by incorporating an eye tracker-based segmentation (Fig.1C). The individual response rate was statistically examined for each stimulus type. The null hypothesis was that the spectral values at the stimulus frequency, across several epochs of the recorded signal, were consistent with random fluctuations (one-sample test) or indifferent from components of a separate baseline sample taken from the non-stimulation phases of the same recording (two-sample test). Conversely, the alternate hypothesis was that this difference was significant and a spectral component related to the evoked response was present in the signal within a given confidence level.

In addition, we performed spatial analysis of the detected responses using the multichannel EEG data from the electrodes positioned across the scalp to (i) ascertain that the responses occur in regions known to involve visual processing, (ii) improve the response detection algorithm by focusing on these areas, and (iii) guide appropriate EEG montage.

Fig. 2 presents a flow chart of the analysis chain that was used in the work starting from the raw EEG and simultaneous eye tracker data to the determination of the response presence. The essential signal processing steps are shared as a downloadable MATLAB package in the Supplementary Information C.1.

Fig. 2.

Flow chart of the method presented in this work. The chart presents the signal processing pipelines that were used in the present work, starting from the raw EEG and eye tracker gaze data to the determination of the response presence using a statistical test. Boxes show all the major processing steps, and their specific parameters are given in parenthesis. The signal processing steps on green-colored blocks can be found in the Supplementary Material of this paper as MATLAB scripts along with brief example dataset of the recordings. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

2.5.1. EEG pre-processing and epoch segmentation

After re-referencing, all EEG channels were processed independently. We first filtered the EEG signals with an eighth-order Butterworth bandpass filter (passband 0.5–30 Hz) with forward and reverse implementation (Braddick et al., 2005, Wattam-Bell et al., 2010). Next, channels with excessively poor signal quality (e.g., loose electrode contact) were excluded from further processing if their standard deviation exceeded 800 μV.

The epoch segmentation was initiated by assessing the EEG of all 0.5-s-long stimulus reversal cycles on the basis of the recorded trigger data. In a simple artefact rejection procedure, the segments were rejected if their peak-to-peak voltage exceeded 200 µV (Braddick et al., 2005, Wattam-Bell et al., 2010).

The accepted EEG segments were then merged to form longer epochs. This was performed using a sliding window (0.5 s step) that calculated the proportion of time when the gaze was directed on the screen according to the eye tracker. This gaze quality index ranged from 0% to 100% and was applied to a fixed threshold (Fig.1C). The tracked gaze coordinates (x and y) were not used in response analyses.

We mainly used epoch length of 1 s with 45% gaze quality threshold, while these parameters were systematically assessed over a wide range of values. The spectral estimate of the EEG was computed using Fourier transform (FFT), which, with 1 s epochs, provides a frequency resolution of 1 Hz.

Finally, we selected baseline epochs from periods without visual stimulation (usually before, after, or in between the stimulus sequences), and the rejection criteria (>200 µV peak-to-peak voltage) was applied for epochs with stimulation. This epoch set acted as an unstimulated reference in the method evaluation and in the two-sample statistical tests.

2.5.2. Computational response detection methods

As with the previous studies using OR, GF, and GM stimuli (Braddick et al., 2005, Wattam-Bell et al., 2010), we used a circular variant of T² test statistic by Hotelling (1931) called T²circ (Victor and Mast, 1991). This statistic measures the signal-to-noise ratio of periodic events in the recorded EEG. Higher T²circ values indicate less variability between the epochs and thus stronger correlation between the stimulation and the recording.

T²circ is a bivariate statistic that utilizes the real and imaginary components of the spectral representation (through the FFT) of the EEG signal at the stimulus frequency, which is assumed to be independent. In the one-sample version of T²circ, the null hypothesis of the test is that no response signal is present, i.e., the spectral properties of the EEG epochs are indifferent to random scatter about zero (noise). Then, T²circ may be calculated from N epochs of EEG using the following equation:

$$T_{\mathit{circ}}^2=N(N-1)\frac{|\langle z{\rangle }_{\mathit{est}}{|}^2}{{\sum }_{j=1}^N|z_j-\langle z{\rangle }_{\mathit{est}}{|}^2}\text{,}$$ (1)

where z_j is the independent and complex-valued spectral values at a given stimulus frequency (2 or 4 Hz) and <z>_est is their empirical mean across epochs.

When defined this way, T²circ is statistically distributed according to a F-distribution (F_[2,2N-2]), which allows the determination of a p-value. Moreover, it induces a dependence on the number of analyzed epochs. When the size of a dataset consisting of N statistically uniform epochs is increased, the resulting T²circ values monotonically increase as a function of N (see T²circ tests with simulated EEG data in the Supplementary Fig. A.2). This is expected in hypothesis testing as smaller differences can be deemed significant when the sample size increases; however, this is an issue as we have variable epoch numbers per infant.

In addition to the conventional T²circ, several other statistical tests were evaluated: one-sample univariate t (Victor and Mast, 1991), one-sample multivariate Hotelling’s T² (Hotelling, 1931), two-sample univariate Mann-Whitney U (Mann and Whitney, 1946), two-sample univariate t, two-sample multivariate Hotelling’s T² (Hotelling, 1931), and two-sample multivariate T²circ (Victor and Mast, 1991). In the two-sample tests, the reference sample consisted of the baseline epochs. The multivariate statistics exploit both the real and imaginary spectral components at the stimulus frequency, whereas univariate tests only use the magnitude of these components.

2.5.3. Detection from multichannel data

Visual responses are conventionally studied using only one recording electrode placed over the occipital area (e.g., at Oz; Odom et al., 2010, Norcia et al., 2015), and the reference electrode is typically placed at the vertex or midfrontally. This fixed placement is very sensitive to inter-subject variability in response topography (Picton et al., 2003). Our prior work on spatial properties in infant EEG has suggested that the scalp-recorded EEG activity may significantly vary within few centimeters (Odabaee et al., 2013, Odabaee et al., 2014). Hence, using multichannel data is likely to improve response detection; however, it also leads to the need for the appropriate correction of multiple statistical comparisons to control family-wise error rate (i.e., positive findings by chance).

We adopted the method in the reference works: the false discovery rate (FDR) correction by Benjamini and Hochberg (1995) that adjusts the selected confidence threshold of the statistical test according to the number of parallel comparisons—in this case, the number of channels analyzed. To avoid performing an overly conservative correction, we decided to limit the number of the channels analyzed by focusing on the postcentral area alone for response detection (specifically T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, O1, Oz, and O2 were included).

2.5.4. Evaluation of the response detection performance

One of the aims of this study was to investigate the various technical factors that could affect the response detection results. In the analysis workflow presented above, variables such as the number of EEG channels, the re-referencing method, the applied statistical test, p-value criterion, epoch length, and the gaze quality threshold of the epoch formation must all be chosen carefully to provide optimal results.

This optimality means a combination of the experimental variables that maximizes the sensitivity and specificity of the classifier (response detector). As there was no “ground truth” information available about the findings in our cohort, we had to assume that all babies elicited a response during stimulus intervals and no response during the non-stimulus. Thus, sensitivity was defined as the percentage of infants with a response at the stimulus frequency during stimulus (true-positive detection rate; TPR) and specificity as the percentage of infants with no response at the stimulus frequency during non-stimulus (true-negative detection rate; TNR).

We used the area under receiver operating characteristic curve (AUC) to combine the measures of sensitivity and specificity. Single-point AUC is defined as

$$\mathit{AUC}=\frac{\mathit{TPR}-\mathit{FPR}+100\%}{2}\text{,}$$ (2)

where FPR is the false-positive rate, which equals 100% − TNR. Infants were included if at least 60 s of valid EEG data were recorded during stimulus.

2.5.5. Response topography

Spatial distributions of the visual responses were examined using all the 20 electrodes. Each channel was analyzed individually. After pooling studies with significant responses, the group-level topographic findings were presented in two ways, separately for each stimulus type: (i) by averaging the calculated T²circ values for each cohort, and (ii) by calculating the channel-specific response rates that show how many of recordings yielded a response for each channel.

3. Results

3.1. Response rates after different detection methods

We first analyzed the EEG using the settings originally proposed by Braddick et al. (2005) (T²circ statistics; p < 0.05; 1 s epochs; only Oz) ignoring the gaze data from the eye tracker, which resulted in AUCs of 92%, 94%, and 78% for OR, GF, and GM, respectively. Incorporating multichannel EEG data (13 postcentral electrodes) and corrections for multiple comparisons improved the AUCs to 94%, 97%, and 92%, respectively. By reducing the statistical detection criterion (p-value), we could alter the ratio of false positives to sensitivity. A detection threshold of p < 0.01 resulted in AUCs of 96%, 99%, and 92% for OR, GF, and GM, respectively, with a FPR of 0% for all stimulus types.

The use of longer epoch length (Supplementary Fig. A.3) or alternative one-sample statistical tests did not significantly improve the AUCs. Hotelling’s T² (AUCs 94–97%) provided almost identical figures for T²circ, and even conventional Student’s t-test showed reasonable performance (AUCs 91–96%). Two-sample tests incorporating a set of baseline EEG epochs did not increase the response rate compared to one-sample tests (best was Hotelling’s T²: TPRs 79–92%). The comprehensive detection results acquired using the other statistical tests described in the section “computational response detection methods” are presented as a table in Supplementary Information C.2.

3.2. Incorporating eye tracking into analysis

Our initial analyses suggested that epoch rejection based on the gaze quality index can result in larger T²circ values, indicating an improvement in detection algorithm efficiency (see Fig. 3 and Supplementary Fig. A.4). To maximize the T²circ per subject, a variable threshold would be required as there was significant variation with subjects and stimulus types (Fig.3D).

Fig. 3.

Optimization of the gaze quality threshold. A To find the optimal threshold level for the gaze quality in the epoch selection, characteristics of the visual responses were studied as a function of variable gaze threshold ranging from 0% to 95%. Average response rates were then calculated for each step using the T²circ method (1-s-long epochs, EEG referenced to frontal average, only posterior channels included, p < 0.01). Recordings during OR, GF, and GM stimuli were processed separately, and their results are depicted with blue, green, and red, respectively, with the black graph denoting the average of all stimuli. A lower limit of 60 epochs was applied for the statistical analyses. If the number of the epochs was lower than 60 (after the gaze quality thresholding), that recording was removed from the calculations. The proportion of accepted recordings for each gaze quality threshold step is represented by the brown area in the background. B The figure shows the incremental loss of EEG epoch quantity when the gaze quality threshold is raised. The distributions of available epochs in all the recordings are presented as boxplots depicting the medians (circles) and interquartile ranges (IQRs; boxes), with the whiskers showing the total range, neglecting the outliers (maximum length 1.5× IQR from the box edge). C Figures exemplify the influence of the eye tracker data on the response analysis by presenting the highest T²circ values for each recording (N = 39) as a function of analyzed epochs quantity with and without the use of fixed 45% gaze quality threshold. OR, GF, and GM recordings are marked with blue circles, green squares, and red triangles, respectively, but the associated trend lines are fitted (in a least-squares sense) for all stimulus types pooled together. Although the slopes of the trend lines are positive in both figures, the use of the gaze threshold produces a stronger increase per epoch, indicating an improvement in the detection algorithm efficiency. A comparative analysis that includes other gaze quality thresholds is presented in Supplementary Fig. A.4. D The bar graph depicts the best thresholds for each stimulation session selected by maximizing the T²circ value in the recordings (epoch number criterion of >60 applied if possible). OR, GF, and GM recordings were processed separately and are depicted with blue, green, and red bars, respectively, with the dotted lines denoting the corresponding averages. The black slashed line represents the mean of all the recordings. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

We performed epoch rejection using a variable gaze threshold that was automatically selected to maximize the T²circ in each case (0–95% in 5% steps, with the additional condition that N > 60). The total epoch quantities before and after this are presented as a table in Supplementary Information C.3. With a detection threshold of p < 0.01 applied, this approach resulted in AUCs of 96%, 100%, and 97% for OR, GF, and GM, respectively, no false positives, and an average rejection rate below 1%.

The increase in AUCs with regard to changes in the analysis method is illustrated in Fig. 4, whereas the findings are presented individually in the Supplementary Fig. A.5 together with detection results calculated using the alternative EEG montages.

Fig. 4.

Evaluation of the response detection performance. Factors affecting response detection were evaluated by comparing “area under receiver operating characteristic curve” (AUC) values calculated from the sensitivity and specificity figures from five different test configuration (HEL #1–5). The corresponding AUCs from the reference works (Braddick et al., 2005, Wattam-Bell et al., 2010; FPR information not available, but presumed to be equal to our findings) are presented leftmost (LND). Proceeding to the right, HEL #1 analysis was performed with only Oz channel (1 s epoch length and p < 0.05 T²circ significance level). Next, HEL #2 shows how the detection improves when all posterior EEG channels are included. In HEL #3, a tighter T²circ criterion of p < 0.01 is adopted, which eliminates all false-positive findings. HEL #4 adds fixation-based epoch segmentation using a fixed gaze quality threshold of 45%. Finally, in HEL #5 we used gaze thresholds that were adjusted to each recording individually (details in the Fig.3D). The proportion of accepted recordings with each test configuration is represented by the brown area in the background. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

3.3. Response topography

Topographical analysis of the responses was performed only at group level (Fig. 5). Each stimulus type elicited a distinct response mainly in the posterior and occipital areas. There were also modest differences in the spatial extent of the response magnitudes and response rates. Typically, the responses for the OR and GF stimulations were more prominent and appeared mainly in the occipital channels, while GM responses covered a broader area including electrodes from the horizontal midline (T3-C3-Cz-C4-T4).

Fig. 5.

Response topography. A Channel-specific response magnitudes (black line; left-sided y-axis) were calculated by averaging the T²circ values from the subset of the study population that yielded significant responses (p < 0.05; frontal average reference; gaze threshold individually adjusted). The inter-subject variability of the response strength at each channel is characterized by the interquartile ranges of measurements, which are depicted as dotted lines in the figures. The colored bars in the background show the average response detection rates at the specific channel (p < 0.05 without FDR correction; right-sided y-axis). B Group averages of channel-specific average T²circ values (left) and response rates (right) are interpolated to fit into a standard infant head model (Delorme and Makeig, 2004) to yield topographic maps of the visual responses in 2D views. Note that the colorbars of the 2D maps are scaled independently, matching the maximum T²circ value in each map. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Additional results from topography analyses, consisting of surface maps with alternative montages, are presented in Supplementary Fig. A.6. They show how the use of grand average reference resulted in the apparent spread of response to frontal regions, which was avoided with frontal average and Fz-referenced montages.

4. Discussion

Our work shows that spectral responses to complex visual stimuli of OR, GF, and GM can be reliably recorded from the infant cortex at few months of age. While the overall finding is fully compatible with those of earlier ERVS studies (Braddick et al., 2005, Wattam-Bell et al., 2010), we show here that the response detection rates can be substantially improved by further development of the recording setup and by optimizing the signal analysis pipeline.

The initial aim of our work was to implement the ERVS method of Braddick et al. (2005) and Wattam-Bell et al. (2010) with three major enhancements: (i) use of multichannel EEG for all stimulus types, (ii) integration of eye tracking information, and (iii) optimization of the analysis protocol. The outcome was evaluated using a cohort of healthy 3-month-old infants, and the resulting AUCs of 96%, 100%, and 97% for OR, GF, and GM, respectively, exceeded our expectations.

We found several essential and adjustable details in the analysis flow. We saw improvements in response rate from (i) the use of conservative classification criterion and (ii) epoch segmentation based on gaze quality. The use of the criterion p < 0.01 eliminated all false-positive findings and preserved true detection rates between 92% and 100%. The introduction of epoch selection using adaptive thresholding of gaze quality permitted the removal of segments with poor gaze quality and predominantly low statistical values, which further improved the detection, thus increasing AUCs by 2% on average. These results should, however, be interpreted in the context of earlier studies where the response rates (p < 0.05) in the corresponding age group were approximately 79%, 50%, and 92% for OR, GF, and GM, respectively (Braddick et al., 2005, Wattam-Bell et al., 2010). Compared to them, our enhancements produced an overall increase of 15% in AUCs on average.

The implemented epoch selection approach should not be confused with the artificial improvement of results. It is, indeed, fully analogous to the manual rejection of artefacts in conventional evoked potential studies (e.g., Luck, 2005, Odom et al., 2010, Picton et al., 2000) that result in improved signal-to-noise ratios. Here, we developed an automated way to perform this optimization, which allows more objective and easier data analysis accounting for the observed variability between the recordings.

In addition to the use of gaze quality for epoch selection, we also found that stimulus presentation can be dynamically guided by the eye tracker to ensure that the infant’s attention is on the screen while stimulating. Such co-operation is obviously crucial for the assessment of visual responses. Our experimenters also found it highly beneficial to have a real-time feedback of infant’s gaze during stimulation so that they could manually control the system as needed. We were unable to directly assess the potential improvement as we did not perform experiments without the use of the eye tracker. However, it certainly helped us to record an adequate amount of EEG from nearly all the participants (>60 epochs in 114/117 sessions) and may also be a reason why our initial analyses exceeded the figures reported in prior studies.

Our topographical analyses showed that the introduction of multichannel data was essential for the detection of GM responses that were spread over a broader postcentral area compared to the more restricted OR and GF responses. These findings are compatible with those of prior work in newborns and adults (Braddick et al., 2003, Braddick et al., 2005, Wattam-Bell et al., 2010), which suggest at least partly different cortical generator areas for the three responses. The OR responses are considered to arise in the V1 area and provide an early shape-selective activity (Braddick et al., 2005) to serve as the basis for object- and pattern-selectivity later in the striate and temporal lobe areas. The GF and GM stimuli are considered to reflect global processing in the ventral and dorsal visual streams (Braddick et al., 2000), respectively. More recent work comparing infant and adult responses (Wattam-Bell et al., 2010) showed that the spatial organization of response topography is still developing in infants; hence, it may not be essential to focus on detailed spatial analysis in the context of the present study. Our topographic data clearly support the presence of different spatial topographies, and this information is useful for designing optimal electrode montages. For instance, the widespread GM responses are inadequately recorded when using a grand average reference, while it can be well presented when using reference at frontal or central areas. Conversely, a simplified recording setup using only occipital electrodes and a central or frontal reference (cf. Braddick et al., 2005) could be sufficient for the detection of occipitally dominant OR and GF responses. Indeed, the use of multiple electrodes, a natural by-product of EEG recording, is helpful for achieving better response detection rates than in earlier literature.

4.1. Clinical practicalities of the ERVS method

The stimulus presentation and the offline response analysis were implemented as semi-automated applications, which allows to perform the study with less technical expertise. In practice, our recordings were performed by our clinical neurophysiology technicians after less than an hour of training.

The cost of eye tracking devices has come down to the level of that of EEG accessories, and they are readily available in many consumer applications. This enables fast and simple integration with any external display that is capable of high-quality visual stimulations.

From the technical point of view, the most challenging part in our experimental setup was the synchronization between stimulus presentation, eye tracker, and EEG recorder. This was successfully solved by using a commercial product that captures luminance change directly from the screen; however, future implementation could build the synchronization directly into the evoked response-recording system.

4.2. Barriers to entry into clinical diagnostics

Applicability of the developed methods at the individual level for clinical evaluation would still require validation, which is conventionally performed by calculating the statistical performance indicators of sensitivity and specificity. The orthodox derivation of these figures would require “ground truths” of the studied features of interest. If this information was available, the p-value criterion could be adjusted to optimize the receiver operating characteristics of the classification, i.e., balance between true and false findings.

In this study, all measured infants were, based on their age and history, principally likely to yield responses. Thus, the observed detection rates, which were high during stimulation and low during baseline segments, hold high promise for the ERVS method’s potential clinical use in differential diagnostics at an individual level. However, because of the challenges mentioned above, the validation will probably come from the test of time by providing a perceived added value to studies seeking early developmental biomarkers. Validation in this manner will be based on larger scale recruitments of both typically and atypically developing infants, which is already underway in our laboratory.

5. Conclusions

In this work, we presented an eye tracker-assisted method for the detection of visual responses to complex stimuli in the EEG of infants. Changes to the experimental setup and analysis protocol and the incorporation of information from an eye tracker resulted in a statistically significant (p < 0.01) response to presented visual stimuli with a negligible amount of false positives in nearly all measured infants. The output of the work is a test protocol suitable for the clinical environment where individual diagnostics are required.

Conflict of interest

None.

Appendix A. Supplementary data

Supplementary Fig. A.1.

VEP responses in time and frequency domain. A Screenshot shows a segment of raw EEG traces before and after the onset of the OR stimulation as observed by the technician. Although in this brief example the 4 Hz component of the VEP response can be visually distinguished at the occipital channels, this was possible only occasionally. B Figure shows the individual averaged EEG responses and their grand average in time (left) and frequency (right) domains. The ERVS responses were calculated for each infant and paradigm from the O2 channel epochs with gaze quality index of >20%. They are each depicted in grey, whereas the black trace presents their grand average. The frequency domain representations of the responses are given as T²circ spectra. These graphs support our preference for the frequency domain-based analysis in the response detection as it is likely to yield higher signal-to-noise ratio.

Supplementary Fig. A.2.

Evaluation of the T²circ analysis method with simulated data. Supplementary T²circ analyses were performed using simulated datasets that included 1-s-long noise and dummy response epochs. The dummy epochs consisted of two additive components, namely 4 Hz sine wave (RMS voltage 4.2 µV) and random pink noise (RMS voltage 31.7 µV), whereas the noise epochs in the datasets included only the latter component. A Sensitivity of the T²circ response detection method was assessed by plotting the T²circ values of the simulation data as a function of the number of noise epochs added to the basis dataset of 30 dummy response epochs. The red dashed line in the figure corresponds to the p = 0.01 significance level. B Similar analyses were also performed conversely by calculating the T²circ values from datasets consisting of increasing number of simulated response epochs. The figure shows how T²circ values rise almost linearly as a function of the dataset size (slope of a linear trend line here would be 0.39). To account for the randomness in the simulation data, the calculations described above were repeated multiple times. In the figures, the solid and the dashed graphs depict averages and ±SD intervals of these repeated measurements, respectively.

Supplementary Fig. A.3.

Influence of the epoch length. A Average response rates calculated for epoch lengths from 1 s to 5 s using the T²circ method (45% gaze quality threshold, EEG referenced to frontal average, postcentral electrodes included, p < 0.01). Recordings using OR, GF, and GM stimuli were processed separately and are depicted with blue, green, and red, respectively, with black graph denoting their average. A lower limit of 60 s was applied for the statistical analyses. If the combined duration of the analyzed epochs was lower than 60 s (after the gaze quality thresholding), that specific recording was rejected and removed from the response rate calculations. The proportion of accepted recordings for each gaze quality threshold step is represented by the brown area in the background. B The number of epochs available for the response analysis decrease when the epoch length is extended. Distributions of the epochs in each recording with fixed duration ranging from 1 s to 5 s are presented as boxplots depicting the medians (circles) and the interquartile ranges (IQRs; boxes), with the whiskers showing the total range, neglecting the outliers (maximum length 1.5 × IQR from the box edge). The performed analyses did not reveal any self-evident choice for the optimal epoch length. Although increasing the epoch duration provides higher spectral resolution, the decline in epoch quantity decreases their statistical usefulness in the process. As these two opposite effects counter-balance each other quite evenly, we opted to keep the epoch length same as that in the reference works, i.e., 1 s (e.g., Wattam-Bell et al., 2010).

Supplementary Fig. A.4.

The effect of gaze quality threshold on the response signal-to-noise ratio. We wanted to test the hypothesis that epochs recorded with good gaze quality contain higher signal-to-noise ratio (SNR) than epochs with less visual attention to the stimulus. As the gaze threshold, the number of available EEG epochs, and the resulting T²circ values are mutually related (Supplementary Fig. A.2), we needed to examine the response strength in these relationships. Hence, we computed the T²circ value for each recording as a function of epoch quantity with 0–95% threshold levels and fitted (in a least-squares sense) trend lines to the resulting distributions (two examples are presented in Fig. 3C). This figure depicts how the slope of the linear model (i.e., “response SNR per epoch”) changes in relation to the gaze quality threshold level (0–95%). Calculations were performed for OR, GF, and GM recordings separately (blue, green, and red, respectively) and for all stimulus types pooled together (black). The analysis shows two results: (i) more number of epochs have a tendency to produce higher T²circ (trend lines always increase) and (ii) the use of higher gaze threshold generally produces higher SNR per epoch (trend line slopes increase) up to 80% level, indicating an improvement in the detection algorithm efficiency.

Supplementary Fig. A.5.

Individual response detection results. Visual responses of 39 participants were determined using the T²circ analysis methods (1-s-long epochs, posterior channels analyzed) for all three stimulus types (OR, GF, and GM). For these comparative figures, the analyses were performed using five different test configurations covering two gaze quality thresholding methods (fixed >45% or individual optimization), two p-value criteria (p < 0.05 or p < 0.01), and four EEG montages (frontal average, grand average of all recorded electrodes, Cz monopolar, and Fz monopolar). In all graphs, the blue “x” and red “+” markers indicate positive and negative findings, respectively. The measurements where the number of available epochs was low (below 60) are enclosed by black squares and excluded from the calculations of the associated response rates (shown on right). Two OR recordings are missing completely from the graph as these sessions had to be stopped prematurely, after GM and GF stimulations, because of infants’ fatigue.

Supplementary Fig. A.6.

Topographical results on different EEG montages. This figure repeats the topographic analyses presented in Fig. 5 and analyses for grand average and Cz- and Fz-referenced montages. In all these cases, the topographic maps of the visual responses were provided for average T²circ values (A) and for average detection rates (B; p < 0.05). The colorbars of the T²circ maps are scaled independently depending on the maximum of each map.

Supplementary Information C.1

MATLAB functions for the response analyses as used in this work (ssvep_fromEpochs.m and calc_t2c.m) and an example dataset containing three occipital EEG channels recorded under 4 Hz orientation reversal stimulation (ssvep_example_data.mat). See headers of the m-files for more information on the usage of the functions.

Supplementary Information C.2

The detection rates (true and false findings) for 48 different analysis configurations differing in channel selection (posterior electrodes or only Oz), epoch length (1 or 2 s), use of eye tracker data (no thresholding, fixed 45% or individually adjusted gaze quality threshold levels), and, most of all, applied statistical test and its significance level (p < 0.01 or 0.05) are presented. In addition, the first row in the table presents the corresponding numbers from the reference works (Braddick et al., 1986, Wattam-Bell et al., 2010), although the false-positive rates here are assumed to be equivalent to that in our analyses. Specifically, the applied statistical tests were as follows: one-sample multivariate T²circ, one-sample univariate t, one-sample multivariate Hotelling’s T², two-sample univariate Mann-Whitney U, two-sample univariate t, two-sample multivariate Hotelling’s T², and two-sample multivariate T²circ. In the two-sample tests, the second sample consisted of the baseline epochs that otherwise were used in the evaluation of the false-positive rates. The multivariate statistics exploit both the real and imaginary components of the FFTs, whereas the univariate tests only use the amplitude of these components. The configurations were evaluated on the basis of the true- and false-positive rates (TPR, FPR) using area under receiver operating characteristic curve (AUC) metric. As there is only one TPR/FPR pair available per configuration, AUC can be calculated as AUC = 0.5 × (TPR-FPR + 100%). Finally, the last columns present the rejection rates in each configuration.

Supplementary Information C.3

The table presents the number of 1 s epochs considered in the response analysis of each individual (#1–39). The data are presented for all visual stimuli with (individual gz thr) and without (no gz thr) eye tracker-based epoch rejection. The former of these cases tells the total recording length during the visual stimulation (excluding epochs with artefacts). Although we aimed to record at least 100 s of EEG for all visual stimuli, in one OR session (#25), we only managed to acquire 40 EEG epochs, which is below the lower limit of 60 that we used for the statistical analyses. In addition, two OR recordings (#12 and #19) are missing completely as these sessions had to be stopped prematurely because of infants’ fatigue.

Supplementary video B.1

Low-quality versions of the videos used in the stimulus presentation encoded in one file comprising 10 s of each stimulus type in succession (order: OR, GF, and GM). The video files with original playback quality are available on request from the author.