A Pilot Study on EEG-Based Evaluation of Visually Induced Motion Sickness

Abstract

The most prominent problem in virtual reality (VR) technology is that users may experience motion sickness-like symptoms when they immerse into a VR environment. These symptoms are recognized as visually induced motion sickness (VIMS) or virtual reality motion sickness (VRMS). The objectives of this study were to investigate the association between the electroencephalogram (EEG) and subjectively rated VIMS level (VIMSL) and find the EEG markers for VIMS evaluation. In this study, a VR-based vehicle-driving simulator (VDS) was used to induce VIMS symptoms, and a wearable EEG device with four electrodes, the Muse, was used to collect EEG data of subjects. Our results suggest that individual tolerance, susceptibility, and recoverability to VIMS varied largely among subjects; the following markers were shown to be significantly different from no-VIMS and VIMS states (P < 0.05): (1) Means of gravity frequency (GF) for theta@FP1, alpha@TP9, alpha@FP2, alpha@TP10, and beta@FP1; (2) Standard deviation of GF for alpha@TP9, alpha@FP1, alpha@FP2, alpha@TP10, and alpha@(FP2-FP1); (3) Standard deviation of power spectral entropy (PSE) for FP1; (4) Means of Kolmogorov complexity (KC) for TP9, FP1, and FP2. These results also demonstrate that it is feasible to perform VIMS evaluation using an EEG device with a small number of electrodes.

Introduction

Virtual Reality (VR) technology has advanced a lot in recent years. Many new devices have been introduced to create impressive games, movies, and other immersive experiences, indicating that they are on their way to becoming mass-market products¹ However, visually induced motion sickness (VIMS, also called virtual reality motion sickness, VRMS) may occur when a person immerses into the VR environment^2–5. VIMS is a motion sickness-like disorder that is often induced by a person exposed to an environment where the visual and proprioceptive motions are decoupled^{6, 7}. A person with VIMS suffers from headaches, stomach awareness, nausea, disorientation, sweating, fatigue, and even vomiting^{2, 4, 6, 8}, which often raises safety and health concerns for current VR platforms^{2, 9}. Therefore, VIMS has been considered as a major hurdle to overcome for a wide acceptance of VR applications.

To investigate any VIMS reduction methods, it is necessary to have tools to evaluate VIMS efficiently and effectively. So far, the simulator sickness questionnaire (SSQ)^{4, 10} has been widely used to measure the amount of VIMS experienced during VR exposure. However, this subjective evaluation method has its own disadvantages: it is usually performed before and after VIMS experiment and due to the large length of the questionnaire, it cannot be carried out in real-time, hence cannot describe the changes of VIMS during the exposure. As a result, it is difficult to detect the occurrence of VIMS or get the details of VIMS development only using this method. Simpler versions of the quick VIMS rating scheme were also introduced for pseudo-real-time VIMS measure^{2, 11}; but they still depend on subjective response, which makes the evaluation susceptible to individuals’ bias. To overcome the limitations of subjective VIMS measures, objective VIMS evaluation methods based on various physiological signals, such as electrogastrogram (EGG)¹², electrocardiogram (ECG)¹³, salivary cortisol level^{4, 14, 15}, blood pressure (BP)¹⁶, pulse rate (PR)¹⁶, electroencephalogram (EEG)^{9, 17, 18}, postural sway¹⁹, electrooculogram (EOG)^{20, 21}, and head movement²⁰, were tested. Such physiological signals mentioned can be measured continuously and precisely.

Please note that different physiological signals may associate with different VIMS theories. For example, the EEG signal is usually related to the sensory conflict theory^{12, 22}. The basic idea of the sensory conflict theory is that all situations that provoke VIMS are characterized by a condition of sensory rearrangement²² in which the motion signals transmitted by the visual and vestibular system (or maybe other proprioceptive systems) are mismatched with one another, or with what is expected from previous experience^{19, 22, 23}. Many researchers measured EEG in driving simulation and motion sickness study based on sensory conflict theory^{6, 24}, trying to make the experimental results more objective^{9, 25–29}. Postural sway is usually related to the postural instability theory³⁰, which predicts that postural activity will differ between persons who are susceptible to VIMS and those who are not, and these differences should exist before the onset of subjective symptoms of motion sickness^{19, 30–33}. The postural instability theory not only provided testable hypotheses but also been used to identify objective measures (based on the center of pressure (COP)¹⁹ and other postural indicators) to predict the occurrence of motion sickness. The EOG signal is usually related to the eye-movement theory^{20, 34}. This theory proposes that reflexive eye movements, such as the optokinetic nystagmus (OKN, can be obtained from EOG signal) during visual yaw rotation, provide eye-muscle afferences that ultimately stimulate the Nervus Vagus^{20, 34}. Support for the involvement of the OKN in VIMS is found in studies showing that VIMS severity correlates with OKN frequency^{20, 35} and OKN slow phase velocity (OKN SPV^{20, 36}). Head movement is usually related to the subjective vertical mismatch theory, which is actually a refinement of the sensory conflict theory proposing that not all sensory conflicts are provocative, but only those associated with the sense of verticality²⁰. This theory argued that VIMS symptoms may arise because subjects make inadvertent head movements while in circular vection. Such head movements cause pseudo-Coriolis effects, which are known to be provocative^{20, 37}.

In this pilot paper, we focus on the potential use of the EEG data as an indicator of VIMS evaluation. According to the sensory conflict theory, the changes in EEG data are consistent with or could be accounted for by conflict mechanisms^{9, 17}, which is believed to be one of the main causes of VIMS^{6, 7, 9, 17, 28, 29, 38, 39}. The changes in EEG signal may be caused by other factors like distress, excitement or tiredness, etc. However, previous studies have shown that EEG dynamics are related to VIMS provoked in VR-based dynamic 3D environment, and the VIMS symptoms are supposed to be the same symptoms as what are induced in the real world^{4, 40}.

Although previous studies have shown that the changes of VIMS symptoms did affect the changes of EEG signal^{9, 17, 41, 42}, the derived details from those studies were not consistent, and, for some cases, they contradicted to each other. For example, Lin et al. claimed that the power spectral density (PSD) of the alpha and gamma bands of the EEG signals can be used as VIMS markers since the correlations between those PSDs and subjective VIMS rating exceed the correlations in other frequency bands in motion sickness-related brain regions⁹ Naqvi et al. reported that the decrease in the power of the EEG alpha band can be a possible VIMS marker⁴¹. However, Chen et al. observed that the increases in the total power of the EEG alpha and theta bands were related to subjective VIMS scoring^{17, 28}.

The fundamental reason why these conclusions varied may be due to the large variability of individual susceptibility to VIMS. It is reported that about 30% viewers are suffered from VIMS when watching a moving scene²¹; However, the prevalence of VIMS can vary from 1% to 70% depending on the apparatus and stimuli²¹. In addition, the VIMS level varies for different viewers^{9, 21}.

In this paper, we describe yet another effort in testing the feasibility of EEG signals analysis for evaluating subject’s VIMS when engaged in a VR-based vehicle-driving simulator (VDS). Both subjective and objective methods were measured to evaluate VIMS. The means and standard deviations of gravity frequency (GF)^{42, 43}, power spectral entropy (PSE)⁴², and Kolmogorov complexity (KC)^{44, 45} were computed from EEG data and tested to determine whether they can be used as VIMS markers. The motivation to choose these measures is that they are reported to be highly correlated with visual fatigue⁴² or mental fatigue⁴⁴, which may be related to VIMS.

Another goal of this paper is to test whether similar results can be achieved with an EEG device with a small number of electrodes. Note that most previous studies collected the EEG data with full-scale clinical EEG equipment, which is usually expensive and inconvenient for the user to wear in a VR environment. To overcome these disadvantages, we used a wearable wireless EEG device, the Muse, for EEG data collection for its affordable price and convenience. This EEG device is a sparse recording device affording only four electrodes for EEG data collection. It supports wireless data transmission (via Bluetooth) and real-time processing. Please note that more electrodes do not always provide better results due to the complication of multi-dimensional signal noises. What’s more, it is often difficult to detect VIMS onset in real-time. In fact, some researchers have tried to reduce the EEG electrodes used in EEG applications. Cai et al. used three-electrode EEG data for depression detection. They argued that compared with 128 channels EEG, their simpler test (three-electrode EEG) can make diagnosis more accessible and widespread, researchers can perform more tests on more patients given the same amount of time and money⁴⁶. To the best of our knowledge, no one has attempted to evaluate VIMS using an EEG device with less than five electrodes. Our subsequent experiments demonstrated that a small number of electrodes are feasible to perform VIMS evaluation.

Materials and Methods

Subjects

Normally sighted (or corrected to be normal vision) subjects of age from 20 to 40 years old were recruited from the Schepens Eye Research Institute. All subjects gave their written informed consent before they participated in the study. The informed content was documented by the Institute Review Board (IRB) of the Schepens Eye Research Institute (SERI). The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Institute Review Board of the SERI of 16–015H. Eight subjects (3 males and 5 females) completed the studies and their data is reported here. Table 1 lists information about the subjects.

Table 1.

Subject information.

					Been trained	Health status
Subject	Sex	Age	Weight (kg)	Handedness	before	Vestibular system	Visual system
S1	M	20 ~ 40	76	right-handed	No	Normal	Normal
S2	M		80	right-handed	No	Normal	Normal
S3	M		77	right-handed	No	Normal	Normal
S4	F		55	right-handed	No	Normal	Normal
S5	F		63	right-handed	No	Normal	Normal
S6	F		72	right-handed	No	Normal	Normal
S7	F		93	right-handed	No	Normal	Normal
S8	F		46	right-handed	No	Normal	Normal

EEG Recording

The Muse™ (InteraXon Inc., Ontario Canada, shown in Fig 1(a)) was used to record the EEG data continually throughout the experiment. There are four electrodes in the Muse, two are located at the frontal lobe areas (FP1 and FP2), and the other two are at the temporal lobe (TP9 and TP10) areas, as shown in Fig 1 (b)⁴⁷. In our experiments, the analog EEG signals were sampled with 10-bit quantization at a sampling rate of 220Hz⁴⁷. The Muse was connected to a computer through Bluetooth; the data output by Muse was recorded and stored on the computer for post analysis.

Fig 1. Muse™ used for EEG recording.

(a) Locations of electrodes in the Muse. (b) Top-down view of the EEG electrode positions on subject’s head.

EEG data are usually contaminated by various artifacts, including eye blinks, muscle movements, and indoor power-line noise⁹. In order to remove these artifacts as much as possible, a notch filter in the Muse was enabled. The Fast Fourier Transform (FFT) coefficients extracted from the filtered signal by the Muse were used for our analysis. In our experiments, the FFT coefficients were used for GF and PSE computation; the raw EEG data measured in microvolts and filtered by the notch filter were used for KC computation.

Note that many studies have shown that the dry contact EEG device (such as the Muse) performs as far as other EEG devices with wet electrodes^48–50. Although the Muse is a sparse recording device (with only four electrodes), as shown in our manuscript, the data collected were shown to be effective for VIMS evaluation.

Driving simulator for inducing VIMS

We used a wide field driving simulator (DE-1500, FAAC Inc. Ann Arbor, MI) to induce VIMS⁴⁷. The VR-based driving simulator comprises a motion seat, a force feedback steering wheel, and five displays, which provides both realistic visual and proprioceptive stimuli to the subjects. All the five displays are 42-inch LCD displays, giving a total horizontal field of view of 220° and vertical field of view of 63°. During the experiment, the subjects were asked to drive the simulator while wearing the Muse on their head. The same driving scenario was used for all subjects. The scenario contains a long winding road (consists of multiple winding sections) that is prone to evoke VIMS symptoms as the subjects drive the VDS through this road. Some studies take the EEG data collected during driving on a straight road as the baseline/control because driving on a straight road induces less motion sickness^{9, 18}. However, it is still questionable whether the data collected during driving on a straight road can correctly serve as a control condition because physical and emotional stimulations of driving on a winding road are clearly different from that of on a straight road. In our experiments, we measured subjective VIMS level and used the scenario contains multiple winding road sections to evoke severe VIMS symptoms. However, the actual onset of VIMS occurred at a few minutes after starting the driving, meaning that what we measured was not just caused by “driving“. Note also that VIMS measure was continued even after the driving is ended. So our within-subject and within-trial segmentation of the physiological/EEG data based on VIMS onset status provides both control (no-VIMS) and effect (VIMS) for comparison.

Experimental protocol

The experiment was carried out in an air-conditioned room with a temperature of 20°C. No subjects knew the VR scenario before the experiment. A three-segment experimental protocol (see Fig 2) was prepared for VIMS evaluation: pre-driving, driving, and post-driving segment. Subjects were asked to complete an SSQ before and after the experiments. This pre- and post-SSQ requirement helps the subjects to establish a more consistent VIMS rate scale by familiarizing with the contents of the SSQ before the experiment.

Fig 2. A typical VIMSL changes during the three-segment experimental protocol.

The x-axis denotes the timeline and the y-axis denotes the measured VIMSL. The blue bars between the experimental sections indicate the transition time between getting in and out of the driving simulator.

In the pre-driving segment, the subjects were required to remain quiet and relaxed, and their baselines of physiological (EEG) state were recorded. In this segment, no VIMS occurred for all subjects.

The driving segment comprised driving on a long winding road, which had been known to induce motion sickness to the subjects^{9, 18}, but it still requires some time to invoke VIMS. Each subject had different VIMS tolerance so actual driving duration varied from several minutes to over 30 minutes. During the driving section, the subjects verbally reported their subjective rating of VIMS level (VIMSL) when they felt there was a change of VIMSL. The VIMSL can be 0 (no-VIMS), 1 (slight VIMS), 2 (moderate VIMS), 3 (severe VIMS), and 4 (very severe VIMS). We used this simple asynchronous VIMSL reporting method to obtain temporal VIMSL changes that the subjects experienced in real-time. In addition, it can avoid as much noise as possible being introduced into the EEG data due to speak. Note that a similar temporal VIMS reporting scheme was used to measure the effect of dynamic (peripheral) visual field size change on VIMS². The subjects continued to drive until they felt very uncomfortable and could not drive anymore.

After stopping driving, the subjects got out of the driving simulator for post-driving measurement. In this segment, they are asked to have a rest to recover from the motion sickness. The duration for recovery varied between individuals.

Note that EEG data and VIMSL were recorded throughout the procedure. There were brief interruptions (e.g. for getting in and out of the driving simulator) of measurements between each segment (less than 1 min), which are labeled as “transition” in Fig 2.

In our experiments, each subject performed only a single trial, then the data from all the subjects were used for the analysis of each potential marker. The reason why each subject did not repeat the trial is that making a subject repeating the trial may change his/her adaptation (i.e. tolerance, susceptibility, or recoverability) to VIMS^51–54. This is actually to have subjects doing adaptation training^{51, 54}. It may have an impact on the subsequent analysis of VIMSL changes. What’s more, those who had ever been trained in VDS were excluded from the recruitment, as described in Table 1.

Data processing

The purpose of our study was to determine whether the EEG signal changes could be used as markers of a person’s VIMS onset in the VR environment. We hypothesized that if VIMS was induced by the perceptual conflicts of the self-motion while interpreting the motion signals from various sensory systems in the brain, the EEG signals between no-VIMS and VIMS states should make (at least) some differences, reflecting the brain’s conflicting state.

In this study, we chose the means and standard deviations of the GF, PSE, and KC of the EEG signals as potential marker candidates for VIMS. For each subject, those potential markers of EEG signals within no-VIMS and VIMS states were computed separately and then compared within a subject. Such pairwise comparisons were done for all subjects to see if there were any significant differences between the states. The increase or decrease of the means of the candidate markers may represent the overall amount of brain activity change, while the standard deviation changes may indicate the amount of brain activity disturbance due to the VIMS. Please note that the lengths of the EEG signals analyzed vary across participants (due to varied onsets and exit times). Computing markers within no-VIMS and VIMS states separately helps to eliminate the effects of variations of EEG signal length on the results. The remainder of this section describes the detailed methods for GF, PSE, and KC computations.

Gravity frequency (GF)

Gravity frequency reflects the transition of EEG power spectral density (PSD)⁴². It allows us to see the temporal changes in brain activity within a given frequency band. It was computed by^{42, 43}

$$GF=\frac{\sum _{f=f_1}^{f_2}(PSD(f)\cdot f)}{\sum _{f=f_1}^{f_2}PSD(f)},f_1\le f\le f_2$$ (1)

where f represents the frequency of the EEG signal, f₁ and f₂ represent the lowest and highest frequency of a given frequency band, and PSD(f) represents the power spectral density corresponding to a given EEG frequency, f.

Fig 3 shows the procedure of GF computation. Note that the PSD describes the power distribution of an EEG signal in the frequency domain for a given time period. A sliding time window of 3 minutes was empirically chosen for computing the PSD because it optimizes the trade-off between temporal resolution and computational complexity. For consistency, these raw data (FFTs) segments of 3 minutes were also used for PSE and KC computation. The PSD and GF were computed for each frequency band, i.e. delta (0–4 Hz), theta (4–8 Hz), alpha (8–12 Hz), beta (12–30 Hz) and gamma (30–50 Hz), of each electrode, i.e. FP1, FP2, TP9, and TP10.

Fig 3. Flowchart of the gravity frequency (GF) computation.

(a) The raw EEG data are segmented into 3-minute data segments. All the blue blocks (no-VIMS) and the red blocks (VIMS) in (a) are grouped by the states and their statistical features are calculated for comparisons of the two states. Data segmentation was done from the beginning of the experiment. This operation repeated until it meets the transitions of the action or the moments of change between no-VIMS and VIMS states. In general, segments less than three minutes are discarded. (b) The power spectral density (PSD) function for each data segment is computed for each electrode. Note that the PSD computation transforms each data segment in the temporal domain to the frequency domain. Therefore, we can separate the brain activity in each frequency band (alpha, beta, delta, …). (c) The representative “center of mass” frequency of a given frequency range, GF, for each frequency band is calculated using Eq. (1) for each electrode. Note that GF computation brings the brain activity of each frequency band back to the temporal domain so that we can monitor the frequency band wise activity monitoring in time. For the illustration purpose, only GFs of FP2 are shown in the bottom figure.

The differences of GFs between the paired electrodes (FP2-FP1 and TP10-TP9) were also computed for each frequency band since Miyazaki et al. suggested that asynchronous bilateral MT+ activation (i.e. between two hemispheric brain areas) could be a marker of VIMS⁶.

Power spectral entropy (PSE)

Power spectral entropy is a measure of complexity reflecting the disorder of time sequence signals and the level of irregularity of multi-frequency components signals⁴². The lower PSE is, the more uniform the signal energy distribution over the whole frequency band is⁴². Note that the PSE also has been shown as a sensitive parameter of brain activity classification featuring a good accuracy in brain-computer interaction (e.g. imaginary hand movements)^{55, 56}. It is very good for the measurement of nonlinear dynamic states, which requires a small amount of data⁵⁶. The previous study has shown that the PSE can be used as one of the features to distinguish different mental tasks (e.g., imagining that the left or right hand is moving)⁵⁶.

The Shannon entropy of the power spectrum of the signal can be defined as⁴²

$$PSE=-\sum _{f=f_1}^{f_2}p(f){\text{log}}_2(p(f))$$ (2)

where the probability of power occurrence for a given frequency, p(f), can be computed as follows:

$$p(f)=\frac{PSD(f)}{\sum _{f=f_1}^{f_2}PSD(f)},f_1\le f\le f_2$$ (3)

Unlike the GF computation, PSE was computed to monitor the overall brain activities over the full frequency range within a given time range. So in this study, we set f₁ = 0 Hz and f₂ = 50 Hz for Eq. (2) and (3), and PSE was computed for each electrode every 3-minute data segment.

Kolmogorov complexity (KC)

Kolmogorov complexity can also be used to quantify the complexity of EEG signals⁴⁴. Note that unlike the PSE, the KC measures the signal complexity directly from the time domain, not from the frequency domain. The KC has been used to measure the mental fatigue and showed encouraging results, where the KC of the EEG decreases as the mental fatigue increases (i.e. signal became less random when a person is in a mental fatigue state)⁵⁷.

KC computation consists of two steps: binary encoding and compression ratio computation. The temporal signals from each electrode were first encoded into a binary string (symbol sequence). A set of unique binary words which could be concatenated to describe the full string were identified, and then, the shortest length binary word sequence composed of the set of the unique binary words were computed. Finally, the ratio of the shortest length (compressed) of the binary word sequence and the binary encoded string length (uncompressed) was computed and used for a measure of the signal complexity⁵⁸. In other words, the KC is a maximum compression ratio of a signal when the signal is encoded into a binary code.

In our KC computation, the same 3-minute data segments used for GF and PSE computation were supplied to the encoding process. For each data segment, the raw EEG data were converted into a binary symbol sequence, x = <x₀, x₁, x₂, …, x_i, …, x_n−1> (0 ≤ i ≤ n − 1) using the following equation:

$$x_i=\{\begin{array}{ll}0\hfill & x_i<\overline{x}\hfill \\ 1\hfill & x_i\ge \overline{x}\hfill \end{array}$$ (4)

where

$$\overline{x}=\frac{1}{n}\sum _{i=0}^{n-1}{x}_i$$ (5)

For each data segment, the complexity of the symbol sequence x of length n, KC, was obtained by

$$KC=\frac{c(n)}{b(n)}$$ (6)

where c(n) is the length of word sequence after the compression of the binary encoded input length of n, b(n) reflects the length of word sequence before the compression, $b(n)=\underset{n\to \infty }{\text{lim}}c(n)\cong \frac{n}{{\text{log}}_{2}n}$^{44, 59}. Note that the KC varies within 0 and 1, where KC = 1 indicates the randomness of the signal reaching the maximum⁴⁵. Similar to PSE, KC was computed for each electrode every three minutes in our study.

Results and Discussions

Subjective VIMSL changes analysis

Let L_max be the highest VIMSL that the subject experienced, T_Total be the total driving duration, T_Occuring be the length of time from the start of the drive to the occurrence of VIMS (the driving duration needed for VIMSL reaching “1”), and T_Recovery be the recovery duration (the length of time from the end of the drive to the VIMSL coming back to “0”). Table 2 and Fig 4 show the distribution of those factors for eight subjects. From these table and figure, we know that the individual differences in tolerance, susceptibility, and recoverability to VIMS varied a lot.

Table 2.

Highest VIMSL that each subject experienced (L_max).

Subject	S1	S2	S3	S4	S5	S6	S7	S8
Lmax	4	4	2	2	4	3	3	2

Fig 4.

Comparison of the durations (T_Total, T_Occuring, and T_Recovery) between different subjects.

From Table 2 and Fig 4 we can find that:

1. The total driving duration varies a lot among the subjects. Generally, larger T_Total indicates higher VIMS tolerance. The T_Total for S2, S6, and S8 are larger than 30 min. These subjects showed higher VIMS tolerance.

2. The variation in T_Occuring indicates that each subject had a different VIMS susceptibility in our study. Generally, smaller T_Occuring indicates that subjects were more likely to get VIMS in a shorter time. The T_Occuring for S1 and S7 are no more than 5 min. They were sensitive to VIMS.

3. The recovery time for each subject also varied a lot. To a certain extent, smaller T_Recovery indicates faster VIMS recoverability. The T_Recovery for S3 is less than 3 min. It may suggest that S3 has a high VIMS recoverability. However, S3 only reached VIMSL of “2”, which may be S3 did not “push” enough until reaching the highest VIMSL. Therefore, we divided each subject’s T_Recovery by their max VIMSL (L_max) to be fair on comparing among the subjects.

First, we investigated the linear relationships between T_Total, T_Occuring, T_Recovery, and L_max. Pearson linear correlation coefficients (PLCC) between them are calculated and presented in Table 3. Table 3 shows no strong (∣r∣ ≤ 0.8) linear relationship between these variables.

Table 3. Pearson linear correlation coefficients between T_Total, T_Occuring, T_Recovery, and L_max.

Correlation coefficient ∣r∣ ∈ [0, 0.8] indicates a relative weak linear relationship here.

Variables	PLCC r
TTotal, TOccuring	0.78
TTotal, TRecovery	−0.10
TTotal, Lmax	−0.15
TOccuring, TRecovery	−0.21
TOccuring, Lmax	−0.11
TRecovery, Lmax	0.09

We applied multivariate nonlinear regression analysis to the factors (T_Total, T_Occuring, T_Recovery, and L_max). Table 4 shows the results of multivariate logistic regression. We can see that P ≥ 0.05 for all cases. This indicates that a multivariate logistic regression model is invalid.

Table 4.

Results of multivariate logistic regression.

Dependent variable	Independent variables	P
Lmax	TTotal, TOccuring, TRecovery	0.24
TRecovery	TTotal, TOccuring, Lmax	0.96
TOccuring	TTotal, TRecovery, Lmax	0.96
TTotal	TOccuring, TRecovery, Lmax	0.96

We also tried other models such as polynomial regression model. Again, P ≥ 0.05. This suggests that no functional relationship exists between T_Total, T_Occuring, T_Recovery, and L_max. As a result, we see the large individual differences in Table 2 and Fig 4. Please note that the T_Total, T_Occuring, T_Recovery, and L_max data came from eight subjects, and did not come from a repeated measures design.

Objective EEG data analysis

We computed the means and standard deviations of the GF, PSE, and KC measured in no-VIMS (VIMSL < 1) and VIMS (VIMSL >= 1) states for the eight subjects. These potential markers were compared between those two states to see if there were any significant differences. Before the analysis, we investigated whether the objective measures (GF, PSE, and KC) would vary simply as a function of time or not. If not, we can hypothesize that those markers with significant differences between the two states may be caused by VIMS. Fig 5 shows the changes of GF, PSE, and KC of Subject S1 (take the data from electrode FP2 for example). From Fig 5 we can see that there is no substantial variation for the potential EEG markers in the pre-driving segment. In this segment, no VIMS occurred for all subjects. This indicates that GF, PSE, and KC may not vary over time without any VIMS. It is reasonable for us to take the data of this segment as the baseline. Similar results can be obtained from the data form other subjects and electrodes.

Fig 5. Temporal changes of the GF, PSE, and KC throughout the experiment, take the data from electrode FP2 of S1 as an example.

(a) GF in five different frequency bands. (b) PSE. (c) KC.

In this section, scatterplots were used to show the distribution of the means (standard deviations) of GF for each frequency band for all channels and the bilateral differences (FP2-FP1 and TP10-TP9). For PSE and KC, only the plots for all channels and the bilateral differences were generated. Each dot in the plots represents a subject’s data. If there is a significant trend of increase or decrease due to the onset of VIMS, the majority of dots should be located above or below the diagonal line, respectively. A pairwise t-test was applied to find out whether the EEG markers were significantly different from no-VIMS state to VIMS state.

Gravity frequency (GF)

Fig 6 shows the distributions of all those mean GFs that have significant changes (P < 0.05) between no-VIMS and VIMS states: theta@FP1, alpha@TP9, alpha@FP2, alpha@TP10, and beta@FP1. The P-value is set by Bonferroni correction. As for these frequency bands and channels, the mean GFs in no-VIMS state are greater than those in VIMS state.

Fig 6. Comparison of the mean of GFs between no-VIMS state and VIMS state, which showed a significant difference (P < 0.05).

The horizontal axis represents the mean GFs [Hz] in no-VIMS state, while the vertical axis represents the mean GFs [Hz] in VIMS state. Each dot in the plots represents a subject’s data.

Similar analyses were carried out for the standard deviations of GFs for each frequency band for all channels. Fig 7 shows the results that turned out to be statistically significant (P < 0.05) between no-VIMS and VIMS states: alpha@TP9, alpha@FP1, alpha@FP2, alpha@TP10, and alpha@(FP2-FP1). In all cases in Fig 7, we can see that the standard deviations of GFs were increased in VIMS state, which indicates the decrease in the dispersion of the GFs during VIMS state. Note that all significant reduction of the dispersion was found in the alpha band.

Fig 7. Comparison of the standard deviation of GFs between no-VIMS and VIMS states, which showed significant differences.

The horizontal axis represents the standard deviation GFs [Hz] in no-VIMS state, while the vertical axis represents the standard deviation GFs [Hz] in VIMS state. Each dot in the plots represents a subject’s data.

It has also been found that these frequency bands have significant connections with the brain activity⁴². In addition, the EEG power spectrum could reflect fluctuations of the vigilant state⁴². The frequency bands that show significant changes (theta, alpha, and beta) are known to be associated with various mental conditions⁶⁰. For example, higher theta power is related to the increased activity in memory and attention processes while higher beta power is associated with the spatial localization processes, higher alpha power in the occipital lobe is associated with the level of relaxation⁶⁰.

In terms of VIMS, previous studies have shown that some of the EEG power-related measures will decrease after the onset of VIMS symptoms. Chen et al. argued that the overall decrease of the GF indicates the decline of subjects’ alertness level⁴², and especially, the alpha power suppression can be referred to the blocking or desynchronization of sensorimotor rhythms in parietal and the right and left motor areas of the brain, which might be influenced by vestibular inputs²⁹. Naqvi et al. believed that the drop in alpha power indicates the visual fatigue or discomfort caused by visual stimulus⁴¹. Our results are consistent with those of the above literature. We suspect the decrease of GFs magnitude and variability in VIMS state might indicate that mental activity was less sensitive.

We also noticed unusual signal noises in all bands and channels from time to time. We suspect that those might be caused by the poor connection between the electrodes and skin. However, since the GF worked as an average filter for the noises, the impact of signal noises in EEG data was automatically reduced in GF analysis.

Power spectral entropy (PSE)

As mentioned in the previous section, PSE is a sensitive brain activity classification parameter, reflecting the spectral structure of EEG signals⁵⁶. Similar to the analyses of GF, we computed the means and standard deviations of the PSE of the EEG signals measured in no-VIMS and VIMS states for four channels. Lower average PSE represents a uniform signal energy distribution over the whole frequency band, while lower standard deviations of PSE indicate a less disturbance or fluctuation in signal energy distribution.

Fig 8 shows the distribution of the standard deviation of PSE for FP1, which turned out to be the only measurement, showing statistically significant (t(7) = 2.36, P = 0.01) difference between no-VIMS and VIMS states. As can be seen, all data points are located below the diagonal line, indicating the standard deviations of PSE for FP1 in no-VIMS state were larger than that in VIMS state. We suspect that when VIMS occurred, the brain activity at FP1 was significantly suppressed (as also shown in GF analysis for alpha and theta wave of the FP1, Fig 7). As a result, the signal energy distribution turned uniform, the signal fluctuation decreased as well. No significant change of mean PSE was observed for VIMS onset.

Fig 8. Scatterplot of the standard deviation of PSE for FP1.

All data points are below the diagonal line, indicating the standard deviations of PSE for FP1 decreases when VIMS onset.

Kolmogorov complexity (KC)

Similar to the analyses of GF and PSE, we again computed the means and standard deviations of the KC in no-VIMS and VIMS states for each channel. As it can be seen in Fig 9, the means of KC for EEG signals collected from all electrodes, except from TP10, showed a significant decrease (all P < 0.05) with the onset of VIMS.

Fig 9.

Comparison of the means of KC for the signal channels (electrodes) which showed significant differences between no-VIMS and VIMS states.

Previous studies have shown that KC of the EEG signal is strongly correlative with mental fatigue^{44, 57}. They found that the values of KC decrease as mental fatigue increases^{44, 57}. In addition, Gao et al. found that the KC sharply drops shortly after the epileptic seizure. They showed that transient EEG signals associated with epileptic seizures contain less random components than normal background EEGs⁵⁸. All these studies showed that KC values would decrease when brain activity changes from normal to abnormal. Our results also supported similar trends such that a decrease of KC occurred with VIMS onset. We suspect that when VIMS occurred, brain activity was significantly suppressed and caused a decrease of KC.

Conclusions

The purpose of this pilot study was to analyze the relationship between the EEG and subjective VIMSL and find possible EEG markers for VIMS evaluation when VIMS was induced by the VR-based vehicle-driving simulator. An EEG device with only four electrodes was used to collect data. We focused on the changes of the mean and standard deviation of various EEG signal descriptors, and compared the values between no-VIMS and VIMS states in an attempt to differentiate whether a subject was in no-VIMS or VIMS state based on those signal descriptors.

Our studies suggest that:

1. The tolerance, susceptibility, and recoverability of a subject to VIMS were quite different from other subjects. It seems there was no functional relationship between each subject’s tolerance, susceptibility, recoverability, and VIMSL.

2. For the following frequency bands and channels of EEG, the means of GF decreased significantly in VIMS state: theta@FP1, alpha@TP9, alpha@FP2, alpha@TP10, and beta@FP1

3. For the following frequency bands and channels of EEG, the standard deviations of GF, which indicates the dispersion of the brain signal, decreased significantly in VIMS state: alpha@TP9, alpha@FP1, alpha@FP2, alpha@TP10, and alpha@(FP2-FP1).

4. A significant reduction of the standard deviation of PSE was observed at FP1 when VIMS occurred.

5. There was a significant reduction in the means of KC for TP9, FP1, and FP2 when VIMS onset.

6. The values of Cohen’s d for each of the statistically significant t-tests were large (no less than 0.80), indicating that the EEG markers are indeed strong indicators of VIMS.

7. The experiments demonstrate that an EEG device with a small number of electrodes (only four) is feasible to perform VIMS evaluation.

All the markers above showed a decrease after the occurrence of VIMS. This may represent our brain’s physiological response to VIMS onset. Note that the physiological markers presented by some other studies also showed decreases after the occurrence of VIMS^{17, 42}. They suspected that it may be caused by the decline of subjects’ alertness level⁴², or brain activity depressing^{44, 57}.

Although we found some significant differences between no-VIMS and VIMS states, it is hard to conclude that all these markers can be used for precise detection of VIMS onset or estimation of VIMS severity in real-time. This is because our analyses focused only on differences occurred between no-VIMS and VIMS states.

Furthermore, the findings of this study are not necessary and sufficient proofs that these physiological markers have a direct casual-effect relation with VIMS. That is to say, if somebody gets VIMS, we can deduce that his or her physiological markers will change significantly with a great probability; but we cannot conclude that he or she must get VIMS. This is because some of the VIMS-like symptoms occur in certain situations may be related to factors other than VIMS (such as alertness level or ‘various mental conditions’), or may not be specific or limited to VIMS, or even have nothing to do with VIMS⁶¹. The proposed EEG markers cannot be used as a “lie detector” to “demonstrate” that a person who says he/she is sick actually is not sick, or a person who says he/she is not sick actually is sick. Furthermore, we do not know the exact physiological mechanism of GF, PSE, and KC changes to the VIMSL. For example, Wei et al. have shown that changes in the alertness level of subjects were monotonically related to changes in the EEG power spectrum in the theta and alpha bands⁶²; Lin et al. have shown the improved behavioral performance was accompanied by concurrent power suppression in the theta and alpha bands in the occipital cortices⁶³. Thus, more evidence needs to confirm the causality of EEG markers changes in future work. However, since these markers can differentiate a person between no-VIMS and VIMS states, we expect that there may be some relationship between these markers and the VIMSL.

In addition, EEG signal changes might be not only induced by visual stimulation but also by driving activities in this study. It may be difficult to determine if the measured physiological difference is caused by the emotional or physical impact of the task (e.g. driving), or truly by the onset of VIMS. Therefore, our results should be verified further in a more controlled experimental design where only passive visual stimulation is provided.

Finally, to make the finding more useful, further analysis method should be developed to enable detection of VIMS onset and estimation of VIMSL level in real-time. Currently, we are working on designing a machine-learning approach to handle such a task. In addition, other physiological signals, such as head and eye movements, can be combined with the EEG to measure the VIMS in future studies. This multimodal approach will be tested in our future work.