Enhanced Multiple Instance Representation Using Time-Frequency Atoms in Motor Imagery Classification

Abstract

Selection of the time-window mainly affects the effectiveness of piecewise feature extraction procedures. We present an enhanced bag-of-patterns representation that allows capturing the higher-level structures of brain dynamics within a wide window range. So, we introduce augmented instance representations with extended window lengths for the short-time Common Spatial Pattern algorithm. Based on multiple-instance learning, the relevant bag-of-patterns are selected by a sparse regression to feed a bag classifier. The proposed higher-level structure representation promotes two contributions: (i) accuracy improvement of bi-conditional tasks, (ii) A better understanding of dynamic brain behavior through the learned sparse regression fits. Using a support vector machine classifier, the achieved performance on a public motor imagery dataset (left-hand and right-hand tasks) shows that the proposed framework performs very competitive results, providing robustness to the time variation of electroencephalography recordings and favoring the class separability.

1. Introduction

Motor Imagery (MI), recorded via electroencephalography (EEG), is the motor intention in the form of enhancement or attenuation of brain activity in μ and β rhythms over the sensorimotor cortex, having a different spatial distribution for each imagined movement (Liang et al., 2016). MI has particular applicability to neurophysical regulation and rehabilitation, unconscious motor preparation (Wang et al., 2019), games and entertainment, and sports training, among others (Suica et al., 2018). In education scenarios, for which the Media and Information Literacy methodology proposed by the UNESCO covers several competencies that are vital for people to be effectively engaged in all aspects of human development (Frau-Meigs, 2007). In this regard, the usage of modern technologies like motor imagery learning may provide tools for measuring the time response of brain activations during imagination of a determined movement, helping to assess the sensorimotor response-ability under very concrete learning stimuli (Guillot and Debarnot, 2019). Nonetheless, in characterizing these event-related potentials (ERP), a significant drawback is the intrinsic variations between individuals and within individual trials, making the EEG energy distribution very dependent on time, frequency, and spatial domains (Yamawaki et al., 2006). Due to this heterogeneity, therefore, finding spatial patterns of neural activation with maximal class separation is a challenging issue and may require a tiresome process (Padfield et al., 2019).

For bi-conditional tasks, Common Spatial Pattern (CSP) filtering remains the most widely used signal projection technique to extract discriminative EEG features, maximizing the variance of one condition while minimizing the alternative one. Still, the effectiveness of CSP is mainly affected by two aspects (Ang et al., 2012): Low robustness of selected channels across trials, and selection of the time-window for piecewise analysis that must cover the interval within a neural pattern is activated, but removing the unrelated sampling points (Miao et al., 2017). However, due to the subject-specific nature of MI tasks, the use of a single time window for a whole subject population diminishes the discriminating ability to extract CSP features and, hence, degrading the classification accuracy (Xu et al., 2015). To improve robustness at low Signal-to-Noise Ratio (SNR), optimal frequency band selection is a common approach that exploits the spectral relationship among the extracted MI features. Namely, sparse regression is applied to find compact representations within a supervised learning framework, determining the significant CSP features with class labels (Zhang et al., 2015). Nonetheless, the enhancement of sparse CSP-based systems should consider the design of an optimized dictionary matrix utilizing subject-adapted frequency-time patterns for increasing the classification accuracy (Shin et al., 2012).

For extracting feature dynamics, the search for adequate time-series similarity is given close attention, centering mainly on finding shape-based relationships. To this end, the point-to-point comparison is the most implemented algorithm, being efficient for short-time series or with periodic waveforms to extract local temporal and/or frequency information. In the case of long sequences, instead, it is more appropriate to consider similarities based on higher-level structures like the bag-of-patterns representations, capturing more accurately the structural dynamics to be fed into a classifier and without having to deal with the raw data (Passalis et al., 2017). Besides, despite ignoring the temporal order of local segments, theses representations enable capturing high-level structural information, relating either local and/or global relationships (Wang et al., 2013). To create bag-of-patterns representations (Chen et al., 2006), similarity distances between sets are used as well as similarity functions between bags and instances (Cheplygina et al., 2015), extracted from histograms (Lin and Li, 2009), codebooks (Gui and Yeh, 2014), or multivariate bag-of-words models (Bailly et al., 2016). Consequently, the construction of bag-of-patterns and selection of a similarity measure are the main aspects to implement a structural relationship between time series, reducing the similarity bias effect because of the presence of outliers in real-world applications and avoiding high dimensional feature vectors that may limit their application for large datasets (Ratanamahatana and Keogh, 2004).

With the aim of improving the distinction of brain dynamics in motor imagery tasks, we propose an enhanced bag-of-patterns representation that is developed through Multiple Instance Representation using Time-Frequency Atoms. In this case, instances are constructed by CSP feature sets extracted over short time windows of the filtered EEG signal. To explore the brain dynamics across time in more detail, augmented instance representations are introduced, which are obtained by the scale and shift operations over the original piecewise CSP feature extraction. Using a multiple-instance learning approach, the relevant bag-of-patterns are constructed and further selected by LASSO regression. From obtained weight vector by LASSO, the brain dynamic is modeled as the weighted appearance probability of each time instant along MI period, highlighting the more relevant time intervals associated with the response variable. The performance of proposed enhanced bag-of-patterns representation is assessed using a Support Vector Machine (SVM) classifier on a public dataset (left hand and right hand MI tasks), showing that the accomplished accuracy is very competitive, providing robustness to the time variation of EEG recordings and favoring the class separability.

The agenda is as follows: Firstly, section 2 describes the mathematical background of CSP feature extraction, optimized multi-instance learning representation, and pattern selection by LASSO regularization for MI classification. The developed experiments and achieved results are described in section 3, respectively. Finally, the discussion and concluding remarks are provided in section 4.

2. Materials and Methods

2.1. Motor Imagery Dataset 2a

This signal collection, publicly available at¹, was recorded from nine subjects using a 22-channel system (with inter-electrode distances of 3.5 cm), corresponding to the international 10−20 system. The EEG trials were recorded at sampling frequency 250 Hz and regarded one of four MI tasks: left hand, right hand, both feet, and tongue. All recordings were performed in six runs separated by short breaks so that each run held 48 trials (each one lasting 7 s). A short beep indicated the trial start, after which a fixation cross appeared on the black screen within the first 2 s. Then, an arrow (or cue) was shown during 1.25 s to indicate the left, right, up, or down directions, stimulating to imagine a left hand, right hand, both feet, or tongue movement, respectively. Next, each subject performed one MI task within the time interval from 3.25 to 6 s, while waiting for the cross to reappear again. The EEG data were labeled (l∈{+1, −1}, left hand or right hand, respectively) and the artifacts removed.

For preprocessing, the raw EEG data are band-pass filtered using N_f overlapped bandwidths, resulting in a matrix $X_{rf}^l={\left[x_{rf}^c:f\in N_f,r\in R\right]}^T$, where the vector $x_{rf}^c\in {ℝ}^{\top }$ denotes each filtered channel c per bandwidth f and trial r. Specifically, we use R = 144 and N_f = 17 five-order overlapped Butterworth filters with bandpass frequencies between 4 and 40 Hz, having a bandwidth of 4 Hz and overlapping rate of 2 Hz as in Zhang et al. (2015). Then, the attained filter-banked signals are time-windowed onto N_t intervals with a 90% overlap of samples, each one lasting t.

As a result, the processing stage provides an input time-frequency matrix $X_{rft}^l={\left[x_{rft}^c:t\in N_t\right]}^{\top }$ per trial, where $x_{rft}^c\in {ℝ}^T$ represents the time-segmented neural activity related to motor imagery within a concrete bandwidth, measured at each channel, lasting T s.

2.2. Construction of Time-Frequency CSP-Based Atoms

Given a labeled trial matrix $X_{rft}^l\in {ℝ}^{T\times C},$ the algorithm of Common Spatial Patterns aims at finding, within each channel c⊂C, the linear transformation vector w ∈ ℝ^C to maximize the Rayleigh Quotient (RQ, noted as J ∈ℝ⁺) that is computed through the mapped data variance between classes as follows:

$$\begin{array}{r}{w}^{*}={\displaystyle \underset{\forall w}{\text{max}}}J(w)=\frac{w^{\top }{\Sigma }^{(+1)}w}{w^{\top }{\Sigma }^{(-1)}w},\mathrm{\text{s}}.\text{t}.:||w|{|}_2=1\end{array}$$ (1)

where ${\Sigma }^{(l)}=E\left\{X_{rft}^l{X}_{rft}^{l\top }:\forall r\in R\right\}$, with Σ^(l) ∈ℝ^C×C, is the simplest estimate of class data variance computed at a frequency f. Notations ||·||_p and E{·:∀r} stand for ℓ_p-norm and expectation operator across variable r, respectively.

Further, the unlabeled EEG sample X_rft is filtered through the learned spatial matrix W ∈ ℝ^C′ × C that holds all C′ piecewise transformation components. Thus, at each time sample t ∈ N_t, the projected data Z_rft = WX_rft, with $Z_{rft}\in {ℝ}^{C^{\prime }\times T},$ is computed by only 2M representative terms of C′ (namely, M first and M last rows), yielding the extracted feature vector d_rt that is calculated across the spectral domain as below:

$$\begin{array}{l}{d}_{rt}=ln(\text{diag}(\mathrm{\text{var}}\{Z_{rft}\})/tr\{Z_{r{f}_{1}t}\})||\cdots \\ ||ln(\text{diag}(\mathrm{\text{var}}\{Z_{r{f}_{N_f}t}\})/tr\{Z_{r{f}_{N_f}t}\})\end{array}$$ (2)

where var{·} and tr{·} denote the variance and trace operator, respectively.

Using the extracted feature from each trial r and overlapped time window t, therefore, we obtain the time-frequency feature array $D_r\in {ℝ}^{2M{N}_f\times N_t}$ (N_t < 2MN_f) with vector columns $d_{rt}\in {ℝ}^{2M{N}_f},$ hereafter we will term as time-frequency atoms.

2.3. Optimization of t-f Atoms for MIL Representation

Relying on the above introduced atoms, we define each column vector d_rt as an instance within a Multi-Instance Learning (MIL) framework, and in turn, we assemble each r-th bag per trial by arranging the whole column in a vector $B_r=\left[\begin{array}{c}{d}_{r1}^{\top }\\ \dots \\ d_{r{N}_t}^{\top }\end{array}\right]$with $B_r\in {ℝ}^{N_t\times 2M{N}_f}$. So, the whole training set, extracted over all trials, is denoted as $\mathcal{B}=\left\{B_r:\forall r\in 2R\right\}$.

To put into effect the concept that makes a bag label either positive or negative, the instance-based embedding is intended to map each bag into a vector space. To this end, we specify each k concept class, noted as d^k, through the single point concept class, assuming as a target concept each instance extracted in Equation (2), i.e., $d^k\subset \mathcal{D}=\left\{d_{rt}:\forall r,t\right\}$.

For improving the target concept set, the CSP feature set extracted by Equation (2) is further refined within each instance in terms of contributing to distinguishing between the labels. In particular, we search the relevant information from each EEG channel through a sparse-based operator over the CSP feature space calculated at each time window. As suggested in Caicedo-Acosta et al. (2019), the Multiple-Instance Logistic Regression estimates a bag label by aggregating each instance label through the following Bernoulli-based logistic regression ${\tilde{l}}_r=\mathcal{I}(\sum _{t\in N_t}Bernoulli(h({\beta }_0+d_{rt}^{\top }\beta ))>0)$, where $\mathcal{I}(x)=1$ if x>0, otherwise, $\mathcal{I}(x)=0$, being h(x) = 1/(1+e^−x) the sigmoid units.

In particular, the parameter set (one bias term β₀ ∈ ℝ and $\beta \in {ℝ}^{2M{N}_f}$) is computed within the following quadratic optimization framework (Chen et al., 2017):

$$\begin{array}{l}{\displaystyle \underset{{\beta }_0,\beta }{\text{min}}}(-{\displaystyle \sum _{r\in 2R,t\in N_t}}{l}_r{\gamma }_{rt}^i({\beta }_0+d_{rt}^{\top }\beta )\\ -\text{log}(1+\text{exp}({\beta }_0+d_{rt}^{\top }\beta ))+\alpha {\displaystyle \sum _{q\in 2M{N}_f}}|{\beta }_q|)\end{array}$$ (3)

where the conditional expectation (given l_r = 1) is computed as ${\gamma }_{rt}=h({\beta }_0+d_{rt}^{\top }\beta )/(1-{\prod }_{t=1}^{N_t}1-h({\beta }_0+d_{rt}^{\top }\beta ))$ and α ∈ ℝ⁺ is a regularization term. Here, we solve the above optimization problem through an iterative coordinate descent algorithm.

As a consequence, by applying the optimizing procedure on MIL representation, we select the t-f atom set with improved properties, noted as ${\hat{d}}_{rt}$, since it holds in Equation (3) that the larger the magnitude of β_q – the more relevant the feature to predict the instance labels. Therefore, the performed atom-based optimization enhances the MIL representation, allowing discarding irrelevant or redundant information in discrimination MI tasks.

2.4. Similarity-Based Enhancement of Expanded MIL Representation

With the aim of implementing the concept introduced before, the instance-based embedding is intended to map each bag into a vector space. As proposed in Chen et al. (2006), the feature vector $p_r\in ℝ{\left[0,1\right]}^{2R}$ extracted from each bag is computed by the conditional probability that k instance belongs to r-bag, $P({\hat{d}}^k|B_r)$, as follows:

$$\begin{array}{r}{p}_r=\left[P({\hat{d}}^k|B_r):\forall {\hat{d}}^k\in \mathcal{D},k\in 2R{N}_t\right].\end{array}$$ (4)

Further, to implement the higher-level time-frequency representation, we accept the mapping of instance-based features with both multiple concepts, either positive or negative, which are extracted from the training dataset, $\mathcal{D}\subset \mathcal{B}$, so that the conditional probabilities in Equation (4) can be approximated as:

$$\begin{array}{r}s({\hat{d}}^k,B_r)=P({\hat{d}}^k|B_r)\propto {\displaystyle \underset{\forall t\in N_t}{\text{max}}}\text{exp}(-\left|\right|{\hat{d}}_{rt}-{\hat{d}}^k|{|}^2/{\sigma }^2)\end{array}$$ (5)

where $s({\hat{d}}^k,B_r)\in ℝ\left[0,1\right]$ is the assessed measure of similarity between concept ${\hat{d}}^k$ and bag B_r, while σ ∈ ℝ⁺ is the bandwidth of exponential square function. Using the measure in Equation (5), therefore, we perform the similarity matrix $S\in ℝ{\left[0,1\right]}^{2R{N}_t\times 2R}$ with elements $s({\hat{d}}^k,B_r),$ ∀k ∈ 2RN_t, ∀r∈ 2R.

Intending to include the mutual influence of other atoms extracted at different time instants, we expand further each similarity representation in Equation (5) by a time-window scale, Δτ, yielding the following enhanced similarity matrix:

$$\begin{array}{r}{\Delta }_{pq}=[S_{p\Delta \tau }||\cdots ||S_{N_{\tau }\Delta \tau }:\forall p,q],\end{array}$$ (6)

where $S_{\tau }=[s({\hat{d}}_{\tau }^k,B_{r,\tau }):\forall r,k]$ is an version of S with elements expanded by a multiple scaling factor τ = pΔτ, p = [q, N_τ], q = [2, N_τ], being N_τ the maximal number of considered time-window lengths τ.

Further, as a powerful preprocessing step for high-dimensional data analysis, feature selection is carried out aiming at discarding irrelevant and redundant features from the expanded similarity representation ${\Delta }_{pq}\in ℝ{\left[0,1\right]}^{2R\times (N_t(N_{\tau }-q))}$ using the LASSO sparse regression model that has been shown to be efficient in MI tasks within the following optimizing framework (Feng et al., 2019):

$$\begin{array}{r}{u}^{*}=arg{\displaystyle \underset{u}{\text{min}}}||{\Delta }_{pq}u-l|{|}_2^2+\lambda ||u|{|}_1\end{array}$$ (7)

where l ∈ ℝ^2R is a vector with the class labels, $u\in {ℝ}^{N_t(N_{\tau }-q)}$ is a sparse vector to be learned, λ ∈ ℝ⁺ is a positive regularization parameter for controlling the sparsity of u. Lastly, a SVM-based algorithm is implemented on the reduced subset of selected features ${\hat{\Delta }}_{pq}$ to train the bag classifier. Algorithm 1 summarizes the similarity approach for MIL representation developed in this section.

Algorithm 1.

Expanded similarity of MIL representation

Ensure:  Optimal bag formed by atom combination from instances having a window-size vector ${\hat{\Delta }}_{pq}$

1:  Δpq = ϕ

2:  for q = [τinit, Nτ] do

3:        for p = [q, Nτ] do

4:              $S_{\tau }=[s({\hat{d}}_{\tau }^k,B_{r,\tau }):\forall r,k]$         Calculate similarity matrix

5:              Δpq ← SpΔτ

6:              ${\tilde{l}}_i^{pq}=\mathrm{\text{SVM}}({\Delta }_{pq})$

7:  ${\hat{\Delta }}_{pq}={\mathrm{\text{arg max}}}_{\forall pq}E\left\{l_i-{\tilde{l}}_i^{pq}E\right\}$

3. Experimental Evaluation

As illustrated in Figure 1, the proposed enhancement of bag-level representation employs a CSP-based feature set, and appraises the following procedures: (i) Frequency-temporal decomposition of EEG raw data; (ii) Feature extraction of t-f atoms through a CSP approach, (iii) Optimization of t-f atoms by Multiple-Instance Logistic Regression; (iv) Extension of similarity assessments for MIL representation over a wider domain; and (v) Feature selection performed by LASSO strategy, feeding a classifier of MI bi-class tasks. Intending to evaluate the contribution of atom optimization, we consider two scenarios of training: including this stage and without it (gray line in Figure 1).

Figure 1.

Scheme of bag-of-patterns representation proposed for classification of bi-class motor imagery tasks. Within the MIL framework using t-f atoms, the suggested improvement is remarked by a dashed box.

3.1. Experimental Set-Up

The experiment is performed for each subject, for which approximately 138 trials per subject are presented. For validation purposes, the feature set is extracted from EEG data using the algorithm of Common Spatial Patterns, for which the filter dimension is adjusted to M = 3, corresponding to the more representative terms of C′ as suggested in Blankertz et al. (2008). Besides, we employ classification accuracy as the performance criterion that is estimated by a support vector machine algorithm through a 10-fold cross-validation scheme. In the proposed expanded representation stage, the investigated range of τ is adjusted to discrete values [0.2−2.0] s to embrace the whole motor imagery period, while the slicing Δτ is empirically adjusted to 100 ms. In Table 1, we report the total number of instances vectors according to τ value. Moreover, in classification stage, the SVM kernel bandwidth and LASSO regularization coefficients were fixed through an exhaustive search within a range of [10⁻³, 10⁻², 10⁻¹, 10⁰, 10¹, 10², 10³] and [10⁻¹⁰−9⁻¹] in a logarithmic scale, respectively.

Table 1.

Amount of instances performed by each tested time-window.

τ[s]	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0	1.1	1.2	1.3	1.4	1.5	1.6	1.7	1.8	1.9	2.0
Nt	91	54	41	29	24	19	16	12	11	9	7	6	5	4	3	2	2	1	1
ΣNt	91	145	185	215	239	258	274	286	297	306	313	319	324	328	331	333	335	336	337

The last row reports the cumulative sum across all range of considered τ.

3.2. Computation of CPS-Based t-f Atoms

Table 2 displays the optimal time window (heuristically determined), performing the highest accuracy for each subject as well as for the whole set. As a result, the optimal span of discrete τ ranges extensively from 0.2 till 2.0 s, implying that each subject differently rules all changes of J and, therefore, posing a difficulty in performing the group analysis across the whole subject set. Note that there is no dependence between the window length and achieved accuracy.

Table 2.

The highest accuracy scores performed by each subject by fixing the optimal time window (τ^*), and across the whole subject set (fixing τ = 2 s).

	A08T	A09T	A03T	A01T	A05T	A07T	A06T	A04T	A02T	Average	Whole set
τ	0.3	1.0	1.5	2.0	1.5	2.0	0.2	1.5	0.2		1.5
$a_{{\tau }^{*}}$	97.7 ± 3.2	100.0 ± 0.0	98.9 ± 2.5	93.7 ± 5.3	90.6 ± 4.9	94.8 ± 6.1	73.2 ± 7.0	65.3 ± 6.5	63.6 ± 5.4	86.4 ± 4.5	89.4 ± 4.7
aτ = 2	95.8 ± 3.3	97.3 ± 3.8	98.0 ± 2.6	93.7 ± 5.3	88.8 ± 4.6	94.8 ± 6.1	67.6 ± 16.4	64.0 ± 9.8	60.3 ± 10.6	84.5 ± 6.9	88.0 ± 5.2

For the sake of illustration, Figure 2 represents the CSP cost function value J (i.e., RQ value) that is estimated over frequency domain at each optimal time window τ. As seen in the left plot, the subject marked as A08 performs the best at a small window τ = 0.3 s, meaning that the rapid dynamics are relevant in differentiating between classes. Instead, the worst case (A02) demands τ = 0.2 s, that is, the relevant dynamics are also fast, but widely scattered over time as observed in the center plot. Therefore, the choice of time window remains crucial to characterize the changes in neural activity through the baseline atom representation.

Figure 2.

Estimation of J using the optimal time window τ. (Left) the subject A08 (achieving the best accuracy), (Center) the patient A02 (worst accuracy), and (Right) the group analysis performed at the admitted value τ = 2 s for validating the tested MI Dataset 2a. Spectral relevance is colored in gray bars.

Another aspect is the spectral relevance that is assessed as the marginal values of J on the frequency bands (bars painted in gray color). Even that left and center plots in Figure 2, indicate that the optimal frequency bands are subject-specific, the right plot shows the Rayleigh quotient of the whole subject set is mostly localized within μ (8–13 Hz) and β (13–25 Hz) rhythms as widely-accepted. The applied group analysis performs concatenation of all single-subject data into a single group array, i.e., ${\bigcup }_{\forall m}{X}_m^l$ from which a latent structure of sources is computed (Padilla-Buritica et al., 2019).

3.3. Performance of MIL Representation Using t-f Atoms

Using the enhanced representation in Equation (6), we validate the present proposal in discriminating against the bi-task MI dynamics. Thus, using the most-likely-cause estimator in Equation (5) as the similarity metric, we investigate the influence of τ (the relationship window between time series) on the produced t-f dynamics that are the most relevant in discriminating between classes.

Figure 3 display the accuracy achieved by estimating the higher-level structure similarity for each one of the available combinations of atom-based instances of time-frequency representation, that is, S_τ, and Δ_pq, for which the graphical meaning is drawn to get a better understanding of the proposed dictionary expansion. The red-box represents the expansion that achieves the best performance in each subject. Due to the symmetry of matrix accuracy in τ, only its upper part is reported and ranked in decreasing order of accuracy achieved by each subject. Namely, the x axis represents the considered time-window sizes, while y axis shows all possible expansions, that is, the different combinations of time-window sizes present in an expanded bag. Thus, the diagonal of the matrix represents the bags composed of instances formed from a single-time-window size. Besides, the last row of the image represents the performance of the three subjects with the lowest performance using the bag optimization stage.

Figure 3.

Accuracy performed at different window length combinations of atom-based instances. The last row beneath the dotted line displays the subject performance with a lower accuracy (A06T, A04T, and A02T) after using the optimization of the atom-based MIL representation stage.

In terms of distinguishing between MI tasks, the measured similarity matrix values allow extracting the relevant dynamics from expanded MIL representations, facilitating an accuracy enhancement over a wide range of τ. Further, we rely on the LASSO fits estimated by the feature selection task in Equation (7), which increases the model interpretability by eliminating irrelevant variables that are not associated with the response variable and this way also reducing the overfitting (Roth, 2004; Fonti and Belitser, 2017). Namely, besides information about feature relevance, some indication is given about the degree up to which a feature is relevant or can be replaced by others. Nevertheless, the solutions tend to be not consistent estimations of the underlying “true” weight vector u, regarding its exact value as quoted in Pfannschmidt et al. (2019). As shown in the Figure 4, therefore, we compute the normalized absolute LASSO weights at each time instant, but including the estimation performed for 10 cross-validated folds. It is worth noting that computation of LASSO fits directly through the Rayleigh quotient in Figure 2 does not provide an understandable representation of brain dynamics at each time instant since the optimal vector u^* holds the extracted CSP features, but contributing across the whole MI period. Instead, the learned sparse vector from the bag-of-patterns representations reveals a dynamic behavior that somehow resembles an elicited ERP waveform, rising at the beginning and declining in the closing periods.

Figure 4.

Temporal dynamics from the absolute LASSO weights performed within the motor imagery period. Each time series is a cross-validated fold. The last row displays the subjects with lower accuracy after the optimization of the atom-based MIL representation stage.

For subjects A08T, A09T, and A03T, the measured similarity values enable an accuracy that is high practically regardless of the examined window of atom-based instances, though each subject has a distinct dynamic behavior. In the case of subjects A01T, A05T, and A07T, their performance accuracy is a bit reduced, while the time range of optimal windows also shrinks. Moreover, the relevant dynamics of enhanced MIL representation are extracted by larger values of τ (close to 2 s) so that the LASSO fits remain constant over a large window of bag representation instances. In the last triad of subjects (A06T, A04T, and A02T), the relevant dynamics are scattered all over the range of considered short-time window τ, but appearing in a noticeably worse accuracy. Note that the dynamic learned by the sparse vector of A02T has low values. The last row in Figure 4 represent the rise of accuracy performed by optimizing the similarity of enhanced MIL representation through the procedure in section 2.4. As a result, the LASSO fits may increase prominently and remain constant over extensive window lengths.

To evaluate the influence of the used optimization procedure, we perform the pairwise similarity between subjects across the trial set. The left plot of Figure 5 presents the case when optimization is omitted, showing an apparent clustering of three groups of patients. However, the last triad of patients, achieving the worst accuracy, turns out to be very similar to the best triad, resulting in an inconsistency. In turn, the right plot degrades the similarity of the worst triad between subjects, as well as with other patients. Therefore, we may hypothesize that the use of Multiple-Instance Logistic Regression allows for improving the performed accuracy by better extracting the discriminating dynamics of MIL representations.

Figure 5.

Pairwise similarity between subjects across the trial set assessed when omitting the optimization of t-f atoms (Left) and using the Multiple-Instance Logistic Regression (Right).

For the sake of comparison, we contrast the accuracy performance achieved the proposed enhancement of bag-level representation using a CSP-based feature set against the following three state-of-the-art approaches based on Filter-Bank preprocessing and CSP-based feature extraction procedure: (i) Sparse Filter Bank Common Spatial Patterns (SFBCSP) (Zhang et al., 2015) used as baseline approach, (ii) Spatial-Frequency-Temporal Optimized Feature Sparse Representation-based Classification(SFTOFSRC) (Miao et al., 2017), which includes a time-decomposition stage in the data preprocessing to generate a column vector of extracted CSP features for sparse representation-based classification, and (iii) Temporally Constrained Sparse Group Spatial Patterns(TSGSP) (Zhang et al., 2018) that adding a LASSO-based regularization term in the time domain. Table 3 displays the accuracy of the compared LASSO-based CSP algorithms reported for each subject, showing that all of them are outperformed by the proposed enhancement of bag-level representation using an SVM instance classifier, at least, in terms of the average across the whole subject set. Furthermore, by optimizing the higher-level structure similarity of bag-based representation, the performed accuracy increases further, improving most of the subjects that perform a low signal-to-noise ratio. Lastly, we evaluate the significance in terms of the disagreement of performing individual accuracy between the proposal and each of the comparison methods. To this end, the paired t-test is conducted at a fixed confidence value of p < 0.1, employing the scores achieved on the cross-validation folds (this information is not available for TSGSP). As seen in Table 3, there are confident differences of performance with the proposal (underlined subjects) for most of the individuals using SFTOFSRC. However, using SFBCSP, only a few subjects achieve a distinctive accuracy, probably because of the obtained higher dispersion of that approach.

Table 3.

Comparison of SVM accuracy achieved by the proposed bag-level representation.

Subject	TSGSP	SFBCSP	SFTOFSRC	Proposal	Proposal*
A08T	95.8	97.0 ± 2.9	96.9 ± 3.4	99.0 ± 1.7	99.2 ± 1.6
A09T	81.3	97.8 ± 3.1	94.6 ± 3.4	98.7 ± 2.1	97.3 ± 3.2
A03T	93.8	98.8 ± 1.7	98.5 ± 1.9	97.7 ± 2.6	99.2 ± 1.6
A01T	87.0	91.8 ± 4.7	91.8 ± 3.9	96.0 ± 1.1	94.8 ± 3.5
A05T	90.4	90.6 ± 3.7	95.7± 2.1	93.0 ± 3.1	95.3 ± 4.4
A07T	91.4	94.7 ± 6.1	76.3 ± 5.7	88.0 ± 4.5	96.2 ± 4.6
A06T	63.9	67.9 ± 6.9	71.0 ± 6.4	74.5 ± 4.8	72.4 ± 7.9
A04T	74.3	63.5 ± 10.6	69.0 ± 7.1	70.3 ± 6.8	69.5 ± 9.3
A02T	64.7	58.4 ± 8.3	62.8 ± 5.9	64.0 ± 5.9	66.2 ± 5.3
Average	82.5	84.5 ± 5.3	84.0 ± 4.4	86.8 ± 3.5	87.8 ± 4.6

Each method performing the best individual accuracy is marked in bold. Abbreviation Proposal denotes the enhanced representation without optimizing the t-f atoms, while notation * includes this procedure. Each underlined subject achieves confident differences of performance with either proposal version.

4. Discussion and Concluding Remarks

Intending to extract patterns of the brain activity that allow improving the class separation, we propose the enhancement of bag-level representation using a CSP-based feature set. For this purpose, we exploit the baseline short-time CSP feature extraction, introducing an expanded atom-based MIL representation, covering more extensive window lengths. The obtained results in a public dataset prove that the accomplished accuracy is very competitive, providing robustness to the time variation of EEG recordings and favoring the class separability. Nevertheless, the following aspects for implementing our proposal are to be mentioned:

• – Construction of CSP-based t-f atoms: As seen in Table 2 and Figure 2, there is not a clear relationship between the window length and achieved accuracy. These facts indicate that a frequency band decomposition of multi-channel EEG combined with an accurate time segmentation is necessary to improve the MI classification accuracy. The alone optimization of frequency bands without considering the influence of time may fail in finding the optimal feature set for classification (Miao et al., 2017). In addition, the obtained results show that the performance of the enhanced bag-level representation is influenced by the CSP-based representation, demanding high values of SNR for the acquired EEG data. This fact is evidenced enough by the subject A02T, whose performance is seriously diminished because of the low SNR.

• – Atom-based MIL representation: With the aim to improve the contribution of time-frequency CSP-based atoms, we perform the Multiple-Instance Logistic Regression, that allows discarding irrelevant or redundant information in discrimination MI tasks. As a result, the performed accuracy importantly increases, as shown for the subjects achieving the worse outcomes without optimization. This improvement may be explained due to the LASSO fits increase prominently and remain constant over extensive window lengths.

• – Accurate similarity of expanded MIL representation: The benefit of mapping the CSP feature space to an expanded bag representation is to disclose the data distribution, capturing/encoding more meaningful information (local and global).

• – Computational burden: As regards the complexity of developed learning algorithms, however, it is so high that real-time processing during training does not apply. The learning procedure includes an exhaustive analysis within the parameter setting stage, which holds approximately 10N_τ≈190 possible expanding configurations, where N_τ is the number of considered time windows τ. Table 4 summarizes the algorithmic complexity of compared approaches. As a result, in comparison with the baseline algorithm with no Multiple-Instance Learning, the computational burden of training procedures, for the tested database, rises as much as 1,500 times! Nevertheless, once the corresponding training procedures are performed, the validating classification algorithms of developed approaches may be suitable for real-time processing.

Table 4.

Algorithm complexity of developed learning procedures.

	Proposal	Proposal*
Time [h]a	36	36Nλ
Complexity	$\mathcal{O}(n^2)$	$\mathcal{O}(N_{\lambda }{n}^2)$

N_τ, number of considered time windows τ; N_λ, amount of searching cycles in LASSO regularization terms.

^aIndicated time per subject.

Consequently, the proposed enhanced bag-of-patterns representation promotes in motor imagery the following two contributions:

• – Accuracy improvement of bi-conditional tasks. The effectiveness of conventional CSP extraction is very affected by the time window of EEG segments due to the significant inter- and intra-subject variation. To cope with this issue, we propose to build representations based om bag-of-patterns using the expanded atom-based instances extracted from the spectral-temporal Rayleigh quotient. As a result, the designed dictionary using the higher-level structures allows capturing the structural dynamics of EEG data more carefully over a wide range of τ. Therefore, it increases the classification accuracy, outperforming the baseline sparse CSP-based systems reported in the literature.

• – Better understanding of dynamic brain behavior. A better understanding of dynamic brain-behavior through the learned LASSO fits. In the designed dictionary of higher-level structures, the model interpretability is increased since the sparse feature selection eliminates irrelevant variables, which are not associated with the response variable. Thus, the learned sparse vector from the bag-of-pattern representations reveals a dynamic behavior that somehow resembles an elicited ERP waveform, rising in the beginning and declining in the closing periods.

However, the Multiple-instance learning algorithms often provide a large number of redundant or irrelevant features, which limits their application for large datasets. Intending to optimize the bag-of-patterns representations, we include a multiple instance Regularization with LASSO penalty and an embedded feature selection that improves further the performed accuracy, increasing the subject performance with a low SNR.

As future work, the authors intend to work out two main issues: computational burden and robustness. To address the former concern, we plan to introduce an instance selection stage, relying on a filter-type measure of performance, like the Rayleigh coefficient. In the latter case, to improve the robustness across trials, the authors are exploring more powerful representations based on bag-of-patterns, using the disgregation/selection of filter-banked components and testing other distances between high-level structures of time series. Further, the computational complexity must be minimized, encouraging validation of the proposed approach on more extensive EEG databases with a higher number of electrodes, multiple labels, and larger populations. Also, we plan to include other motor imagery tasks, extending the present methodology to the multi-class case and applying a set of binary CSP subproblems.

Data Availability Statement

Publicly available datasets were analyzed in this study. This data can be found here: http://www.bbci.de/competition/iv/#download.

Author Contributions

DC-H, JC-A, GC-D, and CA-M conceived of the presented idea. DC-H and JC-A developed the theory based on Multiple Instance Learning Representation and performed the computations. GC-D and CA-M verified the analytical methods. DC-H and GC-D encouraged. JC-A to investigate the influence of the selection of time-window in the feature extraction procedures, and supervised the findings of this work. All authors discussed the results and contributed to the final manuscript.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer FG declared a shared affiliation, with no collaboration, with the authors to the handling editor.