EEG-based texture roughness classification in active tactile exploration with invariant representation learning networks

Abstract

During daily activities, humans use their hands to grasp surrounding objects and perceive sensory information which are also employed for perceptual and motor goals. Multiple cortical brain regions are known to be responsible for sensory recognition, perception and motor execution during sensorimotor processing. While various research studies particularly focus on the domain of human sensorimotor control, the relation and processing between motor execution and sensory processing is not yet fully understood. Main goal of our work is to discriminate textured surfaces varying in their roughness levels during active tactile exploration using simultaneously recorded electroencephalogram (EEG) data, while minimizing the variance of distinct motor exploration movement patterns. We perform an experimental study with eight healthy participants who were instructed to use the tip of their dominant hand index finger while rubbing or tapping three different textured surfaces with varying levels of roughness. We use an adversarial invariant representation learning neural network architecture that performs EEG-based classification of different textured surfaces, while simultaneously minimizing the discriminability of motor movement conditions (i.e., rub or tap). Results show that the proposed approach can discriminate between three different textured surfaces with accuracies up to 70%, while suppressing movement related variability from learned representations.

1. Introduction

Active tactile exploration refers to the process of exploring surrounding objects to retrieve sensory information [1]. Dynamic active movement between human skin and an object’s surface may occur during this process. Active tactile exploration involves several parts of the human body which are characterized by high density sensory receptors, high sensory acuity and large sensory and motor cortical representation [2]. The high density mechanoreceptors on the human skin contribute significantly to the discrimination between different objects varying in pattern, texture and shape [3,4]. Mechanoreceptors in the epidermis and dermis layers of the human fingertips include distinct small, large, slow and rapidly adapting receptive fields which enable high spatial resolution [3,4]. Active tactile object exploration generates simultaneous cutaneous and proprioceptive feedbacks, whose combination is referred as haptic feedback. Proprioceptive signals originate from joint, muscle and skin mechanoreceptors, and are related to joint movement and position [2].

The primary somatosensory cortex (S1) is known to be the main brain region responsible for processing sensory information related to active exploration of objects [3]. This region is also known to have a direct association with the primary motor cortex, which is also involved in processing of active exploratory motor movements. During active tactile exploration, S1 cortex demonstrates neural activity that reflects the activation of several peripheral mechanoreceptors, as well as joint movements [5]. Moreover, movements associated with tactile exploration modulates the transmission of the tactile input to S1 cortex, adding further complications to the processing of somatic sensory signals [5]. It has also been shown that a vast majority of neural activity in S1 cortex significantly contributes to the processing of tactile input during active tactile exploration compared to passive exploration [6,7]. Accordingly, active tactile exploration provides substantial information about the surface properties and demonstrate better discrimination thresholds compared to passive exploration [1,2,8].

Sensorimotor processing is generally visualized as a sequence of function units that are sequentially activated that initiates from sensory input and proceeds until motor execution [9–11]. Several other research studies on human brain function suggest that motor and sensory processing are coupled, where motor actions and perception are in affect with each other [12–14]. A study by Melink et al. [15], investigated these two views of sensorimotor processing (i.e., the classical view and the alternative coupling perspective). Their electroencephalography (EEG) based exploratory study supports the alternative sensorimotor processing hypothesis that sensory perception and motor responses are coupled.

Research studies investigating EEG-based cortical activity in response to active tactile exploration are generally explored within passive touch experimental designs [16–18]. To the best of our knowledge, there exists only a few studies which explores the relation between recorded EEG and the roughness of the explored surface in active touch [19,20]. Yet, EEG-based classification of textured surfaces that vary in their roughness levels, while minimizing the potential influence of motor movement type during active tactile exploration remains to be explored.

In this study, we aim to classify different textured surfaces during active tactile exploration that vary in the roughness level on a single trial basis using simultaneously recorded EEG data. Our experimental study design includes two motor movement conditions, rubbing or tapping the textured surface, and three different levels of surface roughness (i.e., smooth, medium rough and rough). We propose an invariant representation learning neural network that allows classification of textured surface roughness levels, while minimizing the discriminability of the motor movement type in a systematic way using an adversarial training approach [21]. The overarching goal of this study is to use such a framework to develop a system that mimics the sensation of surfaces with varying levels of roughness during active exploration. These systems would ideally be used in a wide range of applications such as teleoperation, surgical training of physicians in virtual environments, or remote control of robotic prostheses [22–26].

2. Experimental study design

2.1. Participants

A total of eight right-handed healthy participants (age: 26 ± 3.4 years) participated in this study. Participants did not have any neurological or somatosensory deficits or physical limitations. Written informed consents (University of Pittsburgh IRB No. PRO17040151) were obtained from all participants.

2.2. Experimental setup

During the experiments the participants sat in front of a computer screen and were instructed to use the tip of their dominant hand index finger while rubbing or tapping three different textured surfaces with varying levels of roughness. An illustration of the experimental setup is provided in Fig. 1(a) (cf. [20] for further information regarding the experimental setup design). The computer screen provided visual cues to the participants via a graphical user interface we developed using the Psychtoolbox software [27]. These cues marked the beginning of each condition. The experiment consisted of a total of six conditions, two movement type (i.e., rub and tap); and three textures surfaces (i.e., flat smooth, medium rough and rough). For each condition we instructed each participant to rub or tap the selected surface. Each complete movement across the textured surface was considered a single complete trial.

Fig. 1.

(a) Illustration of the experimental setup during a trial where the participant is instructed to tap on the texture mounted on the force transducer. (b) Printed medium rough 3D texture mounted on the force transducer.

Three synthesized textured surfaces with varying levels of roughness (i.e., smooth flat, medium rough, and rough) were generated using MATLAB (MathWorks, USA) and fabricated with Stereolithography (Viper SLA System, 3Dsystems, USA). The smooth flat surface is an even regular surface without any form of roughness. The roughness of the medium rough and rough surfaces represented by the power spectral density was controlled by the following expression:

$$\varphi (|k|)=\{\begin{array}{ll}C,\hfill & \text{if }{k}_l<=\left|k\right|<=k_r,\hfill \\ C{\left(\frac{\left|k\right|}{k_r}\right)}^{-2(1+H)},\hfill & \text{if }{k}_r<=\left|k\right|<=k_s,\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$ (1)

where C is the roughness amplitude, k_l, k_r, k_s are the lower roll-off and upper cutoff wave numbers and H is the Hurst roughness exponent. The values for each surface in this study are chosen as follows: the medium rough surface (H = 0.5, C = 10 × 10¹⁰, k_l = k_r = 16, k_s = 64) and rough surface (H = 0.5, C = 10 × 10¹⁰, k_l = k_r = 32, k_s = 256). For the third texture, simply a smooth flat surface was used. Each synthetic textured surface was mounted on a force transducer (as illustrated in Fig. 1(b) for the medium rough texture), and adjusted on a table in reach of the participants.

2.3. EEG data acquisition

Throughout the experiments, EEG data were collected from 14 EEG channels placed over the frontal and somatosensory cortices. These channels were placed according to the international 10–20 system [28] as channel locations: F3, F4, FC3, FC4, C1, C3, C5, CZ, C2, C4, C6, CP1, CPZ and CP2. We used the left mastoid as a reference and FPz as the ground electrode. Force data generated from touching the textured surfaces were also recorded using a force and torque transducer (NANO17 F/T transducer, ATI Industrial Automation, USA). Two g. USBamp amplifiers (g.tec medical engineering GmbH, Graz, Austria) were used to record and synchronize both EEG and force data. The first amplifier, which is used for EEG data acquisition, has a sampling frequency of 1200 Hz and uses a 4th order notch filter with cutoff frequencies 58 and 62 Hz, and an 8th order bandpass filter with cutoff frequencies of 2 and 62 Hz. The second amplifier was connected to the force transducer and used to synchronize the collected force data and digitize it with sampling rate of 1200 Hz. The force data was used to mark the beginning of each trial for the continuously recorded EEG data. We downsampled EEG data to 300 Hz sampling rate offline, and segmented into trials with duration of one third of a second following the touch (i.e., ~330 ms).

3. Methods

Deep neural networks have been widely explored as generic feature extractors for EEG in various classification tasks [29–33]. We recently explored such discriminative models in the context of deep invariant representation learning [34–36], to illustrate potential capabilities of EEG-based neural network models to censor specific nuisance information inherent within the learned feature extractors [37,38,21]. Our proposed neural network aims to learn texture roughness discriminative features while minimizing the influence of movement conditions (rub or tap) during active tactile exploration. We realize this using an adversarial training approach which infers a nuisance (i.e., motor movement activity) invariant latent space within a discriminative setting, hence suppressing motor movement related cortical activity from representations that are capable of classifying texture roughness levels.

3.1. Invariant representation learning network

We consider a subject-specific EEG data set ${\left\{\left({\text{X}}_i,l_{si},l_{mi}\right)\right\}}_{i=1}^n$ with n number of trials, where ${\text{X}}_i\in {ℝ}^{\text{CxT}}$ denotes the multichannel EEG data collected from C channels and T number of discretized time samples, l_si denotes the label for the roughness of a texture surface, and l_mi denotes the binary nuisance variable (i.e., rub or tap motor movement). Since no ordinal relationship exists in the categorical variables l_s and l_m, we represent them as one-hot encoded label vectors. Nuisance variations denoted by l_m are assumed to be involved in the data generation process, and therefore we employ an adversarial training approach that can discriminate l_s invariant of l_m. We premise the underlying data generation process on our assumption that l_s and l_m are disentangled such that l_s ~ p(l_s) and l_m ~ p(l_m) with X ~ p(X|l_s, l_m).

The network flow is outlined in Fig. 2. A deterministic feature encoder network with parameters γ_e projects the time-series EEG data onto a lower dimensional latent feature space, z = f(X; γ_e). This latent vector is passed to a classifier network with parameters γ_c, and an adversary network with parameters γ_a separately. The goal of classifier network is to infer surface texture label l_s, whereas that of adversary is to infer the nuisance variations l_m evoked by motor activity. In other words, the classifier network is trained to maximize the likelihood $q_{{\gamma }_c}\left(l_s\mid z\right)$, and the adversary network maximizes its likelihood $q_{{\gamma }_a}\left(l_m\mid z\right)$ However, the joint feature encoder network parameters are trained using a loss function that combines these two objectives within a conflicting relationship, by maximization of the likelihood $q_{{\gamma }_c}\left(l_s\mid z\right)$ and minimization of the likelihood $q_{{\gamma }_a}\left(l_m\mid z\right)$. This adversarial training behavior of the adversary network enables learning of l_m-invariant feature representations. Overall, the joint feature encoder aims to learn l_s-discriminative features, while the antagonistic adversarial loss enables concealing of l_m-relevant information from the feature encoder. This corresponds to the following min-max objective for adversarial model training:

$$\underset{{\gamma }_e,{\gamma }_c}{\mathit{\text{min}}}\underset{{\gamma }_a}{\mathit{\text{max}}}\mathbb{E}\left[-\mathit{\text{log}}{q}_{{\gamma }_c}\left(l_s\mid f\left(\text{X};{\gamma }_e\right)\right)+\lambda \mathit{\text{log}}{q}_{{\gamma }_a}\left(l_m\mid f\left(\text{X};{\gamma }_e\right)\right)\right],$$ (2)

to obtain the optimal network parameters ${\widehat{\gamma }}_e$, ${\widehat{\gamma }}_c$, ${\widehat{\gamma }}_a$ where λ > 0 is the adversarial regularization weight for nuisance-invariant decoding. λ = 0 would correspond to a traditional neural network training procedure without any manipulation. In case λ < 0, the encoder would be trained to learn particularly motor movement dependent features for texture roughness classification, which is not the intention of this study. Parameter updates are performed alternatingly. At each iteration, firstly the loss of the adversary is back-propagated to update the parameters of the adversary network γ_a towards maximizing its log-likelihood (i.e., max objective). Afterwards, the overall loss is back-propagated to update the encoder and classifier parameters γ_e and γ_c (i.e., min objective). Algorithm 1 outlines the stochastic model training process [21]. Implementations for the invariant EEG representation learning networks are publicly available at: https://github.com/oozdenizci/AdversarialEE GDecoding.

Fig. 2.

Overall adversarial invariant representation learning network flow, illustrating the feature encoder, classifier and adversary networks. Network layer specifications are further provided in Table 1.

3.2. Implementation and evaluation

Table 1 outlines the neural network architecture specifications that we used for EEG-based texture roughness classification. We determined the feature encoder layers in accordance with the EEGNet convolutional neural network (CNN) model [31]. Network input EEG data has dimensionality of 14 channels by 100 discretized time samples (i.e., 1/3 second segments of signals with 300 Hz sampling rate). The feature encoder network sequentially performs: (1) temporal convolutions that resemble to frequency bandpass filtering operations, (2) depthwise spatial convolutions [39] to extract bandpass specific information from distinct cortical sources over all EEG sensors, and (3) spatio-temporal separable convolutions [39] for summarization of all extracted information. Differently from the original EEGNet model [31], we re-selected the temporal and spatial convolution kernel lengths in consistency with our data set structure. Initial temporal convolution kernel size is chosen as 75, indicating that as low as 4 Hz of periodic signal activity will be captured from the input signals sampled at 300 Hz. Spatial convolution kernel length was determined as 14 by the number of EEG channels. All temporal convolution operations involved zero padding to keep the temporal length same. None of the convolution operations had a bias term. We used batch normalization after convolution operations [40], and dropout layers [41] with p = 0.5 after average pooling operations.

Table 1.

Invariant representation learning neural network architecture specifications.

	Network layer	Filters × (kernel size)	Output dim.	# of parameters
Feature encoder	Input EEG: X		(1, 14, 100)
	Conv2D	8 × (1, 75)	(8, 14, 100)	600
	BatchNorm + ELU		(8, 14, 100)	16
	Depthwise Conv2D	2 × (14,1)	(16, 1, 100)	224
	BatchNorm + ELU		(16, 1, 100)	32
	Average pooling	(1, 4)	(16, 1, 25)
	Dropout (p = 0.5)		(16, 1, 25)
	Separable Conv2D	16 × (1, 16)	(16, 1, 25)	512
	BatchNorm + ELU		(16, 1, 25)	32
	Average pooling	(1, 8)	(16, 1, 3)
	Dropout (p = 0.5) + Flatten		(1, 48)
	Dense + Softmax	(48, 3)	(1, 3)	147
	Classifier Output: ${\widehat{l}}_s$		(1, 3)
	Dense + Softmax	(48, 2)	(1, 2)	98
	Adversary Output: ${\widehat{l}}_m$		(1, 2)

Representations z = f(X; γ_e) that are learned at the output of the feature encoder are provided as input to two separate single dense layer networks (i.e., classifier and adversary). Both of these decision making layers perform linear classification. Classifier performs 3-class decoding of surface texture roughness, and adversary performs binary classification of motor movement type (rub or tap).

Evaluations were performed on a within-subject cross-validation basis by learning eight user-specific invariant representation learning neural networks. For each subject, we considered equal number of trials stratified by labels across six class conditions (i.e., two movement types and three textures), with total data set sample sizes of 2190, 426, 768, 438, 570, 762, 678, 708 for subjects 1–8 respectively. We generated the user-specific model training, validation, and test sets with randomly selected 70%, 10% and 20% portions of the trials available from each subject. The 20% test set split is specifically generated in 5-folds and repeated 10 times. Hence, all subject-specific classification analyses are performed by 10 × 5-fold cross-validation.

Networks were trained with a minibatch size of 40 training trials for at most 500 epochs. Early training stopping was performed if the classifier loss of the validation set did not reach a new lowest value for 10 consecutive epochs. We considered the model parameters which resulted in the lowest validation loss after completion of model training. Adam optimizer [42] was used for parameter updates once per batch. The overall network consisted of a total number of 1661 trainable parameters. We implemented the adversarial neural network training protocol using the Tensorflow deep learning library [43] with the Keras API [44].

4. Results

Our main goal is to classify different textured surfaces that vary in their roughness levels based on EEG recordings during active tactile exploration. While we choose to exploit a convolutional neural network model for feature learning and classification, we further benefit from the promise of such models in learning nuisance-invariant discriminative representations. Fig. 3 presents an overview of subject-specific classification accuracies of the classifier and adversary networks on the test sets after model learning. Horizontal axes in Fig. 3 denote the binary adversary network classification accuracies for rub versus tap motor movement type classification, whereas the vertical axes denote the 3-class decoding accuracies for surface texture roughness. Center marks of each colored box denote the averages across 5 training folds and 10 repetitions. The width of the boxes in both dimensions denote ±1 standard deviation intervals of the accuracies.

Fig. 3.

Classifier versus adversary accuracies on the test sets of each subject after model learning. For each colored box, center marks denote the means across 5 training folds and 10 repetitions, and box widths denote ±1 standard deviation intervals in both dimensions. Black dashed lines denote the change level accuracies in both axes. Note that the classifier handles a 3-class decoding problem, whereas the adversary considers a binary classification problem.

In cases where λ = 0, the classifier is trained by the traditional categorical cross-entropy minimization. However we also learn an adversary network solely in parallel, which simply monitors the motor movement variant information leakage on the encoded latent representations z. The non-adversarial model (λ = 0) 3-class classification results lead to an average 56% accuracy across all subjects (maximum is 70.4% for subject 7) as observed by the y-axis values in Fig. 3 plots. Simultaneously, we observed that indeed there is a significant amount of unwanted motor movement related information exploited in these texture roughness classification models. This is reflected by an average adversary network accuracy of 66.4% (as high as 74.7% for subject 2, x-axes in Fig. 3 plots) when λ = 0 for binary decoding of rub versus tap movement using simple linear classification with latent z. This supports our hypothesis on the necessity to censor motor movement relevant nuisance features from the latent representations during discriminative model learning for texture roughness classification.

Our adversarial regularization approach as λ > 0 successfully manipulates latent representations by reducing the adversary accuracies (i. e., leakage) towards minimizing the variance of distinct motor exploration movement patterns, while keeping the classifier performance stable for texture roughness decoding. On average across eight subjects, repetitions and folds, the 66.4% adversary network leakage of regular model learning (λ = 0) can be gradually reduced to 64.6%, 63.8%, 60.8%, 57.8%, 55.9% and 52.9% with adversarial regularization λ values of 0.01, 0.02, 0.05, 0.1, 0.2, 0.5 respectively. Such controllable invariance enables suppression of nuisance-related information down to near chance-level (50%) decoding, while keeping classifier accuracies as stable as possible.

Fig. 4 presents a summary of the differences between accuracies obtained with λ > 0 models and the λ = 0 condition. Horizontal axis represents the models for different λ values, while the vertical axis represents the differences of accuracies averaged across subjects. Note that a very strong adversarial regularization can lead to losing class-discriminative information for the main classification task, hence hindering the texture roughness decoding accuracies (e.g., with λ = 0.5). Here we present our results in an exploratory fashion, by demonstrating performances with all λ values. For further model selection and online use, an optimal λ value based on the validation set classifier and adversary accuracy can be selected [21].

Fig. 4.

Difference of accuracies with respect to λ = 0 (i.e., non-adversarial, regular CNN) for each adversarial regularization weight λ, averaged across subjects. Red denotes the differences in binary adversary network accuracies, and blue denotes the differences in 3-class classifier network accuracies.

5. Discussion

The ultimate goal of this work is to build an EEG-guided system that can provide the sensation of objects with different surface roughness levels. During our daily activities we engage in active exploration of our surroundings, and neural responses to this type of exploration reflect both motor movement and sensory perception related information. The human brain processes motor and sensory related information in a coupled way, where both information are in affect with each other [12–15]. Moreover, this active tactile exploration provides substantial information about the surface properties and demonstrate better discrimination thresholds compared to passive exploration [1,2,8]. To that extent, we highlight the need to investigate discriminative models that are sensitive to sensory information while at the same time invariant to varying motor movement related cortical activity patterns.

In order to empirically assess the feasibility of this problem, we present the current experimental active tactile exploration study with healthy subjects while the participants use their index finger to either rub or tap surfaces with three different texture roughness levels. We propose to use an adversarial invariant representation learning neural network model to classify the three different texture roughness levels using ~330 ms EEG data segments recorded in response to these movements, while minimizing the discriminability of the motor movement type. Our results show that the proposed approach can discriminate between three different textured surfaces with accuracies up to 70%, hypothetically by exploiting event-related potentials generated in response to sensory information [45]. Furthermore we also demonstrate the inherent existence of undesired motor movement related activity embedded in the predictive EEG features (as presented in Fig. 3), pointing to the necessity of our approach in censoring motor movement relevant variability from the texture roughness discriminative representations during model learning. Overall, our proposed adversarial inference approach successfully demonstrates its capability to manipulate latent representations towards minimizing the discriminability of distinct motor exploration movement patterns, while keeping the classifier performance stable for texture roughness decoding.

Natural characteristics of EEG signals are particularly an important factor that motivates: (1) our adversarially learned neural network invariance approach to this complex neural decoding problem, and (2) our specific spatio-temporal feature learning network architecture choices. Our basis argument regarding (1) arises from the fact that EEG signals, by nature, are collected and analyzed as a superposition of multiple task-specific and task-irrelevant neural oscillations [46]. Since multi-channel time-series EEG signals do not trivially dissociate multiple cortical processes occurring at the same time, this characteristic feature took a significant role in motivating the invariant representation learning concept of our model. Our adversarial invariance learning setting is one approach against this phenomena towards disentangling specifically defined nuisance variability, which in this case is simultaneously present motor cortical de-/modulations. In different settings this invariance could address another a priori defined nuisance variability (e.g., rhythmic eye-blinks). Our choices regarding (2) simply indicate that the specific recording and pre-processing pipeline of EEG signals constitute deterministic choices for the network architecture. Our decision to use a generic convolutional network architecture based on EEGNet [31] is motivated by an approach to initially perform temporal convolutions analogous to frequency filtering activities for multi-channel EEG signals, spatial convolutions for localization of task-discriminative cortical sources/regions for each subject, followed by spatio-temporal convolution layers for summarization of extracted temporal and spatial features. Furthermore, as outlined in Section 3.2, our temporal and spatial convolution kernel sizes were chosen in consistency with the utilized EEG signal’s recording and pre-processing pipeline (e.g., temporal convolution kernels were determined as one fourth of the EEG sampling rate to sufficiently capture periodic signal activity above 4 Hz). Hence the EEG signal characteristics implicitly determines the proposed model specifications in our study.

Our use of an invariant representation learning network offers the flexibility to manipulate the learned representations during training to be invariant to particular pre-defined nuisance variability (i.e., active touch movement condition), leading to significantly reduced motor movement related information leakage from the learned texture-discriminative representations. However one important limitation of our approach is that a very strong adversarial regularization can lead to loss of class-discriminative information for the main classification task, hence hindering the texture roughness decoding accuracies (as illustrated in Fig. 4 with λ = 0.5). An immediate potential improvement to our current approach will be to address the systematic subject-specific model selection issue for the optimal λ choice, which yields non-confounding results in terms of task discriminative information loss. Another important limitation of our study is the lack of subject-specific multi-session decoding stability analyses, due to our single-session experimental study design. To proceed along this line, a methodological improvement towards learning both motor movement as well as session-invariant representations can be potentially studied.

6. Conclusion

The overarching goal of this work is to build an EEG guided system that could provide and mimic the sensation of objects with varying levels of roughness. Applications of this system can include teleoperation, training of physicians training in virtual environments or remote robotic prosthesis control. Our daily life activities engage active exploration of our surroundings, and EEG responses to this type of exploration reflects both motor movement related and sensory perception information. Hence, we shed light onto the need to investigate discriminative models that are sensitive to sensory information while at the same time invariant to varying motor movement related cortical activity patterns.

In this paper, we show that it is possible to discriminate surfaces with varying levels of roughness during active tactile exploration using EEG as an input to the proposed neural network architecture. Results show that the proposed approach can discriminate between three different textured surfaces with accuracies up to 70%, while suppressing movement related variability from learned representations. In summary, our aim of this study was to develop a methodology to extract and select EEG features that can classify various textures with different levels of roughness using EEG that are invariant to motor activity. Such EEG features will be used to design principles for model-based optimal EEG-guided haptic feedback system in our future work.