Towards a next-generation hearing aid through brain state classification and modeling

Abstract

Traditional brain-state classifications are primarily based on two well-known neural biomarkers: P300 and motor imagery / event-related frequency modulation. Currently, many brain-computer interface (BCI) systems have successfully helped patients with severe neuromuscular disabilities to regain independence. In order to translate this neural engineering success to hearing aid applications, we must be able to capture brain waves across the population reliably in cortical regions that have not previously been incorporated in these systems before, for example, dorsolateral prefrontal cortex (DLPFC) and right temporoparietal junction. Here, we present a brain-state classification framework that incorporates individual anatomical information and accounts for potential anatomical and functional differences across subjects by applying appropriate cortical weighting functions prior to the classification stage. Using an inverse imaging approach, use simulated EEG data to show that our method can outperform the traditional brain-state classification approach that trains only on individual subject’s data without considering data available at a population level.

I. Introduction

Most young, normal hearing listeners can seamlessly direct their attention to segregate sound sources in crowded environments. However, in face of multiple sound sources, listeners with hearing loss often find such situations overwhelming and intimidating [1]. Hearing aids, especially bilateral aids [2], [3], can help; yet the top complaint from users remains that these aids are not beneficial to them in noisy situations [4]. While directional amplification could selectively amplify sound in a spatial pattern pre-determined by sophisticated algorithms, the listener still would not have a means of controlling that amplification [5]. Shinn-Cunningham and Best [6] proposed that “a revolutionary assistive listening device would use robust source separation algorithms to create auditory objects… and emphasize the desired target of attention… [while] enabling the listener to selectively attend, at will, to different objects in the environment.” Though existing computational auditory scene analysis cannot yet segregate sources as well as normal hearing listeners, this is an active research area [7]. Notwithstanding the above challenge, a next-generation hearing aid design must move away from a pure feed-forward amplification to a system that incorporates a feedback based on the user’s brain state (e.g., maintaining attention to one talker versus switching attention between conversations) that could then dynamically amplify sound based on the user’s attentional focus.

Brain patterns can be captured invasively (e.g., electrocoticography) or non-invasively (e.g., magnetoencephalography; MEG) in order to classify human brain states. In hearing aid design, only technologies that are portable and non-invasive should be considered, such as electroencephalography (EEG). In the past two decades, the BCI research community has concentrated on developing systems that serve the needs of those with severe neuromuscular disabilities. Traditionally, the majority of the brain-state classifications are based on two neural biomarkers: P300 (a positive deflection of the EEG event-related potential at 300 ms that reflects a person’s reaction to a presented “oddball” stimulus) and motor imagery / event-related frequency modulation (frequency-specific power modulation of ongoing EEG activity due to preparing or imagining of a specific movement). However, how do we begin to design new brain-state classification algorithms that move beyond these classic neural biomarkers? Additionally, is there a way to exploit training data across subjects instead of requiring a lengthy calibration session before every use of a BCI? Here, we describe a procedure that would allow us to pool data across subjects to identify new neural biomarkers that are associated with behaviorally relevant cognitive states and to optimally leverage this across-subject information to design new brain-state classification algorithms.

II. Methods

A. Participants

In a previous EEG study, structural MRI scans (using a 1.5-T Avanto scanner, 1 MPRAGE and 2 FLASH scans, ~8 min/scan) were collected for nine healthy subjects (procedures approved by Massachusetts General Hospital). This information was used to compute a three layer boundary element model (BEM) for calculations involving volume conduction. A BEM is a 3 dimensional head model comprised of the brain surface, skull, and scalp. Electrode locations were then coregistered with the structural MRI scan using cardinal landmarks (left and right periauriculars, nasion). FreeSurfer (software: http://surfer.nmr.mgh.harvard.edu/) was used to construct a distributed current model of about 7000 dipoles (Fig. 1 Top) on the cortical mantle oriented normally to the brain surface. These dipoles simulate the activity of macrocolumns in the cerebral cortex [8].

Figure 1. Distributed current model and simulation of activity.

Dipole sources in the brain (Top) are activated in different ROIs (Bottom Left) and projected onto the scalp (as x = Gj, Bottom Middle) of different participants to simulate engagement in a task. These EEG scalp potentials can then be projected back to the cortical surface (as Ĵ = Mx, Bottom Right).

B. Brain activity simulation and estimation

Combining all of this information, each subject’s forward solution gain matrix G was calculated using the BEM model, which gives EEG sensor activity x for any given dipole source activation j at the brain surface where

$$\mathit{x}=\mathit{\text{Gj}}.$$ (1)

The inverse solution to the forward gain matrix M can be determined. M allows the estimation of current at each dipole on the cortical surface Ĵ given recorded EEG activity x where

$$\hat{\mathit{J}}=\mathit{\text{Mx}}.$$ (2)

Simulation of EEG data begins by activating spatial regions of interest (ROIs) in the distributed current source space. The 78 ROIs used come from a standard parcellation of the cortex (using FreeSurfer) and are morphed from an average brain to the subject of interest through spherical morphing [9]. Cortical currents j are simulated at each dipole contained in the specific ROI (Fig. 1 Bottom Left). Currents are normalized based on the ROI size, and only one ROI is activated at a time. Variable white Gaussian noise is added to each dipole to manipulate the signal to noise ratio (SNR) for each set of trials. The simulated currents j are then multiplied by the gain matrix G to obtain simulated EEG sensor activity x for about 70 sensors (see Fig. 1 Bottom Middle). Multiplying the activity by this lead-field matrix G introduces noise correlations as cortical currents are transformed to the sensor space. The pseudoinverse M is then multiplied by the noisy EEG activity x to obtain the dipole estimation Ĵ (Fig. 1 Bottom Right). The inverse matrix M depends only on the lead field (derived from electromagnetic equations [8], [10]) and the noise covariance matrix. The noise covariance used here is only slightly regularized due to a reliable noise covariance estimate. With non-simulated data, a regularization parameter is used to ensure a stable, reasonable inverse solution even for noise covariance estimates that are less reliable.

C. Transforming data between subjects

A transformation matrix T_k→i is also calculated for every pair of subjects. This involves spherically morphing brain spaces between subjects while preserving sulcal-gyral alignment. This allows for the conversion of any subject k’s source space to subject i’s and is necessary for comparing brain activity between subjects in a common cortical space.

Inter-subject training of brain activations is carried out through the morphing of estimated current dipole activity from each subject in a training pool (subjects 1 to N) to the subject of interest (subject N+1) using the transformation matrices. Importantly, this implies that the pooled approach will be training to recognize brain activations for subject N+1 without subject-specific training data from subject N+1 (see Fig. 2). The transformation of an activated ROI from the pooled training on subjects 1 to N is not likely to yield perfect overlap once the corresponding ROI is determined in the subject-of-interest’s source space. Rather than select a subset of the most common current dipoles for classification, we employ a weighted average. The mean position of all the transformed dipoles is calculated in the subject of interest’s spherical brain space and an exponential decay (based on the distance from this centroid) is computed. Dipoles closest to the centroid are assigned a near-unity weight, while dipoles far away are assigned a near-zero weight (see Fig. 2 Bottom Right). Because every individual dipole is assigned a weight, the weight matrix W can be multiplied by G to obtain a weight for the EEG electrodes.

Figure 2. Data from subjects 1 to N are mapped onto Subject N’s space using a spherical morphing procedure.

Individual source spaces (top row) are first morphed into a spherical space (second row). Next, spherical brains from subjects 1 to N are transformed to the N+1 subject’s brain space using sulcal-gyral alignment in spherical space (third row). The ROIs (black, turquoise, blue, and purple patches) are unlikely to match up perfectly, so a center of mass is computed for the spatial exponential weighting. The N+1 subject’s brain is morphed out of spherical space (bottom left). Exponential weighting (bottom right) is used for compensating for inter-subject variability.

In this binary classification (active or rest) scheme, EEG activity in an equal number of conditions is used to train a regularized linear discriminant analysis (LDA) classifier. This classifier was chosen because of its widespread use in BCIs and relatively straightforward implementation. Furthermore, our motivation was rooted in finding relative differences between classifications of the standard and exponential weighting approach and not in the absolute performance achieved by a specific classifier. The exponential weighting approach was evaluated against a standard leave-one-(trial)-out cross-validation benchmark. Briefly, this standard cross-validation approach involves training on all but one trial and testing on the excluded trial in an iterative manner that cycles through the entire data set. These classification schemes were assessed under a variety of SNRs and two training set sizes. SNRs ranged from −25.0 decibels (dB) to −5.0 dB in 2.5 dB steps, with training set sizes of 10 or 40 trials. The lowest training set size (10 trials) and the most common synchronous training set size (40 trials) were selected to show performance gains with few trials and a common training set size, respectively.

III. Results

To determine the effect of exponentially weighting prior to classification, it was evaluated against a standard leave-one-out cross-validation procedure. Brain maps illustrating ROI performance in select conditions are shown in Fig. 3. The six SNRs closest to the 60–80% performance window in the 10- and 40-trial training set sizes are displayed since these correspond to typical motor BCI performance. Note that the largest performance boosts occurred at low SNRs (toward the left) and with small training sets (top row). Furthermore, performance gain varies as a function of cortical location. For example, classification of activities in the dorsolateral prefrontal cortex—an executive control hub—receives a performance boost across all conditions tested, while the premotor cortex does not receive much benefit in this approach.

Figure 3. The performance gains of individual ROIs in the exponentially weighted classification compared to standard leave-one-out.

The rows represent training set sizes of 10 and 40 trials, while the columns are SNRs (−20 to −5 dB in 2.5 dB steps). Hot colors represent a performance gain relative to the standard classifier (yellow is 25%), blue represents a performance deficit (with light blue representing −25%) and gray (see-through) being equal performance. Note performance changes vary as a function of anatomical region (e.g., increases across all set sizes and SNR in the dorsolateral prefrontal cortex [DLPFC]).

IV. discussion

Here we demonstrate that transforming brain activations across subjects improves activity classification when there are few or noisy training trials. While the implementation of a non-uniform weighting approach may yield better performance across cortex, it is especially likely to improve performance in areas that have complex folding patterns. The prefrontal cortex compared to the motor strip, for example, has more complex sulcal-gyral folding, and this folding is more variable across subjects. Using a surface-based nonlinear mapping between subjects could help compensate for these sources of variability. The approach outlined here could potentially be used to translate neuroscience insights regarding significant regions of neural activity that are conserved across subjects (a typical neuroimaging finding) into useable signals for BCI designs. Ideally, the final training solution would require only a structural MRI scan and coregistration with the EEG electrodes to complete the training prerequisite for each new user (given an existing data pool). If classification performance proves comparable in the transition from synthetic data to actual EEG recordings, this methodology would reduce or eliminate the need for a subject-specific trial-based calibration session before every use of a BCI.

Our approach presented here would be useful for incorporating important signals to drive hearing aids in the future. For example, a recent study found that the right temporoparietal junction is more active when a subject switches spatial attention compared to maintaining attention in one location [11]. Through new neuroimaging studies, many hearing aid control signals regarding a user’s intention will be mapped. Although the origin of these control signals is still an active area of research, they are likely to originate from areas such as the prefrontal cortex. While these principles have yet to be developed into a patient-usable device, recent developments in mobile EEG platforms make our brain-state classification approach poised to translate these new neuroscience discoveries into real-world solutions.

V. Conclusion

While the scientific quest to map human brain function has exploded in the last two decades, the ability to link patterns in EEG signals to specific cognitive states remains elusive, owing perhaps to limited crosstalk between the fields of neuroscience and engineering. We hope that the anatomical based approach described here will inspire new innovative brain-state classification methods that can be incorporated in future hearing-aid designs, as well as other augmentative and rehabilitative devices relevant to the speech and hearing sciences domain and beyond.