Real-time brain-controlled selective hearing enhances speech perception in multi-talker environments

Vishal Choudhari; Maximilian Nentwich; Sarah Johnson; Jose L Herrero; Stephan Bickel; Ashesh D Mehta; Daniel Friedman; Adeen Flinker; Edward F Chang; Nima Mesgarani

doi:10.1038/s41593-026-02281-5

. 2026 May 11;29(10):2408–2418. doi: 10.1038/s41593-026-02281-5

Real-time brain-controlled selective hearing enhances speech perception in multi-talker environments

Vishal Choudhari ^1,², Maximilian Nentwich ^3,⁴, Sarah Johnson ^3,⁴, Jose L Herrero ^3,⁴, Stephan Bickel ^3,⁴, Ashesh D Mehta ^3,⁴, Daniel Friedman ⁵, Adeen Flinker ⁵, Edward F Chang ⁶, Nima Mesgarani ^1,^2,^✉

PMCID: PMC13633114 PMID: 42115433

Abstract

Understanding speech in noisy environments is difficult for many people, and current hearing aids often fail because they amplify all sounds rather than the talker of interest. Auditory attention decoding (AAD) offers a potential solution by using the listener’s brain signals to identify and enhance the attended speaker, but it has been unclear whether this can provide real-time perceptual benefits. Here we used high-resolution intracranial electroencephalography in patients undergoing neurosurgical procedures to implement a closed-loop system that achieves the decoding fidelity necessary to dynamically amplify the attended talker. Across multiple experiments, the system improved speech intelligibility, reduced listening effort and was consistently preferred by subjects. It also tracked both instructed and self-initiated attention shifts. By providing direct evidence that a real-time, brain-controlled hearing system can enhance perception, this work establishes a key performance benchmark for future auditory brain–computer interfaces and advances AAD from a theoretical concept to a validated solution for personalized assistive hearing.

Subject terms: Cortex, Attention

This study shows that brain signals can identify and amplify the voice a person wants to hear in a crowded scene. Choudhari et al. provide evidence that brain-controlled hearing can improve speech perception and clarity in noisy environments.

Main

Understanding speech in crowded environments remains one of the most difficult challenges in auditory neuroscience and hearing technology¹. In these situations, listeners rely on selective attention to focus on a target talker while suppressing competing voices and background noise^2,3. Current hearing aids often fail because they cannot infer the listener’s intent. As a result, they amplify all sounds indiscriminately, leading to degraded performance in real-world settings and contributing to low user adoption and social isolation^1,4.

AAD is a brain–computer interface (BCI) approach that decodes a listener’s attentional focus from their neural signals to selectively enhance the attended talker^5,6. This approach is grounded in foundational work showing that the human auditory cortex produces a distinct neural signature for the attended talker^7–10. Over the past decade, hundreds of studies have focused on improving decoding accuracy^6,11–14, while only a few have attempted real-time decoding^15–17 and control in closed-loop settings^18–20 (Supplementary Table 1). Yet one key question has remained unanswered: whether a real-time, brain-controlled hearing system can actually help someone hear better. The absence of perceptual validation reflects a deeper challenge. A real-time AAD system must achieve a high-performance threshold of speed, stability and accuracy; a system that falls short can degrade the listening experience rather than improve it²⁰. Successfully demonstrating a clear behavioral benefit has therefore remained a critical and unachieved milestone for the field of assistive auditory neurotechnology.

Here, we address this longstanding gap using high-resolution intracranial electroencephalography (iEEG) to establish a gold-standard benchmark. We developed a closed-loop AAD system that dynamically amplifies the attended talker in real time. We evaluated its performance across three complementary experiments: (1) testing perceptual benefit when the system is activated mid-trial; (2) measuring responsiveness to instructed attention switches; and (3) assessing the ability to track natural, self-initiated shifts of attention. The system significantly improved speech intelligibility, reduced listening effort and was consistently preferred over the baseline. By providing the first definitive proof of behavioral benefit, this work moves brain-controlled hearing from theoretical promise to validated technology, establishing a critical performance and viability benchmark that confirms the fundamental promise of auditory BCIs and laying the foundation for next-generation assistive and augmented listening technologies.

Results

Four participants (S1–S4) undergoing epilepsy monitoring with iEEG took part in the study (Fig. 1a). All participants had self-reported normal hearing. Electrode coverage varied across individuals based on clinical needs, with recordings spanning the superior temporal gyrus and surrounding auditory cortex, regions known to encode speech-selective neural responses^8,21 (Fig. 1b and Extended Data Fig. 1).

Extended Data Fig. 1 — Anatomical distribution of all speech-responsive electrodes for participants S1–S4, shown on semi-transparent brain surfaces. Each dot represents a single electrode used in decoding, color-coded by participant. Views are shown from lateral, superior, and posterior perspectives to illustrate spatial coverage across cortical surfaces. While coverage varied across individuals, all participants had electrodes over auditory-responsive regions in the temporal lobe, enabling subject-specific decoding models.

Source data

To simulate real-world listening conditions, we constructed multi-talker scenes containing two simultaneous conversations, spatially separated at ±15° azimuth. Each conversation was a naturalistic dialogue between two talkers who took turns speaking about topics such as food, travel and exercise. The gender composition of the competing talkers was varied to produce three levels of acoustic similarity: different gender (male vs female), mixed gender and same gender (male vs male or female vs female), with a balanced number of trials for each. To further increase ecological realism, each trial included either pedestrian noise or multi-talker babble, randomized across trials. To monitor attentional engagement, both conversations included repeated words, but participants were instructed to respond only to repeats in the attended conversation by pressing a button. Participants S2–S4 were native English speakers; S1 was a non-native speaker.

A two-phase design was used to train and test the system. In the offline phase, participants were instructed to attend to a cued talker while ignoring the other as their neural responses were recorded. Trials were presented with the attended talker set at either +9 dB, 0 dB or –6 dB target-to-masker ratio (TMR). Behavioral performance confirmed that participants reliably tracked the attended conversation (Extended Data Fig. 2). Using these data, we trained and validated a participant-specific AAD model. The model is a linear regression decoder based on a stimulus reconstruction approach^5,8,22 that learns to reconstruct the temporal envelope of the attended speech from neural activity. It uses a rich feature set combining high-gamma (70–150 Hz) and low-frequency (1–30 Hz) neural envelopes (Fig. 1a and Extended Data Fig. 3) from all speech-responsive electrodes (Fig. 1b). The attended speech envelope at each time point was reconstructed using the preceding 400 ms of neural data. To make a stable decoding decision, the Pearson correlation between this continuously reconstructed envelope and the actual speech envelopes was then calculated over a 4 s sliding window, which was updated with a 0.5 s hop size. AAD accuracy was defined as the percentage of windows in which the decoded talker matched the cued target. This window size of 4 s was selected based on offline performance to balance real-time responsiveness with decoding stability (Fig. 1d and Extended Data Fig. 3). Single-electrode decoding performance is shown in Fig. 1c (see Extended Data Fig. 4 for subject-level maps). All participants showed decoding accuracy well above chance, with average performance ranging from 72.0% to 90.3% (Extended Data Fig. 5). This variability could be attributed to multiple factors, including subject-specific differences in electrode coverage as well as trial-by-trial variations in a participant’s attentional engagement²³.

Extended Data Fig. 2 — Repeat-word detection accuracy for each participant in the offline phase, separated by attended versus unattended speech streams. Participants were instructed to press a button when they heard repeated words in the attended conversation. Detection accuracy was consistently high for attended speech and near floor for unattended speech, confirming that participants followed attentional instructions.

Source data

Extended Data Fig. 3 — Decoding accuracy plotted as a function of window size (0–12 s) for each participant (S1–S4), using three feature types: low-frequency (LF, 0.5–30 Hz), high-gamma (HG, 70–150 Hz), and the combination of both (LF + HG). Accuracy improves with longer windows across all subjects, and models that combine LF and HG features consistently outperform single-band models. This highlights the complementary information carried by low-frequency phase-locked activity and high-gamma power in tracking attention.

Source data

Extended Data Fig. 4 — Brain maps of individual participants (S1–S4), showing the anatomical location and performance of speech-responsive electrodes used for AAD decoding. Each dot represents a single electrode, colored by its individual decoding accuracy, with darker red indicating higher accuracy. Views are shown from lateral (top) and dorsal (bottom) perspectives of both hemispheres. Data reveal consistent clustering of high-performing electrodes in superior temporal regions, particularly in S1–S3, with more distributed patterns in S4. Accuracy scale ranges from 50% (chance) to 85%.

Source data

Extended Data Fig. 5 — a, Box plots showing AAD accuracies for each participant (S1–S4) during the offline experiment. Mean AAD accuracy is shown on top. All participants showed decoding performance well above chance level (50%, red dashed line). Box plots show the median and interquartile range across trials with whiskers extending to the most extreme data points within 1.5 × IQR (n = 26 trials per subject).

Source data

In the subsequent online phase, the frozen AAD model decoded attention in real time. To test for generalization across both linguistic and acoustic contexts, the speech material used in this phase was different, and of the eight talkers presented, four were new to the model and had not been included in the offline training set. For each sliding window, the model reconstructed the attended speech envelope from neural activity and computed its Pearson correlation with the envelopes of the two competing speech conversations^5,8. The conversation with the higher correlation was assumed to be attended, and this decision was used to guide gain control. To prevent abrupt or distracting volume shifts, we implemented a five-state Markov model that smoothed transitions in the TMR²⁴. The system could dynamically shift gain by up to ±9 dB, with intermediate steps used to create natural, gradual changes. Gain was adjusted by changing the relative levels of the two talkers while holding overall power constant (Methods). Instructed attention trials used a visual cue (Fig. 1a), and participants were not informed that their neural signals were being used to control the audio. Depending on the experiment, trials were divided into segments to allow comparison between baseline and AAD-enhanced conditions, attention-switching periods or natural shifts in focus.

Experiment 1: real-time AAD provides significant and multifaceted perceptual benefits

We first tested the perceptual benefit of the real-time AAD system in an experiment in which the system was activated mid-trial (Fig. 2a). To account for the known build-up of auditory attention, the initial 10 s of each trial served as a stabilization period and were excluded from analysis (unrated segment, gray bar in Fig. 2a). This 10 s window was chosen conservatively, as prior studies have shown that neural attention tracking typically stabilizes within 3–5 s of stimulus onset^25,26. Participants then heard a ‘system off’ segment, during which the target speech was presented at a challenging −6 dB TMR²⁷. In the following ‘system on’ segment, the closed-loop system was activated, selectively amplifying the neurally decoded attended speech up to +9 dB TMR (Supplementary Video 1). At the end of each trial, participants reported which segment they preferred and answered two intelligibility questions probing the target conversation in each condition.

The real-time AAD system accurately tracked attentional focus. The resulting single-trial and average correlation differences between the neurally reconstructed speech envelope and the envelope of each of the two competing conversations showed consistent separation between the attended and unattended conversations (Fig. 2b), yielding an average attention decoding accuracy that was significantly above chance level for all subjects (Fig. 2c). This effective neural decoding directly translated into the system’s gain control; a trial-by-trial analysis confirmed that average AAD accuracy in each trial was a significant predictor of the resulting average TMR (adjusted R² = 0.98, P < 0.001; Extended Data Fig. 6), leading to an average TMR boost of +12 dB between the ‘off’ and ‘on’ segments (Fig. 2d).

Extended Data Fig. 6 — Single-trial average AAD is a robust predictor of the average TMR (n = 80 trials).

Source data

This brain-controlled gain change led to clear and multifaceted perceptual benefits. Participants demonstrated a strong preference for the listening experience when the system was active. A generalized linear mixed-effects model (GLMM) on trial-level ratings, with subjects as random effects, revealed a highly significant preference for the ‘system on’ condition (estimate = 1.55, SE = 0.29, P < 0.001), with individual preference rates for the ‘system on’ condition ranging from 75% to 95% (Fig. 2e). This subjective benefit was mirrored by improved objective performance: a GLMM on the question–answer responses showed that when intelligibility changed between segments, it was significantly more likely to improve (estimate = 0.73, SE = 0.035, P < 0.05; Fig. 2f). Finally, a trial-by-trial analysis of pupil dilation revealed a significant reduction in the ‘system on’ condition for both participants with available data (S2: P < 0.001; S3: P < 0.05; two-sided Wilcoxon signed-rank test), consistent with decreased listening effort^23,28–30, in the ‘system on’ compared to ‘system off’ condition (Fig. 2g).

Decoding accuracy and envelope correlation remained consistent across acoustic conditions that varied in talker acoustic similarity, including trials with same-gender and different-gender competing conversations (P > 0.05; Extended Data Fig. 7). Similarly, decoding performance was stable across background noise types, with no significant differences between babble and pedestrian conditions (P > 0.05; Extended Data Fig. 7). Finally, no significant difference was observed in the decoding accuracy for unseen versus seen speakers (Kruskal-Wallis test, P > 0.05). These results confirm that decoding generalizes well across realistic acoustic variability.

Extended Data Fig. 7 — a–c, (Top): AAD decoding performance as a function of trial difficulty, defined by the gender composition of the competing talkers (easy = male vs. female; medium = mixed-gender; hard = same-gender). Boxplots show (left to right): a, decoding accuracy, b, correlation between reconstructed and attended speech envelopes, and c, average target-to-masker ratio (TMR) achieved by the system. Decoder accuracy and envelope correlation remained consistent across difficulty levels. A Kruskal-Wallis test revealed no significant difference in AAD accuracy across the three talker-gender difficulty conditions (p = 0.940, n = 80 trials), confirming the system’s robustness to this acoustic challenge. a–c, (Bottom): AAD decoding performance under two background noise conditions: babble and pedestrian noise. Boxplots show (left to right): a, decoding accuracy, b, envelope correlation, and c, resulting TMR. System performance was consistent across noise types, indicating robustness of the decoding pipeline and gain control strategy. Red lines indicate medians; boxes show interquartile range; whiskers represent 1.5× IQR. Similarly, a Wilcoxon rank-sum test showed no significant difference in AAD accuracy between trials with babble versus pedestrian background noise (p = 0.29, n = 80 trials), demonstrating the system’s resilience to different noise types.

Source data

To understand the source of trial-by-trial performance variability and its perceptual consequences, we performed two regression analyses. We first tested the hypothesis that the system’s decoding accuracy was directly related to the user’s own attentional engagement. A linear mixed-effect model revealed a significant positive relationship between a participant’s attention tracking performance (the repeat-word detection score) and the AAD accuracy on that trial (estimate = 1.36, P < 0.001; Fig. 3a). This result suggests that variability in system performance is not necessarily a failure of attention decoding, but rather reflects the user’s own internal cognitive state; in trials during which a participant’s attention may have lapsed, the neural signature was correspondingly less distinct, leading to lower decoding accuracy.

Fig. 3 — This figure presents two complementary regression analyses that trace the source and perceptual consequence of trial-by-trial performance variability. a, System accuracy is predicted by the subject’s attentional tracking performance. To evaluate the relationship between objective task engagement and system performance, a two-sided linear mixed-effects model was used to model single-trial AAD accuracy as a function of the participant’s repeat-word detection score. The model included a random intercept for each subject to account for individual differences in baseline performance. System accuracy was significantly predicted by the user’s attentional tracking (estimate = 0.35, SE = 0.09, adjusted R² = 0.23, t₅₈ = 3.75, 95% CI = [0.16, 0.54], P = 0.0004). The shaded region indicates the 95% confidence interval of the fixed effect, and markers represent single-trial results (n = 80 trials). b, Single-trial AAD accuracy predicts user preference. A two-sided GLMM with a binomial distribution and logit link function was used to determine whether single-trial AAD accuracy predicted the binary preference outcome (prefer on vs prefer off). The model included a random intercept for each subject to account for individual differences in baseline preference. Objective decoding accuracy was found to be a highly significant predictor of subjective preference (estimate = 0.042, SE = 0.013, t₇₈ = 3.36, P = 0.0008). Data were grouped into three predefined accuracy bins (0–40%, 40–75% and 75–100%), and the mean probability of preferring the ‘system on’ was computed within each bin. Black circles indicate bin-wise means; error bars, s.e.m across trials. Gray dots represent individual trial outcomes (n = 80 trials), jittered vertically for visibility.

Source data

Crucially, we found that fluctuations in decoder fidelity had a direct and meaningful impact on the user’s subjective experience. A trial-level GLMM analysis revealed that single-trial AAD accuracy was a highly significant predictor of user preference (estimate = 0.042, SE = 0.013, t₇₈ = 3.36, P = 0.0008). This positive coefficient indicates that for every 1% increase in decoding accuracy, the odds of a participant preferring the ‘system on’ condition increased by 4.3%. As visualized in Fig. 3b, trials with higher decoding accuracy were associated with an increasing probability of preferring the ‘system on’ condition. For example, in trials in which the system achieved high accuracy (>75%), participants consistently preferred the system, whereas preference dropped closer to chance levels when accuracy was lower. This confirms that achieving high-fidelity neural decoding, whether through better models or higher quality signals, is not merely a technical objective, but is essential for delivering a consistently beneficial and preferred user experience. Together, these results provide behavioral and physiological evidence that real-time, brain-controlled gain modulation can reliably improve the overall listening experience in challenging acoustic environments.

Experiment 2: the system tracks instructed attention switches

In real-world conversations, listeners often shift attention between talkers. A key concern for closed-loop AAD systems is whether amplifying the currently attended talker in real time might hinder the listener’s ability to switch attention to another, now less audible talker. Specifically, we wondered whether users still switch focus, and whether the system can detect and adapt to the new talker with sufficient speed to support a seamless experience. Our next experiment tested whether the AAD system could track such instructed attention switches and dynamically update the gain in real time.

In each trial, participants were initially cued to attend to one of two overlapping speech streams. Midway through the trial, a visual cue instructed them to switch their focus to the other stream, while the system remained active throughout (Fig. 4a and Supplementary Video 2). The system successfully tracked this shift in focus. As shown in Fig. 4b, single-trial and average correlation differences exhibited a clear transition in neural tracking following the switch cue, resulting in a successful decoding of attentional focus (Fig. 4c). Given that the decoding output controls gain dynamically, the system’s TMR shifted accordingly, favoring the first conversation before the switch and the second conversation after it (Fig. 4d). We quantified the system’s responsiveness using ‘switch time’, defined as the time required for the average TMR to reach 0 dB following the switch cue. Across participants, the mean switch time was 5.1 s (S1, 7.0 s; S2, 4.5 s; S3, 5.0 s; S4, 4.0 s). This latency reflects both the listener’s cognitive response, but more prominently, the system’s design choices, including the 4 s sliding window and the five-state Markov model that smoothed gain transitions to ensure perceptual stability.

Fig. 4 — a, Experimental paradigm for experiment 2. Participants were instructed to attend to conversation 1 and then, following a visual cue mid-trial, switch their attention to conversation 2, while the system operated continuously. b, Single-trial and average correlation difference. The heatmap shows the trial-by-trial neural tracking for each subject, with the average trace below confirming a clear and rapid reversal in decoding following the switch cue at time 0. Error bars, s.e.m. (n = 40). c, Average AAD accuracy per subject. The box plot summarizes the overall decoding accuracy for each participant, showing that all subjects performed well above the 50% chance level (dashed line). Box plots show the median and IQR across trials, with whiskers extending to the most extreme data points within 1.5× IQR (n = 10 trials per subject). d, Single-trial and average attended talker TMR. The system’s gain control successfully followed the decoded attention, with the TMR enhancement flipping from conversation 1 to conversation 2 after the switch, as shown in both the trial-by-trial heatmap and the average trace below. Error bars, s.e.m. (n = 40 trials). e, Repeated word detection accuracy. The behavioral performance (repeat-word detection) for conversation 1 (attended first) was high pre-switch and low post-switch, while the opposite was true for conversation 2. The bars represent the mean detection accuracy across all trials and subjects for each of the four conditions. The lines connect the average performance for each individual subject. Error bars, s.e.m. (n = 40 trials).

Source data

To confirm that participants successfully followed the instructed switch, we analyzed their performance on the repeat-word detection task. The results, visualized in Fig. 4e, show a clear interaction. The plot displays the overall mean performance for each condition (bars) and the average trend for each individual subject (colored lines). A clear ‘X’ pattern is visible: performance for conversation 1 (attended first) is high pre-switch and drops after the cue, while performance for conversation 2 (attended second) is low pre-switch and rises after the cue. To further test this pattern, we performed a linear mixed-effects model analysis. The model included fixed effects for the trial segment (pre vs post switch), the conversation (conversation 1 vs conversation 2) and, critically, their interaction, with subject included as a random effect to account for individual differences. The analysis revealed a highly significant interaction between the trial segment and the attended conversation (estimate = 161.67, SE = 6.92, P < 0.001). This significant interaction shows that participants were able to flexibly and successfully shift their attentional focus on command.

Experiment 3: the system successfully tracks self-initiated attention switches

In natural listening, attention shifts are often internally driven rather than externally cued³¹. This experiment tested whether the AAD system could detect and respond to such self-initiated shifts in attentional focus, without any external timing signal (Fig. 5a). Participants freely redirected their attention between two competing talkers while the system remained active throughout. To provide a behavioral reference, participants indicated their current focus by pointing to the loudspeaker they were attending to. Although this involved a motor gesture, the signal driving the decoder was based on speech envelope tracking, and we confirmed that the motor behavior was uncorrelated with the speech envelopes and therefore could not systematically influence the decoding process (Extended Data Fig. 8).

Extended Data Fig. 8 — Time series showing the correlation between the actual attended speech envelope (s, blue), the neurally reconstructed envelope (rs, red), and either the visual cue or motor input signal (sv, green). The green trace reflects the binary direction of the visual cue (Experiment 2) or the participant’s pointing behavior (Experiment 3). A strong correlation was observed between the actual and reconstructed speech envelopes (ρ = 0.41, p « 0.001), while no significant correlation was found between the visual or motor signals and either s (ρ = 0.014, p > 0.05) or rs (ρ = –0.03, p > 0.05). Importantly, behavioral data (for example, preference ratings and intelligibility) were primarily collected in Experiment 1, where the visual cue was fixed throughout each trial and no motor responses were involved. In Experiment 2, the visual cue switched mid-trial to instruct an attention shift, but no pointing was used. In Experiment 3, participants pointed to indicate their attended talker, allowing motor-related activity to be separately assessed. Together, these results confirm that decoding was not systematically influenced by visual or motor confounds.

The system successfully tracked voluntary shifts in attention. As shown in Fig. 5b, single-trial and average correlation differences between the reconstructed and actual speech envelopes exhibited clear transitions aligned with participants’ self-reported focus. The corresponding TMR adapted in real time, enhancing the newly attended stream while attenuating the previously attended one (Fig. 5c). These results demonstrate that real-time AAD can flexibly and accurately follow endogenous changes in attention without requiring external cues (Supplementary Video 3).

To further test the system’s perceptual impact, we introduced a control condition in which gain control was intentionally reversed without informing the participant. In this condition, the system amplified the unattended speaker and suppressed the attended one. Participants immediately reported increased difficulty and a degraded listening experience (Supplementary Video 4). This condition serves as a critical control: if correct AAD-based gain improves perception, then reversing it should, and did, disrupt it. This further confirms the behavioral relevance and robustness of attention-driven enhancement.

Evaluation by individuals with hearing loss

To assess the translational potential of our system, we conducted a perceptual validation study with 40 individuals with hearing loss. This experiment evaluated whether the AAD-enhanced audio, generated from the neural signals of our iEEG participants with normal hearing, would be perceived as beneficial by this clinically relevant population. Although prior work has shown that increasing the TMR can aid speech comprehension³², it was not clear whether our system’s moment-by-moment, imperfect gain modulation would improve or disrupt perception in listeners with hearing loss. These participants listened to the entire trial output audio from experiment 1, including both the ‘system off’ and ‘system on’ conditions, in which the TMR was dynamically controlled by prior real-time neural decoding of attention from iEEG participants. The results provided a clear validation of the system’s perceptual output (Extended Data Fig. 9). The hearing loss cohort showed a strong subjective preference for the ‘system on’ condition (t₃₉ = 17.53, P < 0.001, Cohen’s d = 1.36), and they demonstrated significantly greater objective improvement in speech intelligibility than the normal-hearing iEEG group (t₃₉ = 9.50, P < 0.001, Cohen’s d = 1.06), probably because the task was more challenging for the hearing-loss group and therefore more sensitive to the perceptual benefits of increased TMR. This finding confirms that brain-driven TMR modulation remains perceptually robust even when applied to an independent population with hearing impairment. Combined with prior evidence that AAD is feasible for individuals with hearing loss^18,33,34, these results support the potential of brain-guided enhancement as a future hearing aid.

Extended Data Fig. 9 — (a) Average audiograms for 40 participants with mild-to-moderate sensorineural hearing loss. Solid lines show mean hearing thresholds across frequencies for the left and right ears; shaded areas represent standard deviation. (b) Subjective preference ratings comparing the AAD “On” vs. “Off” condition, where participants listened to the system-controlled output and indicated which version was easier to understand. Box plots show the median and interquartile range across trials with whiskers extending to the most extreme data points within 1.5 × IQR. (c) Objective intelligibility scores for the same stimuli, measured via multiple-choice comprehension questions. Each line represents a single participant; **p < 0.001 for both comparisons (two-sided paired t-tests, n = 40 subjects). Participants showed a strong preference for the AAD-enhanced audio which significantly improved intelligibility, validating the perceptual quality of the brain-controlled system output. Box plots show the median and interquartile range across trials with whiskers extending to the most extreme data points within 1.5 × IQR.

Source data

Discussion

Establishing a benchmark for brain-controlled hearing

This study addresses a critical and longstanding gap in auditory BCIs by providing the first direct evidence that a real-time, closed-loop AAD system can improve speech perception in multi-talker environments. This work represents a key milestone, demonstrating that brain signals can be used not only to decode attentional focus but also to enhance intelligibility, increase subjective preference and reduce listening effort, advancing AAD from a theoretical concept to a validated perceptual technology. By showing that the system adapts to both instructed and self-directed attention shifts, we establish its flexibility in dynamic listening conditions, an essential requirement for real-world usability. Finally, the finding that the same brain-controlled output enhanced comprehension in individuals with hearing loss highlights the system’s translational potential.

A key aspect of this study was the use of high-resolution intracranial recordings, which provided the signal quality needed to answer the central question of whether a brain-controlled hearing system can yield behavioral benefits. Although iEEG is not an immediate practical solution for widespread use, it was a scientifically necessary choice to establish a gold-standard benchmark for the field. This approach is consistent with other grand challenge problems in BCI research, including recent landmark studies in speech prosthesis development^35–38. Our findings define an upper bound on AAD performance—what is achievable when attentional intent is decoded with high fidelity—and offer a reference point for future, less invasive systems that can be scaled to larger populations. At the same time, the neurotechnology landscape is evolving rapidly toward implantable devices for a variety of neurological and psychiatric conditions, including epilepsy³⁹, depression⁴⁰ and movement disorders⁴¹. Given that disabling hearing loss negatively affects more than 430 million people worldwide⁴², a safe, implantable solution is both plausible and proportionate in scope. This work provides a direct precursor to such a system and establishes a critical benchmark for future translational efforts.

Validation in listeners with hearing loss

Our supplementary validation study provides important evidence for the clinical potential of this technology. Participants with hearing impairment listened to audio that had been dynamically modulated in real time based on the neural signals of a separate user, resulting in moment-by-moment gain adjustments. These brain-driven modulations are inherently variable and would not necessarily improve perception if they were unstable or mistimed²⁰. The finding that listeners with hearing impairment not only preferred the brain-driven audio but also derived a greater objective benefit than individuals with normal hearing is a significant outcome. Consistent with this finding, we observed that objective intelligibility gains were most pronounced for individuals with moderate-to-severe hearing loss, whereas participants with mild hearing loss who operated near the ceiling of performance showed smaller gains, even though they consistently preferred the system-enabled condition. The feasibility of a future closed-loop system for this population is supported by prior work establishing that the necessary neural signatures for AAD are robustly present in hearing-impaired listeners^18,33,34, and that decoding can remain effective even when speech intelligibility is severely degraded^23,43,44. Together, these findings suggest that closed-loop neural decoding systems are viable for this population and may offer the greatest objective benefit to those with the most to gain. As such, they represent a promising direction for the future of assistive hearing technology.

Real-world viability with mixed audio

A critical prerequisite for the translation of any AAD-based hearing device is the ability to operate on mixed audio signals, as clean source signals are unavailable in real-world environments. Although this has long posed a barrier for the field, recent breakthroughs in real-time speech separation^45–49 have made this goal increasingly achievable. Prior work has shown that AAD can be integrated with low-latency separation algorithms to operate on mixed audio without a significant loss in decoding accuracy^{12–14,50–53}. In the present study, we used separated sources strategically to isolate and definitively answer the more fundamental question of whether a brain-controlled system could provide behavioral benefit. To confirm the broader viability of this approach, a control analysis showed that decoding performance was virtually unchanged when using algorithmically separated sources⁴⁶ instead of clean sources (r = 0.97, P < 0.001; Extended Data Fig. 10). Nevertheless, significant challenges remain in ensuring robust performance in complex, real-world conditions, including highly reverberant spaces and scenes with multiple competing talkers⁵⁴. In such scenarios, speech extraction, focusing on enhancing a target speaker rather than fully separating all sources, may offer a more practical and computationally efficient solution⁵¹. The translational potential for AAD-based systems is further supported by ongoing progress in deploying machine learning-based enhancement methods on modern, power-efficient hearing aid hardware, making real-time implementation of AI-driven auditory interfaces increasingly feasible⁵⁵.

Extended Data Fig. 10 — Scatter plot comparing AAD accuracy using clean speech envelopes (cAAD) versus algorithmically separated envelopes (sAAD) across trials. Each point represents a trial from a single subject. The strong linear relationship (Pearson’s r = 0.974, R² = 0.948) indicates that decoding performance is nearly identical between clean and separated audio conditions. Dashed line shows the line of identity (y = x), confirming that gain decisions in real-time deployment can rely on separated audio without degrading accuracy.

Source data

Neural mechanisms and decoding strategies

In this study, we deliberately used a simple linear decoding model^5,22, chosen for its computational efficiency, transparency and robustness in real-time closed-loop operation. Given the high signal-to-noise ratio of iEEG recordings, this approach allowed us to reconstruct the speech envelope from broadband neural responses and compare it to candidate streams using Pearson correlation, a well-established and interpretable framework in auditory neuroscience. Using a lightweight model also minimized the risk of instability or unintended behavior in the feedback loop, ensuring that any observed perceptual benefit could be directly attributed to attentional decoding. Our results further showed that trial-by-trial variations in AAD accuracy were significantly correlated with user preference, reinforcing the importance of decoder performance for perceptual outcomes. To rule out non-auditory confounds, we confirmed that neither the visual cue (experiment 2) nor the motor pointing behavior (experiment 3) correlated with the actual or reconstructed speech envelopes (Extended Data Fig. 8). Moreover, most behavioral outcomes were collected in experiment 1, during which no motor input was present and the visual arrow remained fixed throughout each trial. Finally, we observed that informative electrodes were not restricted to the left superior temporal gyrus but were also found in the right hemisphere and dorsal frontoparietal regions (Fig. 1c). This suggests that the linear decoder can leverage signals from a distributed cortical network, probably recruiting higher-order attentional resources in addition to sensory processing, which contributes to the robustness of the system.

Building on the importance of decoder fidelity, nonlinear and end-to-end models have shown clear potential for improving decoding performance and speed, particularly in lower signal-to-noise ratio environments or more spatially constrained recording settings^{12,34,56–62}. However, we note that more complex nonlinear models often incur higher computational costs, which could increase system latency to a point that diminishes the behavioral benefit. Other approaches have also emerged, such as decoding based on distinct neural population responses rather than temporal correlation between stimulus and response⁶³. These methods may help extend AAD to a broader range of signal modalities, including emerging minimally invasive and wearable technologies that fall between traditional invasive and non-invasive categories⁶⁴. Theoretical work further suggests that current AAD algorithms operate below information-theoretic limits⁶⁵, pointing to substantial headroom for future gains.

System latency and stability constraints

A deeper understanding of how decoding accuracy, latency and model complexity translate into perceptual benefit will be essential for guiding system design and enabling clinical translation. For example, the 5.1 s switching time observed in our attention-switching task is not a neurophysiological limit⁸, but a direct consequence of parameters chosen to prioritize stability: a 4 s decoding window contributes an inherent 2 s latency, and a five-state Markov smoothing filter adds more delay to the transition time. This illustrates the fundamental speed–accuracy trade-off that must be carefully optimized in any practical AAD system. The implementation of such systems must also carefully balance speech enhancement with environmental awareness. Our findings suggest that an approximate gain adjustment of +6 dB in TMR significantly improves intelligibility without fully isolating the listener from their surroundings. However, real-world applications will introduce additional complexities, including complex sound scenes with non-speech sources and unpredictable attention shifts^66,67. Addressing these challenges through further experimental and computational refinement will be vital for successful deployment. We also acknowledge that the fixed trial structure in experiment 1 (‘system off’ followed by ‘system on’) introduces a potential confound regarding attentional build-up. Although we excluded the initial 10 s to account for the rapid formation of auditory streams, it is possible that reduced listening effort in the second segment partially reflects temporal adaptation to the task. However, given the substantial magnitude of the improvement in both intelligibility (+12 dB TMR boost) and subjective preference, we attribute the primary benefit to the system-driven enhancement.

Neural decoding overcomes the limits of external cues

A natural question is whether simpler external cues, such as eye-gaze, head orientation or manual gestures, could provide the same steering information without neural sensing. Although these external cues can be useful in specific scenarios, each has fundamental limitations that a direct neural interface overcomes. Gaze direction, for example, fails when competing talkers are co-located, when there is no visual target accompanying the listening target (for example, visual occlusion or a public-address system). Furthermore, gaze and auditory attention routinely dissociate; listeners often covertly attend to one talker while looking elsewhere⁶⁸. Moreover, as listening becomes effortful, eye movements decrease and spatial gaze width narrows, increasing the difficulty of inferring where attention is deployed from gaze⁶⁹. In live conversations, gaze also serves social functions, regulating turn-taking⁷⁰, reflecting one’s speaking and listening role, which changes with group size⁷¹, further weakening any mapping from gaze direction to attentional focus. The presence of individual gaze differences adds another layer of unpredictability⁷². Related external cues face similar issues: head pose and pointing gestures are constrained by visibility, social norms and task demands, and they impose an explicit behavioral burden that is fatiguing and ill-suited to rapid turn-taking. By contrast, neural attention decoding is covert, continuous and agnostic to spatial arrangement; it seamlessly follows the listener’s internal focus even when gaze and head pose provide no reliable spatial cue, and it requires no behavioral modification in addition to ordinary listening. Moreover, brain signals carry information about higher-order listening goals—what stream is semantically relevant and what sounds are to be suppressed—that eye or head trackers cannot access. Decoding those higher-level intentions opens a path toward future devices that not only amplify the currently attended talker but also can flexibly remix complex soundscapes⁷³. By demonstrating that brain-based steering delivers a perceptual benefit under realistic conditions, we lay the critical groundwork for next-generation assistive technologies that adapt to the listener’s true cognitive intent rather than to an observable, and often unreliable, surrogate.

Subject experience and system perception

In addition to the quantitative behavioral metrics, participant feedback provided valuable insights into the usability of the closed-loop system (Supplementary Video 5). In post-experiment debriefings, users consistently described the adaptive gain control as feeling natural and intuitive, noting that they could shift their attention freely without perceiving the system as resistant or lagging. Crucially, participants reported that the enhanced audio aligned well with their intended focus, reinforcing the objective finding that the decoder accurately tracked their attentional state. These qualitative reports suggest that despite the inherent technical latency of the system, the perceptual experience remained seamless, a critical factor for the long-term adoption of any future neuro-steered hearing technology.

This work also opens broader avenues for integrating cognitive states into assistive hearing technologies, shifting the paradigm from passive amplification to active, personalized, intent-driven auditory enhancement. By establishing the critical performance and viability benchmark for a functional auditory BCI, this study provides a principled framework for designing future systems that may incorporate additional cognitive dimensions, such as perceived effort, listening intent or situational context, enabled by emerging AI frameworks that fuse neural decoding with adaptive machine listening⁶³. By aligning auditory processing more closely with the user’s internal goals, this approach offers a path toward more intuitive, responsive and effective hearing support in everyday environments.

Methods

Inclusion and ethics

All participants provided informed written consent before participation in accordance with protocols approved by Columbia University’s Institutional Review Board and the respective hospitals’ ethics committees. Participants in the iEEG experiments were adult patients undergoing clinical monitoring for epilepsy with implanted intracranial electrodes. The study adhered to all relevant ethical regulations and guidelines for research involving human participants.

Subjects

Four participants (S1–S4; three males aged 24, 29 and 38 years and one female aged 21 years) who were undergoing clinical monitoring for epilepsy with implanted intracranial electrodes (iEEG) were recruited for this study, following approval by the institutional review boards of the participating hospitals. No financial compensation was provided for participation in the experiments. Informed consent was obtained from all participants before enrollment. Electrode coverage varied across individuals. Subjects S1 and S3 had stereoelectroencephalography (sEEG)-depth electrodes and an electrocorticography grid over the left hemisphere, primarily targeting the temporal lobe. Subject S2 had sEEG-depth electrodes in the right hemisphere, while subject S4 had bilateral sEEG-depth coverage. Speech-responsive electrodes were identified based on neural activity recorded during the offline phase. High-gamma power (70–150 Hz) was extracted from each electrode, and speech responsiveness was assessed using t-tests comparing activity during speech versus silence. Based on this analysis, the number of speech-responsive electrodes used in subsequent analyses for each participant was 48 for S1, 38 for S2, 42 for S3 and 21 for S4.

For evaluation in individuals with hearing loss, 40 participants were recruited through the online platform Prolific. Participants were compensated USD $10 for approximately 30 min of participation. All participants were native English speakers. The mean age of participants was 40 years (s.d. = 13 years), with an age range of 20–73 years. The cohort included 25 male participants.

Experimental design and stimuli

The experiment consisted of two phases: an offline phase to collect training data for the AAD model, and an online phase in which the trained model was used in a real-time closed-loop setting. In both phases, participants listened to two spatially separated conversations presented simultaneously through external loudspeakers positioned at ±15° azimuth. Each conversation comprised scripted dialogue between two speakers who took turns discussing everyday topics (for example, food, travel), recorded by professional voice actors to ensure naturalistic but controlled stimuli.

To control acoustic difficulty, trials varied in the gender composition of the competing talkers. Talker pairs were selected to produce three levels of overlap: easy (male vs female), medium (mixed-gender across conversations) and hard (same-gender talkers). The distribution was balanced across the experiment. All dialogues were matched for duration and loudness across conditions. To simulate realistic acoustic environments, diotic background noise was added to each trial, using either pedestrian noise or multi-talker babble (randomized across trials). The background noise level was set 15 dB below the target speech in trials with 0 dB TMR and remained constant across phases. Each trial also included a one-back task: repeated words were inserted into both conversations, and participants were instructed to press a button whenever they detected a repetition in the attended stream. This ensured engagement and compliance with attentional instructions. A central display provided visual cues (an arrow) to instruct participants which stream to attend. In experiment 1, this cue remained static throughout the trial, whereas in experiment 2, it changed mid-trial to signal an attention switch. Additionally, in experiment 3, participants used a hand gesture to indicate their attentional focus for post hoc analysis. A potential concern is that neural activity related to these visual or motor events could contaminate the auditory decoding. However, this is theoretically implausible, as our stimulus reconstruction decoder is trained to reconstruct the temporal envelope of speech, a signal that is not temporally correlated with the change in visual cue (experiment 2) or motor action (experiment 3). We confirmed this empirically by showing a lack of significant correlation between the visual cue signal and the speech envelopes (P > 0.5; Extended Data Fig. 8).

To equalize loudness across trials and talkers, individual speech recordings were first normalized to a root-mean-square amplitude of 1.0, concatenated into full conversation streams and then scaled to −25 dBFS to prevent clipping and maintain consistent signal power. The final mixed audio was presented at 65 dB sound pressure level. In real-time conditions, the gain of each stream was dynamically adjusted to implement the desired TMR, while total acoustic power was held constant across transitions.

Offline phase: data collection and model training

The offline phase was designed to collect labeled training data for subject-specific AAD models. Participants completed 26 trials, each lasting approximately 38 s. To ensure the decoder would generalize across a range of signal conditions, the trial order was randomized across three TMR configurations: in ten trials, the TMR shifted from −6 dB to +9 dB; in another ten trials, it shifted from +9 dB to −6 dB; and in the remaining six trials, the TMR was held constant at 0 dB. At the start of each trial, a visual arrow cued the participant to attend to one of the two conversations, and no attentional switches were prompted. For analysis, data from trials with TMR changes were divided into two TMR-specific segments, while trials without TMR changes were treated as a single segment. Neural data were normalized across all segments, whereas stimulus envelopes were normalized within each segment. Decoders were then trained and evaluated using a leave-one-segment-out cross-validation procedure. Throughout this phase, participants performed the one-back repeat-word detection task, and their behavioral performance confirmed they reliably tracked the attended conversation (Extended Data Fig. 2).

Online experiment

The AAD model trained during the offline phase was deployed to decode attention and enhance the attended conversation in real time. Participants were informed that the online phase was a continuation of the listening task but were not told that their brain activity would control the system, ensuring a blinded experimental design. The online phase included three types of trials: binary forced-choice trials (experiment 1), instructed attention-switching trials (experiment 2) and self-initiated attention-switching trials (experiment 3).

Experiment 1: binary forced-choice trials

Participants completed 20 trials, each lasting 38 s. The initial 10 s of each trial served as an unrated stabilization period to allow for the build-up of auditory attention. Following this period, an on-screen prompt explicitly marked the beginning of ‘segment 1’ (the 14 s ‘system off’ period), during which the TMR was fixed at −6 dB. At the 24 s mark, a second prompt marked the beginning of ‘segment 2’ (the 14 s ‘system on’ period), during which the closed-loop system was activated, dynamically amplifying the attended talker by up to +9 dB TMR. At the end of each trial, participants were asked to rate which of these two explicitly marked segments was easier to follow and answered two multiple-choice comprehension questions. Throughout the trial, they also performed the one-back repeat-word detection task.

Experiment 2: instructed attention-switching trials

In ten additional trials, participants were cued to shift attention from one talker to the other midway through each trial. A continuously displayed arrow on the central screen indicated the cued talker and flipped direction at a predetermined time. Trials lasted 38–47 s. Speech began at 0 dB TMR, and closed-loop gain control was activated 5 s after onset to allow model stabilization before the switch.

Experiment 3: self-initiated switching and control condition

In a third condition, participants were allowed to switch attention between talkers freely and at will, without any external cue. To monitor attentional state, they pointed to the loudspeaker they were attending to, providing a behavioral reference for evaluating the decoder. These trials tested whether the system could track endogenous attention shifts and dynamically adjust enhancement in real time.

As a control condition, some of these trials included a reversal of the gain logic: the system was configured to amplify the unattended talker and suppress the attended one, without informing the participant.

At the end of the session, participants were debriefed and informed that their brain activity had controlled the gain modulation in real time. They were asked about their experience and whether they would consider using such a system in real-world listening scenarios.

Regression analyses of performance and preference

To understand the relationship between user engagement, system performance and subjective preference (Fig. 3), we performed two complementary regression analyses using mixed-effects models to account for within-subject correlations. First, to investigate the source of trial-by-trial variability in system performance (Fig. 3a), we used a linear mixed-effects model. We modeled single-trial AAD accuracy as the dependent variable, with the participant’s repeat-word detection score (a proxy for attentional engagement) as the fixed effect. A random intercept for each subject was included to account for individual differences in baseline decoding performance. Second, to quantify the impact of system performance on subjective preference (Fig. 3b), we used a GLMM. The binary preference outcome (prefer on vs prefer off) was modeled as a function of single-trial AAD accuracy (fixed effect), with a random intercept for each subject. The model specified a binomial distribution with a logit link function. For visualization purposes in Fig. 3b, trials were grouped into three bins based on AAD accuracy. We calculated the proportion of ‘prefer on’ responses within each bin (mean ± s.e.m.) to display the empirical probability of preference across the accuracy range without imposing a sigmoidal curve fit.

Pupillometry

Pupil size was used as a physiological proxy for listening effort, a well-established metric in auditory neuroscience^23,28–30. Larger pupil dilations are associated with increased cognitive load, while smaller dilations reflect reduced effort. In the context of AAD-driven speech enhancement, we reasoned that pupil dilation would be lower when the system successfully enhanced the attended stream (that is, under positive TMR conditions) compared to trials in which the system was inactive or misaligned (negative TMR).

Pupil diameter was recorded using a Tobii Pro Nano eye tracker (60 Hz sampling rate) positioned at chin level beneath the display. Raw traces from the left and right pupils were averaged and aligned to the audio onset. Baseline correction was performed for each trial by computing the mean dilation during the 1 s period preceding trial onset and rescaling the pupil data accordingly. Reliable pupillometry data were obtained from two participants (S2 and S3) across all 20 binary forced-choice trials. Data from the remaining two participants (S1 and S4) were excluded from analysis because of technical limitations in eye-tracker positioning that prevented consistent tracking.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Online content

Any methods, additional references, Nature Portfolio reporting summaries, source data, extended data, supplementary information, acknowledgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at https://doi.org/10.1038/s41593-026-02281-5.

Supplementary information

Supplementary Information^{(349.6KB, pdf)}

Comparison of the current study with prior art in real-time AAD.

Reporting Summary^{(64.3KB, pdf)}

Supplementary Video 1^{(3MB, mp4)}

Experiment 1: system turns on mid-trial.

Supplementary Video 2^{(2.9MB, mp4)}

Experiment 2: system adapts to participants switching attention on instruction.

Supplementary Video 3^{(3.1MB, mp4)}

Experiment 3: system adapts to user-initiated attention switching.

Supplementary Video 4^{(3.8MB, mp4)}

Reversing the system: system suppresses the attended and amplifies the unattended.

Supplementary Video 5^{(5MB, mp4)}

Patient testimonials after de-briefing.

Source data

Source Data^{(132.2KB, zip)}

Zip file of source data files for Figs. 1–5 and Extended Data Figs. 1–7, 9 and 10. Please see the Word file in the zip file for descriptions of the individual files.

Acknowledgements

This work was supported by the National Institute on Deafness and Other Communication Disorders (NIH-NIDCD) grants R01DC014279 (N.M.) and DC018805 (NM). Additional support was provided by grants from the Marie-Josée and Henry R. Kravis Foundation. The authors would also like to acknowledge M. Farinella and D. C. Powell for their help with illustrations and video editing.

Extended data

Author contributions

V.C. and N.M. conceived the project, designed the experiments, evaluated the results and wrote the manuscript. V.C. conducted the experiments with support from M.N., S.J., J.H., S.B., D.F., A.F. and E.F.C. E.F.C. and A.D.M. performed surgeries. N.M. supervised the project. All authors commented on the manuscript.

Peer review

Peer review information

Nature Neuroscience thanks Jonas Obleser and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Data availability

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available owing to privacy and ethical restrictions. Source data are provided with this paper.

Code availability

The code for stimulus reconstruction and the offline analyses presented in this study is publicly available as part of the NapLib toolbox^74,75. MATLAB R2023a was used to perform all data analysis. The custom code used to run the real-time, closed-loop online experiments has been made available on GitHub at https://github.com/naplab/Real-Time-Brain-Controlled-Hearing.

Competing interests

The authors declare no competing interests.

Footnotes

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

is available for this paper at https://doi.org/10.1038/s41593-026-02281-5.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41593-026-02281-5.

References

1.Chung, K. Challenges and recent developments in hearing aids: Part I. Speech understanding in noise, microphone technologies and noise reduction algorithms. Trends Amplif.8, 83–124 (2004). [DOI] [PMC free article] [PubMed] [Google Scholar]
2.Cherry, E. C. Some experiments on the recognition of speech, with one and with two ears. J. Acoust. Soc. Am.25, 975–979 (1953). [Google Scholar]
3.McDermott, J. H. The cocktail party problem. Curr. Biol.19, R1024–R1027 (2009). [DOI] [PubMed] [Google Scholar]
4.Shukla, A. et al. Hearing loss, loneliness, and social isolation: a systematic review. Otolaryngol. Neck Surg.162, 622–633 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
5.O’Sullivan, J. A. et al. Attentional selection in a cocktail party environment can be decoded from single-trial EEG. Cereb. Cortex25, 1697–1706 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Geirnaert, S. et al. Electroencephalography-based auditory attention decoding: toward neurosteered hearing devices. IEEE Signal Process. Mag.38, 89–102 (2021). [Google Scholar]
7.Power, A. J., Foxe, J. J., Forde, E. J., Reilly, R. B. & Lalor, E. C. At what time is the cocktail party? A late locus of selective attention to natural speech. Eur. J. Neurosci.35, 1497–1503 (2012). [DOI] [PubMed] [Google Scholar]
8.Mesgarani, N. & Chang, E. F. Selective cortical representation of attended speaker in multi-talker speech perception. Nature485, 233–236 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Ding, N. & Simon, J. Z. Emergence of neural encoding of auditory objects while listening to competing speakers. Proc. Natl Acad. Sci. USA109, 11854–11859 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Golumbic, E. M. Z. et al. Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party”. Neuron77, 980–991 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
11.de Cheveigné, A. et al. Decoding the auditory brain with canonical component analysis. Neuroimage172, 206–216 (2018). [DOI] [PubMed] [Google Scholar]
12.Pan, Z., Borsdorf, M., Cai, S., Schultz, T. & Li, H. NeuroHeed: neuro-steered speaker extraction using EEG signals. IEEEACM Trans. Audio Speech Lang. Process.32, 4456–4470 (2024). [Google Scholar]
13.O’Sullivan, J. et al. Neural decoding of attentional selection in multi-speaker environments without access to clean sources. J. Neural Eng.14, 056001 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Han, C. et al. Speaker-independent auditory attention decoding without access to clean speech sources. Sci. Adv.5, eaav6134 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Ha, J., Baek, S.-C., Lim, Y. & Chung, J. H. Validation of cost-efficient EEG experimental setup for neural tracking in an auditory attention task. Sci. Rep.13, 22682 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Baek, S.-C., Chung, J. H. & Lim, Y. Implementation of an online auditory attention detection model with electroencephalography in a dichotomous listening experiment. Sensors21, 531 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Zink, R., Baptist, A., Bertrand, A., Van Huffel, S. & De Vos, M. On-line detection of auditory attention in a neurofeedback application. In Proc. 8th International Workshop on Biosignal Interpretation (eds Yamamoto, Y. et al.) 1–4 (2016); https://www.p.u-tokyo.ac.jp/~bsi2016/proceedings.php
18.Hjortkjær, J. et al. Real-time control of a hearing instrument with EEG-based attention decoding. J. Neural Eng.22, 016027 (2025). [DOI] [PubMed] [Google Scholar]
19.Haro, S., Beauchene, C., Quatieri, T. F., & Smalt, C. J. A brain-computer interface for improving auditory attention in multi-talker environments. IEEE Access 10.1109/ACCESS.2025.3623842 (2025). [DOI] [PMC free article] [PubMed]
20.Aroudi, A., Fischer, E., Serman, M., Puder, H. & Doclo, S. Closed-loop cognitive-driven gain control of competing sounds using auditory attention decoding. Algorithms14, 287 (2021). [Google Scholar]
21.O’Sullivan, J. et al. Hierarchical encoding of attended auditory objects in multi-talker speech perception. Neuron104, 1195–1209.e3 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Mesgarani, N., David, S. V., Fritz, J. B. & Shamma, S. A. Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. J. Neurophysiol.102, 3329–3339 (2009). [DOI] [PMC free article] [PubMed] [Google Scholar]
23.He, X., Raghavan, V. X. & Mesgarani, N. Neural speech tracking in noise reflects the opposingin fluence of SNR on intelligibility and attentional effort. Imaging Neurosci. (Camb.)126, IMAG.a.126 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Geirnaert, S., Francart, T. & Bertrand, A. An interpretable performance metric for auditory attention decoding algorithms in a context of neuro-steered gain control. IEEE Trans. Neural Syst. Rehabil. Eng.28, 307–317 (2020). [DOI] [PubMed] [Google Scholar]
25.Elhilali, M., Xiang, J., Shamma, S. A. & Simon, J. Z. Interaction between attention and bottom-up saliency mediates the representation of foreground and background in an auditory scene. PLoS Biol.7, e1000129 (2009). [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Lu, K., Dutta, K., Mohammed, A., Elhilali, M. & Shamma, S. Temporal-coherence induces binding of responses to sound sequences in ferret auditory cortex. iScience28, 111991 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]
27.Brungart, D. S., Simpson, B. D., Ericson, M. A. & Scott, K. R. Informational and energetic masking effects in the perception of multiple simultaneous talkers. J. Acoust. Soc. Am.110, 2527–2538 (2001). [DOI] [PubMed] [Google Scholar]
28.Egeth, H. & Kahneman, D. Attention and effort. Am. J. Psychol.88, 339 (1975). [Google Scholar]
29.Neagu, M.-B. et al. Investigating the reliability of pupillometry as a measure of individualized listening effort. Trends Hear.27, 23312165231153288 (2023). [Google Scholar]
30.McCloy, D. R., Larson, E. & Lee, A. K. C. Auditory attention switching with listening difficulty: behavioral and pupillometric measures. J. Acoust. Soc. Am.144, 2764–2771 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
31.Tang, X., Wu, J. & Shen, Y. The interactions of multisensory integration with endogenous and exogenous attention. Neurosci. Biobehav. Rev.61, 208–224 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
32.MacPherson, A. & Akeroyd, M. A. Variations in the slope of the psychometric functions for speech intelligibility: a systematic survey. Trends Hear.18, 2331216514537722 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Fuglsang, S. A., Märcher-Rørsted, J., Dau, T. & Hjortkjær, J. Effects of sensorineural hearing loss on cortical synchronization to competing speech during selective attention. J. Neurosci.40, 2562–2572 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
34.Tanveer, M. A., Skoglund, M. A., Bernhardsson, B. & Alickovic, E. Deep learning-based auditory attention decoding in listeners with hearing impairment. J. Neural Eng.21, 036022 (2024). [DOI] [PubMed] [Google Scholar]
35.Anumanchipalli, G. K., Chartier, J. & Chang, E. F. Speech synthesis from neural decoding of spoken sentences. Nature568, 493–498 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Willett, F. R. et al. A high-performance speech neuroprosthesis. Nature620, 1031–1036 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Metzger, S. L. et al. A high-performance neuroprosthesis for speech decoding and avatar control. Nature620, 1037–1046 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D. & Mesgarani, N. Towards reconstructing intelligible speech from the human auditory cortex. Sci. Rep.9, 874 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Chen, W.-M. et al. A fully integrated 8-channel closed-loop neural-prosthetic CMOS SoC for real-time epileptic seizure control. IEEE J. Solid-State Circuits49, 232–247 (2014). [Google Scholar]
40.Dandekar, M. P., Fenoy, A. J., Carvalho, A. F., Soares, J. C. & Quevedo, J. Deep brain stimulation for treatment-resistant depression: an integrative review of preclinical and clinical findings and translational implications. Mol. Psychiatry23, 1094–1112 (2018). [DOI] [PubMed] [Google Scholar]
41.Benabid, A. L. Deep brain stimulation for Parkinson’s disease. Curr. Opin. Neurobiol.13, 696–706 (2003). [DOI] [PubMed] [Google Scholar]
42.Mathers, C., Fat, D. M. & Boerma, J. T. The Global Burden of Disease: 2004 Update (World Health Organization, 2008).
43.He, X., Raghavan, V. S. & Mesgarani, N. Decoding the unintelligible: neural speech tracking in low signal-to-noise ratios. In ICASSP 2025: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 1–5 (IEEE, 2025); 10.1109/ICASSP49660.2025.10889263 [DOI]
44.Das, N., Bertrand, A. & Francart, T. EEG-based auditory attention detection: boundary conditions for background noise and speaker positions. J. Neural Eng.15, 066017 (2018). [DOI] [PubMed] [Google Scholar]
45.Luo, Y., Chen, Z. & Mesgarani, N. Speaker-independent speech separation with deep attractor network. IEEEACM Trans. Audio Speech Lang. Process.26, 787–796 (2018). [Google Scholar]
46.Luo, Y. & Mesgarani, N. Conv-TasNet: surpassing ideal time–frequency magnitude masking for speech separation. In IEEEACM Transactions on Audio, Speech, and Language Processing (ed Lee, T.) 1256–1266 (IEEE, 2019). [DOI] [PMC free article] [PubMed]
47.Hershey, J. R., Chen, Z., Le Roux, J. & Watanabe, S. Deep clustering: discriminative embeddings for segmentation and separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 31–35 (IEEE, 2016). [DOI] [PMC free article] [PubMed]
48.Han, C., Luo, Y. & Mesgarani, N. Binaural speech separation of moving speakers with preserved spatial cues. In Proc. Interspeech 2021 3505–3509 (ISCA, 2021); 10.21437/Interspeech.2021-1372 [DOI]
49.Han, C., Luo, Y. & Mesgarani, N. Online deep attractor network for real-time single-channel speech separation. In ICASSP 2019: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 361–365 (IEEE, 2019); 10.1109/ICASSP.2019.8682884 [DOI] [PMC free article] [PubMed]
50.Choudhari, V. et al. Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments. Adv. Sci.11, 2401379 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
51.Ceolini, E. et al. Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception. Neuroimage223, 117282 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
52.Borgström, B. J., Brandstein, M. S., Ciccarelli, G. A., Quatieri, T. F. & Smalt, C. J. Speaker separation in realistic noise environments with applications to a cognitively-controlled hearing aid. Neural Netw.140, 136–147 (2021). [DOI] [PubMed] [Google Scholar]
53.Van Eyndhoven, S., Francart, T. & Bertrand, A. EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses. IEEE Trans. Biomed. Eng.64, 1045–1056 (2017). [DOI] [PubMed] [Google Scholar]
54.Zmolikova, K. et al. Neural target speech extraction: An overview. IEEE Signal Process. Mag.40, 8–29 (2023). [Google Scholar]
55.Sonova, A. G. Phonak elevates sound quality with new Infinio portfolio, introducing a paradigm shift with real-time AI for speech clarity in noise in Audéo Sphere Infinio. Audiology23, 35 (2024). [Google Scholar]
56.Ciccarelli, G. et al. Comparison of two-talker attention decoding from EEG with nonlinear neural networks and linear methods. Sci. Rep.9, 11538 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
57.Vandecappelle, S. et al. EEG-based detection of the locus of auditory attention with convolutional neural networks. Elife10, e56481 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
58.EskandariNasab, M., Raeisi, Z., Lashaki, R. A. & Najafi, H. A GRU–CNN model for auditory attention detection using microstate and recurrence quantification analysis. Sci. Rep.14, 8861 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
59.Xu, Z. et al. Decoding selective auditory attention with EEG using a transformer model. Methods204, 410–417 (2022). [DOI] [PubMed] [Google Scholar]
60.Accou, B., Vanthornhout, J., Hamme, H. V. & Francart, T. Decoding of the speech envelope from EEG using the VLAAI deep neural network. Sci. Rep.13, 812 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
61.Su, E., Cai, S., Xie, L., Li, H. & Schultz, T. STAnet: a spatiotemporal attention network for decoding auditory spatial attention from EEG. IEEE Trans. Biomed. Eng.69, 2233–2242 (2022). [DOI] [PubMed] [Google Scholar]
62.Han, C., Choudhari, V., Li, Y. A. & Mesgarani, N. Improved decoding of attentional selection in multi-talker environments with self-supervised learned speech representation. In 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) 1–5 (IEEE, 2023); 10.1109/EMBC40787.2023.10340191 [DOI] [PubMed]
63.Jiang, X. et al. AAD-LLM: neural attention-driven auditory scene understanding. In Proc. 63rd Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers 10.18653/v1/2025.acl-long.1257 (Association for Computational Linguistics, 2025). [DOI]
64.Mahoney, T. B., Grayden, D. B. & John, S. E. Sub-scalp brain-computer interface device design and fabrication. Preprint at https://arxiv.org/abs/2504.12578 (2025).
65.Abeysekara, R., Smalt, C. J., Karunathilake, I. D., Simon, J. Z. & Babadi, B. Information-theoretic limits on the performance of auditory attention decoders. In 57th Asilomar Conference on Signals, Systems, and Computers 1479–1483 (IEEE, 2023).
66.Apoux, F. et al. Effects of hearing loss and hearing aids on the perception of natural sounds and soundscapes: a survey of hearing care professional opinions. Am. J. Audiol.34, 281–295 (2025). [DOI] [PubMed] [Google Scholar]
67.Lelic, D., Picou, E., Shafiro, V. & Lorenzi, C. Sounds of nature and hearing loss: a call to action. Ear Hear.46, 298–304 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]
68.Foulsham, T. & Kingstone, A. Covert orienting: the dark matter of social attention. Trends Cogn. Sci.29, 597–599 (2025). [DOI] [PubMed] [Google Scholar]
69.Cui, M. E. & Herrmann, B. Eye movements decrease during effortful speech listening. J. Neurosci.43, 5856–5869 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
70.Degutyte, Z. & Astell, A. The role of eye gaze in regulating turn taking in conversations: a systematized review of methods and findings. Front. Psychol.12, 616471 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
71.Maran, T. et al. Visual attention in real-world conversation: gaze patterns are modulated by communication and group size. Appl. Psychol.70, 1602–1627 (2021). [Google Scholar]
72.Rogers, S. L., Speelman, C. P., Guidetti, O. & Longmuir, M. Using dual eye tracking to uncover personal gaze patterns during social interaction. Sci. Rep.8, 4271 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
73.Jiang, X., Han, C., Li, Y. A. & Mesgarani, N. Listen, chat, and remix: text-guided soundscape remixing for enhanced auditory experience. IEEE J. Sel. Topics Signal Process. 10.1109/JSTSP.2025.3570103 (2025). [DOI]
74.Khalighinejad, B., Nagamine, T., Mehta, A. & Mesgarani, N. NAPLib: an open source toolbox for real-time and offline neural acoustic processing. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 846–850 (IEEE, 2017); 10.1109/ICASSP.2017.7952275 [DOI] [PMC free article] [PubMed]
75.Mischler, G., Raghavan, V., Keshishian, M. & Mesgarani, N. naplib-python: Neural acoustic data processing and analysis tools in Python. Softw. Impacts17, 100541 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Information^{(349.6KB, pdf)}

Comparison of the current study with prior art in real-time AAD.

Reporting Summary^{(64.3KB, pdf)}

Supplementary Video 1^{(3MB, mp4)}

Experiment 1: system turns on mid-trial.

Supplementary Video 2^{(2.9MB, mp4)}

Experiment 2: system adapts to participants switching attention on instruction.

Supplementary Video 3^{(3.1MB, mp4)}

Experiment 3: system adapts to user-initiated attention switching.

Supplementary Video 4^{(3.8MB, mp4)}

Reversing the system: system suppresses the attended and amplifies the unattended.

Supplementary Video 5^{(5MB, mp4)}

Patient testimonials after de-briefing.

Source Data^{(132.2KB, zip)}

Zip file of source data files for Figs. 1–5 and Extended Data Figs. 1–7, 9 and 10. Please see the Word file in the zip file for descriptions of the individual files.

Data Availability Statement

[CR1] 1.Chung, K. Challenges and recent developments in hearing aids: Part I. Speech understanding in noise, microphone technologies and noise reduction algorithms. Trends Amplif.8, 83–124 (2004). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR2] 2.Cherry, E. C. Some experiments on the recognition of speech, with one and with two ears. J. Acoust. Soc. Am.25, 975–979 (1953). [Google Scholar]

[CR3] 3.McDermott, J. H. The cocktail party problem. Curr. Biol.19, R1024–R1027 (2009). [DOI] [PubMed] [Google Scholar]

[CR4] 4.Shukla, A. et al. Hearing loss, loneliness, and social isolation: a systematic review. Otolaryngol. Neck Surg.162, 622–633 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR5] 5.O’Sullivan, J. A. et al. Attentional selection in a cocktail party environment can be decoded from single-trial EEG. Cereb. Cortex25, 1697–1706 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR6] 6.Geirnaert, S. et al. Electroencephalography-based auditory attention decoding: toward neurosteered hearing devices. IEEE Signal Process. Mag.38, 89–102 (2021). [Google Scholar]

[CR7] 7.Power, A. J., Foxe, J. J., Forde, E. J., Reilly, R. B. & Lalor, E. C. At what time is the cocktail party? A late locus of selective attention to natural speech. Eur. J. Neurosci.35, 1497–1503 (2012). [DOI] [PubMed] [Google Scholar]

[CR8] 8.Mesgarani, N. & Chang, E. F. Selective cortical representation of attended speaker in multi-talker speech perception. Nature485, 233–236 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR9] 9.Ding, N. & Simon, J. Z. Emergence of neural encoding of auditory objects while listening to competing speakers. Proc. Natl Acad. Sci. USA109, 11854–11859 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR10] 10.Golumbic, E. M. Z. et al. Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party”. Neuron77, 980–991 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR11] 11.de Cheveigné, A. et al. Decoding the auditory brain with canonical component analysis. Neuroimage172, 206–216 (2018). [DOI] [PubMed] [Google Scholar]

[CR12] 12.Pan, Z., Borsdorf, M., Cai, S., Schultz, T. & Li, H. NeuroHeed: neuro-steered speaker extraction using EEG signals. IEEEACM Trans. Audio Speech Lang. Process.32, 4456–4470 (2024). [Google Scholar]

[CR13] 13.O’Sullivan, J. et al. Neural decoding of attentional selection in multi-speaker environments without access to clean sources. J. Neural Eng.14, 056001 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR14] 14.Han, C. et al. Speaker-independent auditory attention decoding without access to clean speech sources. Sci. Adv.5, eaav6134 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR15] 15.Ha, J., Baek, S.-C., Lim, Y. & Chung, J. H. Validation of cost-efficient EEG experimental setup for neural tracking in an auditory attention task. Sci. Rep.13, 22682 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR16] 16.Baek, S.-C., Chung, J. H. & Lim, Y. Implementation of an online auditory attention detection model with electroencephalography in a dichotomous listening experiment. Sensors21, 531 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR17] 17.Zink, R., Baptist, A., Bertrand, A., Van Huffel, S. & De Vos, M. On-line detection of auditory attention in a neurofeedback application. In Proc. 8th International Workshop on Biosignal Interpretation (eds Yamamoto, Y. et al.) 1–4 (2016); https://www.p.u-tokyo.ac.jp/~bsi2016/proceedings.php

[CR18] 18.Hjortkjær, J. et al. Real-time control of a hearing instrument with EEG-based attention decoding. J. Neural Eng.22, 016027 (2025). [DOI] [PubMed] [Google Scholar]

[CR19] 19.Haro, S., Beauchene, C., Quatieri, T. F., & Smalt, C. J. A brain-computer interface for improving auditory attention in multi-talker environments. IEEE Access 10.1109/ACCESS.2025.3623842 (2025). [DOI] [PMC free article] [PubMed]

[CR20] 20.Aroudi, A., Fischer, E., Serman, M., Puder, H. & Doclo, S. Closed-loop cognitive-driven gain control of competing sounds using auditory attention decoding. Algorithms14, 287 (2021). [Google Scholar]

[CR21] 21.O’Sullivan, J. et al. Hierarchical encoding of attended auditory objects in multi-talker speech perception. Neuron104, 1195–1209.e3 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR22] 22.Mesgarani, N., David, S. V., Fritz, J. B. & Shamma, S. A. Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. J. Neurophysiol.102, 3329–3339 (2009). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR23] 23.He, X., Raghavan, V. X. & Mesgarani, N. Neural speech tracking in noise reflects the opposingin fluence of SNR on intelligibility and attentional effort. Imaging Neurosci. (Camb.)126, IMAG.a.126 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR24] 24.Geirnaert, S., Francart, T. & Bertrand, A. An interpretable performance metric for auditory attention decoding algorithms in a context of neuro-steered gain control. IEEE Trans. Neural Syst. Rehabil. Eng.28, 307–317 (2020). [DOI] [PubMed] [Google Scholar]

[CR25] 25.Elhilali, M., Xiang, J., Shamma, S. A. & Simon, J. Z. Interaction between attention and bottom-up saliency mediates the representation of foreground and background in an auditory scene. PLoS Biol.7, e1000129 (2009). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR26] 26.Lu, K., Dutta, K., Mohammed, A., Elhilali, M. & Shamma, S. Temporal-coherence induces binding of responses to sound sequences in ferret auditory cortex. iScience28, 111991 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR27] 27.Brungart, D. S., Simpson, B. D., Ericson, M. A. & Scott, K. R. Informational and energetic masking effects in the perception of multiple simultaneous talkers. J. Acoust. Soc. Am.110, 2527–2538 (2001). [DOI] [PubMed] [Google Scholar]

[CR28] 28.Egeth, H. & Kahneman, D. Attention and effort. Am. J. Psychol.88, 339 (1975). [Google Scholar]

[CR29] 29.Neagu, M.-B. et al. Investigating the reliability of pupillometry as a measure of individualized listening effort. Trends Hear.27, 23312165231153288 (2023). [Google Scholar]

[CR30] 30.McCloy, D. R., Larson, E. & Lee, A. K. C. Auditory attention switching with listening difficulty: behavioral and pupillometric measures. J. Acoust. Soc. Am.144, 2764–2771 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR31] 31.Tang, X., Wu, J. & Shen, Y. The interactions of multisensory integration with endogenous and exogenous attention. Neurosci. Biobehav. Rev.61, 208–224 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR32] 32.MacPherson, A. & Akeroyd, M. A. Variations in the slope of the psychometric functions for speech intelligibility: a systematic survey. Trends Hear.18, 2331216514537722 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR33] 33.Fuglsang, S. A., Märcher-Rørsted, J., Dau, T. & Hjortkjær, J. Effects of sensorineural hearing loss on cortical synchronization to competing speech during selective attention. J. Neurosci.40, 2562–2572 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR34] 34.Tanveer, M. A., Skoglund, M. A., Bernhardsson, B. & Alickovic, E. Deep learning-based auditory attention decoding in listeners with hearing impairment. J. Neural Eng.21, 036022 (2024). [DOI] [PubMed] [Google Scholar]

[CR35] 35.Anumanchipalli, G. K., Chartier, J. & Chang, E. F. Speech synthesis from neural decoding of spoken sentences. Nature568, 493–498 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR36] 36.Willett, F. R. et al. A high-performance speech neuroprosthesis. Nature620, 1031–1036 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR37] 37.Metzger, S. L. et al. A high-performance neuroprosthesis for speech decoding and avatar control. Nature620, 1037–1046 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR38] 38.Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D. & Mesgarani, N. Towards reconstructing intelligible speech from the human auditory cortex. Sci. Rep.9, 874 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR39] 39.Chen, W.-M. et al. A fully integrated 8-channel closed-loop neural-prosthetic CMOS SoC for real-time epileptic seizure control. IEEE J. Solid-State Circuits49, 232–247 (2014). [Google Scholar]

[CR40] 40.Dandekar, M. P., Fenoy, A. J., Carvalho, A. F., Soares, J. C. & Quevedo, J. Deep brain stimulation for treatment-resistant depression: an integrative review of preclinical and clinical findings and translational implications. Mol. Psychiatry23, 1094–1112 (2018). [DOI] [PubMed] [Google Scholar]

[CR41] 41.Benabid, A. L. Deep brain stimulation for Parkinson’s disease. Curr. Opin. Neurobiol.13, 696–706 (2003). [DOI] [PubMed] [Google Scholar]

[CR42] 42.Mathers, C., Fat, D. M. & Boerma, J. T. The Global Burden of Disease: 2004 Update (World Health Organization, 2008).

[CR43] 43.He, X., Raghavan, V. S. & Mesgarani, N. Decoding the unintelligible: neural speech tracking in low signal-to-noise ratios. In ICASSP 2025: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 1–5 (IEEE, 2025); 10.1109/ICASSP49660.2025.10889263 [DOI]

[CR44] 44.Das, N., Bertrand, A. & Francart, T. EEG-based auditory attention detection: boundary conditions for background noise and speaker positions. J. Neural Eng.15, 066017 (2018). [DOI] [PubMed] [Google Scholar]

[CR45] 45.Luo, Y., Chen, Z. & Mesgarani, N. Speaker-independent speech separation with deep attractor network. IEEEACM Trans. Audio Speech Lang. Process.26, 787–796 (2018). [Google Scholar]

[CR46] 46.Luo, Y. & Mesgarani, N. Conv-TasNet: surpassing ideal time–frequency magnitude masking for speech separation. In IEEEACM Transactions on Audio, Speech, and Language Processing (ed Lee, T.) 1256–1266 (IEEE, 2019). [DOI] [PMC free article] [PubMed]

[CR47] 47.Hershey, J. R., Chen, Z., Le Roux, J. & Watanabe, S. Deep clustering: discriminative embeddings for segmentation and separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 31–35 (IEEE, 2016). [DOI] [PMC free article] [PubMed]

[CR48] 48.Han, C., Luo, Y. & Mesgarani, N. Binaural speech separation of moving speakers with preserved spatial cues. In Proc. Interspeech 2021 3505–3509 (ISCA, 2021); 10.21437/Interspeech.2021-1372 [DOI]

[CR49] 49.Han, C., Luo, Y. & Mesgarani, N. Online deep attractor network for real-time single-channel speech separation. In ICASSP 2019: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 361–365 (IEEE, 2019); 10.1109/ICASSP.2019.8682884 [DOI] [PMC free article] [PubMed]

[CR50] 50.Choudhari, V. et al. Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments. Adv. Sci.11, 2401379 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR51] 51.Ceolini, E. et al. Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception. Neuroimage223, 117282 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR52] 52.Borgström, B. J., Brandstein, M. S., Ciccarelli, G. A., Quatieri, T. F. & Smalt, C. J. Speaker separation in realistic noise environments with applications to a cognitively-controlled hearing aid. Neural Netw.140, 136–147 (2021). [DOI] [PubMed] [Google Scholar]

[CR53] 53.Van Eyndhoven, S., Francart, T. & Bertrand, A. EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses. IEEE Trans. Biomed. Eng.64, 1045–1056 (2017). [DOI] [PubMed] [Google Scholar]

[CR54] 54.Zmolikova, K. et al. Neural target speech extraction: An overview. IEEE Signal Process. Mag.40, 8–29 (2023). [Google Scholar]

[CR55] 55.Sonova, A. G. Phonak elevates sound quality with new Infinio portfolio, introducing a paradigm shift with real-time AI for speech clarity in noise in Audéo Sphere Infinio. Audiology23, 35 (2024). [Google Scholar]

[CR56] 56.Ciccarelli, G. et al. Comparison of two-talker attention decoding from EEG with nonlinear neural networks and linear methods. Sci. Rep.9, 11538 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR57] 57.Vandecappelle, S. et al. EEG-based detection of the locus of auditory attention with convolutional neural networks. Elife10, e56481 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR58] 58.EskandariNasab, M., Raeisi, Z., Lashaki, R. A. & Najafi, H. A GRU–CNN model for auditory attention detection using microstate and recurrence quantification analysis. Sci. Rep.14, 8861 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR59] 59.Xu, Z. et al. Decoding selective auditory attention with EEG using a transformer model. Methods204, 410–417 (2022). [DOI] [PubMed] [Google Scholar]

[CR60] 60.Accou, B., Vanthornhout, J., Hamme, H. V. & Francart, T. Decoding of the speech envelope from EEG using the VLAAI deep neural network. Sci. Rep.13, 812 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR61] 61.Su, E., Cai, S., Xie, L., Li, H. & Schultz, T. STAnet: a spatiotemporal attention network for decoding auditory spatial attention from EEG. IEEE Trans. Biomed. Eng.69, 2233–2242 (2022). [DOI] [PubMed] [Google Scholar]

[CR62] 62.Han, C., Choudhari, V., Li, Y. A. & Mesgarani, N. Improved decoding of attentional selection in multi-talker environments with self-supervised learned speech representation. In 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) 1–5 (IEEE, 2023); 10.1109/EMBC40787.2023.10340191 [DOI] [PubMed]

[CR63] 63.Jiang, X. et al. AAD-LLM: neural attention-driven auditory scene understanding. In Proc. 63rd Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers 10.18653/v1/2025.acl-long.1257 (Association for Computational Linguistics, 2025). [DOI]

[CR64] 64.Mahoney, T. B., Grayden, D. B. & John, S. E. Sub-scalp brain-computer interface device design and fabrication. Preprint at https://arxiv.org/abs/2504.12578 (2025).

[CR65] 65.Abeysekara, R., Smalt, C. J., Karunathilake, I. D., Simon, J. Z. & Babadi, B. Information-theoretic limits on the performance of auditory attention decoders. In 57th Asilomar Conference on Signals, Systems, and Computers 1479–1483 (IEEE, 2023).

[CR66] 66.Apoux, F. et al. Effects of hearing loss and hearing aids on the perception of natural sounds and soundscapes: a survey of hearing care professional opinions. Am. J. Audiol.34, 281–295 (2025). [DOI] [PubMed] [Google Scholar]

[CR67] 67.Lelic, D., Picou, E., Shafiro, V. & Lorenzi, C. Sounds of nature and hearing loss: a call to action. Ear Hear.46, 298–304 (2025). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR68] 68.Foulsham, T. & Kingstone, A. Covert orienting: the dark matter of social attention. Trends Cogn. Sci.29, 597–599 (2025). [DOI] [PubMed] [Google Scholar]

[CR69] 69.Cui, M. E. & Herrmann, B. Eye movements decrease during effortful speech listening. J. Neurosci.43, 5856–5869 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR70] 70.Degutyte, Z. & Astell, A. The role of eye gaze in regulating turn taking in conversations: a systematized review of methods and findings. Front. Psychol.12, 616471 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR71] 71.Maran, T. et al. Visual attention in real-world conversation: gaze patterns are modulated by communication and group size. Appl. Psychol.70, 1602–1627 (2021). [Google Scholar]

[CR72] 72.Rogers, S. L., Speelman, C. P., Guidetti, O. & Longmuir, M. Using dual eye tracking to uncover personal gaze patterns during social interaction. Sci. Rep.8, 4271 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR73] 73.Jiang, X., Han, C., Li, Y. A. & Mesgarani, N. Listen, chat, and remix: text-guided soundscape remixing for enhanced auditory experience. IEEE J. Sel. Topics Signal Process. 10.1109/JSTSP.2025.3570103 (2025). [DOI]

[CR74] 74.Khalighinejad, B., Nagamine, T., Mehta, A. & Mesgarani, N. NAPLib: an open source toolbox for real-time and offline neural acoustic processing. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 846–850 (IEEE, 2017); 10.1109/ICASSP.2017.7952275 [DOI] [PMC free article] [PubMed]

[CR75] 75.Mischler, G., Raghavan, V., Keshishian, M. & Mesgarani, N. naplib-python: Neural acoustic data processing and analysis tools in Python. Softw. Impacts17, 100541 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Real-time brain-controlled selective hearing enhances speech perception in multi-talker environments

Vishal Choudhari

Maximilian Nentwich

Sarah Johnson

Jose L Herrero

Stephan Bickel

Ashesh D Mehta

Daniel Friedman

Adeen Flinker

Edward F Chang

Nima Mesgarani

Abstract

Main

Results

Fig. 1. Experimental paradigm and offline AAD performance.

Extended Data Fig. 1. Electrode coverage across participants.

Extended Data Fig. 2. Behavioral performance confirms attentional compliance during repeat-word detection.

Extended Data Fig. 3. AAD decoding accuracy as a function of integration window size and frequency band.

Extended Data Fig. 4. Spatial distribution of electrode-level decoding accuracy.

Extended Data Fig. 5. Average AAD decoding accuracy across participants for the offline phase.

Experiment 1: real-time AAD provides significant and multifaceted perceptual benefits

Fig. 2. Real-time, brain-controlled hearing enhancement improves speech perception and reduces listening effort.

Extended Data Fig. 6. Real-time gain is systematically modulated by decoding strength.

Extended Data Fig. 7. System performance is robust across trial difficulty and background noise conditions.

Fig. 3. The user’s attentional state is a key source of system performance variability, which in turn predicts subjective benefit.

Experiment 2: the system tracks instructed attention switches

Fig. 4. The AAD system successfully tracks instructed shifts in a listener’s attention.

Experiment 3: the system successfully tracks self-initiated attention switches

Fig. 5. The AAD system successfully tracks natural, self-initiated shifts in a listener’s attention.

Extended Data Fig. 8. Visual and motor cues do not confound AAD decoding.

Evaluation by individuals with hearing loss

Extended Data Fig. 9. Perceptual validation of the system using individuals with hearing loss.

Discussion

Establishing a benchmark for brain-controlled hearing

Validation in listeners with hearing loss

Real-world viability with mixed audio

Extended Data Fig. 10. Decoding performance is preserved with separated audio sources.

Neural mechanisms and decoding strategies

System latency and stability constraints

Neural decoding overcomes the limits of external cues

Subject experience and system perception

Methods

Inclusion and ethics

Subjects

Experimental design and stimuli

Offline phase: data collection and model training

Online experiment

Experiment 1: binary forced-choice trials

Experiment 2: instructed attention-switching trials

Experiment 3: self-initiated switching and control condition

Regression analyses of performance and preference

Pupillometry

Reporting summary

Online content

Supplementary information

Source data

Acknowledgements

Extended data

Author contributions

Peer review

Peer review information

Data availability

Code availability

Competing interests

Footnotes

Extended data

Supplementary information

References

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases