A Fine-grained Spatiotemporal ECoG Dataset during Speech Perception in Tonal Language

Abstract

High-density intracranial recordings during naturalistic language processing are critical for advancing models of speech perception. However, open, well-annotated high-density ECoG resources for tonal languages such as Mandarin remain scarce. We present a publicly available high-density ECoG dataset from four participants undergoing awake craniotomy who listened to continuous, sentence-level Mandarin drawn from the Annotated Speech Corpus of Chinese Discourse (ASCCD). Signals were recorded with 128–256-channel subdural grids and synchronized with the audio; ECoG signals were down-sampled to 400 Hz, filtered in the high-gamma range (70–150 Hz), and used to derive high-gamma amplitude. The release follows BIDS-iEEG and is distributed as NWB files, with derivatives including high-gamma amplitude; word- and syllable-level alignments; Pinyin; lexical tone and stress tiers; prosodic break indices; mel-spectrograms; F0 and formants; and electrode localization on individual anatomy with projections to MNI space. This resource supports fine-grained investigations of lexical tone, syllabic structure, and higher-level linguistic representations during naturalistic listening.

Background & Summary

Rapid advances in cognitive neuroscience and brain-inspired computation demand neural recording methods that are both direct and high resolution. Among available approaches, intracranial recordings (e.g., electrocorticography, ECoG, and stereotactic EEG, sEEG) offer an irreplaceable window into human cortical function by providing millimeter-scale spatial sampling, millisecond temporal precision, and high signal-to-noise measurements via cortical surface or depth electrodes, which are more authentic and reliable than those obtained by non-invasive methods. These properties are crucial for elucidating the cortical computations that support speech perception and language processing^1–4.

Previous studies and datasets were mainly based on non-invasive techniques^5–10, which are constrained by limited spatial or temporal resolution, making it difficult to resolve the rapid neural dynamics within specified sub-regions of cortex, especially underlying naturalistic speech^5,11. High-density ECoG resolves rapid spectrotemporal dynamics and fine-grained functional organization within perisylvian language networks while maintaining broad cortical coverage^12–15. High-density ECoG studies of language vividly demonstrate the unique value of intracranial recordings for revealing neural mechanisms that are rarely accessible or verifiable with traditional non-invasive methods. In particular, they illuminated the production and perception of natural speech at the fine spatial and temporal scales characteristic of eloquent cortex^{12,13,15–18}, which enables the encoding of fast phonetic cues, prosodic structure, and—critically for Mandarin and other tonal languages—lexical pitch contours. Similarly, the fine-grained organization of semantic and syntactic processing observed with high-density ECoG has been difficult to establish using non-invasive neural recordings¹⁹, underscoring the indispensability of high-density ECoG for mechanistic studies of human language.

However, publicly available, well-annotated, and standardized high-density ECoG datasets capturing sentence-level auditory comprehension, remain exceedingly rare. This scarcity stems mainly from the clinical constraints that data can only be collected in medically indicated cases with electrode coverage dictated by clinical need, and their tasking time is limited. Robust public release also requires labor-intensive curation (e.g., time-aligned transcripts, tone/phoneme labels, electrode localization, and rich metadata). Additionally, many existing intracranial datasets prioritize constrained, non-naturalistic paradigms (e.g., isolated phonemes, syllables or single words), which are suboptimal for studying naturalistic language processing.

To address this gap, we publicly release a standardized, richly-annotated high-density ECoG dataset specifically targeting the perisylvian language network, obtained from four clinical participants during sentence-level auditory comprehension in Mandarin. Participants listened to a naturalistic yet rigorously controlled Mandarin corpus while audio and high-density ECoG were recorded simultaneously, enabling precise, sample-level alignment of neural and acoustic signals (Fig. 1a,b). The dataset is curated to mitigate the challenges outlined above. With de-identification and multi-tiered annotations (including precise word/syllable alignments, lexical tones, and prosodic features) providing time-locked, high-resolution intracranial signals during naturalistic sentence processing, this dataset offers an access for more profound analyses focusing on the perception neural mechanisms of naturalistic tonal sentences.

Fig. 1.

Schematic overview of the data acquisition and dataset composition. (a) High-density subdural electrocorticography (ECoG) grids were placed on the cortical surface according to clinical indications. Speech stimuli from the corpus were delivered via a sound amplifier, while neural signals were acquired with a multichannel amplifier linked to a digital signal processor (Tucker-Davis Technologies). A laptop simultaneously logged ECoG signals and ambient audio via the processor and a microphone, enabling precise alignment of neural activity to the audio stimuli for downstream analyses. (b) ECoG signals were first down-sampled to 400 Hz and filtered in the high-gamma range (70–150 Hz). High-gamma amplitude was then extracted using the Hilbert transform and temporally aligned to the corpus audio together with hierarchical linguistic annotations: tier 1, vowel/consonant markers; tier 2, Pinyin transcription; tier 3, Break Index (BI) indicating the prosodic hierarchy of Mandarin; tier 4, Stress (ST) reflecting Mandarin’s hierarchical stress patterns. (c) Participant 1 and Participant 2 (left hemisphere) completed 6 sessions selected from the corpus, and Participant 3 and Participant 4 (Right hemisphere) completed 4 sessions. Each session contained multiple 10–60 s corpus items selected from the ASCCD stimulus pool. The duration of each session is indicated in the panel. (d) The word and sentence counts were computed for every session, the mean counts were 800.3 words and 27.3 sentences, respectively.

Methods

Participants

This study included four participants (55-year-old male PA1 left-hemisphere, 47-year-old female PA2 left-hemisphere, 51-year-old male PA3 right-hemisphere, 34-year-old female PA4 right-hemisphere) undergoing awake language or motor mapping as part of their brain tumor surgeries²⁰ at Huashan Hospital, Shanghai, China. All the participants are native speakers of Mandarin, right-handed and left dominant (Fig. 1c).

The study was approved by the Huashan Hospital Institutional Review Board of Fudan University (HIRB, KY2017-437), and the conducting of research complied with all relevant ethical considerations. Participants were asked to participate in this study only when awake surgery is needed to protect eloquent area. Participants signed their names on informed consent the day before surgery and were told that the task was only for research purposes, which could be terminated at any time during the task.

Experimental task

Participants listened to natural, continuous Mandarin Chinese speech drawn from the Annotated Speech Corpus of Chinese Discourse (ASCCD; Chinese Linguistic Data Consortium, http://paslab.phonetics.org.cn/?p = 1763; see Corpus Information). The stimulus set comprised 68 10–60 s corpus items spanning diverse discourse structures (e.g., narrative, prose; Fig. 1c,d)²¹. These corpus items were separated by 0.5 s of silence to provide clear boundaries. The task was organized into four or six sessions of approximately 5 minutes each.

Corpus information

The Annotated Speech Corpus of Chinese Discourse (ASCCD)²¹ is a multi-component resource that comprises (i) discourse texts, (ii) studio-quality speech recordings, and (iii) expert phonetic annotation. It was developed to support research in phonetics and linguistics, the development of speech technologies, and foundational instruction in Standard Mandarin, with systematic design and validation to ensure scientific utility.

The corpus includes 18 articles spanning common discourse genres—narration, argumentation, reportage, and prose—read by 10 carefully screened native speakers of Standard Mandarin from the Beijing region (5 female, 5 male). Recordings are distributed as uncompressed WAV files (16 kHz sampling rate, 16-bit resolution, stereo), totaling approximately 1.5 GB. Manual phonetic annotations include Pinyin transcriptions, initial–final segmentation, and prosodic labels such as stress patterns.

In this study, we selected a subset of ASCCD corpus items to provide natural, continuous, sentence-level Mandarin speech stimuli for neural recording and analysis.

Data acquisition and signal processing

During awake craniotomy, one 128-channel high-density, subdural electrode grid (16 × 8) or two 128-channel grids (2 × 16 × 8), depending on clinical need, was temporarily placed onto the surface of the brain. Grid placement was performed by an experienced neurosurgeon, and the location was determined based on clinical exposure and avoidance of the tumor. ECoG data and audio sound (environment sound) were recorded simultaneously through the Tucker-Davis Technologies ECoG system (Fig. 1a). The ambient audio was used to align the neural recordings with the speech stimuli (Fig. 1a). The sampling rates of ECoG and audio are 3052 Hz and 24414 Hz, respectively.

The original broadband intraoperative ECoG voltage recordings are not publicly redistributed because they were acquired in a clinical surgical setting and are subject to patient-privacy and institutional data-protection restrictions. For the public release, the ECoG recordings were first down-sampled to 400 Hz. Line noise was then removed with notch filters at 50, 100, and 150 Hz. The down-sampled ECoG signals were further filtered in the high-gamma range of 70–150 Hz using eight Gaussian band-pass filters with logarithmically spaced center frequencies and semi-logarithmically increasing bandwidths. These 400-Hz, 70–150 Hz band-pass-filtered ECoG signals are provided in the NWB files.

High-gamma amplitude was extracted from the 400-Hz, 70–150 Hz band-pass-filtered ECoG signals using the Hilbert transform. The analytic amplitude envelopes from the filtered sub-bands were combined to generate a broadband high-gamma amplitude time series for each electrode. The high-gamma amplitude was then z-scored within each recording session on every electrode^22–24. We studied the high-gamma amplitude because it has been shown to correlate well with multi-unit firing rates²⁵. The time–frequency maps showed the strongest response-related modulation in the high-gamma range (70–150 Hz). Therefore, subsequent analyses focused on high-gamma activity as the primary neural feature for speech processing (Figure S1a,b). The Hilbert-transform-derived high-gamma amplitude data are provided in the derivatives/HighGammaAmplitude folder and were used for the subsequent speech-responsiveness, tone-discriminant, decoding, and TRF analyses reported in this manuscript.

Anatomical labeling

Electrodes on each individual brain (Fig. 2a–d) were marked on preoperative T1 MRI in BrainLab neuronavigation system, which were double-checked by a neurosurgeon via intra-operative photos²⁶. The reconstruction of cerebral surface were created through FreeSurfer, and anatomical labeling and plotting were performed through img_pipe Python package²⁷. The electrodes on individual brain were also projected to the MNI (Montreal Neurological Institute) template, PA1 PA2 on left hemisphere and PA3 PA4 on right hemisphere respectively (Fig. 2e,f).

Fig. 2.

Electrode coverage, speech responsiveness, tone selectivity, and decoding performance. (a–d) Electrode coverage on the native cortical surfaces of four participants. Dark red dots denote electrodes exhibiting significant speech responsiveness. Orange- circled electrodes indicate the subset that both respond to speech and statistically differentiate among Mandarin lexical tones. (e,f) Group-level visualization of electrode locations after projecting each participant’s contacts to the MNI template surface. Participants PA1 and PA2 (left hemisphere coverage) and PA3 and PA4 (right hemisphere coverage) are displayed separately to illustrate hemispheric sampling. (g) Event-related high-gamma responses from two example electrodes (Electrode 64 on PA3 and Electrode 32 on PA4) time-locked to speech onset. Black horizontal ticks mark time intervals with significant deviations from baseline assessed by nonparametric permutation testing. (h) Tone-selective high-gamma responses from two example electrodes (Electrode 70 on PA1 and Electrode 24 on PA2), aligned to vowel onset. Curves show the mean response for each lexical tone; shaded bands represent mean ± s.e.m. Black ticks indicate time intervals where responses differ significantly across tones (P < 0.05; F test, two-tailed; Bonferroni-corrected). (i) Aggregate electrode distributions on the MNI152 template, shown separately for left and right hemispheres, highlighting spatial coverage across cortical regions. (j) Receiver operating characteristic (ROC) curves for participant-specific speech detectors trained to discriminate speech versus silence from high-gamma activity. (k) Confusion matrices summarizing per-participant classification performance of the speech detector.

Data Records

The dataset is organized following the BIDS-iEEG²⁸ data structures. The released 400-Hz, 70–150 Hz band-pass-filtered ECoG signals are provided in Neurodata Without Borders (NWB; https://www.nwb.org/) format. All data and metadata conform to the BIDS naming conventions and folder hierarchy. The dataset is publicly available at https://doi.org/10.57760/sciencedb.27618⁴⁰. Codes for processing the data and using tutorials are also available at https://github.com/yuanningli/ECoG_Tonal_Speech_Perception.

The root folder (Fig. 3a) contains participant metadata (participants.tsv), subject-specific folders (sub-0x), and a derivatives folder (derivatives). For each participant/session, the subject-specific iEEG folder contains NWB files storing the 400-Hz, 70–150 Hz band-pass-filtered ECoG signals aligned to the speech stimulus, with dimensions of time samples × channels (128 or 256 channels, depending on the participant). Electrode localization information is provided through BIDS-compatible electrode and coordinate-system files, including native-space and MNI-space coordinate information where applicable. Anatomical data used for electrode localization and visualization are provided in the corresponding anatomical and derivative folders.The T1 MRI was also contained in sub-0x/anat/ for reconstruction of the brain by using FreeSurfer or img_pipe²⁷ Python package.

Fig. 3.

Structure and contents of the ECoG dataset. (a) Organization of the ECoG_Tonal_Speech_Perception_dataset folder. The derivatives folder contains six subfolders, namely Annotations, Brain2D, BrainFile, mel_dB, formants and HighGammaAmplitude. These subfolders provide processed data products in .npy format. Each participant’s iEEG (intracranial electroencephalography) data was stored in an individual folder, labelled sub-0x (participant x), which contained two subfolders named anat and ieeg, respectively. The Annotations subfolder provides time-aligned word-level Pinyin annotations and syllable-level annotations, lexical-tone labels, break index (BI), and stress tier (ST). Electrode localization information is provided in BIDS-compatible electrode and coordinate-system files, with anatomical visualization and reconstruction files available in Brain2D and BrainFile. (b) Audio features: Mel-spectrogram (dB) are stored in mel_dB/sub-0x/sub-0x_ses-0x_task-audio_mel_dB.npy and were shown by using librosa Python package. (c) Linguistic annotations: Annotations/sub-0x/sub-0x_ses-0x_task-audio_pinyin.npy contains three parallel arrays—onset_list (float), pinyin_list (string), and offset_list (float), providing token-level timing and Pinyin transcription aligned to ECoG recordings. (d) Visualization: The Brain2D folder provides participant-specific cortical images and corresponding 2D electrode coordinates, alongside projections to the MNI152 template, enabling reproducible visualization of electrode coverage in native and standardized spaces. (e) ECoG data aligned with stimuli are provided in NWB files as 400-Hz, 70–150 Hz band-pass-filtered signals. Hilbert-transform-derived high-gamma amplitude data are provided in the derivatives/HighGammaAmplitude folder. The example channel illustrated exhibits robust speech-evoked responses.

The derivatives folder (Fig. 3a) contains six subfolders, namely Annotations, Brain2D, BrainFile, mel_dB, formants and HighGammaAmplitude. The Annotations subfolder includes vowel and consonant markers in _syllable.npy for each participant, word-level Pinyin transcription and timing in _pinyin.npy (Fig. 3c), break index (BI) in _BI.npy which annotates the prosodic hierarchy of Mandarin speech, stress (ST) tier in _ST.npy, reflecting Mandarin’s hierarchical stress patterns²¹ (Figure S3). Brain2D subfolder (Fig. 3d) provides pictures and electrodes’ relative positions on them. Original electrodes on individuals’ brain were warped onto MNI152 template by using FreeSurfer and img_pipe²⁷ Python package. BrainFile subfolder contains processed brain information including pial surface cortical meshes of the left and right hemisphere (Meshes/_lh_pial.mat and _rh_pial.mat), the brain anatomy (mri/brain.mgz) and other relevant files for spatial positioning and plan plotting generated from T1 MRI by using FreeSurfer and img_pipe²⁷ Python package. mel_dB subfolder (Fig. 3b) contains 100 Hz mel-spectrogram²⁹ of the audio of the corpus participants listening to during awake craniotomy. We also provide formants (from F1 to F6) and pitch (F0) by using Parselmouth³⁰ Python package, Praat³¹ software and codes from Github³². High Gamma Amplitude were extracted and saved in HighGammaAmplitude for downstream analyses.

Codes for processing these data were also included in folder code as the format of.ipynb and .py files. Researchers can use Python to readily run them.

Technical Validation

The recorded data were validated by showing that there exists speech-responsive electrodes and tone-discriminant electrodes in superior temporal gyrus (STG) that respond to the onsets of speech and lexical tone, respectively¹². Also, we trained a decoder discriminating between speech and silence using high-gamma activity recorded from speech-responsive electrodes.

Checking for acoustic contamination

It has been reported that intracranial neural signals can directly detect acoustic signals, rather than through neural activities^33,34. To assess potential acoustic contamination of the ECoG recordings, we compared the time-frequency power of the audio signal with that of each ECoG channel following Roussel et al. Audio and ECoG signals were transformed into spectrograms using a 200-ms Hamming window at 50-Hz temporal resolution, and Pearson correlations were computed between sound-frequency and ECoG-frequency power time courses. Because direct acoustic contamination is expected to produce a narrow diagonal structure in the cross-frequency correlation matrix, each channel was tested for enhanced diagonal correlation within the 70–150 Hz band using 10,000 frequency-label permutations. Channels significant after Benjamini-Hochberg FDR correction (q < 0.05) were excluded (Figure S5), and these rejected channels were documented as bad channels in the corresponding BIDS channel files. This procedure removed 0, 5, 10, and 1 channels from PA1-PA4, respectively. After channel rejection, the cross-frequency correlation matrices, lag-correlation curves, and permutation tests were recomputed using the retained channels (Fig. 4). No narrow-band diagonal pattern or prominent zero-lag peak remained after rejection, indicating that the retained ECoG signals were not systematically driven by direct acoustic contamination.

Fig. 4.

Acoustic-contamination analysis before and after channel rejection. For each participant, cross-frequency correlations between sound power and ECoG power, lag-correlation curves, and permutation-test results are shown before and after removal of contaminated channels. (a–d) Results for PA1-PA4, respectively. In each panel, the top row shows the maximum cross-frequency correlation matrix across channels before and after channel rejection. The middle row shows biased cross-correlations between sound and ECoG power in the 70–150 Hz band across temporal lags. The bottom row shows the permutation-test distribution for the mean diagonal correlation in the 70–150 Hz band; vertical lines indicate the observed statistic. After channel rejection, no participant showed a significant diagonal correlation structure or a prominent zero-lag peak.

Speech-responsive and Tone-discriminant electrodes

Subsequent validation analyses were performed on the 400-Hz high-gamma amplitude signals extracted from the 70–150 Hz band-pass-filtered ECoG data using the Hilbert transform.

To find speech-responsive electrodes, we first aligned high-gamma responses to the onset of the speech, which was preceded by at least 200 ms of silence. We then used a nonparametric permutation test to assess whether the average high-gamma response following speech onset (0–600 ms) was significantly elevated relative to baseline activity expected by chance. Specifically, for every electrode we generated 1,000 surrogate datasets by locking to randomly chosen “pseudo-onsets” that were at least 400 ms away from any real speech, thereby forming an empirical null distribution of high-gamma activity. Observed responses were converted to z-scores relative to this distribution, and samples exceeding ±3 STD (two-sided P = 0.0027) were marked significant. Only the electrodes with at least 40 consecutive significant time points (100 ms) were defined as speech-responsive electrodes (Fig. 2a–fred electrodes, Fig. 2g).

To find speech-responsive electrodes that also discriminate between lexical tones, we first aligned high-gamma responses to the onset of the tones (vowel onsets). Then, a one-way F test (ANOVA) was used to test whether the mean high-gamma responses of the four tones were significantly different¹². Specifically, F statistic for every time point from 0 to 600 ms relative to the vowel onsets (240 points in total) was computed and two-sided P < 0.05 threshold using Bonferroni correction was used to determine significant time points¹². Only the electrodes with at least 40 consecutive significant time points (100 ms) were considered as tone discriminant electrodes (Fig. 2a–fyellow electrodes, Fig. 2h).

Speech-responsive and tone-discriminant electrodes mainly locate in the middle and posterior STG, shown on cvs_avg35_inMNI152 template (Fig. 2g,h), consistent with previous studies of speech perception^{12,13,16,35,36}. Also, kernel density estimation (KDE) maps of speech-responsive electrodes showed a predominant spatial concentration over the superior temporal gyrus (STG) in MNI space (Figure S2a–d).

Speech decoding model

To demonstrate the utility of our ECoG dataset for neural decoding tasks, we trained a binary classification model to determine whether a 1000 ms period ECoG data (high-gamma activity) can reflect auditory stimuli, specifically, speech present or absent.

During training, a sliding window strategy firstly was used on 400 Hz high-gamma ECoG signals with window length of 1000 ms and step size of 100 ms. The ground truth labels were derived from the aligned annotations of the stimulus corpus. Each 1-second segment was labeled as 1 (speech present) if the cumulative speech duration within the window exceeded 500 ms; otherwise, it was labeled as 0 (speech absent).

We used the Torch 2.10.0 + cu128 Python package to train the speech detection model³⁷. The event classification architecture was a CNN structure followed by a stack of two GRU layers with a latent dimension size of 128 and a dropout of 0.5 applied at each layer³⁸. To concentrate on speech-responsive electrodes, only speech-responsive electrodes (red electrodes on Fig. 2a–d) were used to train the model for each participant.

All results are obtained using 6-fold cross validation (CV) for PA1 and PA2, 8-fold cross validation (CV) for PA3 and PA4 during which each session was split into two parts and then the sliding strategy was performed for the avoidance of overlapping in validation sets and test sets. Early-stopping strategy was used during validation. To evaluate the performance of our speech decoder, the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve for each participant (Fig. 2j) was calculated and plotted, with all participants over 0.9. Confusion matrices of every participant were also shown (Fig. 2k), with high True Positive Rate (TPR) and high True Negative Rate (TNR). The decoding results of our dataset show solid and discriminative power in distinguishing linguistic stimuli.

Tone Decoding Model

To further demonstrate that tone can be discriminated by neural activity, we developed a neural network classifier using tone-discriminant electrodes to decode the four lexical tones. The model takes the 1 s high-gamma amplitude as input. We also used Torch 2.10.0 + cu128 Python package to train this 4-way classification model. The network architecture consists of two sequential 1-D convolutional blocks-each comprising a convolution layer, batch normalization, Exponential Linear Unit (ELU) activation, average pooling, and dropout-followed by a fully connected layer.

To evaluate the model rigorously, we employed a 5-fold stratified cross-validation framework and the overall decoding performance was quantified using balanced accuracy based on out-of-fold predictions. Also, to determine the statistical significance, we performed a permutation test. The results showed that tone decoding performance was significantly above theoretical chance level (Figure S4).

Temporal Receptive Field (TRF) Analyses

To determine the degree to which the response could be predicted with phonetic features, we used time-delayed linear encoding models known as TRF models³⁹. TRF models allow us to predict neural activity based on stimulus features in a window of time preceding neural activity. In particular, we fit the linear model $y\left(t\right)={\sum }_{f=1}^F{\sum }_{\tau =0}^T{\beta }_f^T\left(\tau \right)x_f\left(t-\tau \right)+ϵ$ for each electrode, where $y$ is the high-gamma activity recorded from the electrode, $x_f\left(t-\tau \right)$ is the stimulus representation vector of feature set $f$ at time $t-\tau$, ${\beta }_f\left(\tau \right)$ is the regression weights for feature set $f$ at time lag $\tau$, and $ϵ$ is the gaussian noise.

In the full TRF model, we included features for provided mel-spectrogram, pinyin onset time, syllable onset time, break index and stress tier. To prevent model overfitting, we used L2-norm regularization and cross-validation. Specifically, we divided the data into three mutually exclusive sets of 80%, 10%, and 10% of samples. The first set of 80% was used as the training set. The second set was used to optimize the L2 regularization hyperparameter, and the final set was used as the test set. We evaluated the models using the correlation between actual and predicted values of neural activity on held out testing data. We performed this procedure five times and the performance of the model was taken as the mean of performance across all test sets.

Results (Fig. 5a,b) show that electrodes with stronger TRF performance were concentrated primarily in bilateral temporal cortex, especially superior and middle temporal regions. Figure 5c illustrates a representative 10-s stimulus segment, showing the temporal alignment of mel-spectrogram structure with word/syllable onsets and BI/ST event streams used by the model.

Fig. 5.

TRF feature design and spatial distribution of encoding performance. (a) Cortical distribution of electrodes from both hemispheres. Black dots indicate all implanted electrodes, and blue dots indicate electrodes with TRF model performance above threshold (R² > 0.10). Circle size illustrates example marker scaling for R² values. (b) Regional electrode counts (LH+RH) summarized by anatomical labels. Gray bars show all electrodes, and blue bars show electrodes with R² > 0.05, highlighting strongest contributions in temporal regions. (c) Representative 10-s stimulus segment (sub-01, session 01; 3.0–13.0 s) used for TRF fitting. Top: mel-spectrogram. Bottom: aligned event streams on separate rows, including word onset (blue), syllable onset (purple), BI onset (green), and ST span (red). (d) Cross-validation confusion matrix (proportion) across 4 BI categories of 4 subjects. Colors represent the normalized prediction proportion according to the true category, and numbers represent percentages.

Break Index Decoding

To quantify break-index information across participants, we pooled all subjects’ trials and used 1-s ECoG windows on both sides of each break index for joint training of a 4-class decoder with stratified 5-fold cross-validation. For each fold, a convolutional neural network was optimized with class-balanced cross-entropy to mitigate label imbalance. Out-of-fold predictions were aggregated across folds to compute a normalized confusion matrix (row-wise proportions), which summarizes classification performance for each break-index class. Examples of the four-level BI mel-spectrograms are shown in Figure S3.

Usage Notes

Aligned ECoG data

Codes for processing ECoG signals are provided on our Github. They are also contained in folder code.

Anatomical data

To run the img_pipe²⁷ Python package and electrodes on individual and MNI152 brain, you should put folder derivatives/BrainFile into /freesurfer/subjects/ where you downloaded the FreeSurfer application and renamed the folder and refer to their Github (https://github.com/ChangLabUcsf/img_pipe) for further tutorials.

Recommended use cases & caveats

To facilitate responsible and effective reuse of this dataset by the research community, we outline the primary recommended applications and important limitations to consider. We recommend this dataset used in Speech Perception Dynamics. Researchers can utilize the high spatiotemporal resolution of these ECoG recordings to investigate the fine-grained spatiotemporal dynamics of continuous speech processing, particularly within the superior temporal gyrus (STG). In Caveats and Limitations, this dataset includes recordings from four participants. While standard for invasive ECoG studies and sufficient for robust within-subject modeling and proof-of-concept decoding, the data should not be used to draw broad population-level inferences or demographic comparisons. Besides, electrode coverage is clinically determined and primarily concentrated over the perisylvian language areas (e.g., STG). The dataset is therefore not suitable for whole-brain network analyses or investigating brain regions outside the recorded auditory and language networks.

Besides, the original broadband ECoG voltage recordings are not publicly redistributed. Although the released NWB files provide 400-Hz, 70–150 Hz band-pass-filtered ECoG signals and the derivatives folder provides high-gamma amplitude time series, the dataset is not suitable for analyses that require the original broadband recordings, such as alternative low-frequency preprocessing pipelines or analyses outside the released high-gamma range.

Author contributions

J.W., J.L. and Y.L. designed the experiment and recorded the data. H.Z. analyzed the data. D.Z. designed the neural network. H.Z. wrote the codes for the analyses and tutorials. H.Z. and D.Z. wrote and revised the manuscript. All authors reviewed and approved the manuscript.

Data availability

The dataset is organized following the BIDS-iEEG²⁸ data structures. The released 400-Hz, 70–150 Hz band-pass-filtered ECoG signals are provided in Neurodata Without Borders (NWB; https://www.nwb.org/) format. All data and metadata conform to the BIDS naming conventions and folder hierarchy. The dataset is publicly available at https://doi.org/10.57760/sciencedb.27618⁴⁰.

Code availability

The Python code for processing ECoG and aligning it with ASCCD annotations can be found on Github: https://github.com/yuanningli/ECoG_Tonal_Speech_Perception. The code relies mainly on the following packages in python: numpy⁴¹, scipy⁴², torch³⁷, pandas⁴³, librosa⁴⁴ and matplotlib⁴⁵.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41597-026-07619-z.