Figure 4. Left hemispheric ROIs demonstrate two-tiered hierarchical organization of syllables and phonemes.
a. Graphical representation of hierarchical speech representations with syllables coded at higher levels and phonemes are coded at lower levels. The hierarchical organization allows syllable frames to be coded earlier in time, followed by phonemes into the syllable frame. Neural codes for syllables and phonemes were characterized by neural decoding analyses. Heatmaps indicate confusion matrices (SVD-LDA, 10-fold cross-validation) for syllables (2-way; chance 50%) and phonemes (9-way, chance: 11.11%) in speech significant electrodes (2895 total electrodes, 69 mean electrodes per subject). b. Left: The temporal evolution of syllables and phoneme activations were obtained using time-resolved neural decoding (n = 20, 200 ms window; 10 ms step size). Solid lines indicate average across 20 instances of decoding iterations, and shaded bars indicate standard errors across instances. Grey traces indicate mean and standard errors of chance decoding which was obtained from randomly shuffling syllable and phoneme labels (250 shuffled decoding models). Horizontal bars indicate durations of significant decoding (one-sided temporal permutation cluster test, cluster forming threshold p < 0.05; 250 shuffled decoding models). Colored histograms indicate distributions of decoding onsets of syllables (violet) and phonemes (green), with syllables detectable earlier than phonemes (n = 20, one-sided Mann-Whitney U test, p < 0.001). Middle: Colored histograms indicate distributions of onset decoding times (n = 20) for both syllable and phoneme for each ROI network. Significant differences in decoding distributions indicate a temporal progression across the ROI networks for both syllables and phonemes (n = 20, one-way ANOVA with post-hoc Tukey HSD test, two-sided, p < 0.01, for exact p-values see Main text). Syllable coding was earlier in time than phoneme coding (n = 20, p < 0.05, one-sided Mann-Whitney U test, for exact p-values see Main text), demonstrating a network-level organization in accordance with the proposed hierarchical representation of syllables and phonemes (see Fig. 1a). Right: The difference in activation timing of the hierarchical representation varied for each ROI group, with maximum difference between syllable and phoneme activation during planning (n = 400, 250–350 ms), followed by articulation (n = 400, 110 – 130 ms), and then minimally (n = 400, 17 – 25 ms) during monitoring (n = 400 pairwise differences between syllables and phoneme times per group, one-sided pairwise t-test, planning vs. articulation, onset: t(798) = −10.7, p < 0.001, articulation vs. monitoring: onset: t(798) = 18.2, p < 0.001; planning vs. monitoring: onset: t(798) = −8.04, p < 0.001). Error bars indicate mean differences (± SEM) between syllable and phoneme times for each speech network. c. Hierarchical organization at micro-scale population level: μECoG electrode array was implanted over sensorimotor cortex in an awake patient (S3). The electrode array had 128 electrodes with 1.33 mm inter-electrode distance (purple grid). d. Electrode-specific decoding characteristics visualized on μECoG grid for syllables (left) and phonemes (right), indicating peak decoding accuracies of electrodes with significant decoding. Syllable decoding was broadly distributed across the array, while phoneme decoding was spatially restricted, indicating a hierarchical topographic organization. e. Paired comparison of decoding onset times between syllables and phonemes across electrodes with significant decoding for both features (n = 29 electrodes). Syllable decoding onsets (median = −0.46 s) were consistently earlier than phoneme onsets (median = −0.26s). Box plots show the distribution of decoding onsets for syllables and phonemes (25th – 75th percentile), and the whiskers indicate extreme data points within 1.5 × inter-quartile range of the nearest quartile. The lines connect paired estimates from the same electrodes. Syllable decoding onsets significantly preceded phoneme onsets across the electrodes with a hierarchical lag of 200 ms across the array (T_Syllable – T_Phoneme = −200 ms, one-sided Wilcoxon signed-rank test, p < 0.001). f. Spatial map of the onset timing difference across the array. Each circle represents a single electrode with valid syllable and phoneme onset estimates. Color indicates the magnitude of the onset differences, revealing a distributed gradient of hierarchical encoding.