Skip to main content
. 2024 Nov 11;20(11):e1012537. doi: 10.1371/journal.pcbi.1012537

Fig 2. EEG is more accurately predicted by a contextualized speech model than standard acoustic or lexical surprisal representations, with accuracy increasing with model layer depth.

Fig 2

Top row line plots: The speech model (Whisper) complemented Speech Envelope-based measures, a Log-Mel Spectrogram (Whisper’s input) and lexical surprisal in predicting EEG data. Models were considered to be complementary if scalp-average EEG prediction derived from concatenating model pairs (Union = [Whisper, Surprisal]) were more accurate than constituent models (signed-ranks tests, Z and FDR corrected p-values are displayed at the top of each plot). Whisper Layers 1 to 6 complemented every competing predictor. Lexical surprisal and the envelope-based predictors, but not the spectrogram also made independent contributions to prediction. Corresponding signed-rank test statistics (Z, FDR(p)) are the red and black numbers at the top of each plot). Individual-level scalp-average results for Env&Dv, Log-Mel Spectrogram and Word Surprisal are in Fig E in S1 Text. Bottom row scalp maps: Post hoc electrode-wise analyses mapped scalp-regions that were sensitive to the speech-to-language model. Each electrode was predicted using the Union of Whisper Layer 6, the Envelope-based measures and Lexical Surprisal (because all three had made independent predictive contributions in the primary scalp-average analyses, unlike the spectrogram which was excluded). Whisper’s independent contribution was estimated by partitioning the variance predicted by the three-model union (including Whisper) minus the variance predicted by a joint model excluding Whisper (Envelope and Surprisal). The square root of the resultant variance partition was taken to provide a pseudo-correlation estimate on the same scale as other results, as is a common procedure [31,45]. Electrodes that are typically associated with low-level acoustic processing were especially sensitive to Whisper. This is visible as the red band that straddles central scalp from ear to ear in the leftmost scalp map.