Fig. 2. LLM-based linear prediction and decoding of brain activities.

a, The linear encoding model captures selectivities of different brain regions. We contrasted the brain activity predicted from five novel people- versus place-related sentences (left) and five food- versus people-related sentences (right; significance threshold set by a two-tailed t-test across participants (N = 8) with P = 0.05; without FDR correction). These contrasts highlight brain areas known to be selective for people, places and food (people and place areas are localized as part of NSD (left); food areas described by ref. ⁵⁵ shown as white outlines (right)). b, Decoding captions from visually evoked brain responses. Top: we fit a linear model to predict LLM embeddings (MPNet) from fMRI voxel activities. We then use a nearest-neighbour look-up to generate a caption for each image. Bottom left: kernel density estimate plot of the prediction score for each participant on a held-out test set (see Supplementary Fig. 5 for a t-SNE projection of the training and testing sets), quantified using Pearson correlation between predicted and target embedding. The noise ceiling is computed as the consistency between the five human-generated captions for each image. Bottom right: target (blue), decoded (pink) and nearest training (green) caption examples from different participants on the held-out test set, spanning the range of prediction scores. The decoder is not simply looking up the closest training item, but instead provides another adequate caption. The rank refers to the prediction score of the shown sample (that is, rank 0 is the best prediction for this participant, while rank 514 is the worst). Note that, for copyright reasons, we cannot show the real COCO images we used; hence, they have been replaced by similar copyright-free images. EBA, extrastriate body area; FBA1/2, posterior/anterior section of fusiform body area; FFA1, posterior section of fusiform face area; FFA2, anterior section of fusiform face area; PPA, parahippocampal place area; pSTS face, posterior superior temporal sulcus face area; OFA, occipital face area; OPA, occipital place area. References: Allen et al.⁴⁶, Pennock et al.⁵⁵.