Fig. 3. Representations of image category and familiarity (new/old).

(A-B) Example cells that (A) represent image category and (B) that differentiate between new and old stimuli. (C) Decoding accuracy of image category from all recorded cells was significantly higher in the HA relative to the MFC (Δ_true = 49%, p<0.001). (D) Decoding of new vs. old (ground truth) was similarly accurate in HA and MFC (Δ_true = 7%, p = 0.13). For new vs. old decoding, trials with images of monkeys were excluded since the recognition performance for these images was at chance (Fig. S1B). (E) Population activity of all recorded HA (left) and MFC (right) cells, plotted in 3D using MDS. Individual points show the mean activity of the population for that specific condition. The highlighted plane contains all locations of state space occupied by a given task for the case of fruits vs. faces as the binary category distinction (for illustration only; all analysis uses all categories). The geometry of the representation allows for a decoder that is trained on one task to generalize to the other task (see Fig. S4C for example decoder hyperplanes). (F) Approach used for the cross-condition generalization analysis. Color indicates task (blue=categorization, red=memory). (Top) We trained a decoder to discriminate between new vs. old trials on categorization trials and then tested its performance on new vs. old stimuli encountered during the memory condition (and vice-versa). (Bottom) Similarly, a decoder that is trained to discriminate between image categories (in this example face vs. fruits, all results include all 6 possible pairs) on categorization trials, was tested on memory trials. (G) Cross-condition generalization performance for image category. (H) Cross-condition generalization performance for new vs. old. (I) Difference in cross-task generalization decoding accuracy for familiarity and image category between HA and MFC. Difference is computed between the average cross-task performances in each area (i.e. average of memory → categorization and categorization → memory). The null distribution for the average was estimated from the empirical null estimated by shuffling the labels used to train the decoders. For both variables, decoding from HA had significantly greater cross-task generalization performance than decoding from MFC (difference in both cases is positive and outside of the 95^th percentile of the null distribution). (J) Generalization index (see Methods) for memory (left two data points) and image category (right two data points). For both image category and familiarity, generalization across task was higher in the HA population than the MFC population (see figure for statistics).