Extended Data Fig. 3. Optimization fidelity vs. human recognition of model metamers.

a,b, Optimization fidelity for visual model metamers at the metamer generation stage (a) and at the final model stage corresponding to a categorization decision (b; N = 219 model stages). Visual models are those in Figs. 2–4 and Fig. 6g. Note that most data points are very close to 1 for the final stage correlation metrics (for example 209/219 stages exceed an average Spearman ρ of 0.99). Each point corresponds to a single stage of a single model. c,d, Optimization fidelity for auditory model metamers at the metamer generation stage (c) and at the final model stage corresponding to a categorization decision (d). Auditory models are those in Fig. 2 and Fig. 5 (N = 127 model stages). Note that most data points are again very close to 1 for the final stage correlation metrics (all 127 stages exceed an average Spearman ρ of 0.99). In all cases, optimization fidelity is high for both the metamer generation stage and the final stage, and human recognition is not predicted by the optimization fidelity, or only weakly correlated with the optimization fidelity for the generated model metamers (accounting for a very small fraction of the variance). Error bars on each data point are standard deviation across generated model metamers that passed the optimization criteria to be included in the psychophysical experiment. e,f, Final stage optimization fidelity plotted vs. model metamer generation stage for CochCNN9 auditory model (e) and CochResNet50 auditory model (f). Note the y axis limits, which differ across plots to show the small variations near 1 for the correlation measures. It is apparent that for any given model, some stages are somewhat less well optimized than others, but these variations do not account for the recognizability differences found in our experiments (compare these plots to the recognition plots in Figs. 2 and 5).