Making models disagree to learn how brains compute

Abstract

Computational hypotheses about brain information processing can be expressed in neural network models. Neuroscientists have begun to compare such models in terms of their alignment with neural and behavioral data. The high parametric capacity of these models is essential to their ability to capture cognitive processes but also enables them to approximate arbitrary functions, making distinct models difficult to discriminate experimentally. Model comparisons using stimuli sampled from the training distribution often fail to reveal differences. This challenge can be met by optimizing stimulus sets for model discrimination and by leveraging out-of-distribution generalization as a severe test. This review explains the emerging methods for the design of experiments that adjudicate among neural network models. These methods seek stimuli that are controversial among the models: maximizing model disagreement in terms of predictions of the data. We discuss the choices involved in designing such stimulus sets, including a prior over candidate stimuli (e.g. naturalistic images) and procedures for selecting or synthesizing stimulus sets that maximize alternative measures of power. Whereas natural stimuli promise greater ecological validity, artificial stimuli can provide greater power for model comparison. Tempered by a prior, controversial stimuli offer a synthesis of these contrasting established approaches, combining their advantages.

1. Introduction

To learn how brains compute, neuroscience must compare alternative theories of brain computation in light of experimental data. Computational theories can be fully specified and rendered empirically testable by implementation in models that simulate the information processing function investigated [1–10]. One approach to the experimental evaluation of such models is to present samples of natural stimuli to animal or human subjects in neurophysiological and behavioral experiments [11]. The same stimuli can be presented to the models so as to compare their internal representations and behavior to the neural representations and behavior of the subjects. Using natural stimuli to evaluate models ensures that the model evaluation is ecologically valid: Ultimately, models should explain neural activity and behavior under natural conditions, so evaluation on natural stimuli is essential [4, 12–14]. If a model perfectly captured the neural mechanism producing the neural and behavioral responses, however, it should make good predictions not only for natural stimuli, but for all stimuli. If researchers would like models to make good predictions for all stimuli, they should test models also on non-natural stimuli. If we are open to evaluating models with non-natural stimuli, we have greater flexibility to choose stimuli that provide good power for model comparison, which will typically not be natural stimuli [15].

Historically, psychophysicists and neuroscientists seeking to evaluate computational models have often relied on hand-crafted artificial stimuli, designed to provide critical tests of model predictions. Such an approach is feasible when the models are simple enough for researchers to design the stimuli. However, complex perceptual, cognitive, and motor control functions require much knowledge about the world, which any adequate model must have the capacity to store. For example, visual recognition of a wide range of objects requires knowledge of the appearance of these objects. The uncomfortable consequence of the need for knowledge is that models must combine hypothetical cognitive and neuroscientific mechanisms with high parametric capacity and machine learning. Neural network models enable us to express computational hypotheses while also providing the required capacity to store the knowledge needed to capture intelligence [5, 16]. However, their complexity makes it difficult to hand-craft stimulus sets that enable powerful model comparisons. The field’s reliance on natural stimuli can be motivated in terms of ecological validity, but it is also a choice of convenience in the context of models that make it difficult to design critical experiments. This review describes how neuroscience can leverage optimization techniques to design experimental stimuli for more powerful comparisons among neural network models.

Optimization of stimulus sets for model comparison offers a synthesis of the contrasting established approaches of using artificial or natural stimuli. Stimuli optimized for model comparison will be driven by model predictions and will not in general conform to the statistics of natural stimuli. However, when the models are neural network models trained on natural images, stimulus sets optimized for model comparison reflect the knowledge the models have acquired about the natural world and therefore often have certain features of natural stimuli. In addition, the stimulus space can also be restricted to ensure that model comparisons rely only on natural stimuli or on stimuli considered sufficiently naturalistic to make the experiment ecologically valid.

2. Why is it challenging to compare parameter-rich models?

Recent research on the relationship between artificial neural networks and cortical representations has repeatedly revealed two observations. (1) Internal activations of parameter-rich neural network models are highly predictive of animal and human neural responses to natural stimuli, even when these models are not directly fitted to neural data [2, 3]. (2) Multiple distinct neural network models achieve similar levels of predictivity [17, 18]. While observation (1) suggests that neural network models hold promise for understanding brain computation, observation (2) casts doubt on the prospects for theoretical progress through iterative model building and model comparison.

The two observations are evident across domains, when models are trained and tested on randomly sampled natural stimuli. The “Brain-Score” project [19, 20] benchmarks many neural network models by evaluating their predictions of diverse neural and behavioral datasets using fitted linear mappings from model representations to neural and behavioral responses. Qualitatively distinct models often achieve similar scores on Brain-Score. For example, a range of artificial neural network architectures align equally well with neural recordings from V2, V4, and IT [21]. Models designed to implement specific neuroscientific motifs, such as recurrence [22] or V1-like early processing [23], do not systematically outperform neural networks adopted from machine learning without further modification [20] (but see [24]). Comparing human functional magnetic resonance imaging (fMRI) and hidden-layer artificial neural network activations indicates similar brain alignment of various convolutional neural network architectures trained on object recognition, once linear reweighting of representations is allowed [17]. A recent, larger-scale survey generalizes the finding of equivalent neural prediction accuracy from convolutional to transformer models and to models trained on different visual tasks [18]. In the language domain, also covered by Brain-Score using human neuroimaging data, some models predict the available imaging benchmarks as well as the inter-subject reliability permits [19, 20]. In general, studies on the alignment of artificial neural networks with human fMRI stimulus representations often report model performance well above chance, yet statistical comparisons between models are typically absent, insignificant, or uncorrected for multiple testing. Within the domain of neural representations of human faces, studies using primate single-unit recordings [25], human intracranial recordings [26], fMRI [27], and human similarity judgments [28] all report tied predictivity scores across two or more distinct computational models. Importantly, even when the models under comparison are not neural-network representations of natural stimuli but instead classic machine vision image descriptors [29] or human-labeling-based stimulus features [30, 31], evaluations on randomly sampled natural stimuli frequently yield statistically indistinguishable performance. This lack of differentiation indicates that the models make predictions that are too similar in their fit to the data, relative to the sensitivity of our experimental and data-analytical methods for model comparison. It remains a matter of debate to what extent the similarity of the models or the sensitivity of our methods is to blame.

Some recent papers argue that indistinguishable models should be considered implementations of a single universal representation. The argument for universality is based on the observation that the most predictive representational features are in fact shared between models [32–34]. Even before this hypothesis was stated, there was already a concern that high-parametric deep neural network models may act as mere function approximators that do not provide predictive value beyond a data summary [17, 18]. Whether alternative models make empirically distinguishable predictions can be assessed using model-recovery simulations, where data is simulated using each of the models, such that the ground truth is known. If the simulated data enable identifying the true data-generating model, then the models make predictions that are sufficiently distinct to enable adjudication given the noise level simulated. Using only naturalistic stimuli in the context of vision, model-recovery simulations give a mixed picture. The “glass half full” interpretation is that they often enable us to discriminate models. The “glass half empty” interpretation is that they also regularly fail to discriminate even models that have strikingly different architectures [21, 35].

We can reconcile the universalist and particularist perspectives by noting that diverse models and brains all need to learn to represent major variance-explaining latent factors of the sensory data. All visual systems, after all, must represent the same visual world. Even a vision and a language model [32] will come to reflect some of the same latent factors that organize the world. Consider, for example, that forks and knives will tend to co-occur in images and also in text documents, which could give rise to a latent factor related to “cutlery” in vision and language models alike. We should expect that the shared features are those that explain a lot of variance in the sensory input, so they will also account for much of the neural predictivity of models when tested on natural stimuli. To the extent that diverse systems have ample capacity to capture the statistical structure of the world, they may converge toward a platonic representation where their diverse architectures, objectives, and learning rules cease to substantially affect their representations [32]. However, all brains and AI models are vastly less complex than the world [36], so the platonic ideal they are driven toward is bound to remain out of reach. Thus, while models share major variance-explaining features, they also acquire unique features that reflect their distinct architectures, objectives, and learning rules, and the corners they must cut to achieve good performance with limited resources.

Studies may fail to detect differences in predictive performance when the models are evaluated on samples from a shared training distribution (e.g. randomly sampled natural stimuli) [17, 18]. For model comparison to drive theoretical progress, we may therefore need to go beyond the training distribution. Empirical studies using stimuli optimized for model discrimination often reveal considerable performance differences among high-parametric models [29–31, 34, 37–41]. We refer to such stimuli as controversial stimuli because they are controversial among the models: They make the models disagree in their predictions. Like adversarial examples, controversial stimuli result from optimization using an objective defined on the responses of the models. However, adversarial examples aim to make a single model err, whereas controversial stimuli aim to elicit distinct predictions from alternative models. Although this approach is still new, the early empirical results suggest that optimized stimulus sets can disentangle the predictions even of strong function approximators.

The power to discriminate models depends not only on our experiments but also on our data-analysis techniques. Neural network models can be evaluated with neural or behavioral data using encoding models [4, 42] or representational similarity analysis (RSA) [43, 44]. Both approaches have seen substantial recent advances [45–51]. Encoding models and RSA are closely related and complementary, and together provide a powerful set of tools for model comparison [45, 52, 53].

In this review, we discuss how to optimize stimuli for discriminating high-parametric neural network models. The stimulus set optimization techniques we describe are relevant for a broad range of analyses that use encoding models or RSA, or combine elements of both approaches. The review is grounded in the normative foundations of experimental design. We begin by examining the key choices that an experimenter must make when optimizing a stimulus set to discriminate models. We then review neuroscientific studies that have optimized stimulus sets to elicit disagreement and adjudicate among neural network models. We conclude by discussing outstanding questions and future directions.

3. Generating stimulus sets that make models disagree

Selecting or synthesizing stimulus sets that elicit distinct predictions in alternative models is an optimization problem. To optimize the experimental stimuli, researchers need to define four components (Fig. 1): (1) the alternative models, each mapping stimuli to predictions about experimental measurements of neural or behavioral responses, (2) an optimization objective that quantifies how controversial a stimulus set is among the models, (3) a stimulus search space that specifies a parameterization for the stimulus or stimulus set to be optimized (Fig. 1a-c) and the set of admissible stimuli within that space, and (4) an optimization procedure, which enables us to navigate the search space toward a controversial stimulus set. We now explain these four components in turn.

Figure 1: Design choices in stimulus optimization for model discrimination. Left (a–c): The choice of stimulus search space determines the balance between model-comparison power and ecological validity.

Researchers can choose the distribution that the experimental stimuli are constrained to. The candidate stimulus distribution can have small or large support, can be more or less ecological, and may match the training distribution of the brain-computational model or pose an out-of-distribution generalization challenge. (a) Selecting stimuli from a stimulus bank via discrete optimization allows maximal control. (b) Optimizing visual stimuli in pixel space (or stimuli for other sensory modalities in an analogous low-level description) poses only minimal restrictions (e.g. resolution and color space) on the search space. (c) Generating a stimulus set by optimizing the latent inputs to a specific stimulus generator (e.g., a GAN [54], a diffusion model, or a differentiable renderer [38]) provides a compromise between control and flexibility: the search space is continuous but confined to a generator-defined manifold. (a–c) In all three cases, the resulting stimuli may undergo additional transformations, such as the addition of noise, to simulate realistic perceptual conditions. For visual stimuli, for example, spatial jitter can simulate imprecise fixation and improve the robustness of model predictions. Right (d–f): The nature of the predicted empirical observations determines what optimization objectives are appropriate. (d) When behavioral responses are predicted, model discrimination experiments can be designed by finding stimulus sets that produce distinct predicted behavioral response distributions. (e) When representational geometries are predicted, model discrimination experiments can be designed by finding stimulus sets that yield dissimilar RDMs. (f) When neural responses are predicted using linear mappings from the model units, model discrimination experiments can be designed by finding stimulus sets that yield dissimilar predicted neural responses according to the alternative models. (d–f) Each of these three approaches requires operationalization of the concept “dissimilar predictions” by defining a controversiality objective (circle on the right, see Boxes 1–3 for mathematical definitions). Importantly, in each of these approaches, researchers may need to account for measurement noise and uncertainty about the parameters of the model-to-brain or model-to-behavior mappings.

3.1. Neural network models predicting neural or behavioral responses

Current work often repurposes neural network models from the machine-learning literature and maps their internal representations or outputs to neural or behavioral measurements. We use the term “neural network model” to include both repurposed machine learning models and models built or adapted to implement neuroscientific hypotheses. As the field matures, we expect models to gain more neuroscientifically motivated features.

Each model, indexed by $m$, predicts responses $\mathit{y}$ to stimuli $\mathit{\xi }$. In the simplest case, each model $m$ defines a conditional probability distribution, $p(\mathit{y}\mid m,\mathit{\xi })$. $\mathit{\xi }$ specifies the stimulus set and $\mathit{y}$ represents a specific experimental outcome (i.e., a set of neural or behavioral responses). However, predictions are often conditional not only on the identity $m$ of the model, but also on its parameters ${\theta }_m$, defining a probability distribution $p\left(\mathit{y}\mid m,{\mathit{\theta }}_m,\mathit{\xi }\right)$. These parameters may include both the neural network weights and parameters of the mapping from the neural network's hidden-layer activations or outputs to the predicted neural or behavioral data. For predicting behavioral choices, the outputs of the neural networks can often be mapped directly to behavioral decision probabilities (Fig. 1d) without the introduction of additional parameters. For example, the softmax output of a neural network model trained to classify inputs can be interpreted as predicting classification response probabilities.

Linear encoding models.

When linear encoding models are used, the predicted outcomes $\mathit{y}$ are the responses of $C$ individual neurons or voxels (Fig. 1f). Each measured response is predicted separately, and the noise is typically assumed to be independent. The encoding model specifies the likelihood $p(\mathit{y}\mid m,{\mathit{\theta }}_m,\mathit{\xi })$, usually using a Gaussian [42] or Poisson [55] noise model [4]. Typically, the number of stimuli with corresponding neural responses is small relative to the number of weights to be fitted, and the responses are affected by substantial trial-to-trial variability and measurement noise. To mitigate overfitting, regularization is generally required. A popular choice is to use a zero-mean Gaussian prior on the encoding weights (i.e. ridge regression). The performance of each model at predicting neural or voxel responses is evaluated for an independent set of test stimuli. Generalized linear models can also be fitted to predict behavioral data, for example using the softmax to predict probabilities of discrete behavioral choices (e.g., [56–58]).

Representational similarity analysis.

RSA characterizes the representational geometry of a set of stimuli by the multivariate distances among the stimuli in the neural population response space. The predicted experimental outcomes $\mathit{y}$, then, correspond to these distances (one for each pair of stimuli; Fig. 1e). The distance estimates for all pairs of stimuli can be assembled in a square representational dissimilarity matrix (RDM, where the more general term dissimilarity is chosen to include distance estimators that are not distances in the mathematical sense [52]). Alternatively, the pairwise relationships can be captured using a similarity or kernel matrix, e.g. when using the linear centered kernel alignment [59]. RSA is also applicable to behavioral experiments, such as similarity judgments, that yield representational similarity or dissimilarity estimates for pairs of stimuli. The likelihood $p\left(\mathit{y}\mid m,{\mathit{\theta }}_m,\mathit{\xi }\right)$ is typically not explicitly defined (but see [60–62]). Instead, RSA requires that we choose an RDM estimator and an RDM comparator [50], which define which pairs of representations are considered equivalent and what aspects of the representational geometry matter [48, 63]. For example, the vectors of representational dissimilarities from model and brain can be compared using the whitened Pearson correlation or the whitened cosine similarity, which account for the correlational structure of the dissimilarity estimates and approximate the power of likelihood-based model comparison [45, 46]. When the RDM estimator is the squared Euclidean distance and the RDM comparator is the whitened cosine similarity, then the comparison is equivalent [46, 49] to the linear centered kernel alignment [59]. Alternative RDM estimators and comparators differ in their bias and variance at the level of the distance estimates [47, 64] and comparator outcomes [65].

For both encoding models and RSA, we have to consider the parametric flexibility of the mapping from the model to the neural or behavioral measurements [66]. By default, encoding models require $P\times C$ weights, one for each combination of the $P$ model units (predictors) and the $C$ measured response channels (e.g. neurons or voxels). The factor $P$ can be reduced by using a structural prior, for example, a separable encoding model [51, 67], where the unit weights for each response are defined as an outer product of a weight vector across feature maps and a weight vector across spatial locations. Whereas encoding models still always require a substantial number of parameters, RSA enables the evaluation of neural representational predictions without any parameters for a model-to-brain mapping [43, 44, 47, 50]: The RDMs for each model layer and brain region are indexed only by the stimuli and can thus be directly compared. However, some flexibility is often still desired. For example, we may assume that a model has the right set of features (tuning curves), but that the relative prevalences of the features in the neural population are unknown. Weighted RSA enables modeling the relative prevalence of each model feature using one weight per unit (so $P$ rather than $P\times C$ weights) or one weight for each set of units (e.g., for each feature map or layer) [47, 68–71]. If full linear flexibility is desired, RSA can also be used to evaluate representational geometry predictions of linear encoding models [69].

In addition to the parameters involved in mapping the model to the empirical predictions, neural network models have many connection weight parameters, whose values affect model predictions. We can set these parameters by training models to achieve good task performance. Alternatively or additionally, we can fit models to neural and/or behavioral data.

When planning a new experiment, researchers need to consider their uncertainty about both the neural network parameters and the parameters defining the mapping from neural network units to experimental outcomes. Their uncertainty about the parameters determines how vaguely or specifically they can define the scientific hypotheses they wish to adjudicate among. More flexible models make less specific predictions and will tend to be harder to distinguish empirically [35]. However, a model that implements one specific version of a hypothesis may fail to fit the data, even if the hypothesis is correct.

If researchers are certain about all parameters prior to experimental design (a scenario we refer to as “no flexibility” below), controversial stimuli will be relatively easy to generate, since the predicted responses for each stimulus set are unequivocally defined. However, if researchers are still considering adjusting either the parameters of a linear model-to-brain or model-to-behavior mapping (scenario: linear flexibility) or the parameters of the neural network model itself (scenario: nonlinear flexibility), a more complicated approach may be needed. The resulting three scenarios (no flexibility, linear, and nonlinear flexibility) are summarized in Table 1. The design of controversial stimuli critically depends on the level of parametric flexibility that is to be allowed when analyzing the neural or behavioral responses to the controversial stimuli. We now discuss the three scenarios in turn.

Table 1:

Three levels of parametric flexibility of neural network models.

	no flexibility (prefitted, fixed parameters)	linear flexibility (linear mapping-models still to be adjusted)	nonlinear flexibility (end-to-end nonlinear models still to be adjusted)
Will the neural network model be fitted to the new data?	No, the final parameters are considered known.	No, the final parameters are considered known.	Yes, the model will be fitted to neural data.
Will a linear mapping model be fitted to the new data?	No (specific feature weighting assumed)	Yes, a mapping model will be fitted to neural data.	Yes (subsumed in model)
Hypothesis to be evaluated	Architecture, training objective, learning rule, training data, feature weights	Architecture, training objective, learning rule, training data	Architecture
Application of controversial stimuli	Controversial stimuli are straightforward to generate.	Mapping flexibility needs to be accounted for when generating controversial stimuli and evaluating responses to these stimuli.	Controversial stimuli can help reveal whether the architectures are functionally distinct. (If not, training to explain responses to the controversial stimuli will efficiently drive models into alignment.)

3.1.1. No flexibility: all parameters prefitted and considered certain

Prefitted models can be obtained by training each neural network model on a task (e.g., image classification or next-token prediction). If a model-to-brain or a model-to-behavior mapping is needed (e.g. to predict individual responses of neurons or voxels with encoding models or to allow some flexibility in RSA), then this mapping must also be prefitted. Once all parameters are fitted, they are considered known and fixed. Each model’s internal representations or outputs can then be compared to neural or behavioral data without further parameter adjustment. When designing controversial stimuli, the likelihood function $p(\mathit{y}\mid m,\xi )$ of each prefitted model $m$ is straightforward to evaluate because we need not consider different possibilities for parameters ${\theta }_m$. The likelihood depends only on the model $m$ and the stimuli $\mathit{\xi }$ and can be plugged into one of the controversiality objectives described in Section 3.2.

Prefitted models make very specific predictions. By evaluating such a model, researchers test a computational hypothesis that specifies not only the architecture, training objective, learning rule, and training data, but also the relative prevalence in the neural population of each tuning function (and the specific linear encoding if any). Their specific predictions make prefitted models easy to evaluate by experiment. In particular, our professed certainty about the parameters simplifies model comparisons and the optimization of controversial stimuli. Because prefitted models do not hedge their bets, but make strong, specific predictions, they are often easy to reject experimentally.

The main limitation of using prefitted models is that a model might be rejected based on a premature commitment to particular parameter settings, even when the computational hypothesis the model is meant to represent is correct. A model may correctly capture the computations but not the relative prevalence in the neural population of different tuning functions. For example, a qualitatively correct model of V1 that has a quantitatively incorrect receptive-field distribution may fail to accurately predict neural or behavioral responses. Model predictions are often improved substantially when some flexibility is allowed. Even low-parametric flexibility, such as a weighting of the representational features (reflecting the unknown prevalences of different tuning functions in the neural population), can improve prediction accuracy substantially [17, 68–71]. Theoretical progress through model comparison requires that we are honest about our uncertainty about the parameters.

3.1.2. Linear flexibility: model-to-brain or model-to-behavior mapping considered uncertain

Researchers usually cannot be certain about the linear mapping from the trained neural network model to the brain or behavioral data. The parametric flexibility of linear mapping models ranges from a small number of parameters (e.g. one weight per layer or feature map in weighted RSA (e.g. [68]), through a larger number of parameters in a separable encoding model (e.g. [51]), to the full P-unit-by-C-response linear encoding model [4, 42, 72] (Fig. 1e,f).

Models with linear mapping flexibility make less specific predictions than prefitted models. By evaluating such a model, researchers are testing a computational hypothesis that specifies the architecture, training objective, learning rule, and training data, but not the specific linear encoding or relative prevalence in the neural population of each tuning function. Generating controversial stimuli for models with flexible model-to-brain or model-to-behavior mappings is more challenging since the uncertainty about the mapping parameters must be taken into account. Ideally, the stimuli should be controversial for any particular choice of the unknown parameter settings. In order to account for linear flexibility of the models, the unknown parameters can either be marginalized over (ideally analytically [73]) or chosen such that models are maximally confusable (the worst-case scenario). How to achieve this in practice will be described in the section on optimization objectives below.

3.1.3. Nonlinear flexibility: weights of the neural network model considered uncertain

The highest level of model flexibility is to consider all non-linear parameters of the models uncertain at the time of experimental design (e.g., [74–77]). This approach is sometimes referred to as “end-to-end fitting” (emphasizing that all parameters are being trained) or “finetuning” (when a task-pretrained model is further trained to fit the experimental data by adjusting its neural network weights). Models with all parameters still to be fitted make the least specific predictions. By evaluating such a model fitted to the new data starting from random weights, researchers are testing a computational hypothesis that specifies only the architecture (e.g., testing whether a certain architecture can approximate the neural data well [77]). When the model is first pre-trained on a task and then finetuned, the hypothesis may extend to the task as well.

The greater flexibility of end-to-end fitting on the new data renders the models harder to distinguish, thus creating a greater need for controversial stimuli, but it also makes controversial stimuli harder to generate. Marginalizing over the unknown parameters or finding parameters that render models maximally confusable may be intractable in practice. It is attractive to consider iterative experiments with controversial stimuli, where new stimuli are generated in each round to uncover differences among models as the parameters converge to their best settings across experiments.

3.2. Optimization objectives

To pick a stimulus set ($\mathit{\xi }$), researchers need to define an objective to be optimized. Intuitively, the objective should ensure that the models make predictions that are as distinct as possible. However, deriving an objective that quantifies this intuitive notion of distinct predictions requires explicitly articulating the underlying goal.

The goal may be to learn which of a number of brain-computational models best accounts for a brain representation. If researchers assume that their models include the true model, then the question becomes: Which is the true model? Given a data set of outcomes ($\mathit{y}$) generated by an experiment using the particular stimulus set $\mathit{\xi }$, this question is elegantly addressed by probabilistic inference about the identity of the true model: Researchers calculate the likelihood of each model and use Bayes’ rule to update their prior belief about which model is true. But how should they pick a stimulus set $\mathit{\xi }$ that would best inform this update of their beliefs?

In the Bayesian framework, a natural criterion for evaluating a potential experiment is the expected information gain (EIG) [78–80]. The EIG is the expected reduction in our uncertainty about which model is true, where our uncertainty is measured as the entropy of the posterior distribution over model identity (Box 1). The EIG can also be understood as the Shannon mutual information $I(M;Y)$ between the experimental outcome $Y$ and model identity $M$ (capital letters signify random variables here). Because of the symmetry of mutual information, the expected information gain that the experimental outcome would provide about the true model index is equal to the expected information gain that an oracle revealing the true model would provide about the experimental outcome:

$$I(M;Y)=H\left(M\right)-H(M\mid Y)=H\left(Y\right)-H(Y\mid M)$$ (1)

Box 1: Maximizing the experiment’s expected Shannon information gain.

The normative Bayesian perspective on experimental design derives from probabilistic inference and information theory. Say researchers performed an experiment using design $\xi$ and obtained data $y$. The normative approach to assessing which model is true is to use Bayes' rule to infer our posterior over the model index $m$:

$$p(m\mid \mathit{y},\mathit{\xi })=\frac{p(\mathit{y}\mid m,\mathit{\xi })p(m)}{\sum _{m^{\prime }}p\left(\mathit{y}\mid m^{\prime },\mathit{\xi }\right)p\left(m^{\prime }\right)},$$ (2)

where $p(m)$ is the prior probability over the models. If the models have parameters, we can compute the likelihood $p(\mathit{y}\mid m,\mathit{\xi })$ used in Eq. 2 by marginalizing over the model parameters ${\mathit{\theta }}_m$:

$$p(\mathit{y}\mid m,\mathit{\xi })=\int p\left(\mathit{y}\mid {\mathit{\theta }}_m,m,\mathit{\xi }\right)p\left({\mathit{\theta }}_m\right)\mathrm{d}{\mathit{\theta }}_m.$$ (3)

Our goal is to reduce our uncertainty about which model is correct, so the utility of an experiment $\xi$ is the expected information gain as we update our prior in light of the data. Computing this expectation requires marginalizing across all possible true models, their parameters, and all possible datasets they might generate. Before we marginalize over these variables, let us first consider the information gain that experiment $\mathit{\xi }$ would bring us if it yielded a particular dataset $\mathit{y}$. The information gain is the reduction of the entropy (which measures the uncertainty) of our posterior resulting from seeing the data $\mathit{y}$. We call this entropy reduction from the prior $p(m)$ to the posterior $p(m\mid \mathit{y},\mathit{\xi })$ the local utility $u(\mathit{\xi },\mathit{y})$ [78]:

$$\begin{gathered} u(\mathit{\xi },\mathit{y})=H(M)-H(M\mid \mathit{y},\mathit{\xi }) \\ =-\sum _{m}p(m)\cdot \mathrm{l}\mathrm{o}\mathrm{g}p(m)+\sum _{m}p(m\mid \mathit{y},\mathit{\xi })\cdot \mathrm{l}\mathrm{o}\mathrm{g}p(m\mid \mathit{y},\mathit{\xi }). \end{gathered}$$ (4)

By averaging the local utility $u(\mathit{\xi },\mathit{y})$ over all possible outcomes, we obtain a scalar global utility $U(\mathit{\xi })$ for experiment $\mathit{\xi }$:

$$U(\mathit{\xi })=\int p(\mathit{y}\mid \mathit{\xi })\cdot u(\mathit{\xi },\mathit{y})\mathrm{d}\mathit{y}$$ (5)

Because the experimental outcome depends on the experiment $\mathit{\xi }$, the data-generating model $m$ and its parameters ${\theta }_m$, the global utility is an expectation over possible data-generating models and their parameters:

$$U(\mathit{\xi })=\underset{m,{\mathit{\theta }}_m,\mathit{y}}{\mathbb{E}}[u(\mathit{\xi },\mathit{y})]=\sum _{m}p(m)\cdot \int p\left({\mathit{\theta }}_m\right)\cdot \int p\left(\mathit{y}\mid \mathit{\xi },m,{\mathit{\theta }}_m\right)\cdot u(\mathit{\xi },\mathit{y})\mathrm{d}\mathit{y}\mathrm{d}{\mathit{\theta }}_m,$$ (6)

Note that the local utility defined in Eq. 4 does not depend on the true model. In the ideal world of Bayesian inference, the reduction in the entropy of our posterior belief over the models is all we need to consider: If our inferences give us well-calibrated uncertainty, approaching certainty leads us to the correct model. An optimal experiment is $\mathit{\xi }\mathrm{*}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathit{\xi }}{\mathrm{m}\mathrm{a}\mathrm{x}}U(\mathit{\xi })$.

The experimental outcome is unobserved at the design stage, so calculating this criterion requires taking the expectation over all possible outcomes (Fig. 2).

Figure 2: Estimating and optimizing expected information gain using a Monte Carlo approach.

In this simulation, we find a set of stimulus orientations (dashed gray lines in left panels) such that measured responses of an orientation-selective neuron will enable us to adjudicate among three toy models (tuning curves in blue, orange, green). Each toy model predicts Poisson-distributed spike counts with a mean firing rate parameterized by a scaled von Mises tuning curve over stimulus orientation. A particular sample of eight stimulus orientations (“Random stimuli”, dashed gray lines) drawn from a uniform distribution over the interval [0, 180) yields simulated spike counts (black dots) that do not discriminate between model 2 and model 3. Maximizing expected information gain (Box 1) via differential evolution (“Controversial stimuli”, dashed gray lines) yields a stimulus set concentrated at orientations where the three models disagree in their predicted rates. The visualizations show the outcome of a single simulation for each data-generating model and each stimulus set ($\xi$). The global utility was estimated by averaging 3,000 repeats. Across these repeats, model-recovery accuracy was near-perfect for the controversial stimulus set (99.83%) but substantially lower for the randomly sampled stimulus set (73.80%). Log-likelihood values were calculated using the natural logarithm.

If there is remaining uncertainty about model parameters, the utility defined in Eq. 1 must marginalize over each model’s potential parameter settings [79]. For example, for linear encoding models, this entails marginalizing over all potential feature mixing matrices.

When designing experiments to discriminate among models, the aim is to identify the data-generating model, not to most accurately estimate its parameters. Therefore, model parameters can be considered nuisance variables whose influence must be accounted for although they are not the object of the inference.

The optimization criterion of Shannon information gain about the identity of the true model marginalizes over the parameters. Optimizing this criterion will entail a reduction of uncertainty about model parameters only to the extent that reducing parameter uncertainty for the models helps infer which of them is the true model.

The normative Bayesian approach is important as an ideal to aspire to, but it can be difficult to implement in practice. For example, estimating $H(M\mid Y)$ requires computing the expectation across all possible experimental outcomes $\mathit{y}$ of the entropy of the posterior $p(M\mid \mathit{y})$. The experimental outcomes $\mathit{y}$ predicted by the models could be (1) behavioral responses to different stimuli, (2) neural responses of different cells to different stimuli, or (3) representational distances in a neural population for different pairs of stimuli. We consider these scenarios more concretely in the following Section (Controversial stimuli in practice). For models with many parameters (i.e., models with linear or non-linear flexibility, see Sections 3.1.2 and 3.1.3), inference of $p(M\mid \mathit{y})$ is often already too difficult even for a single, specific experimental outcome $y$, unless the model is chosen wisely to make such inference tractable [73]. Data analyses aiming to compare models with many parameters therefore often use cross-validated performance measures for maximum-likelihood estimates of model parameters rather than Bayesian inference. These complications motivate consideration of alternative optimization objectives that are computationally tractable and account for the actual inferential procedure that is to be used to analyze the experimental results.

One such alternative approach that maintains important elements of the Bayesian framework is to maximize the probability of recovering the true data-generating model (Box 2), which we refer to as the recovery accuracy. This method is highly versatile because the inferential procedure used for comparing models given the experimental outcome is not limited to Bayesian inference. However, to estimate the recovery accuracy, researchers must still specify priors for the models and their parameters and marginalize over these unknowns.

Box 2: Maximizing the experiment's expected model-recovery accuracy.

A natural alternative to maximizing Shannon information gain is to maximize the probability of selecting the true data-generating model. This is equivalent to maximizing the expected model-recovery accuracy or minimizing model recovery "0–1 loss" (e.g., [87]). Here, the local utility is defined as success in correctly identifying the data-generating model given a design $\mathit{\xi }$, ground-truth model $m$, experimental outcome $y$, and a model comparison procedure. Specifically, if the fit of each model to the simulated data $\mathit{y}$ is quantified by a score $S(\mathit{y},\mathit{\xi },m)$, the local utility can be defined as:

$$u(\mathit{\xi },\mathit{y},m)=\mathrm{𝟙}\left[\forall m^{\prime }\ne m:S(\mathit{y},\mathit{\xi },m)>S\left(\mathit{y},\mathit{\xi },m^{\prime }\right)\right]$$ (7)

(i.e., 1 if the model with the highest score is the true model, and 0 otherwise). Note that if the model set includes a model that is not distinguishable from the true model, Eq. 7 will always return 0.

A more permissive criterion is the fraction of alternative models that the true model dominates, which is the average of the model-recovery accuracy over all pairwise model comparisons:

$$u(\mathit{\xi },\mathit{y},m)=\frac{1}{\left|ℳ\right|-1}\sum _{m^{\prime }\ne m}\mathrm{𝟙}\left[S(\mathit{y},\mathit{\xi },m)>S\left(\mathit{y},\mathit{\xi },m^{\prime }\right)\right].$$ (8)

The global utility is the expected value of the local utility (Eq. 7 or Eq. 8) over all possible data-generating models, their parameters, and the resulting experimental outcomes:

$$U(\mathit{\xi })=\sum _{m}p(m)\cdot \int p\left({\mathit{\theta }}_m\right)\cdot \int p\left(\mathit{y}\mid \mathit{\xi },m,{\mathit{\theta }}_m\right)u(\mathit{\xi },\mathit{y},m)\mathrm{d}\mathit{y}\mathrm{d}{\mathit{\theta }}_m.$$ (9)

Note that this criterion is considerably more versatile than Shannon information gain. The model performance score $S$ can reflect posterior model probabilities, but alternatively, it can also reflect non-Bayesian measures such as the goodness or predictive performance of maximum likelihood fits, or even non-probabilistic measures such as the rank prediction accuracy (Spearman or Kendall rank correlation coefficient) or a cross-validated measure of generalization performance.

Finally, one can adopt a frequentist approach and optimize the expected discrepancy between each model’s predicted distribution of experimental outcomes and the predictions of alternative models (Box 3). If a data-generating model is compared to an alternative model using the two models’ log-likelihood ratio, then the expected discrepancy is the Kullback-Leibler (KL) divergence between the distributions predicted by the two models [81]. If the alternative model has unknown parameters, these can be adjusted to minimize the discrepancy, so as to consider the worst-case scenario of model confusability. Stimuli are then chosen to maximize this minimum, turning the optimization into a maximin problem. If the data-generating model has unknown parameters, the expectation must be taken with respect to their possible values (via either integration or sampling). Because the true model is unknown during experimental design, researchers may sample the model from a flat prior and take the expected value. When comparing each model to multiple alternative models, researchers can either average the divergences across all pairwise comparisons or consider the most challenging comparison (i.e., the smallest divergence). Maximizing the expected discrepancy of the predicted distributions over experimental outcomes is a highly intuitive objective, and in some cases, it enables researchers to capitalize on analytical solutions to the optimization problem [82–85]. Note, however, that the three optimization objectives discussed (Boxes 1–3) are distinct in general. An experimental design that maximizes the expected discrepancy between two models’ predicted distributions does not necessarily maximize the model-recovery accuracy or the Shannon information gain.

Box 3: Maximizing the expected KL divergence between model-predicted distributions.

The frequentist perspective on optimal experimental design suggests directly maximizing the discrepancy between the predictions of different models. Assume that we compare two models, $m_1$ and $m_2$, and that, without loss of generality, we designate model $m_1$ as the true model and a parameter vector ${\mathit{\theta }}_1$ as the true parameters of this model. In this setting, the KL optimality criterion is defined as the Kullback-Leibler divergence $\left(D_{\mathrm{K}\mathrm{L}}\right)$ between the predictions of the two models, with the parameter vector ${\mathit{\theta }}_2$ chosen to minimize the divergence [88]:

$$I_{\mathrm{2,1}}\left(\mathit{\xi },{\mathit{\theta }}_1\right)=\underset{{\mathit{\theta }}_2}{\mathrm{m}\mathrm{i}\mathrm{n}}{D}_{\mathrm{K}\mathrm{L}}\left[p\left(Y\mid \mathit{\xi },m_1,{\mathit{\theta }}_1\right)\Vert p\left(Y\mid \mathit{\xi },m_2,{\mathit{\theta }}_2\right)\right]$$ (10)

$$=\underset{{\mathit{\theta }}_2}{\mathrm{m}\mathrm{i}\mathrm{n}}{\int }_{\mathit{y}\in 𝒴}p\left(\mathit{y}\mid \mathit{\xi },m_1,{\mathit{\theta }}_1\right)\cdot \mathrm{l}\mathrm{o}\mathrm{g}\frac{p\left(\mathit{y}\mid \mathit{\xi },m_1,{\mathit{\theta }}_1\right)}{p\left(\mathit{y}\mid \mathit{\xi },m_2,{\mathit{\theta }}_2\right)}\mathrm{d}\mathit{y}.$$ (11)

The KL optimality criterion can be viewed as the expected log-likelihood ratio between the data-generating model and the alternative model, considering the worst-case setting of the alternative model's parameters (those that render the alternative model most confusable with the true model). The KL optimality criterion greatly simplifies if we assume that the models share a multivariate Gaussian error distribution, that is, ${\mathit{y}}_1={\eta }_1\left(\mathit{\xi },{\mathit{\theta }}_1\right)+{\mathit{ϵ}}_1$ and ${\mathit{y}}_2={\eta }_2\left(\mathit{\xi },{\mathit{\theta }}_2\right)+{\mathit{ϵ}}_2$, where ${\eta }_m$ is a function that maps an experimental design and model parameters to the predictions of model $m$, and ${ϵ}_1$ and ${ϵ}_2$ are vectors of normally distributed residuals, ${\mathit{ϵ}}_1,{\mathit{ϵ}}_2\sim N(0,\mathbf{\Sigma })$. In this case, the KL optimality criterion reduces to the minimal squared Mahalanobis distance between the two vectors:

$$I_{\mathrm{2,1}}\left(\mathit{\xi },{\mathit{\theta }}_1\right)=\frac{1}{2}\underset{{\mathit{\theta }}_2}{\mathrm{m}\mathrm{i}\mathrm{n}}\left[{\left({\eta }_1\left(\mathit{\xi },{\mathit{\theta }}_1\right)-{\eta }_2\left(\mathit{\xi },{\mathit{\theta }}_2\right)\right)}^T{\mathbf{\Sigma }}^{-1}\left({\eta }_1\left(\mathit{\xi },{\mathit{\theta }}_1\right)-{\eta }_2\left(\mathit{\xi },{\mathit{\theta }}_2\right)\right)\right]$$ (12)

Note that the covariance matrix $\mathbf{\Sigma }$ may depend on the experimental design $\mathit{\xi }$. If we assume that the covariance matrix $\mathbf{\Sigma }$ is a scaled identity matrix ${\sigma }^2\mathbf{I}$ and that ${\sigma }^2$ is independent of the experimental design $\mathit{\xi }$, then the KL optimality criterion is proportional to the classic T-optimality criterion [82, 83, 88]:

$$T_{\mathrm{2,1}}\left(\mathit{\xi },{\mathit{\theta }}_1\right)=\underset{{\mathit{\theta }}_2}{\mathrm{m}\mathrm{i}\mathrm{n}}{‖{\eta }_1\left(\mathit{\xi },{\mathit{\theta }}_1\right)-{\eta }_2\left(\mathit{\xi },{\mathit{\theta }}_2\right)‖}^2$$ (13)

In the frequentist perspective, the natural way to generalize this objective to the more realistic case where the true model and its parameters are unknown is to take the minimum over these choices:

$$\begin{gathered} U(\mathit{\xi })=\underset{m}{\mathrm{m}\mathrm{i}\mathrm{n}}\underset{{\mathit{\theta }}_m}{\mathrm{m}\mathrm{i}\mathrm{n}}\underset{m^{\prime }\ne m}{\mathrm{m}\mathrm{i}\mathrm{n}}{I}_{m^{\prime },m}\left(\mathit{\xi },{\mathit{\theta }}_m\right) \\ =\underset{\begin{array}{c}m,m^{\prime },{\mathit{\theta }}_m,{\mathit{\theta }}_{m^{\prime }}\\ m\ne m^{\prime }\end{array}}{\mathrm{m}\mathrm{i}\mathrm{n}}{D}_{\mathrm{K}\mathrm{L}}\left[p\left(Y\mid \mathit{\xi },m,{\mathit{\theta }}_m\right)\Vert p\left(Y\mid \mathit{\xi },m^{\prime },{\mathit{\theta }}_{m^{\prime }}\right)\right] \end{gathered}$$ (14)

This objective characterizes the worst-case scenario for model discrimination. Note the analogy to Eq. 6, where models and parameter settings are instead weighted by their probabilities and averaged over. If researchers have trustworthy estimates of the prior over the models $(p(m))$ and the prior over the parameter settings $\left(p\left({\mathit{\theta }}_m\right)\right)$ for each model, they can take expectations instead of considering the worst case. Researchers can also interpolate between averaging and minimization by choosing which variables to average over and which ones to take the minimum over. If researchers do not have trustworthy priors over models, they can use an unweighted average over all pairwise model comparisons as in Eq. 8. All combinations of averaging and minimization yield sensible optimization objectives. As for the other objectives (Boxes 1 and 2), an optimal experiment is $\mathit{\xi }\mathrm{*}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathit{\xi }}{\mathrm{m}\mathrm{a}\mathrm{x}}U(\mathit{\xi })$.

A simpler and more computationally affordable approach to addressing uncertainty about model parameters is to apply iterative cycles of model fitting, stimulus optimization, and experiment. Such an iterative approach has been employed successfully in studies manipulating neural responses to test the predictions of a single flexible model [67, 74, 86]. The iterative approach simplifies the computations by settling for point estimates of the model parameters at each step. However, it updates these estimates iteratively, such that, ideally, each model comes to be represented by its optimal fitted instance. The fitting and evaluation at each iteration should take advantage of all previously recorded responses, including responses to natural stimuli, to ensure that the models do not drift away from explaining responses to ecologically relevant stimuli. Unlike fully Bayesian adaptive experimental design, iterative fitting is not guaranteed to converge to the correct solution. Future work should explore how well such heuristic approaches work in practice.

3.3. Stimulus search spaces

Next, one must define the space within which the optimization procedure searches for a stimulus set. This definition includes the admissible stimuli (the set of potential solutions) and the stimulus parameterization (the optimized variables). Neuroscientific computational models should explain brain function under natural conditions, so the natural distribution of stimuli (those subjects encounter during their daily lives) has a privileged status. One approach is to restrict the experimental stimuli to stimuli that are either natural or naturalistic, i.e., not too far from the natural distribution. Such stimulus sets are described as ecologically valid. Moreover, a model may have a specific stimulus domain over which it is intended to operate. Models of reading or face identification, for example, may be intended only to make predictions for text or face images, respectively. As noted above, however, to the extent that researchers take a model seriously as a mechanistic model, it should also make valid predictions when challenged with stimuli outside its designated domain.

A simple way to define the admissible stimuli is as a predefined collection of stimuli, which could be restricted to those relevant for the model (Fig. 1a). The experiment can then be optimized by selecting a subset of the admissible set that maximizes a model-discrimination objective [30, 31, 37, 39, 89]. This approach can be interpreted as employing a discrete parameterization in which each stimulus in the collection is represented by a binary parameter that indicates its inclusion in the experimental stimulus set.

Instead of selecting stimuli among a predefined collection, the optimization process can select stimuli from a richer discrete or continuous parameter space. This approach is sometimes called stimulus synthesis because the stimulus space contains novel possibilities. Such larger stimulus spaces offer greater flexibility than choice among a predefined collection (see Fig. 3 for examples). However, researchers need to consider whether the resulting stimuli are sufficiently ecologically valid.

Figure 3: Stimuli synthesized for model discrimination: existing work.

(a) Maximum differentiation competition [29]. A reference natural image is subjected to an initial distortion and then optimized in pixel space to generate a pair of images to which one perceptual model assigns maximally different distortion strengths (‘Best’ and ‘Worst’), while the predictions of the other model are held fixed. When the distortion strength predicted by structural similarity (SSIM) is manipulated while the distortion strength predicted by mean-squared-error (MSE) is fixed (vertical arrows), human judgments of the perceived quality of the resulting images change more strongly than in the converse scenario (horizontal arrows), supporting the SSIM model. (b) Controversial stimuli for classification models [37]; a random noise image is optimized to elicit distinct classifications in pairs of classifiers (e.g., the digit 3 in one model and 7 in another). The labels assigned by models with generative components, especially the Schott analysis by synthesis (ABS) model, are best aligned with human perception. (c) A reference stimulus s (digit “7", middle) is perturbed in a direction that maximally distorts local geometry in one model while minimally affecting it in another (left), and vice versa (right) [40]. (d) A stimulus perturbed along a “principal distortion” axis, which maximizes variability in models’ sensitivity to local geometry distortion [41]. (e) Sets of face pairs are synthesized using Basel Face Space to elicit distinct face-dissimilarity ranking predictions from alternative models [38]. (f) Natural English sentences drawn from a corpus (white dots) are considered similarly probable by two language models (axes). Sentence synthesis by discrete optimization (iteratively replacing words) yields controversial sentences (gray dots) that elicit contrasting probabilities in the two language models [39].

A key choice in stimulus synthesis is the parameterization of the stimuli. For images, one simple parameterization is pixel intensities (Fig. 1b). When optimizing pixel intensities, the admissible stimuli may include all displayable images. Stimulus synthesis can be guided more strongly by using a different parameterization, such as the latent representation of a stimulus-generating model (Fig. 1c). For example, researchers can search for controversial stimuli within a manifold of naturalistic images by optimizing the stimuli in the latent representation of a neural-network-based generator, such as a Generative Adversarial Network (GAN) trained to generate naturalistic images [54]. Such an approach can be understood as assuming a naturalistic prior that excludes many possible stimuli. For instance, the manifold of possible images spanned by a model trained to transform a vector of 200 elements into an image is dramatically smaller than the space of all possible images (Figure 12.20 in [90]). If the models have a limited scope, the admissible stimuli can be restricted by using a domain-specific generator. For faces, for example, the latent variables of the Basel Face Model (BFM, [91]) define a continuous space of 3D shapes and textures for human faces [38, 92] (Fig. 3e). Recent advances in generative AI, including GANs as well as diffusion and flow-based models [93, 94], can enable researchers to design highly naturalistic controversial stimulus sets by continuous optimization.

Note, however, that using a generator does not automatically guarantee ecological validity. In contrast to randomly sampling stimulus latents, optimizing latents can strain the generator and yield stimuli that lie outside the intended stimulus manifold. The optimized latents are then like adversarial examples for the generator: making it fail at its task of generating natural stimuli. Additional safeguards—such as manual or automatic inspection—may therefore be necessary.

Restricting the admissible set, e.g., to human face images, imposes a hard prior that excludes many stimuli from consideration. The parameterization additionally imposes a soft prior by shaping the optimization trajectory such that some stimuli are more likely to be included than others. The optimization is usually set up for a predefined number of stimuli. Each stimulus is represented by a row of the parameter matrix. The rows are typically initialized with independent random samples from a suitable prior. When using local optimization procedures such as gradient descent to find local maxima of controversiality in the space of stimulus sets, solutions that are closer to the initialization are more likely to be selected than solutions that are further away. Even when the set of admissible stimuli is held constant, the parameterization matters because it determines the distance of different solutions from the initialization and the gradients that shape the trajectory. An invertible linear transform of the parameterization, for example, can change which of multiple local optima the search converges to [95].

Stimulus synthesis can be further regularized by introducing penalty terms to the optimization objective. For example, [67] used a total variation penalty to promote spatially smooth images. The process can also be regularized by inserting a stochastic stimulus presentation model into the pipeline that links the stimulus set parameters to the controversiality objective. Stochastic stimulus presentation (Fig. 1, a-c) can help create stimulus sets that are robustly controversial. For example, if images are jittered during optimization [95], they will elicit contrasting responses from the models irrespective of their precise placement in the input array, accounting for random variation in stimulus presentation during an experiment, where the animal may not fixate perfectly. Penalty terms and stochastic variation are both forms of regularization that change the loss landscape, thus altering the final solution.

3.4. Optimization procedures

Given a stimulus search space and an optimization objective, finding a controversial stimulus set is an optimization problem that can be approached using standard optimization procedures. Since neural network models are complex and nonlinear, it will usually be impossible to find a set of controversial stimuli that is guaranteed to be globally optimal. However, it is often possible to find stimulus sets that are far more controversial than random samples and yield substantial power increases for inferential model comparison. In this respect, the problem of optimizing an experiment resembles the problem of optimizing a neural network model: Both are nonconvex optimization problems for which iterative methods can yield powerful solutions that are not guaranteed to be optimal.

A central distinction for the choice of optimizer is whether the space of candidate stimulus sets is continuous or discrete. Even if the space is discrete and finite, we will not be able to try all possible stimulus sets and thus still rely on iterative optimization. The continuous and discrete scenarios are similar in that the optimizer finds a solution by taking small steps in parameter space that increase the controversiality. When the process converges on a local optimum, it will not be possible to find further improvements. However, continuous and discrete scenarios differ substantially in terms of the steps one can take. In a continuous space of stimulus sets, we can take arbitrarily small steps and, if the controversiality objective is differentiable, its gradient gives us the direction in which controversiality rises. In a discrete space, a step may correspond to exchanging a stimulus for a different stimulus, or to a discrete modification of one of the stimuli. We therefore consider completely different classes of optimization procedures in the continuous and discrete scenarios.

For the discrete stimulus-selection problem, where an experimental stimulus set is to be assembled from a large collection of candidate stimuli, even simple approaches, such as starting with a random set of the desired size and greedily replacing stimuli to improve the objective, can be effective [30, 34, 39]. If the optimization objective is noisy (e.g., its estimation involves Monte Carlo sampling), more principled stochastic optimization approaches [96] or evolutionary algorithms [97] may yield better results.

For continuous stimulus parameterizations, gradient descent and its variants can be highly effective for both deterministic and stochastic objectives. However, gradient descent requires that the objective be differentiable with respect to the stimulus parameters, so all intervening components, including all models and the stimulus generator (if any) must be differentiable. If the gradient is not available in closed form, we can use a local estimate of the gradient based on taking small steps around the current best guess. This approach may not work well when the evaluation of the optimization objective is noisy, because the effect of the small steps can become undetectable under the noise. A versatile and derivative-free alternative for navigating both continuous and discrete stimulus spaces is evolutionary optimization (e.g. as applied in [31, 54]).

Because all optimization techniques we discuss here are local, the initial stimulus set, the stimulus parameterization, and the chosen optimization algorithm can influence the outcome. Approximate optimization procedures are not “neutral” or “prior-free” methods to choose the best stimulus set for a given model discrimination task. Rather they impose their own bias, which the researcher should be aware of.

Although beyond the scope of this review, recent advances in the Bayesian experimental design literature offer efficient new ways to optimize experiments (for a review, see [98]). Briefly, these advances include unbiased Monte Carlo estimators of the expected information gain (Eq. 1) [99]; variational bounds that can sidestep the computational costs associated with estimating and optimizing this utility [100]; and even amortizing the optimization with a separate neural network model that designs the next trial given previous data [101]. These methods have yet to be integrated into stimulus selection and synthesis, where an additional challenge is that the search space (the space of stimulus sets) is typically of higher dimension.

4. Controversial stimuli in practice

Now that we have a sense of the four requirements for optimizing stimuli for model comparison (the models, the optimization objective, the stimulus search space, and the optimization procedure), we review studies that have applied the method of controversial stimuli and made particular choices for each. The stimuli in these studies are either synthetic (Fig. 3) or selected from a larger set (Fig. 4). Related methods that are beyond the scope of this review are (a) low-dimensional stimulus-set optimization for model discrimination (e.g. optimizing the brightness or timing, a setting thoroughly addressed by the literature on optimal experimental design and adaptive psychophysics, e.g., [102]) and (b) high-dimensional stimulus-set optimization with other objectives that do not measure disagreement between models. The latter class includes methods that generate stimuli that are predicted by a single model to elicit (1) extreme responses [54, 67, 74, 92, 103], (2) responses that are very distinct from those elicited by close-by reference stimuli [86, 104–106], (3) responses that are identical to those elicited by reference stimuli (such that the synthesized and reference stimuli are metamers) [107, 108], or (4) visual illusions [109, 110]. These four approaches employ optimization to elicit bold predictions from a single model. The distinguishing feature of controversial stimuli is that they are designed to make multiple models disagree in their predictions about neural or behavioral responses.

Figure 4: Stimuli selected for model discrimination: existing work.

(a) RDMs of natural scene images selected to decorrelate the predictions of three models: human judgments of functions (i.e., whether certain actions can be performed in a scene), human judgments of object categories present in the images, and the penultimate-layer activations of a CNN in response to the same images. [30] (b) Average representational agreement (1 + RDM correlation) between seven language models, for random English sentences (above diagonal) and sentences selected to decorrelate model predictions (below diagonal) [34]. (c) Example visual stimuli from a similar selection procedure, aiming to increase (left) or decrease (right) the correlation between the RDMs of visual neural network models [34]. (d) Spearman-correlation-coefficient matrices (left) for five subjective measures of object animacy. The top matrix depicts correlations for randomly sampled stimuli, and the bottom for a stimulus set selected to decorrelate the five measures. Brighter shades indicate weaker correlation [31]. The right panel shows example stimuli from the controversial (i.e. the predictor-decorrelating) set. Stimuli are ordered by their rank on each of the subjectively rated measures. (e) A triplet of natural-scene images selected such that each image is the odd-one-out according to a different image property [89].

For practical reasons, studies using optimization of high-dimensional stimuli for model discrimination have relied on heuristics that do not exactly implement the normative objectives described in Boxes 1–3. However, the normative objectives are our north star and help clarify the assumptions or approximations implicit to the heuristic approaches that have been used in practice.

Across domains, the studies reviewed below illustrate a common practical benefit: stimulus sets optimized for model discrimination can target conditions under which candidate models make divergent predictions, thereby increasing empirical discriminability relative to standard stimulus sampling. In some cases, the resulting data support decisive comparisons among models, revealing clear winners; in others, they expose hidden shortcomings of all models. Either way, controversial stimuli can reduce ambiguities that arise when multiple high-capacity models achieve similarly high scores on naturalistic test sets by providing more informative empirical tests.

4.1. Discriminating models with comparative behavioral judgments

Consider the problem of selecting a model for predicting which of two corrupted images will be judged as having a higher image quality. Such models can help assess alternative methods for lossy image compression. Wang and Simoncelli [29] developed an efficient method called Maximum Differentiation (MAD) competition for distinguishing two models of perceived image quality (Fig. 3a). They considered the mean-squared-error (MSE) model and the structural similarity (SSIM, [111]) model, which provide two ways of measuring the dissimilarity between the original and the corrupted image. In a 2-alternative-forced-choice experiment, subjects viewed a natural image along with two subtly corrupted images. The task was to pick the less perceptually corrupted image. To compare the models, MAD generated two kinds of corrupted-image pairs. In some pairs, the MSE is matched between the two corrupted images, while the SSIM differs maximally. In other pairs, the SSIM is matched, and the MSE differs maximally. The SSIM model better predicted which image corruptions people are more sensitive to [29].

The MAD approach can drive high-dimensional stimulus optimization, enabling efficient comparison of models of perceptual judgments or neural responses. MAD can be extended to compare multiple models in a pairwise fashion [112]. MAD stimuli are not necessarily optimal according to any of the normative objectives described in Boxes 1–3. However, holding one model’s prediction constant while manipulating the other’s prediction provides a highly interpretable experimental design, where individual comparison trials can reveal differences between the models.

The restriction of holding the prediction of one model fixed between the two stimuli can be relaxed to an inequality constraint: When modifying a stimulus to reduce the predicted quantity according to one model, the prediction of the other model can be constrained to remain at or above its original level. This relaxation allows more options, which can make the optimization easier. When the search space is discrete, for example, the optimizer may fail to find any stimulus modification that keeps one model’s predictions exactly constant. Golan and colleagues [39] used an inequality constraint to compare language-model predictions of human judgments (Fig. 3f). Human subjects were presented with pairs of English sentences and judged which of the two is more likely to be encountered in natural English (linguistic acceptability judgment). Neural network language models assign probabilities to sentences, and each model was taken to select the sentence to which it assigned the higher probability. Pairs of sentences were generated by drawing a natural sentence from a corpus and iteratively replacing words, so as to decrease the probability of the sentence according to one model while keeping it at least equal according to the other model. All models performed well and indistinguishably for random natural sentence pairs. Controversial sentences revealed that the generative pretrained transformer model (GPT-2) better predicts the human judgments than alternative recurrent neural networks, discriminative language models, and simple n-gram models.

Greene and Hansen [89] constructed 100 triplets of natural-scene images such that, within each triplet, each of the three images served as the odd-one-out along a different human-labeled attribute—material (e.g., wood), affordance (e.g., related to eating), or surface property (e.g., glossy). They found that human odd-one-out judgments (i.e., selecting the least similar image) were predominantly driven by affordances.

4.2. Discriminating models with behavioral judgments of single stimuli

In the studies above, subjects made comparative judgments given stimulus pairs or triplets (e.g., picking the higher-quality image or the sentence more likely to be encountered). An alternative approach is to have subjects judge each stimulus independently (e.g. estimating the image quality or the probability of a sentence on a continuous scale or assigning a label to each stimulus). Judgments of individual stimuli have been successfully used to compare models of image classification. Consider the task of classifying an image as depicting a cat or a dog. A researcher can select or synthesize an image classified as a dog by one model and as a cat by another model. Let’s assume one model discriminates cats and dogs by their ears, and the other by their mouths. Both models may recognize cats and dogs well, and both may predict human classifications accurately for natural images of cats and dogs. Controversial images can reveal the different mechanisms employed by the two models: the internal features discriminating cats from dogs by either ears or mouths. The human response to a stimulus that two models disagree over is guaranteed to provide evidence against at least one of the models. It can provide evidence against both models if humans do not perceive either a cat or a dog.

Golan and colleagues [37] showed that such individually controversial images provide a powerful tool for distinguishing the distinct perceptual mechanisms employed by alternative image classification models. They created controversial images for each combination of a pair of models and a pair of image labels (Fig. 3b). Human classifications for handwritten digits and small natural object images were better predicted by classification models that incorporated image-generative components than by purely discriminative classification models.

The method of controversial stimuli as used in [37] shares some important features with MAD. Like MAD, it is not equivalent to maximizing the Shannon information gain, the model-recovery accuracy, or the KL divergence of the predictive distributions. However, the controversiality objective renders each stimulus and the human response it elicits highly interpretable. Future studies may use alternatives to the pairwise approach to compare multiple models. For example, one could generate stimulus sets that jointly maximize Shannon information gain or model-recovery accuracy, taking into account the entire set of candidate models. Optionally, rather than relying on pre-designated class pairs, the choice of class pairs to leverage for model comparison could be left up to the optimization algorithm.

4.3. Discriminating models with measurements of representational geometry

Most studies that compare models in terms of the accuracy with which they predict representational geometries (RSA) use randomly sampled or manually selected stimuli. However, a few pioneering studies have begun to explore how stimulus sets can be algorithmically selected or generated such that different models make more distinct predictions of representational dissimilarities [30, 34, 38].

How can researchers best make models disagree in their predictions of the representational geometry? Exact implementations of the theoretical objectives detailed in Boxes 1–3 remain outstanding, but a number of effective heuristic approaches have instead been used. One heuristic is to minimize the correlation for each pair of predicted RDMs [30, 34]. For example, Groen and colleagues [30] employed discrete optimization to minimize the correlation between the RDMs of three distinct image representations: neural network features, human-derived action labels, and human-derived object labels (Fig. 4a). The resulting stimulus set was used in a behavioral and an fMRI experiment. Both experiments were analyzed with RSA, revealing that behavioral judgments were consistent with both action labels and neural network features, whereas representations in scene-selective cortical regions were explained solely by neural network features. Hosseini and colleagues [34] employed a similar approach to find a sentence set that elicits decorrelated RDMs in different language models (Fig. 4b) or to find an image set that elicits decorrelated RDMs in different visual neural network models (Fig. 4c). The controversial stimuli elicited significantly lower correlations between model and brain RDMs, revealing shortcomings of the models that were not apparent with randomly sampled stimuli.

The RDM-decorrelation approach is intuitive, straightforward, and conceptually related to maximizing the discrepancy between model predictions (Box 3). However, it neglects consideration of the noise. Consider the scenario where the dissimilarities of one of the models are similar across pairs of stimuli. Such a representational geometry can arise either when the model does not distinguish any pair of stimuli (all stimuli are metamers for the model) and the RDM contains only zeros, or when the model distinguishes all pairs of stimuli, but each pair has approximately the same dissimilarity. Even if the model is true, the correlation of its predicted RDM with the empirical RDM will be close to 0, since measurement noise will dominate the RDM correlation. We refer to this issue—where model dissimilarities are approximately equal—as the flat-RDM problem. When using a correlation coefficient as the RDM comparator in model comparisons, the model RDMs need to have sufficient dissimilarity signal variance relative to the noise, in addition to distinct patterns of dissimilarities. Here, the signal variance is the variance across the predicted model dissimilarities, and the noise is the error variance of the empirical dissimilarity estimates. Merely minimizing the correlation between model RDMs does not ensure that the dissimilarity signal-to-noise ratio is sufficient. The optimization procedure can drive two model RDMs to be uncorrelated or even anticorrelated, but if this entails a reduction of the spread of dissimilarities in one or both of the model RDMs relative to the noise, then the models might not perform significantly differently at predicting the empirical RDM.

The flat-RDM problem can be avoided by taking the signal-to-noise ratio into account. The optimization must ensure that, if a model represents the ground truth, its RDM correlation with a noisy empirical RDM will be higher than that of other models [38]. There are multiple sources of noise that can be considered: measurement noise, stochasticity of the predicted responses, and inter-participant variability, which may be modeled as parameter uncertainty or using variability between neural network realizations [38, 113].

The approaches mentioned above compare models based on their global representational geometry—the relations among the representations of clearly distinct stimuli. Alternatively, models can be compared based on their local representational geometry—how the representation of a given stimulus changes when the stimulus is slightly altered [40, 41]. Zhou and colleagues [40] investigated the local geometry by identifying a direction in input space along which a perturbation of the stimulus maximizes the change of the representation in one model relative to the change of the representation in the other model (Fig. 3c). Conceptually, this approach shares the motivation of MAD competition to change the prediction of one model as much as possible while minimally affecting the prediction of the other model. Unlike MAD, this local perturbation approach relies on linearizing model responses around the response elicited by a given stimulus and can be computed for vector representations (whereas MAD is defined only for scalar outputs). In two example applications, the approach identified shortcomings in two chromatic threshold models—motivating a third, improved model—and showed that human discrimination aligned more closely with a sparse autoencoder rather than a dense one. The local perturbation objective can be generalized to target more than two models [41] (Fig. 3d).

4.4. Discriminating models with neural response measurements

When models predict individual neural responses (rather than representational geometries), stimuli can be optimized to make those predictions diverge. Bashivan and colleagues [67] fitted linear mappings to predict the responses of individual V4 neurons from the representation in a single neural network model. Using this model, synthetic stimuli were generated to activate specific neurons while suppressing the responses of other neurons. This study aimed to control neural activity using one model rather than compare multiple models, so it is not an example of controversial stimuli. However, similar techniques could be used in the future to synthesize controversial stimuli: stimuli predicted by one model to activate a specific neuron, but predicted by another model to suppress the same neuron (and vice versa). A controversiality objective could also be defined for a set of recorded neurons to measure the extent to which a candidate stimulus drives two models to predict distinct neural activity patterns. Such an objective could further be generalized to multiple stimuli, multiple responding neurons, and multiple models, to ensure that each model can be discriminated from each other model. The resulting controversial stimuli for encoding models may enable us to reveal differences in the ability of different neural network models to account for a neural population code, even when the models all perform similarly (given the flexibility of their linear mappings to the neural recordings or fMRI data) at explaining responses to natural stimuli.

A tempting choice of controversiality objective is to minimize the correlation between the response patterns predicted by alternative models. However, the controversiality objective needs to account for measurement noise. A stimulus that elicits predictions from two models that are decorrelated but not sufficiently distinct given the noise is unlikely to support accurate model selection, whether the predictions are representational dissimilarities (in RSA, as discussed above) or responses of individual neurons or voxels (encoding models). Experiments employing stimuli designed to be controversial among encoding models have only appeared in preliminary results [114, 115].

5. Outlook

5.1. Neural network models enable comparisons of computational hypotheses

Artificial neural networks can express distinct hypotheses about neural mechanisms [1, 5, 16], yet they are also high-capacity function approximators. Their capacity to store knowledge is essential to their ability to explain intelligent behavior, but their high flexibility can make it challenging to empirically adjudicate among models so as to make theoretical progress. Despite their flexibility, however, two models implementing different neural mechanisms will have distinct inductive biases that determine how the models generalize beyond the training distribution. Which is a better model of a neural mechanism can therefore be revealed by out-of-distribution challenges, where generalization transcends mere curve fitting. A model trained to predict responses to one category of natural stimuli, but evaluated on responses to another category, is making out-of-distribution predictions. Similarly, a model trained on natural stimuli but tested on artificial stimuli or stimuli optimized for model comparison is making out-of-distribution predictions. Hidden-layer representational predictions of a model trained solely on a behavioral task objective (see Section 3.1.1) also constitute an out-of-distribution generalization challenge because the model was not fitted to predict neural representations ([2, 3]). Such out-of-distribution prediction challenges can reveal the inductive biases implicit in the architecture, objective, and learning rule [6], even for models with millions or billions of parameters [37, 39].

5.2. Finding stimuli that expose meaningful model differences

When comparing the representations learned by two or more neural network models, representational features can be conceptually grouped into two sets: (1) features shared between the models and (2) features that differ between the models. The latter set can be further subdivided into (2a) features that differ because of meaningful differences between models in terms of architecture, training task, training data, or learning rule, and (2b) features that differ between models because of nuisance variability—for example, due to different weight initializations or training-example presentation orders.

Recently, there has been a growing interest in representational similarities found in qualitatively distinct models [32–34]. Neural networks with different architectures or trained on different tasks do not only exhibit similar predictivity of brain-activity data [17, 18]. Diverse models also acquire features they all share [32], and it appears that these features shared among the models are also more likely to be shared with biological representations [33]. The shared features likely reflect regularities of the environment from which the data was sampled [32]. For example, object animacy often explains a considerable portion of the variance in higher-order visual representations in diverse neural network models, as well as in primate cortex (e.g., [116, 117]).

The observation that features shared among model representations are also often shared with brain representations has led some authors to suggest that differences between models primarily reflect nuisance variability: models disagree on features that the brain does not care about [33, 34]. If this hypothesis were correct in a strong sense (i.e., feature set 2a above is empty), then generating controversial stimuli would be futile: Any feature not shared across models would not be present in brains. An experiment with controversial stimuli would then fail to reveal significant differences between models in terms of their brain alignment. However, when optimizing controversial stimuli for models that are qualitatively distinct—for example, models trained on different tasks—the results of experiments with controversial stimuli often do reveal that certain models are significantly more brain-aligned than others [37–39]. This indicates that the differences among the models do lend them distinct inductive biases and that some of their distinct features are present also in the brain representations, enabling us to adjudicate among the models by experiment.

In order for model comparisons to drive theoretical progress, the models must implement distinct computational hypotheses. Controversial stimulus generation should capitalize on meaningfully distinct features (feature set 2a mentioned above) rather than on nuisance variation of features (feature set 2b) that can arise, for example, from training the same architecture with the same objective and learning rule from different random initializations. To ensure meaningful model controversy, it is attractive to optimize stimulus sets such that they are controversial among ensembles of models. Each ensemble represents one computational hypothesis and comprises a set of model instances that have different weights (e.g., trained using different initializations) and, optionally, variations of architecture, objective, and learning rule that are not of theoretical interest. More generally, the set of models should capture variation of theoretical interest (e.g., vision-language vs. vision-only models) as well as nuisance variation (e.g. initialization, small changes of architectural hyperparameters), and analyses should reveal how the factors of theoretical interest affect predictive performance.

In the longer term, it will be essential that we develop distributions of models that capture our uncertainty about the computational mechanisms in a more comprehensive fashion. This includes representing model parameters as random variables [118–120], as well as capturing not only group-mean tendencies but also intersubject variability in neural representations and behavior [121]. Ideally, neuroscientists should compare theories corresponding to distributions of mechanistic models with data sampled from distributions of individual animals. Controversial stimuli should therefore generalize across the idiosyncrasies of model instances and individuals. The model-comparative power of such stimulus sets will be measured by their ability to discriminate among ensembles of models, even when new model instances are used to generate the predictions and responses are sampled from new individual subjects.

5.3. Theoretical progress when the true model is elusive

Each candidate model represents a hypothesis about the mechanism underlying a particular cognitive function. Ideally, researchers would discover the model that implements the true mechanism generating the neural and behavioral responses to the stimuli. This true model would make accurate predictions of brain representations across stages of processing and behavioral responses for all stimuli, natural or artificial. However, for a given study that tries to make progress on a complex cognitive function such as visual recognition or spatial navigation, it is unrealistic to assume that the true model is among the models under consideration. Some of our models may be closer to the truth than others, but in general, the true model (i.e., the actual data-generating mechanism) will not be among the candidate models tested.

All models are wrong; especially in cognitive computational neuroscience. This reflects both the stage of development of our field and the intrinsic complexity of brains. Cognitive computational neuroscience finds itself in what is known as the open-set, or M-open scenario (Section 6.1.2. in [122]), which is distinct from the closed-set system-identification framework assumed in optimal experimental design (Boxes 1–3). The absence of the true model precludes strict Bayesian inference. If all models are misspecified, we also cannot get to the truth by refuting alternatives. Instead, the field must navigate the space of models seeking progress toward models that are less incorrect [8, 16, 123, 124]. This motivates a focus on statistical comparison of alternative imperfect models.

A consequent challenge is that different models may dominate for different sets of stimuli: Two experiments with different stimulus sets may yield apparently inconsistent model comparison results. Model A may perform significantly better than model B in experiment 1, and model B may perform significantly better than model A in experiment 2. Neither model, thus, dominates the other. If the true model were among our models, it would not be beaten by any other model, no matter what stimulus set is used for the comparison. In particular, the true model would be among the winning models for naturalistic as well as non-naturalistic stimuli. However, in the absence of the true model, there is only a set of imperfect models, each exhibiting a specific pattern of systematic prediction errors. A particular experiment will reveal model performance for a limited portion of stimulus space where one model may do better than others even if it fails badly elsewhere. For example, Golan and colleagues [37] found that discriminative neural networks predicted human recognition of handwritten digits slightly more accurately than alternative models when tested with standard MNIST digits. However, when synthetic controversial stimuli were used, a generative neural network [125] predicted human digit recognition substantially more accurately (Fig. S8 in [37]).

To eliminate ambiguities in model ranking, researchers might be tempted to commit to a single stimulus set, such as a large set of natural stimuli or a combination of all available natural and artificial stimulus sets. Using a single stimulus set, researchers can combine all the evidence, so as to reduce each model’s performance profile to a single number. This approach will not falsely eliminate the true model. However, it tacitly commits to a relative weighting of different portions of stimulus space and may misguide researchers in assessing the relative merits of imperfect models. Models that underperform overall may excel on particular stimuli and may even come to displace the current winners when a different weighting of the data sets is chosen or additional data sets are added to the benchmark.

Rather than prematurely reducing complex profiles of performance to scalar scores, researchers can embrace multiple evaluations of each model across stimulus sets and prediction targets (e.g. brain regions and behavioral tasks). Embracing multiple evaluations enables researchers to gauge the scope of stimuli and the scope of responses over which each model predicts well. This information is essential in choosing models for particular applications. Analyses of failure modes can also provide powerful guidance for improving models (e.g., [126]). Explainable AI methods can help researchers understand why failures occur and how a model might be improved. Approaching computational modeling as a multi-objective optimization problem, rather than a contest on a single benchmark, ensures that distinct failure modes are not masked by aggregation.

An important tool for handling the ambiguity that comes with multiple objectives is the concept of Pareto optimality. Rather than attempting to identify a single winning model, a Pareto set can be identified. A model is in the Pareto set if it is non-dominated—that is, if no single other model performs better across all objectives (i.e. all stimulus sets and/or prediction targets). Modeling should aim to narrow the Pareto set of non-dominated models, which requires models that outperform others across all objectives. However, new experiments may uncover new model failure modes, which can bring non-competitive (i.e., dominated) models back into the Pareto set. Identifying a Pareto set instead of averaging model performance across stimulus sets avoids the arbitrary weighting of different stimulus sets and prediction targets.

While the field should strive for unified mechanistic explanations that make correct predictions under a wide range of conditions (including natural stimuli of different categories, as well as adversarial and controversial stimuli), particular studies must compare imperfect models that are typically far from offering unified explanations. To navigate this gap between the ideal and the real, researchers can adjust the scope of their explanatory ambition, narrowing or expanding the stimulus domain and range of prediction targets across which candidate models are expected to perform well. By defining the space of admissible stimuli for experimental design, they can choose a scope that pushes toward more unified explanations while acknowledging the limitations of current models.