MAME: Multidimensional adaptive metamer exploration with human perceptual feedback

Abstract

Alignment between human brain networks and artificial models has become an active research area in vision science and machine learning. A widely adopted approach is identifying “metamers,” stimuli that are physically different yet perceptually equivalent within a system. However, conventional methods lack a direct approach to searching for the human metameric space. Instead, researchers first develop biologically inspired models and then infer about human metamers indirectly by testing whether model metamers also appear as metamers to humans. Here, we propose the multidimensional adaptive metamer exploration (MAME) framework, enabling direct, high-dimensional exploration of human metameric spaces through online image generation guided by human perceptual feedback. MAME modulates reference images across multiple dimensions based on hierarchical neural network responses, adaptively updating generation parameters according to participants’ perceptual discriminability. Using MAME, we successfully measured multidimensional human metameric spaces within a single psychophysical experiment. Experimental results using a biologically plausible convolutional neural network (CNN) model showed that human discrimination sensitivity was lower for metameric images based on Gram-matrix representations derived from low-level CNN features than for those derived from high-level CNN features. The finding suggests a relatively worse alignment between the metameric spaces of humans and the CNN model for low-level processing compared with high-level processing. Counterintuitively, given recent discussions on alignment at higher representational levels, our results highlight the importance of early visual computations in shaping biologically plausible models. Our MAME framework can serve as a future scientific tool for directly investigating the functional organization of human vision.

Introduction

Recent advancements in artificial neural networks have not only improved their ability to solve visual tasks but have also positioned them as valuable analogies for understanding human visual processes. In particular, deep convolutional neural networks (CNNs) have been noted for their structural and functional similarities to the hierarchical processing observed in biological vision. These models have demonstrated considerable success in approximating the information flow from the primary visual cortex to higher visual areas in the ventral visual stream (Yamins & DiCarlo, 2016). However, numerous studies have reported significant divergences between the behavioral responses of humans and advanced neural network models (Geirhos et al., 2018a; Geirhos et al., 2018b; Szegedy, 2013). Consequently, extensive research efforts have been directed toward achieving a functional alignment between human vision and artificial models (Geirhos et al., 2021).

Metamer exploration as a tool for functional understanding of cognitive neural processes

One widely adopted approach for functionally aligning models with humans is the identification of “metamers.” Metamers are sets of input stimuli that are physically different yet processed as equivalent by a given system, such as humans or models. Here, consider an unknown system whose functional properties we aim to uncover. If the system produces the same output for different sets of physical inputs, that is, metamers, it implies the existence of common factors to which the system is sensitive. Identifying these common factors would provide insight into what the system functionally computes.

Through this metamer-based approach, various visual functions have been uncovered in the history of vision sciences. A classic example is the finding that human color vision is mediated by three types of sensors (reviewed by Gegenfurtner, 2003). The discovery that humans do not directly sense light as a function of wavelength was based on experimental results showing that physically different spectral compositions of light can appear identical to human observers, that is, they form metamers.

Furthermore, this approach has also been used to investigate hierarchical information processing in the brain. In the context of texture information processing, researchers have explored the features critical for texture perception by examining metamers and linking them to hierarchical processing in the brain (Bergen & Adelson, 1988; Freeman & Simoncelli, 2011; Freeman, Ziemba, Heeger, Simoncelli, & Movshon, 2013; Julesz, 1962; Okazawa, Tajima, & Komatsu, 2015; Ziemba et al., 2024). For example, Freeman et al. (Freeman & Simoncelli, 2011; Freeman et al., 2013) identified a model capable of generating metamers at mid-level stages of visual processing, such as V2, using texture synthesis algorithms. The exploration of metamers has been applied not only to texture perception, but also to other visual processes, such as peripheral vision (Rosenholtz, 2016; Wallis et al., 2019) and visual search (Rosenholtz, Huang, Raj, Balas, & Ilie, 2012), as well as to other sensory modalities, including audition (McDermott & Simoncelli, 2011) and haptics (Kuroki, Sawayama, & Nishida, 2021).

The approach of using metamers for functional understanding has further advanced with the use of deep neural networks (DNNs) as artificial systems. Gatys, Ecker, and Bethge (2015) proposed an image generation technique that uses intermediate features in CNNs, summarizing them with statistical representations such as Gram matrices. This method has been particularly beneficial for cognitive neuroscience researchers interested in higher-order hierarchical visual processing. For example, in the context of object recognition research, a metamer-based approach combining this image algorithm with functional magnetic resonance imaging (fMRI) measurements has demonstrated that object-selective regions in higher ventral visual areas exhibit texture-like representation, computed by Gram matrices (Jagadeesh & Gardner, 2022). Furthermore, there have been attempts to evaluate the alignment between humans and models by assessing the extent to which metamers generated by DNN models align with human metamers (Feather, Durango, Gonzalez, & McDermott, 2019; Feather, Leclerc, Madry, & McDermott, 2023).

Challenges in exploring high-dimensional human metameric spaces

A key limitation of approaches that use DNNs to explore human metamers is that these approaches are inherently indirect. This limitation becomes evident when examining how previous studies have explored human metamers (Figure 1).

Figure 1.

A conceptual illustration of the MAME framework. In previous studies, human metamers were explored indirectly by generating model metamers from noise images and presenting them to human observers. In contrast, our approach proposes a method for directly measuring human metamers by performing multidimensional exploration starting from a reference image. The orange and blue regions represent the human and model metameric spaces, respectively.

In previous studies, model metamers are first synthesized from noise to match a target image in a given DNN, and human observers are then asked whether these images appear to be equivalent to the original target (Figure 1, left). In this approach, the effectiveness of human metamer exploration critically depends on how well the model used for synthesis aligns with human visual perception. Indeed, adversarially trained CNNs, which are known to exhibit human-like robustness across various tasks (Salman, Ilyas, Engstrom, Kapoor, & Madry, 2020), have been shown to generate metamers that better account for human peripheral vision (Harrington & Deza, 2021).

However, this approach can fully reveal the structure of human metameric representations only if a suitably aligned model is available in advance. In this sense, human metamer exploration remains an indirect approach. Instead, could we adopt a more direct approach to measuring human metamers? If such an approach were possible, we could first measure the human metameric space and then design models that align with it, ultimately achieving human–model functional alignment. In this case, rather than generating images from noise using a model, as in conventional studies, we could directly measure human metamers by freely exploring the human metameric space from a reference stimulus, as illustrated in Figure 1 (right). Furthermore, directly exploring human metamers is itself crucial for understanding human information processing, as exemplified by the discovery of color vision sensors.

Indeed, several previous studies have explored perceptual representations by probing distortions defined with respect to a reference stimulus. An example is the maximum differentiation (MAD) competition framework, which constructs stimulus pairs that maximize disagreement between competing computational models of perceptual quantities (Ma et al., 2016; Ma et al, 2018; Wang & Simoncelli, 2008). Other approaches have further characterized the local sensitivity structure of models using analytically derived distortions, such as eigen- or principal-distortions, which approximate directions of maximal or minimal sensitivity in a representation space (Berardino, Laparra, Ballé, & Simoncelli, 2017; Feather, Lipshutz, Harvey, Williams, & Simoncelli, 2024). In addition, recent studies have shown that deep generative models can synthesize stimuli that induce specific perceptual effects, such as visual illusions (Gomez-Villa, Martín, Vazquez-Corral, Bertalmío, & Malo, 2022).

In many of these studies, the primary objective is to identify models that better approximate human perceptual judgments. To ensure experimental feasibility, these approaches typically reduce the stimulus space to a small set of dimensions. This strategy is effective for comparing candidate models and selecting those that align more closely with human perception. However, it is not primarily designed to reveal the intrinsic multidimensional structure of the human metameric space itself. Rather, the direct exploration of human metameric space requires a framework that enables multidimensional, online measurement of metameric boundaries guided by human perceptual feedback.

Proposed approach: The MAME framework

In this study, we propose a framework for exploring high-dimensional human metameric spaces, called the multidimensional adaptive metamer exploration (MAME) framework (Figure 2). The MAME framework enables multidimensional exploration of human perceptual boundaries along model-defined directions by using online image generation based on human feedback, thereby relaxing the constraints that a human-aligned model is needed beforehand. Specifically, in the MAME framework, the model is used solely to determine the search direction for human metamers. Although it is better for the model to be biologically plausible when determining the direction, the final human metameric boundary can be determined independent of the model’s metamers. This aspect relaxes the constraints on human metamer exploration. Furthermore, to enhance the efficiency of multidimensional exploration, we use online image generation guided by trial-wise human feedback, enabling the adaptive search for metameric boundaries.

Figure 2.

Exploring human metamers using the MAME framework. The exploration space is defined by the model (in this study, ICA components of CNN activations). Stimuli are synthesized along the defined directions, and their perceptual equivalence to a reference image is evaluated via a human behavioral task. Based on the feedback, the estimated human metamer is adaptively updated, and the task is repeated until convergence.

In the experiment, we tested our MAME framework using a biologically plausible CNN, an adversarially trained ResNet50, on a natural image dataset, ImageNet. The key contributions of our MAME framework can be summarized as follows.

• •It can explore multiple model-defined directions relevant to human metameric perception simultaneously in one psychophysical experiment.
• •Using a guiding CNN, we found that the human discrimination sensitivity was lower for metameric images based on Gram-matrix representations derived from low-level CNN features than for those derived from high-level CNN features. This finding suggests a weaker alignment between low-level processing in humans and the model, highlighting a potential guideline for designing models that better align with human perception.
• •The experimental environment is highly flexible because online image generation is performed on a Python server using a GPU, while the human experiment is conducted via a web browser with JavaScript.

The source code used in the experiments, as well as the analysis scripts required to reproduce the results, are available at Zenodo (https://doi.org/10.5281/zenodo.20789863) and GitHub (https://github.com/amano-k-lab/mame_hil). In addition, the experimental results data are available at OSF (https://doi.org/10.17605/OSF.IO/3TFHU).

Methods

The MAME framework

The MAME framework is designed to adaptively explore the boundaries of the human metameric space. This exploration is achieved by systematically perturbing a given reference image in multiple dimensions (Figure 1, right). The execution of the MAME framework relies on the following requirements.

• (1)Definition of the exploration direction of human metameric space. The human metameric space, initially defined in an input image space ${\mathbb{R}}^{W\times H\times C}$, must be compressed to a dimension practical for exploration (on the order of tens of dimensions) and defined using a coordinate system. Using CNNs or biologically plausible models coupled with compression methods, one can define the subspace direction to explore human metameric boundaries.
• (2)Online stimulus generation. After determining the direction of exploration, an image-generation algorithm is required that is capable of generating stimuli at any position within the defined metameric space. The stimulus generation must be completed within a time frame that ensures compatibility with the human feedback process in the MAME framework. For instance, when incorporated into an ABX test (Freeman & Simoncelli, 2011), this means generation within 2 to 3 seconds.
• (3)Feedback from tasks performed by humans or animals. Finally, our MAME framework requires feedback from tasks performed by humans or animals. As described in the Introduction, metamers refer to images that are physically different but equivalent within a given system. However, the definition of “equivalence” changes across studies. Some adopt categorical equivalence based on behavioral responses (Feather et al., 2019), while others define it in terms of perceptual appearance (Deza, Jonnalagadda, & Eckstein, 2017; Freeman & Simoncelli, 2011) using ABX tasks. Our method can be applied to feedback from any type of task if the computation speed of image generation is fast enough for real-time online experiments.

Depending on the research objective, different methods can be selected for each requirement. In the present study, our aim was to demonstrate the effectiveness of MAME, a novel comprehensive framework, under specific conditions. To this end, we evaluated its effectiveness using a canonical approach within the context of metamer exploration research. Specifically, the MAME framework was implemented by 1) defining the direction of human metameric space using a CNN and independent component analysis (ICA), 2) implementing a stimulus generation method that perturbs the reference stimulus via gradient descent, and 3) incorporating ABX tasks as feedback. The specific motivations for selecting each method are described in detail in the corresponding sections of the Methods.

Definition of the exploration direction of human metameric space

In this section, we describe how the metameric search space was defined in the present study. We constructed the search space based on the activations of a robust variant of ResNet50 (He, Zhang, Ren, & Sun, 2016) (ResNet50 robustness ImageNet L2-Norm ϵ = 3.0) (Engstrom et al., 2019). Among the many available DNN models, we selected this specific model because previous work has shown it to best approximate human metamers under indirect exploration paradigms (Feather et al., 2023). In our MAME framework, the model is used to define the direction of exploration. Therefore, to efficiently estimate human metamers under a limited number of conditions, it is preferable that the model be biologically plausible. In particular, we note that manipulating high-level features is more difficult in other models, as shown in the comparison results provided in the Supplementary Material (Supplementary Figure S1).

Next, we detail how the search space was concretely defined (Figure 3). First, natural images were passed through the robust ResNet50, while extracting Gram matrices from the activations of each layer as position-invariant features. Here, we used Gram matrices because the present study explored metamers based on the equivalence of perceptual appearance and focused on summary statistics like textures representations (Gatys, Ecker, & Bethge, 2015; Gatys, Ecker, & Bethge, 2016; Jagadeesh & Gardner, 2022).

Figure 3.

Definition of exploration directions in the present study. An input image is passed through a robust ResNet50 model, and activations are extracted from the conv1, layer3, and avgpool layers. For each layer, filter-by-filter Gram matrices are computed and vectorized. Using these vectorized features from the ImageNet dataset, ICA is applied, and the top three components with the highest EV are selected as exploration axes for each layer.

A Gram matrix represents the similarity of activations across multiple filters, summed over spatial positions. Given the activation of the l-th layer of a CNN for an input image as F_jk, where j denotes the filter index and k denotes the spatial position, the Gram matrix G_ij is defined as:

$$\begin{array}{c}\hfill G_{ij}=\sum _k{F}_{ik}{F}_{jk}.\end{array}$$

In this study, Gram matrices were extracted from multiple layers of the model using 1,000 images randomly selected from the ImageNet validation dataset (Deng et al., 2009).

After extracting the Gram matrix features, ICA was performed on them, and components with high explanatory power were selected, thereby considering images as points in the ICA coordinate space. The exploration direction was defined along the axes of ICA from the reference stimulus.

Although we used ICA to determine the exploration directions, several dimensionality reduction methods could be considered. In preliminary analyses, we also extracted components using principal component analysis (Supplementary Figure S2). However, the image patterns associated with different principal components tended to be visually similar. For this reason, we adopted ICA for dimensionality reduction in the present analysis. In addition, ICA normalizes components to have unit variance, which facilitates comparisons of representations across different layers.

In the following, we describe in detail how dimensionality reduction of the metameric space was performed using ICA. ICA assumes non-Gaussianity of signals and computes a linear transformation to separate observed signals into statistically independent components. The Gram matrices of the target layers were vectorized to form feature vectors x for each image. Using the feature vectors of 1,000 images, an input data matrix $\mathbf{X}\in {\mathbb{R}}^{N\times D}$ was constructed, where N = 1,000 denotes the number of stimuli and D the dimensionality of the flattened Gram features.

Before ICA, the data were centered and whitened. Whitening was performed using a principal component analysis-based transformation, which decorrelates the data and normalizes the covariance structure. Specifically, if the covariance of the centered data is decomposed as $\Sigma =\mathbf{E}\Lambda {\mathbf{E}}^{\top }$, the whitening transformation corresponds with

$${\mathbf{X}}_w=(\mathbf{X}-\mathbf{1}{\mu }^{\top })\mathbf{E}{\Lambda }^{-1/2},$$

which yields Cov(X_w) = I.

ICA was then applied to the whitened data using FastICA (scikit-learn), yielding an unmixing matrix W such that

$$\mathbf{S}={\mathbf{X}}_w{\mathbf{W}}^{\top }.$$

With the unit-variance option, each recovered independent component has zero mean and unit variance across stimuli, that is, Cov(S) = I. For a single stimulus x, its ICA representation is obtained as s = Wx_w. The corresponding mixing matrix A allows reconstruction in the original feature space:

$$\mathbf{X}\approx \mathbf{S}{\mathbf{A}}^{\mathsf{T}}.$$

The number of independent components was preset to 100, and the top 3 components with the highest contribution were selected. The contribution of each component was quantified using the explained variance (EV) computed in the original (non-whitened) feature space as

$${\mathrm{EV}}_i=1-\frac{\parallel \mathbf{X}-{\widehat{\mathbf{X}}}_i{\parallel }_{\text{F}}^2}{{\parallel \mathbf{X}\parallel }_{\text{F}}^2},$$

where

$${\widehat{\mathbf{X}}}_i={\mathbf{S}}_i{\mathbf{A}}^{\mathsf{T}},$$

and S_i retains only the i-th independent component (Supplementary Figure S4). Although ICA components have unit variance in the whitened space, their contributions differ in the original Gram feature space due to the mixing transformation.

Finally, we computed the metameric space based on features extracted from the conv1, layer3, and avgpool layers of the robust ResNet50. Each feature dimension is (64 × 64) for conv1, (1,024 × 1,024) for layer3, and (2,048 × 2,048) for avgpool. These layers were selected to examine early-, intermediate-, and higher-level stages of visual processing, while avoiding layers affected by skip connections in the ResNet architecture. Both positive and negative directions of each ICA component were used in the exploration.

Generation of target images

This section describes the method for generating images at designated positions using gradient descent. The image modification process was performed by specifying the target layer (conv1, layer3, and avgpool), the ICA component of interest (components 0, 1, and 2), the direction of change (positive or negative), the magnitude of change (a positive scalar corresponding with the target value), and the reference image (i.e., the initial image of the modification). The image generation procedure is as follows.

• (1)Compute the ICA components of the reference image for the target layer as y_o = Wx_{o, w}, where x_{o, w} is the whitened vectorized Gram matrix of the target layer.
• (2)
  Define the target component as p and the target value as t. The modified ICA component y_t is given by:

  $$\begin{array}{c}\hfill {\mathbf{y}}_{\text{t}}={\mathbf{y}}_{\text{o}}+t{\mathbf{e}}_{\text{p}},\end{array}$$

  where e_p is a unit vector whose p-th element is 1 and 0 elsewhere. This operation modifies only the target component while keeping all others unchanged. If the change direction is negative, the unit vector is multiplied by −1.
• (3)In each iteration, compute the ICA components of the current image as y_c = Wx_c,w.
• (4)Compute the mean squared error between y_t and y_c as the loss function.
• (5)Perform gradient descent using the Adam optimizer to minimize the loss and update the pixel values of the current image while keeping the CNN model parameters fixed.

Image optimization was performed for 50 iterations using the Adam optimizer implemented in PyTorch (Paszke et al., 2019) (torch.optim.Adam, β₁ = 0.9, β₂ = 0.999, ϵ = 10⁻⁸). The learning rate was determined as a function of the target value t. Specifically, for each layer, we first sampled a set of target values in log-space and optimized the learning rate independently for each value using Optuna (Akiba, Sano, Yanase, Ohta, & Koyama, 2019). The resulting discrete set of optimal learning rates was then interpolated using cubic interpolation to obtain a continuous function of t.

To assess optimization fidelity, we computed the normalized final loss for each layer, component, and direction across 100 randomly sampled images. The normalized final loss remained consistently low across all conditions (Supplementary Table S1), indicating that the optimization converged reliably and that the main results are unlikely to be explained by differences in optimization fidelity.

Examples of image generation for the three layers and three components used in the experiment are shown in Figure 4 and Supplementary Figure S3.

Figure 4.

An example of the images used in the experiment, generated with three layers and three components each. The numbers (in red and blue) below the layer names represent the target ICA values used during image generation. The magnitude of modulation in this visualization is scaled according to the experimental results.

The selection of reference images for initialization was based on two criteria. First, we aimed to use images located near the zero point of the ICA components. Second, to prevent participants in the behavioral experiment from memorizing individual images, we needed to sample many images. For this purpose, from 50,000 images in the ImageNet validation dataset, we sampled images for 3 target layers whose ICA component distances from the origin were within the lowest 20%. Among these, 614 images that met the selection criteria across all three layers were used as experimental stimuli (Table 1).

Table 1.

ICA component criteria and the number of selected images.

Criterion*	Layer	Threshold	No. of images
Lowest 20%	conv1	1.22	614
	layer3	1.73
	avgpool	0.79

^*An image satisfies a given criterion if its Euclidean distance in each layer is below the corresponding threshold.

Feedback from the behavioral task

As feedback in the present experiment, we used an ABX task to measure perceptual similarity. Although various forms of feedback could be considered—such as neural similarity or categorical judgments—our goal was to demonstrate the effectiveness of the novel MAME framework under specific conditions. In general, measuring perceptual appearance psychophysically is more time consuming than categorical judgment in one trial. In contrast, assessing equivalence using neural or blood-oxygen-level-dependent responses in neurophysiological or fMRI studies requires a time course comparable to that of perceptual-appearance experiments, although the experimental cost is substantially higher. The ABX task was chosen because it offers a comparable trial duration to neural similarity experiments, while being less costly to implement. This makes it a suitable choice for assessing the feasibility of the MAME framework in an online experimental environment.

The experiment consisted of an ABX test comparing generated images with reference images. Based on the participants’ response data, each image was updated until the exploration of human metamer boundaries converged.

Eight participants took part in the experiment. The study was conducted with approval by the Ethics and Safety Committee of the University of Tokyo. The MAME framework was implemented using a Python web framework (Django) and a browser-based experimental environment with JavaScript (Adolphe et al., 2022). Image generation was performed on a server machine with an NVIDIA RTX A2000 GPU, and the generated images were presented in the Google Chrome browser via a local server.

The experiment was conducted in a dark room, with participants’ heads fixed at a distance of 70 cm from the monitor. In all tasks, participants were instructed to fixate on a fixation point (Thaler, Schütz, Goodale, & Gegenfurtner, 2013) at the center of the monitor (Figure 5).

Figure 5.

Overview of the ABX test. Stimuli A and B were the reference and generated images, presented in random order, and stimulus X was identical to either A or B. Each stimulus was presented for 200 ms and followed by a 500-ms blank period. Participants reported whether X matched the first or second stimulus.

Furthermore, to ensure that participants were indeed fixating on the designated point during the experiment, gaze tracking was performed using the Tobii Pro Spark (Tobii, Stockholm, Sweden). Trials in which fixation was not maintained were not used for parameter updates.

The procedure of the ABX task is as follows (Figure 5). In each trial, participants are presented with three stimuli in sequence: stimulus A, stimulus B, and stimulus X. The first two stimuli (A and B) are the reference and generated images in a random order. The third stimulus (X) is a test stimulus that is the same as A or B. Participants are required to identify whether X is the same as A or B. The presentation position was 4°, 8°, or 12° from the center depending on the eccentricity condition. The size of the presentation image was 4° × 4° in the visual angle.

Based on the results of each ABX test, the target value was updated using the staircase method (2-down, 1-up rule). Specifically, if a participant correctly answered two consecutive trials in the ABX task, the boundary of the human metameric space was narrowed by one step. Conversely, if the participant made an error in a single trial, the boundary was shifted in the direction of expansion. Here, one step was defined as a fixed value for each layer, determined based on preliminary experiments: 10 for conv1, 0.3 for layer3, and 0.02 for avgpool.

For the final threshold estimate, we fitted a sigmoidal psychometric function to the trial-by-trial responses collected during the staircase procedure using psignifit (2-alternative forced choice task). The responses were treated as Bernoulli trials (correct/incorrect), and the threshold was defined as the stimulus level corresponding with 70.7% correct, which matches the convergence point of the two-down, one-up staircase.

The stimulus values generated during image synthesis did not always exactly match the predefined staircase step values. To quantify this discrepancy, we performed a post hoc analysis of all stimulus images used in the experiment. The root mean square (RMS) error between the target and actual stimulus values was 2.088 for conv1, 0.025 for layer3, and 0.005 for avgpool, all of which were substantially smaller than a single staircase step.

When fitting the psychometric functions, we therefore used the actual stimulus values rather than the predefined target staircase values. Specifically, for each staircase step, we computed the mean actual stimulus value across trials and fitted the psychometric function based on participants’ responses to these stimulus levels. Examples of psychometric function fits are provided in Supplementary Figure S6.

In addition, because the stimulus values generated during image synthesis did not always exactly match the predefined staircase step values, we also estimated thresholds directly from the staircase trajectory by averaging the last five reversal points for each condition. This estimate reflects the actual stimulus levels presented during the experiment and was used as a complementary measure (see Supplementary Table S2).

In summary, the experimental conditions consisted of three layers, three ICA components each explored in both positive and negative directions, and three levels of stimulus eccentricity (4°, 8°, and 12°). This resulted in a total of 54 conditions (3 × 3 × 2 × 3).

The experiment was conducted over 2 days for each participant. Each participant engaged in 15 experimental blocks, where each block consisted of 90 trials and took approximately 9 to 12 minutes to complete. In each block, the stimulus presentation position was fixed, and 18 conditions were presented five times each in a randomized order. Across the 15 blocks, 3 different stimulus presentation positions were used, with each position appearing in five blocks. The order of these positions was randomly determined at the start of the experiment. Following this protocol, each condition was tested 25 times using the ABX test.

Results

The MAME framework was applied to each subject, and their metameric boundaries were obtained as ICA component values.

Furthermore, analyses based on image metrics were conducted, allowing the metameric boundaries to be computed using these indices.

Metameric boundaries in ICA components

The metameric boundaries represented by ICA components, obtained directly from the experiment, are shown in Table 2 and Supplementary Figure S5. Because the effects of the ICA component and direction conditions were small (Supplementary Figure S5), we first computed within-subject averages across these conditions. These mean values were averaged across different participants and are shown in Table 2.

Table 2.

Mean threshold values computed by averaging across ICA components, search directions, and subjects for each layer and eccentricity. The standard deviation represents intersubject variability, computed from the threshold values that were first averaged within each participant over components and directions (n = 8).

Layer	Eccentricity (°)	Mean	Standard deviation
conv1	4	50.7380	9.9466
	8	67.4927	12.8030
	12	70.8025	9.6864
layer3	4	0.5124	0.1042
	8	0.6474	0.0631
	12	0.7165	0.0885
avgpool	4	0.1700	0.0219
	8	0.1701	0.0272
	12	0.1913	0.0269

To quantitatively evaluate the effects of layer and eccentricity on the metameric threshold, we performed a two-way repeated-measures analysis of variance (ANOVA) with Layer (conv1, layer3, avgpool) and Eccentricity (4°, 8°, 12°) as within-subject factors. The analysis showed the significant main effects of Layer and Eccentricity, F(2, 14) = 317.46, p < 0.001, ${\eta }_g^2=0.96$ and F(2, 14) = 17.46, p < 0.001, ${\eta }_g^2=0.18$. For the main effect of Layer, post hoc pairwise comparisons with Bonferroni correction showed that the thresholds were significantly higher in the order of conv1, layer3, and avgpool (p < 0.001). The interaction was also significant, F(4, 28) = 16.71, p < 0.001, ${\eta }_g^2=0.30$. Based on the significant interaction, we further examined the simple effects of eccentricity within each layer.

For the conv1 layer, thresholds increased significantly with eccentricity, F(2, 14) = 16.96, p < 0.001, ${\eta }_g^2=0.39$. Post hoc pairwise comparisons with Bonferroni correction showed that thresholds at 8° and 12° were significantly higher than those at 4°, whereas the difference between 8° and 12° was not significant. For the layer3 layer, the effect of eccentricity was also significant, F(2, 14) = 14.46, p < 0.001, ${\eta }_g^2=0.49$. Pairwise comparisons revealed that thresholds at 8° and 12° were significantly higher than those at 4°, while the difference between 8° and 12° was not significant. For the avgpool layer, the effect of eccentricity was weaker but still significant, F(2, 14) = 5.37, p = 0.019, ${\eta }_g^2=0.13$. However, pairwise comparisons showed that none of the pairs were statistically significant (p > 0.05).

The observed differences in thresholds across layers suggest that the sensitivity of the metameric boundary defined by conv1 is lower than that of layer3 and avgpool. However, this result should be interpreted carefully when considering ICA dimensionality selection. When ICA is applied to the features of each layer, whitening is performed with respect to the natural image distribution, implying that the values of the ICA components are scaled within that distribution for each layer. Nevertheless, as shown in Supplementary Figure S4, the proportion of variance in the original Gram matrix explained by the independent components differs substantially across layers. This suggests that comparing sensitivity across layers requires evaluating the threshold images using multiple image metrics in addition to ICA-based thresholds. We address this in the following section.

Evaluation by multiple image metrics

We examined how much the threshold images obtained under each experimental condition differed from the original images using several image quality metrics. Here, the threshold images obtained for each condition can be interpreted as being equally different from the original image for human observers across different stimulus dimensions. If the directions explored by a model are well aligned with human metamers, smaller perturbations from the original image are expected to be sufficient for detection by human observers. This is because even a slight perturbation breaks the metameric equivalence for the model, meaning that the perturbed image is no longer a metamer from the model’s perspective.

First, we conducted an evaluation using RMS contrast. RMS contrast is an index that quantifies the overall contrast strength of a grayscale image. In the analysis, for each layer and eccentricity, grayscale difference images were generated by subtracting the reference image from the perturbed image using the target ICA component values at the experimentally determined thresholds (Figure 6a). The RMS contrast of these difference images was then computed. Each threshold image was computed separately for each participant.

Figure 6.

(a) Examples of threshold images for each layer (displayed in grayscale). The target value t is determined based on the threshold for each layer. The rightmost column shows the difference images. (b) The RMS contrast of the difference image was computed. A total of 30 images were generated based on the threshold values obtained from each participant in the behavioral experiment and were averaged across images. The RMS contrast values were then averaged across participants, and the mean and standard error are plotted. Lower values indicate smaller contrast differences. Similarly, (c) SSIM, (d) LPIPS, and (e) DISTS between the original and perturbed images were computed, and the mean and standard error are plotted. Higher values for SSIM and lower values for LPIPS and DISTS indicate higher similarity between the original and perturbed images.

The results of the RMS analysis are shown in Figure 6b. As in the previous section, we performed a two-way repeated-measures ANOVA with Layer (conv1, layer3, and avgpool) and Eccentricity (4°, 8°, and 12°) as within-subject factors. The analysis revealed significant main effects of Layer and Eccentricity, F(2, 14) = 428.12, p < 0.001, ${\eta }_g^2=0.87$; F(2, 14) = 40.70, p < 0.001, ${\eta }_g^2=0.36$. The interaction was also significant, F(4, 28) = 4.84, p < 0.01, ${\eta }_g^2=0.15$.

Simple effects analyses with Bonferroni-corrected pairwise comparisons showed that, for both conv1 and layer3, the values at 4° were significantly lower than those at 8° and 12° (p < 0.05, Bonferroni corrected), whereas no significant differences were found between 8° and 12° (p > 0.05, Bonferroni corrected). In contrast, for the avgpool layer, no significant differences were found across eccentricities (p > 0.05, Bonferroni corrected).

For comparisons across layers, the same analyses revealed that only the difference between layer3 and avgpool at 4° was not statistically significant (p > 0.05, Bonferroni corrected), whereas all other between-layer comparisons showed significant differences (p < 0.05, Bonferroni corrected). Consistent with the ICA threshold analysis, these results suggest that human sensitivity is lower when the metameric boundary is explored using lower-level image features, such as those in conv1.

Next, in addition to RMS contrast, we evaluated the threshold images using several other image quality metrics. Specifically, we used the structural similarity index (SSIM) (Wang, Bovik, Sheikh, & Simoncelli, 2004), learned perceptual image patch similarity (LPIPS) (Zhang, Isola, Efros, Shechtman, & Wang, 2018), and deep image and texture similarity (DISTS) (Ding, Ma, Wang, & Simoncelli, 2020). The SSIM assesses local structural similarity within a sliding window. LPIPS and DISTS are recent perceptual metrics that capture high-level feature similarities based on DNN representations. For LPIPS and DISTS, evaluations were performed using the original color images.

The results of these analyses are shown in Figure 6c–e. Across all metrics, the overall trends were consistent with those observed for RMS contrast and the ICA-based thresholds. Note that, for SSIM, higher values indicate greater similarity to the original image. The results of the two-way repeated-measures ANOVA for each metric are summarized in Supplementary Tables S3, S4, and S5. The conditions showing significant effects in the ANOVA were consistent with those observed for RMS contrast.

Taken together with the ICA threshold results, these findings suggest that the Gram-matrix representations derived from conv1 are less well aligned with human sensitivity than those derived from layer3 and avgpool. Furthermore, threshold images generated from higher-level features, such as avgpool, appear to be less affected by eccentricity.

Discussion

The present study proposed the MAME framework, an experimental framework to directly explore human metamers using online feedback by humans with weak supervision of artificial models. The effectiveness of this framework was demonstrated by successfully estimating multidimensional image metamer boundaries. Experimental results indicated that, among image generation layers, the threshold in conv1 was higher based on the magnitude of ICA thresholds, RMS contrast, SSIM, LPIPS, and DISTS, suggesting that changes in this layer were less noticeable compared with others. Furthermore, the effect of eccentricity on sensitivity was less pronounced for higher-level features.

To contextualize our findings, it is important to revisit what model metamers represent in the current study. In our MAME framework, when an image is shifted even slightly in the model’s exploration direction from the reference image, the shifted image is no longer a model metamer for the model. This is because exploration is carried out in the model’s activation space, where model metamers are defined as stimuli that are distinct in image space but equivalent in the target activation space. As a result, a single step in this space exceeds the model’s metameric boundary in image space.

This relationship implies that the lower mean thresholds observed in human responses at a given hierarchical level indicate a closer alignment between the metamer boundaries of the human and the model. Given this relationship, the experimental results showed that threshold images generated from conv1 exhibited higher ICA thresholds than those from layer3 and avgpool. Moreover, evaluations based on image quality metrics indicated that these differences cannot be readily explained by differences in the scale of ICA components across layers. These findings suggest that Gram-matrix representations derived from lower-level CNN layers are less well aligned with human perception.

Alignments of higher and lower visual processing in humans and models

The finding related to the alignment of higher-level image features is consistent with recent studies using robust CNNs (Feather et al., 2019; Feather et al., 2023; Gaziv, Lee, & DiCarlo, 2023). For instance, it has been shown that adversarial attacks on high-level activations can induce category modulations that are also perceptible to humans (Gaziv, Lee, & DiCarlo, 2023). Furthermore, the alignment of high-level features has been a topic of intense discussion in both visual and large language model literature. For example, the Platonic representation hypothesis (Huh, Cheung, Wang, & Isola, 2024) proposes that “neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.” From this perspective, the strong alignment of high-level representations between humans and modern neural models, trained with various data, might be a straightforward consequence.

However, our experimental results also suggest a functional misalignment between models and humans in low-level processing. Given that CNNs are originally inspired by the information processing of simple and complex cells in the primary visual cortex (Fukushima, 1980), this result may seem counterintuitive. One reason is that the first-layer filters in ResNet50 have a high-resolution size of 7 × 7 pixels and lack low-frequency components.

Indeed, previous work by Berardino et al. demonstrated that, when perturbations aligned with directions of maximal or minimal model sensitivity (eigen-distortions) were applied to reference stimuli, models that mimicked early visual processing stages, such as the lateral geniculate nucleus, showed closer alignments to human perceptual sensitivity than standard CNN architectures (Berardino et al., 2017). Furthermore, attempts to incorporate such early-stage processing into CNNs have been reported to improve model robustness and generalization (Dapello et al., 2020). Taken together, these findings and our present results suggest that reconsidering the early-stage architecture of models may be crucial for improving functional alignment between humans and artificial systems.

Model-guided distortions, limitations, and future applications

Several previous studies have investigated human perceptual sensitivity by modulating images with respect to a reference stimulus. For example, eigen-distortions and principal-distortions (Berardino et al., 2017; Feather et al., 2024) use the Fisher information matrix to characterize the local geometry of a model’s representation and to quantify how sensitivity to specific perturbation directions compares between models and human observers. These approaches are typically designed to identify models whose local sensitivity structure better approximates human perception. To ensure experimental feasibility, they often evaluate sensitivity along a relatively small number of analytically defined directions.

A key difference between these previous approaches and our framework is how the exploration directions are defined. Rather than deriving perturbation directions from local sensitivity structures around a particular image, we derive exploration directions from independent components estimated across a large set of natural images. This approach enables the extraction of multiple exploration directions from a single reference image and facilitates multidimensional exploration within a practical experimental setting. Given that human metameric representations are likely to be multidimensional, we believe that this strategy provides a reasonable starting point for exploring candidate metameric boundaries.

Although Gram-matrix–based ICA is not the only possible strategy, it offers several practical advantages. By performing ICA over natural image distributions, the resulting directions reflect dominant statistical variations across images and can partially mitigate differences in scale across hierarchical layers. This property is particularly beneficial when comparing modulation effects across layers with inherently different activation magnitudes.

At the same time, the effectiveness of this approach may differ across levels of visual representation. If perceptual sensitivity at early stages of visual processing is more image dependent than at later stages, exploration directions estimated across many images may be less effective for characterizing sensitivity in early representations, whereas directions derived from local sensitivity structures around individual images may provide a more appropriate description. Future work should investigate how multidimensional exploration approaches such as the MAME framework can be integrated with image-specific local sensitivity measures, including those derived from Fisher information matrices.

More generally, the explored stimulus conditions in the present study do not comprehensively cover all possible directions of human metamer space. Furthermore, because the ICA components were estimated from only 1,000 images in a high-dimensional feature space, larger and more diverse image sets may reveal additional informative directions. Exploring alternative feature representations, image datasets, and computational models will be important for obtaining a more complete characterization of human metameric representations.

Despite these limitations, our approach provides a flexible framework for studying perceptual representations. Because the framework can be executed in a web browser, it is largely free from hardware constraints and can be easily combined with a wide range of experimental environments, including neuroimaging settings such as fMRI, as well as large-scale crowdsourcing experiments. Therefore, we believe that the MAME framework can serve as a useful tool for future investigations of perceptual representations across diverse domains and methodologies.

Conclusions

In this study, we proposed the MAME framework, which combines hierarchical neural network–based image generation with human-in-the-loop exploration, to directly map the multidimensional human metameric space. Our evaluation experiment demonstrated that the MAME framework can measure this space within a single psychophysical experiment. The results further revealed that a recent human-aligned model exhibits a substantial functional mismatch between human perception and Gram-matrix representations derived from lower-level CNN layers, despite closer alignment for representations derived from higher layers. Because the user interface of MAME operates entirely within a web browser, it can be applied in fMRI environments and large-scale crowdsourcing experiments, making it a diverse tool for exploring metamers in human vision and for advancing interpretable artificial intelligence.