Figure 1. The link between modeling, mouse-tracking, and multivariate fMRI.

An illustration of the link between the three modalities is depicted using an example of stereotype influences on emotion perception based on [10]. Happy and Angry categories are each associated with a unique pattern across a neuronal population, with certain neurons highly active during a ‘Happy state and other neurons highly active during an ‘Angry’ state. These states are low-energy attractors, into which the system is compelled to settle (similar to how a ball must roll down a hill). These states would be associated with distinct multi-voxel patterns using fMRI. Mouse-tracking can provide a real-time behavioral index of how the perceptual process settles over time into one of the two categories. The computer screen during mouse-tracking may serve as a two-dimensional proxy for higher dimensional neural state space. The mouse-tracking paradigm is depicted at the center, overlaid onto a hypothetical energy landscape describing the energy at all states in the system. The two energy minima (attractors) are shown, corresponding to the Happy and Angry response locations and ‘Happy and ‘Angry’ neural states (and corresponding multi-voxel patterns). At the beginning of the perceptual process, the system is in an unstable, high-energy state. As the process evolves over hundreds of milliseconds, the neuronal population gradually settles into a low-energy attractor state, i.e., Happy or Angry category, just as the hand settles into one of the response locations. Due to automatic stereotype-based expectations linking Black people to hostility, for a happy Black face, during the perceptual process (e.g., mid-trajectory) the neuronal pattern would approximate the Angry pattern to a greater extent and the hand would be more attracted toward the Angry response (e.g., relative to a happy White face). Because the multi-voxel pattern in response to such a face would reflect an average over this time period (as fMRI’s temporal sensitivity is limited), it would exhibit a degree of greater pattern-similarity to the Angry category (e.g., relative to a happy White face), as shown in previous neuroimaging work [10]. This example shows the link between the multidimensional space of a neuronal population/model, the two-dimensional space of a computer screen with mouse-tracking, and the multidimensional space of multi-voxel response patterns.