Interhemispheric multisensory perception and Bayesian causal inference

Summary

In daily life, our brain needs to eliminate irrelevant signals and integrate relevant signals to facilitate natural interactions with the surrounding. Previous study focused on paradigms without effect of dominant laterality and found that human observers process multisensory signals consistent with Bayesian causal inference (BCI). However, most human activities are of bilateral interaction involved in processing of interhemispheric sensory signals. It remains unclear whether the BCI framework also fits to such activities. Here, we presented a bilateral hand-matching task to understand the causal structure of interhemispheric sensory signals. In this task, participants were asked to match ipsilateral visual or proprioceptive cues with the contralateral hand. Our results suggest that interhemispheric causal inference is most derived from the BCI framework. The interhemispheric perceptual bias may vary strategy models to estimate the contralateral multisensory signals. The findings help to understand how the brain processes the uncertainty information coming from interhemispheric sensory signals.

Graphical abstract

Highlights

• •Spatial disparity affects the processing of interhemispheric sensory signals
• •The Bayesian framework dominates interhemispheric causal inference
• •Perceptual bias varies strategy models to estimate contralateral multisensory signals

Biological sciences; Sensory neuroscience; Computing methodology

Introduction

In daily life, we are exposed to a barrage of sensory signals, which consistently provide our brains with uncertain information about the external environment and ourselves. Importantly, the brain should separate out irrelevant information and integrate sensory signals only when they pertain to the same event.^1,2 It is, therefore, a challenge for our brain to infer sensory signals that are caused by common or independent sources.

The causal inference and probabilistic interpretation of multisensory combination (probabilistic population code) used in psychophysics have enhanced an overall understanding about the inference in the brain.^3,4,5 Early studies proposed maximum likelihood estimation (MLE) to implicitly assume that the sensory signals stem from a common cause, resulting in multisensory forced fusion (FF) (integration).^6,7 Under the FF model (or MLE), observers integrate sensory signals approximately weighted by their relative precisions, which leads to intersensory biases and perceptual illusions.^8,9 Nevertheless, the sensory signals are not always integrated between each other.^2,6,10,11 Once large intersensory conflicts such as temporal asynchrony,^12,13 spatial disparity,^14,15,16 or numeric disparity¹⁷ were presented, multisensory integration breaks down and cross-modal biases are attenuated. The brain may then perceive these sensory signals as independent sources and segregate multisensory cues responding to the conflict.^11,18,19 Recently, Bayesian causal inference (BCI) models were used to account for an observer’s uncertainty that whether sensory signals come from common or independent sources.^{6,20,21,22,23} In the BCI framework, the pre-existing expectation (i.e., prior) and sensory representation (i.e., likelihood distribution) are coded independently and combined following Bayes rule.^16,24 A final perceptual estimate is then obtained by combining the estimates under the assumptions of common or independent sources, according to a variety of strategy models.⁶ BCI models are considered to be a rational strategy to arbitrate between sensory integration and segregation,^2,14,17,19 which was supported by psychophysical evidence.^15,25,26,27 In the audiovisual localization task, for example, observers flexibly transited between the integration and segregation of sensory audiovisual signals, following a function of the spatial disparity based on BCI model.^9,14,16 The integration of sensory signals occurred only when a common cause was inferred. The rubber hand paradigm, involving in the integration of visual and proprioceptive signals, further supported the causal framework. In this paradigm, biases of hand location (proprioceptive drift) were strongly related to the spatial disparities between the vision and proprioception, which was well estimated by the BCI model.^15,26

Evidence for BCI models was mainly from paradigms involved in ipsilateral tasks. However, most human activities require the involvement of coordinated movements with both sides of the body. In order to fill water into a cup from a bottle, for example, one hand must hold the cup while the other hand picks up the bottle and tilts it. Meanwhile, the eyes must be kept on the cup and bottle to guide the water pouring. In such coordinated movements, sensory signals from both hemispheres of the brain are integrated. Few paradigms, to our best knowledge, have given concerns on bilateral activities, which are related to interhemispheric transmission with the complex processing of sensory signals (e.g., sensory mirroring,^28,29 lateral-side dominance^30,31,32). The uncertainty of perception associated with interhemispheric transmission may strengthen the estimated sensory bias, which is suggested to be considered in full explanations of the behavioral preformation.¹⁶ For example, lateral-side dominance leads to asymmetries in target matching, resulting in a significantly larger perceptual bias (or matching error) when matching the contralateral position compared with matching the ipsilateral position.³³ However, BCI models assume that the sensory perception is unbiased.⁶ It is not yet clear whether the causal inference related to interhemispheric transmission of sensory signals could be well predicted by BCI models.

Here, we presented a bilateral hand-matching task to understand the causal structure of interhemispheric sensory signals. The task included unisensory and multisensory matching tasks involved in the processing of bilateral sensory signals, in which proprioceptive and visual signals were provided by an exoskeleton robot and a virtual reality (VR) headset, respectively. We used BCI models to fit behavior data under both unbiased and biased-estimation assumptions. We first obtained the perceptual bias of unisensory matching task by matching only visual or proprioceptive angle (simulation). In the BCI model with biased estimation, the mean value of sensory representation is the matching center obtained by the unisensory matching task. In contrast, the mean value of the unbiased estimate is centered on the stimulus location. Our results indicated that BCI models formalized the processing of interhemispheric multisensory signals and demonstrated that the visual and proprioceptive perception formed into a unified representation on the contralateral activity. Moreover, the interhemispheric perceptual bias may make a difference in strategy models for estimating the contralateral signals.

Results

Behavioral performance under unisensory and multisensory stimulation

We designed a bilateral hand-matching task to characterize the causal structure of interhemispheric sensory signals (Figure 1B). In this task, participants were asked to match ipsilateral visual (VM) or proprioceptive cues (PM) using the contralateral hand in the context of unisensory (either visual or proprioceptive stimulation) and multisensory (both visual and proprioceptive stimulations) stimulation, respectively (Figure 2A). To examine interhemispheric perceptual bias (see also Figure S1 for the definition of perceptual bias), we first calculated the relative matching accuracy (RMA) in the unisensory matching task. Except for one minimal stimulation condition (Wilcoxon matched pairs signed-rank test, vision matching vs. proprioception matching in 10°, Z = −0.31, p = 0.75), RMAs in the vision matching were significantly lower than those in the proprioception matching (Wilcoxon matched pairs signed-rank test, vision matching vs. proprioception matching in 20°, 30°, 40°, 50°, median: [0.584, 0.508, 0.426, 0.443] vs. [0.963, 0.883, 0.929, 0.966], Z < -3.60, p < 0.001) (Figure 2B), which means that PM has better matching accuracy. In the multisensory matching task, the visual weight index W_V and RMA were analyzed as a function of spatial disparity between visual and proprioceptive stimulus. This visual weight index (W_V) ranges from pure visual (1) to pure proprioception (0) influence. The W_V of vision matching was significantly higher than that of proprioception matching both in low- and high-spatial-disparity groups (Wilcoxon matched pairs signed-rank test, vision matching vs. proprioception matching, high group, median: 0.629 vs. 0.306, Z = 3.90, p < 0.001; low group, median: 0.684 vs. 0.307, Z = 3.19, p = 0.001), which means that participants rely more on visual information to complete matching in VM. Although not significant, the W_V has an increasing trend in proprioception matching from high- (mean ± SD = 0.255 ± 0.141) to low (mean ± SD = 0.330 ± 0.181)-spatial-disparity group (Figure 2D). As shown in Figure 2E, the measured RMAs decreased with narrowing disparity, but the slope has obvious distortion under the large spatial disparity (−40, −30, −20), indicating the segregation of multisensory cues. Therefore, the multisensory integration broke down for large spatial disparity, when visual and proprioceptive stimuli were more likely to be caused by independent sources.

Figure 1.

Experimental setup

(A) A participant sat upright in front of a height-adjustable table; and his arms were mounted on a bilateral exoskeleton robot. The participant wore a head-mounted display in which a virtual left hand was presented from the first-person perspective. His real hands, meanwhile, were hidden by the display device. The robot drove the participant’s left hand with a flexion movement. The participant was then instructed to move his right hand to match with his left hand.

(B) The virtual and real left (ipsilateral) hands passively moved to one of five pre-set angles (10°, 20°, 30°, 40°, 50°), respectively. The right (contralateral) hand actively moved to match the angle of the virtual or real (ipsilateral) hand.

Figure 2.

Matching task and behavioral performance

(A) The tasks involved the unisensory (either visual or proprioceptive stimulation) and multisensory (both visual and proprioceptive stimulation) matching tasks. In the unisensory matching task, participants located the ipsilateral (virtual or real) hand using the contralateral hand based on the virtual or real hand angle. In the multisensory matching task, participants were asked to match either a virtual hand (vision matching, VM) or a real hand (proprioception matching, PM) in the presence of both visual and proprioceptive signals.

(B) The relative matching accuracy (RMA) in the unisensory matching task.

(C) A total of 25 stimulus conditions (five visual stimulation × five proprioceptive stimulation) were included in the multisensory matching task. The experimental design factorially manipulated the angle of virtual hand (visual stimulation), the angle of real hand (proprioceptive stimulation), and the task relevance of the sensory modality (VM vs. PM). The spatial disparity was calculated by sensory stimulation (VM: the angle of visual stimulation minus the angle of proprioceptive stimulation; PM: the angle of proprioceptive stimulation minus the angle of visual stimulation). To characterize the behavior on spatial disparity, we reorganized stimulus conditions into a two (task relevance: VM vs. PM) × nine (spatial disparity: −40, −30, −20, −10, 0, 10, 20, 30, 40) factorial design. The high (spatial disparity ≥20 or spatial disparity ≤ −20, light gray block) and low spatial disparity (spatial disparity ≥ −10 and spatial disparity ≤10, dark gray block) were defined.

(D) The visual weight index W_V is shown as a function of spatial disparity (high vs. low spatial disparity) and task relevance (VM vs. PM). W_V = 1 for purely visual influence, and W_V = 0 for purely proprioceptive influence.

(E) The variation of RMA with spatial disparity. The error bars represent the standard error.

Bayesian modeling under the unbiased-estimation and biased-estimation assumption

To analyze the causal structure of interhemispheric integration and segregation about multisensory, we fitted the BCI framework to each participant’s behavioral responses under the unbiased-estimation assumption and biased-estimation assumption (Figure 3A). Regardless of unbiased-estimation assumption or biased-estimation assumption, the log model evidence shows the main effect (Friedman non-parametric ANOVA, unbiased-estimation assumption, χ²_{(6, 20)} = 87.94, p < 0.001; biased-estimation assumption, χ²_(6, 20) = 63.34, p < 0.001) between strategy models. The log model evidence was significantly larger in BCI framework (model 1, 2, 3) than in the other models (model 4, 5, 6, 7) (Post hoc test, p < 0.05) (Figure 3B). When different models are compared, the BCI-MA (model averaging) model (model 1) has more participants' mode attributes in unbiased-estimation assumption (mean ± SD = 0.759 ± 0.427), while the more mode attributes were observed in BCI-MS (model selection) model (model 2) under biased-estimation assumption (mean ± SD = 0.607 ± 0.420) (Figure 3C). According to the estimated model frequencies, the maximum frequencies were found in BCI-MA (model 1) under unbiased-estimation assumption (mean ± SE = 0.730 ± 0.008) and BCI-MS (model 2) under biased-estimation assumption (mean ± SE = 0.556 ± 0.011) (Figure 3D). Bayesian model comparison demonstrated that the BCI model outperformed the other models (98% variance explained, exceedance probability of 0.99 in BCI-MA under unbiased-estimation assumption; 96% variance explained, exceedance probability of 0.94 in BCI-MS under biased-estimation assumption) (Figure 3E). The comparison of posterior common source probabilities (P_COM) and subjective scores (Likert scale) showed an approximatively consistent trend both in unbiased-estimation assumption and biased-estimation assumption (Figure 3F). In other words, the causal structure was well predicted by BCI model in both unbiased-estimation assumption and biased-estimation assumption, and human observers integrate multisensory signals predominantly in a low spatial disparity which may come from a common source.

Figure 3.

Models of BCI framework under the assumptions of unbiased estimation and biased estimation

(A) The assumptions of unbiased estimation and biased estimation. The μ_V and μ_p is the expectation of perceptive position X_V (X_P) (unbiased estimation: μ_V = S_V, μ_P = S_P; biased estimation: μ_V ≠ S_V, μ_P ≠ SP). The blue and brown lines represent the estimated (matched) position ${\hat{\mathrm{S}}}_{\mathrm{V}}\left({\hat{\mathrm{S}}}_{\mathrm{P}}\right)$ in multisensory task under unbiased-estimation and biased-estimation assumption, respectively.

(B) Log model evidence. Seven different strategy models were examined in our study, including three BCI models (model 1, model averaging; model 2, model selection; model 3, probability matching), the forced-fusion model (model 4), unimodel vision model (model 5), unimodel proprioception model (model 6), and the full-segregation model (model 7). Log model evidence = −2× (LL − 0.5×m×ln(n)), LL = log likelihood, m = number of parameters, n = number of data points. Error bars represent the standard errors.

(C) The mode attributes were obtained in terms of the posterior probability of each model to characterize each participant’s performance optimally.

(D) Estimated model frequencies were obtained by the first- and second-order moments of a Dirichlet density.

(E) The exceedance probability refers to the probability that a given model is more likely than any other model.

(F) The posterior common source probabilities (P_COM) and subjective scores (Likert scale). The posterior common source probabilities (P_COM) were calculated by the optimal model under unbiased-estimation and biased-estimation assumption. The subjective scores represent a consistent feeling between the visual perception and the proprioceptive perception. (−3: complete inconsistency, 0: neither consistency nor inconsistency, +3: complete consistency). The error bars represent the standard deviation.

Comparison of the unbiased-estimation and biased-estimation assumption

When comparing unbiased-estimation and biased-estimation assumption, the interhemispheric perceptual bias seems to affect the arbitrate strategies in BCI framework, resulting in different strategy models to behavioral response (BCI-MA under unbiased-estimation assumption vs. BCI-MS under biased-estimation assumption). As shown in Tables 1 and 2, two complete different prior distributions (BCI-MA vs. BCI-MS, μ_pr: Z = −2.930, p = 0.002; σ_pr: Z = −2.272, p = 0.023) are found between unbiased-estimation assumption (BCI-MA, mean ± SD, μ_pr = 7.43 ± 9.47, σ_pr = 7.95 ± 1.63) and biased-estimation assumption (BCI-MS, mean ± SD, μ_pr = 15.44 ± 10.04, σ_pr = 8.39 ± 1.31). To characterize the spatial features of behavioral and BCI framework, we construct representational dissimilarity matrices (RDMs) based on behavioral data (Figure 4A) and model estimation (Figures 4B and 4C). We found that the similarity of RDM was significantly greater in unbiased-estimation assumption than in biased-estimation assumption (One-way ANOVA, unbiased-estimation assumption.vs. biased-estimation assumption, mean ± SD: 0.924 ± 0.031.vs. 0.914 ± 0.035, F_(1,361) = 16.85, p < 0.001), which means that the BCI model based on unbiased-estimation assumption better account for behavioral data (Figure 5A). When comparing the RMA of visual stimulation between behavior data (vision matching) and model estimation, we found that the estimated RMA of unbiased-estimation assumption was significantly better than that of biased-estimation assumption across spatial disparity (mean prediction error across spatial disparity, unbiased-estimation assumption vs. biased-estimation assumption, median: 0.140 vs. 0.287, Z = −3.266, p = 0.001) (Figures 5B and 5C). Overall, the BCI framework under unbiased-estimation assumption is more consistent with behavioral outcomes.

Table 1.

Results of the model comparison based on unbiased-estimate assumption

	PCOM	${\mu }_{\mathit{pr}}$	${\sigma }_{\mathit{pr}}$	${\sigma }_{\mathrm{P}}$	${\sigma }_V$	R2	relBIC	pEP	%win
BCI-MA	0.13 ± 0.24	7.43 ± 9.47	7.95 ± 1.63	8.20 ± 1.39	6.93 ± 1.49	0.98 ± 0.02	0	1	99.95
BCI-MS	0.26 ± 0.41	8.00 ± 8.96	7.79 ± 1.65	8.23 ± 1.25	6.95 ± 1.39	0.98 ± 0.02	102.17	0	0.02
BCI-PM	0.33 ± 0.49	7.09 ± 8.28	7.41 ± 1.49	7.54 ± 1.34	6.56 ± 1.42	0.97 ± 0.02	251.36	0	0
FF	–	5.41 ± 9.35	8.20 ± 1.58	9.09 ± 0.01	8.74 ± 0.55	0.91 ± 0.11	2464.78	0	0
UV	–	11.01 ± 8.70	8.68 ± 1.03	–	9.09 ± 0.01	0.90 ± 0.09	3151.97	0	0
UP	–	10.77 ± 8.77	8.78 ± 0.84	9.09 ± 0.01	–	0.79 ± 0.27	4133.67	0	0
FS	–	2.23 ± 1.48	0.32 ± 0.13	8.97 ± 0.38	8.61 ± 1.12	0.89 ± 0.13	3426.05	0	0.03

Model averaging, BCI-MA; Model selection, BCI-MS; Probability matching, BCI-PM, Forced-fusion, FF; Unimodel vision, UV; Unimodel proprioception, UP; Full-segregation, FS. R², coefficient of determination; relBIC, Bayesian information criterion at the group level; pEP, protected exceedance probability, i.e. the probability that a given model is more likely than any other model; % win, percentage of participants in which a model won the within-participant model comparison based on BIC.

Table 2.

Results of the model comparison based on biased-estimate assumption

	PCOM	${\mu }_{\mathit{pr}}$	${\sigma }_{\mathit{pr}}$	${\sigma }_{\mathrm{P}}$	${\sigma }_V$	R2	relBIC	pEP	%win
BCI-MA	0.65 ± 0.44	15.81 ± 11.82	7.75 ± 1.21	7.09 ± 1.31	7.44 ± 1.42	0.89 ± 0.19	703.29	0	0.03
BCI-MS	0.86 ± 0.45	15.44 ± 10.04	8.39 ± 1.31	7.32 ± 1.61	7.38 ± 1.59	0.96 ± 0.05	0	1	94.74
BCI-PM	0.87 ± 0.45	15.52 ± 9.99	8.21 ± 1.28	7.25 ± 1.66	7.10 ± 1.48	0.96 ± 0.05	23.47	0	0.43
FF	–	15.04 ± 11.17	8.92 ± 0.54	8.49 ± 1.46	8.69 ± 0.88	0.93 ± 0.05	1775.92	0	0
UV	–	11.72 ± 7.10	8.78 ± 0.89	–	9.07 ± 0.10	0.89 ± 0.12	2242.56	0	0
UP	–	23.98 ± 8.78	9.09 ± 0.02	8.60 ± 0.94	–	0.85 ± 0.17	2692.67	0	0
FS	–	1.83 ± 1.28	0.43 ± 0.19	7.90 ± 1.76	8.56 ± 1.23	0.95 ± 0.04	772.72	0	4.80

Model averaging, BCI-MA; Model selection, BCI-MS; Probability matching, BCI-PM, Forced-fusion, FF; Unimodel vision, UV; Unimodel proprioception, UP; Full-segregation, FS. R², coefficient of determination; relBIC, Bayesian information criterion at the group level; pEP, protected exceedance probability, i.e. the probability that a given model is more likely than any other model; % win, percentage of participants in which a model won the within-participant model comparison based on BIC.

Figure 4.

Representational dissimilarity matrices (RDMs) for behavior data and the estimates of BCI model

(A) Participants’ visual and proprioceptive matched outcomes (across-participants’ mean; n = 20) are plotted as a function of the true angle of visual (V#) and proprioceptive (P#) stimuli, separately for VM (top) and PM (bottom). The VM is more strongly influenced by the visual stimuli, while the PM is more strongly influenced by the proprioceptive stimuli. Cross-modal biases are also presented.

(B) RDM (across-participants’ mean) showing the absolute differences in participants’ matching outcomes between all pairs of the 25 experimental conditions. The angle of visual and proprioceptive stimuli for each condition is indicated by the black block.

(C) RDMs (across-participants’ mean) are from the angle estimates and the estimate of the posterior common source probabilities using the BCI ‘model average’ model under unbiased-estimation assumption.

(D) RDMs (across-participants’ mean) are from the angle estimates and the estimate of the posterior common source probabilities using the BCI ‘model select’ model under biased-estimation assumption.

Figure 5.

Comparison of the BCI framework between unbiased-estimation and biased-estimation assumption

(A) The RDM similarity between BCI framework and behavior outcomes. A high similarity (unbiased estimation vs. biased estimation, mean ± SD, 0.924 ± 0.030 vs. 0.914 ± 0.035, F_(1,361) = 16.85, p < 0.001) represents a strong interpretation of behavioral outcomes.

(B) Estimates (model) and matches (behavior) of visual cues as a function of spatial disparity. The RMA obtained under biased-estimation assumption tended to decrease when the spatial disparity increased. The RMA obtained under unbiased-estimation assumption EA is more consistent (mean prediction error across spatial disparity, unbiased estimation vs. biased estimation, median, 0.140 vs. 0.287, Z = −3.266, p = 0.001) with behavioral outcomes.

(C) Estimates (model) and matches (behavior) of proprioceptive cues as a function of spatial disparity. Both unbiased-estimation and biased-estimation assumptions have excellent estimates of proprioceptive cues. The error bars represent the standard deviation. ∗∗∗, p < 0.005.

Discussion

To form a coherent representation of the surrounding, the human brain needs to integrate signals arising from a common cause but segregate signals from independent causes.^{2,11,17,20,34} Perception thus relies crucially on the causal structure of the signal source. BCI has been considered to be a rational strategy to arbitrate between sensory integration and segregation in perception.^{14,21,22,23,35} The paradigm of audiovisual localization without laterality^17,36,37 and ipsilateral rubber hand localization^15,26 support the view that human behavior is approximately in line with BCI model. However, most human activities are of bilateral interaction involved in processing of interhemispheric sensory signals. It remains unknown whether the BCI could still explain human behavior in the context of interhemispheric perception. What is even less clear is whether the interhemispheric perceptual biases lead to different causal inferences. Our study focused on the interhemispheric multisensory perception and the causal inference. To characterize the cause structure of interhemispheric perception, we presented participants with a bilateral hand-matching task that varied in spatial disparity. Our results demonstrated that a unified representation of visual and proprioceptive signals could be formed on the contralateral side according to BCI framework and that a different strategy model was highlighted when considering interhemispheric perceptual bias.

Previous study has used BCI framework in predicting proprioceptive drift of the rubber hand illusion (RHI),^15,26 which also involves the integration of visual and proprioceptive cues. The study of computational account of the RHI has shown that the proprioceptive drift and the perception of body ownership are driven by Bayesian sensory inference.^15,26 In the RHI, proprioceptive drift was usually measured by the participant using their visible hand (or mouse) to point the location of the invisible hand.^15,26 Visual cues were used to locate the invisible hand, which does not involve interhemispheric sensory signals. In other words, the integration and segregation of multisensory signals in the RHI paradigm occur on the same side. However, many bilateral activities are related to interhemispheric transmission with the complex processing of sensory signals (e.g., sensory mirroring,^28,29 lateral-side dominance^30,31,32). For example, lateral-side dominance lead to asymmetries in target matching, resulting in a significantly larger perceptual bias (or matching error) when matching the contralateral position than when matching the ipsilateral position.³³ The uncertainty of perception associated with interhemispheric transmission may strengthen the estimated sensory bias, which is suggested to be considered in full explanations of the behavioral preformation.¹⁶ Our experimental paradigm is the first study to consider this interhemispheric multisensory perception. We first obtained the perceptual bias of unisensory matching task by matching only visual or proprioceptive angle (simulation). In the bilateral hand-matching task, participants were asked to match ipsilateral visual or proprioceptive cues with the contralateral hand.

The behavioral findings in our unisensory matching task showed that the perceptual bias was significantly greater in vision matching than in proprioception matching. A potential explanation for large biases in visual perception derives from the reference frame transformations.^38,39 When tracking ipsilateral visual cues with contralateral proprioceptive receptors, visual signal additionally needs to be transformed into proprioceptive information in the brain. In addition, the uncertainty of sensory estimates may also lead to distortions in perception of visual cues. For instance, inaccuracies in visual estimates of both egocentric distance estimates and object shapes have been well documented.⁴⁰ Crucially, one study demonstrated that the visual bias dominated the magnitude of the central bias in the audiovisual localization task.¹⁶ It also revealed that the perceptive biases in unisensory were also necessary for BCI framework in their study.¹⁶ Inspired by this study, we therefore quantified BCI models under unbiased-estimation assumption and biased-estimation assumption to investigate the impact of interhemispheric perceptual biases on inference framework.

Our results showed that the BCI models (BCI-MA, BCI-MS, BCI-PM [probability matching]) outperformed the other models (FF, unimodal vision [UV], unimodal proprioception [UP], full-segregation [FS]) in both unbiased-estimation assumption and biased-estimation assumption. Extensive research has demonstrated the superiority of the BCI framework in the tasks such as audiovisual localization^17,36,37 and hand localization.^15,26 Notedly, we focused on the integration and segregation of multisensory signals on the contralateral side. Our results support that interhemispheric causal inference is most well derived from the BCI framework. This indicates that the principle of spatial consistency was obeyed when a uniform estimate of multisensory signals formed on the contralateral side. Under this principle, human integrated multisensory information when cross-modal signals are coherent and tended to segregate the signals as the disparity progressively increases.^15,17 Unlike previous studies, we also found that a different BCI strategy model was selected when considering interhemispheric perceptual biases. The optimal strategy model is the “model average” under unbiased-estimation assumption and is the “model select” under biased-estimation assumption. Most previous studies have supported the BCI model under unbiased-estimation assumption with “model average” (BCI-MA) as the optimal strategy,^5,17,20,41 which is consistent with our results. However, when considering interhemispheric perceptual bias, the brain selectively integrates or segregates multisensory signals according to the magnitude of the posterior probability. Therefore, the interhemispheric perceptual bias may influence the BCI framework, under which the brain adopts a different strategy to estimate the contralateral multisensory signal.

To obtain the optimal estimation model, we compared the BCI models (BCI-MA vs. BCI-MS) under the two assumptions (unbiased-estimation assumption and biased-estimation assumption) through a similarity analysis in RDM. We found that the “model average” strategy under unbiased-estimation assumption had greater similarity for behavioral data than the “model selection” strategy under biased-estimation assumption. Furthermore, the estimation bias increased with the spatial disparity when the visual cues were estimated using the “model selection” strategy under biased-estimation assumption. This finding suggests that the biases of interhemispheric visual perception dominate the goodness of the estimation. In fact, the magnitude of visual perceptual bias in unisensory matching task increases as the stimulus angle enlarged, resulting in a smaller matching angle for a larger stimulus angle. It led to a greater estimation bias in the “model selection” strategy because the estimated location was obtained by the unisensory visual perception and the prior distribution. In contrast, a smaller bias was observed when matching proprioceptive cues in the unisensory matching task, resulting in a better estimation for both unbiased-estimation assumption and biased-estimation assumption. The findings further support that the bias of interhemispheric visual perception is responsible for a lower accuracy of the “model selection” strategy for biased-estimation assumption. The visual perception is characterized by a less matching accuracy under larger visual stimulus angles, which would seem to be a suboptimal strategy. One possible explanation is that the suboptimal strategy represents an advantageous trade-off between accuracy and the movement speed needed for a good matching.^42,43 A high-quality performance is associated with the cost of diminished movement speed.^44,45 We speculated that participants could need more time consumption to improve their performance when matching a large visual angle. Additionally, the sensory representation of visual cues but a priori perception may be biased.^16,46 For instance, one study concluded that sensory likelihoods (i.e., sensory representations) become distorted through a process of “efficient encoding” in the perception of visual orientation and spatial frequency.⁴⁶ Overall, our results suggest that visual perceptual bias deteriorates the BCI framework for biased-estimation assumption. The visual bias of unisensory perception seems to reduce the accuracy of BCI framework. One possible explanation is that sensory representation is partially or completely inconsistent between unisensory and multisensory conditions. It has been reported that the multisensory integration is dominated by the visual bias, which tends to increase under the unisensory condition.¹⁶ Therefore, the perceptual bias in the unisensory macthing is not a superior substitute for the sensory representation to optimize the BCI framework.

In conclusion, the interhemispheric causal inference for both unbiased-estimation assumption and biased-estimation assumption follows the BCI framework. The interhemispheric perceptual bias may make a difference in strategy models to estimate the contralateral multisensory signal. However, the visual perceptual bias in unisensory conditions could form a suboptimal model under biased-estimation assumption, which cannot be regarded as a superior input for optimizing the BCI framework. The “model average” strategy under unbiased-estimation assumption is best for understanding the causal structure of interhemispheric multisensory cues.

Limitations of the study

This study focused primarily on interhemispheric multisensory perception and BCI model. In our tasks, participants were required to perform an operational practice in VR environment. A previous study focusing on user experience with VR has revealed that the prior experience has an effect on self-assessment of performance and pragmatic quality.⁴⁷ Therefore, it is possible that the different prior experiences among participants would have effect on the perception of hand motion in VR. To diminish the individual experience, we provided a pre-experiment practice for participants until they felt being familiar with the virtual environment. Despite this, there is also a little difference in the prior experience of VR, which could not be eliminated absolutely. Another limitation is that our task presented the visual stimulation in the ipsilateral side, which deviated the center of visual field. Many studies have demonstrated that the visual periphery has less processing resolution that weakens capacities for motion perception⁴⁸ and color sensitivity.⁴⁹ The weak motion perception of visual periphery may enlarge the perceptual bias in our task. Future research will be needed to improve the visual presentation to reduce the impact of visual periphery.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and algorithms
Matlab 2019b	MathWorks	www.mathworks.com
VBA -toolbox	MBB team	https://mbb-team.github.io/VBA-toolbox/wiki/

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Xiaoyu Liu (x.y.liu@buaa.edu.cn).

Materials availability

This study did not generate new unique reagents.

Experimental model and subject details

Participants

We included 20 right-handed, healthy participants (10 males aged 24.2 ± 1.62 years and 10 females aged 25.7 ± 3.26 years) with normal binocular vision or corrected visual acuity. This study adhered to the tenets of the Declaration of Helsinki, and ethical approval was provided by the Ethics Committee, Beihang University. We provided all the participants written and verbal information about the experiment’s purpose and procedure, and each participant signed an informed consent statement.

Apparatus

A bilateral exoskeleton robot was developed to provide and obtain the movement of hands (Figure 1A). Participants sat upright in front of a height-adjustable table on which we mounted the bilateral exoskeleton robot. The left exoskeleton robot drove the real left-hand (ipsilateral to visual hand) with a certain angle in a completely passive motion. The right performed active mode to obtain the matching angle of right-hand (contralateral to visual hand). A three-dimensional (3D) virtual left-hand was created and placed in a position spatially overlapping the position of the participants’ individual left-hand. Participants wore a head-mounted display (HMD, VIVE Pro, HTC Corporation) in which the visual hand was presented from the first-person perspective.

Method details

Procedure

Experiment 1- unisensory matching task

To examine the interhemispheric bias in unisensory perception, participants were asked to operate the robot with their right hand to match the visual or real hand on the left in the two blocks. Participants were asked to perform the matching task in the hand flexion (also called wrist flexion). In the first block, only the visual stimuli of virtual hand were presented, and participants had to match their right hand with the virtual left. Five different visual stimuli were created by flexing the virtual hand at different angles: 10°, 20°, 30°, 40° and 50°. The visual stimulus was selected from five different angles in a pseudo-random algorithm. Each angle was selected six times, so that each participant had to complete a total of 30 trials (five stimuli × six times) in the first block. In the second block, only the proprioceptive stimuli of real hand were presented, and participants had to match their right hand with their left real hand. First, the left hand was passively flexed by the left robot at one of five different angles: 10°, 20°, 30°, 40° and 50°. The proprioceptive stimulus was selected from five different angles in a pseudo-random algorithm.⁵⁰ The participant then had to manipulate the right robot to perform a flexion motion to match the proprioceptive stimulus. Each angle was matched six times, so that a total of 30 trials (five stimuli × six times) was completed in the second block. The relative matching accuracy (RMA) was calculated by the matching angle over the stimuli angle.

Experiment 2-multisensory matching task

Participants performed the matching tasks under condition with visual and proprioceptive stimulus were synchronously presented. The visual and proprioceptive stimuli was selected from five different angles: 10°, 20°, 30°, 40° and 50°. A total of 25 stimulus conditions (five visual stimulation × five proprioceptive stimulation) was structured in the multisensory matching task. The order of stimulus conditions was counterbalanced across runs to decorrelate matching responses from previous stimulus in a pseudo-random algorithm.⁵⁰ Across experimental runs, we instructed participants to selectively match either the visual hand or real hand and to ignore the stimuli in the task-irrelevant modality. They were asked to match their right hand (contralateral) with the left hand according to the multisensory stimulus in two blocks. In the first block, participants had to match their right hand with the virtual left hand. Each stimulus condition was presented three times, and a total of 75 trials (25 stimulus combination × three times) were completed in the first block. In the second block, participants had to match their right hand with the real left hand. Each stimulus condition was presented three times, and each participant was required to complete 75 trials (25 stimulus combination × three times) in the second block. The relative matching accuracy (RMA) was calculated by the matching angle over the stimuli angle of task-relevant modality.

The experimental design factorially manipulated the angle of virtual hand (visual stimulation), the angle of real hand (proprioceptive stimulation) and the task relevance of the sensory modality (vision matching vs. proprioception matching). The spatial disparity was calculated by subtracting the angle of task irrelevance sensory stimulation from the angle of task relevance sensory stimulation (vision matching: the angle of visual stimulation minus the angle of proprioceptive stimulation; proprioception matching: the angle of proprioceptive stimulation minus the angle of visual stimulation). To characterize the behavior on spatial disparity, we reorganized these conditions into a two (task relevance: vision matching vs. proprioception matching) × nine (spatial disparity: -40, -30 -20 -10, 0, 10, 20, 30, 40) factorial design (Figure 2C). We defined the high (spatial disparity ≥ 20 or spatial disparity ≤ -20, light grey block) and low spatial disparity (spatial disparity ≥ -10 and spatial disparity ≤ 10, dark grey block).

The participants' subjective feelings were assessed using a -3/+3 Likert scale⁵¹ (-3 represented complete inconsistency, 0 neither consistency nor inconsistency, and +3 complete consistency). When matching was complete in each trial, participants were required to report a rating regarding consistency between the visual and proprioceptive stimuli. The time interval between two blocks and two experiments was more than 15 minutes to eliminate potential interaction effects. There was no limit to the finish time in all trials.

Quantification and statistical analysis

Visual weight index W_V

In the multisensory matching task, we calculated the visual weight index (W_V) to investigate the contribution of visual and proprioceptive perception. The W_V was defined as a function of matching angle and stimulus angle, as follows:

$$\begin{array}{c}{W}_v=\frac{\left|{\theta }_M-{\theta }_{P\#}\right|}{\left|{\theta }_{P\#}-{\theta }_M\right|+\left|{\theta }_{V\#}-{\theta }_M\right|}\end{array}$$ (Equation 1)

The ${\theta }_M$ is the matching angle of participants. ${\theta }_{V\#}$ and ${\theta }_{P\#}$ represent the angle of visual stimulation and proprioceptive stimulation, respectively.

Bayesian causal inference model

We used Bayesian Causal Inference (BCI) model^2,5,6,17,23 to investigate the characteristics and perceptual components of human spatial processing in contralateral matching (location) task. The stimulus position S_V and S_P are considered to originate from a normal prior distribution $\mathrm{N}\left({\mu }_{\mathrm{pr}},{\sigma }_{\text{pr}}\right)$. Under the assumption of unbiased-estimation, we introduced sensory noise by drawing X_V (X_P) from normal distributions centered on the stimulus position S_V (S_P) with parameters ${\sigma }_{\mathrm{V}}$ (${\sigma }_{\mathrm{P}}$). For participants, multisensory signals were either from a common cause (C=1) or from the independent causes (C=2). Thus, the probability of common sources (p(C=1) = p_com) follows a binomial distribution. The brain makes an inference about whether the sensory signals come from the same source (C=1) and should be integrated or the signals come from different sources (C=2) and should be segregated.^5,6 The posterior probability of the underlying causal structure can be inferred by combining the causal prior with the sensory evidence according to Bayes rule:^5,6

$$\begin{array}{c}p\left(C|X_V,X_P\right)=\frac{p\left(X_V,X_P|C\right)p\left(\mathrm{C}\right)}{p\left(X_V,X_P\right)}\end{array}$$ (Equation 2)

$$\begin{array}{c}p\left(C=2|X_V,X_P\right)=1-p\left(C=1|X_V,X_P\right)\end{array}$$ (Equation 3)

$$\begin{array}{c}p\left(C=1|X_V,X_P\right)=\frac{p\left(X_V,X_P|C=1\right)p_{com}}{p\left(X_V,X_P|C=1\right)p_{com}+p\left(X_V,X_P|C=2\right)\left(1-p_{com}\right)}\end{array}$$ (Equation 4)

If there is a common cause (S_V = S_P=S), the formula reduces to:

$$p\left(X_V,X_P|C=1\right)=\int p\left(X_V,X_P|S\right)p\left(S\right)dS=\int p\left(X_V|S\right)p\left(X_P|S\right)p\left(S\right)dS=\frac{1}{2\pi \sqrt{{\sigma }_V^2{\sigma }_P^2+{\sigma }_V^2{\sigma }_{\mathit{pr}}^2+{\sigma }_P^2{\sigma }_{\mathit{pr}}^2}}\mathit{exp}\left[-\frac{1}{2}\frac{{\left(X_V-X_P\right)}^2{\sigma }_{\mathit{pr}}^2+{\left(X_V-{\mu }_{\mathit{pr}}\right)}^2{\sigma }_P^2+{\left(X_P-{\mu }_{\mathit{pr}}\right)}^2{\sigma }_V^2}{{\sigma }_V^2{\sigma }_P^2+{\sigma }_V^2{\sigma }_{\mathit{pr}}^2+{\sigma }_P^2{\sigma }_{\mathit{pr}}^2}\right]$$ (Equation 5)

If there are the independent causes, draw X_V from $\mathrm{N}\left({\mathrm{S}}_{\mathrm{V}},{\sigma }_{\mathrm{V}}\right)$ and X_P from $\mathrm{N}\left({\mathrm{S}}_{\mathrm{P}},{\sigma }_{\mathrm{P}}\right)$, respectively.

$$p\left(X_V,X_P|C=2\right)=\iint p\left(X_V,X_P|S_V,S_P\right)p\left(S_V,S_P\right)d{S}_{V}d{S}_P=\left(\int p\left(X_V|S_V\right)p\left(S_V\right)d{S}_V\right)\left(\int p\left(X_P|S_P\right)p\left(S_P\right)d{S}_P\right)=\frac{1}{2\pi \sqrt{\left({\sigma }_V^2+{\sigma }_{\mathit{pr}}^2\right)\left({\sigma }_P^2+{\sigma }_{\mathit{pr}}^2\right)}}\mathit{exp}\left[-\frac{1}{2}\left(\frac{{\left(X_V-{\mu }_{\mathit{pr}}\right)}^2}{{\sigma }_V^2+{\sigma }_{\mathit{pr}}^2}+\frac{{\left(X_P-{\mu }_{\mathit{pr}}\right)}^2}{{\sigma }_P^2+{\sigma }_{\mathit{pr}}^2}\right)\right]$$ (Equation 6)

When the sensory signals have a common cause, the optimal solution is:

$$\begin{array}{c}{\hat{\mathrm{S}}}_{\mathrm{V},\mathrm{C}=1}={\hat{\mathrm{S}}}_{\mathrm{P},\mathrm{C}=1}=\frac{\frac{X_V}{{\sigma }_V^2}+\frac{X_P}{{\sigma }_P^2}+\frac{{\mu }_{\mathit{pr}}}{{\sigma }_{\text{pr}}^2}}{\frac{1}{{\sigma }_V^2}+\frac{1}{{\sigma }_P^2}+\frac{1}{{\sigma }_{\text{pr}}^2}}\end{array}$$ (Equation 7)

In the case of independent causes (C=2), the optimal estimates of the unisensory visual and proprioceptive stimuli are independent:

$$\begin{array}{c}{\hat{\mathrm{S}}}_{\mathrm{V},\mathrm{C}=2}=\frac{\frac{X_V}{{\sigma }_V^2}+\frac{{\mu }_{\mathit{pr}}}{{\sigma }_{\text{pr}}^2}}{\frac{1}{{\sigma }_V^2}+\frac{1}{{\sigma }_{\text{pr}}^2}}\end{array}$$ (Equation 8)

$$\begin{array}{c}{\hat{\mathrm{S}}}_{\mathrm{P},\mathrm{C}=2}=\frac{\frac{X_P}{{\sigma }_P^2}+\frac{{\mu }_{\text{pr}}}{{\sigma }_{\text{pr}}^2}}{\frac{1}{{\sigma }_P^2}+\frac{1}{{\sigma }_{\text{pr}}^2}}\end{array}$$ (Equation 9)

Arbitrate strategies based on different model

We fitted seven different strategy model^2,5,6,17,23 as decision functions to behavioral results, which including the unimodal vision (UV), the unimodal proprioception (UP), the full-segregation (FS), the forced-fusion (FF) and three Bayesian Causal Inference model with model averaging (BCI-MA), model selection (BCI-MS) and probability matching (BCI-PM). Using model comparison, we then assessed which of these models is the best explanation for interhemispheric multisensory perception.

According to BCI-MA model (model 1), the brain combines the estimates weighted in proportion to the posterior probabilities of their underlying causal structures.

$$\begin{array}{c}\{\begin{array}{c}{\hat{S}}_V=p\left(C=1|X_V,X_P\right){\hat{S}}_{V,C=1}+\left(1-p\left(C=1|X_V,X_P\right)\right){\hat{S}}_{V,C=2}\\ {\hat{S}}_P=p\left(C=1|X_V,X_P\right){\hat{S}}_{P,C=1}+\left(1-p\left(C=1|X_V,X_P\right)\right){\hat{S}}_{P,C=2}\end{array}\end{array}$$ (Equation 10)

According to BCI-MS model (model 2), the brain obtains the estimate selectively from the more likely causal structure.

$$\begin{array}{c}\{\begin{array}{c}{{{\hat{S}}_P=\hat{S}}_V=\hat{S}}_{V,C=1};ifp\left(C=1|X_V,X_P\right)>0.5\\ {\hat{S}}_V={\hat{S}}_{V,C=2},{\hat{S}}_P={\hat{S}}_{P,C=2};ifp\left(C=1|X_V,X_P\right)\le 0.5\end{array}\end{array}$$ (Equation 11)

According to BCI-PM model (model 3), the brain obtains the estimate of one causal structure stochastically selected in proportion to its posterior probability. Therefore, the cutoff probability α obeys the uniform distribution $U\left(\mathrm{0,1}\right)$.

$$\begin{array}{c}\{\begin{array}{c}{{{\hat{S}}_P=\hat{S}}_V=\hat{S}}_{V,C=1};ifp\left(C=1|X_V,X_P\right)>\alpha ,\alpha \sim U\left(\mathrm{0,1}\right)\\ {\hat{S}}_V={\hat{S}}_{V,C=2},{\hat{S}}_P={\hat{S}}_{P,C=2};ifp\left(C=1|X_V,X_P\right)\le \alpha ,\alpha \sim U\left(\mathrm{0,1}\right)\end{array}\end{array}$$ (Equation 12)

The visual and proprioceptive stimulus were forced to integration in the FF model (model 4). The multisensory estimates can be computed a reliability-weighted linear average of the two unisensory estimates.

$$\begin{array}{c}{\hat{S}}_V={\hat{S}}_P=\frac{{\sigma }_V^2}{{\sigma }_P^2+{\sigma }_V^2}{X}_V+\frac{{\sigma }_P^2}{{\sigma }_P^2+{\sigma }_V^2}{X}_P\end{array}$$ (Equation 13)

The UV model (model 5) estimates stimulus relies only on visual cues.

$$\begin{array}{c}\{\begin{array}{c}{\hat{S}}_V=X_V\\ {\hat{S}}_P=X_V\end{array}\end{array}$$ (Equation 14)

The UP model (model 6) estimates stimulus relies only on proprioceptive cues.

$$\begin{array}{c}\{\begin{array}{c}{\hat{S}}_V=X_P\\ {\hat{S}}_P=X_P\end{array}\end{array}$$ (Equation 15)

The FS model (model 7) estimates stimulus independently for vision and proprioception.

$$\begin{array}{c}\{\begin{array}{c}{\hat{S}}_V=X_V\\ {\hat{S}}_P=X_P\end{array}\end{array}$$ (Equation 16)

Model optimization

According to previous studies, the fitting data is introduced by sampling the likelihoods from a normal distribution centered at the sensory stimulus, plus a bias term that scales linearly with the variance of stimulus.³⁶ If participants’ sensory representations are biased against the sensory stimulus, this bias term could potentially reflect this systematic shift in the likelihoods.¹⁶ Considering our behavioral results, which reveal interhemispheric bias in the contralateral matching tasks, it appears that the model would need to be enhanced to allow flexibility in the representation of likelihoods when considering interhemispheric multisensory perception. In this case, the visual and the proprioceptive signal are draw X_V from $\mathrm{N}\left({\mu }_{\mathrm{V}},{\sigma }_{\mathrm{V}}\right)$ and X_P from $\left({\mu }_{\mathrm{P}},{\sigma }_{\mathrm{P}}\right)$. The ${\mu }_{\mathrm{V}}$ and ${\mu }_{\mathrm{P}}$ can be obtained by unisensory matching task.

To explore the effects of interhemispheric perception bias, we fitted decision models separately under unbiased and biased estimation conditions. Under the unbiased-estimation assumption, the predicted distributions were generated by simulating X_V and X_P 5000 times (i.e. continuous variables sampled from Gaussian distributions) for each stimulus condition (25 total) from a normal distribution centered at the sensory stimulus. Under the biased-estimation assumption, the predicted distributions were generated by simulating X_V and X_P 5000 times for each stimulus condition (25 total) from a normal distribution centered at the matched mean angle in the unisensory matching task. We computed the log likelihood of participants’ perceived location (matching angle) and summed the log likelihoods across different stimulus condition. The log likelihood was given by:

$$\begin{array}{c}\log L\left(model|\left\{n_{i,j}\right\}\right)=\sum _{i=\left\{S_V,S_P\right\}}\sum _{j=1}^5{n}_{i,j}\log p_{i,j}+constant\end{array}$$ (Equation 17)

where $n_{i,j}$ is the total number of trials in the condition $i$ and the model $j$, and $p_{i,j}$ are the respective modeled response probabilities. To obtain maximum likelihood estimates for the five parameters of the models (p_com, μ_pr, σ_pr, σ_V, σ_P; formally, the FF and FS models assume p_com=1 or 0, respectively), we used a non-linear simplex optimization algorithm as implemented in Matlab’s fmincon function (Matlab 2019b) to obtain the optimal solution of log-likelihood function in variable space. The model fit for behavioral data was assessed by the coefficient of determination R² defined as^14,15,52:

$$\begin{array}{c}{R}^2=1-\mathit{exp}\left[-\frac{2}{n}\left(l\left(\hat{\beta }\right)-l\left(0\right)\right)\right]\end{array}$$ (Equation 18)

were $l\left(\hat{\beta }\right)$ and $l\left(0\right)$ denote respectively the log likelihoods of the fitted and the null model, and $n$ is the number of data points. For the null model, we assumed that participants would match the perceived hand angle randomly over the disparity range where form the leftmost to the rightmost. Thus, a uniform distribution over this span was predicted. To identify the optimal model for explaining participants’ data, We used the Bayesian information criterion (BIC) as an approximation to the model evidence.^53,54

$$\begin{array}{c}\mathrm{B}\mathrm{I}\mathrm{C}=-\ln \left(\hat{L}\right)+\ln \left(n\right)\ast k\end{array}$$ (Equation 19)

where $\hat{L}$ denotes the likelihood, $n$ the number of data points and $k$ the number of parameters. The relative Bayesian information criterion at the group level (relBIC) was the summation of all participants’ BIC relative to the causal inference model.^2,17 The log model evidence was calculated as:⁵⁵

$$\begin{array}{c}\mathit{Log}ModelEvidence=-\frac{1}{2}\ast BIC\end{array}$$ (Equation 20)

Final, we obtained the model attribution, estimated model frequencies and exceedance probability using the Variational Bayesian Analysis (VBA -toolbox), which is a fully Bayesian toolbox for model-based data analyses, to identify the better model at the group level.

Model comparison

We used the representational similarity analysis to obtain the behavioral optimal model. To define the representational dissimilarity matrices (RDM) in behavior, we computed the pairwise absolute distance in participants’ angle matching between all pairs of the 25 conditions in the multisensory matching task. The angle estimates of model was calculated by final BCI model (unbiased-estimation assumption: BCI-MA; biased-estimation assumption: BCI-MS) across all 25 conditions individually for each participant and then averaged those across participants. Like behavioral RDM, the model RDM was constructed using the angle estimates.

We used a Perceptual Hash Algorithm^56,57 for subtle RDM differences identification between behavior and model. Instead of a conventional pixel compression, we selected 32×32-pixel matrix in the original RDM matrix step by step (361 group selection total), which preserves as much detail as possible. Then, we converted the image (RGB value) of RDM matrix to a grayscale picture, which caused the hash from 32×32 pixels to a total of 32×32 colors. The Perceptual Hash algorithm computes hashing (a 64-bit hash) on top of Discrete Cosine Transform that transforms data from spatial domain to frequency domain. The hamming distance⁵⁶ was obtained by the comparison of the binary hash between two images. Thus, the similarity between behavior data and model estimates⁵⁸ was calculated as:

$$\begin{array}{c}Similarity=\frac{64-Hammingdistance}{64}\end{array}$$ (Equation 21)

Finally, we compared the similarity of RDM between Unbiased-estimation assumption and biased-estimation assumption (see Figure S2). And we characterized the estimates feature of model as a function of spatial disparity to obtain the conformity to behavioral data.

In our study, non-parametric tests with the median were used to analyze non-normal data, while parametric tests with the mean were used to analyze normal data.

Published: April 20, 2023

Data and code availability

All data reported in this paper will be shared by the lead contact upon request. This paper does not report original code. Any additional information required to reanalyze the data reported in this paper is available form the lead contact upon request.