Optimizing Visual Cortex Parameterization with Error-Tolerant Teichmüller Map in Retinotopic Mapping

Abstract

The mapping between the visual input on the retina to the cortical surface, i.e., retinotopic mapping, is an important topic in vision science and neuroscience. Human retinotopic mapping can be revealed by analyzing cortex functional magnetic resonance imaging (fMRI) signals when the subject is under specific visual stimuli. Conventional methods process, smooth, and analyze the retinotopic mapping based on the parametrization of the (partial) cortical surface. However, the retinotopic maps generated by this approach frequently contradict neuropsychology results. To address this problem, we propose an integrated approach that parameterizes the cortical surface, such that the parametric coordinates linearly relates the visual coordinate. The proposed method helps the smoothing of noisy retinotopic maps and obtains neurophysiological insights in human vision systems. One key element of the approach is the Error-Tolerant Teichmüller Map, which uniforms the angle distortion and maximizes the alignments to self-contradicting landmarks. We validated our overall approach with synthetic and real retinotopic mapping datasets. The experimental results show the proposed approach is superior in accuracy and compatibility. Although we focus on retinotopic mapping, the proposed framework is general and can be applied to process other human sensory maps.

1. Introduction

There is a great interest to understand, quantify, and simulate the human retinotopic mapping, i.e. the mapping between the visual field on the retina to the cortical surface. Since the first time functional magnetic resonance imaging (fMRI) was introduced to measure human retinotopic maps in vivo [1, 2], many improvements have been made: New experimental protocol were carried out, especially the traveling wave experiment [3]; The population receptive field model (pRF) was proposed to better interpret fMRI data [4]. Those researches bring inspiring insights to understand human vision systems.

Besides the great scientific interest, human retinotopic maps are also applied in several fields. In ophthalmology, there is a great need to evaluate the visual organization of amblyopia, which affects ~2% of all children [5] and may cause significant visual impairment if untreated. The retinotopic map is a better detection method and guides a proper treatment for amblyopia patients even for adults, which is usually believed incurable [6–8]. In neurology, fMRI signal and retinotopic maps have been adopted to help register cortical surfaces and discover more brain visual-related areas [9]. The retinotopic maps are also adopted in computer vision, e.g. a retinotopic Spiking Neural Network is inspired to recognize moving objects [10].

Although great progress has been made, there are several key characteristics, e.g. cortical magnification, have not been quantified precisely for human retinotopic maps due to several challenges. The fundamental challenge is the fMRI signal is of low signal-noise ratio and low spatial resolution, which makes the retinotopic maps noisy and blurry. Meanwhile, the cortical surface is complicated and convoluted, which makes the spatial smoothing extremely difficult for the noisy retinotopic maps.

Previous works tackle these challenges in separated steps [11–13]: the cortical surface is parametrized to a 2D domain, then smoothing methods are applied to the noisy retinotopic data, based on fitting smooth function on the parametrization domain. Although intuitive, the smoothed results by this approach are not compatible with the neuroscientific results: the retinotopic maps are topological/diffeomorphic (preserve the neighboring relationship) within each visual area in neuroscientific results [12, 14], yet the smoothed result does not preserve the neighboring relationship.

We address the parametrization and smoothing simultaneously and propose a novel cortical parametrization approach. The reason one shall combine them is that the parametrization influences the smoothing complexity, and vice versa. The smoothing is processed on the parametrized coordinates, where a small change of the coordinates may significantly reduce the complexity of smoothing in the 2D domain; On the other hand, high-quality visual coordinates can be used to guide the desired parametrization. Therefore, the combined approach has the potential to balance the difficulties of surface parametrization and smoothing. In specific, we first formulate the problem as a parameterization optimization problem. Then we solve the problem iteratively with Laplacian smoothing [15] and the proposed Error-Tolerant Teichmüller Map (ETTM). The ETTM can handle self-contradicting landmarks, so one can set landmarks for the parametrization with errors. This is the first approach that aligns the retinotopic parameters to the stimulus visual field and provides canonical space for retinotopic maps. The ETTM is proposed in general so it can be adapted to other sensory cortex like auditory maps [16].

2. Method

2.1. Background on Retinotopic Maps

We briefly explain the retinotopic mapping and introduce the notations. Suppose the visual stimulation at position v = (v⁽¹⁾, v⁽²⁾) is s(t, v). Note we take the polar angle system for the visual position, i.e. v⁽¹⁾ is the radical distance to the origin (i.e. eccentricity), and v⁽²⁾ is the polar angle in the visual field (i.e. polar angle). The visual system will perception the stimulation and eventually activate a population of neurons, illustrated in Fig. 1. The main purpose of retinotopic mapping is to find the center v and the extent σ ∈ R⁺ of its receptive field for each point $P=(X,Y,Z)\in {ℝ}^{\mathcal{3}}$ on the visual cortex. fMRI provides a noninvasive way to determine v and σ for P, based on the following procedure: (1) Design a stimulus s(t; v), such that it is unique respect to location, i.e., s(t;v₁)≠s(t;v₂),∀ v1≠v₂; (2) Present the stimulus sequence to an individual and record the fMRI signals during the stimulation; (3) For each point P on cortical, collect the fMRI signal along the time, y(t:P); (4) Determine the parameters, including its central location v and its size σ, that most-likely generated the fMRI signals. Specifically, one assumes the neurons’ spatial response r(v′;v, σ) model, and the hemodynamic function h(t) , to predict the fMRI signal by, $\widehat{y}(v,\sigma )=\beta \left(\int r\left(v^{\prime };v,\sigma \right)s\left(t,v^{\prime }\right)d{v}^{\prime }\right)*h(t)$, where β is a coefficient that converts the units of response to the unit of fMRI activation. The parameters v and σ are estimated by minimizing the prediction error, i.e. $(v,\sigma )=\text{arg}\text{min}\mid \widehat{y}(v,\sigma )-{y(\text{t};P)|}^2$. The retinotopic maps are obtained when (v, σ) is solved for every point on the cortical surface. Fig. 1(b) shows a typical retinotopic mapping decoded and rendered on the inflated cortical surface. Besides the retinotopic maps, one can further evaluate the goodness of retinotopic parameters for each location by computing the variance explained, $R^2=\int |\widehat{y}-\overline{y}{|}^{2}dt/\int |y-\overline{y}{|}^{2}dt$.

Fig. 1.

(a) Illustration of the human visual system and retinotopic mapping procedure: The Visual Stimulus is presented in front of the subject’s visual field, and then recording the fMRI signal during the process; (b) The retinotopic coordinates v=(v⁽¹⁾,v⁽²⁾) are decoded, and rendered on the inflated cortical surface by population receptive field (pRF) analysis: The top and bottom are the visual eccentricities v⁽¹⁾ and visual polar angles v⁽²⁾ on the cortical surface, respectively.

2.2. Problem Statement

We wish to parametrize the visual cortex such that the parametric coordinates, $\widehat{u}=\left({\widehat{u}}^{(1)},{\widehat{u}}^{(2)}\right)$ in polar coordinates linearly relate the retinotopic coordinates v = (v⁽¹⁾, v⁽²⁾). Namely, $\widehat{\rho }=k_1{v}^{(1)}+b_1,\widehat{\theta }=k_2{v}^{(2)}+b_2$, where $\widehat{\rho }=\left|{\widehat{u}}^{(1)}+i{\widehat{u}}^{(2)}\right|$, $\widehat{\theta }=\text{arg}\left({\widehat{u}}^{(1)}+i{\widehat{u}}^{(2)}\right)$, k = (k⁽¹⁾, k⁽²⁾), and b = (b⁽¹⁾, b⁽²⁾) are constants. However, the raw retinotopic coordinates v is noisy. It will violate the topological condition by simply enforcing the coordinate $\widehat{u}$ to the noisy coordinates. We propose a method to generate a smooth and topological parametrization. Mathematically, the problem is to find the minimum of energy,

$$(\widehat{u},\widehat{v})=\text{arg}\text{min}E=\int {\left(\widehat{\rho }-k_1{\widehat{v}}^{(1)}-b_1\right)}^2+{\left(\widehat{\theta }-k_2{\widehat{v}}^{(2)}-b_2\right)}^2+{\lambda }_1{\left|{\nabla }_{\widehat{u}}\widehat{v}\right|}^2+{\lambda }_{2}w|\widehat{v}-v{|}^{2}ds,\hspace{1em} s.t\text{. }\left|{\mu }_{u\to \widehat{u}}\right|<1,$$ (1)

where $\widehat{u}=\left({\widehat{u}}^{(1)},{\widehat{u}}^{(2)}\right)$ is the desired parametrization coordinates, $\widehat{v}=\left({\widehat{v}}^{(1)},{\widehat{v}}^{(2)}\right)$ is the desired smoothing retinotopic coordinates, ${\nabla }_{\widehat{u}}$ is the gradient operator defined on the parametric domain $\widehat{u}$, i.e. ${\nabla }_{\widehat{u}}=\left(\partial /\partial {\widehat{u}}^{(1)},\partial /\partial {\widehat{u}}^{(2)}\right)$, λ₁, λ₂ are constants, w is a weight coefficient with the purpose of emphasis high-quality points, and ${\mu }_{u\to \widehat{u}}$ is the Beltrami coefficient [17] associated with the mapping from the initial parametrization u to the desired $\widehat{u}$. The first two terms in Eq. (1) are introduced to linearly align the parametric coordinates to the smoothed retinotopic coordinates. Likewise, the last two terms are introduced with the purpose of smooth the noisy v. Lastly, the constraint, $\left|{\mu }_{u\to \widehat{u}}\right|<1$, is introduced to ensure the new parametric coordinates is topological respect to the cortical surface [18].

2.3. The Iterative Parametrization.

The optimization of energy in Eq. (3) is difficult as the $\widehat{u}$ and $\widehat{v}$ have influences on each other. We adopt the ADMM [19] to separate the problem into two sub-problems, namely the smoothing problem and the parametrization problem, then solve the sub-problems iteratively. In practice, the cortical surface is discretized as triangular mesh consists of triangular faces and vertices, denoted by S = (F, V). The retinotopic parameters (v, σ, R) are solved by the pRF method [4, 20] for each vertex. We denote the surface with retinotopic parameters by (F, V, v, R), which is the input of the method.

The overall pipeline of the parametrization is illustrated in Fig. 2. (1) Given the surface with retinotopic parameters, as illustrated in Fig. 2(a), we first find the region of interest (ROI). In this paper, we select the region of interest as the V1/V2/V3 complex; (2) Then we find a geodesic patch that contains the ROI. Specifically, pick a point as the center, find geodesic distances from the center point to all vertices, and keep the portion of the surface whose geodesic distance is within certain value r. We call this patch as the geodesic patch, P = (F_P, V_P, v_P, R_P). The purpose to bound the patch by geodesic distance is to reduce distance distortion; (3) Later, we parametrize the patch to unit disk by conformal [21] mapping c, denoted by u = c(P). This is the initial coordinates for our method, as illustrated in Fig. 2 (b); (4) The retinotopic coordinates are smoothed on the domain of u. Specifically, with the initial parametrization u and the raw retinotopic coordinates v, fitting a smooth function $\widehat{v}(u)$ to approach the raw coordinates. Fig. 2(d) shows the smoothed result from the raw result Fig. 2(c) in the ROI; (5) Adjust the parametric coordinate from u to $\widehat{u}$ such that ${\widehat{u}}^{\prime }s$ polar coordinates linearly relate to the visual coordinates v, illustrated in Fig. 2(e). During the adjusting, we will ensure the adjusted coordinates still have one-to-one mapping to the cortical surface, namely keeping the topological condition by enforcing $\left|{\mu }_{u\to \widehat{u}}\right|<1$; (6) The adjusting step (5) will break the ideal linearity, so procedure (4)(5) are repeated until the error is within tolerance. The key part of the pipeline is the Error-Tolerant Teichmüller Map (ETTM) in Step (5). The idea to move coordinates from u to the desired location $\widehat{u}$ is by setting landmarks and move accordingly.

Fig. 2.

The pipeline of the iterative parametrization: (a) Select the ROI, i.e. V1/V2/V3 complex and compute a geodesic disk patch contains the ROI; (b) Map the patch (enclosed by the blue curve in the occipital lobe) to the 2D unit disk; (c) A zoomed view of the ROI; (d) Apply Laplacian smoothing on the retinotopic data; (e) Adjust the parametric coordinates by ETTM according to the smoothed data. The ellipse which encloses (d)(e) means the steps are repeated.

Error Tolerant Teichmüller Map.

To move coordinates toward the target, we set some landmarks. Specifically, all the points within the ROI are selected as landmarks. The landmark target is set by ${\widehat{u}}^{(1)}=k_1{\widehat{v}}^{(1)}+b_1$ (similar for u⁽²⁾). Although $\widehat{v}$ is smoothed, the landmarks may still have errors. Previous work, e.g. Teichmüller map (T-map) [22] cannot handle this situation. The Error Tolerant Teichmüller Map (ETTM) is proposed to enhance the T-map to tackle landmarks with errors. The key idea of ETTM is to check the topology condition, $\left|{\mu }_{u\to \widehat{u}}\right|<1$, and seek the most similar alternative parametrization that is topological. ${\mu }_{u\to \widehat{u}}$ is defined as ${\mu }_{u\to \widehat{u}}=\left(\frac{\partial \widehat{u}}{\partial u^{(1)}}+i\frac{\partial \widehat{u}}{\partial u^{(2)}}\right)/\left(\frac{\partial \widehat{u}}{\partial u^{(1)}}-i\frac{\partial \widehat{u}}{\partial u^{(2)}}\right),i=\sqrt{-1}$. The partial derivatives are approximated by piecewise linear interpretation of the discrete values. If the topology is violated, i.e. $\left|{\mu }_{u\to \widehat{u}}\right|>1$, we will shrink the magnitude, ${{\mu }^{\prime }}_{u\to \widehat{u}}=\alpha {\mu }_{u\to \widehat{u}}$ to ensure $\left|{{\mu }^{\prime }}_{u\to \widehat{u}}\right|$ close but less than 1. ${{\mu }^{\prime }}_{u\to \widehat{u}}$ corresponds to the topological map that is closest to the previous non-topological map. Once the proper ${{\mu }^{\prime }}_{u\to \widehat{u}}$ is given, we recovery from ${{\mu }^{\prime }}_{u\to \widehat{u}}$ to $\widehat{u}$ by Linear Beltrami Solver (LBS). We explain the LBS in brief and refer readers to [22] for the details. Denote ${{\mu }^{\prime }}_{u\to \widehat{u}}=\rho +i\tau$, according to the definition, we have $-\frac{\partial {\widehat{u}}^{(1)}}{\partial u^{(2)}}={\alpha }_1\frac{\partial {\widehat{u}}^{(2)}}{\partial u^{(1)}}+{\alpha }_2\frac{\partial {\widehat{u}}^{(2)}}{\partial u^{(2)}}$ and $\frac{\partial {\widehat{u}}^{(1)}}{\partial u^{(1)}}={\alpha }_1\frac{\partial {\widehat{u}}^{(2)}}{\partial u^{(1)}}+{\alpha }_2\frac{\partial {\widehat{u}}^{(2)}}{\partial u^{(2)}}$ , where ${\alpha }_1=\frac{{(\rho -1)}^2+{\tau }^2}{1-{\rho }^2-{\tau }^2},{\alpha }_2=-\frac{2\tau }{1-{\rho }^2-{\tau }^2}$ and ${\alpha }_3=\frac{1+2\rho +{\rho }^2+{\tau }^2}{1-{\rho }^2-{\tau }^2}$. Apply $\frac{\partial }{\partial u^{(1)}}$ on the first equation, and plus $\frac{\partial }{\partial u^{(2)}}$ on the second one, one can write, $\nabla \cdot A\nabla {\widehat{u}}^{(1)}=0$, where $A=\left(\begin{array}{ll}{\alpha }_1\hfill & {\alpha }_2\hfill \\ {\alpha }_2\hfill & {\alpha }_3\hfill \end{array}\right)$. Then one can solve ${\widehat{u}}^{(1)}$ with boundary conditions. Similarly, ${\widehat{u}}^{(2)}$ can be solved. We summarize ETTM in Alg. 1, and the overall procedure in Alg. 2. Pick a point as center and compute the geodesic distance to this point.

3. Dataset and Evaluation Method

3.1. Synthetic Data

The problem of the real dataset is there is no ground truth. We generate a synthetic dataset to compare the performance. Specifically, we take the average of cortical surface (with cortical registration) in the Human Connectome Project (HCP) [20] dataset (only use the anatomical surface), map it to the 2D by orthographical projection. Then we generate the ground truth retinotopic coordinates by the complex-log-model with the formula, $u=k\text{ln}\left({\widehat{v}}^{(1)}{e}^{i{\widehat{v}}^{(2)}}+a\right)$, with k = 1, a = 1. The complex-log-model is a good approximation of the retinotopic mapping introduced by Schwartz [23]. Finally, add noise to $\widehat{v}$, i.e. $v=\widehat{v}+\mathit{\text{noise}}$.

3.2. Evaluation Metric

The evaluation is, given noisy v, which algorithm can recovery best visual coordinates respect to the ground truth $\widehat{v}$, cortical magnification, and the compatibility to neuroscientific results. (1) The visual coordinates error is computed with Euclidean distance; (2) The cortical magnification factor M is defined as the area of corresponding cortical surface A(P) divided by the area of visual field $A(\widehat{v})$ for a small patch σ around the center $\widehat{v}$, namely $M=\underset{\sigma \to 0}{\text{lim}}\frac{A(P)}{A(\widehat{v})}$. In the discrete triangular mesh (F, V), for each point P ∈ V, we select the dual cell for P as the patch σ. Then compute the dual cell area A(P), and its corresponding area in visual field $A(\widehat{v})$. (3) We use the number of non-topological points $\tilde{T}$, i.e. the point moves out of the polygon consisted of neighbors’ points, to quantify the compatibility. The ideal result is no such non-topological points exist, i.e. $\tilde{T}=0$.

3.3. HCP Dataset

The Human Connectome Project (HCP) is a high-quality retinotopy dataset [20]. The data is collected on a modern MRI machine of 7 Tesla magnetitic field with carefully designed visual stimuli. Even so, the result is still noisy. We also applied our method and compare it with harmonic parametrization [24], orthographic projection [12], angle-preserving parametrization [21], area-preserving parametrization [25], and nearly isometric parametrization [26], based on same smoothing setting.

4. Result

4.1. Synthetic data

We first show the method on synthetic data. The proposed method is compared with other parametrization methods with the same smoothing setting. Methods and their performances are listed in Tab. 1. We find: (1) the proposed method can recover the best visual coordinates, evaluate the best cortical magnification factor, and most compatible with neuroscientific results; (2) Angle-preserving and the nearly isometric methods are in the second rank. However, the nearly isometric method is more time consuming; (3) Area-preserving map, harmonic map, and orthographic projection are in the third rank with similar performance.

Table 1.

Compare different methods by three metrics, visual coordinate $\widehat{v}$ difference, the cortical magnification factor M difference relative to the ground truth, and the number of topology violations. The metrics are evaluated for a small noise level (PSNR=20) and a big noise level (PSNR=10) (with the separation symbol ‘/’). “Avg.” is the average difference, and “SD” is the standard deviation of the difference.

Method	$\widehat{v}$ error		M error		$\tilde{T}$	Time/s
	Avg.	SD	Avg.	SD
Raw Data	0.385/0.694	0.225/0.392	27179/27628	13602/13788	645/680	0.0/0.0
Harmonic	0.343/0.616	0.205/0.362	26992/27567	13518/13759	646/685	1.9/1.8
Orthographic	0.346/0.620	0.201/0.355	27022/27580	13550/13770	645/686	2.0/2.0
Angle-Preserving	0.341/0.614	0.205/0.359	26985/27567	13514/13761	650/689	2.3/2.2
Area-Preserving	0.342/0.618	0.198/0.351	27019/27576	13560/13776	649/683	16.2/16.2
Nearly-Isometric	0.339/0.613	0.197/0.353	27010/27576	13567/13774	649/683	32.1/33.5
Proposed	0.142/0.216	0.119/0.136	18728/21422	12291/12629	564/677	7.1/6.4

4.2. HCP data

We apply the proposed algorithm to the HCP data. The raw retinotopic maps of the first subject (in the left hemisphere) is shown in Fig. 3(a). Fig. 3(b)–(c) are for angle-preserving and area-preserving respectively. The proposed method is in Fig. 3(d). The same data are overlaid on the inflated cortical surface in Fig. 3(e)–(h), respectively. Visually, the proposed result is smoother and closer to neuroscientific results.

Fig. 3.

First subject’s visual coordinates contour: (a) The conformal parameterization with the raw retinotopic coordinates; (c) Smoothed on the conformal parameterization; (d) Smoothed based on area-preserving parametrization; (d) Our method; (e)-(h) Results on the inflated surface by same data of (a)-(h)’s, respectively. Adjacent blue lines are drawn with an 5° eccentricity interval, and the black lines are drawn with a 10° polar angle interval.

Although no ground-truth is available, we try to evaluate the results indirectly by two aspects: (1) whether the smoothed result is compatible with the neuropsychology result; and (2) whether the cortical magnification factor (CMF), agrees to public records. For the first subject, we report the violation numbers are: $\tilde{T}=189$ (raw), $\tilde{T}=90$ (harmonic) , $\tilde{T}=101$ (orthographic), $\tilde{T}=85$ (angle-preserving), $\tilde{T}=120$ (area-pre-serving), $\tilde{T}=83$ (nearly-isometric) , and $\tilde{T}=\mathbf{20}$ (proposed). We see the method generated the least number of topology violations, which means the proposed method generates more reasonable results than other methods.

We further estimate CMF by our method. Fig. 4(a)–(c) shows the first three CMF overlaid on the parametric space. The extra benefit of the proposed method is the parametric coordinates have been aligned if the constants k, b are the same for all subjects. To take the constants k, b from the first subject, our method can directly take an average of the parametric domain across the subjects, which is shown in Fig. 4(d). Previous methods [12, 27] can only estimate CMF as a function of eccentricity in the periphery. Contrast to previous work where the CMF is assumed to be symmetric along the polar angle [12], we provide the first quantification of CMF along with different polar angles near the fovea. It shows the CMF is not symmetric, which is compatible to the overall observation that the visual field is compressed vertically [28].

Fig. 4.

(a)-(c): the first three CMF; (d) The average CMF for the first three subjects.