Determining Functional Units of Tongue Motion via Graph-regularized Sparse Non-negative Matrix Factorization

Abstract

Tongue motion during speech and swallowing involves synergies of locally deforming regions, or functional units. Motion clustering during tongue motion can be used to reveal the tongue’s intrinsic functional organization. A novel matrix factorization and clustering method for tissues tracked using tagged magnetic resonance imaging (tMRI) is presented. Functional units are estimated using a graph-regularized sparse non-negative matrix factorization framework, learning latent building blocks and the corresponding weighting map from motion features derived from tissue displacements. Spectral clustering using the weighting map is then performed to determine the coherent regions—i.e., functional units—defined by the tongue motion. Two-dimensional image data is used to verify that the proposed algorithm clusters the different types of images accurately. Three-dimensional tMRI data from five subjects carrying out simple non-speech/speech tasks are analyzed to show how the proposed approach defines a subject/task-specific functional parcellation of the tongue in localized regions.

1 Introduction

The relationship between the structural and functional components of the tongue is poorly understood partly due to the complex tongue anatomy and muscle interactions. The human tongue is a volume preserving structure with highly complex, orthogonally oriented, and interdigitated muscles. The tongue muscles interact with one another in order to carry out the oromotor behaviors of speaking, swallowing, and breathing, which are executed by deforming local functional units in this complex muscular array. Tongue motions are synergies created by locally deforming regions, or functional units [1]. Functional units are regions of the tongue that exhibit homogeneous motion during the execution of the specific task. Therefore, identifying functional units and understanding the mechanisms of coupling among them can identify motor control strategy in both normal and adapted speech (e.g., tongue motion after tongue cancer surgery).

To understand the function of the tongue, magnetic resonance imaging (MRI) has played a pivotal role in imaging both tongue surface motion using cine-MRI [2, 3] and internal tissue motion using tagged-MRI (tMRI) [4]. Despite the rich data on internal tissue motion that is available from tMRI, there has been very little research on its analysis to determine functional units. A key previous report is that of Stone et al. [5] who presented a method to determine functional segments using ultrasound and tMRI. Another key report is that of Ramanarayanan et al. [6], who used a convolutive non-negative matrix factorization (NMF) algorithm to determine tongue movement primitives from electromagnetic articulatory. Our work is inspired by both of these approaches, but we use far richer tMRI-derived data (3D displacement fields) and the NMF approach with the addition of sparsity and intrinsic data geometry in defining functional units.

Modeling a data matrix as sparse linear combinations of basis vectors is a popular approach to understanding speech production. Among them, NMF and variants involving sparsity have received substantial attention since the seminal work by Lee and Seung [7]. NMF is a matrix factorization method that focuses on data matrices whose elements are non-negative. NMF is based on a parts-based representation inspired by psychological and physiological observations about the human brain [8]. However, since standard NMF assumes a standard Euclidean distance measure for its data, it fails to discover the intrinsic geometry of its data [8].

In our work, we assume a manifold of the data within an NMF approach, which thereby captures the intrinsic geometry of the motion features derived from tMRI. In particular, we propose a new approach to determine functional units of tongue motion from tMRI using graph-regularized sparse NMF with spectral clustering. The method integrates a regularization term that encourages the computation of distances on a manifold rather than the whole of Euclidean space in order to preserve the intrinsic geometry of the observed motion data. The use of NMF is important because it does not allow negative combinations of basis vectors. This is consistent with the analysis of muscles, which either have positive activation or no activation, not negative activation. Both quantitative and qualitative evaluation results demonstrate the validity of the proposed method and its superiority to conventional clustering algorithms.

2 Proposed Approach

2.1 Problem Statement

Consider a set of P internal tongue tissue points each with n scalar quantities (e.g., magnitude and angle of each track) tracked through F time frames. These quantities characterize each point and are used to group them into functional units. The location of the p-th tissue point at the f-th time frame can be written as ($x_f^p,y_f^p,z_f^p$). The tongue motion can then be represented by a 3F×P spatio-temporal feature matrix N = [n₁, …, n_P] ∈ $\mathbb{ℝ}$^3F×P , where the p-th column is given by

$${\mathbf{n}}_p={[x_1^p,\cdots ,x_F^p,y_1^p,\cdots ,y_F^p,z_1^p,\cdots ,z_F^p]}^T.$$ (1)

We cast the problem of determining the functional units as a motion clustering problem. Thus, the goal is to determine a permutation of the columns to form [N₁| N₂| … |N_c] , where the submatrix N_i comprises point tracks associated with the i-th submotion—i.e., the i-th functional unit. We provide the proposed approach in more detail. The overall algorithm is shown below.

2.2 Extraction of Motion Quantities

The first step in our algorithm is to extract the motion features that characterize the cohesive motion patterns over time. We extract motion features including the magnitude and angle of the track as in [9] described as

$$m_f^p=\sqrt{{(x_{f+1}^p-x_f^p)}^2+{(y_{f+1}^p-y_f^p)}^2+{(z_{f+1}^p-z_f^p)}^2}$$ (2)

$$c{z}_f^p=\frac{x_{f+1}^p-x_f^p}{\sqrt{{(x_{f+1}^p-x_f^p)}^2+{(y_{f+1}^p-y_f^p)}^2}}+1$$ (3)

$$c{x}_f^p=\frac{y_{f+1}^p-y_f^p}{\sqrt{{(y_{f+1}^p-y_f^p)}^2+{(z_{f+1}^p-z_f^p)}^2}}+1$$ (4)

$$c{y}_f^p=\frac{z_{f+1}^p-z_f^p}{\sqrt{{(z_{f+1}^p-z_f^p)}^2+{(x_{f+1}^p-x_f^p)}^2}}+1$$ (5)

where $m_f^p$ denotes the magnitude of the track and $c{z}_f^p$, $c{x}_f^p$, and $c{y}_f^p$ represent the cosine of the angle projected in the z, x, and y axes plus one, respectively, which are in the range of 0 to 2. For clustering, we gather all the motion features into a 4(F − 1) × P non-negative matrix $\mathbf{U}=[{\mathbf{u}}_1,\dots ,{\mathbf{u}}_n]\in {\mathbb{ℝ}}_{+}^{m\times n}$, where the p-th column can be expressed as

$${\mathbf{u}}_p={[m_1^p.\cdots ,m_{F-1}^p,c{z}_1^p,\cdots ,c{z}_{F-1}^p,c{x}_1^p,\cdots ,c{x}_{F-1}^p,c{y}_1^p,\cdots ,c{y}_{F-1}^p]}^T.$$ (6)

These features are always non-negative and can therefore be input to NMF.

2.3 Graph-regularized Sparse Non-negative Matrix Factorization

NMF

Given a non-negative data matrix U and k ≤ min(m,n), let $\mathbf{V}=\left[v_{ik}\right]\in {\mathbb{ℝ}}_{+}^{m\times n}$ be the building blocks and let $\mathbf{W}=\left[w_{ik}\right]\in {\mathbb{ℝ}}_{+}^{m\times n}$ be the weighting map. The goal of NMF is to learn building blocks and corresponding weights such that the input U is approximated by a product of two non-negative matrices (i.e., U ≈ VW). A typical way to define NMF is to use the Frobenius norm to measure the difference between U and VW [7] given by

$$\epsilon (\mathbf{V},\mathbf{W})=\Vert \mathbf{U}-\mathbf{V}\mathbf{W}{\Vert }_F^2=\sum _{i,j}{(u_{ij}-\sum _{k=1}^K{v}_{ik}{w}_{kj})}^2$$ (7)

where ‖·‖_F denotes the matrix Frobenius norm. The solution can be found through the multiplicative update rule [7]:

$$\mathbf{V}\leftarrow \mathbf{V}.\ast \mathbf{U}{\mathbf{W}}^T.∕\mathbf{V}\mathbf{W}{\mathbf{W}}^T$$ (8)

$$\mathbf{W}\leftarrow \mathbf{W}.\ast {\mathbf{V}}^T\mathbf{U}.∕{\mathbf{V}}^T\mathbf{V}\mathbf{W}$$ (9)

Sparsity Constraint

In this work, we impose a sparsity constraint on the weighting map W. The sparsity constraint allows to encode the high-dimensional motion data using only a few active components, thereby making the weighting map easy to interpret. In particular, the weighting map obtained this way will represent the simplest tongue behavior that could generate the observed motion. In the NMF framework, it has been reported that a fractional regularizer using the L_1/2 norm outperformed the L₁ norm regularizer and gave sparser solutions [10]. Thus, we incorporate the L_1/2 sparsity constraint into the NMF framework, which can be expressed as

$$\epsilon (\mathbf{V},\mathbf{W})=\frac{1}{2}\Vert \mathbf{U}-\mathbf{V}\mathbf{W}{\Vert }_F^2+\eta \Vert \mathbf{W}{\Vert }_{1∕2},$$ (10)

where the parameter η ≥ 0 controls the sparseness of W and ‖W‖_1/2 is defined as

$$\Vert \mathbf{W}{\Vert }_{1∕2}={\left(\sum _{i=1}^k\sum _{j=1}^n{w}_{ij}^{1∕2}\right)}^2.$$ (11)

Manifold Regularization

Many human motions lie on low-dimensional manifolds that are non-Euclidean [11]. NMF with the L_1/2 norm sparsity constraint, however, produces a weighting map based on a Euclidean structure in the high-dimensional data space. Thus, the intrinsic and geometric relation between motion features may not be reflected accurately. To remedy this, we incorporate a manifold regularization that respects the intrinsic geometric structure as in [8,12,13]. The manifold regularization favors the local geometric structure and also serves as a smoothness operator, which reduces the interference of noise. Our final objective function incorporating both the manifold regularization and the sparsity constraint can then be given by

$$\epsilon (\mathbf{V},\mathbf{W})=\frac{1}{2}\Vert \mathbf{U}-\mathbf{V}\mathbf{W}{\Vert }_F^2+\frac{1}{2}\lambda \mathrm{Tr}\left(\mathbf{W}\mathbf{L}{\mathbf{W}}^T\right)=\eta \Vert \mathbf{W}{\Vert }_{1∕2},$$ (12)

where λ is a balancing parameter of the manifold regularization, Tr(·) denotes the trace of a matrix, Q is a heat kernel weighting, D is a diagonal matrix where ${\mathbf{D}}_{jj}={\sum }_l{\mathbf{Q}}_{jl}$, and L = D − Q, which is the graph Laplacian.

Minimization

The objective function in Eq. (12) is not convex in both V and W and therefore it is not possible to find the global minima. In order to minimize the objective function, we use a multiplicative iterative method similiar to that used in [13]. Let Ψ = [ψ_mk] and Φ = [φ_kn] be the Lagrange multiplier subject to v_mk ≥ 0 and w_kn ≥ 0, respectively. By using the definition of the Frobenius norm, ‖U‖_F = (Tr(U^TU))^1/2, and matrix calculus, the Lagrangian ℒ is given by

$$\begin{array}{c}\hfill ℒ=\frac{1}{2}\mathrm{Tr}\left(\mathbf{U}{\mathbf{U}}^T\right)-\mathrm{Tr}\left(\mathbf{U}{\mathbf{W}}^T{\mathbf{V}}^T\right)+\frac{1}{2}\mathrm{Tr}\left(\mathbf{V}\mathbf{W}{\mathbf{W}}^T{\mathbf{V}}^T\right)\hfill \\ \hfill +\frac{\lambda }{2}\mathrm{Tr}\left(\mathbf{W}\mathbf{L}{\mathbf{W}}^T\right)+\mathrm{Tr}(\Psi {\mathbf{V}}^T)+\mathrm{Tr}(\Phi {\mathbf{W}}^T)+\eta \Vert \mathbf{W}{\Vert }_{1∕2}.\hfill \end{array}$$ (13)

The partial derivatives of ℒ with respect to U and V are

$$\begin{array}{cc}\hfill & \frac{\partial ℒ}{\partial \mathbf{V}}=-\mathbf{U}{\mathbf{W}}^T+\mathbf{V}\mathbf{W}{\mathbf{W}}^T+\Psi \hfill \\ \hfill & \frac{\partial ℒ}{\partial \mathbf{W}}=-{\mathbf{V}}^T\mathbf{U}+{\mathbf{V}}^T\mathbf{V}\mathbf{W}+\lambda \mathbf{W}\mathbf{L}+\frac{\eta }{2}{\mathbf{W}}^{-1∕2}+\Phi \hfill \end{array}$$ (14)

By using Karush-Kuhn-Tucker conditions—i.e., Ψ_mkV_mk = 0 and Φ_knW_kn = 0—the final update rule becomes

$$\begin{array}{cc}\hfill & \mathbf{V}\leftarrow \mathbf{V}.\ast \mathbf{U}{\mathbf{W}}^T.∕\mathbf{V}\mathbf{W}{\mathbf{W}}^T\hfill \\ \hfill & \mathbf{W}\leftarrow \mathbf{W}.\ast ({\mathbf{V}}^T\mathbf{U}+\lambda \mathbf{W}\mathbf{Q}).∕({\mathbf{V}}^T\mathbf{V}\mathbf{W}+\frac{\eta }{2}{\mathbf{W}}^{-1∕2}+\lambda \mathbf{W}\mathbf{D}).\hfill \end{array}$$ (15)

2.4 Spectral Clustering

The non-negative weighting map provides a good measure of regional tissue point similarity. Thus spectral clustering using the weighting map is adopted to determine the cohesive motion patterns as spectral clustering outperforms traditional clustering algorithms such as the K-means algorithm [14].

Once W is determined from Eq. (15), an affinity matrix A is constructed:

$$\mathbf{A}(i,j)=\exp (-\frac{\Vert w\left(i\right)-w\left(j\right){\Vert }_2}{\sigma }),$$ (16)

where w(i) is the i-th column vector of W and σ denotes the scale (we set σ = 0.02 in this work). The column vectors of W form nodes in the graph, and the similarity A computed between column vectors of W form the edge weights. On the affinity matrix, we apply a spectral clustering technique using a normalized cut algorithm [15].

3 Experimental Results

3.1 Experiments Using 2D Data

Since there is no ground truth in our in vivo data, we used two 2D datasets to demonstrate the clustering performance of the proposed method. The first dataset is the COIL20 image library, which contains 20 classes (32×32 gray scale images of 20 objects). The second dataset is the CMU PIE face database, which has 68 classes (32×32 gray scale face images of 68 persons). In order to compare the performance of the different algorithms, we used a K-means clustering method (K-means), a normalized cut method (N-Cut) [15], standard NMF with K-means clustering (NMF-K), graph-regularized NMF with K-means clustering (G-NMF-K) [8], graph-regularized NMF with spectral clustering (G-NMF-S), graph-regularized sparse NMF with K-means clustering (GS-NMF-K), and our method (GS-NMF-S). Two metrics, the Normalized Mutual Information (NMI) and the accuracy (AC), were used to measure the clustering performance as used in [8]. Table 1 lists the NMI and AC values, demonstrating that the proposed method outperformed other methods. We also compared the L1/2 and L1 norms experimentally, and the L1/2 norm had slightly better results.

Table 1.

Clustering Performance: NMI and AC

NMI (%)	K-means	N-Cut	NMF-K	G-NMF-K	GS-NMF-K	G-NMF-S	Ours
COIL20 (K=20)	73.80%	76.56%	74.36%	87.59%	90.11%	90.24%	90.63%
PIE (K=68)	54.40%	77.13%	69.82%	89.93%	89.95%	90.95%	91.74%
AC (%)	K-means	N-Cut	NMF-K	G-NMF-K	GS-NMF-K	G-NMF-S	Ours
COIL20 (K=20)	60.48%	66.52%	66.73%	72.22%	83.75%	84.58%	85.00%
PIE (K=68)	23.91%	65.91%	66.21%	79.3%	79.93%	80.60%	84.31%

3.2 Experiments Using In Vivo Tongue Data

We also tested our method using a simple non-speech protrusion task and a speech task: “a souk”. Four subjects said “a souk” and one subject performed the protrusion task. All MRI scanning was performed on a Siemens 3.0 T Tim Treo system (Siemens Medical Solutions, Malvern, PA) with 12-channel head and 4-channel neck coil. The tMRI datasets were collected using Magnitude Imaged CSPAMM Reconstructed images [16]. The datasets had a 1 second duration, 26 time-frames with a temporal resolution of 36 ms for each phase with no delay from the tagging pulse, 6 mm thick slices (6 mm tag separation), 1.875 mm in-plane resolution with no gap. The field-of-view was 24 cm. We acquired 2D orthogonal stacks of tMRI and used harmonic phase (HARP) to track internal tissue points. The incompressible deformation estimation algorithm was then used to combine 2D tracking data to produce the 3D tracking result with an incompressibility constraint [17].

Fig. 1 shows the protrusion task. The outer tongue layer expands forward and upward (but not backward), and the region near the jaw has little motion. Functional units, based on magnitude and angle, have been extracted for two (Fig. 1(b)) and three clusters (Fig. 1(c)). Fig. 1(b) is a good representation of forward protrusion (blue) vs no motion (red), but the three cluster output introduces noise, suggesting there are only two clusters, or functional units.

Fig. 1.

Illustration of protrusion showing (a) 3D displacement field, (b) functional units (2 clusters), (c) functional units (3 clusters)

Fig. 2 shows the motion from /s/ to /u/ during the word “a souk”. The functional units were determined using our method for two clusters (Fig. 2(b)) and three clusters (Fig. 2(c)), respectively. Note that the three clusters better represent the motions of the tongue. These motions include backward motion of the tongue tip (blue), upward motion of the tongue body (green), and forward motion of the posterior tongue (red).

Fig. 2.

Illustration of motion from /s/ to /u/ showing (a) 3D displacement field, (b) functional units (2 clusters), (c) functional units (3 clusters)

4 Discussion and Conclusion

In this work, inspired by recent advances in sparse NMF and manifold learning, we presented a novel method for determining functional units from tMRI. Unlike previous algorithms, this proposed work aims at identifying the internal, coherent manifold structure of high-dimensional motion data to determine functional units. The contributions of this work are two-fold. In an NMF framework, we formulate a new clustering problem, that of a learning latent weighting map as well as spectral clustering and we give an efficient algorithm to solve this problem. Our method performed better than K-means, N-Cut, NMF-K, G-NMF-K, GS-NMF-K, and G-NMF-S using 2D data. In a tongue motion analysis context, we define functional units from tMRI, which opens new vistas to study speech production. The identified functional units are visually assessed and further studies using biomechanical stimulations are needed to co-validate our findings due to the lack of ground truth in in vivo data. The proposed method gives a principled method for defining subject/task-specific functional units, which can be potentially used to elucidate speech-related disorders.