Differentiating post-cancer from healthy tongue muscle coordination patterns during speech using deep learning

Abstract

The ability to differentiate post-cancer from healthy tongue muscle coordination patterns is necessary for the advancement of speech motor control theories and for the development of therapeutic and rehabilitative strategies. A deep learning approach is presented to classify two groups using muscle coordination patterns from magnetic resonance imaging (MRI). The proposed method uses tagged-MRI to track the tongue's internal tissue points and atlas-driven non-negative matrix factorization to reduce the dimensionality of the deformation fields. A convolutional neural network is applied to the classification task yielding an accuracy of 96.90%, offering the potential to the development of therapeutic or rehabilitative strategies in speech-related disorders.

1. Introduction

The human tongue is a highly complex muscular structure composed of several paired intrinsic and extrinsic muscles that are critical for speaking, swallowing, and breathing (Kier and Smith, 1985). In order to produce intelligible speech, a variety of internal tongue muscle groupings, i.e., functional units, are temporarily formed, rapidly and nimbly, in a highly coordinated fashion (Green and Wang, 2003; Ramanarayanan et al., 2013; Stone et al., 2004; Woo et al., 2019). For individuals who have difficulty with speech production following tongue cancer, the location and size of functional units obtained from the same speech movements are likely to be different from those of healthy controls because of surgical modifications or compensatory strategies used to maximize speech intelligibility (Stone et al., 2014). In addition, the extent to which tongue function for speech is impaired due to resected muscle groups as in tongue cancer (Goldsmith and Jacobson, 2015) varies depending on the location and size of tumors resected, which leads to a large variability in the local muscle coordination patterns during speech. Due to the large variability and complexity of tongue motion during speech for both healthy controls and patients, it remains challenging to detect and characterize the similarities and differences of functional units between controls and patients. Differentiating post-cancer from healthy tongue muscle coordination patterns is, however, an important part of the development and verification of speech motor control theories and in the development of new therapeutic, surgical, or rehabilitative strategies.

In recent years, machine/deep learning has produced state-of-the-art results in a large number of tasks such as recognition, prediction, and classification from complex sets of data (LeCun et al., 2015). In the context of tongue motion analysis for speech using various imaging and motion capture techniques, several researchers have studied the classification of speech movements using machine learning. For example, Tang et al. (2011) investigated spatiotemporal gestural descriptors from ultrasound for classifying speech movements using a support vector machine (SVM). Wang et al. (2016) examined a set of flesh points on the tongue and lips that are optimal to classify speech movements via a SVM from electromagnetic articulographs. To the best of our knowledge, no attempts have been made for the classification task using tagged-magnetic resonance imaging (MRI) previously. For an extensive review on scientific and clinical applications of speech movement analysis using machine learning, readers can refer to the paper by Green (2015) and the references therein.

In this work, we propose a deep learning classification framework to differentiate post-cancer from healthy motion coordination patterns after tongue cancer treatment using tagged-MRI (Parthasarathy et al., 2007) and voxel level tracking (Xing et al., 2017) for the first time. To achieve this goal and mitigate the aforementioned challenges, we use a four-dimensional (4D) (three-dimensional space with time) atlas of tongue motion from tagged-MRI (Woo et al., 2017) and atlas-driven non-negative matrix factorization (NMF) (Woo et al., 2019), to characterize the standardized muscle coordination patterns, and a deep learning framework to build an accurate and interpretable classification framework. The use of a 4D atlas of tongue motion in combination with NMF is crucial because it provides a benchmark to compare differences of internal muscle coordination patterns in an objective manner. We show that this classification occurs within the convolutional neural network (CNN) with the implicit yet interpretable structures of muscle coordination patterns derived from atlas-driven NMF.

2. Materials and methods

2.1. Subjects and MRI acquisition

A total of 26 subjects including 18 healthy controls and 8 tongue cancer patients after treatment participated in this study. Each subject was trained prior to the scan to speak a simple utterance (“ə-suk”) in synchrony with a repetitive (metronome-like) sound. During the scan, the subjects repeated the utterance following the repetitive sound while MRI scans were performed. Both T2-weighted two-dimensional (2D) dynamic cine-MRI and tagged-MRI images were acquired using a Siemens 3.0 T Tim Trio system with a 16-channel head and neck coil (Siemens Medical Solutions, Erlangen, Germany). These dynamic acquisitions were acquired 3 times each—in axial, coronal, and sagittal orientations—at 26 frames per second for a duration of 1 s on identical slice positions. Table 1 shows the characteristics of the patients including speech intelligibility, tumor size, and closure type.

Table 1.

Characteristics of patients.

Subjects	Speech Intelligibility	Tumor Size	Closure Type
Patients 1–5	100%	T1	Primary
Patient 6	99%	T2	Radial forearm free flap
Patient 7	95%	T2	Primary
Patient 8	98%	T2	Primary

2.2. Our method

Tissue point tracking from tagged-MRI. In order to track each voxel within the tongue from tagged-MRI, we use the previously developed method called phase vector incompressible registration algorithm (PVIRA) (Xing et al., 2017). First, cubic B-spline interpolation is used to generate dense 2D slices from the sparsely acquired tagged-MRI slices. All three orientations (axial, coronal, and sagittal acquisitions) are interpolated onto the same digital grid. A harmonic phase filter (Osman et al., 1999) is then used to yield three harmonic phase (HARP) volumes from the interpolated results. Finally, the HARP phase volumes are input into an iLogDemons algorithm (Mansi et al., 2011) (modified to accept three phase volumes) to find accurate correspondences across time frames. This tracking method yields accurate and incompressible motion fields that capture the dynamic motion of the tongue during speech.

4D atlas building of tongue motion. We build a Lagrangian 4D atlas of tongue motion (ə-suk) from 18 healthy controls as follows. First, we use a super-resolution volume reconstruction technique (Woo et al., 2012) to combine three orthogonal stacks of cine images, yielding 26 volumes with isotropic resolution for each subject. Second, we establish a common Lagrangian coordinate system across subjects by using the first time frame (representing a neutral position) of the 18 cine super-resolved volumes. This coordinate system is found using a symmetric diffeomorphic groupwise registration with a cross-correlation similarity measure (Avants et al., 2008). Third, we transform the motion fields computed by PVIRA into this common coordinate system, thus putting them into correspondence. Finally, we average all the motion fields registered at each voxel to yield the final 4D Lagrangian atlas.

Weighting map estimation from atlas-driven NMF. Once we obtain the 4D atlas in the common spatial coordinate system, we extract motion features from 26 time frames including the magnitude and angle of point tracks (Woo et al., 2019). Those features are aggregated to form an input matrix for graph-regularized sparse non-negative matrix factorization (GS-NMF). The GS-NMF produces two output matrices: (1) spatial building blocks reflecting muscle groups, which changes over time, and (2) the associated weighting map. The weighting map is then used to identify the coherent regions, thus revealing spatiotemporally varying functional units (Woo et al., 2019). The 4D motion atlas of tongue motion using the same speech task in a healthy population represents standardized motion patterns. Therefore, the building blocks and their weighting maps identified from the 4D atlas motion can be seen as standardized building blocks with weighting maps that can generate the standardized motion patterns. More specifically, let U_A be a non-negative feature matrix constructed by the motion features from the 4D atlas as shown in Fig. 1, where K ≤ min(m, n), and let ${\mathbf{V}}_A=[v_{\mathit{i}\mathit{k}}]\in {ℝ}_{+}^{m\times K}$ be the building blocks and ${\mathbf{W}}_A=[w_{\mathit{k}\mathit{j}}]\in {ℝ}_{+}^{K\times n}$ be their weighting map, respectively. The goal of GS-NMF is to find two matrices while maintaining intrinsic and geometric relationship between features given by

$${\mathbf{U}}_A\approx {\mathbf{V}}_A{\mathbf{W}}_A.$$ (1)

Fig. 1.

(Color online) A schematic illustration of obtaining weighting maps used for the classification task. Once we obtain the building blocks from the 4D motion atlas, we fix the building blocks to obtain objective weighting maps across controls and patients. The obtained objective weighting maps are used for the subsequent classification task.

Because the building blocks obtained from the 4D atlas and GS-NMF can be interpreted as the standardized aggregated information that defines the muscle group types given the speech task, the weighting map provides the tissue coordinate location and weights of each muscle group type to generate the phonemic movements that produce the speech. We then compute weighting maps for each subject by using the same building blocks from the 4D atlas so that the obtained weighting maps provide objective weights, i.e., a proxy structure containing implicit muscle coordination patterns, associated with the standardized muscle groups given by

$${\mathbf{U}}_{S\to A}\approx {\mathbf{V}}_A{\mathbf{W}}_{S\to A},$$ (2)

where U_S→A represents an input feature matrix derived in the atlas space for each subject and W_S→A represents the obtained weighting map, while V_A remains fixed. This facilitates the objective comparison of muscle coordination patterns across controls and patients. We obtain the weighting maps using different combinations of phonemes including (1) /ə/ - /s/, (2) /ə/ - /u/, (3) /ə/ - /k/, (4) /s/ - /u/, (5) /s/ - /k/, and (6) /u/ - /k/, each subject having six unique weighting maps.

Deep learning-based classification. The numbers of the weighting maps estimated for controls and patients are 108 and 48, respectively. With this limited training data, it is likely that many of the complicated relationships could be attributed to sampling noise and therefore they will exist in the training data only but not in test data, which thus leads to overfitting. To address this, we perform data augmentation on weighting maps using combinations of scaling and rotation, thereby yielding 936 weighting maps in total (648 weighting maps for controls and 288 weighting maps for patients).

The original and augmented weighting maps are then input into a CNN classification system. The architecture of our CNN classification system is summarized in Fig. 2. The architecture contains six learned layers—five convolutional and one fully-connected. We follow each convolutional layer with a ReLU activation function, a batch normalization layer, and a max-pooling layer, except for the fully-connected layer where a softmax activation function is used. Dropout (Hinton et al., 2012) is not used, following the suggestion in Ioffe and Szegedy (2015), and for optimization we use Adam optimization with the default learning rate. We use binary cross-entropy as the loss function. Our CNN is trained on an GTX 1080 Ti GPU (NVIDIA, Santa Clara, CA) using 936 weighting maps from controls and patients. Our implementation is based on Keras with Tensorflow as a backend.

Fig. 2.

(Color online) A schematic illustration of our deep learning architecture.

3. Validation

Our method was compared to three other classification methods including a SVM with a radial basis function kernel, a random forest (RF), and a multilayer perceptron (MLP). All three were optimized independently using the same training data that we used to train our CNN in order to achieve the best classification performance. To measure the classification accuracy, we used five-fold cross-validation. The results using our approach and other competing approaches were statistically compared using the Student t-test with a level of significance set at p < 0.05.

4. Results

Example motion fields from our 4D atlas of ə-suk are shown in the top row of Fig. 3 and three clusters of example functional units for the transitions /ə/ - /s/, /s/ - /u/, and /u/ - /k/ are shown in the bottom row of Fig. 3. The bottom row shows the location of three functional units that reflect and characterize the averaged motions in the top row. The motion from /ə/ to /s/ includes minimal motion in the tongue base, forward motion in the posterior tongue, and forward/upward motion in the tip and central body. The functional units capture these three motion regions and create clear borders between them. The /s/ to /u/ motion shows functional divisions between the anterior and posterior tongue. The movement regions in the posterior are not well delineated, suggesting integrated regions of motion in the back that differ from the one in the front. The functional units for the /u/ to /k/ motion capture three different functional units, showing the upper tongue divided into an anterior and posterior region, each moving toward the center as the tongue body is elevated.

Fig. 3.

(Color online) Illustration of the motion fields of a 4D motion atlas, ə-suk, in a Lagrangian coordinate system (top row), and identified functional units (bottom row). The motion fields and the functional unit results are depicted relative to the neutral tongue position. The cone size is the magnitude of the motion fields, and red, green, and blue colors are left–right, front–back, and up–down directions of the motion fields in the top row, respectively. Of note, the colors in the bottom row represent different units, not directions.

Figure 4 illustrates the motion fields, functional units, and associated weighting maps for the transitions /ə/ - /s/, /s/ - /u/, and /u/ - /k/ for a control and a patient (patient 4). The second and third row display the location of three functional units and associated weighting maps that reflect and characterize the averaged motions in the first row, respectively. For the weighting maps, each axis represents a different parameter. The left axis represents spatial locations within the tongue, the right axis represents reduced motion features (in this work, the number was set to 20), and the height axis represents the magnitude of weights. For the control, similar to Fig. 3, the motion from /ə/ to /s/ includes motion in the tongue base, forward motion in the posterior tongue, and forward/upward motion in the tip. The /s/ to /u/ motion shows functional divisions between the anterior and posterior tongue. The functional units for the /u/ to /k/ motion show the upper tongue divided into an anterior and posterior region. For the patient, the motion from /ə/ to /s/ includes motion in the tongue base, forward motion in the posterior tongue, and forward/upward motion in the tip and lower body. The /s/ to /u/ motion shows functional divisions among the tip, posterior region, and base of the tongue. The functional units for the /u/ to /k/ motion capture three different functional units, showing the tip, base, and body of the tongue. As shown in Fig. 4, it can be clearly observed that the tongue's tip and base of the patient have a tendency to act as separate functional units over time. However, for the control, at certain time periods (such as from /s/ to /u/), the tongue's tip and base were identified as one unified unit. This could be contributed by the fact that the control's internal tongue muscles are more active than the patient so that they could cooperate more freely, even connecting the tip and base. However, the patient appeared to lose that capability for a broader muscle connection due to the surgical removal of tumors.

Fig. 4.

(Color online) Illustration of the motion fields (first row), three clusters of functional units (second row), and associated weighting maps (third row) for the transitions /ə/ - /s/, /s/ - /u/, and /u/ - /k/ in a Lagrangian coordinate system for a control and a patient.

We validated the effectiveness of our approach using five-fold cross-validation. In short, we first divided the whole sample into five disjoint subsets of approximately equal size. We then trained each model 5 times in which for each run, four of the subsets were used for training, while leaving out one of the subsets from the training, and using only the left out subset to compute the predictive ability of the model. Accuracy was measured by how effectively we could classify motion patterns into healthy versus patient groups. Our approach achieved an accuracy of 96.90 ± 2.12%, while SVM, RF, and MLP achieved an accuracy of 83.33 ± 1.92%, 84.40 ± 2.66%, and 84.72 ± 2.90%, respectively. The Student's t-test results demonstrate that there is a statistically significant difference (p < 0.05) in the classification accuracy between our method and each competing method.

5. Discussion and summary

In this work, we presented a method to discriminate tongue motion patterns during speech in tongue cancer patients and healthy controls from tagged-MRI via a deep learning classification approach. To overcome the greater complexity and variability inherent in internal tongue muscle coordination patterns, a 4D atlas from tagged-MRI and atlas-driven NMF was used to yield a low-dimensional yet interpretable and referential structure to compare the motion patterns of the tongue between patients with tongue cancer and healthy controls. Our results show that our method achieves over 95% classification accuracy and is statistically superior to competing approaches. Upon analysis of the different speech mechanisms revealed by our approach, we may better understand the underlying mechanisms of muscle coordination patterns and potentially contributing to the development of therapeutic, rehabilitative, and surgical strategies.

In order to learn physiologically interpretable feature representation as an input to our classification model instead of a variety of individual motion features, an NMF-based approach (Woo et al., 2019) was used for dimension reduction while retaining interpretability through weighting maps which implicitly contain muscle coordination patterns. This is partly attributed to the fact that NMF offers parts-based representation of motion features resulting from muscle activity that are inherently non-negative. Of note, it is, however, difficult to visually assess the number of clusters, the locations and sizes of the function units directly from the weighting maps since the weighting maps are low-dimensional structures. Also due to the high variability in the identified functional units, it is challenging to directly compare and contrast the weighting maps between the control and the patient as shown in Fig. 4. It is, therefore, necessary to use computerized algorithms including the spectral clustering algorithm and deep learning to identify the size and location of the functional units and classify between controls and patients, respectively. This is in contrast with other dimension reduction methods such as Principal Component Analysis (PCA) (Xing et al., 2016) that is used as a versatile tool for dimension reduction and visualization and Independent Component Analysis (ICA) that is used for separation of the data into a set of statistically independent components. The decomposition results derived from PCA and ICA are, however, limited to orthogonal and independent basis vectors, thereby lacking physiological interpretability of the underlying data.

In the present work, we employed a Lagrangian configuration as in Woo et al. (2017) based on material coordinates. This Lagrangian coordinate system in conjunction with the 4D atlas greatly simplified all the computations of motion fields, weighting maps, and subsequent comparison using classifiers. The improved classification accuracy can, in part, be attributed to the minimization of across-subject difference in tongue motor function, which presumably allowed for the identification of common patterns of tongue muscle coordination within each participant group. The Lagrangian coordinate system used to simplify tongue motion data analysis is analogous with the coordinate system previously established in other organs including the “motionless movies” concept (Wedeen et al., 1995) to analyze myocardial strain-rates of the heart or referential structures of the brain (Wedeen et al., 2012) to map multimodal and multiscale data to characterize brain structure and function at different scales from a population of interest.

Although the patient population studied in this work was limited to patients who underwent a partial glossectomy, we anticipate that the novel methods used to identify and characterize tongue impairment will be relevant to a large number of individuals with anatomic and neurologically-based speech impairments such as Amyotrophic Lateral Sclerosis (ALS). For example, the extent to which tongue function is impaired due to muscle weakness as in ALS varies depending on the severity of the disease, which also leads to large variability in the local muscle coordination patterns. Likewise, our proposed method could be applied to the ALS population for characterizing functional units and subsequently distinguishing between persons with ALS and healthy controls.

There are a few ways to expand on the present work. First, deep learning methods are typically trained on a large number of datasets and deep learning architectures have many layers. However, the present work is limited by the number of subjects used for training and testing. In future work, we will increase the number of subjects to build our model and test our algorithm. In addition, we will stratify the patient group based on the location and size of tumors resected for tongue cancer patients, or different disease states of ALS. This more granular participant stratification will allow us to investigate the multi-class categorization problem. Second, in this work, we used one speech word to build the model and to test our algorithm. In future work, we will also diversify speech words to establish a larger database and further build training models for various machine learning tasks such as classification, recognition, and prediction. Third, we only used motion quantities including magnitude and angle derived from displacements. In future work, we will use principal strains to investigate various cause-and-effect relationships among muscle groups (Buchaillard et al., 2008).