A fast and flexible MRI system for the study of dynamic vocal tract shaping

Abstract

Purpose

To develop and evaluate a MRI-based system for study of dynamic vocal tract shaping during speech production, which provides high spatial and temporal resolution.

Methods

The proposed system utilizes (a) custom eight-channel upper airway coils that have high sensitivity to upper airway regions of interest, (b) 2D golden angle spiral gradient echo acquisition, (c) on-the-fly view-sharing reconstruction, and (d) off-line temporal finite difference constrained reconstruction. The system also provides simultaneous noise-cancelled and temporally aligned audio. The system is evaluated in three healthy volunteers, and one tongue cancer patient, with a broad range of speech tasks.

Results

We report spatio-temporal resolutions of 2.4×2.4 mm² every 12ms for single-slice imaging, and 2.4×2.4 mm² every 36 ms for three-slice imaging, which reflects roughly 7-fold acceleration over Nyquist sampling. This system demonstrates improved temporal fidelity in capturing rapid vocal tract shaping for tasks such as producing consonant clusters in speech, and beat-boxing sounds. Novel acoustic-articulatory analysis was also demonstrated.

Conclusions

A synergistic combination of custom coils, spiral acquisitions, and constrained reconstruction enables visualization of rapid speech with high spatio-temporal resolution in multiple-planes.

Introduction

Speech production involves complex spatio-temporal coordination of several vocal organs in the upper and lower airways. Modalities to study speech production include real-time MRI (RT-MRI), electro-magnetic articulography (EMA), electropalatography (EPG), ultrasound, and X-ray, or video-fluoroscopy (1). In comparison to alternate modalities, (RT-MRI) provides distinct advantages in terms of (a) non-invasiveness, as opposed to X-rays, video-fluoroscopy, and (b) ability to image in arbitrary planes and visualize deep structures (eg. epiglottis, glottis), which are not possible with ultrasound and EMA. Applications of RT-MRI in speech science and vocal production research are numerous; these include addressing open questions pertaining to understanding the goals of language production, language timing, speech errors, and other topics in phonetics and phonology (1-8) as well as vocal production of song (9-10). It also has the potential to manage and inform treatment plans in several clinical applications, such as clinical assessment of velo-pharyngeal insufficiency (11-13), cleft palate repair and management (14-15) and surgical planning and post-treatment functional evaluation of speech and swallowing (16-17) in head and neck cancer.

The rates of movements of articulators are highly dependent on the speech task, and the subject's speaking style (3,18-19). For example, in the articulation of sustained sounds such as during singing, the spatial position of the articulators change on the order of seconds, while in tasks involving flaps/trills, production of consonant clusters, the motion of articulators occur at a much faster rate on the order of few ms (also see Figure 1) (3). While modalities such as EMA can operate up-to time resolution of 1ms, the imaging speed of RT-MRI is restricted by challenges posed due to device physics.

Figure 1.

Spatial versus temporal resolutions trade-offs in RT-MRI using short interleaved spiral trajectories. In comparison to full sampling, sparse sampling reduces spatio-temporal trade-offs and enables improved visualization of several speech tasks both in single plane and multi-plane imaging. The clouds in the above figure represent a recently reported consensus opinion among speech imaging researchers and linguists (3).

Several schemes have been proposed to improve the imaging speed of RT-MRI for speech studies. These can be classified as on-the-fly or off-line schemes (3). On-the-fly schemes are referred to those that allow for immediate visualization of the reconstructed images (with latency less than 500ms), while off-line schemes are referred to those where the reconstructions are implemented off-line. (20) utilized on-the-fly imaging with Cartesian trajectories, and demonstrated temporal resolutions between (111ms-50ms) at spatial resolution of (1.6-2.7mm²) for velo-pharyngeal closure. (1,21-22) utilized short spiral readouts to acquire images at a native time resolution of (54-78 ms), and a spatial resolution of (3-2.4mm²), and visualize at 24 frames/s using view-sharing. View-sharing was also used with radial imaging in (23-24). (25) compares Cartesian, radial, spiral trajectories with view-sharing, and demonstrated spirals to provide the best compromise in terms acquisition efficiency, motion robustness, and signal to noise.

Iterative constrained reconstruction schemes have shown to enable greater acceleration factors. Utilizing radial trajectories, (23) proposed non-linear temporal regularized reconstruction to enable a temporal resolution of 33 ms, and a spatial resolution of 1.5mm², and studied a variety of tasks (vowel, consonant sounds, co-articulations events). More recently, (26) demonstrated an on-the-fly implementation of the iterative reconstruction by exploiting parallelization within the reconstruction along with efficient use of graphical processing units. This work used a 64-channel brain coil and demonstrated 100 frames/sec at 1.5 mm². (24) applied off-line spatial total-variation regularization with radial trajectories to provide spatial resolution of 1.8mm², and a native time resolution of 55ms, and analyzed morphometric measurements of the vocal tract. (27) utilized off-line reconstruction with the partial separable (PS) model (28-29) along with spatial-spectral sparsity constraint (30), and demonstrated a frame rate of 102 frames/sec at a spatial resolution of 2.2mm². The low-rank (via the PS model) constraint is essentially a data-driven retrospective binning technique (31), and was fully exploited in (27) by utilizing repeated speech utterances. Low rank constraints are unlikely to apply to short speech utterances with no repetitions, or stimuli involving infrequent distinct movements such as swallowing.

In this work, we develop a MRI-based system for dynamic study of vocal tract shaping during speech production to provide high spatio-temporal resolutions. We propose a system that utilizes (a) a novel eight-channel custom upper-airway coil which has improved sensitivity in upper-airway regions of interest, (b) a flexible slice selective spiral spoiled gradient echo acquisition with golden angle time interleaving, (c) on-the-fly view-sharing reconstruction, (d) off-line temporal finite difference constrained reconstruction, (e) simultaneous audio acquisition, and off-line temporal alignment of noise-cancelled audio with the reconstructions. The innovation of our system lies in the synergistic combination of the above components. We chose custom upper-airway coil design for its ability to provide superior signal to noise (SNR) across all articulators of interest. This is advantageous as it can provide an important boost in SNR while operating at 1.5 T. Its combination with spirals is complementary, as it enables improved SNR at low fields, low off-resonance related spiral artifacts, and high sampling efficiency. Multi-shot short spirals (read out length of 2.4ms) are used to reduce off-resonance artifacts. Our rationale of choosing spirals is that they have shown to provide improved motion robustness and efficiency over alternate trajectories. Temporal finite difference constraint was chosen as it exploits redundancy based on the prior that the desired information is contained in the moving edges, which directly fits to the end goal assessment of dynamics of air-tissue interfaces. Also, since it exploits similarities amongst neighboring time frames, it is applicable to a wide variety of speech tasks, and does not impose restrictions on the imaging task.

We present several examples with the proposed system for rapid RT-MRI of speech including visualization of interleaved consonant and vowel sounds, fluent speech sentences that contain consonant clusters, as well as beat-boxing sounds that involve rapid coordination between various vocal articulators, on three healthy volunteers, and one tongue cancer patient referred to glossectomy.

Methods

Simulation of spatial v/s time resolution trade-offs with spiral sampling

A multi-shot short spiral readout spoiled gradient echo pulse sequence (flip angle: 15 degrees, slice thickness: 6mm, readout time: 2.5ms, TR=6.004ms), which was used in our previous studies, was adapted in this study (32-33). We chose spiral trajectories over alternate trajectories as they have shown to provide a superior tradeoff amongst spatial resolution, time resolution, and robustness to motion artifacts. The spiral trajectories were designed to make maximum use of gradients (40 mT/m maximum gradient amplitude and 150 mT/m/ms slew rate). Simulations were performed to investigate the spatial versus time resolution trade-offs for a field of view (FOV) of 20 cm² at Nyquist (full) sampling, and rate 6.5 fold under-sampling for single slice and concurrent three-slice imaging (Figure 1). Nyquist sampling was determined by varying the number of spiral interleaves such that the maximum spacing between interleaves equaled the reciprocal of the unaliased FOV. The Nyquist definition was defined based on the assumption of spiral interleaving with a uniform angle distribution. Figure 1 also duplicates the schematic placement of various speech tasks according to their spatial and temporal resolution requirements, as reported in (3).

Single slice, and multi-slice golden angle spiral time interleaving

A previously proposed golden angle time interleaved sampling pattern scheme in which successive spiral interleaves are separated by the golden angle 2π × 2/(√5+1) was adapted in this work (32). The sampling schedule was repeated after 144 interleaves. Two single slice sequences with spatial resolutions of 2.4 mm² and 1.76 mm² were realized, which respectively corresponded to 13 and 21 spiral interleaves/frame for Nyquist sampling. A flexible multi-slice time interleaved sequence (33) was also adapted to realize an arbitrary three-slice select sequence at 2.4 mm². The golden angle increment for the three-slice sequence occurred every 3TR. It should be noted that the unaliased FOV with golden angle sampling slightly differs with that of uniform density sampling used for the simulation in Figure 1 (32). Specifically, the point spread function of golden angle spiral sampling provides more reduced side-lobe energies, and provides improved trade-off of the achievable unaliased FOV in comparison to uniform density sampling (32).

Custom upper-airway coil

All of our experiments were performed on a GE Signa Excite 1.5 T scanner with a custom 8-channel upper airway receiver coil that has four elements on either side of the jaw. The elements were designed to be spatially localized and to be in close proximity to the upper-airway to provide high signal to noise over all the important upper airway structures. We chose custom coil design due its ability to provide superior SNR across all articulators of interest. This is advantageous as it provides an important boost in SNR while operating at 1.5 T, and enables efficient combination with spirals is highly complementary, as it together enables improved SNR at low fields, low off-resonance related spiral artifacts, and high sampling efficiency. The custom coil was developed for two sizes for adult and child arrays, although all the experiments in the current study were performed on adults using the adult sized array.

Reconstruction

On-the-fly

Data acquisition was implemented within custom RT-hawk software (34). A view-sharing scheme was used, where data for each frame was combined respectively from 13 and 21 subsequent interleaves, respectively for the 2.4 mm² and 1.76 mm² sequences. Images were reconstructed on-the-fly by using a fast implementation of the gridding algorithm within RT-hawk. The minimal latency allowed for instant feedback to the operator, and enabled efficient scan plane localizations. In addition, it enabled on-the-fly adjustment of the center frequency to minimize off-resonance blurs. Specifically, the subject being scanned was asked to open their mouth, and in the mid-sagittal plane, the operator qualitatively adjusted the center frequency such that the air-tissue (majorly: air-tongue, air-velum, air-lip) boundaries were sharp.

Off-line

A sparse SENSE temporal finite difference constrained reconstruction scheme was implemented offline. This constraint exploits redundancy based on the fact that the desired information is contained in the moving edges. This directly fits to the application of speech imaging, where the end goal is the assessment of interaction and timing of various articulators, or assessment of the dynamics of the air-tissue interfaces (moving edges). Spatial, and temporal finite difference constraints have also been previously used in several studies (eg. 35-39).

The reconstruction is formulated as:

$${min}_{f(\mathrm{x},t)}{\Vert A(f)-\mathbf{b}\Vert }_2^2+\lambda {\Vert D_t(f)\Vert }_1$$ [1]

where b is a concatenated vector containing the spiral noisy k-t measurements from each coil. f(x, t) is the dynamic data to be reconstructed at a retrospectively specified time resolution. A models coil sensitivity encoding as well as Fourier encoding on the specified spiral trajectory in each time frame; the coil sensitivities were assumed to be time-invariant, and were estimated by an Eigen decomposition method using time averaged image data from each coil (40), and the non-uniform Fourier Transform (nuFFT) implementation by (41) was used. The l₁-sparsity based temporal finite difference (D_t) penalty is used to penalize pixels with rapidly varying pixel time profiles. λ is the regularization parameter that controls the balance between the sparsity penalty and the data-fidelity. [1] was solved by a non-linear conjugate gradient (CG) algorithm (42). The algorithm was initialized with the f=A^H(b) estimate, and was terminated at 40 iterations, where qualitatively there was no noticeable change in image quality. The reconstructions were implemented within MATLAB on an Intel Core i7 3.5 GHz machine with 32 GB memory.

In-vivo experiments and speech tasks

Three healthy volunteers (2 male, 1 female, median age: 29), and one male tongue cancer patient (62 years) were scanned. The patient was scanned prior to clinical treatment. All the stimuli were presented in scanner using a mirror projector set up. A variety of speech tasks were considered. The mid-sagittal orientation was used for single-slice sequences, while orientations for the multi-slice sequence differed according to the speech task. With the 2.4mm² single-slice sequence, volunteer 1 (male Indian English speaker) was scanned without any speech stimuli on two separate instances: (a) using an eight-channel head coil, and (b) using the custom eight-channel upper-airway coil. With the custom upper-airway coil, the same volunteer was scanned with both the single-slice sequences, using the stimuli: “one-two-three-four-five” at a normal speech rate followed by a rapid speech rate (approximately four times faster). A task to produce interleaved consonant and vowel sounds by the repetition of the phrase: “loo-lee-laa-za-na-za” at the normal speech rate was considered on volunteer 1, and imaged using the three-slice sequence (one mid-sagittal, one axial plane at the level of mid-pharyngeal airway, and one coronal plane at the middle of the tongue). Volunteer 2 (male Chinese English speaker) was scanned with the 1.76 mm² single-slice sequence, while producing the sentence: “She had your dark suit in greasy wash water”, which involves producing sounds that involve rapid articulatory movements (e.g. co-articulation events as a part of running speech). Volunteer 3 (female American English speaker) was a beat-boxer, and was scanned while producing a variety of beat-boxing sounds. The 2.4 mm² single and concurrent three-slice (one mid-sagittal slice, one axial slice at the level of velum, one axial slice at the level of glottis) sequences were considered. The particular axial cuts were chosen to capture the rapid velar, and glottis movements during beat-boxing. Volunteer 4 (male American English speaker) was a tongue cancer patient, and was scanned with the single slice 2.4 mm² sequence. Speech stimuli comprising words, sentences, and a passage were presented, and the ability to produce speech was analyzed. A small subset of these stimuli is presented in this work, which pertain to short words that contain vowels interleaved by consonants: “beat, bit, bait, bet, bat, pot, bought, boat”.

Simultaneous audio collection

For 3 of 4 volunteers scanned, audio recordings were obtained simultaneously at a sampling frequency of 20 KHz inside the scanner, while the subjects were being imaged, using a commercial fiber optic microphone (Optoacoustics) (43), and a custom recording setup. Noise cancellation was performed using a data-driven adaptive signal processing algorithm, which is blind to the acoustic properties of noise (44). The final noise cancelled audio was synchronized with the reconstructed real-time MRI data to facilitate acoustic-articulatory analysis.

Analysis

Comparison of SNR between coils

The SNR properties of the custom eight-channel upper-airway (UA) coil were qualitatively compared with a commercial eight-channel head (brain) coil. The single slice (2.4 mm²) sequence was used, where volunteer 1 was scanned with no speech, and 55 interleaves were used to reconstruct f=A^H(b). The region of interest (ROI) SNR in different upper-airway regions were quantified as SNR_ROI = μ (S) / σ (n), where S is a vector with image intensities from the ROI containing a specific upper-airway structure, and n is a vector with intensities from an ROI in the background capturing only noise. A total of 10 ROIs were defined: 1) upper lip, 2) lower lip, 3) front tongue, 4) middle tongue, 5) back tongue, 6) hard palate, 7) velum, 8) pharyngeal wall, 9) epiglottis, and 10) glottis. A relative measure of SNR between the two coils was evaluated by the factor SNR_UA/SNR_head.

Choosing regularization parameter in constrained reconstruction

The regularization parameter in the constrained reconstruction was chosen empirically, with the best trade-off between artifact removal, and temporal stair-casing. With the single-slice 2.4 mm² sequence, and a 12 ms time resolution reconstruction, the effect of regularization parameter on the reconstructions was studied. L-curves were obtained for two datasets with different speech stimuli from different volunteers: volunteer 1 with fast speech stimuli of counting numbers, and volunteer 3 with beat-boxing. For generating the L-curve, the CG iterations were set to a very high number of 120 to ensure no bias in the norm calculations due to small numerical errors, however in practice a lower number of iterations of about 40 were sufficient for convergence, where qualitatively, there was no noticeable change in image quality.

Comparison of constrained reconstructions at various reduction factors

Golden-angle time interleaving allows for retrospective reconstruction with arbitrary time resolution. Using the speech stimuli of counting numbers at a rapid pace, reconstructions were performed at 5 TR, 3 TR, 2 TR, 1 TR for the 2.4 mm² and 1.76 mm² single-slice sequences. This corresponded to reduction factors (R) between 2.6-13 fold for the 2.4 mm² sequence, and 4.2-21 fold for the 1.76 mm² sequence. The regularization parameters were chosen empirically for all cases. A λ_2.4mm2=0.1, λ_1.76mm2=0.3 respectively for the 2.4mm² and 1.76 mm² sequence was used for the 5, 3, 2 TR cases, while a higher λ_2.4mm2=0.2, λ_1.76mm2=0.4 was used for the 1 TR case. The trade-off between residual aliasing artifacts, and temporal blurring (due to large temporal foot print), and reconstruction time was analyzed in all the cases.

Visualization of various stimuli with different reconstructions

To qualitatively depict the gains in improved time resolution, and its utility in capturing fast articulatory movements, constrained reconstructions (from sub-sampled data) were qualitatively compared to gridding reconstructions obtained at Nyquist (full) sampling using the stimuli of counting numbers (normal pace, followed by rapid pace). A reduction factor of 6.5-7 fold was used in constrained reconstruction, which corresponded to a temporal foot-print of 2 TR, and 3 TR, respectively for the single slice 2.4 mm², and 1.76 mm² sequences. Constrained reconstructions were also qualitatively compared to the view-sharing reconstruction, which was used in our previous work (21-22). To ensure consistent comparisons, the step size in view-sharing was matched to the native time resolution in the constrained reconstruction. Acoustic-articulatory analysis along with articulatory image segmentation was performed on the patient dataset. An upper-airway image segmentation algorithm (45) was modified to handle image space data, and was applied to segment all the important articulators in all the frames; this enabled tracking of the articulators across time.

Results

Simulation of spatial v/s time resolution trade-offs with spiral sampling

Figure 1 shows the improved spatial and temporal resolution trade-off with rate 6.5 under-sampled spiral imaging over fully-sampled spiral imaging. Fast articulatory movements such as consonant constrictions, co-articulation events, and flaps, trills are expected to be well depicted at rate 6.5 fold single place spiral imaging over fully sampled spiral imaging. Multi-plane imaging compromises the spatial and time resolutions for additional slice coverage. At rate 6.5 fold under-sampling, 3-plane imaging offers spatio-temporal resolutions of up to 2.4 mm², and 36 ms/frame, which enables capture of speech tasks such as consonant constrictions, co-articulation events, and all tongue movements. In contrast, with full sampling, multi-plane imaging is notoriously slow, for its efficient use in capturing fast articulatory shaping.

Comparison of SNR between coils

Figure 2 shows the qualitative comparison of the improvement offered by the custom upper-airway 8-channel coil (adult sized array) over an 8-channel commercial head coil. The individual coil images provide high sensitivity over all relevant upper-airway regions. The upper-airway coil offered improved SNR for all articulators. The relative SNR gain over the commercial head coil in different upper-airway regions were: 2.2 – velum; 2.6 – pharyngeal wall; 2.9 – hard palate; 3 – upper lip; 4.3 – lower lip; 4.5 – mid tongue; 4.6 – back tongue; 5.2 – glottis; 5.4 – front tongue; 5.9 – epiglottis.

Figure 2.

The custom eight channel upper-airway coil and its placement on a subject (a). As shown in (d), the individual coil images from all the channels depict high signal on all relevant upper-airway regions. (b) depicts the coil combined image that demonstrate high SNR on all the upper-ariway regions. (c) depicts the relative SNR gain map over a commercial eight-channel head coil, where SNR gains between 2 to 6 fold are observed in all the upper-airway regions of interest.

Choosing regularization parameter in constrained reconstruction

Figure 3 shows consistent behavior of L-curve for two subjects with two stimuli that were different: (a) counting numbers at a rapid pace (from volunteer 1); and (b) producing beat boxing sounds (from volunteer 3). The reconstructions produced expected image quality and artifacts with varying choice of λ. The choices of λ>0.4 resulted in greater denoising within the tissue (e.g. interior of tongue), but this occurred at the expense of temporal blurring of the fast articulators (e.g. lips tongue tip). The choice of extremely low λs (<0.05) resulted in residual noise, and aliasing. The choice of λ=0.1 was empirically chosen – which resolved all aliasing while maintaining temporal fidelity of fast moving articulators, and slight residual noise in the interiors of the tissue. This choice correlated with a bias toward data-consistency in the L-curve, and a slightly higher sparsity norm in comparison to the inflection point on the L-curve.

Figure 3.

Determining the regularization parameter λ: The L-curve determines the choice of λ that best controls the bias between the sparsity constraint and data-consistency on a global perspective. Our choice however was motivated by the local spatio-temporal fidelity of the images. To find the best λ, we relied on an empirical based approach where the temporal fidelity of fast articulators was qualitatively analyzed. The choices of λ greater than 0.5 resulted in greater denoising within the tissue (eg. interior of tongue), but occurred at the expense of temporal blurring of the fast articulators (eg. lips, tongue tip as shown from the image time profiles above). λ=0.1 was chosen which resolved all aliases with maintained temporal fidelity of the articulators, and slight residual noise in the interiors of the tissue. This choice correlated with a bias towards data-consistency in the L-curve, and a slightly higher sparsity norm in comparison to the inflection point of the L-curve. The behavior was consistent across datasets from different volunteers, with different speech tasks.

Comparison of constrained reconstructions at various reduction factors

Figure 4 demonstrates constrained reconstructions at 1 TR, 2 TR, 3 TR, and 5 TR time resolutions with the 2.4 mm² and 1.76 mm² single slice sequences. While the reconstructions with 5 TR demonstrate blurring of the articulatory movements such as the tongue tip touching the hard palate during the formation of the sound /n/, the reconstructions with time resolutions of at least 3 TR qualitatively demonstrates capturing these events well. This is depicted as increased sharpness along the image time profiles for the 3, 2, 1 TR reconstructions versus the 5 TR reconstructions in Figure 4. The unresolved aliasing artifacts were qualitatively substantial only at very high sub-sampling factors, which corresponds to 1 TR reconstruction with the 1.76 mm² sequence (i.e., 21 fold acceleration level). Mild aliasing artifacts were present with the 2.4 mm² – 1 TR; 1.76 mm² – 2TR reconstructions. Reconstruction times respectively for reconstructing a 5 second sample with 2.4/1.76 mm² sequences were: a) 5TR – (12/17 minutes); b) 3 TR – (19/26 minutes); c) 2 TR – (24/36 minutes); d) 1 TR – (45/76 minutes). Based on the tradeoffs between temporal resolution (due to large temporal footprint), residual alias artifact energy, and reconstruction times, a conservative acceleration factor of 6.5-7 fold was empirically chosen in this study.

Figure 4.

Evaluation of constrained reconstruction at different sub-sampling factors using a 5 sec speech stimuli of counting numbers at a rapid pace. Constrained reconstructions at 5, 3, 2, 1 TR time resolutions are compared with the (a) 2.4 mm², and (b) 1.76 mm² single slice sequences. The spatial frame depicts the event of producing the sound /n/ in the word “one”, where the tongue hits the hard palate. The image time profile corresponds to the time evolution of the cut depicted by the horizontal dotted arrows in (a) and (b). As depicted by the white arrows in the spatial frames, the 5 TR reconstructions produce temporal blurring in capturing the formation of the /n. sound, while the 3, 2, 1 TR reconstructions depict this event well. The temporal blurring in the 5 TR reconstructions is also evident in the image time profiles (see yellow arrows). As the acceleration factor is increased, alias artifacts are predominant in the images as seen with R∼21 in the 1 TR reconstruction at 1.76 mm² spatial resolution (green arrows on the image time profiles). With R∼13, mild alias artifacts were observed in the 1 TR reconstruction at 2.4 mm² spatial resolution. Based on these observations, an acceleration factor of 6.5-7 fold is chosen for rest of the experiments in this work.

Comparison of under-sampled constrained reconstruction with fully-sampled gridding reconstructions without and with view sharing

Figure 5 compares fully-sampled (Nyquist) acquisitions using gridding reconstructions against under-sampled acquisitions with constrained reconstruction. As depicted in the image time profiles, the proposed reconstruction enables marked improvement in temporal fidelity, and enables robust visualization of the fast articulatory shaping such as tongue tip movements, and opening and closure of lips.

Figure 5.

Demonstration of improved temporal resolution using sparse sampling and constrained reconstruction: A speech task of repeated counting of numbers “one-two-three-four-five” at a normal pace followed by a rapid pace (∼4 times faster) was performed. The top and bottom rows show the comparisons against Nyquist spiral sampling and gridding reconstruction with the 2.4 mm² and 1.76 mm² sequences respectively, where the time profiles corresponds to the cut marked by the white arrow in the spatial frames (first column). Note the marked improvement in the temporal fidelity (in terms of crispness along time profiles) of the accelerated reconstructions in comparison with the fully sampled reconstructions. As demonstrated in the last column, the gains in time resolution ensure the capture of fast articulatory movements.

Figure 6 demonstrates comparisons of constrained reconstruction against view-sharing using the 1.76 mm² single slice sequence. While view-sharing reconstructions qualitatively demonstrated minimal temporal blurring with normal speech rate stimuli, it showed temporal blurring with fast speech rate (approximately 4 times faster than normal pace). For instance, the events of opening of lips, and raising of tongue tip towards the hard palate during the production of initial vowel sound in “one” and the lip movement during the sound /f/ in “five”, were considerably blurred with view-sharing, while these events were robustly depicted with constrained reconstruction (Fig 6).

Figure 6.

Demonstration of improved spatio-temporal fidelity with constrained reconstruction, over view sharing reconstruction. The top row shows two image time profiles corresponding to the cuts marked in the spatial frame of the bottom row. View-sharing depicted good image quality in articulators that moved slowly such as velar motion; however showed motion artifacts of faster articulators such as those involving lips and the tongue tip. In comparison, constrained reconstruction showed improved overall temporal fidelity (see crispness along time profiles in the top row). The bottom row shows examples of motion blurs in view-sharing due to fast lip and tongue movements (see white and yellow arrows).

Figure 7 demonstrates one more example from volunteer 2 with free running speech. Similar performances are shown in this figure, where constrained reconstruction showed improved temporal fidelity over view-sharing. In particular, in capturing the production of consonant clusters such as [kr], which involves rapid transitions of articulatory positions such as the tongue base touching the velar region followed by tongue tip touching hard palate.

Figure 7.

Single slice RT-MRI acquisition aligned with a simultaneous acquired audio of speech after acoustic gradient noise cancellation: The speech task was the phrase “bright sunshine shimmers across the ocean” at the subject's normal pace. This phrase contains several consonant clusters and fricatives, which involve fast articulatory movements. For instance, the time window highlighted by the red arrow depicts the formation of the consonant cluster [kr], where the tongue retracts back, and then touches the hard palate. The bottom row shows every fourth frame (for compactness) of the reconstructions to depict the articulatory movement during this sound. Note the constrained reconstruction based images show less motion blur and loss of temporal fidelity in comparison to view-sharing (see yellow arrows).

Multi-plane imaging of interleaved consonant and vowel sounds

Figure 8 demonstrates an example of producing the sequence “loo-lee-la-za-na-za” simultaneously in mid-sagittal, coronal, and axial planes at 36 ms/frame with constrained reconstruction. Concurrent coronal, and axial imaging enabled capturing of articulatory shaping that is complementary to mid-sagittal orientation, such as tongue grooving in the coronal plane, and airway opening and closures in two-dimensions at the lower pharyngeal airway level.

Figure 8.

Simultaneous visualization of sagittal, coronal, and axial planes during production of loo-lee-la-za-na-za sounds using constrained reconstruction at a time resolution of 36 ms, and a spatial resolution of 2.4 mm². Note movements such as tongue grooving, and pharyngeal airway shaping in two dimensions can be best visualized in the coronal and axial planes, and add complementary information to the sagittal view.

Axial imaging of the glottis and velum

Figure 9 shows examples of axial imaging in the regions of the glottis and the velum, concurrently with the midsagittal plane during recording of a short beatboxing session. Fig. 9(a), shows the changing area of the velopharyngeal port (also see arrows). During the production of the beatboxing sound, the velum rises to close against the posterior pharyngeal wall, and the lateral pharyngeal walls move to close against the soft palate, which results in a sphincter-type closure that obstructs the flow of air towards the nasal cavity. The third row of Fig. 9(b), gives insight to the dynamics of the glottal cross-sectional area. This area has been modeled as the sum of two components: a slow-varying component (speed comparable to the other speech articulators), and a fast-varying component (vibrating at typically >100Hz during voiced sounds) (46). The slow-varying component distinguishes between voiced (small cross-sectional area) and unvoiced (large area) sounds, and is expected to spread temporally with co-articulation. The time-course of the slow-varying glottal component is typically only inferred (47-48). While the fast-varying glottal component (which determines pitch) is too fast to be observed by MRI, we posit that the slow-varying component can be observed.

Figure 9.

Demonstration of axial imaging at the areas of the glottis and the velum. The two example sequences are drawn from a short beatboxing section and correspond to imitations of percussion instruments. Note the potential of imaging the dynamics of the area of the velo-pharyngeal port (which is critical for the characterizing nasal vs. oral speech segments), and the dynamics of the glottal aperture (critical for characterizing voiced vs. unvoiced segments).

Articulatory-Acoustic analysis of a pre-operative tongue cancer patient

Figure 10 shows the result from an articulatory segmentation algorithm, which segments, the: nose, upper-lip, lower lip, hard palate, tongue, epiglottis, glottis, velum, pharyngeal wall. This allows for identification and tracking of subtle articulatory motion patterns, such as the tongue contour shape changing, and opening of lips during the production of the word [bit]. ROI analysis is also shown, which gives a more compact view of the articulatory kinematics along certain parts of the vocal tract. Three averaged ROI time profiles are aligned with the noise-cancelled spectrogram of simultaneously acquired audio. In all the words, the first “red” segment in the spectrogram for a healthy subject should correspond to the vowel and the second red segment to the release of the plosive consonant. Plosive consonants like /p/,/k/,/t/,/b/,/g/,/d/ have a) a closure phase; which corresponds to a silence phase in the spectrogram; and b) a release phase due to the sudden opening of the airway.

Figure 10.

Single slice RT-imaging of a preoperative tongue cancer patient at 12ms/frame: The patient's ability to produce consonant and vowel sounds was studied. Speech stimuli of several short words with the consonant sounds interleaved with vowel sounds as consonant-vowel-consonant was used. Region of interest averaged time course profiles at the airway between lips (ROI 1), tongue tip (ROI 2), and airway between back of velum and pharyngeal wall (ROI 3) are shown at the native time resolution of 12 ms. Note the high time resolution depicts the formation of all the sounds with excellent fidelity, and correlates well with the simultaneously acquired audio signal.

This example was drawn from the recording of a pre-operative cancer patient, who was able to produce normal speech while producing the short words listed in Fig 10. The patient's articulatory trajectories followed normal patterns as depicted by the closure phase of /b/ being marked by a local maximum in the lip trajectory, the closure phase of the /t/ sound being marked by a local maximum in the ROI 2(tongue tip) trajectory and its release by the sudden drop in the same ROI 2 (tongue tip) trajectory.

Discussion

The proposed MRI based system combining custom upper-airway coil acquisition, flexible multi-plane interleaved fast spiral readouts, constrained reconstruction, simultaneous audio recording and noise cancellation demonstrated improved visualization of rapid vocal tract dynamics with time resolutions of up to 12 ms, and 36 ms respectively for single slice, and three slice imaging. In comparison to fully sampled spiral imaging with gridding reconstruction, under-sampled spiral imaging with temporal finite difference constrained reconstruction demonstrated marked improvement in visualization of articulators during fast speech (e.g. consonant constrictions, co-articulation events). View-sharing reconstruction were adapted as a part of on-the-fly reconstruction during data acquisition. Comparisons of the off-line constrained reconstruction against view-sharing demonstrated improvements in preserving true temporal resolution with constrained reconstruction, by reducing the temporal foot print of the number of interleaves required to form a single frame. These differences were visually evident with single slice 1.76 mm² sequence, and the multi-slice 2.4 mm² sequences.

A direct comparison of the proposed system against state of the art methods (eg. 26, 27) has not been performed due to practical challenges in implementation at our site (eg. implementation at 3T, custom k-t sampling requirements (27), acquisition with a 64 channel head coil, and on-the-fly reconstruction with GPUs (26)). Comparisons of some of the individual components used in the proposed work against alternate components have been performed in the literature. For instance, the SNR benefits, with a custom coil have been utilized toward improved parallel imaging in (49). The trade-off amongst spatial resolution, time resolution, artifacts, with different trajectories have been reported in (3, 25).

With a single temporal finite difference constraint, high acceleration factors (>10 fold) demonstrated only mild artifacts (Fig 4). We attribute this to the improved SNR offered by the custom airway coil and SNR-efficient spiral acquisitions. Performance at these high accelerations may be further improved by enforcing additional spatial sparsity constraints, and/or suitable post-processing methods such as median filtering (26). Note that additional constraints also introduce additional tuning of regularization parameters. Reconstruction times in this work may also be improved by the use of coil compression (eg. (50)), efficient optimization algorithms such as alternating direction of method of multipliers (51), advanced parallelization of reconstruction (26), use of graphical processing units (26, 52-54). Automatic tuning of regularization parameters as demonstrated in (55) may also be feasible. On our systems we have found gradient errors to be insignificant with very short spiral readouts (2.4 ms) as used in this work, because there is insufficient time for these to accumulate into significant k-space trajectory errors. However, we acknowledge that a controlled experiment should be performed to confirm this, and that performance may vary among MRI vendors or with gradient subsystems. The coil map estimation in this work assumed the coil maps to be time invariant, and were estimated using the eigen decomposition method prior to reconstruction. Joint estimation of time varying coil maps along with the reconstruction may further improve the well-posedness of the problem, and may improve the image quality (eg. 56). All the above extensions require detailed investigations, and are scope of future work.

In two of the four subjects, we observed a cardiac pulsation artifact (Figure 10). Although not shown in this manuscript, we have observed this in several prior studies, and have confirmed that the artifact relates to the precise tilt of the subjects head. This artifact occurs when the mid-sagittal airway plane also intersects with the heart and/or large vessels far outside the imaging FOV. This artifact can typically be corrected on-the-fly by adjusting the scan plane. Off-line constrained reconstruction was observed to partially suppress this artifact relative to view-sharing reconstruction .

In this work, image quality from constrained reconstruction was assessed qualitatively as quantitative assessment is challenging in several ways due to the lack of ground truth, and the non-linear nature of the reconstruction. Ground truth images obtained from mechanically rotating phantoms with known velocities have been previously used to assess temporal fidelity of temporal constraints (57). Rotating phantoms however do not fully describe articulatory motion patterns during speech production. It may be possible to extract a ground truth from higher temporal resolution modalities such as EMA, or high-speed camera imaging for plainly visible articulators such as the lips and tongue. Comparison of the articulatory motion profiles obtained from constrained reconstruction with real time MRI against these modalities may be appropriate for investigation of speech tasks over a broad range of speech rate.

Example data from three normal subjects, and one patient are shown in this work. The supporting videos (S1-S6) contain real-time MRI movies from parts of our ongoing speech science studies. Additional video material is hosted at http://sail.usc.edu/span/fastmri.html. These include example reconstructions from linguistically driven Puerto-Rican Spanish study, and a comprehensive study of acquiring all sounds in the entire International Phonetic Alphabet (IPA) chart from four experienced phoneticians.

Supplementary Material

Supp Video S1. Supporting Video S1.

Single slice RT-imaging using the 2.4 mm² sequence at 12 ms time resolution. The speech task involved counting numbers at a normal pace followed by a rapid pace (approximately four times faster). The video also shows the result from an articulatory segmentation algorithm, which segments, the: nose, upper-lip, lower lip, hard palate, tongue, epiglottis, glottis, velum, pharyngeal wall. Segmentation of articulators allow for tracking

Supp Video S2. Supporting Video S2.

Rapid beat boxing sounds captured with 2.4 mm² spatial resolution at 12 ms time resolution.

Supp Video S3. Supporting Video S3.

Vowel sounds produced by an expert phonetician and captured at 12 ms time resolution. These sounds are a part of a study, which involved imaging all the sounds in the entire International Phonetic Alphabet (IPA) chart.

Supp Video S4. Supporting Video S4.

Consonant sounds produced by an expert phonetician and captured at 12 ms time resolution. These sounds are a part of a study, which involved imaging all the sounds in the entire International Phonetic Alphabet (IPA) chart.

Supp Video S5. Supporting Video S5.

Fluent speech (rainbow passage) produced by an expert phonetician and captured at 12 ms time resolution.

Supp Video S6. Supporting Video S6.

Beat-boxing sounds imaged concurrently in mid-sagittal, and 2 axial planes at the level of velum, and glottis at 36 ms/frame.