Automatic extraction of abnormal lip movement features from the alternating motion rate (AMR) task in ALS

Abstract

Purpose:

With the long-term goal to develop a clinically feasible tool for assessing articulatory involvement in ALS, we designed an algorithmic approach to automatically extract lip movement features during an alternating motion rate (AMR) task and assessed their efficacy for detecting and monitoring articulatory involvement in ALS.

Method:

Twenty three spatial, temporal, and spatiotemporal AMR features were extracted from 161 samples of lip movements (139 from participants with ALS; 22 from neurologically-intact participants). The diagnostic value of these features was assessed based on their (1) sensitivity for detecting early bulbar motor involvement, and (2) associations with accepted clinical measures of bulbar disease progression.

Result:

Among all AMR features, two temporal features were the most affected - temporal variability and syllable frequency, which (1) showed large changes during early disease stages and (2) predicted the progression of bulbar motor involvement and speech intelligibility decline. Spatial features were in general, less sensitive to early bulbar motor involvement.

Conclusion:

The findings provided preliminary support for the algorithmic approach to quantifying articulatory features predictive of bulbar motor and speech decline in ALS. The differential disease effects on spatial and temporal AMR features might shed light on the mechanism of articulatory involvement during ALS progression.

Introduction

Amyotrophic lateral sclerosis (ALS) is a motor neuron disease characterised by the gradual paresis of speech muscles, resulting in progressive loss of speech communication. In the current study, we investigated the diagnostic efficacy of articulatory movement analysis for assessing bulbar motor involvement due to ALS. The long-term goal of our research program is to develop a clinically feasible assessment tool for detecting and monitoring bulbar motor involvement in ALS. Our motivation for assessing articulatory movement is that, among the four physiological speech subsystems (i.e. respiratory, phonatory, articulatory, and resonatory), impairments to the articulatory subsystem appear to have the most significant impact on speech intelligibility (Rong et al., 2016). Moreover, the onset of articulatory involvement is detected early in the disease prior to perceptually detectable changes in speech (Rong, Yunusova, Wang, & Green, 2015; Rong et al., 2016). Multiple changes in articulatory movement have been associated with the progression of ALS including spatial (e.g. movement displacement; Green, Yunusova, et al. (2013); Hirose (1986); Yunusova et al. (2010)), temporal (e.g. movement time; Tjaden and Turner (2000); Weismer, Jeng, Laures, Kent, and Kent (2001); Yunusova et al. (2010)), and spatiotemporal (i.e. spatiotemporal variability; Kuruvilla-Dugdale and Mefferd (2017)). Monitoring of these articulatory features during the early and late stages of ALS could have important clinical implications including (1) improving early diagnosis of bulbar motor involvement, (2) determining behavioural treatment targets in clinical interventions (e.g. compensatory strategies to maintain speech for as long as possible), and (3) improving predictions of the timing of the impending speech loss and the needs for assistive technologies.

Although the analysis of articulatory movement may provide a useful diagnostic target for bulbar ALS, clinical implementation has been slow, in part, because of the prohibitive cost of optical and electromagnetic tracking. As the price barrier to lip and jaw motion capture has recently been removed by the emergence of low-cost 3D depth sensing cameras, tracking lip and jaw movements has become possible in clinical environments. In this study, we investigated the feasibility and clinical efficacy of an automated analysis of lip movement during rapid syllable repetitions, which is commonly known as the alternate motion rate task (AMR), for assessing bulbar motor involvement. Our focus on movements of the lips is because (1) they are accessible using depth-sensing cameras, unlike those of the tongue, and the accuracy of lip movement tracking using depth-sensing cameras is substantially higher than jaw movement tracking (Bandini, Green, Zinman, & Yunusova, 2017); and (2) our previous studies have demonstrated that changes in lip movements precede detectible declines in speech intelligibility and speaking rate (Rong, Yunusova, Wang, et al., 2015; Rong et al., 2016; Yunusova et al., 2010).

Rationale for testing articulatory impairment in ALS using AMR.

To assess bulbar motor involvement, we chose a behavioural task - AMR - that was both physically demanding and easy to administer. Previous studies have shown that tasks with high motoric load, such as AMR, reveal declines in speech motor performance prior to clinically-observable speech changes during conversation and reading (Nishio & Niimi, 2000; Rong, Yunusova, Wang, et al., 2015; Rong et al., 2016). This is consistent with prior studies showing that reduced contraction rate of orofacial muscles was one of the most severe oromotor deficits underlying speech motor involvement in ALS (Langmore & Lehman, 1994).

Besides being a physically demanding task, AMR is also less affected by cognitive-linguistic deficits than conversational speech or reading (Wang, Kent, Duffy, & Thomas, 2009; Wang, Kent, Duffy, Thomas, & Weismer, 2004). In ALS, 10% of patients exhibit symptoms of frontotemporal dementia (FTD) and up to 50% show signs of cognitive-linguistic deficits in addition to speech motor symptoms (Goldstein & Abrahams, 2013; Yunusova et al., 2016). Yunusova et al. (2016) showed that several speech measures that are commonly used to index bulbar severity such as speaking rate and pausing time during passage reading were affected by both cognitive-linguistic and bulbar motor deficits. The confounding effect of cognitive-linguistic deficits could compromise the sensitivity of these measures for detecting bulbar motor deficits. These advantages of AMR as well as its clinical feasibility support its potential as a clinical tool for assessing articulatory involvement in ALS.

The extent to which AMR represents speech capacity, however, has been called in to questions. One criticism is that AMR may not recruit the same neural structures as does speech because of its simple phonological structure, its focus on maximum performance, and its lack of communication intent. These task differences have fueled long-standing theoretical debates about the differences in task demands and the neural networks that govern speech and speechlike/nonspeech oral motor behaviours (Ballard, Robin, Solomon, Moon, & Folkins, 2009; Kent, 2015; Maas, 2017; Staiger, Schölderle, Brendel, Bötzel, & Ziegler, 2017; Weismer, 2006; Westbury & Dembowski, 1993; Ziegler, 2003). The task-dependent model (TDM) (Ziegler, 2003) emphasises that motor control is task-dependent, and that speech and nonspeech are fundamentally different in both neural control and behavioural characteristics (Staiger et al., 2017). In contrast, the integrative model (IM) incorporates speech and volitional nonspeech motor control into the functioning of a general motor system (Ballard, Robin, & Folkins, 2003). The IM holds the view of a gradient distinction between speech and nonspeech with some shared components in both neural and behavioural systems. While the TDM minimises the diagnostic value of speechlike and nonspeech tasks for speech assessment, the IM suggests that carefully-designed tasks with shared speech and nonspeech components (e.g. AMR) are useful for speech motor assessment (Ballard et al., 2003; Maas, 2017). Despite the controversy on the usefulness of speechlike tasks in assessing speech production and associated intelligibility deficit, they remain a standard component in the assessment of oral-motor performance and detection of lesion-specific characteristics in neurological diseases (Duffy, 2013).

Rational for articulatory movement-informed assessment of early bulbar motor involvement.

Our enthusiasm for an articulatory movement-informed bulbar motor assessment is, in part, based on the observation of Wang et al. (2009) showing that more than one third of the AMR acoustic recordings from their subjects with dysarthria were nonexecutable - a finding which underscores challenges of acoustic-based approaches in automated analysis of AMR. In an earlier study, Wang and colleagues (2004) noted that the validity of computing acoustic features from abnormal speech samples was significantly compromised by the presence of explosive speech quality, breathy voice, abnormal stop bursts, and continuous voicing, which worsen with dysarthria severity. The confounding effect of voice or resonance impairments could compromise the sensitivity of acoustic measures for detecting articulatory deficits. For example, abnormal voice features, as commonly found in dysarthria (Ramig, Scherer, Klasner, Titze, & Horii, 1990), increase the difficulty for determining the onset and offset of AMR cycles and thus, impact the accuracy of the acoustic measures (Wang et al., 2004).

Performance features extracted from lip movement recordings

In this study, we examined the diagnostic value of 23 lip movement features, which were automatically extracted by an algorithmic approach, for detecting and tracking bulbar motor involvement in ALS. These features comprehensively characterised the spatial, temporal, and spatiotemporal features of lip movement during the AMR task (see Method for rationale for each measure and Supplementary materials for other details about these variables). The diagnostic value of the AMR features was assessed based on their (1) sensitivity for detecting bulbar motor involvement and especially early bulbar motor involvement, and (2) associations with the clinical gold standards of bulbar motor assessment.

Method

Participants

Fifty-seven participants diagnosed with possible, probable or definite ALS (33 male and 24 female; aged from 40 to 81 years old) and 22 healthy participants with normal speech-language-hearing functions (10 male and 12 female; aged from 45 to 73 years old) took part in this study. All participants met the following criteria to be recruited: (1) spoke fluent English, (2) able to read at the grade 5 level, (3) able to follow the instructions, (4) reported no hearing impairment or visual impairment preventing them from reading, (5) had adequate cognitive function to perform the tasks (as indicated by MoCA or ALS-CFB), and (6) were not on medications known to affect speech production (Gawel, 1981). Because an important goal of this study was to identify sensitive markers for detecting early bulbar motor involvement in individuals with ALS before they develop clinical bulbar signs, presence of dysarthria was not considered as an inclusion criterion for participants with ALS.

Twenty-seven of the participants with ALS were studied longitudinally (range: 2–13 sessions). The duration between the first and last sessions ranged from 60 to 681 days (SD = 197 days). The interval between sessions varied depending on participant availability. The other 30 participants with ALS and all healthy participants attended only one session. As a result, a total 161 sessions were recorded from all participants, including 139 from participants with ALS (57 initial sessions, 82 follow-up sessions) and 22 from healthy participants.

Among all participants with ALS, 8 reported bulbar onset, 44 spinal onset, 4 mixed bulbar and spinal onset, and 1 had an unknown onset site. Disease duration varied among participants. In the first session, participants were on average 12 months post diagnosis (SD = 14 months). The severity of ALS and its bulbar presentation also varied among participants, as assessed by the ALS Functional Rating Scale – Revised (ALSFRS-R) (Cedarbaum et al., 1999). The ALSFRS-R score (0~48) was obtained from 12 survey questions that assess the degree of functional impairments with the score on each question ranging from 4 – least impaired to 0 – most impaired. The ALSFRS-R scores of the 57 participants with ALS in the initial session ranged between 18 and 48 with a mean of 35 and SD of 8. The bulbar subscore, estimated based on the first three questions of the scale, assessed the bulbar motor functions of speech, swallowing, and chewing with a maximum score of 12 (range 3 to 12, M = 10, SD = 2).

Stratifying subjects at early and late stages of bulbar motor involvement in ALS.

To assess behavioural changes during the progression of bulbar ALS, we stratified the early and late stages of bulbar ALS based on a bi-phasic nonlinear mixed effects (NLME) model (nlmefit, MATLAB R2016a). This model included a fixed effect that predicted speech intelligibility using AMR as a predictor and a random effect that accounted for subject effects. The cutoff of the two phases of this model was used to stratify the early and late stages of bulbar ALS. Following this criterion, 109 sessions from participants with ALS were stratified as early ALS (ALSFRS-R bulbar subscore: M = 10.68, SD = 1.63) and 30 sessions from the participants with ALS were stratified as late ALS (ALSFRS-R bulbar subscore: M = 7.31, SD = 2.7) based on a cutoff of AMR < 2.64 (cycles / sec). As shown in Figure 4d (and see Results for more details), speech intelligibility (1) remained above 96% during the early phase when AMR >= 2.64 (cycles / sec) and (2) declined rapidly during the late phase when AMR < 2.64 (cycles / sec).

Figure 4.

Scatter plots showing the relations of the AMR variables to speech intelligibility (a–d) and speaking rate (e–h) in subjects with ALS. Circles correspond to data from subjects with a single-session recording; dots correspond to data from subjects with multiple-session (longitudinal) recordings. The lines correspond to the fit of the bi-phasic NLME models (a–d) and the LME models (e–h).

Data acquisition

Lip kinematic data acquisition.

Data during the AMR task were drawn from a pre-existing database collected in a larger project that involved longitudinal assessments of all the speech subsystems using a variety of kinematic, acoustic, aerodynamic, and auditory perceptual approaches in a large group of patients with ALS. Details of the data acquisition protocols were provided in Yunusova, Green, Wang, Pattee, and Zinman (2011). Briefly here, the movements of upper and lower lips were recorded in 3 dimensions (3D) using an electromagnetic or optical tracking system (electromagnetic: WAVE, NDI, and EMA, Carstens; optical: Motion Capture System, MotionAnalysis Corp.). Because lower lip is located on the mandible, the lower lip movement captured the combined movement of lower lip and jaw.

AMR task.

Participants repeated the syllable /ba/ as fast and accurately as possible on one breath group. To ensure the participants perform the task properly, the experimenter instructed the participants to (1) give their full effort by repeating the syllables as fast as they can and (2) produce as many syllables as possible until they run out of breath. Due to the fatiguing nature of this task, only one trial was performed by each participant, after a single practice run.

Kinematic data processing.

Prior to extracting the articulatory performance features, we derived the 3D Euclidean distance between upper lip and lower lip, which was low-pass filtered at 15 Hz. The filtered kinematic signals were used for all of the following data analyses.

Speech measurements

In addition to movement traces, speech intelligibility and speaking rate were assessed using the Speech Intelligibility Test (SIT) (Yorkston, Beukelman, & Hakel, 1996). In the test, participants were asked to read a list of 11 sentences of varying length (5 to 15 words) randomly generated by the SIT software. SIT recordings were transcribed by a naïve listener who was unfamiliar with either the test or the speech profiles of the participants. Based on the transcriptions and the speech recordings, the SIT software automatically calculated (1) speech intelligibility, which was the percentage of words correctly transcribed out of the total number of words, and (2) speaking rate, which was the number of words read per minute (WPM). Speech intelligibility has been used as a measure of functional communication decline and dysarthria severity in ALS (Green, Yunusova, et al., 2013; Rong, Yunusova, Wang, et al., 2015; Rong et al., 2016). Speaking rate was used as an index of bulbar disease severity because previous studies have described its association with bulbar ALS progression and relatively linear decline over time (Green, Yunusova, et al., 2013; Yorkston, Strand, Miller, & Smith, 1993). Both speaking rate and speech intelligibility were examined because they have distinct trajectories of decline - while speaking rate declines linearly, speech intelligibility declines non-monotonically, as characterised by two phases – an early slow declining phase and a late rapid declining phase (Rong, Yunusova, Wang, et al., 2015; Rong et al., 2016).

Automatic extraction of lip movement features

Semi-automatic parsing of lip movement.

An algorithmic approach was developed in MATLAB to segment the opening and closing phases of the lip distance trace time-series that were associated with each cycle of syllable repetitions (see Supplementary materials: Figure S1a). The opening phase started at the time point corresponding to the minimum distance between upper and lower lips during the closure of /b/ and ended at the time point corresponding to the maximum distance between upper and lower lips during the vowel /a/. The closing phase started from the maximum opening and ended at the time point corresponding to the minimum distance between upper and lower lips during the closure of /b/ in the next syllable.

To identify the opening and closing phases in each cycle, our approach first automatically detected all peaks and troughs from the lip distance trace time-series during the entire sequence using an automatic peak detection algorithm. As shown in Figure S1a, because lip movement followed a cyclic pattern in this example, the automatic peak detection algorithm effectively identified the onset and offset of the opening and closing phases of each cycle. However, in some severely impaired subjects such as the example in Figure S1b, lip movements became irregular due to articulatory impairment, resulting in local peaks and troughs that disrupted the cyclic pattern. These local peaks and troughs should be distinguished from the global peaks and troughs that corresponded to the onset and offset of the opening and closing phase.

To address this issue, our approach automatically identified the cycles with large spatial or temporal deviations from the average pattern across all cycles. Specifically, (1) all automatically-detected cycles showing a peak-to-trough distance beyond ±2 standard deviations from the mean distance and (2) all automatically-detected cycles whose duration was beyond ±2 standard deviations from the average cycle duration were identified. For the AMR sequences containing these irregular cycles, the time-series of lip distance was displayed graphically, and the cycles with large deviations as identified above were manually corrected. In the manual correction, the “false” peaks and troughs with relatively small amplitudes, which reflected irregular lip movement, were discarded and the most prominent peaks and troughs were manually identified as the onset and offset of each phase. Figure S1c shows the manually corrected onsets and offsets of the opening and closing phases of all cycles for the same file as shown in Figure S1b. Among the 161 files of lip movements, only three required manual logging, which yielded 2% of the total number of files analysed.

Automatic extraction of lip movement features.

Based on the lip movements during the AMR task, 21 variables that characterised the spatial, temporal, and spatiotemporal features of lip movement and two additional variables indexing the overall lip AMR performance were automatically extracted by the algorithmic approach (see Supplementary materials: Table S1). Spatial features were characterised by (1) three variables representing the time-dependent changes in lip movement displacement throughout the AMR sequence, (2) six variables indexing spatial variability across cycles, and (3) three variables indicating the extent of lip movement displacement. Temporal features were characterised by (1) one variable indicating syllable repetition frequency, (2) two variables corresponding to the overall duration and average cycle movement time, and (3) three variables indexing temporal variability across cycles. Spatiotemporal features were characterised by three variables including the maximum velocity of lip movement, the spatiotemporal variability across cycles, and the spatiotemporal irregularity of lip movement. Of all variables, eleven (i.e. Slp1, Slp2, Slp_d, D_max_open, D_max_close, D_mean, Freq, Dur, Max_vel, STI, D_dtw) were extracted from the movement pattern of the entire AMR sequence; twelve (i.e. Sse1, Sse2, Sse, Scanning_d1, Scanning_d2, Scanning_d, T_mean, Tsd, Tsd_cv, Jitter, Ncyc, AMR) were extracted from individual cycles of movement.

The rationale for these variables is described in the following. Due to the loss of both upper and lower motor neurons, ALS leads to decreases in muscle strength and contraction rate (DePaul & Brooks, 1993; Dworkin, Aronson, & Mulder, 1980; Langmore & Lehman, 1994), which result in reduced and slowed movement (Green, Wang, & Wilson, 2013; Yorkston et al., 1993). Therefore, we examined both the displacement (i.e. D_max_open, D_max_close, D_mean, Slp1, Slp2, Slp_d) and velocity (i.e. Max_vel) of lip movement to indicate the effect of motor neuron loss on articulatory movement. Among the six variables quantifying displacement, three measured the overall range of movement (i.e. D_max_open, D_max_close, D_mean); and three measured time-dependent changes in movement extent (i.e. Slp1, Slp2, Slp_d), which tested the effect of disease-induced fatigue on the extent of articulatory movement when the articulatory system was operating toward its maximum capacity.

In addition to changes in movement displacement and velocity, prior work also reported decreases in the spatiotemporal variability of articulatory movements in individuals with mild-to-moderate dysarthria due to ALS, which were moderately associated with speech severity (Kuruvilla-Dugdale & Mefferd, 2017; Mefferd, Pattee, & Green, 2014). As spatiotemporal variability is inversely related to stiffness, decreased spatiotemporal variability might indicate active control over articulatory stiffness, which serves as an adaptive strategy to improve articulatory precision (Kuruvilla-Dugdale & Mefferd, 2017). To determine the physiologic factors underlying the changes in spatiotemporal variability, we assessed the spatial and temporal variability of lip movement separately because spatial and temporal articulatory features were found to be differentially affected by ALS (Langmore & Lehman, 1994). As a result, to quantify spatial and temporal variability, six spatial variables (i.e. Sse1, Sse2, Sse, Scanning_d1, Scanning_d2, Scanning_d) and three temporal variables (i.e. Tsd, Tsd_cv, Jitter) were adopted from the speech science literature (Ramig et al., 1990; Tjaden & Watling, 2003; van den Berg, 2018; Wang et al., 2009; Westbury & Dembowski, 1993). Also, two variables quantified overall spatiotemporal variability (i.e. STI, D_dtw), reflecting disease effects on articulatory control.

To determine an individual’s articulatory capacity to perform the maximum rate task, we examined the frequency/rate (i.e. Freq/AMR) of syllable repetitions and three other relevant variables (i.e. T_mean, Dur, Ncyc). The rate of syllable repetitions assessed the neuromotor control of temporal coordination of muscle activation, which is different from the speed of articulatory movement (Westbury & Dembowski, 1993). The number of syllables in AMR was found to correlate with the progression of bulbar ALS in our previous studies (Rong, Yunusova, & Green, 2015; Rong, Yunusova, Wang, et al., 2015). The combination of these variables assessed individuals’ ability to fulfill the maximum rate requirements in the AMR task.

Data analysis

Assessing the validity of the algorithmic approach.

Twenty files (15 for ALS and 5 for controls) were randomly selected and manually analysed. Specifically, each file was first manually parsed into individual cycles; and the same movement analysis was applied to the manually parsed cycles, which yielded 23 AMR variables for each manually analysed file. These variables were compared with the automatically-extracted AMR variables using Pearson’s correlations. In addition, the mean percentage of error between the automatic and manual measurements (i.e.$\%error=\left|V_{auto}-V_{manual}\right|/V_{manual}$) was calculated for each variable.

Assessing the diagnostic value of AMR features.

To assess the efficacy of the AMR variables for detecting bulbar motor involvement in ALS, we compared participants with ALS and healthy controls. First, the effect size for the group difference for each AMR variable (as shown in Table S1) was calculated using Cohen’s D. Receiver operating characteristic (ROC) analyses were then applied to assess the diagnostic efficacy of each AMR variable for differentiating participants with ALS from healthy controls. Based on the analyses, the area under the curve (AUC) was calculated as an index of overall diagnostic efficacy. The sensitivity and specificity of each variable was then calculated at an optimal cut point, which was determined for each variable to maximise the value of (sensitivity² + specificity²).

Detecting early bulbar motor involvement based on AMR variables.

Each variable among three subgroups – early ALS, late ALS, and healthy controls – was compared using repeated measure ANOVA followed by lsmeans pairwise comparisons (R 3.3.1). Because this study is exploratory in nature, α < 0.05 was used for all group comparisons.

Association with progression of bulbar motor involvement.

To screen the AMR variables for their potential efficacy as an index of bulbar motor impairment severity, Pearson’s correlation coefficients were calculated between each variable and speaking rate in participants with ALS. For each variable showing a significant correlation with speaking rate, a linear mixed-effects (LME) model (lmer, R 3.3.1) was applied, which included a fixed-effect that predicted speaking rate based on the AMR variable and a random effect that accounted for subject effects. Based on the LME models, all AMR variables that provided more than 80% statistical power for predicting speaking rate at a level of p < 0.05 were regarded as predictors of the progression of bulbar motor involvement in ALS.

Prediction of impending speech intelligibility decline.

To screen the AMR variables for their predictive power of estimating the time course of the nonlinear speech intelligibility decline (Rong et al., 2016), Spearman’s rank correlation coefficients were computed to determine the correlation between each AMR variable and speech intelligibility in participants with ALS. For each AMR variable showing a significant correlation with speech intelligibility, a bi-phasic nonlinear mixed-effects (NLME) model (nlmefit, MATLAB R2016a) was applied, which was comprised of a fixed effect that predicted speech intelligibility based on each AMR variable and a random effect that accounted for subject effects. Based on these NLME models, all AMR variables that provided more than 80% statistical power for predicting intelligibility at a level of p < 0.05 were regarded as predictors of speech intelligibility decline in ALS.

Result

Validity of the algorithmic approach

Based on Pearson’s correlations, automatic and manual measures of all AMR variables were highly correlated across the 20 selected files (r > 0.99, p < 0.001). The percentage of error between automatic and manual measurements ranged between 0 and 1.73% across all variables, suggesting that the algorithmic approach was valid for extracting variables that accurately quantified spatial, temporal, and spatiotemporal deficits of lip movement during the AMR task.

Diagnostic value of AMR variables

Among all 23 AMR variables, high correlations were detected between Freq and AMR (r = 0.98, p < 0.001), and between Freq and T_mean (r = −0.9, p < 0.001), which were consistent with the theoretical relationships among these three variables (i.e. Freq = AMR = 1/T_mean) and suggested that the temporal information about maximum syllable repetitions was captured by any of these variables. In the following, we only discussed the results for Freq given its simplicity of calculation (i.e. Freq was automatically extracted from the whole AMR sequence, whereas AMR and T_mean required parsing of individual cycles).

Sensitivity for detecting bulbar motor involvement.

The effect sizes for the difference of each AMR variable between the participants with ALS and healthy controls are shown in Figure 1. Among all 21 variables, four showed a large effect size (Cohen’s D > 0.8), including three temporal variables (Jitter: Cohen’s D = 0.99; Freq: Cohen’s D = 1.20; Dur: Cohen’s D = 0.97), and one overall variable (Ncyc: Cohen’s D = 1.34). The AUC, sensitivity, and specificity for all AMR variables are provided in Table S2. Figure 2 shows the ROC curves for 12 AMR variables with AUC > 0.5, including Slp_d, Sse1, Sse, Scanning_d1, Scanning_d, D_max_open, D_max_close, D_mean, Tsd, Tsd_cv, Jitter, and D_dtw. Among these 23 variables, Jitter showed the largest AUC and sensitivity.

Figure 1.

Spider chart that shows the effect sizes (Cohen’s D) of group differences between the participants with ALS and healthy controls based on 21 AMR variables.

Figure 2.

Receiver operating characteristic curves for 12 AMR variables with an AUC >0.5.

A further examination of the differences of all AMR variables among the three subgroups – early ALS, late ALS, and controls – suggested that 11 variables (i.e. Jitter, Freq, Dur, Ncyc, Sse2, Sse, Scanning_d2, D_max_open, D_max_close, D_mean, Max_vel) showed a significant difference across the three subgroups. The bar graphs of these 11 AMR variables and the two speech measures (i.e. speaking rate, speech intelligibility) are shown in Figure 3 and the statistics for all variables are provided in Table 1.

Figure 3.

Bar graphs of 11 AMR variables and two speech measures (i.e. speaking rate, speech intelligibility) showing significant differences among early ALS, late ALS, and healthy controls.

Table 1.

Mean, standard deviation, and comparative statistics for all AMR variables, speaking rate, and speech intelligibility (Refer to note for Table S1 for definitions of terms).

Variables	Control		Early ALS		Late ALS		ANOVA		Post-hoc pairwise comparisons
	Mean	SD	Mean	SD	Mean	SD	F	p	Control vs. Early ALS	Control vs. Late ALS	Early ALS vs. Late ALS
Slp1	0.01	0.15	0	0.2	-0.07	0.19	0.58	0.56
Slp2	-0.03	0.07	-0.05	0.09	-0.01	0.09	2.4	0.09
Slp_d	0.03	0.14	0.05	0.21	-0.06	0.17	0.16	0.86
Sse1	1	0.35	1.17	0.32	0.99	0.31	1.88	0.16
Sse2	0.58	0.15	0.58	0.19	0.48	0.17	8.66	< 0.001	n.s.	0.03	< 0.001
Sse	0.83	0.24	0.93	0.24	0.79	0.22	3.59	0.03	n.s.	n.s.	0.03
Scanning_d1	0.98	0.4	1.17	0.4	0.97	0.34	1.09	0.34
Scanning_d2	0.54	0.16	0.53	0.19	0.5	0.14	4.07	0.02	n.s.	n.s.	0.02
Scanning_d	0.76	0.26	0.85	0.25	0.74	0.21	2.13	0.12
D_max_close	10.13	3.13	13.21	3.8	15.89	3.19	3.97	0.02	0.03	0.03	n.s.
D_max_open	10.15	3.11	13.25	3.8	15.87	3.14	3.66	0.03	0.03	0.03	n.s.
D_mean	6.76	2.34	9	2.89	11.69	3.02	3.79	0.03	0.04	0.02	n.s.
Freq	5.8	0.92	4.5	1.07	2.39	0.41	47.9	< 0.001	0.001	< 0.001	< 0.001
Dur	18.66	5.61	14.3	5.82	13.27	4.24	8.04	< 0.001	0.03	< 0.001	0.03
Jitter	0.18	0.02	0.24	0.05	0.44	0.07	101.13	< 0.001	< 0.001	< 0.001	< 0.001
Tsd	0.03	0.02	0.03	0.02	0.04	0.03	2.34	0.10
Tsd_cv	0.15	0.12	0.15	0.13	0.1	0.1	1.08	0.34
Max_vel	165.63	39.34	167.77	45.09	141.43	26.35	4.1	0.02	n.s.	n.s.	0.02
STI	10.08	2.77	8.13	2.37	6.71	2.38	2.52	0.08
D_dtw	1.52	1.11	2.6	1.88	2.71	1.66	2.09	0.13
Ncyc	98.71	27.4	58.95	30.65	27.71	9.07	27.24	< 0.001	< 0.001	< 0.001	< 0.001
Speaking rate	174.78	26.63	166.05	42.85	106.73	44.85	17.55	< 0.001	n.s.	< 0.001	< 0.001
Intelligibility	99.57	1.06	97.74	3.35	66.89	31.55	35.67	< 0.001	n.s.	< 0.001	< 0.001

Among all 11 AMR variables as shown in Figure 3, seven variables showed monotonic changes throughout the early and late stages of ALS. These variables include (1) three spatial variables (i.e. D_max_open, D_max_close, D_mean), which showed a progressive increase; (2) three temporal variables (i.e. Freq, Dur, Jitter), among which Freq and Dur showed a progressive decrease and Jitter showed a progressive increase; and (3) one overall variable (i.e. Ncyc), which showed a progressive decrease.

The pattern of change over the early and late stages of ALS appeared to be non-monotonic for the remaining four AMR variables (i.e. Sse2, Sse, Scanning_d2, and Max_vel). As shown in Figure 3, the disease-related changes in these variables accelerated significantly in the late stages of ALS. The changes in speaking rate and speech intelligibility had a similar pattern. Both measures showed (1) no change in early ALS relative to controls, and (2) a significant decrease in late ALS relative to both controls and early ALS.

Association with the progression of bulbar motor involvement

Based on Pearson’s correlations and the fit of the LME models (i.e. as indicated by conditional R² and statistical power), four of the 21 AMR variables showed more than 80% statistical power for predicting speaking rate at a level of p < 0.05. These variables included (1) two temporal variables: Jitter (r = −0.47, p < 0.001; R² = 0.75), Freq (r = 0.56, p < 0.001; R² = 0.72), (2) one spatiotemporal variable: Max_vel (r = 0.26, p = 0.004; R² = 0.71), and (3) one overall variable: Ncyc (r = 0.41, p < 0.001; R² =0.67). The scatter plots in Figure 4e–h showed the linear relationships between each of these variables and speaking rate.

Prediction of speech intelligibility decline.

Based on Spearman’s rank correlations and the NLME models, three of the 21 AMR variables showed more than 80% statistical power for predicting speech intelligibility at a level of p < 0.05. These variables included (1) two temporal variables: Jitter (rho = −0.38, p < 0.001; R² = 0.92), Freq (rho = 0.52, p < 0.001; R² = 0.94) and (2) one overall variable: Ncyc (rho = 0.50, p < 0.001; R² = 0.95). The nonlinear relationships between each of these variables and speech intelligibility were shown in Figure 4a–c, all of which were characterised as having two phases and modeled by the NLME approach.

As shown in Figure 4a–c, the three NLME models based on Jitter, Freq, and Ncyc respectively as well as the NLME model based on AMR, which was used for disease stage stratification, uniformly suggested that (1) during the early stages of the disease, speech intelligibility remained relatively high (i.e. > 96%), whereas (2) during the late stages of the disease, speech intelligibility declined rapidly, which was significantly correlated with the decrease in Freq, the increase in Jitter, the decrease in Ncyc, and the decrease in AMR. The two disease stages were distinguished by the following cutoffs for each of these variables: Freq < 2.79 (cycles/s); Jitter > 0.38 (s); Ncyc < 35 (cycles); AMR < 2.64 (cycles/s).

To validate the disease stage stratification in this study (i.e. based on AMR), the results were compared with stratifications based on Freq, Jitter, and Ncyc. Comparisons suggested strong agreement with stratifications based on Freq (98.58% of agreement) and Jitter (97.16% of agreement), and moderate agreement with stratification based on Ncyc (76.6% of agreement).

Discussion

In this study, we estimated the validity and diagnostic value of an algorithmic approach for automatic extraction of lip movement features during an AMR task. Based on the ROC analyses and effect sizes for the difference between participants with ALS and healthy controls, we examined the sensitivity of 23 automatically-extracted AMR features to the presence of ALS. Further, we assessed the efficacy of these features for detecting early bulbar motor involvement by examining the changes of these features in the early stages of ALS relative to the late stages of ALS and healthy controls. We also examined the relation of these features to the time course of bulbar motor involvement and speech intelligibility decline.

Our findings suggested that, compared to the automated acoustic analysis in Wang et al. (2009), which was only able to process two thirds of their acoustic samples, our algorithmic approach based on lip kinematics provided valid measurements for all samples. Overall, lip AMR performance declined relatively early during bulbar motor involvement, and this change (1) effectively differentiated persons with ALS from healthy controls before speech intelligibility and speaking rate started to decline; (2) was associated with the progression of bulbar motor involvement; and (3) predicted the timing of speech intelligibility loss.

Our findings further suggest that the temporal features of lip motor performance were more affected than the spatial features. Among the temporal features, increased temporal variability across cycles of the AMR task was the most sensitive indicator of early bulbar involvement and one of the strongest predictors of the impending speech intelligibility loss. Longitudinal changes in spatial features were not only less evident than temporal features but also appeared to be non-monotonic.

The findings of this study provide preliminary support for the algorithmic approach to quantifying AMR features predictive of bulbar motor and speech decline in ALS. In the future, we will use this algorithmic approach to analyse tongue AMR features as well. With the current findings demonstrating the sensitivity and predictive power of lip AMR features, we expect that, when the technology barrier for tongue movement tracking in clinical environments is resolved, the tongue AMR features will be even more efficacious for clinical bulbar motor assessment.

AMR and articulatory assessment

This study provided evidence that, with appropriate selection of performance measures, an AMR task is efficacious for detecting bulbar motor involvement in ALS and predicting the progression of bulbar motor involvement and speech intelligibility decline across a range of severity. The current findings also suggest that, an adequate representation of articulatory involvement requires features that distinctly represent temporal and spatial aspects of articulatory movement.

Our finding that temporal features were most affected is consistent with previous findings of various neurological disorders including ALS (Langmore & Lehman, 1994), Multiple Sclerosis, and Parkinson’s Disease (Tjaden & Watling, 2003). As shown in Figures 1 & 4, among all spatial and temporal variables, jitter and frequency (1) showed large effect sizes for the difference between participants with ALS and healthy controls, and (2) predicted a significant portion of variance in both speaking rate, representative of the overall bulbar motor decline, and speech intelligibility. The observed decreases in frequency indicated the deterioration of the bulbar neuromotor system in producing rapid, repetitive oral muscle contractions required in the AMR task (Westbury & Dembowski, 1993). The observed increase in temporal variability as captured by the jitter variable further suggested that ALS may negatively affect bulbar neuromotor control when the oromotor system operated near its performance limit in the AMR task.

Different from the temporal features, which showed monotonic changes during disease progression, several spatial and spatiotemporal features showed non-monotonic changes over the early and late stages of ALS. Specifically, spatial variability of lip opening (i.e. Sse2, Scanning_d2), overall spatial variability of lip movement (i.e. Sse), and peak velocity of lip movement (i.e. Max_vel) all showed no to minimal change until the late disease stages in the AMR task. The finding that peak velocity remained relatively high during the early disease stages despite significant decreases in syllable repetition rate confirmed the distinction between articulatory speed and syllable rate as raised by Westbury and Dembowski (1993). However, in contrast to their finding showing that healthy participants tended to achieve faster syllable rates by truncating articulatory gestures, our finding suggested that the habitual, but relatively slow syllable rates of talkers with ALS were associated with articulatory overshoot, an increase of movement extent (as indicated by the three displacement measures in Figure 3).

The disease-related changes in the spatial, temporal, and spatiotemporal features of lip movement as discussed above suggested that the lip motor performance in our participants with ALS during the AMR task (i.e. increased movement extent, decreased syllable rate, and increased temporal variability) was more similar to the performance observed during speech tasks than speechlike tasks in healthy controls (Westbury & Dembowski, 1993). This finding indicates changes in motor control of AMR tasks as a result of the combined impacts of the primary disease effects on lip control and the articulatory demands required by the maximum syllable rate task. With further comparisons of disease effects on the spatial and temporal aspects of speech and AMR performance warranted, the current finding may provide new insights into the theoretical debate about task-related differences in speech and non-speech motor control.

Moreover, the behavioural changes as captured by the spatial and temporal variables in this study might shed light on the mechanisms of articulatory involvement during ALS progression. The early-stage articulatory deficits were primarily characterised by deteriorated neuromotor control of temporal coordination of oral muscles, shown by the decrease of frequency and the increase of temporal variability. The late-stage articulatory deficits were characterised by both the weakness of articulators, shown by the reductions of maximum velocity and spatial variability, and a further deterioration of neuromotor control of temporal coordination. The differential disease effects on the spatial and temporal features of articulatory movements may explain why spatiotemporal measures such as STI and D_dtw failed to detect a group difference. In addition, the deterioration in neuromotor control of temporal coordination and the weakness of articulators detected at different stages of ALS may be associated with different subtypes of dysarthria (e.g. spastic, flaccid), which need to be investigated further.

Early diagnostic markers of bulbar motor involvement in ALS

Due to the rapid progression of ALS after the onset of clinical bulbar signs and symptoms (Green, Yunusova, et al., 2013; Rong, Yunusova, Wang, et al., 2015), early detection of bulbar changes is critical. The results in Figure 3 and Table 1 suggested that frequency, temporal variability, total duration, and number of cycles all showed significant changes during the early stages of ALS, when speaking rate and speech intelligibility were yet unaffected. Specifically, intelligibility remained above 96% and speaking rate was above 150 WPM, while the AMR features were starting to show the effects of bulbar motor involvement. Thus, frequency, jitter, total duration, and number of syllables might be more sensitive to early bulbar changes than current assessments based on patient reported outcomes, clinical ratings, speech intelligibility, or speaking rate (Allison et al., 2017; Green, Yunusova, et al., 2013; Yorkston et al., 1993).

Among the four AMR features, jitter has the largest AUC and highest sensitivity based on the ROC analyses. This finding suggests that the increase in temporal variability was the most robust sign of bulbar motor involvement detected in ALS. Thus, jitter may have the greatest potential as a diagnostic marker of early bulbar motor involvement. Similar findings of increased temporal variability in alternating motion tasks have also been reported in persons with dysarthria due to TBI (Wang et al., 2004), multiple sclerosis (Tjaden & Watling, 2003), and Parkinson’s disease (Tjaden & Watling, 2003). Prior work has also shown that faster than normal rates in persons with ALS induce spatiotemporal instability in speech movements (Mefferd et al., 2014). Additional research is required to determine (1) if the observed change in the temporal variability that was captured by the jitter measure are also evident during functional speech, and (2) if so, its potential impact on intelligibly.

The finding that jitter showed the greatest potential as a diagnostic marker supports our initial motivation to record and analyse lip kinematic data during the AMR task beyond acoustic recording. Lip kinematic data provided important information regarding the onset of AMR cycles. That is, the timing of lip closure during /b/ relative to the end of the preceding vowel /a/ and the beginning of the following /a/ in each AMR cycle. This information is essential for determining cycle duration as needed for calculating jitter but is not available from acoustic recording because of the interaction between articulation and voicing. In a case study by Kent et al. (1991), continue voicing was detected throughout the CV syllable during the AMR task, making it difficult to determine the onset of each syllable based on acoustic recording.

Monitoring of bulbar motor involvement in ALS

In addition to detecting early bulbar involvement, jitter, frequency, and number of cycles were significantly correlated with speaking rate. Because speaking rate decreases linearly with bulbar ALS progression (Rong et al., 2016), the strong linear correlations (Figure 4e–g) between these three AMR features and speaking rate suggest that these AMR features also decline linearly with bulbar ALS progression. The finding of linear decline of syllable frequency is different from the finding of Kent et al. (1991) based on a single case, which suggested that AMR rate did not change consistently during disease progression. This discrepancy is most likely attributed to the methodological differences between the two studies – a group analysis vs. a case study. As shown in Figure 4f, although syllable frequency showed an overall linear decrease with disease progression, there was some variability across individuals. As a result, for an individual case such as in Kent et al. (1991), syllable frequency may not be as sensitive to disease progression as other measures such as jitter. This points to the importance of assessing different aspects of the AMR performance (e.g. jitter, syllable frequency, total number of syllable) at an individual level, which could provide a more comprehensive assessment of bulbar and speech motor involvement across individuals. Consistently, Kent et al. (1991) suggested that a single measure of syllable frequency is not sufficient for capturing the underlying bulbar/speech motor deficits.

Although speaking rate has been used as a clinical index of bulbar disease severity, the AMR features have a distinct advantage over the speaking rate measure. The AMR variables are less affected by cognitive-linguistic processing deficits, which often co-occur with motor deficits in ALS (Goldstein & Abrahams, 2013) and, thus, may be more direct measures of bulbar motor capacity. Given this advantage of the AMR task, jitter, frequency, and number of cycles derived from this task may be used to monitor the progression of bulbar motor involvement clinically.

In addition to monitoring bulbar disease progression, AMR features might also be efficacious for stratifying fast and slow progressors, which has important clinical implications for informing patients about their expected rate of decline and for participant selection in experimental therapeutic trials. Our prior study compared the efficacy of four commonly-used clinical measures in ALS assessment (i.e. ALSFRS-R scores, speech intelligibility, speaking rate, AMR) for stratifying patients based on their disease progression rates (Rong, Yunusova, & Green, 2015). The findings suggested that fast progressors showed significantly lower AMR and faster decline in AMR performance overtime than did slow progressors when all other measures were still within the normal limits (Rong, Yunusova, & Green, 2015). The finding of the current study further indicated that, in addition to the AMR measure, other movement features during the AMR task (e.g. jitter) might provide additional efficacious markers for disease stratification.

Predicting impending speech intelligibility loss in ALS

Speech intelligibility declines relatively late in ALS (Green, Yunusova, et al., 2013; Westbury & Dembowski, 1993). However, monitoring and predicting its decline is essential for bulbar disease management. Monitoring AMR performance on measures of frequency, jitter, and number of cycles may allow the prediction of the impending speech intelligibility changes. Specifically, intelligibility remained above 96% in the early stages of ALS, while (1) frequency decreased from over 5 cycles / sec to 2.8 cycles / sec; (2) jitter increased from less than 0.1 second to 0.38 second; and (3) the number of cycles decreased from over 100 cycles to 35 cycles. These changes in the AMR variables preceded the decline of intelligibility and might serve as early predictors of the late-stage intelligibility decline.

Among the three AMR variables, frequency may have the greatest potential as a clinical predictor of impending speech loss. The measure of frequency is advantageous for its simplicity of calculation, which does not require parsing individual cycles. Another advantage is its robustness to respiratory involvement. Unlike the number of cycles, which may be affected by respiratory involvement (e.g. decreased respiratory volume and slowed articulatory movement could both contribute to decreased number of cycles), frequency provides a time-normalised measure of the average number of cycles / sec, which may minimise the confounding effect of respiratory involvement. Thus, movement frequency may be more sensitive to articulatory involvement than the number of cycles. This suggestion was supported by the disease stage stratification results, which indicated that the stratification based on frequency agreed well with the stratifications based on jitter and AMR, whereas the stratification based on number of cycles agreed less well with the stratifications based on the other variables. These advantages of the frequency measure elevate its potential as a clinical predictor of the needs for assistive communication. With further testing warranted, the current finding suggested that assistive communication device training should occur long before syllable frequency drops to 2.8 cycles / sec, which may signal the impending rapid loss of functional speech.

Future directions

Lip movement tracking has recently become feasible in clinical settings because of the proliferation of low-cost depth-sensing cameras, such as Intell’s RealSense ® (Bandini et al., 2017; Green, Richburg, Markan, & Berry, 2016). The automated analysis is also essential for decreasing the burden of data reduction and allowing for use by non-experts. The findings of this study provide preliminary support for using automated analysis of lip AMR kinematics to assess articulatory involvement in ALS. In the future, we will test the algorithmic approach with movement data from different articulators (e.g. tongue, jaw, lips) in individuals with a greater range of severity of bulbar motor involvement as well as healthy controls.

As disease severity increases, speech intelligibility is expected to decline rapidly, making transcription more challenging. Therefore, we will have multiple listeners transcribe the speech samples and assess the reliability of speech intelligibility. Our long-term goal is to establish an assessment standard that informs both the norms and the milestones of disease progression, aiming at assessing baseline performance and tracking disease progression.

Conclusion

A set of speech movement features was automatically extracted from the lip movements of individuals with ALS during the alternating motion rate task using an algorithmic approach. Among these features, syllable frequency and jitter that measured temporal variability were most affected. Jitter might potentially serve as an early diagnostic marker of bulbar motor involvement and syllable frequency might serve as an efficacious predictor of the impending functional speech loss in ALS. A comparison of the temporal and spatial AMR features suggested that early articulatory involvement was primarily attributed to the deterioration of overall neuromotor control of the temporal coordination of oral muscles. Articulatory weakness was not detected until later stages of the disease when bulbar motor neuron loss progressed to a critical level, which led to rapid, precipitous declines of speech intelligibility.