Automatic Speech Recognition in Primary Progressive Apraxia of Speech

Abstract

Introduction:

Transcribing disordered speech can be useful when diagnosing motor speech disorders such as primary progressive apraxia of speech (PPAOS), who have sound additions, deletions, and substitutions, or distortions and/or slow, segmented speech. Since transcribing speech can be a laborious process and requires an experienced listener, using automatic speech recognition (ASR) systems for diagnosis and treatment monitoring is appealing. This study evaluated the efficacy of a readily available ASR system (wav2vec 2.0) in transcribing speech of PPAOS patients to determine if the word error rate (WER) output by the ASR can differentiate between healthy speech and PPAOS and/or among its subtypes, whether WER correlates with AOS severity, and how the ASR's errors compare to those noted in manual transcriptions.

Method:

Forty-five patients with PPAOS and 22 healthy controls were recorded repeating 13 words, 3 times each, which were transcribed manually and using wav2vec 2.0. The WER and phonetic and prosodic speech errors were compared between groups, and ASR results were compared against manual transcriptions.

Results:

Mean overall WER was 0.88 for patients and 0.33 for controls. WER significantly correlated with AOS severity and accurately distinguished between patients and controls but not between AOS subtypes. The phonetic and prosodic errors from the ASR transcriptions were also unable to distinguish between subtypes, whereas errors calculated from human transcriptions were. There was poor agreement in the number of phonetic and prosodic errors between the ASR and human transcriptions.

Conclusions:

This study demonstrates that ASR can be useful in differentiating healthy from disordered speech and evaluating PPAOS severity but does not distinguish PPAOS subtypes. ASR transcriptions showed weak agreement with human transcriptions; thus, ASR may be a useful tool for the transcription of speech in PPAOS, but the research questions posed must be carefully considered within the context of its limitations.

Supplemental Material:

https://doi.org/10.23641/asha.26359417

Primary progressive apraxia of speech (PPAOS) is a neurodegenerative motor speech disorder that is characterized by slow speech rate; abnormal prosody; and the distortion, substitution, addition, and prolongation of speech (Duffy, 2006; Josephs et al., 2012). Although apraxia of speech (AOS) can occur with the nonfluent/agrammatic variant of primary progressive aphasia (Gorno-Tempini et al., 2011), PPAOS can also be the sole manifestation of neurodegeneration (Duffy, 2006; Josephs et al., 2012, 2013), with at least two phenotypical presentations. Its phonetic subtype is characterized by predominance of distorted sounds and sound substitutions, deletions, and additions, whereas its prosodic subtype shows dominance of slow and prosodically segmented speech (with increased pauses between words and syllables of multisyllabic words). When there is the presence of both phonetic and prosodic features but the predominance of neither, the term “mixed” is applied; mixed PPAOS is typically associated with severity (mild or severe) but may also be associated with lack of consensus agreement about the predominating speech characteristic (Josephs et al., 2013; Utianski et al., 2018, 2023). The phonetic and prosodic subtypes may have different clinical trajectories and underlying pathologies. Patients with phonetic PPAOS are more likely to develop more severe aphasia and have underlying corticobasal degeneration, whereas patients with prosodic PPAOS are more likely to develop additional motor-related symptoms and have underlying progressive supranuclear palsy (Josephs et al., 2021).

Evaluating spoken expression is an important part of the diagnosis of speech and language disorders because it captures differences between what is produced by affected versus healthy populations. One method for doing this in a quantifiable manner is transcribing speech, writing it out either orthographically or phonetically, to capture both the content (language) and accuracy of production (speech). The speech errors, or deviations present in disordered speech, may not be fully captured by traditional scoring methods of clinical tests, highlighting the underutilized usefulness of speech samples for clinical use. Typically, speech samples are manually transcribed by a trained individual. Because this process can be laborious, subjective, and subject to interrater variance, interest in using automatic speech recognition (ASR) systems, which transcribe speech to text, has grown.

Many ASR systems are readily available for use. The accuracy of current ASR models for healthy populations is high, with word error rates (WERs) of less than 5% (i.e., greater than 95% accuracy; Szymański et al., 2020); the definition of WER differs between studies, but we use WER to mean number of word additions, deletions, and substitutions divided by the total number of intended words. However, since these models are trained on typical, nondisordered speech, their efficacy is limited when used in clinical populations. Work to date has evaluated the influence of dysarthria on WERs, which tend to be high. For example, De Russis and Corno (2019) compared ASR accuracy across three easily accessible ASR systems with speech from patients with dysarthria; they showed that the ASR systems had WERs of between 78% and 89% and a low percentage of correctly transcribed sentences (0%–1.2%). Other studies investigating WER in disordered speech have found similar results (Coleman & Meyers, 1991; Raghavendra et al., 2001; Rudzicz, 2007, 2010). Accuracy is better when using personalized ASR models, but this requires a customized model for each individual (Green et al., 2021), which would preclude across participant comparisons. Although a personalized model could be invaluable to individuals with speech disorders in their daily interactions and communication, it is not practical for clinical use, where analysis relies on accurate or exact transcriptions of speaker utterances rather than the communicative intent of the utterance. One study to date evaluated ASR in AOS and aphasia from stroke and reported excellent agreement between human transcriptions and ASR output based on recordings of read sentences (Jacks et al., 2019).

Even within typical, nondisordered speech, variation in the speech signal can affect the output of ASR systems both in the overall accuracy and in the types of inaccuracies made (Benzeghiba et al., 2007; Mustafa et al., 2015; Wiepert et al., 2023); we will use the term “error” to reflect speech errors in the acoustic signal (e.g., saying “aluminalum” for aluminum) and “inaccuracy” for ASR output that does not match what was produced (e.g., “catapulp” as output for accurately produced catapult). Deviations in prosody or word stress and in speech rate result in higher WERs (Goldwater et al., 2010; Greenberg & Chang, 2000), but sound errors in the speech such as phoneme substitutions, deletions, or insertions are even more likely to negatively affect ASR accuracy (Greenberg & Chang, 2000). These different speech deviations also yield different types of ASR inaccuracies, such as word boundary errors versus phoneme differences, the nature of which is not fully captured in a measure like WER.

Disordered speech tends to show more variation in articulation, beyond those accounted for by healthy, typical speech variation, which might explain why ASR accuracy is so low for disordered speech. Despite this, ASR accuracy has correlated with overall speech intelligibility in populations with and without speech disorders, where reduced intelligibility is correlated with higher WERs (Jacks et al., 2019; McHenry & LaConte, 2010; Riedhammer et al., 2007). ASR systems have also been used to grade the severity of various speech impairments. For example, Tu et al. (2016), Mustafa et al. (2015), and Christensen et al. (2013) showed that the accuracy of an ASR system positively correlated with the clinical judgment of dysarthria. Additionally, Gutz et al. (2022) found that ASR was less successful than humans in word recognition of speakers with dysarthria and was inadequate for grading dysarthria severity. It is possible that the nature of speech deficits (e.g., greater voice than articulation disturbances) and/or the individual ASR algorithm influenced results, although more details about the algorithms themselves would be necessary to fully understand the interaction between ASR type (e.g., deep learning–based or not; language-layered or not) and speech difficulties.

Recent work has shown that phonetic and prosodic PPAOS can be distinguished based on the number and type of speech errors produced in a relatively simple word repetition task when transcribed manually (Tetzloff et al., 2024). No study to date has investigated the efficacy of ASR in patients with PPAOS in accurately transcribing their speech productions (and not necessarily just the intended productions as in previous studies of intelligibility). PPAOS is an ideal disease model for evaluating how AOS influences ASR transcriptions, without confounds of aphasia or dysarthria. If effective, ASR could help in the diagnosis of PPAOS and differentiation of its subtypes, which would lessen the need for manual transcriptions of speech samples. Given that (a) PPAOS subtypes present with different types of speech errors and (b) that ASR systems have been shown to make different proportions and types of inaccuracies based on the deviations in the speech stream, we sought to evaluate how ASR accuracy and types of inaccuracies reflect PPAOS and/or its subtypes. An automatic transcription-based scoring could be a useful and efficient supplemental tool for clinicians in diagnosing PPAOS and determining AOS subtype that does not rely on expert perception and the limitations of manual transcriptions. ASR may also provide an objective index of severity, disease progression, or effects of medical or behavioral intervention. As an initial step toward those ends, the first goal of this study was to evaluate the relationship between WER from a readily available ASR system and PPAOS severity; it was hypothesized that more severe AOS would correlate with higher WERs (i.e., less accurate ASR transcriptions than for neuronormal speakers). The second goal was to determine if the inaccuracies noted by ASR transcription align with PPAOS subtype diagnosis; we hypothesized that ASR would generate more word boundary inaccuracies for prosodic PPAOS and more segment-level (i.e., sound) inaccuracies for phonetic PPAOS, consistent with their predominant speech deficits.

Method

Participants

Forty-five patients with PPAOS (21 mild, 15 mild–moderate, seven moderate-marked, one severe) and 22 healthy controls were included in this study, all of whom were recruited into a National Institutes of Health–funded study through the Neurodegenerative Research Group at Mayo Clinic, Rochester. All patients underwent a detailed speech-language examination by a speech-language pathologist (SLP), described below, and were given a clinical diagnosis of phonetic, prosodic, or mixed PPAOS via consensus among at least two experienced SLPs who are board certified by the Academy of Neurological Speech and Communication Disorders and Sciences. Patients must have had isolated PPAOS without evidence of dysarthria or aphasia to be included in this study. All participants were native speakers of American English. This study was approved by the institutional review board at Mayo Clinic (Approval Numbers 17–010087 and 17–002468), and all participants provided informed consent of their participation.

Speech and Language Evaluation

All patients underwent a speech-language battery that included the Western Aphasia Battery–Revised, which is composed of various tasks that evaluate lexical content, fluency, repetition, naming, and language comprehension and yielded an aphasia quotient score (WAB-AQ; Kertesz, 2007). Additional tasks evaluated motor speech functioning, including vowel prolongation, speech alternating motion rates (e.g., rapid repetition of a single syllable “puh-puh-puh”), speech sequential motion rates (e.g., rapid repetition of three syllables “puh-tuh-kuh-puh-tuh-kuh”), and word and sentence repetition. The Apraxia of Speech Rating Scale–Version 3 (ASRS-3; Duffy et al., 2023; Strand et al., 2014) and Motor Speech Disorders Severity Rating Scale (MSDRS; adapted from Hillel et al., 1989, and Yorkston et al., 1993) were also scored. The ASRS-3 is a rating scale that quantifies the frequency and severity of different characteristics that are associated with PPAOS and has been shown to correlate with clinical judgments of AOS severity (Duffy et al., 2023; Strand et al., 2014). Items 1–4 capture speech problems associated with the phonetic subtype, and Items 5–8 capture the prosodic subtype (Utianski et al., 2018). The ASRS-3 total score was used as a proxy of PPAOS severity for this study. The MSDRS (adapted from Hillel et al., 1989; Yorkston et al., 1993), is a scale that assesses success of communication (i.e., functional impairment) and has been used as a measure of speech intelligibility in PPAOS (Utianski et al., 2020). These tasks were video-recorded, and a PPAOS diagnosis and subtype designation was made via consensus among at least two SLPs (Utianski et al., 2018).

For the experimental task, patients and controls were asked to produce three repetitions of 13 words that varied in length and phonetic complexity as part of a bigger clinical research study (see Appendix for word list); these words were taken from the Articulatory Error Score word list (Duffy et al., 2015). A word repetition task was selected as it is thought less influenced by language and/or cognitive difficulties and highlights the motor speech difficulties. The SLP said the word once, and participants then repeated the word 3 times in succession. There were three patients who did not produce any repetitions of one word, for a total of 1,746 word repetitions by participants with PPAOS and 936 word repetitions by control participants. These word repetitions were manually transcribed by author KAT, a trained phonetician with extensive transcription experience, using the International Phonetic Alphabet (IPA), and sound substitutions or distortions, additions, and deletions were totaled, in addition to within-word segmentation errors. For sound substitutions and distortions, the IPA symbol closest to the produced sound was used, along with any necessary diacritics for indicating aspiration or palatalization., and so forth, although, such variations were not considered phonetic errors. Segmentations were identified in accordance with clinical judgments as a longer-than-expected duration or pause between syllables. Given that they were performing an isolated word repetition task, segmentations between words were not considered as errors; each segmentation within a word was considered a separate error. The sound substitutions or distortions, additions, and deletions were considered “phonetic” errors, and between-syllable segmentations were considered “prosodic” errors.

The number of phonetic and prosodic inaccuracies recorded from the ASR was used to compare phonetic, prosodic, and mixed PPAOS versus controls to determine if ASR output can differentiate AOS from nonimpaired speech and distinguish among PPAOS subtypes as well as human transcription can (Tetzloff et al., 2024). The accuracy of the ASR for detection of speech errors was also measured by comparing the number of phonetic and prosodic errors across all participants to the same types of errors noted with manual transcriptions.

ASR System

For the ASR system, based on prior research and the desire for a free and publicly available package, we used wav2vec 2.0 from HuggingFace to obtain word transcriptions (Baevski et al., 2020; Sowjanya, 2021); namely, this system has been shown to reflect more accurately what was said at a phoneme level in disordered speech, particularly in cases of PPAOS (Wiepert et al., 2023). wav2vec 2.0 is a type of self-supervised speech model, meaning that the model is first trained with a large quantity of unlabeled speech data with a goal of learning general representations of speech and then further fine-tuned with labeled data for a particular task. In the case of wav2vec 2.0, the model is fine-tuned for ASR. wav2vec 2.0 learns to recognize speech at the subword level, predicting discrete speech units (i.e., graphemes/characters) across an audio file. The model used learns English graphemes (a–z) along with tokens for padding, utterance beginning and end, unknown sounds, apostrophe, and word boundaries. One of these speech units is predicted for every ~20-ms increment, with consecutive repeated tokens eventually being merged. The process of transcription is that an audio file is fed first to a feature encoder, which serves as input to the main wav2vec 2.0 transformer model. The output of the model is a series of probabilities for each speech token occurring at a timestep (i.e., the output size is [batch size, number of time steps, number of tokens − 32]). This is then decoded into a transcription by getting the highest probability token at each timestep and processing to remove repeated tokens where necessary (Baevski et al., 2020).

The ASR model selected does not include a language model, meaning that the transcriptions can include nonwords. We intentionally selected such a model so that it would be better at capturing what was said by the patient and not predict what was intended to be said. The same word repetitions were given to the ASR system that were transcribed by hand using the above-described approach. Individual .wav files of all words concatenated were submitted to the ASR, each containing the three productions of the 13 words, with each different word separated by a pause. We combined all transcriptions for each participant, and both automatically and manually calculated the sum of inaccuracies. The automatic sum was calculated as the WER, which is calculated as the sum of word errors (word deletions, insertions, substitutions) against a target transcription, which, in our case, should contain 39 total words (three repetitions of 13 words). A score of 1 or higher indicates more inaccuracies than the number of words in the target (> 39 errors). The ASR transcriptions were then used to manually tally the number of phonetic inaccuracies (sound substitutions, additions, and deletions), as well as prosodic inaccuracies (within-word segmentations), in the same manner that was done with the manual transcriptions; in other words, all sound additions, substitutions, and deletions were tallied as “phonetic inaccuracies” and segmentation inaccuracies were tallied as “prosodic inaccuracies.” To evaluate the relative proportion of errors, we calculated an inaccuracy predominance score [(phonetic – prosodic inaccuracies) / (phonetic + prosodic inaccuracies)].

Reliability

Ten percent of the total word productions for both patient and controls were independently hand-transcribed and evaluated by author RLU (an experienced SLP); there was 96% agreement between transcribers for the number and type of errors produced, and the Cohen's kappa was .87, which indicates almost perfect agreement (Landis & Koch, 1977), as previously reported in Tetzloff et al. (2024). Ten percent of the ASR transcriptions for both patients and controls were also independently coded by a second transcriber (R.L.U.), a board-certified SLP. There was 97.4% point-to-point agreement between K.A.T. and R.L.U. for the number and types of errors produced. The Cohen's kappa coefficient was calculated for interrater reliability for both phonetic and prosodic inaccuracies; the coefficient was .94 for phonetic inaccuracies and .98 for prosodic inaccuracies, which signifies almost perfect agreement (Landis & Koch, 1977).

Statistics

Analyses of variance were used to compare all PPAOS groups and controls on continuous demographic and clinical data; Fisher's exact tests were used to compare binary variables.

A spearman rank correlation was used to correlate ASRS-3 scores (i.e., AOS severity) with the WERs from the ASR output. A linear regression was then used to evaluate if WER alone could distinguish PPAOS subtype from one another and controls. The area under the operating curve (AUROC) was also calculated to evaluate the reliability of WER being able to differentiate between patients and controls, along with the point at which maximal sensitivity and specificity is achieved. Additionally, a spearman rank correlation was run to compare the proportion of words with errors from the human transcriptions (i.e., the number of words with at least one error divided by the total number of words) to the WER from the ASR.

Linear mixed-effects regressions were run to compare the total number of inaccuracies recorded by the ASR transcription, as well as phonetic and prosodic inaccuracies separately. For each of these models, the total number of phonetic or prosodic inaccuracies were used as the dependent variable, with PPAOS subtype as the predictor. Random by-participant intercepts and word-given-repetition slopes were included in each of the models. Additionally, an inaccuracy predominance score was calculated for each participant, which was the number of phonetic errors minus the number of prosodic errors, divided by the total number of errors. This score was also compared between groups using a linear regression. All p values reported are model outputs, but those marked with an asterisk remained significant after correction for multiple comparisons using false discovery rate correction, with all 13 p values across Tables 1 and 2 included in this correction. Cohen's Kappa coefficients were also calculated to compare the number of phonetic and prosodic errors counted in human transcriptions against the phonetic and prosodic inaccuracies of the ASR transcriptions.

Table 1.

Demographics and clinical data of all participants.

Feature	Phonetic (13)	Prosodic (23)	Mixed (9)	All PPAOS (45)	Controls (22)	p value
Age	60.9 (57.2, 68.3)	75.4 (70.8, 79.5)	73.7 (69.2, 74.1)	69.8 (62.3, 76.1)	66.3 (62.6, 69.9)	< .0001
Disease duration	2.5 (2.3, 6.8)	3.4 (2.8, 4.9)	5.0 (3.8, 5.8)	3.8 (2.5, 5.6)	NA	.41
Sex	46% female	74% female	33% female	58% female	64% female	.16
WAB-AQ	98.8 (96.6, 99.2)	98.0 (97.0, 99.1)	97.2 (96.0, 98.8)	98.0 (96.6, 99.2)	NA	.85
ASRS-3	14 (8, 23)	16 (12, 18)	14 (11, 28)	15 (11, 22)	NA	.83
ASRS-Phonetic	7 (4, 12)	4 (2.5, 4)	4 (4, 10)	4 (3, 7)	NA	.0005
ASRS-Prosodic	4 (4, 6)	7.25 (5, 10)	4 (4, 10)	7 (4, 10)	NA	< .0001
MSDRS	6 (6, 8)	7 (6.25, 7.75)	7 (5, 8)	6.7 (7, 8)	NA	.93

Note. Data are shown as median (Q1, Q3). Values in bold indicate a significant difference across all groups. PPAOS = primary progressive apraxia of speech; ASRS-3 = Apraxia of Speech Rating Scale–Version 3; WAB-AQ = Western Aphasia Battery Aphasia Quotient score; MSDRS = Motor Speech Disorders Severity Rating Scale; NA = not applicable.

Table 2.

The mean WER for all participants, along with the mean number of total ASR inaccuracies, phonetic inaccuracies, and prosodic inaccuracies by group, for those who underwent error analysis.

Variable	Phonetic	Prosodic	Mixed	All PPAOS	Controls
WER – all participants	0.89 (0.48)	0.87 (0.33)	0.90 (0.49)	0.88 (0.40)	0.33 (0.09)
Mean inaccuracies per word	1.87 (2.14)	1.51 (1.70)	1.73 (2.27)	1.66 (1.96)	0.21 (0.52)
Mean phonetic inaccuracies per patient	69.00 (44.98)	47.67 (26.99)	60.17 (39.37)	NA	5.64 (5.33)
Mean prosodic inaccuracies per patient	4.3 (5.03)	7.33 (5.87)	8.00 (9.36)	NA	3.14 (1.83)
(Phonetic – prosodic inaccuracies)/(phonetic + prosodic inaccuracies)	0.88 (0.11)	0.71 (0.19)	0.83 (0.13)	0.78 (0.17)	0.10 (0.64)

Note. Data are shown as mean (standard deviation). Values in bold indicate that controls had a significantly lower value than all PPAOS groups. There are no statistically significant differences among PPAOS subgroups. WER = word error rate; ASR = automatic speech recognition; PPAOS = primary progressive apraxia of speech; NA = not applicable.

Results

Demographics

The demographic and clinical characteristics of the participants are summarized in Table 1. Overall, the phonetic PPAOS group was significantly younger than the prosodic PPAOS, mixed PPAOS, and control groups. There were no differences between PPAOS subtypes on disease duration, sex, WAB-AQ, ASRS-3, or MSDRS scores, indicating well matched groups on those variables. Consistent with the gestalt clinical judgment of subtype, the phonetic group had a higher score on the portion of the ASRS-3 reflecting phonetic-type errors, whereas the prosodic group had a higher score on the portion of the ASRS-3 reflecting prosodic-type errors. Patients overall, and across groups, had mild–moderate AOS.

ASR in PPAOS Speech

The mean WER rate, along with the mean number of total inaccuracies, phonetic inaccuracies, and prosodic inaccuracies by group, transcribed by the ASR system, can be seen in Table 2. The WER was correlated with the ASRS-3 total scores for patients with PPAOS, which showed a significant correlation (r = .53, p = .0004; see Figure 1).

Figure 1.

Scatterplot of Apraxia of Speech Rating Scale–Version 3 (ASRS-3) total scores, scored by experienced speech-language pathologists, against the word error rate (WER) output, derived from the automatic speech recognition system.

A linear regression revealed that controls had a significantly better (i.e., lower) WERs (mean WER of 0.33) than the PPAOS subtypes and the PPAOS group as a whole (mean WER of 0.88; p < .0001), but there were no differences among PPAOS subtypes based on WER (p > .90). Additionally, the AUROC was .96 (see Figure 2), indicating that WER results in near perfect discrimination between controls and patients with PPAOS (with comparable results for separately identifying phonetic and prosodic patients); the threshold that maximized group classification accuracy (maximizes Youden's Index) was a WER of 0.44. The WER from the ASR was also correlated with the proportion of words produced with errors from the human transcriptions. This yielded an r value of .71 and a p value of < .0001.

Figure 2.

Receiver operating characteristic curve for controls against patients with primary progressive apraxia of speech, using word error rate.

Inaccuracy Analysis

Although all participants' recordings were transcribed by the ASR, only 34 patients with PPAOS (75.6%, 11 phonetic, 17 prosodic, six mixed) and 22 controls (100%) were included in the analysis of inaccuracies transcribed, as the output was indiscernible for the remaining patients (see Appendix for example) and inaccuracies could not reliably be tallied for these patients; this was, in part, but not entirely, attributable to the severity of their motor speech disorder (see Supplemental Material S1). All patients excluded from inaccuracy analysis had more than mild AOS; six of the eleven unintelligible outpatients had prosodic PPAOS (along with three mixed and two phonetic). While this is in and of itself informative, it interfered with a primary objective of this study which was characterization of the ASR inaccuracies.

For the included 34 patients with PPAOS and 22 controls (demographic and clinical data in Supplemental Material S2), the number of total inaccuracies transcribed by the ASR were tallied and separated into phonetic (sound additions, deletions, and substitutions) and prosodic inaccuracies (within word segmentations). The mean number of overall, phonetic, and prosodic inaccuracies per word are shown in Table 2 and Figure 3, separated by group. The results of a regression analysis showed that the ASR produced significantly fewer inaccuracies for controls than for patients with PPAOS (p < .0001), but there were no significant differences between PPAOS subtypes (p > .70). When looking at phonetic and prosodic inaccuracies separately, similar results were observed. The relative proportion of phonetic to prosodic inaccuracies for each group can also be seen in Table 2; overall, phonetic inaccuracies were much more prevalent in all PPAOS groups, including the prosodic group. The mean phonetic inaccuracies per patient in the phonetic group was 69 compared to 47.67 in the prosodic group and 5.64 in controls, consistent with the result that controls produced fewer phonetic inaccuracies than all patients with PPAOS (p < .0001). The mean prosodic inaccuracies per patient in the phonetic group was 4.3 compared to 7.3 in the prosodic group and 3.14 in controls. Although the ASR system made more phonetic inaccuracies for phonetic and mixed PPAOS than for prosodic PPAOS, and more prosodic inaccuracies for prosodic and mixed PPAOS than phonetic PPAOS, these differences were not statistically significant.

Figure 3.

Mean number of each type of inaccuracy per word across participant groups from automatic speech recognition. PPAOS = primary progressive apraxia of speech.

The predominance scores, shown in Table 2, showed that all patients and controls had more recorded phonetic than prosodic inaccuracies. The controls had the most balanced asymmetry index (.1). Prosodic PPAOS showed a higher proportion of prosodic inaccuracies (.88) compared to phonetic (.71) and mixed PPAOS, although this difference was not statistically significant.

Cohen's kappa coefficients were calculated to determine the association between the ASR transcriptions and human transcriptions. For phonetic errors and inaccuracies, the kappa coefficient was .22, and for prosodic, it was .10. This indicates “fair” agreement for phonetic errors/inaccuracies and “slight” agreement for prosodic errors/inaccuracies (Landis & Koch, 1977).

Discussion

The goal of this study was twofold: to determine if ASR-determined WER correlates with PPAOS severity and to determine if the speech errors transcribed by the ASR can distinguish between controls and patients with PPAOS and among PPAOS subtypes. Our results suggest that using ASR to determine a WER may be clinically useful in informing whether a patient may have PPAOS or another motor speech disorder, given that WER alone was predictive of whether the participant was in the control versus PPAOS group (p < .0001), supporting our hypotheses. This conclusion is strengthened by the high AUROC value. That is, if a patient in this cohort had a WER of over 0.44, there was a high likelihood the patient had PPAOS. This WER value is higher than expected intuitively, but this is because the ASR made several errors on healthy speech. This finding requires replication in a larger cohort, with a broader set of speech samples to identify how to best optimize not just sensitivity but also specificity for detecting AOS. Our results also showed that WER moderately and positively correlates with PPAOS severity, as measured by the ASRS-3 total score, so it may be useful as a proxy of AOS severity. This is in accordance with previous results with patients with dysarthria, where higher WERs correspond to more severe dysarthria (Christensen et al., 2013; Mustafa et al., 2015; Tu et al., 2016). Relatedly, there was a strong correlation between WER from the ASR and the proportion of words produced with an error from the human transcriptions (r = .72, p < .0001). This similarly supports that WER may be a good proxy for severity of the problem cross-sectionally; further investigation if it tracks longitudinally within patients is ongoing.

WER alone was not useful in determining PPAOS subtype, as all PPAOS groups had comparable WERs. This is not surprising given that the groups did not differ on overall severity, which overall correlated with WER. WER does not, however, provide insight into the nature of the inaccuracies made by the ASR, which is what our second research question addressed.

Manually counting the number of phonetic and prosodic inaccuracies from the ASR transcriptions reveals more differences between PPAOS subtypes, although the utility of doing so appears limited. That is to say, although the prosodic and mixed groups had more recorded segmentation inaccuracies made by the ASR than phonetic PPAOS, and phonetic and mixed PPAOS transcriptions had more phonetic inaccuracies than those from patients with prosodic PPAOS, the differences were not statistically significant. Even with manual sorting of error types, the ASR system used in this study was not sufficient for determining PPAOS subtype diagnosis. This contrasts with manual transcription, where the distribution of coded error types recorded differentiated among PPAOS subtypes, where the phonetic group produced more phonetic-type errors compared to the prosodic group and the prosodic group produced more segmentations compared to the phonetic groups (Tetzloff et al., 2024).

Table 3 shows examples of the ASR output and error counting compared to that of manual transcriptions. In reviewing the respective ASR and manual transcriptions, there was no overt pattern or identifiable source of the errors (e.g., although /t/ at the end of “catapult” was clearly produced to the listener's ear, it was transcribed as a /p/ by the ASR). So, while ASR output appears useful in differentiating disordered and healthy speech, evidence of this system's ability to classify between disorders is currently limited. More work is needed to compare output across more heterogeneous disease groups.

Table 3.

Example transcription from a prosodic PPAOS patient, including: the ASR transcription, the number of tallied ASR transcription inaccuracies, a phonetic transcription of the recording by a trained phonetician, and the number of errors noted in the manual transcription.

ASR transcription	No. of ASR inaccuracies by type	IPA transcription	No. of errors by type
alumanlum alumanlum alumanlm	Phonetic: 6 Prosodic: 0	[ɑ:lum:ɪnəm:əm əlumɪn:məm əlumnə̃]	Phonetic: 5 Prosodic: 0
ar tilri ar tilri ar tilei	Phonetic: 3 Prosodic: 3	[ɑɹtʰɪ̤lɚi ɑɹtʰɪ: lɚ:i ɑɹtʰɪ lɚi]	Phonetic: 0 Prosodic: 2
cat cat cat	Phonetic: 0 Prosodic: 0	[kʰæʔ kʰæʔ kʰæʔ]	Phonetic: 0 Prosodic: 0
cat pulp cato pulp cata pulp	Phonetic: 5 Prosodic: 3	[kæɾə pʰʊltʰ kæɾɚ pʰʊltʰ kʰæɾə pʰʌltʰ]	Phonetic: 1 Prosodic: 3
atatrafi catastref catastrefi	Phonetic: 5 Prosodic: 0	[kətʰæstɹɪfi kə tʰæstɹɪfi kʰə tʰæstɹəfi]	Phonetic: 0 Prosodic: 2
catnem cat nem cat nem	Phonetic: 6 Prosodic: 2	[kʰæt nɪb kʰæʔ nɪp kʰæʔ nɛm]	Phonetic: 3 Prosodic: 3
harmana c hrmane c armane ca	Phonetic: 6 Prosodic: 3	[hɑɹ:mɑ:nɪ:kaw hɑɹ maɾɪ kʰɑ hɑɹ mɑ:nɪ: kʰə]	Phonetic: 2 Prosodic: 4
r naseris ry nasores ry nasres	Phonetic: 3 Prosodic: 3	[ɹɑɪnɑ:sɚɪs ɹɑɪnɑ:sɚ:ɪs ɹɑɪnɑ:sɚɪ:s]	Phonetic: 0 Prosodic: 0
snman snoman snman	Phonetic: 2 Prosodic: 0	[snoʊmɛn snoʊmæn snoʊmæn]	Phonetic: 1 Prosodic: 0
speseiic spstific spesific	Phonetic: 3 Prosodic: 0	[spɪsɪ:f:ɪk spəsɪf: ɪk spəzɪf: ɪk]	Phonetic: 1 Prosodic: 2
stestistic stetisti statistic	Phonetic: 6 Prosodic: 0	[stə tʰɪs tʰɪk stə tʰɪs tʰɪk stə tʰɪs tʰɪk]	Phonetic: 3 Prosodic: 6
stethascop stethascp stath a cop	Phonetic: 3 Prosodic: 2	[stɛ θɛzskoʊp stɛ θɛskoʊp stɛθ ðə skoʊp]	Phonetic: 2 Prosodic: 4
val cnno val canu valcano	Phonetic: 5 Prosodic: 2	[vɔl kʰeɪnoʊ vɔl kʰɑnoʊ vɑl kʰeɪnoʊ]	Phonetic: 1 Prosodic: 3

Note. PPAOS = primary progressive apraxia of speech; ASR = automatic speech recognition; IPA = International Phonetic Alphabet.

It is important to again note that the inaccuracies recorded were counted from the ASR transcriptions and not from the audio itself. Therefore, the inaccuracies made by the ASR system may reflect a combination of the errors made by the speakers and those made by the ASR system, as many errors are only partially captured in such models; that is to say, a high rate of ASR inaccuracy means that even when speaker errors are present, it may not faithfully capture what the speaker said (Wiepert et al., 2023). For example, even in controls, the ASR transcriptions showed some segmentations between syllables in the two compound words “snowman” and “catnip” (e.g., snow man, cat nip) or sound substitutions (e.g., aluminam for “aluminum”) when the speaker made no such error by manual transcription; this would be an ASR inaccuracy and not a speech error, although it was still counted as an inaccuracy in these analyses. In this case, the source of the inaccuracy is considered attributable to the model. Regarding segmentation inaccuracies, it was surprising how few were reported, particularly for patients with prosodic PPAOS who have predominantly segmented speech. It is possible this reflects the reduced opportunity for segmentation within words (e.g., three in the word “catastrophe”); interword segmentation was not evaluated in this word repetition task. The other features of prosodic PPAOS—overall slow rate and reduced pitch modulation—were also not captured. It is possible there were more segmentation errors in the patients who were not included in the error analysis, but it does not explain everything; patients with moderate to mild PPAOS who were included in the error analysis had perceptually identified word segmentations that were not captured by the ASR system. This could help explain why there was such little agreement between the prosodic errors in the ASR versus human transcriptions. It is unclear, however, why the ASR was so unreliable at capturing segmentation errors that were judged present and marking them when they were perceptually absent.

On the other hand, the sound deletions or substitutions detected by the ASR, particularly in the patients with phonetic and mixed PPAOS, are likely representative of the speech that was produced, consistent with those identified in the manual transcriptions of these speech samples (Tetzloff et al., 2024). For example, one production of the words “aluminum” and “rhinoceros” were transcribed as “elumnem” and “renaserous,” respectively by the ASR, which qualitatively matches the audio files and their manual transcriptions quite accurately. However, upon comparing the ASR to human transcriptions, there was little direct concordance (although there is an apparent relationship between the number of words with manually detected errors and the WER; see Supplemental Material S3). Therefore, the specific phonetic errors captured by the ASR are likely different from those captured by the manual transcription. It is possible some of the differences are due to the ASR transcribing orthographically, while manual transcriptions used IPA symbols with associated diacritics; this requires further investigation. We intentionally chose the wav2vec model that produced orthographic transcriptions because the phonemic model does not have the capability to segment words at all. The different sources of inaccuracies—including the model, speaker, and audio file—is an important area of investigation that requires further research (Wiepert et al., 2023). Of note, that study showed that wav2vec had lower model inaccuracy and higher “speaker error” (i.e., accurately transcribed what the speaker said, even if it did not match the intended target), which aligns with the hopes of this and other studies of ASR for clinical use.

The results of this study should be interpreted in the context of the study's limitations. First, inaccuracy analysis could not be completed on 25% of patients, as the transcripts produced by the ASR system were unintelligible, which again may reflect a combination of speaker and model error. Nonetheless, this was only present in patient and not control speakers, suggesting it may also serve as an index of abnormality. If all cases were subject to inaccuracy analysis, it is likely even more phonetic inaccuracies would be reported due to the extreme distortions of speech sounds; however, given the ASR system's reduced detection of prosodic errors, more segmentations may not have been reported. There was no clear pattern of difficulty, including audio recording quality or type or severity of AOS that suggests why the ASR performed the way it did. It is also possible that by feeding the ASR system individual words rather than all words concatenated together may have made the 25% slightly more intelligible; although there was a pause between unique words, so it is unclear if this would have made a meaningful difference. Further exploration is warranted. Another weakness is that only one ASR system was evaluated. The one that was selected (wav2vec) does not include a language model; this means the model's transcription may include nonwords; this was selected intentionally, as it was desired for the ASR to capture what was said by the patient as opposed to predict what was intended. Different ASR systems, particularly those that include a language model, might instead predict what the patient intended to say, thereby removing diagnostically valuable information (Gutz et al., 2022; Wiepert et al., 2023). Additional future research will evaluate the efficacy of ASR with a broader range of the ever-increasing number of available ASR algorithms, across sentence repetition and connected speech, and in a broader range of AOS severity, to evaluate the potential for differential diagnosis, as well as longitudinally to better characterize the potential for serving as a proxy for disease severity.

This is only the second study to evaluate efficacy of ASR in the AOS population (Jacks et al., 2019) and the first to do so in the context of neurodegenerative AOS. Our study has demonstrated that ASR can be useful for detecting the presence of mild to moderate PPAOS (vs. healthy controls) and estimating AOS severity, as the automatically calculated WER was correlated with a measure of symptom severity (ASRS scores). However, researchers should proceed with caution when considering ASR for more detailed analysis of disordered speech, because the ASR transcriptions did not distinguish between PPAOS subtypes, despite the sensitivity of human transcriptions to subtype distinctions (Tetzloff et al., 2024). This weakness is amplified by the fact that there was little agreement between the errors transcribed by the ASR system compared to human transcription. While ASR may be helpful to augment clinical care, more research is needed before integrating ASR into clinical practice tools to discriminate between healthy and disordered speech (based on the presence of errors), among communication disorders (according to the nature of errors), as well as track progression (via the proportion of errors).

Data Availability Statement

Data will be made available by the corresponding author upon reasonable requests from qualified investigators.

Appendix

Example Transcription, From a Patient With Severe AOS, for Which Error Analysis Was Not Completed (WER = 1)

ah no no no ah o go noawoÊ o ah goooo a go o gooo ah go g go go god god go dos o goye go dos o go go gos go gooy god nodgo god nd gos od go ah nao go go a o go goah a no go go go go do s wo go d oos od g go oo do no do n do no go do ak so god do gk ods go g gs do dake dod go a dodgo d dos dos o go do do ago o do gogo ah go do oh gooÊ a gooo

Targets (word list):

aluminum aluminum aluminum artillery artillery artillery cat cat cat catapult catapult catapult catastrophe catastrophe catastrophe catnip catnip catnip harmonica harmonica harmonica rhinoceros rhinoceros rhinoceros snowman snowman snowman specific specific specific statistics statistics statistics stethoscope stethoscope stethoscope volcano volcano volcano

Funding Statement

This work was supported by the National Institutes of Health Grants R01 AG 83832 (PI: Botha), R01-DC12519 (PI: Whitwell), R01-DC14942 (PI: Utianski/Josephs), and R01-DC010367 (PI: Josephs).