Prosodic Preferences of Surface Electromyography-based Subvocal Speech for People with Laryngectomy

Abstract

Introduction:

People who undergo a total laryngectomy lose their natural voice and depend on alaryngeal technologies for communication. However, these technologies are often difficult to use and lack prosody. Surface electromyographic-based silent speech interfaces are novel communication systems that overcome many of the shortcomings of traditional alaryngeal speech and have the potential to seamlessly incorporate individualized prosody. The purpose of this study was to (1) validate the ability of alaryngeal silent speech to effectively incorporate pitch modulations—a key prosodic element in natural speech—into synthesized speech assessed through listening experiments, and (2) determine the key features of these communication devices according to core users.

Methodology:

People with laryngectomy (PWL, n=15) and their primary communication partners (n=5) listened to synthesized sentences with differing prosodic content generated from deep regression neural networks developed in our prior work. Specifically, the fundamental frequency $\left(f_o\right)$ contour of each sentence was manipulated in 4 ways: (1) flattened to the average $f_o$, (2) altered to discrete sentence-level classification of muscle activity, (3) altered to continuous mapping of muscle activity, and (4) filtered to emulate speech from an electrolarynx (EL). Listeners ranked the $f_o$ contours of each sentence in terms of speech naturalness and the importance of various speech aid features.

Results:

Continuous contours rated higher than all other types of contours, and monotonic EL contours rated the lowest. Speech aid features were rated highest to lowest in the following order: sound quality, intelligibility, pitch, delay, volume, handsfree, maintenance, cost, wearability, training, and visibility.

Conclusion:

These results will help inform future development of silent speech interfaces and shape priorities of communication devices toward the preferences of their users.

Introduction

Voice is one of the most unique and ubiquitous methods of transmitting identity. A short utterance provides effortless self-expression, informing a speaker’s gender, personality, mood, desires, and beyond. Consequently, people who lose the ability to speak naturally suffer negatively altered self-perception, social withdrawal, and lower quality of life, among other severe negative psychosocial concomitants.^1,2

Total laryngectomy involves complete removal of the larynx and, thus, an individual’s ability to speak naturally. This surgical procedure is the primary treatment for advanced-stage laryngeal cancer and the standard of care when nonsurgical management of laryngeal cancer fails.³ Without their vocal folds to produce voice, people with laryngectomy (PWL) must rely on alternative vibration sources to produce speech; however, these technologies remain inferior to natural laryngeal speech. Two common methods involve vibration of the pharyngoesophageal (PE) segment: tracheoesophageal speech (TE, requires intensive maintenance and additional surgery⁴) and esophageal speech (ES, requires extensive training, and few master the technique^5,6). An electrolarynx (EL) is a common, non-invasive method that offers a rapid learning curve. As such, it often serves as a primary tool for communication or as a backup option for patients using tracheoesophageal (TE) or esophageal speech (ES). When placed against the neck, the handheld device supplies electromechanical vibration to the vocal tract. Yet the constant use of a patient’s hand can be excessively cumbersome and especially challenging for patients with impaired manual dexterity.^4,7

Physical limitations aside, alaryngeal options provide most PWLs with a serviceable, albeit unnatural, voice. Still, many do not hear their voice: naïve listeners and PWLs consistently rate all alaryngeal options as sounding less intelligible and more artificial compared to pre-surgical voices.² The lack of naturalness can be partially attributed to these technologies’ lack of effective dynamic pitch modulation capabilities. Modulating pitch is an essential prosodic aspect of verbal communication that facilitates meaning, attitude, emotion, and more. With TE and ES, excitation of the PE segment has unstable oscillation patterns and lower periodicity compared to natural vocal folds, all leading to a much lower fundamental frequency $\left(f_o\right)$, inconsistent pitch contours, unnatural spectral shaping, and an overall rough-sounding voice.⁸ EL speech is often perceived as inferior to TE and ES in terms of naturalness as it has a very mechanical and monotone sound quality.⁵ Attempts to incorporate dynamic pitch modulation to ELs have had mixed outcomes. Studies have shown intelligibility was slightly higher for speakers using a pitch-controlled EL (e.g., a TrueTone EL) compared to those with flattened frequencies.^9,10 However, commercially available pitch-controlled ELs use buttons and/or switches that are arguably impractical for real-time modulation of pitch due to system complexity and high cognitive loads.⁷ Additionally, despite the increase in $f_o$ control, speakers do not think that their pitch-modulated EL speech represents how they sounded before total laryngectomy.¹¹ Other efforts for improving EL naturalness provide hands-free control via biosignals associated with vocal $f_{0.}$ ^(12–16) Despite success within the research community, these devices are not yet commercially available since they still fundamentally sound like an EL (e.g., mechanical or robot-like).

Constraints of existing alaryngeal options have inspired the emerging field of silent speech interfaces (SSIs). SSIs enable oral communication in the absence of vocalization by decoding physiological signals associated with speech production.¹⁷ Without requiring new vocalization or articulation techniques, SSIs are ideal for PWLs. Various sensory modalities have been used to capture these signals,^18–27 but surface electromyography (sEMG)-based SSIs are the most promising due to their non-invasiveness, portability, real-time applicability, and high efficacy.^11,28,29

Surface EMG-based SSIs capture articulatory muscle activity of the face and voice-related activity of the neck during subvocal speech (inaudibly mouthing words) via non-invasive electrodes affixed to the skin surface. Previous work has demonstrated these systems’ ability to accurately translate sEMG signals to continuous speech with low latency, and thus show promise as a viable alternative to conventional alaryngeal speech.³⁰ Yet incorporating prosody into synthetic speech is a developing effort for sEMG-based SSIs. While a natural relationship between extrinsic laryngeal muscles and pitch modulation has been established,^{12–14,31–33} effectively translating this relationship into synthetic voice synthesis remains an area of exploration.

Among various signal-processing approaches applied to date, machine learning techniques have had the most success in extracting pitch from sEMG signals. Nakamura et al.³⁴ first demonstrated the potential of using sEMG signals from the neck to predict $f_o$ using Gaussian mixture models, achieving a moderate correlation between observed and predicted $f_o$ of $r=.49$. Diener et al.³⁵ took a different approach by using a feed-forward neural network to predict $f_o$ contours from a quantization approach rather than a continuous sequence, but did not observe good performance ($r=.27$). Building off these previous studies, our recent work achieved the highest correlation between observed and predicted $f_o(r=.92)$ using deep regression neural networks.³⁶ Although these recent developments show improvements in the objective accuracy of $f_o$ prediction from an sEMG signal, evaluating whether prosody is effectively incorporated into synthetic speech requires subjective assessments from listeners. Few SSI studies have explored these outcomes,^11,35 and it remains unclear the extent to which different types of $f_o$ cues (e.g., fine-tuned or generic) affect the reception of synthetic speech. Systems pursuing a fine-tuned approach of $f_o$ estimation developed user-specific models that at least initially rely on acoustic information.^35–37 Such systems are incompatible with most PWL as they typically do not seek speech aids until after laryngectomy surgery. Furthermore, no study has investigated whether these user-specific systems generate more natural-sounding prosodic cues compared to generic systems. For instance, it is possible that a PWL would prefer generically generated prosodic information instead of a fine-tuned $f_o$ contour, as they do not require acoustic input and resulting computational overhead (e.g. increased delay and design constraints). It is additionally unclear how the preferences of PWL compare to their communication partners, who typically play a central role in facilitating effective communication. Yet, to our knowledge, previous investigations have only involved perceptual preferences from naïve listeners rather than the intended target population.^11,35 As such, one of the primary goals of this work was to elucidate the preferences of both PWLs and their primary communication partners regarding methods of $f_o$ generation in synthetic speech. We utilized the synthetic speech outputs from sEMG-based SSIs described in our prior work^11,36 to understand how manipulations to—or the absence of—a time-varying $f_o$ contour affect the perceived naturalness of sEMG-based synthetic speech. For this investigation, we compared listener preferences across three types of $f_o$ contours:

1. A flat $f_o$ contour that does not vary with time,

2. A generic $f_o$ contour that is based on discrete classifications of phrase-level muscle activity, and

3. A fine-tuned $f_o$ contour that is generated by continuous mapping of time-varying muscle activity to $f_o$.

We additionally included a synthetic version of EL speech as a reference in our analysis, given its popularity as an alaryngeal speech option for PWLs. We hypothesized that the $f_o$ contour of the more nuanced approach of continuous mapping of muscle activity would be preferred by PWLs and their primary communication partners over the flat contour, discrete mapping, and synthesized EL speech.

The second goal of this work was to determine the characteristics of speech aids (e.g., SSI or EL) that PWLs deemed most important for effective communication. Factors such as intelligibility,³⁸ voice quality,³⁹ cost,^40–42 and ease of use,⁴² among others, are known to affect speech rehabilitation following total laryngectomy; however, recent empirical evidence as to how these characteristics are prioritized by PWLs and their primary communication partners is limited. This lack of information raises concerns that current research into high-tech speech aids, such as SSIs, might be prioritizing aspects that are less important to their intended users. For instance, a speech aid might offer excellent voice quality but be challenging to use, leading to low adoption rates among PWLs. Conversely, a less sophisticated device might be favored due to its relative cost and ease of use, even if it provides lower voice quality. It therefore remains unclear which characteristics should be prioritized. Thus, a secondary goal of our work was to address these gaps by surveying the importance of various speech aid characteristics to these core users.

The development of SSIs or other high-tech speech aids that incorporate the most preferred prosodic qualities of synthetic speech may be an important step toward improving the quality of life of PWLs. Beyond discerning the preferred prosodic qualities of synthetic speech for PWLs and their primary communication partners, we provide a list of essential considerations to guide future research and development in communication strategies and technologies for PWLs.

Methods

Development of Speech Stimuli

Collection of sEMG and Acoustic Signals

Speech stimuli were sourced from a retrospective analysis, the methods of which are described in Vojtech et al.³⁶ In brief, speakers with typical voices were seated in a quiet room where they were instructed to perform a series of speech tasks as sEMG and acoustic data were concurrently recorded. The speakers produced sustained vowels, legatos, syllables, phrases, reading passages, questions, and monologues to induce a range of vocal behaviors and articulatory patterns. The sEMG signals were collected via two wireless Trigno Quattro sensors (Delsys, Natick, MA, USA), which provided eight single-differential electrodes. Each sensor was placed over a distinct region of the face and neck as described in Meltzner et al.^43,44 Acoustic signals were recorded using a headset omnidirectional microphone (Movo LV-6C XLR) that was positioned at a 45-degree angle from the midline and 4–7 cm from the lips. Microphone signals were pre-amplified (ART Tube MP Project Series) and digitized at 44.1 kHz (National Instruments USB NI-6251). Time-aligned sEMG and acoustic signal acquisition was performed using a triggering setup in EMGworks software (Delsys, Natick, MA, USA) with a custom trigger module to connect the data acquisition board and sEMG base station trigger port.

For the purposes of the current study, we focused on the sEMG and acoustic data from four participants who were native English speakers (2 female, 2 male; M = 29.8 years, SD = 9.6 years). None of the speaker participants had a history of speech, language, or hearing disorders. Three speech tokens of varying phrasal stress patterns—initial word stress, final word stress, and unstressed—were chosen from the retrospectively collected speech corpus for our analysis. These tokens included: (1) “THAT won’t make a difference,” (2) “Are you SERIOUS?,” and (3) “There is nothing I can do about it.” These selections were based on empirical assessments conducted by the authors to confirm phrasal stress on the intended word, ⁴⁵ indicated by capitalized words.

Manipulating ${\mathit{f}}_{\mathit{o}}$ Contours

Using the concurrently collected acoustic and sEMG data from each speech token, we generated a series of synthetic speech stimuli for subsequent perceptual evaluation. Three types of $f_o$ contours were synthesized (flat, discrete, and continuous) and overlaid each acoustic signal (see Fig. 1 for example). Flat $f_o$ contours had uniform pitch variation across the speech signal and served as a baseline condition. Discrete and continuous $f_o$ contours were synthesized to examine the effects of pitch variation type—either distinct pitch variations associated with specific stress patterns or natural pitch variations that mirror typical speech inflections and prosodic contours — on perceived speech naturalness. Generation of these contours is described as follows:

Figure 1.

Illustration of discrete (purple), continuous (gold), and flat (green) $f_o$ contours generated for one speaker from the phrase “Are you SERIOUS?”

1. Flat Contours: Flat speech stimuli were generated using the “monotonize” function of the Praat Vocal Toolkit,^46,47 which automatically analyzes the $f_o$ values across an acoustic recording input and then sets all $f_o$ values to a consistent median value, which was set per stimulus as the mean $f_o$ of the original signal.

2. Discrete Contours: Speech stimuli with discrete $f_o$ contours were synthesized using a modified version of the sEMG-based SSI method described in Vojtech et al.¹¹ As phrasal stress categorization and lexical recognition is outside the scope of the current study, we developed a streamlined text-to-speech algorithm inspired by the referenced methodology to generate an $f_o$ contour via prototypical representations of phrasal stress.⁴⁵ Stress on the first word (“THAT”) or last word (“SERIOUS”) was achieved through empirically derived gain modifications to increase the $f_o$ of the intended word.^48–50 Pitch manipulations were implemented to overlay the generated $f_o$ contours onto flat speech stimuli using routines based on the Pitch Synchronous Overlap and Add (PSOLA) algorithm.^51,52

3. Continuous Contours: Continuous speech stimuli were generated using the deep regression neural networks described in Vojtech et al.,³⁶ which were already trained on the concurrent sEMG and acoustic data of the four speakers. In brief, the $f_o$ contour was extracted from each speech token using the autocorrelation-based Praat method of the Parselmouth^46,53 package in Python (v.3.8). Minimum and maximum $f_o$ values were set to 65 Hz and 475 Hz, respectively, ^54–56 and the time step for this algorithm was set to default (0.75/minimum $f_o$). The acoustic and sEMG data were time-aligned using a dynamic time-warping algorithm from the linmdtw package⁵⁷ in Python with a hop value of 0.010 s. All acoustic and sEMG signals were then windowed at a frame size of 40 ms with a 20-ms step shift; for each sEMG signal window, a set of 20 features was calculated using procedures detailed in Vojtech et al.³⁶ Deep regression neural networks were trained for each speaker using the set of sEMG features and observed $f_o$ from their signals, then predicted $f_o$ for each window. Using these trained models for each of the four speakers, we used existing routines from the Parselmouth package in Python to replace the existing $f_o$ contour of each speech token with the predicted $f_o$.

4. EL Speech: In addition to the three types of $f_o$ contours that were overlaid onto existing speech tokens, we synthesized speech stimuli to mimic the acoustic characteristics of EL speech. The EL speech stimuli were derived using the vocoder and EQ 10 bands functions of the Praat Vocal Toolkit.^46,47 The mean $f_o$ value of the original speech token was first estimated using the autocorrelation-based Praat method with minimum and maximum $f_o$ values set to 65 Hz and 475 Hz; all other parameters were kept at their default settings. The speech token was then filtered via source-filter synthesis with a pulse train carrier waveform at a frequency matching that of the identified mean $f_o$. The spectral characteristics of the resulting speech were additionally adjusted using the automated equalizer function from the Praat Vocal Toolkit, with an emphasis on 1) boosting the lower frequencies to enhance the $f_o$ prominence and 2) attenuating the higher frequencies to replicate the acoustic profile of EL speech. To mimic the recording conditions of original acoustic samples (and hence, the discrete and continuous synthetic samples) that were collected in a low-noise environment, and to make the synthetic EL speech more like true EL speech,^58–60 white noise was added to the processed EL speech stimuli at a signal-to-noise ratio of 25 dB.

Evaluation of Speech Stimuli

Listeners

Fifteen PWLs (10 male, 5 female; M = 54 years, SD = 4.13 years) and five primary communication partners (1 male, 4 female; M = 59 years, SD = 9.56 years) were recruited for the study through in-person outreach at community-based support groups by author J.T.H. All participants spoke English and were naïve to the purpose of the study. Refer to Table 1 for demographic information of PWLs and their primary communication partners. Prior to participation, all listeners provided informed, written consent in compliance with the Western Institutional Review Board (Protocol #20182089).

Table 1.

Demographic information of people with laryngectomy (PWLs) and primary communication partners (“PCP”).

			Time Since Laryngectomy
ID	Sex	Age	Years	Months	Primary Method of Communication	Other Methods Used (if applicable)
PWL01	Male	58	6	0	TE	EL
PWL02	Female	59	2	8	ES
PWL03	Male	62	0	10	TE
PWL04	Male	51	15	6	TE	EL
PWL05	Female	52	1	6	EL, TE
PWL06	Male	51	0	5	EL
PWL07	Female	57	4	7	TE	EL, ES
PWL08	Male	57	0	4	TE
PWL09	Female	50	3	2	EL, TE
PWL10	Male	52	4	8	TE	EL, Dry-erase board, Text-to-speech apps
PWL11	Male	56	0	9	EL	EL, Text-to-speech apps
PWL12	Female	57	2	3	TE
PWL13	Male	51	2	9	EL, TE	ES
PWL14	Male	48	1	10	EL
PWL15	Male	37	23	11	EL	ES
PCP01	Female	52	N/A	N/A	N/A	N/A
PCP02	Female	67	N/A	N/A	N/A	N/A
PCP03	Female	69	N/A	N/A	N/A	N/A
PCP04	Female	62	N/A	N/A	N/A	N/A
PCP05	Male	57	N/A	N/A	N/A	N/A

Note. TE=tracheoesophageal speech, ES=esophageal speech, EL=electrolaryngeal speech.

Online Survey

The listening procedure was completed remotely by all participants using the online behavioral research platform Gorilla Experiment Builder (www.gorilla.sc). Listeners were instructed to complete both study tasks in a single session in a quiet environment. Headphones were recommended to provide optimal acoustic fidelity but not required.. Each listening session began with a volume adjustment from the Gorilla open materials repository to allow listeners to adjust their device volume to a comfortable level during the presentation of a sample speech token. Once they set the volume, they were instructed to keep the volume constant throughout the entire experiment.

Listeners were presented with a visual sort-and-rate (VSR) task, in which they were instructed to rate the overall naturalness of speech stimuli relative to one another. VSR has been reported with high reliability for ratings of naturalness in speakers with voice disorders⁶¹ and for synthesized speech tokens.⁶² A definition of speech naturalness was provided for the entire duration of the task as “Speech naturalness is not judged by how well you understand the words that are being said, but rather your preference for how it sounds in terms of rate, rhythm, intonation and voice quality”.⁶³ This definition remained visible throughout the naturalness rating tasks.

In the VSR task, listeners could play each audio file as many times as needed and were not instructed as to the order in which to rate the naturalness of the synthetic speech tokens. A progress bar displayed at the bottom of the screen guided listeners through each task page, where they rated four synthetic speech tokens at a time. Each token featured the same phrase (“THAT won’t make a difference,” “Are you SERIOUS?,” or “There is nothing I can do about it.”) spoken by the same speaker, but synthesized using one of four manipulation methods (flat, discrete, continuous, EL). Listeners assessed the naturalness of each audio file by adjusting a continuous scale graded from 0 to 100, anchored between “least natural” (0) and “most natural” (100). After rating the naturalness of all four tokens, listeners were instructed to play each audio file again and adjust their ratings relative to each other, ensuring that no two tokens received the same score. Once they were comfortable with their responses, they were instructed to press “Next” to move onto the next page of the task. The VSR task involved evaluating a series of 48 unique synthetic speech tokens (1 phrase × 3 stress types × 4 manipulation methods × 4 speakers). Additionally, 12 tokens were repeated in a pseudorandom order (1 phrase × 3 stress types × 4 manipulation methods) to assess intralistener reliability, resulting in a total of 60 synthetic speech token judgements. The order of each task page was randomized to minimize biased responses from anticipating speakers’ voices.

After completing the VSR task, listeners were asked to rate the degree of importance of various features of speech aids or alaryngeal speaking techniques on a continuous scale graded from 0 to 100, anchored between “least important” (0) and “most important” (100). Listeners were asked to consider all features prior to rating them. As in the VSR task, no two features were allowed to have the same rating. Definitions of each feature were provided to participants and are described in Table 2.

Table 2.

Definitions of each speech aid feature.

Speech Aid Feature	Definition
Device Cost	Some devices might not be fully covered by insurance (e.g., Medicare). How important is it to avoid out-of-pocket costs (for example, $1000 initial cost)
No Wearable Sensors	Some devices might require you to wear small sensors near your mouth or under your chin. How important is it to avoid having skin-placed sensors?
Device Visibility	Some devices are visible to people around you. How important is it to have a device not visible or obvious to others?
Device Volume	Some devices or alaryngeal speech techniques are louder than others. How important is it for you to speak loudly when you want?
Pitch Modulation	Some devices or alaryngeal speech techniques are monotone (flat pitch) while others allow pitch control. How important is it for you to control your vocal pitch?
Speech Intelligibility	Some devices or alaryngeal speech techniques are easier to understand than others. How important is it to have everything you say understood?
Sound Quality	Some devices or alaryngeal speech techniques sound better than others. How important is it to have a normal, natural-sounding voice and speech quality?
Hands-free Device	Some devices or alaryngeal speech techniques require the use of one hand. How important is it to be able to speak hands-free?
No Speech Delay	Some devices have a 1- to 3-second delay between when you start to speak and when a synthetic version of your speech is generated. How important is it that your speech has no delay?
No Training Required	Some devices or alaryngeal speech techniques require training and practice before performing well. How important is it not to need training or practice?
Ease of Device Maintenance	Some devices or alaryngeal speech techniques require routine maintenance (e.g., battery changes, valve replacement, etc.). How important is it to avoid maintenance?

Statistical Analysis

Naturalness ratings (0–100) and speech aid feature importance scores (0–100) were extracted directly from the VAS scores. First, intrarater reliability was calculated for the naturalness rating task using two-way random effects intraclass correlation coefficients (ICCs) to assess consistency of values. Listeners with poor reliability scores (ICC < 0.50) were removed from further analysis.⁶⁴ Due to the unbalanced sample sizes (15 PWLs and 5 primary communication partners), Levene’s test for homogeneity of variance was calculated for both analyses.

A two-way mixed effects analysis of variance (ANOVA) was performed to evaluate how mean naturalness ratings were impacted among the four types of $f_o$ contours (flat, discrete, continuous, EL) and group (PWL, primary communication partner) across participants. Type of contour, group, and the interaction between contour type and group were fixed factors. An alpha level of .05 was used for significance tests. A squared partial curvilinear correlation $\left({{\eta }_p}^2\right)$ was used to calculate effect sizes. Fixed effects tests used the Satterthwaite approximation and variance estimation was calculated via maximum likelihood. Tukey simultaneous post hoc tests were used to evaluate pairwise comparisons of mean naturalness ratings across the types of $f_o$ contours using 95% confidence intervals.

A two-way ANOVA was performed to evaluate differences in mean importance score between speech aid features and listener group. An alpha level of .05 was used for significance tests. A squared partial curvilinear correlation was used to calculate effect sizes. Tukey simultaneous post hoc tests were then performed to evaluate pairwise comparisons of the mean importance score between speech aid features using 95% confidence intervals. Effect sizes were calculated via Cohen’s $d$. All analyses were completed using Minitab version 20.

Results

Naturalness Ratings

One participant (PWL15) was found to have low intrarater reliability, and their data was not included in the naturalness rating analysis. For the remaining 19 participants, the average intrarater reliability was ICC = .87 (SD = .13, range: .45–.99). There were no significant differences in the variance of naturalness ratings between PWLs or primary communication partners (p = .27). Table 3 shows the results of the two-way mixed effects ANOVA. The model showed a significant, small interaction effect of $\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{o}\mathrm{u}\mathrm{r}\times \mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{p}\left(p<.001,{n_p}^2=.02\right)$ and a significant effect of contour on the naturalness ratings (p < .001). A Tukey post-hoc test revealed that PWLs rated continuous (M = 62.63, 95% C.I. = [59.50, 65.76]), discrete (M = 58.69, 95% C.I. = [55.56, 61.82]), and flat (M = 58.83, 95% C.I. = [55.70, 61.96]) contours significantly higher than EL speech (M = 6.61, 95% C.I. = [3.48, 9.74], all p < .001). Primary communication partner also rated EL speech (M = 2.74, 95% C.I. = [−1.71, 7.01]) significantly lower than all other contours (p < .001). However, Tukey post-hoc tests found primary communication partners rated continuous contours (M = 74.73, 95% C.I. = [70.37, 79.10]) significantly higher than flat contours (M = 62.53, 95% C.I. = [58.17, 66.90], p = .004), but discrete contours (M = 69, 95% C.I. = [64.64, 73.36]) did not rate significantly different from continuous (p = .634) or flat contours (p = .478; Fig. 2).

Table 3.

Results of analysis of variance test performed for speech naturalness ratings.

Factor	df	${n_p}^2$	F	p
Participant	18	0.74	0.25	< .001
Contour	3	0.21	513.14	< .001
Group	1	—	1.35	.260
Contour x Group	3	0.86	7.54	< .001

Figure 2.

Mean ratings of speech naturalness for each type of $f_o$ contour for PWL (maroon) and communication partners (peach).Error bars represent 95% confidence intervals. Post-hoc pairwise comparisons revealed statistical significance as indicated by the brackets (*p < .05).

Speech Aid Feature Preferences

The Levene’s test showed that the variance of the feature importance scores between PWL and primary communication partners was not significantly different (p = .50). The results from the two-way ANOVA showed that only speech aid feature demonstrated a significant, large effect in the model (p < .001, ${{\eta }_p}^2=0.32$; Table 4). The Tukey post-hoc analysis showed that sound quality was the most important speech aid feature, with a significantly higher preference than maintenance, cost, wearability, training, and visibility (Fig. 3a). However, sound quality was not statistically significantly higher than intelligibility, pitch, delay, volume, handsfree, or maintenance (which received the subsequent greatest preference scores in the listed order; Fig. 3b). Visibility demonstrated the lowest score for importance. All statistically significant comparisons showed a strong effect size (d > 0.8) except for training, which showed a medium effect with handsfree, intelligibility, and pitch (d > 0.6) and small effect with volume and delay (d > 0.2).

Table 4.

Results of analysis of variance test performed for speech aid feature ratings

Factor	df	${n_p}^2$	F	p
Feature	10	0.32	9.32	< .001
Group	1	—	0.12	.732
Feature x Group	10	—	0.99	.456

Figure 3.

a) Mean of scores for each speech aid feature. Error bars represent 95% confidence intervals.

b) Statistical significance matrix of speech aid features. Orange squares=p < .05, teal squares=p > .05, gray square indicates no comparison was performed.

Discussion

This study aimed to elucidate how a core group of potential SSI users perceive different modulations of prosodic cues in synthesized speech as well as their preferred features in speech aids. Three sentences were synthesized from a state-of-the-art, sEMG-based SSI. Each synthesized sentence had four variations of $f_o$ contours: flat, discrete, continuous, and EL. Twenty listeners, consisting of PWLs and primary communication partners, rated their perceived naturalness of each variation of each sentence. Additionally, listeners provided ratings of 11 common speech aid features in terms of which they believed to be the most or least important. The results of this study may serve as a guide for future development of SSIs to prioritize features that meet PWL needs regarding speech prosody and device characteristics.

Perceived Speech Naturalness

We hypothesized that the continuous pitch contours would be rated the highest among listeners. Our results supported this conclusion as continuous pitch contours were rated higher than all other types. This result was unsurprising as the continuous contours were personalized to the speaker’s muscle activity rather than generically generated based on the grammatical content of the sentence.

Interestingly, we found no difference between discrete and flat contours. This finding contradicts our previous work which found that synthesized speech was perceived as more natural with sentence-level variations in $f_o$ than with a flat $f_o$.⁶² These differences may be explained by the listener group of the current study comprising experienced users of alaryngeal speech while our previous study involved naïve listeners. Since PWLs and their primary communication partners may be more acquainted with the prosodic limitations of alaryngeal speech, the flat $f_o$ contour may sound more familiar thus leading it to have a similar naturalness rating to the discrete contours. This phenomenon has been observed in several other types of disordered speech where naïve listeners rate atypical speech more harshly than patients or clinicians.^65–67 While it is reasonable to speculate this to be an underlying cause, more work is needed to investigate this notion, specifically for synthesized speech from PWLs. Our contradictory results may additionally be explained by different methodologies to produce synthetic speech: a generic voice bank as used in our previous work instead of signal mapping based on the individual’s own acoustic characteristics as used here. Future work should aim to more comprehensively examine the reception of these different methodologies by both SSI users and listeners.

There was a significant interaction effect between listener group and the different types of contours on the mean naturalness rating. Both PWLs and their primary communication partners agreed that continuous contours were the most natural, EL speech was the least natural, and that there was no significant difference between discrete and flat contours. Yet overall naturalness rating of the continuous contours differed between listener groups. For PWLs, continuous contours were rated the highest, but not significantly different from the other two $f_o$ contours. In contrast, primary communication partners found continuous contours significantly preferable than flat contours but did not perceive them as different from discrete contours. The discrepancy in statistical significance may be explained by the imbalance in group size (14 PWLs, 5 partners). Additionally, the difference may arise from the varying perspectives on speech naturalness. PWLs may have rated speech naturalness relative to their own monotonic alaryngeal voice, whereas primary communication partners likely assessed naturalness of the synthesized sentences relative to typical speech. This effect parallels findings of our previous work, where EL users preferred using a monopitch EL despite having access to an EL with pitch control; in this case, users reported that ELs with pitch control were cognitively complex and not worth the additional effort to master.¹¹ Over time, PWLs may identify with their relatively monotonic voices and no longer associate modulation with naturalness, thereby resulting in a weak differentiation between flat and dynamic $f_o$ contours. Likewise, their communication partners might become accustomed to diminished prosody and thereby discount nuanced differences between continuous and discrete $f_o$ patterns, despite appreciating flat contours as different from continuous. These subtle shifts in $f_o$ contour perception could have been accentuated by the marked difference between our synthesized EL versus laryngeal speech samples, causing both listener groups to perceptually lump the non-EL samples together. Given that EL speech is consistently perceived as highly unnatural (as verified in our results), future work might better distinguish among $f_o$ modulation strategies for synthesized SSI speech by eliminating comparisons to EL samples and instead using natural speech as a point of comparison.

Speech Aid Feature Preferences

Most studies on speech aids—particularly those focusing on speech-generating devices—prioritize speech quality and intelligibility as primary outcomes. However, there is a notable gap in research concerning other characteristics that may be critical to end users, especially from the perspective of current needs and technological advancements. To our knowledge, no recent studies have systematically identified and rated the relative importance of various features of speech aids according to users aside from sparse user-specific case studies. Addressing this gap is essential for identifying common trends in preferred features and understanding which features vary in importance among different users. To initiate this discourse, we surveyed a subset of individuals who use speech aids (i.e., PWLs) and their primary communication partners to rate the importance of several different speech aid features. Across participants, features were rated in the following order: sound quality, intelligibility, pitch, delay, volume, handsfree, maintenance, cost, wearability, training, and visibility.

The 11 features can be categorized into two distinct groups: communication reception and physical operation. The results of our study indicate that users of speech aids prefer effective communication (intelligibility, sound quality, pitch, volume, and delay) over operational considerations (training, handsfree functionality, maintenance, cost, wearability, and visibility). Specifically, all five features associated with communication reception were rated as more important than features associated with the physical operation of the speech aid. These results underscore the need for alaryngeal speech aid development to focus on enhancing communication effectiveness, such as incorporating prosodic cues in the speech output of these devices. Broadly, these findings resonate with the perspectives observed among augmentative and alternative communication (AAC) users, in that the benefits of “being understood” outweigh the challenges of using a communication aid.⁶⁸ One focus group discussing the research priorities of AAC users and their communication partners similarly concluded that such systems should be responsive to individual needs and allow users to communicate successfully in specific situations.⁶⁹ The results of our study further reinforces these values while also providing new quantitative evidence of what PWLs prioritize in their speech aids.

Limitations and Future Work

Our current study demonstrates how a core set of SSI users who rely on alaryngeal speech prefer synthesized versions of typical speech relative to synthesized EL speech. Since PwL and their primary communication partners are experienced listeners, we acknowledge the potential for bias in their preferences compared to naïve listeners. Future research should include both experienced and naïve listeners to better understand how exposure to alaryngeal speech influences speaker and listener preferences. Furthermore, it is important to note that speech and voice have a wealth of characteristics that were outside the scope of this study. For one, prosody includes elements beyond $f_o$ modulation—including intensity, rate, and other spectral characteristics—which future work should consider. Although speech naturalness includes various elements of prosody, other metrics are also relevant for assessing speech function. In this study, we selected speech naturalness as a primary measure to balance the depth of information obtained with the overall study duration. Yet other measures, such as acceptability and communication efficiency, may provide insights into additional aspects of speech health.^62,70

While the purpose of this study was to assess a key aspect of synthetic speech perception (i.e., naturalness), future studies should consider other metrics and a range of social contexts to capture a comprehensive assessment of the speech. Beyond prosodic elements, other variables affect the rating of naturalness. Dall et al.⁷¹ found that spontaneous conversational speech was consistently rated more natural-sounding than speech read from a prompt (as was done in our study). Additionally, the ratings were significantly different even when listeners were instructed to perceive the speech “as if they were having a conversation” or “as if somebody was reading aloud.”⁷¹ Using conversational speech was outside the scope of our current study given our focus on the perceptual impact of pitch contours, but broader speaking contexts should be studied. Future work should also examine a larger range of $f_o$ contours and synthesis approaches. For example, although our discrete $f_o$ contours were designed to best represent natural and appropriate pitch patterns, other patterns with wider or narrower $f_o$ ranges may have been perceived differently. Likewise, the continuous $f_o$ contours derived directly from speaker participants’ sEMG recordings could have mapped more widely or narrowly onto $f_o$ ranges instead of strictly relating to their acoustic recordings. Ultimately, sEMG-based prosodic control will likely be individualized for PWLs working with their healthcare team during their initial SSI fitting and instruction.

Some patients might do best with pitch contours selected through sEMG-based pattern recognition—particularly if their treatment includes radical neck dissection and radiation therapy, thereby limiting pitch-related sEMG content. In contrast, other patients might prefer pitch control continuously mapped onto their sEMG signals. In either case, dynamic $f_o$ range could be easily adjusted to best serve a user’s language, dialect, speaking context, and mood all afforded by the powerful flexibility of speech synthesis. The speakers in our study were pseudorandomly selected from a larger sample (n = 10) to ensure an equal male-to-female ratio while balancing statistical power and the duration of the perceptual study. However, it is important to note that these speakers had normal neck anatomy, thereby generating sEMG signals that may not accurately reflect the physiological variations present in individuals with altered neck anatomy (e.g., PWL). The selected speakers were also able to produce $f_o$ contours for model training, which may not be attainable for PWL who rely on alternative speech production methods. Consequently, while our study focused on perceived outcomes rather than the specific inputs of SSIs, our findings may not be broadly applicable to the PWL population. The limited sample size of speakers (n = 4) further compounds this issue, as it restricts the diversity of speaker characteristics and experiences represented in the results. As a result, caution should be exercised when extrapolating these findings to the wider PWL community. Future research with a more diverse and representative sample of speakers is essential to enhance the generalizability of the findings and ensure they are applicable to individuals with varying speech production challenges.

Our study gained some valuable initial insight into what speech aid features are most important to their users. Although PWLs and primary communication partners did not exhibit clear differences in their preferences, it is possible that other factors influence these ratings. For example, in the development of hearing aids—another assistive technology with extensive consumer research due to its broader market—studies have found that user preferences vary based on demographic factors and the severity of hearing loss.⁷² As such, future work should adopt similar considerations to investigate what factors influence consumers’ preferences in both SSIs and their application in speech aids.

Conclusion

As SSIs continue to advance, obtaining feedback from target users is essential to guiding the effective development of these technologies. Incorporating the preferences of both users and listeners is key for the clinical viability and acceptance of sEMG-based SSIs for alaryngeal speech. Our study shows that PWLs and their primary communication partners strongly preferred the synthetic speech with prosody generated by our models over monotone EL speech. Although it is well documented that prosodic cues involving pitch are an important factor in perceived speech naturalness, the method in which these cues are introduced into synthetic speech had not yet been investigated; our findings indicate that continuous mapping of $f_o$ to time-varying muscle activity was preferred over discrete, sentence-level mapping. However, future studies should investigate how these preferences hold across different types of speech, such as conversational speech. Finally, we also obtained a list of speech aid features that PWLs and their primary communication partners valued most. Having quality communication was deemed more important than reducing physical burdens as quality of speech, intelligibility, pitch control, little delay, and volume control rated as the most important features. Taken together, these listener preferences and speech aid design feature rankings provide foundational insight for SSI development.