Skip to main content
Scientific Reports logoLink to Scientific Reports
. 2025 Jan 8;15:1361. doi: 10.1038/s41598-024-84728-y

Artificial intelligence empowered voice generation for amyotrophic lateral sclerosis patients

Stefano Regondi 1,4,✉, Giordana Donvito 1, Emanuele Frontoni 1,2, Milutin Kostovic 1, Fabio Minazzi 3, Sébastien Bratières 3, Massimiliano Filosto 4,5, Raffaele Pugliese 1,✉
PMCID: PMC11711320  PMID: 39779800

Abstract

Amyotrophic Lateral Sclerosis (ALS) is a neurodegenerative disease that can result in a progressive loss of speech due to bulbar dysfunction, which can have significant negative impact on the patient’s mental well-being. Alternative Augmentative Communication (AAC) strategies based on synthetic voices have been shown to assist patients in maintaining communication and improving their Quality of Life (QoL). However, such synthetic voices are often perceived as impersonal and fail to capture the unique voice and identity of the patient. To tackle this issue, combining voice banking (VB) and artificial intelligence (AI) has emerged as a more natural communication strategy, enabling individuals to preserve their voice for use with AAC devices as needed. This involves recording speech samples to generate a synthetic voice closely resembling the individual’s own. Despite the increasing interest in VB, there’s a lack of clear strategies for its effective implementation in rapidly progressing diseases like ALS. Additionally, the perceptual quality of VB on patients with preserved speech, especially when offered early in the disease, remains poorly understood. In light of these challenges, this study aims to assess the effectiveness and the perceptual impact of AI-generated voices on ALS patients with preserved speech, utilizing a personalized voice synthesis system based on machine learning. The AI-generated patient-specific voice is achieved through voice recording, followed by fine-tuning using a Generative Adversarial Network for Efficient and High Fidelity Speech Synthesis (HiFi-GAN), resulting in a model capable of producing speech highly similar to the patient’s own voice, with exceptional expressive and audio quality. By addressing these aspects, this study intends to offer valuable insights into the potential benefits and challenges of combining VB with AI voices to enhance communication support for ALS patients.

Keywords: Amyotrophic lateral sclerosis, Artificial intelligence, HiFi-GAN, Voice banking, Synthetic voice, Voice generation, Alternative augmentative communication

Subject terms: Neurological disorders, Amyotrophic lateral sclerosis

Introduction

Amyotrophic Lateral Sclerosis (ALS) is a neurodegenerative disease that affects both upper and lower motor neurons, resulting in motor and extra-motor symptoms1,2. One of the most common symptoms of ALS is a decline in the ability to speak, which can occur regardless of the type of onset3. This weakening of the ability to communicate naturally leads to an speech production impairments (dysarthria), leading to devastating social and psychological consequences that negatively impact the patient’s quality of life (QoL)4,5. Communication is a crucial aspect of personal identity and social context. Its impairment can devastate a person’s role in their environment and negatively affect their emotional balance6.

Indeed, communication is inherently a collaborative process in which participants jointly construct and interpret meaning through various channels, including verbal, non-verbal, and contextual cues. For individuals living with ALS, sustaining effective communication becomes progressively more difficult as the disease advances. While speech intelligibility is often the primary concern, communication effectiveness extends beyond this to include factors such as reduced speech rate, increased fatigue during prolonged interactions, shifts in social participation, and the impact on both personal and professional relationships. Addressing these broader challenges is crucial to preserving meaningful communication for people with ALS.

Patients with communicative vulnerability are often trained in alternative augmentative communication (AAC) strategies, which involve both low- and high-technology devices based on residual abilities and cognitive profile. Speech Generation Devices (SGDs) are computer-based systems that enable controlled communication in a home environment, thus helping individuals with communicative vulnerability to communicate effectively. Different studies have shown that about 70% of ALS patients could benefit from AAC7,8.

However, the synthetic voice produced by these AAC devices can be perceived as impersonal and lacking in expressivity and emotion, which is a significant problem for many patients9,10. To address this issue, voice banking (VB) has been introduced as a more natural communication strategy, allowing individuals to bank their voice and access it through an AAC device whenever they need to11. This process involves recording samples of a person’s speech to create a synthetic voice that sounds like their own. Although synthetic voice technology primarily targets accurate acoustic reproduction, its contribution to communication extends far beyond sound. By maintaining familiar vocal characteristics, it enhances communication effectiveness by aiding partner comprehension, preserving the speaker’s personal identity, and reducing cognitive effort for both the speaker and the listener. Additionally, it enables individuals to remain engaged in diverse social contexts and supports emotional expression through the retention of familiar voice qualities, making interactions more meaningful and authentic.

Despite the growing interest in VB, there is currently no clear strategy for introducing it effectively in the context of a progressive and relatively rapid disease such as ALS11,12. Additionally, the perceptual impact of VB on patients with preserved speech is not well understood, especially when the option to record a personal voice synthesis is offered to ALS patients in the early stages of the disease, when speech production is still preserved13,14.

This study aims to evaluate the effectiveness of AI-generated voices in facilitating communication for ALS patients with preserved speech. Specifically, we focus on analyzing the technical quality and acoustic fidelity of synthetic voices generated through the HiFi-GAN model by examining key voice features including intensity, pitch, formants, and Mel Cepstral Distance (MCD). Our analysis compares these features between natural and synthetic voices to assess how closely the AI-generated voices match the original vocal characteristics of ALS patients. Additionally, we evaluate the perceptual quality of the synthetic voices using Mean Opinion Score (MOS) testing, providing quantitative measures of naturalness and similarity to the original voices. Through this comprehensive technical evaluation, we aim to establish the viability of HiFi-GAN as a voice banking solution for ALS patients, contributing to the development of evidence-based strategies for implementing AI-driven voice synthesis in ALS. Overall, the findings from this study have important implications for improving communication support tools for individuals with progressive neurological disorders like ALS. However, other aspects of communication, such as the speed of interaction and integration with external devices, are not directly impacted by voice banking alone. Communication is multifaceted, and effective support must also consider factors like device usability, communication rate, and the interaction dynamics with communication partners. The insights gained here primarily enhance the emotional and personal resonance of synthetic voices, but further technological advancements are needed to address other key aspects of communication.

Voice banking

VB is the process of recording a person’s voice and saving it for later use, e.g. for text-to-speech playback or in text-to-speech synthesis. Individuals with degenerative diseases, such as ALS, who may lose their ability to speak over time, often use it. Recording one’s own voice while still able to do so enables them to later use text-to-speech systems to communicate with a synthetic voice that sounds like their own. In recent years, VB has garnered significant attention as a potential solution to the communication challenges faced by people with degenerative diseases. For instance, VB has been shown to provide several benefits for people with ALS. In the study reported by Cave and Bloch11, participants who used VB reported feeling more in control of their communication and more positive outlook on their future. Further, they reported that their custom synthetic voice sounded more like their own voice than other synthetic voices.

Despite the benefits of VB, synthetic voices generated from voice banks may not always effectively convey the full spectrum of human emotion, which limits their naturalness and expressiveness. Recent advances in VB technology aimed to address some of these challenges. For instance, some systems currently allow the recording of expressive speech, such as laughter and crying, to enhance the naturalness of the synthetic voice. Moreover, deep learning algorithms have been used in some systems to synthesize more natural-sounding voices with smaller amounts of recorded speech data.

To emphasize, VB has the potential to be used in several healthcare applications apart from aiding communication for individuals with degenerative diseases. Synthetic voices generated from voice banks can be used to assist people with speech disorders, as well as providing voice prompts for medical devices. Indeed, recent studies have explored the potential healthcare applications of VB-based-technology15,16. For example, it has been explored the use of synthetic voices generated from voice banks to deliver personalized health messages to patients15. The study found that patients who received personalized messages with synthetic voices were more likely to engage with the messages and follow through with their health goals.

In our study, instead of synthesizing a caretaker’s or a third party’s voice with high fidelity, we focus on recreating the user’s own voice for artificial speech generation, which is crucial for improving user satisfaction and maintaining personal identity.

In conclusion, VB has several benefits for individuals with degenerative diseases, but also presents challenges in terms of implementation and naturalness of synthetic voices. Advances in VB technology have sought to address these challenges, and VB has potential applications in healthcare beyond communication for individuals with degenerative diseases.

Voice generation

Voice generation, also known as text-to-speech synthesis, is the process of generating synthetic speech from text. The field of speech synthesis has witnessed significant advancements over the past few decades. Early approaches to speech synthesis include concatenate and parametric text-to-speech (TTS) techniques. Concatenate TTS involves the construction of synthetic speech by concatenating pre-recorded speech segments17. In contrast, parametric TTS utilizes mathematical models to generate speech waveforms based on input parameters, such as pitch and duration18. Although these methods were widely adopted in their time, they often resulted in synthetic speech with limited quality and naturalness.

More recently, neural speech synthesis has emerged as the state of the art, largely supplanting concatenate and parametric TTS techniques19. Neural TTS systems rely on deep learning models to generate high-quality, natural-sounding speech. One prominent example is WaveNet, an autoregressive generative model that directly generates raw audio waveforms20. Another is Tacotron, an end-to-end TTS model that converts text to speech using an attention mechanism21,22. These advancements in neural TTS have enabled more realistic and expressive synthesized speech, even in challenging conditions.

In this study, we selected a neural architecture composed of HiFi-GAN and FastPitch to perform speech synthesis for ALS patients. HiFi-GAN is a generative adversarial network that generates high-fidelity speech waveforms23, while FastPitch is an efficient pitch extraction model that enables real-time synthesis with expressive control24. This combination of models allows us to generate high-quality, personalized speech utterance for our target population. The HiFi-GAN model was trained using a subset of the recorded voice samples, while a separate set of phrases, not included in the training, was reserved for testing. This approach simulates real-world scenarios where the synthesized voice must generate phrases that were never directly recorded by the patient, demonstrating the model’s ability to generalize beyond the training data.

To create a personalized voice model, we perform neural network fine-tuning of a pre-trained voice model. This pre-trained model has been trained on 20 h of speech data from the same language, but with different speakers. Fine-tuning the pre-trained model with the target speaker’s voice data enables us to adapt the model to the specific characteristics of their speech, resulting in a more accurate and natural-sounding synthesized voice.

Materials and methods

Dataset

This study includes a cohort of 10 ALS patients, with a median age at evaluation of 67.5 years (range: 29–77 years) and a male-to-female ratio of 1:1, fulfilling a diagnosis of probable or definite ALS according to the revised El Escorial criteria25. Demographic characteristics: 70% of the ALS patients were from Northern Italy, while 30% were from Central Italy.

All ALS patients included in the study reported preserved speech at the time of enrolment, with an ALS-FRS language score of ≥ 1. Patients with a severe cognitive and behavioral impairment and with an ALS-FRS language score of ≤ 1 at enrollment were excluded from the study. The study followed the principles outlined in the World Medical Association Declaration of Helsinki received approval from Territorial Ethics Committee Lombardia 6, I.R.C.C.S. Policlinico “San Matteo” of Pavia (project number 0003700/24). Informed consent was obtained from all participants following a thorough explanation of the study’s objective and methodology. To simplify and ensure anonymity, we referred to the 10 ALS patients enrolled in the study as Users (U1–U10).

Voice banking procedure

Prior to the VB activity, ALS patients were given instructions on how to perform voice recordings. The guideline included suggestions and recommendations on various aspects, such as posture, distance, and position to the microphone, as well as how to read the default message list in a natural and spontaneous way. Participants were advised to record their voice in a proper environment (i.e., a silent room) and to use speech strategies to preserve the quality of their voice (i.e., taking pauses and drinking water during each recording session). Each recording session lasted approximately 45–60 min, with breaks offered every 15–20 min or as needed by the participant to minimize fatigue and maintain voice quality. Participants were encouraged to provide feedback about their comfort levels and voice fatigue during the sessions. We provided water and encouraged rest periods when participants showed signs of fatigue or voiced discomfort.

For this study, participants were asked to record a group of sentences in order to generate an individual synthetic voice that sounded like their own one.

Three different scripts varied based on the number of required messages to record. Each script required recording between three hundred and five hundred sentences. The average sentence length is approximately 65 characters, with the shortest sentence around 15 characters and the longest reaching up to 150 characters. On average, each sentence contains about 12 words. The content of the scripts spans various everyday topics, including personal conversations, family matters, daily routines, health and wellness, travel, event planning, education, and work. Linguistically, the scripts predominantly feature an informal register, incorporating a mix of questions and statements, frequent colloquial expressions, and a high usage of first- and second-person pronouns. Additionally, modal verbs appear often, enhancing the conversational tone. Participants recorded batches of four or five sentences at a time and were advised to read the content of the sentence as if they were conversing spontaneously. The sentences were of varying lengths and composed of simple terms, and were recorded using a microphone and the open-source audio processing program Audacity. No time limit was given to complete the registration of the script. At the end of the recording phase, the elapsed time for each session and for the entire recording was noted, along with the number of sentences completed.

Cloud infrastructure for voice management and delivery

The creation and delivery of voices in this study is based on a cloud infrastructure, due to the following considerations: (1) recorded samples of voice should be made available at any time to their users; (2) in order to provide high quality synthesized voices, recent neural network algorithms were selected, requiring over 300 MB of storage per voice; (3) to synthesize speech in real time the voice models require high-end GPUs, offering a latency of below 500 ms after the user input.

The architecture selected is described below. At high-level we see the space for storing the banked voices, the deep learning training, the API for voice generation, and the privacy and security services. Figure 1 visually represents the main components.

Fig. 1.

Fig. 1

Summary of the main components of Cloud Architecture.

Data storage: the first component of the cloud architecture is a data storage system. This system provides scalable, secure, and high-performance storage for recorded voice data. The system supports versioning and backups to protect against data loss.

Deep learning training: the second component of the cloud architecture is a deep learning platform for training voice generation models. This platform is able to scale up and down depending on the computational requirements of the training process. We implemented a semi-automatic process, planning to scale up the process when more than 4 voice models need to be created every day. We currently manually launch training processes and quality controls. In the future, the platform will also be able to automatically spin up new training servers when necessary and spin down idle servers to minimize costs.

API for voice generation: the third component of the cloud architecture is an API for generating synthetic voices. This API is designed to make the trained models available to the patients in a seamless way. The overall system is designed to be highly available, fault-tolerant, and secure.

Privacy and security services: the fourth component of the cloud architecture is a set of privacy and security services to protect the sensitive voice data. This includes encryption of the data in transit and at rest, user authentication and authorization, and access control mechanisms to ensure that only authorized users can access the data. Only the designated user can use the assigned voice.

Monitoring and logging system is the final component of the cloud architecture. This system provides real-time monitoring of the health and performance of the various components in the architecture. It should also provide logs that can be used for auditing, troubleshooting, and analysis.

In terms of specific services and technologies that can be used to implement this architecture, here are some suggestions:

Data storage - Services like Amazon S3 or Google Cloud Storage can provide scalable, secure, and high-performance data storage.

Deep learning training - Services like Amazon SageMaker or Google Cloud AI Platform can provide a scalable, managed platform for deep learning training.

API for voice generation - Services like Amazon API Gateway or Google Cloud Endpoints can provide a scalable, secure, and highly available API for generating synthetic voices.

Privacy and security services - Services like Amazon Cognito or Google Cloud Identity can provide user authentication and authorization, while encryption can be provided by services like Amazon Key Management Service or Google Cloud KMS. Monitoring and logging - Services like Amazon CloudWatch or Google Cloud Logging can provide real-time monitoring and logging of the various components in the architecture.

The specific services and technologies used will depend on the specific requirements and constraints of the project. However, the overall architecture described here should provide a scalable, secure, and reliable cloud infrastructure for voice banking, deep learning training, API for voice generation, and privacy and security services.

Experimental setup: voice recording and artificial voice generation

To ensure that the quality of the recording was adequate in capturing the timbre and intonation of the individual’s voice, the voice recording was conducted in a soundproof room where test recordings were previously taken. The laptop provided had the Audacity software installed, which is required for recording audio and converting it to a digital format. Additionally, a professional microphone with a table arm was set up for recording (MV7 Shure). Three different “scripts” were prepared, consisting of brief and straightforward sentences. Speaking instructions were carefully crafted, to ensure prosody as natural and fluid as possible. Once recorded it was necessary to preprocess the dataset by cleaning, normalizing, down-sampling to 22 kHz and formatting the data to ensure that it is top quality and compatible with the voice generation system.

Subsequently the voice generation model is trained using the preprocessed dataset. The training process may involve optimizing the model architecture, hyper-parameters, and training algorithm to achieve the best possible performance. Once the training of the voice generation model is completed, the subsequent stage involves producing artificial voices utilizing the model.

Comparative method and evaluation metrics

The comparative method used to evaluate the quality and resemblance of AI-generated synthetic voices by comparing them against their human counterparts is based on a series of voice features, which act as evaluation metrics. These features include Pitch (fundamental frequency in Hz), Intensity (sound intensity in dB), formants (formant frequencies in Hz), Mel frequency cepstral coefficients (MFCC), and Mel Cepstral Distance (MCD). Each features offers a unique perspective on vocal characteristics, allowing for detailed and comprehensive analysis of the voices in the dataset. Furthermore, the Mean Opinion Score (MOS) testing was conducted to assess perceived naturalness and similarity of the synthetic voices to the original recordings. While MOS provides valuable data about voice quality perception, it should not be interpreted as a measure of psychological impact.

Praat, a freely available software package designed for speech analysis in phonetics, served as the primary tool for the analysis of pitch, intensity, and formants. Default settings within Praat were employed for the extraction of these features across the recorded speech samples.

Mel-Cepstral Distortion analysis was performed utilizing a Python script26.

Results and discussion

Pitch, intensity, and formats evaluation

To evaluate the quality, similarities, and differences between synthetic and natural voices of ALS patients, we conducted a comprehensive analysis using several key vocal biomarkers: pitch, intensity, and formants (F1 and F2). Figure 2 illustrates the extraction of vocal features from both natural and synthetic voices using HiFi-GAN.

Fig. 2.

Fig. 2

Extraction of audio waveforms, spectrograms, pitch, and intensity features from both natural and synthetic voices using HiFi-GAN across various test dataset scripts. The analyzed sentences include: U1 - Choose what to do once in Naples. U2 - Please, turn on the TV for me. U3 - Would you mind if I came too? U4 - I was thinking we could rent a car once in Rome. U5 - The cheeks were flat and not protruding. U6 - Let’s go back to the previous page for a moment. U7 - Choose what to do once in Naples, come on. U8 - Have you already packed your bags? U9 - The man recognized the importance of the finding. U10 - I want an ochre-colored wool sweater. These sentences, selected for their similarities in average sentence length, character count, content, and linguistic features, demonstrate the generalizability and robustness of the extracted features across both natural and synthetic speech.

The pitch values of synthetic voices closely matched those of the natural voices of ALS patients, indicating that the expressive pitch quality of the natural human voice was well-preserved even under pathological conditions (Fig. 3). Specifically, natural voices exhibited an average pitch of 155 ± 33 Hz, whereas synthetic voices generated by HiFi-GAN displayed an average pitch of 155.20 ± 35 Hz (Fig. 3A). The percentage difference in pitch between natural and synthetic voices, calculated using the following formula:Inline graphic

Fig. 3.

Fig. 3

(A) Comparison of natural and synthetic voices in ALS patients shows that HiFi-GAN can closely replicate natural voice pitch (AI) with a minimal average difference of 1.94%. (B) Synthetic voices generated by HiFi-GAN exhibit higher average intensity compared to natural voices, (BI) resulting in a 12.84% increase.

showed a variation range of ± 5%, with an average difference of 1.94% (Fig. 3AI), demonstrating a better pitch preservation compared to the study reported by Yamagishi et al.27.

For intensity, natural voices had an average intensity of 60.93 ± 5.39 dB, while synthetic voices demonstrated a higher average intensity of 68.29 ± 4.04 dB, resulting in a percentage difference of 12.84% (Fig. 3B-BI). This finding is particularly significant as it suggests that despite the presence of dysarthria in ALS patients, the HiFi-GAN model can generate synthetic voices with enhanced intensity compared to natural voices.

Such data reveals that the HiFi-GAN model effectively replicates the pitch of natural voices while enhancing the intensity, potentially improving voice quality for ALS patients. This could have significant implications for preserving certain aspects of expressiveness in ALS patients’ voices, as the synthetic voices replicate the pitch quality of their natural voices. However, while pitch is important for maintaining elements of emotional expression and vocal identity, we acknowledge that emotional expressiveness also depends on context and other factors, such as tone, rhythm, and interaction dynamics. In addition, the increased intensity in synthetic voices could improve the audibility and clarity of speech, which is often compromised in ALS patients due to dysarthria and aphasia.

Next, we focused on analyzing the F1 and F2 formants, as illustrated in Fig. 4. For this analysis, we specifically targeted the vowel ‘a’ extracted from words where this vowel was positioned between two consonants. The selected vowel portions had an average duration of 100 to 150 milliseconds, ensuring consistent and reliable comparisons between natural and synthetic voice samples.

Fig. 4.

Fig. 4

Comparison of lower (F1) and higher (F2) formant frequencies between natural and synthetic voices. (A) The average F1 values for natural voices were 774.71 Hz, while synthetic voices averaged 776.02 Hz, (A1) showing a minimal percentage difference within ± 10%. (B) For the F2 formant, natural voices averaged 1485.68 Hz and synthetic voices averaged 1496.67 Hz, (B1) with a minimal difference of ± 0.62%.

The average F1 values showed remarkable similarity between natural and synthetic voices, with averages of 774.71 Hz and 776.02 Hz, respectively (Fig. 4A). The percentage difference between the two datasets fluctuated within ± 10%, indicating a high degree of correspondence in the lower formant frequencies (Fig. 4AI).

For the F2 formant, the average values were 1485.68 Hz for natural voices and 1496.67 Hz for synthetic voices (Fig. 4B). The percentage difference between the two was minimal, at ± 0.62%, demonstrating that the HiFi-GAN model accurately replicates the higher formant frequencies as well (Fig. 4BI).

The close match in both F1 and F2 formant frequencies between natural and synthetic voices underscores the effectiveness of the HiFi-GAN model in capturing the acoustic characteristics of human speech. This is particularly significant because formants play a crucial role in vowel articulation and overall speech intelligibility. The ability to replicate these formants accurately ensures that the synthetic voices are not only similar in pitch and intensity but also in their phonetic details, contributing to a more natural and comprehensible synthetic speech output.

Furthermore, such results showed an improvement in formant matching accuracy compared to the findings of Creer et al.28 who reported a deviation of ± 15%, highlighting an enhancement in performance.

Overall, the detailed analysis of F1 and F2 formants reaffirms the capability of the HiFi-GAN model to produce high-quality synthetic voices that closely resemble the natural voices of ALS patients, thereby enhancing communication effectiveness and preserving vocal identity.

Finally, we conducted a statistical analysis of the vocal biomarkers—pitch, intensity, and formants (F1 and F2)—obtained from the natural and synthetic voice samples, as shown in Fig. 5. The analysis was performed using a two-way ANOVA. Values were expressed as means ± SD, with *p < 0.1 considered significant.

Fig. 5.

Fig. 5

Statistical analysis of voice features between natural and synthetic voices. (A) No statistically significant difference in pitch values was observed, confirming the model’s ability to preserve the original pitch quality. (B) Intensity levels showed a significant difference (**p < 0.01), with synthetic voices exhibiting higher intensity compared to natural voices. (C–D) Analysis of formant frequencies F1 and F2 indicated no significant differences, demonstrating the model’s accuracy in replicating natural formant structures, highlighting HiFi-GAN’s proficiency in producing natural-sounding, intense synthetic speech, which can improve speech intelligibility for ALS patients.

As anticipated, there was no statistically significant difference between the pitch values of natural and synthetic voices (Fig. 5A). This confirms that the HiFi-GAN model effectively preserves the pitch quality of the original voices, even under pathological conditions. In contrast, the intensity biomarker (Fig. 5B) revealed a significant difference between the natural and synthetic voices (**p < 0.01). This indicates that while the synthetic voices are able to maintain similar pitch, they tend to have higher intensity levels compared to natural voices.

Regarding the formants F1 (Fig. 5C) and F2 (Fig. 5D), the statistical analysis confirmed that there were no significant differences between the natural and synthetic voice samples. This reinforces the finding that the HiFi-GAN model accurately replicates the acoustic characteristics of human speech, maintaining the natural formant structure.

Such results underscore the proficiency of the HiFi-GAN model in generating synthetic voices that closely match natural voices in terms of pitch and formant frequencies, while exhibiting an increased intensity. These findings are critical as they demonstrate the model’s potential to produce synthetic speech that is both natural-sounding and more intense, which could benefit ALS patients by improving speech intelligibility and audibility, thus highlighting the advancements in voice synthesis technology for individuals with speech impairments.

Mel cepstral distance (MCD) evaluation

The next step was to integrate the extraction and study of the Mel Frequency Cepstral Coefficients (MFCC) and the Mel Cepstral Distance (MCD) into the analysis of real and synthetic voices for ALS patients.

MFCCs are widely used in audio signal analysis and speech recognition29. These coefficients were extracted from the audio signal spectrograms of the dataset samples through the following steps: (1) pre-processing—the audio signal is divided into short time segments, typically 20 to 40 milliseconds each; (2) fourier Transform—each segment undergoes a Fourier transform to obtain its frequency representation; (3) Mel Filtering—the spectral power values are transformed into a Mel frequency scale, which approximates human auditory perception; (4) Logarithmization calculated for each Mel frequency band; (5) Discrete Cosine Transform (DCT)—The DCT is applied to the logarithms of the power values, producing the Mel Cepstral coefficients.

The MCD is derived from the spectral characteristics of the MFCC coefficients and serves as a metric to measure the distance between two sets of Mel Cepstral coefficients. This distance can be calculated using various metrics, such as Euclidean or Mahalanobis distance. MCD is particularly useful for comparing the diversity between two vocal tracks, such as a natural voice and a synthetic voice. The MCD is a critical parameter for distinguishing natural voices from synthetic ones because it captures differences in intonation, prosody, harmonics, and other vocal attributes. By measuring the distance between the MFCC vectors of a synthetic voice and a natural voice, we can evaluate the similarity or dissimilarity of the synthetic voice in terms of vocal characteristics.

Hence, from the spectral characteristics of the MFCC coefficients (Fig. 6A), we extrapolated the MCD values (Fig. 6B). These values provide a quantitative measure of how closely the synthetic voices generated by the HiFi-GAN model resemble the natural voices of ALS patients.

Fig. 6.

Fig. 6

Spectral analysis using MFCC coefficients and MCD values for natural and synthetic voices. (A) The MFCC coefficients highlight the spectral characteristics of the voices. (B) MCD values, ranging from 12.80 to 23.83 with an average of 16.23 ± 3.6, quantify the resemblance of synthetic voices to natural voices.

The integration of MFCC and MCD into our analysis allows for a more nuanced assessment of synthetic voice quality. It offers a comprehensive evaluation of how well the synthetic voices maintain the natural characteristics of human speech, which is crucial for developing effective communication aids for ALS patients. The synthetic voice samples generated using the HiFi-GAN model exhibit MCD values ranging from 12.80 to 23.83, with an average of 16.23 ± 3.6 (Fig. 6B). These results align with findings from other AI models in the literature, such as SV2TTS30, FastSpeech24, and V2C31, which report average MCD values of 17.41, 12.08, and 11.79, respectively. This comparison indicates that the HiFi-GAN model performs competitively with existing state-of-the-art models in producing high-quality synthetic voices.

To further evaluate the quality of generated speech, we conducted an analysis by using a subjective evaluation metric, following the settings reported by Chen et al.31, using the Mean Opinion Score (MOS) evaluation approach based on subjective listening tests. The enrolled patients primarily evaluated the AI-generated speech for naturalness and similarity, focusing on whether the synthesized speech effectively conveyed the speaker’s identity and emotional expression. As reported by Chen et al.31, in order for the quality of the generated speech to be perceived as natural, the MOS-naturalness values ​​must be high (i.e., MOS ≥ 4). Experimental results show that our HiFi-GAN model produces synthetic voices that listeners rate as highly similar to natural speech, with an average MOS score of 6.013 ± 0.77. While MOS scores primarily assess the perceived naturalness and similarity of synthetic speech, they do not directly measure psychological impact. Nevertheless, the recognition of familiar voice qualities by ALS patients can significantly enhance their well-being. Preserving a patient’s natural voice fosters a sense of personal identity in social interactions, reduces anxiety about future communication limitations, and helps maintain a sense of autonomy as their condition progresses. This psychological support underscores the broader value of AI and VB technologies beyond technical assessments alone.

Conclusion

In conclusion, our comprehensive analysis of synthetic voices of ALS patients, generated using the HiFi-GAN model, revealed several key findings. In particular, the synthetic voices closely match the pitch values of natural voices, indicating that the expressive pitch quality of the natural human voice is well-preserved even under pathological conditions. Also, the synthetic voices exhibit a higher intensity than natural voices. This enhancement is particularly significant for ALS patients, as it could improve the audibility and clarity of their speech, which is often compromised due to dysarthria and aphasia. Additionally, the obtained results on the formants underscore the HiFi-GAN model’s effectiveness in accurately replicating the lower and higher formant frequencies, which are crucial for vowel articulation and overall speech intelligibility. Lastly, using the integration of MCD and MOS into our analysis demonstrated the HiFi-GAN effectiveness in generating voices that are perceived as natural and similar to the natural voices of ALS patients. Overall, the HiFi-GAN model demonstrates significant potential in generating high-quality synthetic voices that closely resemble the natural voices of ALS patients. These findings could have important implications for enhancing communication aids for individuals with speech impairments, providing them with a means to preserve their vocal identity and improve speech intelligibility and audibility. This study advances the current state of research in synthetic voice generation, offering a robust framework for future developments in this field.

While this study demonstrates technical success in voice synthesis, a key limitation is that our evaluation focused primarily on technical metrics, without incorporating feedback from the individuals with ALS who provided their voices. Although our analysis of voice banking and artificial intelligence shows promising results in reproducing voices, we acknowledge that technical measurements alone cannot fully capture the practical utility or emotional impact of these synthetic voices.

To address this, future research must prioritize user-centered assessments and real-world applications. We plan to include a systematic evaluation of user experience, the perspectives of communication partners, long-term effectiveness in preserving meaningful communication, ease of use in daily conversations, and performance across different environmental conditions. This comprehensive approach will provide deeper insights into whether synthetic voices can maintain the emotional connection essential for everyday conversations, ultimately guiding the development and implementation of voice banking technologies for people with ALS.

Acknowledgements

We are grateful to AISLA - Associazione Italiana Sclerosi Laterale Amiotrofica for supporting NeMO Lab “Voice of purpose” research projects. We are also thankful for the support from the speech synthesis team at Translated in the development of personalized voice models.

Author contributions

Conceptualization, S.R., R.P., F.M., S.B.; methodology, S.R., R.P., M.K., F.M., S.B.; software, S.R., R.P., M.K., F.M., S.B.; validation, S.R., G.D., E.F., M.K., F.M., S.B. M.F., R.P.; formal analysis, S.R., G.D., E.F., M.K., F.M., S.B. M.F., R.P.; investigation, S.R., G.D., E.F., M.K., F.M., S.B. M.F., R.P.; data curation, S.R., M.K., R.P.; writing—original draft preparation, R.P.; writing—review and editing, S.R., G.D., E.F., M.K., F.M., S.B. M.F., R.P.; visualization, R.P.; supervision, S.R., F.M, R.P.; project administration, S.R., F.M, R.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Data availability

Data is provided within the manuscript.

Declarations

Competing interests

The authors declare no competing interests.

Ethics approval and consent to participate

The study followed the principles outlined in the World Medical Association Declaration of Helsinki received approval from Territorial Ethics Committee Lombardia 6, I.R.C.C.S. Policlinico “San Matteo” of Pavia (project number 0003700/24). Informed consent was obtained from all participants following a thorough explanation of the study’s objective and methodology.

Footnotes

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Stefano Regondi, Email: stefano.regondi@nemolab.it.

Raffaele Pugliese, Email: raffaele.pugliese@nemolab.it.

References

  • 1.Feldman, E. L. et al. Amyotrophic lateral sclerosis. Lancet400, 1363–1380. 10.1016/S0140-6736(22)01272-7 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Hardiman, O. et al. Amyotrophic lateral sclerosis. Nat. Rev. Dis. Primers. 3, 17071. 10.1038/nrdp.2017.71 (2017). [DOI] [PubMed] [Google Scholar]
  • 3.Brown, R. H. & Al-Chalabi, A. Amyotrophic lateral sclerosis. N. Engl. J. Med.377, 162–172. 10.1056/NEJMra1603471 (2017). [DOI] [PubMed] [Google Scholar]
  • 4.Tomik, B. & Guiloff, R. J. Dysarthria in amyotrophic lateral sclerosis: A review. Amyotroph. Lateral Scler.11, 4–15. 10.3109/17482960802379004 (2010). [DOI] [PubMed] [Google Scholar]
  • 5.Maffei, M. F. et al. Acoustic measures of Dysphonia in Amyotrophic lateral sclerosis. J. Speech Lang. Hear. Res.66, 872–887. 10.1044/2022_JSLHR-22-00363 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Genuis, S. K., Luth, W., Bubela, T. & Johnston, W. S. What do people affected by amyotrophic lateral sclerosis want from health communications? Evidence from the ALS Talk Project. Muscle Nerve. 68, 286–295. 10.1002/mus.27935 (2023). [DOI] [PubMed] [Google Scholar]
  • 7.Beukelman, D., Fager, S. & Nordness, A. Communication support for people with ALS. Neurol. Res. Int.2011(714693). 10.1155/2011/714693 (2011). [DOI] [PMC free article] [PubMed]
  • 8.Linse, K., Aust, E., Joos, M. & Hermann, A. Communication matters-pitfalls and promise of hightech communication devices in palliative care of severely physically disabled patients with amyotrophic lateral sclerosis. Front. Neurol.9, 603. 10.3389/fneur.2018.00603 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Broomfield, K., Harrop, D., Jones, G. L., Sage, K. & Judge, S. A qualitative evidence synthesis of the experiences and perspectives of communicating using augmentative and alternative communication (AAC). Disabil. Rehabil Assist. Technol. 1–15. 10.1080/17483107.2022.2105961 (2022). [DOI] [PMC free article] [PubMed]
  • 10.Elsahar, Y., Hu, S., Bouazza-Marouf, K., Kerr, D. & Mansor, A. Augmentative and alternative communication (AAC) advances: A review of configurations for individuals with a Speech disability. Sens. (Basel). 19. 10.3390/s19081911 (2019). [DOI] [PMC free article] [PubMed]
  • 11.Cave, R. & Bloch, S. Voice banking for people living with motor neurone disease: Views and expectations. Int. J. Lang. Commun. Disord. 56, 116–129. 10.1111/1460-6984.12588 (2021). [DOI] [PubMed] [Google Scholar]
  • 12.Brent, J. R., Franz, C. K., Coleman, J. M., Ajroud-Driss, S. & 3rd & ALS: Management problems. Neurol. Clin.38, 565–575. 10.1016/j.ncl.2020.03.013 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Peters, B., O’Brien, K. & Fried-Oken, M. A recent survey of augmentative and alternative communication use and service delivery experiences of people with amyotrophic lateral sclerosis in the United States. Disabil. Rehabil Assist. Technol.19, 1121–1134. 10.1080/17483107.2022.2149866 (2024). [DOI] [PubMed] [Google Scholar]
  • 14.Neto, L. L., Constantini, A. C. & Chun, R. Y. Communication vulnerable in patients with amyotrophic lateral sclerosis: A systematic review. NeuroRehabilitation40, 561–568. 10.3233/NRE-171443 (2017). [DOI] [PubMed] [Google Scholar]
  • 15.Oosthuizen, I., Dada, S., Bornman, J. & Koul, R. Message banking: Perceptions of persons with motor neuron disease, significant others and clinicians. Int. J. Speech Lang. Pathol.20, 756–765. 10.1080/17549507.2017.1356377 (2018). [DOI] [PubMed] [Google Scholar]
  • 16.Simon, J. & Nicola, H. Voice banking for individuals living with MND: A service review. Technol. Disabil.34 (2022).
  • 17.Hunt, A. J. & Black, A. W. Unit selection in a concatenative speech synthesis system using a large speech database. ICASSP-96. Conference Proceedings 1 (1996).
  • 18.Nagaraj, A. & Prasanna, R. M. Acoustic features modelling for statistical parametric speech synthesis: A review. IETE Tech. Rev.36.
  • 19.Naihan Li et al. Neural speech synthesis with transformer network. arXiv (2019).
  • 20.Aaron et al. WaveNet: A generative model for raw audio. arXiv (2016).
  • 21.Yu, J. et al. DIA-TTS: Deep-inherited attention-based text-to-Speech Synthesizer. Entropy (Basel). 2510.3390/e25010041 (2022). [DOI] [PMC free article] [PubMed]
  • 22.Yuxuan et al. Tacotron: Towards end-to-end speech synthesis. arXiv (2017).
  • 23.Jiaqi Su, Z., Finkelstein, A. & HiFi-GAN Jin high-fidelity denoising and dereverberation based on speech deep features in adversarial networks. arXiv (2020).
  • 24.Łańcucki, A. & FastPitch parallel text-to-speech with pitch prediction. arXiv (2021).
  • 25.Brooks, B. R., Miller, R. G., Swash, M., Munsat, T. L. & World Federation of Neurology Research Group on Motor. Neuron, D. El Escorial revisited: Revised criteria for the diagnosis of amyotrophic lateral sclerosis. Amyotroph. Lateral Scler. Other Motor Neuron Disord. 1, 293–299. 10.1080/146608200300079536 (2000). [DOI] [PubMed] [Google Scholar]
  • 26.< https: //github.com/MattShannon/mcd>.
  • 27.Yamagishi, J., Veaux, C. & Renals, S. Simon King Speech synthesis technologies for individuals with vocal disabilities: Voice banking and reconstruction. Acoust. Sci. Tech.33 (2012).
  • 28.Sarah Creer, S., Cunningham, P. G. & Yamagishi, J. Building personalised synthetic voices for individuals with severe speech impairment. Comput. Speech Lang.27 (2013).
  • 29.Shashidhar, G., Koolagudi, D. R. & Rao, K. S. Identification of Language using Mel-frequency cepstral coefficients (MFCC) author links open overlay panel. Procedia Eng.38 (2012).
  • 30.Hu, W. & Zhu, X. A real-time voice cloning system with multiple algorithms for speech quality improvement. PLoS One. 18, e0283440. 10.1371/journal.pone.0283440 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Qi et al. V2C: Visual Voice Cloning. arXiv (2021).

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Data is provided within the manuscript.


Articles from Scientific Reports are provided here courtesy of Nature Publishing Group

RESOURCES