Computing nasalance with MFCCs and Convolutional Neural Networks

Andrés Lozano; Enrique Nava; María Dolores García Méndez; Ignacio Moreno-Torres

doi:10.1371/journal.pone.0315452

. 2024 Dec 31;19(12):e0315452. doi: 10.1371/journal.pone.0315452

Computing nasalance with MFCCs and Convolutional Neural Networks

Andrés Lozano ^1,^*, Enrique Nava ¹, María Dolores García Méndez ², Ignacio Moreno-Torres ²

Editor: Laura Morett³

PMCID: PMC11687758 PMID: 39739659

Abstract

Nasalance is a valuable clinical biomarker for hypernasality. It is computed as the ratio of acoustic energy emitted through the nose to the total energy emitted through the mouth and nose (eNasalance). A new approach is proposed to compute nasalance using Convolutional Neural Networks (CNNs) trained with Mel-Frequency Cepstrum Coefficients (mfccNasalance). mfccNasalance is evaluated by examining its accuracy: 1) when the train and test data are from the same or from different dialects; 2) with test data that differs in dynamicity (e.g. rapidly produced diadochokinetic syllables versus short words); and 3) using multiple CNN configurations (i.e. kernel shape and use of 1 × 1 pointwise convolution). Dual-channel Nasometer speech data from healthy speakers from different dialects: Costa Rica, more(+) nasal, Spain and Chile, less(-) nasal, are recorded. The input to the CNN models were sequences of 39 MFCC vectors computed from 250 ms moving windows. The test data were recorded in Spain and included short words (-dynamic), sentences (+dynamic), and diadochokinetic syllables (+dynamic). The accuracy of a CNN model was defined as the Spearman correlation between the mfccNasalance for that model and the perceptual nasality scores of human experts. In the same-dialect condition, mfccNasalance was more accurate than eNasalance independently of the CNN configuration; using a 1 × 1 kernel resulted in increased accuracy for +dynamic utterances (p < .000), though not for -dynamic utterances. The kernel shape had a significant impact for -dynamic utterances (p < .000) exclusively. In the different-dialect condition, the scores were significantly less accurate than in the same-dialect condition, particularly for Costa Rica trained models. We conclude that mfccNasalance is a flexible and useful alternative to eNasalance. Future studies should explore how to optimize mfccNasalance by selecting the most adequate CNN model as a function of the dynamicity of the target speech data.

Introduction

Hypernasality (HN) is one of the consequences of Velo-Pharyngeal Dysfunction (VPD). It refers to the presence of excessive nasal resonance when producing oral speech [1]. HN may be caused by anatomical malformations, as in patients with cleft palate (CP), or speech motor-disorders [2]. Evaluating HN is most relevant for making clinical decisions and planning effective interventions, particularly in children with CP [3].

Traditionally, HN has been evaluated perceptually, with one or more expert Speech & Language Therapists (SLTs) rating the degree of nasality of the patient (e.g. on a 0–3 scale; [4]). Perceptual evaluation is still considered the gold standard for HN assessment [5]. However, perceptual evaluation is a demanding task that requires extensive training. This has motivated researchers to develop objective HN evaluation tools to support subjective evaluation. One important tool is Nasometry [6], which was developed half a century ago, but continues to be a well-considered method among SLTs.

Nasometry requires that speech is recorded using a split channel pair of microphones separated by a plate that records oral and nasal signals separately. Based on these dual-channel recordings, it is possible to compute Nasalance, which is the ratio of nose energy to mouth and nose energy in a small 300 Hz band centered around 600 Hz [5, 7]. In the remainder of this study, we refer to this measure as eNasalance. From a clinical perspective, Nasometry has two important characteristics that may partly explain its relative success: 1) it provides an intuitive percentage score that is easily interpretable by a clinical expert; 2) it can be used in any language or dialect, and for any utterance of interest for SLTs.

However, from today’s perspective, eNasalance seems clearly limited, both from perceptual and technical perspectives. From a speech perception perspective, the acoustic information used to compute eNasalance is only a small portion of the information that the human listener has access to. For instance, vowel nasalization may impact the spectrum up to 3 kHz or more [8], a value much larger than the small band used to compute eNasalance [6]. Furthermore, while Nasometry is based only on the summed energy of the speech signal, humans may use varied spectral information [8]. This may partly explain the broad variance in studies that have computed the Pearson correlation between the eNasalance and perceptual nasality (with r values ranging between 0.88 and 0.42) see [9–11]. From a technical perspective, research in the last half-century has proposed multiple speech features that have been shown to be highly effective in processing speech. One good example is Mel Frequency Cepstrum Coefficients (MFCCs), which model speech perception in humans and are commonly used in Automatic Speech Recognition tools. Furthermore, there are currently multiple machine learning (ML) algorithms that may serve to classify complex feature sets, such as MFCCs (see [12–14]), or high-speed nasopharyngoscopy [15]. Thus, it seems convenient to attempt to redefine nasalance by considering the technical and scientific advances in the last 50 years. Given that many studies in the last two decades have used advanced speech processing techniques to evaluate nasality using monophonic signals (i.e., a single speech signal), we will briefly review these models before making our own proposal.

Previous ML proposals

The development of ML models requires a training process, during which the model is fed with the same type of information that is expected to evaluate. In the case of HN ML models, many have been trained with healthy and hypernasal utterances; in most cases, the utterances were sustained vowels (for example, [16, 17]) whereas a few studies have used monosyllabic words [18], target words [19], or a few sentences [20, 21]. This means that these models can only be used to detect/quantify nasality in these specific utterances (and hence in one language), and they lack the flexibility of Nasometry.

Other studies have adopted flexible approaches. Carignan et al. [14] uses MFCCs in combination with other acoustic features to train an ML algorithm to create nasalance-like signals and compares the results with Nasometry measures. However, they do not analyze the interest of this approach for speech assessment, which is the main aim of this study. Siriwardena et al. [15] uses the full acoustic waveform to train an ML algorithm to model nasalance and compare the results with nasopharyngoscopy. However, in speech therapy, both Nasometry and nasopharyngoscopy are instrumental methods, whose validity needs to be compared with the ground truth (i.e. perceptual evaluation; [1, 22]). In contrast, Mathad et al. [23] trained their model, a Deep Neural Network (DNN), exclusively with speech samples from healthy speakers, without any clinical data, and aim to measure hypernasality in children with HN pathology. The speech samples were part of a large oral corpus that had already been phonetically transcribed. This allowed the authors to classify speech sounds into four major groups: nasal consonants, nasal vowels, oral consonants, and oral vowels. Note that nasal vowels do not exist as separate phonemes in English (or in many other languages). However, because vowels in contact with nasal consonants tend to be nasalized (universally, although the degree varies according to language or dialect), they used these vowel fragments to create a large set of nasal vowels. After the speech data were classified into these four classes, the speech signal was segmented into 25 ms windows from which the 13 MFCCs and the first and second derivatives were computed. The resulting 39 coefficients served to train the DNN. The trained model was used to evaluate speech samples from healthy children and children with hypernasal speech, providing a nasality score per child. Finally, the Pearson correlation was computed between the DNN scores, and the perceptual scores produced by human experts. The authors found that the correlation was .80 with the Americleft database, a value on par with trained clinicians on this dataset.

From the perspective of the present study, Mathad et al.’s proposal has the advantage of being utterance independent. However, some aspects of this process require further consideration. One is that their model used a very large, annotated speech corpora, and that such corpora are available only for a small number of languages. In addition, the authors did not present data on alternative objective evaluation methods, such as Nasometry. Thus, it remains to clarify whether or not their model is more effective than Nasometry. Finally, from a technical perspective, this model favors spectral analysis over temporal analysis. This approach is compatible with the common assumption that nasality is a spectral phenomenon, which is certainly true in the case of sustained speech sounds. However, in the case of running speech, we may expect nasality characteristics (e.g., nasal formants) to exhibit important temporal variation. Note also, that as standard speech evaluation protocols use different types of utterances (i.e. from sustained sounds to running speech [4]), it is possible that the type of acoustic information (e.g. spectral, temporal, or spectral-temporal) might depend on the target utterance. Thus, it seems necessary to explore whether or not the accuracy of HN ML models is influenced by the type of utterance. One type of ML algorithm that seems ideal for this task are Convolutional Neural Networks (CNNs), as it is possible use one specific architecture for different types of acoustic cues (i.e. by selecting the appropriate kernel shapes). Also, CNN have been shown to successfully detect nasality, though only in mono-syllabic words [24]. These considerations led us to use CNNs in the present study.

Finally, one issue that has received little attention is whether or not HN MLs model trained with data from one dialect can be used with other dialects. This is relevant specifically for dialects that may vary in the degree of nasality. For instance, in the case of Spanish language, it has been observed that speakers from Central America show a relatively strong nasalization tendency, as compared with speakers from other areas (e.g. Spain, Argentina, Chile, etc. [25]). This raises the possibility speakers’ subjective evaluation of HN varies between dialects, and that a ML model trained for a non-nasal dialect (e.g. Spain) might show reduced accuracy when tested with data from a nasal dialect (and vice versa). More data is needed to clarify this issue.

The main questions to be investigated in the present study are: 1) is nasalance, computed with CNNs and MFCCs (i.e. mfccNasalance) closer to human perception than the traditional eNasalance? 2) is it possible to use one model for all utterance types or, alternatively, utterance-type specific models are needed? and 3) does dialectal variation influence the accuracy of mfccNasalance? (i.e. is it possible to use one single model for multiple dialects or, alternatively, dialect-specific models should be trained?

Our approach

As noted above, in this study, we adopted Convolutional Neural Network (CNN) as the ML model. CNNs were originally used for image classification [26]. However, when speech signals are converted into images (e.g., spectrograms), CNNs can be used to classify these images.

One important advantage of CNNs is that the information process is divided into two phases. In one phase, a series of convolutions detect speech features (which might be viewed as equivalent to acoustic cues in speech signals). In the second phase, the proper model is trained to classify the features identified in the convolutions. The features depend on the shape of the kernels used to convolve the images (see Section Kernels and phonetic information). Therefore, the type of kernel should be adapted to the type of information to be classified. In their model [24] concluded that the kernel producing the most optimal results was a spectral one. As noted above, this result is compatible with the common assumption that nasality is a spectral phenomenon. However, assuming that in running speech, spectral cues become dynamic [27], it is possible that temporal kernels might be more appropriate; furthermore, given that HN evaluation protocols include utterances with varying dynamicity, it is possible that the CNN results may vary depending on the utterance-type/model combination. For instance, a CNN model with spectral kernels might be appropriate for static utterances (e.g., sustained vowels and possibly short words), and one with temporal kernels might be appropriate for dynamic utterances (e.g., sentences and diadochokinetic syllables). Note that this distinction is compatible with evidence that humans have separate spectral and temporal processing skills [28].

In order to clarify the impact of dialectal variation, we explored two different train-test conditions: same-dialect and different-dialect. In the same-dialect condition, one dialect was used in both training and test databases. In the different-dialect condition, the CNN model was trained with one dialect and evaluated using a different dialect. To this end, as a part of this study, we collected three training databases respectively from Spain, Chile, and Costa Rica.

Overview of the mfccNasalance model

Fig 1 shows an overview of the mfccNasalance model. Below, we describe the steps of the model.

Audio input and preprocessing

The speech signal was recorded using a Nasometer (icSpeech, Rose Medical Solutions Ltd., Canterbury, UK), which produces two-channel audio signals. These signals are divided into 250 ms windows with 150 ms overlap (i.e., every window starts 100 ms after the previous one). For each window, 39 MFCCs were computed using the Librosa package in Python, with a 25 ms frame length a 10 ms overlap. This results in a three-dimension matrix: 2 (channels) × 39 MFCCs × 26 timeframes. These matrices (or two-color pictures) were input into the CNN model.

Basic architecture of the neural networks

The network model is illustrated in Fig 2. The network parameters were initialized using a standard Glorot uniform initializer. The batch size and training epochs were set to 64 and 100, respectively. The number of filters used is 16. As part of this study, we tested multiple kernels of size i × j (1 ≤ i ≤ 8, 1 ≤ j ≤ 8), as explained below. Independent of the kernel shape, the stride was 1 and padding was set to same. A linear activation function was applied to each convolutional layer. Subsequently, batch normalization and a leaky Rectified Linear Unit were applied. In the pooling layers, a 2 × 2 kernel with stride 1 and the same padding was used; the method employed was max pooling. A 128-units fully connected layer with a linear activation was used. The output layer contains a softmax layer and a classification layer.

To train each model, 80% of the speakers were included in the training dataset and the remaining 20% were used to validate the model. A cross-validation process with 5 k-fold was performed, and the average of the correlation results with test data was used. The Adam optimizer was used, with a learning rate of 1e-5. The training used a single Nvidia RTX A100 graphics card. The authors thankfully acknowledge the computer resources, technical expertise, and assistance provided by the SCBI (Supercomputing and Bioinformatics) Center of the University of Malaga. Script files can be found in public repository: https://github.com/Caliope-SpeechProcessingLab/Nasalance-MFCC-CNN.

Kernels and phonetic information

As the MFCCs input images used in this study represent time in the horizontal axis and MFCCs in the vertical axis, kernel shapes can be grouped as follows (see Fig 3):

Spectral (cepstral): (2 × 1), (3 × 1), (4 × 1), etc.
Temporal: (1 × 2), (1 × 3), (1 × 4), etc.
Spectral-temporal: (2 × 2), (3 × 3), (4 × 4), (2 × 4), (4 × 2), etc.

Note that in the default setup, the same kernel was used for both the convolution layers. However, we decided to explore one further combination using a 1 × 1 kernel in the first layer, combined with any kernel shape in the second layer. This approach aims to reduce the dimensionality of the model [29, 30]. We assumed that it might serve to compare the oral and nasal signals for each MFFC time point (i.e., similar to energy computation in traditional eNasalance). In total, we computed 126 kernel combinations; 63 of these combinations used the same kernel in both layers, and 63 more were obtained by using 1 × 1 in the first layer and any kernel size in the second layer. Of these kernels, 14 were spectral, 14 were temporal, and 98 were spectral-temporal.

Training

The training data consisted of the MFCCs input images described above, together with the nasality value (which could be either nasal or oral) of the corresponding 250 ms window. Fig 4 shows the steps performed to determine the nasality value. First, the speech samples of the speaker were annotated using Praat v6.1.53 [31] and a phonetic transcription was obtained using the Montreal Forced Alignment v1.0.0 tool [32]. The phonetic codes were then reclassified as oral or nasal. At this stage two groups of sounds were classified as nasal: 1) the three nasal consonants (/m/, /n/ and /ɲ/); and 2) a section of the vowels that are in contact with nasal consonant (e.g. the end part of “a” in / an.tes /). The duration of this section varies with vowel duration. In this study, we assumed that the entire vowel was nasalized if it was shorter than 60 ms, 50% if it was between 60 and 90 ms, and 30% if it was longer than 90 ms. These limits are based on experiments conducted in the early design phases of this study. In the third step, the speech signal was segmented using a moving window of 250 ms length and 100 ms overlap. Next, the accumulated duration of the frames annotated as nasal was computed and divided by the total duration (i.e., 250). If the resulting value is higher than .30 (i.e. at least 75 ms), the window is classified as nasal; otherwise, it is classified as oral. Note that this percentage was adopted after conducting multiple experiments, suggesting that this value was optimal.

Testing and hypernasality computation

Once the model is trained, it can be used to test new MFCC input images (from 250 ms speech samples). Testing produces the posterior probability of being nasal for each input image, with values between 0 and 1. The mfccNasalance for a speaker is computed by averaging the posteriors probabilities of the speech fragments of that speaker. Once the full test database has been tested (i.e. mfccNasalance scores have been obtained for all speakers), the speaker scores are transformed into a 0–3 scale to provide clinical meaning, like the perceptual scale. To transform the probabilities into a 4-level scale, we had to determine the boundaries between the categories. Since we already knew how many children were perceptually classified as oral or nasal, and specifically as oral, mild, moderate or severely nasal, we assumed that the same distribution would apply to the nasalance measurements and make a scale transformation maintaining the same group distribution. One limitation of this approach is that bias may have been artificially avoided. However, as we analyzed only the accuracy and the same approach was used for eNasalance and mfccNasalance, we assumed that it would be valid for this study.

Methods

Speech samples

The present study analyzed speech samples from a large number of healthy and hypernasal speakers. All participants, or parents/tutors in the case of minors, signed an informed consent form. The recruitment period for this study began on July 1^st, 2023, and concluded on October 15^th, 2023.The study was conducted in accordance with the guidelines of the Declaration of Helsinki and was approved by the Ethics Committee of the University of Málaga (protocol code 67-2023-H, June 13, 2023).

Three Spanish language speech databases were used in this study: European Spanish, Costa Rican Spanish, and Chilean Spanish. All speech samples were recorded using the Nasometer device described in Section” Audio input and preprocessing”, and consisted of recordings of young, healthy women while reading four texts. Three of these four texts include a balanced representation of the inventory of Spanish language phonemes [33–35]. The fourth text was written as a part of this study and included many instances of the three nasal consonants in various contexts. A total of 50 speakers were recorded in Spain, 42 in Costa Rica, and 32 in Chile. All speakers were young adult females (age range: 18–30). The motivation to use this age range was to minimize the differences in fundamental frequency with that of the target children.

The test database included speech samples from 38 children with hypernasal speech and from 11 healthy children. The patients were recruited from Málaga and Barcelona (Spain). The children were regularly followed up by interdisciplinary teams in a clinical setting. Children from Málaga were recruited from the Materno Infantil Hospital. Children from Barcelona were recruited with the help of the local CP Association (FICAT). There were 20 males (aged 5–13 years) and 18 females (age range 4–21 years). The main criterion for inclusion was the existence of a history of HN speech. For male participants, a second criterion was having a fundamental frequency of 180 Hz or higher. Some of the patients were recorded on two occasions (N = 5) or three occasions (N = 2), for which the total number of speech samples from the HN group was 47.

The control group consisted of 11 participants with no history of speech disorders. Seven of them were male and four were female; their ages ranged from 5 to 12 years old. Individuals with upper respiratory tract infection, mixed nasal resonance, dysphonia, hoarseness of voice, or hyponasality were excluded from the study. All participants spoke Spanish as their native language.

The test recordings were obtained using the Nasometer described above. Each participant produced 45 utterances as part of a repetition task that is routinely used in our lab to evaluate children with resonance disorders. Only a subset of these 45 utterances was used for the present study, as shown in Table 1. Note that syllable repetition includes as many repetitions of the syllable as the user is able to perform during the recording time.

Table 1. Utterances in test data.

Category	Utterances
Diadochokinetic syllable repetition (+ dynamic)	papapa…, pipipi…, tatata…, tititi…, kakaka…, kikiki…
Words (-dynamic)	boca, pie, llave, dedo, dedo, gafas, silla, sol, casa, pez
Sentences (+dynamic)	A David le duele el dedo
	Al gato de Agatha le gusta el yogur
	Uy, hay algo ahí
	Si me llevo la llave
	Susi sale sola
	Fali fue a la feria
	Los zapatos de Cecilia
	La jirafa de Jesús
	Toda tu taza de té
	Papá puede pelar a Pili
	Quique coge el papel de calco

Open in a new tab

Dialect conditions

Fig 5 shows the combinations of the training and test dialects used in this study: 1) same-dialect and 2) different-dialect.

Perceptual ratings

A three-point scale of nasality was used to rate each utterance as follows: 0 (normal), 1 (at least one instance of vowel nasalization), and 2 (at least one instance of consonant nasalization). The scores were then summed and divided by the maximum score (i.e., 2 × the number of utterances). This resulted in scores ranging between 0 (maximally oral) and 1 (maximally nasal). Perceptual ratings were calculated by two SLTs who had, respectively, two and four years of experience on patients with resonance disorders. Whenever the two SLTs disagreed, a third experienced SLT determined the nasality score.

Finally, the percentage ratings were recorded using a 4-level scale. Oral: nasality ≤ 0.05; Mild: 0.05 < nasality ≤ 0.25; Moderate: 0.25 < nasality ≤ 0.50; Severe: nasality > 0.50. Note that this rating scale, based on the expert’s own clinical experience, does not differentiate between closed and open vowel nasalization, which is a common distinction in many protocols [4, 36]. However, as close vowels were included in only approximately 50% of the utterances, it was assumed that the results would reflect the close/open distinction.

eNasalance ratings

Nasalance was computed using a script developed by our team using Praat software [26]. The script produced one nasalance measure per utterance following these steps: 1) the nose and mouth signals were filtered, and only a 300 Hz wide band centered at 600 Hz was retained; 2) nasalance was computed as the ratio of nasal acoustic RMS energy to the sum of oral and nasal RMS energy. For each participant, the mean nasalance value was computed by averaging the nasalance values for each utterance. The percent eNasalance scores were recoded into a 0–3 scale following the same procedure used for the mfccNasalance scores.

Statistical analyses

Three measures were obtained for each test speaker: perceptual nasality, eNasalance and mfccNasalance. Furthermore, a new mfccNasalance score was obtained for each trained model and different test utterances. To compare eNasalance with the different mfccNasalance measures, we computed the Spearman correlation between these scores and perceptual ratings. Note that Pearson correlation was not used because the data did not comply with the normally distributed required condition of this statistical test. This is known by applying the Shapiro-Wilk test to the scores from syllables, words, sentences, and all together, with a p-value less than 0.05 in each case. The results for different sets of simulations (e.g., same-dialect vs different dialect) were compared using student t-test. In order to analyze the optimal CNN setup Two-way ANOVAs were run using the kernel shape (spectral, temporal, spectral-temporal) and the presence of the point-wise 1 × 1 kernel.

Results

The results are organized to answer the three research questions noted above. First, we present the result for the mfccNasalance approach in the same-dialect condition and compare those results with eNasalance. Second, results are presented for multiple kernel shape × utterance type combinations (in the same-dialect condition). Finally, we analyze the results for the different-dialect condition.

mfccNasalance vs. eNasalance: Same-dialect condition

In order to compare the two measures, we trained and tested five CNN models (i.e. using a cross-correlation) per kernel combination (N = 126). Thus, the total number of models was 630. In all these cases, the train and test dataset used was European Spanish (i.e. same-dialect condition). For each of the 630 models, we computed mfccNasalance as well as its correlation with perceptual scores separately for the diadochokinetic syllables, words, and sentences. Finally, mean for the five scores per of each kernel combination (N = 126) × utterance type (N = 3) was obtained, which results in 378 measures (i.e. 126 per utterance type).

Fig 6 (left) shows one boxplot for each utterance type. For syllables and sentences, the 126 correlational measures for mfccNasalance were higher than eNasalance. This means that, independently of the kernel combination, mfccNasalance is closer to human perception than eNasalance. In the case of words, the advantage of mfccNasalance is smaller than in the case of syllables, but the scores are higher than eNasalance in most cases (i.e. all except the lower whisker). For sentences, mfccNasalance scores are better than in the case of words, though not as good as for syllables.

Kernel combinations and utterance types in the same-dialect condition

Next, we analyzed to what extent the correlation with perceptual scores varied as a function of the kernel setup in the CNN models. Two factors were analyzed: the presence (or not) of a 1 × 1 kernel in the first layer (k11), and the kernel shape used in the second layer (and also in the first one if 1 × 1 kernel was not used). Two-way ANOVA were computed separately for syllables, words and sentences. The results showed that using a 1 × 1 kernel resulted in increased accuracy for syllables (p < .000) and sentences (p < .000), but nor for words. In contrast, the kernel shape had a significant impact for words (p < .000), but not for syllables or sentences. Post-hoc pair-wise comparisons for the word-level data showed that the difference was significant among all groups of CNN models (temporal > spectral > spectral-temporal). Fig 6 (right) shows the box-plots obtained for the optimal CNN models. The results indicate that selecting the optimal CNN setup does not have a major impact on the mean correlation; however, it reduces clearly the variability. Table 2 summarizes the results obtained in the two-way ANOVA.

Table 2. Two-way ANOVA for Spain-trained models and optimal configuration per utterance type.

Spain
Category	k11 in first layer	Kernel shape	Interaction k11 and kernel	Optimal configuration
Syllable	p< .000 (post-hoc: k11)	ns	p < .01	k11 = True
Words	ns	p< .000 (Temporal > Spectral > Spectral-Temporal)	ns	Kernels = Temporal
Sentences	p< .000 (post-hoc: k11)	ns	p < .01	k11 = True

Open in a new tab

ns: not significant. k11: kernel 1 × 1 in the first layer of the CNN model.

mfccNasalance vs. eNasalance: Different-dialect condition

Next, we run the same analyses as above for the different-dialect condition (i.e. Chile and Costa Rica). First, we examined the scores for the full set of models N = 126 per dialect. The results of the Costa Rica-trained models are shown in Fig 7 (top left). Note that, compared with Spain-trained models, the results of the Costa Rica-trained models are relatively poor. For instance, for approximately half the models mfccNasalance for words was lower than eNasalance for the same words. As for Chile (Fig 7 down left), the results seem to be comparable to those of Spain. In order to confirm this apparent contrast, we computed paired T-test between, on the one hand, Spain-trained models and, on the other hand, Chile or Costa Rica-trained models. For the pair Spain-Costa Rica, the difference was significant in all cases (p < .000) For the pair Spain-Chile, the difference was significant for syllables (p < .000) but not for words or sentences.

Fig 7 — Top is for Costa Rica trained models, down for Chile. Left figures shows the results for all the CNN configurations. The right figures show the results for the CNNs using the optimal configuration (Costa Rica: Syllable k11 = True, Words Kernels = Temporal, Sentences k11 = True. Chile: Syllable Kernels = Spectral, Words k11 = False, Sentences Kernels = Spectral).

Next, we computed two-way ANOVAS separately for each dialect × utterance type combination. The results are summarized in Table 3. In the case of Costa Rica, using a 1 × 1 kernel had a significant and positive impact both for syllables and sentences (p < .000 in both cases), but nor for words (n.s.) In contrast, the kernel shape had a significant effect for words (p < .000), but not for syllables or sentences. Post-hoc analyses confirmed that there were significant differences among all groups (Spectral > Spectral-Temporal > Temporal). In the case of Chile, the presence of a 1 ×1 kernel produced a significant and negative effect for words (p < .000) but no effect at all for syllables or sentences. In contrast there was a main effect of the kernel shape for syllable and sentences (p < .000 in both cases) but not for words. Post-hoc analysis for the three kernel shape groups showed that differences were significant in all cases (Spectral > Spectral-Temporal > Temporal).

Table 3. Two-way ANOVA for Costa Rica (top) and Chile (down) trained models.

Costa Rica
Category	k11 in first layer	Kernel shape	Interaction k11 and kernel	Optimal configuration
Syllable	p< .000 (post-hoc: k11)	ns	ns	k11 = True
Words	ns	p< .000 (Spectral > Spectral-Temporal > Temporal)	ns	Kernels = Temporal
Sentences	p< .000 (post-hoc: k11)	ns	ns	k11 = True
Chile
Category	k11 in first layer	Kernel shape	Interaction k11 and kernel	Optimal configuration
Syllable	ns	p< .000 (Spectral > Spectral-Temporal > Temporal)	p < .01	Kernels = Spectral
Words	p< .000 (post-hoc: no k11)	ns	ns	k11 = False
Sentences	ns	p< .001 (Spectral > Spectral-Temporal > Temporal)	ns	Kernels = Spectral

Open in a new tab

ns: not significant. k11: kernel 1 × 1 in the first layer of the CNN model.

Finally, we compared the results of the Costa Rica (Fig 7 top right) and Chile (Fig 7 bottom right) trained optimal configuration models with the corresponding optimal Spain-trained models. As the number of models was different in dialect it was not possible to use the paired samples method; instead, we used the independent samples t-student test. The mfccNasalance scores obtained with European Spanish data (same-dialect) were higher than the corresponding mfccNasalance obtained in the different-dialect condition (both for Chile and Costa Rica). For the pair Spain-Costa Rica, this difference was significant for words and sentences (p < .000). For the pair Spain-Chile, the difference was significant only for syllables (p < .000).

Discussion

The main results of this study are as follows: first, we found that, globally, mfccNasalance is closer to human perception than traditional eNasalance. Second, our results indicate that the different utterance types may require different CNN configurations. Third, the accuracy of mfccNasalance varies as a function of the dialect pair used, with results being somewhat better in the same-dialect condition than in the different-dialect condition. We discuss these results below.

mfccNasalance vs eNasalance

The main result of this study was that the correlation between mfccNasalance and perceptual scores was higher than the correlation between eNasalance and the same perceptual scores. In other words, mfccNasalance seems closer to human perception than eNasalance. This result was especially clear in the same dialect condition: independently of the CNN configuration mfccNasalance was higher than eNasalance (see Fig 6 left). These results are compatible with the fact that nasalance is a poor approximation to perceptual nasality, as studies comparing the nasalance scores and perceptual scores have obtained correlations ranging from non-significant to strong [37]. Nasalance computes a short spectral bandwidth [6], while Nasality processes the full spectrum [8]; furthermore, while Nasometry is based only on the summed energy in the predefined bandwidth, mfccNasalance, like humans, has a filterbank that can split the spectrum into a set of energy measures. Thus, it is not surprising that mfccNasalance is closer to human perception than eNasalance and, hence, that the former is a better approach to evaluate nasality than the latter.

Our proposed method has some advantages compared with that in [22]. In the first place, while Mathad et al. used a large (100 h long) database, we used relatively small datasets of speech samples (between 2h and 4h per dialect). This result is relevant because, obviously, creating one annotated speech corpora of 100 h is much more costly than creating 2-hours speech corpora. Furthermore, note that the approach used by Mathad et al. required very precise phonetic annotations (which means that it is necessary to make manual revision of the oral databases); in contrast, as we only need to know that a nasal sound was present (and not the exact location, due to the use of 250 ms speech samples), manual revision of training data can be bypassed. Altogether, this means that it might be notably easier to adapt mfccNasalance approach to any language or dialect than to adapt Mathad et al.’s model.

Optimal kernel configuration and utterance types

Our second research question was if it was possible to use one model for all utterance types or, alternatively, utterance-type specific models are needed. More specifically, we speculated that as dynamic utterances (i.e. diadochokinetic syllables and sentences) are characterized by temporal variability, hypernasality would be more precisely captured by temporal kernels, while spectral kernels might be more useful for -dynamic utterances. The results in the same dialect condition revealed a contrast between the optimal models for +dynamic and–dynamic utterances, though it was not in the way we had anticipated (i.e. dynamic utterances being optimally classified with temporal kernels). Rather, the key factor was the presence or not of the 1 x 1 kernel in the first layer: in the case of dynamic utterances, 1 x 1 kernels in the first layer increased the accuracy; in the case of -dynamic utterances (i.e. words), it had no impact. As for the kernel shape in the second layer, it was relevant only in the case of words.

The 1 × 1 convolution kernel, also known as pointwise convolution, is generally used to increase the depth of feature maps without altering the spatial dimensions. Pointwise convolution enables the network to learn combinations of input feature maps, thus creating new feature maps that represent a blend of both the mouth and nose input channels. As shown in previous studies [29, 30] this effect is particularly useful for integrating and abstracting features across the channel dimension, leading to more robust feature representations.

Thus, the one possible interpretation for the relevance of pointwise convolution with +dynamic utterances, but not with -dynamic ones is that in the former case the system is confronted with highly heterogeneous signals; being so heterogeneous, feature abstraction and integration may help to identify the relevant acoustic data and, hence, increase the accuracy of the system. As for words, given that variability in the signal is reduced, selecting a 1 x 1 kernel does not produce any advantage; however, the fact that temporal kernels are the most relevant suggest that nasality information is dynamically produced in short bi-syllabic words in Spanish. Note that this result (i.e. the good results with temporal kernels) do not agree with those obtained in the only previous study that used CNN to compute nasality [18]. The authors found that spectral kernels produced the best results. The difference between the previous study and ours might be explained as follows. Nasality information is relatively weak as compared with oral information, for which the latter may partially masked the former in monophonic signals [38]. Naturally, masking cannot be complete, but it seems reasonable to assume that temporal changes (e.g. formant transitions) are more easily masked than more stable sections (e.g. center of nasal consonants). Second [18], analyzed monosyllabic words in Chinese, and these words produced in isolation may be even less dynamic than the two-syllable words used in the present study, for which they might be better classified using temporal or spectral-temporal kernels. To summarize, while in [16], the most relevant nasality information was spectral, in our case temporal information may be much more relevant.

Dialect-effect

As for the results for utterance types and dialects, one caution must be exercised. Given the limited number of speakers per dialect, it was not possible to fully confirm whether these results reflected the properties of the set of participants for each database or the general characteristics of the dialect. However, given what we know about nasality in these dialects, a tentative explanation can be proposed for the complex pattern of the results for dialects and utterances. The most relevant result is that, independently or the kernel setup, the results in the same dialect-condition were significantly better than the results in the different-dialect conditions. Furthermore, the difference was clearer for the Spain-Costa Rica pair than for the Spain-Chile pair. This result is compatible with what we know about nasality in the three dialects. As noted above, Costa Rican Spanish is known to be highly nasal, while European and Chilean Spanish are comparatively similarly oral. Thus, it seems that the phonetic differences between these two dialects may explain the results. However, the fact that the Chile-Spain pair produced poorer results than the same dialect condition indicates that other dialectal characteristics may be relevant apart from nasality.

Another interesting result is that the optimal kernel shapes are not identical for the different-dialect condition as for the same dialect condition. Furthermore, there were differences between the two different-dialect condition. In the case of Costa Rica, the pointwise convolution has a significant positive effect for +dynamic utterances exclusively (i.e. as in the case of Spain); then, for words, spectral kernels produce the best results. In the case of Chile, the pointwise convolution produces no impact for +dynamic utterances, and a negative impact for words.

Two aspects of these results require further attention: the pointwise convolution and the preference for specific kernel shapes. As to the pointwise convolution, the results in Costa Rica are identical to those in Spain. One possible explanation for this result is that, even the two dialects might differ in nasality expression spectrally (i.e. with nasalization more pronounced in Costa Rica) but not temporally. As for Chile, the lack of significance of the pointwise convolution suggests that we find the different pattern: the two dialects (European and Chilean) might be similar in spectral terms, but not so much in temporal terms. Inasmuch as the Chilean samples are less dynamic than the European ones, the effect of the convolution might be negative in the case of -dynamic utterances, and neutral in the case of +dynamic utterances. Thus, the different-dialect data further confirms the relevance of the pointwise convolution. As to the kernel shapes, it is intriguing that temporal kernels produced the best results in the same dialect condition, and spectral kernel shapes in the different-dialect condition. Unfortunately, the fact that we found three different kernel configurations for three different dialects makes it impossible to make any generalization. To summarize, dialectal data further confirms that a key aspect of the CNN setup is the pointwise convolution: it tends to have a significant and positive impact with highly heterogenous data, and it may also have a negative impact with less heterogeneous data (i.e. short -dynamic utterances).

Future lines of research

The present proposal shows that mfccNasalance preserves the advantages of Nasometry while simultaneously incorporating more recent speech processing techniques. It also shows that in order to increase accuracy, it is convenient to use utterance and dialect specific kernel setup, and that the use of a pointwise convolution has the strongest influence on the final accuracy. These results imply that in order to develop a nasality evaluation tool that is valid for the varied utterance types used in clinical protocols (e.g. words, sentences…) and also for speakers from varied dialectal backgrounds, it might be necessary build more than one nasality model. Furthermore, it implies that a preprocessing stage will be needed to decide, for each target speech fragment, which is the most appropriate model. Note that this is compatible with information about speech perception by humans, as we can dynamically favor spectral or temporal speech processing [39]. Future studies should analyze to what extent it is possible to predict, for a given speech fragment, which model is the most optimal one. For this task, it seems that one alternative would be to look for inspiration from recent neurocognitive models of speech perception (e.g., [40]).

Acknowledgments

The authors would like to express their gratitude to FICAT (Catalonia) and ASAFILAP (Andalusia).

Data Availability

All autor-generated code script files can be found in public repository: https://doi.org/10.6084/m9.figshare.26762818.v1 https://doi.org/10.6084/m9.figshare.26762842.v1 https://doi.org/10.6084/m9.figshare.26762833.v1.

Funding Statement

This research was funded by the Spanish MINISTERIO DE CIENCIA, INNOVACIÓN YUNIVERSIDADES, grant number PID2021-126366OB-I00. This funding received by Enrique Nava and Ignacio Moreno-Torres. This research was also funded by the Spanish JUNTA DE ANDALUCIA, grant number UMA18FEDERJA021. This funding received by Enrique Nava and Ignacio Moreno-Torres.

References

1.Kummer A., "Cleft Palate and Craniofacial Anomalies: Effects on Speech and Resonance, ed 3. Clifton Park," ed: Delmar Publishing, 2013. [Google Scholar]
2.Kuehn D. P. and Moller K. T., "Speech and language issues in the cleft palate population: the state of the art," The Cleft palate-craniofacial journal, vol. 37, no. 4, pp. 1–35, 2000.10670881 [Google Scholar]
3.Howard S. and Lohmander A., Cleft palate speech: assessment and intervention. John Wiley & Sons, 2011. [Google Scholar]
4.John A., Sell D., Sweeney T., Harding-Bell A., and Williams A., "The cleft audit protocol for speech—augmented: A validated and reliable measure for auditing cleft speech," The Cleft palate-craniofacial journal, vol. 43, no. 3, pp. 272–288, 2006. doi: 10.1597/04-141.1 [DOI] [PubMed] [Google Scholar]
5.Bettens K., Wuyts F. L., and Van Lierde K. M., "Instrumental assessment of velopharyngeal function and resonance: A review," Journal of communication disorders, vol. 52, pp. 170–183, 2014. doi: 10.1016/j.jcomdis.2014.05.004 [DOI] [PubMed] [Google Scholar]
6.Fletcher S. G., Sooudi I., and Frost S. D., "Quantitative and graphic analysis of prosthetic treatment for “nasalance” in speech," The Journal of Prosthetic Dentistry, vol. 32, no. 3, pp. 284–291, 1974. doi: 10.1016/0022-3913(74)90032-8 [DOI] [PubMed] [Google Scholar]
7.Gildersleeve-Neumann C. E. and Dalston R. M., "Nasalance scores in noncleft individuals: why not zero?," The Cleft palate-craniofacial journal, vol. 38, no. 2, pp. 106–111, 2001. doi: 10.1597/1545-1569_2001_038_0106_nsiniw_2.0.co_2 [DOI] [PubMed] [Google Scholar]
8.Carignan C., "Using ultrasound and nasalance to separate oral and nasal contributions to formant frequencies of nasalized vowels," The Journal of the Acoustical Society of America, vol. 143, no. 5, pp. 2588–2601, 2018, doi: 10.1121/1.5034760 [DOI] [PubMed] [Google Scholar]
9.Keuning K. H., Wieneke G. H., Van Wijngaarden H. A., and Dejonckere P. H., "The correlation between nasalance and a differentiated perceptual rating of speech in Dutch patients with velopharyngeal insufficiency," The Cleft palate-craniofacial journal, vol. 39, no. 3, pp. 277–284, 2002. doi: 10.1597/1545-1569_2002_039_0277_tcbnaa_2.0.co_2 [DOI] [PubMed] [Google Scholar]
10.Brancamp T. U., Lewis K. E., and Watterson T., "The relationship between nasalance scores and nasality ratings obtained with equal appearing interval and direct magnitude estimation scaling methods," The Cleft Palate-Craniofacial Journal, vol. 47, no. 6, pp. 631–637, 2010. doi: 10.1597/09-106 [DOI] [PubMed] [Google Scholar]
11.Khwaileh F. A., Alfwaress F. S., Kummer A. W., and Alrawashdeh M. m, "Validity of test stimuli for nasalance measurement in speakers of Jordanian Arabic," Logopedics Phoniatrics Vocology, vol. 43, no. 3, pp. 93–100, 2018. doi: 10.1080/14015439.2017.1370724 [DOI] [PubMed] [Google Scholar]
12.McKechnie J., Ahmed B., Gutierrez-Osuna R., Monroe P., McCabe P., and Ballard K. J., "Automated speech analysis tools for children’s speech production: A systematic literature review," International journal of speech-language pathology, vol. 20, no. 6, pp. 583–598, 2018. doi: 10.1080/17549507.2018.1477991 [DOI] [PubMed] [Google Scholar]
13.Dhillon H. et al., "Current applications of artificial intelligence in cleft care: a scoping review," Frontiers in medicine, vol. 8, p. 676490, 2021. doi: 10.3389/fmed.2021.676490 [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Carignan C., "A practical method of estimating the time-varying degree of vowel nasalization from acoustic features," The Journal of the Acoustical Society of America, vol. 149, no. 2, pp. 911–922, 2021. doi: 10.1121/10.0002925 [DOI] [PubMed] [Google Scholar]
15.Siriwardena Y. M. et al., "Speaker-independent speech inversion for recovery of velopharyngeal port constriction degree," The Journal of the Acoustical Society of America, vol. 156, no. 2, pp. 1380–1390, 2024. [DOI] [PubMed] [Google Scholar]
16.He L., Zhang J., Liu Q., Yin H., Lech M., and Huang Y., "Automatic evaluation of hypernasality based on a cleft palate speech database," Journal of medical systems, vol. 39, pp. 1–7, 2015. [DOI] [PubMed] [Google Scholar]
17.Akafi E., Vali M., Moradi N., and Baghban K., "Assessment of hypernasality for children with cleft palate based on cepstrum analysis," Journal of medical signals and sensors, vol. 3, no. 4, p. 209, 2013. [PMC free article] [PubMed] [Google Scholar]
18.Wang X., Tang M., Yang S., Yin H., Huang H., and He L., "Automatic hypernasality detection in cleft palate speech using cnn," Circuits, Systems, and Signal Processing, vol. 38, pp. 3521–3547, 2019. [Google Scholar]
19.Fagniart S. et al., "Consonant and vowel production in children with cochlear implants: acoustic measures and multiple factor analysis," Frontiers in Audiology and Otology, vol. 2, p. 1425959, 2024. [Google Scholar]
20.Moreno-Torres I., Lozano A., Nava E., and Bermúdez-de-Alvear R., "Which Utterance Types Are Most Suitable to Detect Hypernasality Automatically?," Applied Sciences, vol. 11, no. 19, p. 8809, 2021. [Google Scholar]
21.Carignan C., Chen J., Harvey M., Stockigt C., Simpson J., and Strangways S., "An investigation of the dynamics of vowel nasalization in Arabana using machine learning of acoustic features," Laboratory Phonology, vol. 14, no. 1, pp. 1–31, 2023. [Google Scholar]
22.Kummer A. W., "Evaluation of Speech and Resonance for Children with Craniofacial Anomalies," Facial plastic surgery clinics of North America, vol. 24, no. 4, pp. 445–451, 2016. doi: 10.1016/j.fsc.2016.06.003 [DOI] [PubMed] [Google Scholar]
23.Mathad V. C., Scherer N., Chapman K., Liss J. M., and Berisha V., "A deep learning algorithm for objective assessment of hypernasality in children with cleft palate," IEEE Transactions on Biomedical Engineering, vol. 68, no. 10, pp. 2986–2996, 2021. doi: 10.1109/TBME.2021.3058424 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Wang X., Yang S., Tang M., Yin H., Huang H., and He L., "HypernasalityNet: Deep recurrent neural network for automatic hypernasality detection," International Journal of Medical Informatics, vol. 129, pp. 1–12, 2019. doi: 10.1016/j.ijmedinf.2019.05.023 [DOI] [PubMed] [Google Scholar]
25.Bongiovanni S., "Acoustic investigation of anticipatory vowel nasalization in a Caribbean and a non-Caribbean dialect of Spanish," Linguistics Vanguard, vol. 7, no. 1, p. 20200008, 2021. [Google Scholar]
26.Fukushima K., "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biological cybernetics, vol. 36, no. 4, pp. 193–202, 1980. doi: 10.1007/BF00344251 [DOI] [PubMed] [Google Scholar]
27.Elvin J., Williams D., and Escudero P., "Dynamic acoustic properties of monophthongs and diphthongs in Western Sydney Australian English," The Journal of the acoustical society of America, vol. 140, no. 1, pp. 576–581, 2016. doi: 10.1121/1.4952387 [DOI] [PubMed] [Google Scholar]
28.Zatorre R. J. and Belin P., "Spectral and temporal processing in human auditory cortex," Cerebral cortex, vol. 11, no. 10, pp. 946–953, 2001. doi: 10.1093/cercor/11.10.946 [DOI] [PubMed] [Google Scholar]
29.He K., Zhang X., Ren S., and Sun J., "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. [Google Scholar]
30.Szegedy C. et al., "Going deeper with convolutions," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9. [Google Scholar]
31.Boersma P., "Praat, a system for doing phonetics by computer," Glot. Int., vol. 5, no. 9, pp. 341–345, 2001. [Google Scholar]
32.McAuliffe M., Socolof M., Mihuc S., Wagner M., and Sonderegger M., "Montreal forced aligner: Trainable text-speech alignment using kaldi," in Interspeech, 2017, vol. 2017, pp. 498–502. [Google Scholar]
33.Bruyninckx M., Harmegnies B., Llisterri J., and Poch-Oiivé D., "Language-induced voice quality variability in bilinguals," Journal of Phonetics, vol. 22, no. 1, pp. 19–31, 1994. [Google Scholar]
34.Martínez-Celdrán E., Fernández-Planas A. M., and Carrera-Sabaté J., "Castilian spanish," Journal of the International Phonetic Association, vol. 33, no. 2, pp. 255–259, 2003. [Google Scholar]
35.Ortega-Garcia J., Gonzalez-Rodriguez J., and Marrero-Aguiar V., "AHUMADA: A large speech corpus in Spanish for speaker characterization and identification," Speech communication, vol. 31, no. 2–3, pp. 255–264, 2000. [Google Scholar]
36.Sweeney C., The perceptual and instrumental assessment of nasality and nasal airflow errors associated with velopharyngeal dysfunction. Trinity College Dublin, 2000. [Google Scholar]
37.Liu Y., Lee S. A. S., and Chen W., "The correlation between perceptual ratings and nasalance scores in resonance disorders: A systematic review," Journal of Speech, Language, and Hearing Research, vol. 65, no. 6, pp. 2215–2234, 2022. doi: 10.1044/2022_JSLHR-21-00588 [DOI] [PubMed] [Google Scholar]
38.Moreno-Torres I., Lozano A., Bermúdez R., Pino J., Méndez M. D. G., and Nava E., "Unmasking Nasality to Assess Hypernasality," Applied Sciences, vol. 13, no. 23, p. 12606, 2023. [Google Scholar]
39.Xu L. and Zheng Y., "Spectral and temporal cues for phoneme recognition in noise," The Journal of the Acoustical Society of America, vol. 122, no. 3, pp. 1758–1764, 2007. doi: 10.1121/1.2767000 [DOI] [PubMed] [Google Scholar]
40.Hickok G. and Poeppel D., "Neural basis of speech perception," Handbook of clinical neurology, vol. 129, pp. 149–160, 2015. doi: 10.1016/B978-0-444-62630-1.00008-1 [DOI] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0315452.r001

Decision Letter 0

Laura Morett

2 Jul 2024

PONE-D-24-18061Computing nasalance with MFCCs and Convolutional Neural NetworksPLOS ONE

Dear Dr. Lozano Durán,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

This submission is interesting and timely, but several aspects of the methods and results require clarification and/or modification. Provided that the authors are willing to address them, they will be sent back to the original reviewers for re-assessment to determine suitability of the manuscript for publication.

Please submit your revised manuscript by Aug 16 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Laura Morett

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, we expect all author-generated code to be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse."

3. Thank you for stating the following financial disclosure:

[This research was funded by the Spanish MINISTERIO DE CIENCIA, INNOVACIÓN Y UNIVERSIDADES, grant number PID2021-126366OB-I00.].

Please state what role the funders took in the study. If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.""

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

4. We note that you have indicated that there are restrictions to data sharing for this study. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

Before we proceed with your manuscript, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., a Research Ethics Committee or Institutional Review Board, etc.). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

https://journals.plos.org/plosone/s/recommended-repositories. You also have the option of uploading the data as Supporting Information files, but we would recommend depositing data directly to a data repository if possible.

We will update your Data Availability statement on your behalf to reflect the information you provide.

5. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please delete it from any other section.

6. Please upload a copy of Supplementary files to which you refer in your text on page 12. Please amend the file type to 'Supporting Information'. If the Supplementary file is no longer to be included as part of the submission please remove all reference to it within the text.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Questions:

"Based on these dual-channel recordings, it is possible to compute Nasalance, which is the ratio of nose energy to mouth and nose energy in a small 300 Hz band centered around 600 Hz. [5, 7]."

Q1) Why the center frequency reduced to 500Hz (in method) instead of 600Hz according to the reference [5, 7] as mentioned in the Introduction?

Q2) According to the method described by Mathad et al. (2021), MFCCs and their first and second deltas were utilized to train the DNN. In contrast, your method includes only MFCCs without the deltas, which requires further explanation. [if there is no justification provided, please implement R2]

For Revision:

R1) In comparison to the reference study by Mathad et al. 2021, which utilized a substantial 100-hour database, the training model in that study is robust enough to mitigate any subset effects. However, the current paper relies on only 2 hours of data without employing cross-validation. To enhance the reliability of the results, I recommend implementing cross-validation and conducting a straightforward comparison of accuracy (%) means to strengthen the statistical validity of the findings.

R2) The deltas of MFCCs are crucial, as demonstrated in numerous studies, for capturing the temporal dynamics that are often lost during time windowing. By including the deltas, the current spectral dimension would be expanded to a three-dimensional matrix: 2 (channels) × 39 MFCC features (13 MFCCs + 13 first deltas + 13 second deltas) × 26 timeframes. This addition would enhance the model's ability to recover temporal information.

Typos/Corrections:

T1) Ensure that all graphs include labels on the y-axis. Currently, they are missing.

T2) Address the numerous typos throughout the paper, with particular attention to consistent capitalization and formatting of "MFCCs."

Reviewer #2: The utilization of CNN on MFCC for classification and nasal and oral radiated energy based classification are well-established. So, the article has lack of novelty. The description of the optimized kernel is confusing. The author should represent the kernel with respect to individual MFCC and their derivatives. The word and phrase selection should be unified. For an example, the spectrogram and MFCC are used same purpose, which made the reader confused.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Md Mahbub Hasan

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2024 Dec 31;19(12):e0315452. doi: 10.1371/journal.pone.0315452.r002

Author response to Decision Letter 0

16 Aug 2024

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

1: Manuscript have been modified to meet PLOS ONE’s style requirements..

2: All autor-generated code script files can be found in public repository:

https://doi.org/10.6084/m9.figshare.26762818.v1

https://doi.org/10.6084/m9.figshare.26762842.v1

https://doi.org/10.6084/m9.figshare.26762833.v1

3. Thank you for stating the following financial disclosure: [This research was funded by the Spanish MINISTERIO DE CIENCIA, INNOVACIÓN Y UNIVERSIDADES, grant number PID2021-126366OB-I00.]. Please state what role the funders took in the study. If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."" If this statement is not correct you must amend it as needed. Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

3: Added “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.” in the financial disclosure.

Before we proceed with your manuscript, please address the following prompts:

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see https://journals.plos.org/plosone/s/recommended-repositories. You also have the option of uploading the data as Supporting Information files, but we would recommend depositing data directly to a data repository if possible.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4: Minimal anonymized data set can be found in public repository:

https://doi.org/10.6084/m9.figshare.26762818.v1

https://doi.org/10.6084/m9.figshare.26762842.v1

https://doi.org/10.6084/m9.figshare.26762833.v1

5. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please delete it from any other section.

5: Ethics statement only appear in Method section.

6. Please upload a copy of Supplementary files to which you refer in your text on page 12. Please amend the file type to 'Supporting Information'. If the Supplementary file is no longer to be included as part of the submission, please remove all reference to it within the text.

6: Supplementary files reference has been removed within the text.

Response to Reviewer 1 Comments

Comments made by the reviewer have been applied to improve the reliability of the results and the robustness of the methods applied. In particular, the use of 39MFCC, the cross-validation of the data, and an improvement in the depth of the statistical analysis have led to a substantial change in the results obtained.

Therefore, in this revised version of the article, extensive changes have been made in the Results and Discussion sections, modifying the images generated and the way of showing and justifying the results obtained.

Questions

Question 1: Why the center frequency reduced to 500Hz (in method) instead of 600Hz according to the reference [5, 7] as mentioned in the Introduction?

Response 1: Changed to: “600 Hz” in Methods.

Question 2: According to the method described by Mathad et al. (2021), MFCCs and their first and second deltas were utilized to train the DNN. In contrast, your method includes only MFCCs without the deltas, which requires further explanation. [if there is no justification provided, please implement R2]

Response 2: In the present version, training is run using 39 MFCCs coefficient with delta and second-delta in the method section, as suggested by both reviewers.

Revisions

Revision 1: In comparison to the reference study by Mathad et al. 2021, which utilized a substantial 100-hour database, the training model in that study is robust enough to mitigate any subset effects. However, the current paper relies on only 2 hours of data without employing cross-validation. To enhance the reliability of the results, I recommend implementing cross-validation and conducting a straightforward comparison of accuracy (%) means to strengthen the statistical validity of the findings.

Response 1: Cross-validation process has been implemented for train and validation speakers.

Revision 2: The deltas of MFCCs are crucial, as demonstrated in numerous studies, for capturing the temporal dynamics that are often lost during time windowing. By including the deltas, the current spectral dimension would be expanded to a three-dimensional matrix: 2 (channels) × 39 MFCC features (13 MFCCs + 13 first deltas + 13 second deltas) × 26 timeframes. This addition would enhance the model's ability to recover temporal information.

Response 2: Thanks for your suggestion. 39 MFCCs coefficient with delta and second-delta are computed to generate a 2-channel image 2 (channels) × 39 MFCC × 26 timeframes as input data of CNN.

Typos

Typos 1: Ensure that all graphs include labels on the y-axis. Currently, they are missing.

Response 1: Graphs has been modified to include y-axis labels.

Typos 2: Address the numerous typos throughout the paper, with particular attention to consistent capitalization and formatting of "MFCCs."

Response 1: Typos corrected to use consistent “MFCCs”

Response to Reviewer 2 Comments

Questions

Question 1: The utilization of CNN on MFCC for classification and nasal and oral radiated energy-based classification are well-established. So, the article has lack of novelty. The description of the optimized kernel is confusing. The author should represent the kernel with respect to individual MFCC and their derivatives. The word and phrase selection should be unified. For an example, the spectrogram and MFCC are used same purpose, which made the reader confused.

Response 1: The description of the optimized kernel has been modified in the Results section.

MFCCs features with delta and second delta has been computed and added as input of CNN.

Word and phrase selection remains unchanged because the results show different behaviour in terms of the optimal kernel configuration for each type of utterance, which we consider worthy of distinction and discussion.

Spectrogram and MFCC are no longer used for the same purpose. Input data generated with MFCCs coefficients are defined as ‘MFCCs input images’ in the text.

Attachment

Submitted filename: Response_to_reviewers.docx

pone.0315452.s001.docx^{(22.2KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0315452.r003

Decision Letter 1

Laura Morett

8 Oct 2024

PONE-D-24-18061R1Computing nasalance with MFCCs and Convolutional Neural NetworksPLOS ONE

Dear Dr. Lozano Durán,

Because neither of the original reviewers were able to re-review this manuscript, a third reviewer was recruited. This reviewer raises some important points that should be addressed, including coverage of relevant literature, data coding and analysis, and interpretation of results. Please revise the manuscript to address the points raised by R3 and it will be reassessed for suitability for publication in PLOS One.

Please submit your revised manuscript by Nov 22 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

We look forward to receiving your revised manuscript.

Kind regards,

Laura Morett

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #3: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #3: Yes

**********

6. Review Comments to the Author

Reviewer #3: This study attempt to create a nasalance-like metric directly from acoustics, using deep learning models. The study is quite nice in many respects, but there are issues that would need to be addressed before publication can be considered. These issues are listed below in the order (broadly) that they appear in the paper.

Abstract, Table 1, elsewhere: "The test data were recorded in Spain and included short words (-dynamic), sentences (+dynamic), and diakinetic syllables (+dynamic)"

-What do these +/- labels mean? Why is the /ka/ in "ka" considered "dynamic" but the /ka/ in "boka" isn't? These terms are never defined or justified.

-I am not familiar with the term "diakinetic". Do you mean "diadochokinetic"?

p.3: "To our knowledge, only one ML model has the flexibility of Nasometry [18]"

-The authors may want to be aware of similar works that have previous been carried out, for example Siriwardena et al. (2024) and Carignan (2021), the latter of which has been extended to low-resource languages (Carignan et al., 2023) and hyper-nasality in children (Fagniart et al., 2024), two goals which the current study also aims to meet.

Carignan, C. (2021). "A practical method of estimating the time-varying degree of vowel nasalization from acoustic features", J. Acoust. Soc. Am. 149 (2), 911–922. doi: https://doi.org/10.1121/10.0002925

Carignan, C., Chen, J., Harvey, M., Stockigt, C., Simpson, J. & Strangways, S. (2023). "An investigation of the dynamics of vowel nasalization in Arabana using machine learning of acoustic features”, Laboratory Phonology 14(1). doi: https://doi.org/10.16995/labphon.9152

Fagniart, S., Charlier, B., Delvaux, V., Huberlant, A., Harmegnies, B.G., Piccaluga, M., & Huet, K. (2024). "Consonant and vowel production in children with cochlear implants: acoustic measures and multiple factor analysis", Front. Audiol. Otol. 2:1425959. doi: 10.3389/fauot.2024.1425959

Siriwardena, Y.M., Boyce, S.E., Tiede, M.K., Oren, L., Fletcher, B., Stern, M., & Espy-Wilson, C.Y. (2024). "Speaker-independent speech inversion for recovery of velopharyngeal port constriction degree", J. Acoust. Soc. Am. 156 (2), 1380–1390. doi: https://doi.org/10.1121/10.0028124

-You say that European Spanish is "a non-nasal dialect" (p.3) and that "Costa Rican Spanish is known to be highly nasal, while European and Chilean Spanish are comparatively similarly oral" (p.15), but how do these statements align with the statement that "vowels in contact with nasal consonants tend to be nasalized (universally)" (p.2)?

p.4: "These signals are divided into 250 ms windows with 100 ms overlap (i.e., every window starts 100 ms after the previous one)"

-Based on this description, it would seem that every window starts 150 ms (not 100 ms) after the previous one.

p.6: "(/m/, /n/ and /ñ/)"

-"/ñ/" is not a symbol used in the IPA to denote any known consonant

p.6: "In this study, we assumed that the entire vowel was nasalized if it was shorter than 60 ms, 50% if it was between 60 and 90 ms, and 30% if it was longer than 90 ms."

-Based on what evidence?

p.7: "The transformation from a percentage to a 4-level scale required deciding the limits among classes. As we knew in advance how many children were classified perceptually as oral or nasal, or as Oral, Mild, Moderate or Severely nasal, we assumed that the same numbers would apply in the case of nasalance measures. Thus, after every simulation, the results were sorted per mfccNasalance, with the first N cases being categorized as Healthy, the next M cases as Mild, etc."

-This should be explained much more clearly, I don't understand this at all. Does this just meant that you assigned identical labels as the perceptual labels?

p.9: "Fiinally, the percentage ratings were recoded using a 4-level scale. Oral: nasality ≤ 0.05; Mild: 0.05 < nasality ≤ 0.25; Moderate: 0.25 < nasality ≤ 0.50; Severe: nasality > 0.50."

-Based on what evidence?

p.9: "Nasalance was computed as the ratio of nasal acoustic energy to the sum of oral and nasal energy"

-Please specify that you used RMS energy, as shown in the Praat script.

Fig.6, etc.: Why not two sets of box plots, or two sets of violin plots, etc.? Why one box plot and one line?

Fig.6, etc.: The caption says "Correlation between e-Nasalance [...] and mfccNasalance", but this is not accurate. They are separate correlations between e-Nasalance and perceptual scores, and mfccNasalance and perceptual scores.

p.10: "the scores are higher than eNasalance in most cases (i.e. all except the lower 25th percentile)."

-Are you referring to the lower whisker? The whisker is not the 25th percentile.

Table 2: Do you mean "p" instead of "s"? Also, what is "k11"?

p.13: "(Fig. 7 down right)" -> "bottom right"

p.14: "These results are compatible with the fact that nasalance is a poor approximation to perceptual nasality"

-Citation needed. Nasalance arose as a methodology precisely because it approximates perceptual nasality.

p.14: "in contrast, as we only need to know that a nasal sound was present (and not the exact location)"

-Maybe I've misunderstood, but this doesn't seem correct, because you based your labels off of phone-wise segmentation. In order to have phone-wise segmentation, you need to know the exact location of that phone in the speech stream.

-Punctuation errors throughout.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

**********

PLoS One. 2024 Dec 31;19(12):e0315452. doi: 10.1371/journal.pone.0315452.r004

Author response to Decision Letter 1

17 Oct 2024

Response to Reviewer 3 Comments

Comments made by the reviewer have been applied to correct erroneous terminology used, complement the referenced bibliography, and clarify parts of the paper for better understanding.

Questions

Question 1: Abstract, Table 1, elsewhere: "The test data were recorded in Spain and included short words (-dynamic), sentences (+dynamic), and diakinetic syllables (+dynamic)"

-What do these +/- labels mean?

- Why is the /ka/ in "ka" considered "dynamic" but the /ka/ in "boka" isn't? These terms are never defined or justified.

- I am not familiar with the term "diakinetic". Do you mean "diadochokinetic"?

Response 1: Label means: more (+) and less (-). Abstract is modified to include clarification.

Utterance in Table 1, Diakinetic syllable repetition is modified. Instead of /ka/ is marked as /kakaka…/, and a paragraph is added to clarify that it represents many repetitions of syllable /ka.

The term diakinetic is replaced by diadochokinetic.

Question 2: p.3: "To our knowledge, only one ML model has the flexibility of Nasometry [18]"

-The authors may want to be aware of similar works that have previous been carried out, for example Siriwardena et al. (2024) and Carignan (2021), the latter of which has been extended to low-resource languages (Carignan et al., 2023) and hyper-nasality in children (Fagniart et al.,2024), two goals which the current study also aims to meet.

Fagniart, S., Charlier, B., Delvaux, V., Huberlant, A., Harmegnies, B.G., Piccaluga, M., & Huet,K. (2024). "Consonant and vowel production in children with cochlear implants: acoustic measures and multiple factor analysis", Front. Audiol. Otol. 2:1425959. doi:10.3389/fauot.2024.1425959

Siriwardena, Y.M., Boyce, S.E., Tiede, M.K., Oren, L., Fletcher, B., Stern, M., & Espy-Wilson,C.Y. (2024). "Speaker-independent speech inversion for recovery of velopharyngeal port constriction degree", J. Acoust. Soc. Am. 156 (2), 1380–1390. doi: https://doi.org/10.1121/10.0028124

Response 2: These related works have been added in the Introduction section.

Question 3: You say that European Spanish is "a non-nasal dialect" (p.3) and that "Costa Rican Spanish is known to be highly nasal, while European and Chilean Spanish are comparatively similarly oral"(p.15), but how do these statements align with the statement that "vowels in contact with nasal consonants tend to be nasalized (universally)" (p.2)?

Response 3: The fact that vowels in contact with a nasal tend to nasalise is universal, although dialects and languages vary in the degree to which this occurs, as is in the case of European and Chilean Spanish.

Changes in text are added: “However, because vowels in contact with nasal consonants tend to be nasalized (universally, although the degree varies according to language or dialect), they used these vowel fragments to create a large set of nasal vowels”.

Question 4: p.4: "These signals are divided into 250 ms windows with 100 ms overlap (i.e., every window starts 100 ms after the previous one)"-Based on this description, it would seem that every window starts 150 ms (not 100 ms) after the previous one.

Response 4: Every window starts 100 ms after the previous one. Text modified to be accurate to 150 ms overlap.

Question 5: p.6: "(/m/, /n/ and /ñ/)"

-"/ñ/" is not a symbol used in the IPA to denote any known consonant.

Response 5: /ñ/ is replaced by the IPA symbol /ɲ/ .

Question 6: p.6: "In this study, we assumed that the entire vowel was nasalized if it was shorter than 60 ms, 50% if it was between 60 and 90 ms, and 30% if it was longer than 90 ms."

-Based on what evidence?

Response 6: This distribution of vowel nasalisation length is based on experiments conducted in the early design phases of the present study. A subset of the training group is selected and trained with an automatic classification system, the aim being to select different nasalised vowel lengths so that the output corresponds to the annotation produced by the speech specialist. The results shown in the paper are the ones that obtain an optimal result.

Text is modified to add clarification.

Question 7: p.7: "The transformation from a percentage to a 4-level scale required deciding the limits among classes. As we knew in advance how many children were classified perceptually as oral or nasal, or as Oral, Mild, Moderate or Severely nasal, we assumed that the same numbers would apply in the case of nasalance measures. Thus, after every simulation, the results were sorted per mfccNasalance, with the first N cases being categorized as Healthy, the next M cases as Mild, etc."

-This should be explained much more clearly, I don't understand this at all. Does this just meant that you assigned identical labels as the perceptual labels?

Response 7: The CNN output produces values between 0 and 1, which have no clinical value. Therefore, the output is transformed into a 4-value scale. This transformation is done keeping the original training case distribution, from which we know the classification as oral, mild, moderate or severely nasal.

Text has been modified to clarify it.

Question 8: p.9: "Finally, the percentage ratings were recoded using a 4-level scale. Oral: nasality ≤ 0.05;Mild: 0.05 < nasality ≤ 0.25; Moderate: 0.25 < nasality ≤ 0.50; Severe: nasality > 0.50."

-Based on what evidence?

Response 8: This measure is supported by clinical practice experience in our lab. Experience shows that:

- Any healthy speaker can nasalize occasionally (hence 5% seems a reasonable limit).

- Nasalizing over 50% of the speech utterances is evidence of severe hypernasality.

- Intervals between 5% and 50% are divided equally.

A clarification is added in this section.

Question 9: p.9: "Nasalance was computed as the ratio of nasal acoustic energy to the sum of oral and nasal energy”.

-Please specify that you used RMS energy, as shown in the Praat script.

Response 9: RMS energy use is specified.

Question 10: Fig.6, etc.: Why not two sets of box plots, or two sets of violin plots, etc.? Why one box plot and one line?

Response 10: The line represents the eNasalance correlation with the perceptual nasality scores of human experts, it only has one measure, that’s why is represented as a line.

The boxplots represent the mfccNasalance correlation with human experts. They have 126 results per utterance type, as we made a k-fold = 5 cross-correlation with 126 kernel combinations and compute the mean value for kernel.

The main purpose of Figure 6 was to illustrate the correlation between eNasalance and human experts, and mfccNasalance with human experts. Separate images are created to illustrate the cases with all available data (left), while the figure on the right shows the cases with the optimal configuration: Spain: Syllable k11 = True, Words Kernels = Temporal, Sentences k11 = True.

Question 11: Fig.6, etc.: The caption says, "Correlation between e-Nasalance [...] and mfccNasalance", but this is not accurate. They are separate correlations between e-Nasalance and perceptual scores, and mfccNasalance and perceptual scores.

Response 11: Fig. 6 and 7 captions are modified to include: “Correlation between e-Nasalance and perceptual scores (orange rectangle), and mfccNasalance and perceptual scores (blue)”

Question 12: p.10: "the scores are higher than eNasalance in most cases (i.e. all except the lower 25thpercentile)."

-Are you referring to the lower whisker? The whisker is not the 25th percentile.

Response 12: Sentence is modified to indicate: “i.e. all except the lower whisker”.

Question 13: Table 2: Do you mean "p" instead of "s"? Also, what is "k11"?

Response 13: “s” is changed to “p”.

k11 denoted the presence of a 1 × 1 kernel in the first layer of the CNN model. Clarification added in Table 2 and Table 3.

Question 14: p.13: "(Fig. 7 down right)" -> "bottom right"

Response 14: Text is corrected.

Question 15: p.14: "These results are compatible with the fact that nasalance is a poor approximation to perceptual nasality"

-Citation needed. Nasalance arose as a methodology precisely because it approximates perceptual nasality.

Response 15: Citation is added. Studies comparing the nasalance scores and perceptual scores have obtained correlations ranging from non-significant to strong.

Liu, Y.; Lee, S.A.S.; Chen,W. The correlation between perceptual ratings and nasalance scores in resonance disorders: A systematic review. J. Speech Lang. Hear. Res. 2022, 65, 2215–2234. [CrossRef] [PubMed]

Question 16: p.14: "in contrast, as we only need to know that a nasal sound was present (and not the exact location)"

Response 16: In contrast to Mathad et al., who use a 25 ms window, the present work uses a CNN with a larger window, 250 ms, which allows for less accurate annotation.

In the present work the classification of a sound as oral/nasal is based on the percentage of nasality present, so an error in the precise location of the nasal sound does not prevent the 250ms fragment from being classified correctly.

A clarification is added in this section.

Question 17: Punctuation errors throughout.

Response 17: Punctuation errors are corrected.

Attachment

Submitted filename: Response_to_reviewers.pdf

pone.0315452.s002.pdf^{(141.8KB, pdf)}

PLoS One. doi: 10.1371/journal.pone.0315452.r005

Decision Letter 2

Laura Morett

6 Nov 2024

PONE-D-24-18061R2Computing nasalance with MFCCs and Convolutional Neural NetworksPLOS ONE

Dear Dr. Lozano Durán,

Please submit your revised manuscript by Dec 21 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

We look forward to receiving your revised manuscript.

Kind regards,

Laura Morett

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

I thank the authors for their attention to the points raised in the last round of reviews. I agree with R3 that the revisions largely address the points raised by the reviewers, and I ask that they address the remaining points raised by R3. Provided they do so, I will render a decision without re-sending the manuscript for review.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #3: Yes

**********

6. Review Comments to the Author

Reviewer #3: Some minor issues still remain from my previous comments, but by and large the authors have done an admirable job with the revision!

p.2: "Furthermore, there are currently multiple machine learning (ML) algorithms that may serve to classify complex feature sets, such as MFCCs (see [12-14]), or high-speed nasopharyngoscopy [15]."

-The authors seem to have missed the point in providing these additional references, which are not simply for "classify[ing] complex feature sets". Carignan (2021) uses MFCCs in combination with other acoustic features to train an ML algorithm to create nasalance-like signals, and compares the results with nasometry measures. Siriwardena et al. (2024) use the full acoustic waveform to train an ML algorithm to create nasalance-like signals, and compare the results with nasopharyngoscopy. Therefore, both of these works have previously used speech acoustics to train ML algorithms to create nasalance-like signals, in a similar manner to Mathad et al. (2021) and to the current study, which is somewhat at odds with the claim that "only one ML model has the flexibility of Nasometry" (p. 2); both Carignan (2021) and Siriwardena et al. (2024) were carried out very specifically to generate nasalance-like signals from speech acoustics using ML algorithms.

Response 13: “s” is changed to “p”

-"s" still remains in the text at many different points

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

**********

PLoS One. 2024 Dec 31;19(12):e0315452. doi: 10.1371/journal.pone.0315452.r006

Author response to Decision Letter 2

8 Nov 2024

Response to Reviewer 3 Comments

New comments made by the reviewer have been applied to correct issues that remain from previous review. Minor typos are corrected, and additional text has been added.

Questions

Question 1: p.2: "Furthermore, there are currently multiple machine learning (ML) algorithms that may serve to classify complex feature sets, such as MFCCs (see [12-14]), or high-speed nasopharyngoscopy [15]."

-The authors seem to have missed the point in providing these additional references, which are not simply for "classify[ing] complex feature sets".

Carignan (2021) uses MFCCs in combination with other acoustic features to train an ML algorithm to create nasalance-like signals and compares the results with nasometry measures.

Siriwardena et al. (2024) uses the full acoustic waveform to train an ML algorithm to create nasalance-like signals and compare the results with nasopharyngoscopy.

Therefore, both of these works have previously used speech acoustics to train ML algorithms to create nasalance-like signals, in a similar manner to Mathad et al. (2021) and to the current study, which is somewhat at odds with the claim that "only one ML model has the flexibility of Nasometry" (p. 2); both Carignan (2021) and Siriwardena et al.(2024) were carried out very specifically to generate nasalance-like signals from speech acoustics using ML algorithms.

Response 1: Thank you for the clarification, text has been adapted to the suggestions mentioned above.

Text added in section “Previous ML proposals”:

Question 2: Response 13: “s” is changed to “p”

-"s" remains in the text at many different points

Response 2: The remaining instances of “s” have been changed to “p”.

Attachment

Submitted filename: Response_to_reviewers.docx

pone.0315452.s003.docx^{(19.1KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0315452.r007

Decision Letter 3

Laura Morett

26 Nov 2024

Computing nasalance with MFCCs and Convolutional Neural Networks

PONE-D-24-18061R3

Dear Dr. Lozano Durán,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Laura Morett

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

I thank the authors for their revisions in response to the remaining points raised by R3. The manuscript can now be accepted for publication in PLOS One.

Reviewers' comments:

PLoS One. doi: 10.1371/journal.pone.0315452.r008

Acceptance letter

Laura Morett

6 Dec 2024

PONE-D-24-18061R3

PLOS ONE

Dear Dr. Lozano Durán,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Laura Morett

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Attachment

Submitted filename: Response_to_reviewers.docx

pone.0315452.s001.docx^{(22.2KB, docx)}

Attachment

Submitted filename: Response_to_reviewers.pdf

pone.0315452.s002.pdf^{(141.8KB, pdf)}

Attachment

Submitted filename: Response_to_reviewers.docx

pone.0315452.s003.docx^{(19.1KB, docx)}

Data Availability Statement

[pone.0315452.ref001] 1.Kummer A., "Cleft Palate and Craniofacial Anomalies: Effects on Speech and Resonance, ed 3. Clifton Park," ed: Delmar Publishing, 2013. [Google Scholar]

[pone.0315452.ref002] 2.Kuehn D. P. and Moller K. T., "Speech and language issues in the cleft palate population: the state of the art," The Cleft palate-craniofacial journal, vol. 37, no. 4, pp. 1–35, 2000.10670881 [Google Scholar]

[pone.0315452.ref003] 3.Howard S. and Lohmander A., Cleft palate speech: assessment and intervention. John Wiley & Sons, 2011. [Google Scholar]

[pone.0315452.ref004] 4.John A., Sell D., Sweeney T., Harding-Bell A., and Williams A., "The cleft audit protocol for speech—augmented: A validated and reliable measure for auditing cleft speech," The Cleft palate-craniofacial journal, vol. 43, no. 3, pp. 272–288, 2006. doi: 10.1597/04-141.1 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref005] 5.Bettens K., Wuyts F. L., and Van Lierde K. M., "Instrumental assessment of velopharyngeal function and resonance: A review," Journal of communication disorders, vol. 52, pp. 170–183, 2014. doi: 10.1016/j.jcomdis.2014.05.004 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref006] 6.Fletcher S. G., Sooudi I., and Frost S. D., "Quantitative and graphic analysis of prosthetic treatment for “nasalance” in speech," The Journal of Prosthetic Dentistry, vol. 32, no. 3, pp. 284–291, 1974. doi: 10.1016/0022-3913(74)90032-8 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref007] 7.Gildersleeve-Neumann C. E. and Dalston R. M., "Nasalance scores in noncleft individuals: why not zero?," The Cleft palate-craniofacial journal, vol. 38, no. 2, pp. 106–111, 2001. doi: 10.1597/1545-1569_2001_038_0106_nsiniw_2.0.co_2 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref008] 8.Carignan C., "Using ultrasound and nasalance to separate oral and nasal contributions to formant frequencies of nasalized vowels," The Journal of the Acoustical Society of America, vol. 143, no. 5, pp. 2588–2601, 2018, doi: 10.1121/1.5034760 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref009] 9.Keuning K. H., Wieneke G. H., Van Wijngaarden H. A., and Dejonckere P. H., "The correlation between nasalance and a differentiated perceptual rating of speech in Dutch patients with velopharyngeal insufficiency," The Cleft palate-craniofacial journal, vol. 39, no. 3, pp. 277–284, 2002. doi: 10.1597/1545-1569_2002_039_0277_tcbnaa_2.0.co_2 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref010] 10.Brancamp T. U., Lewis K. E., and Watterson T., "The relationship between nasalance scores and nasality ratings obtained with equal appearing interval and direct magnitude estimation scaling methods," The Cleft Palate-Craniofacial Journal, vol. 47, no. 6, pp. 631–637, 2010. doi: 10.1597/09-106 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref011] 11.Khwaileh F. A., Alfwaress F. S., Kummer A. W., and Alrawashdeh M. m, "Validity of test stimuli for nasalance measurement in speakers of Jordanian Arabic," Logopedics Phoniatrics Vocology, vol. 43, no. 3, pp. 93–100, 2018. doi: 10.1080/14015439.2017.1370724 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref012] 12.McKechnie J., Ahmed B., Gutierrez-Osuna R., Monroe P., McCabe P., and Ballard K. J., "Automated speech analysis tools for children’s speech production: A systematic literature review," International journal of speech-language pathology, vol. 20, no. 6, pp. 583–598, 2018. doi: 10.1080/17549507.2018.1477991 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref013] 13.Dhillon H. et al., "Current applications of artificial intelligence in cleft care: a scoping review," Frontiers in medicine, vol. 8, p. 676490, 2021. doi: 10.3389/fmed.2021.676490 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0315452.ref014] 14.Carignan C., "A practical method of estimating the time-varying degree of vowel nasalization from acoustic features," The Journal of the Acoustical Society of America, vol. 149, no. 2, pp. 911–922, 2021. doi: 10.1121/10.0002925 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref015] 15.Siriwardena Y. M. et al., "Speaker-independent speech inversion for recovery of velopharyngeal port constriction degree," The Journal of the Acoustical Society of America, vol. 156, no. 2, pp. 1380–1390, 2024. [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref016] 16.He L., Zhang J., Liu Q., Yin H., Lech M., and Huang Y., "Automatic evaluation of hypernasality based on a cleft palate speech database," Journal of medical systems, vol. 39, pp. 1–7, 2015. [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref017] 17.Akafi E., Vali M., Moradi N., and Baghban K., "Assessment of hypernasality for children with cleft palate based on cepstrum analysis," Journal of medical signals and sensors, vol. 3, no. 4, p. 209, 2013. [PMC free article] [PubMed] [Google Scholar]

[pone.0315452.ref018] 18.Wang X., Tang M., Yang S., Yin H., Huang H., and He L., "Automatic hypernasality detection in cleft palate speech using cnn," Circuits, Systems, and Signal Processing, vol. 38, pp. 3521–3547, 2019. [Google Scholar]

[pone.0315452.ref019] 19.Fagniart S. et al., "Consonant and vowel production in children with cochlear implants: acoustic measures and multiple factor analysis," Frontiers in Audiology and Otology, vol. 2, p. 1425959, 2024. [Google Scholar]

[pone.0315452.ref020] 20.Moreno-Torres I., Lozano A., Nava E., and Bermúdez-de-Alvear R., "Which Utterance Types Are Most Suitable to Detect Hypernasality Automatically?," Applied Sciences, vol. 11, no. 19, p. 8809, 2021. [Google Scholar]

[pone.0315452.ref021] 21.Carignan C., Chen J., Harvey M., Stockigt C., Simpson J., and Strangways S., "An investigation of the dynamics of vowel nasalization in Arabana using machine learning of acoustic features," Laboratory Phonology, vol. 14, no. 1, pp. 1–31, 2023. [Google Scholar]

[pone.0315452.ref022] 22.Kummer A. W., "Evaluation of Speech and Resonance for Children with Craniofacial Anomalies," Facial plastic surgery clinics of North America, vol. 24, no. 4, pp. 445–451, 2016. doi: 10.1016/j.fsc.2016.06.003 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref023] 23.Mathad V. C., Scherer N., Chapman K., Liss J. M., and Berisha V., "A deep learning algorithm for objective assessment of hypernasality in children with cleft palate," IEEE Transactions on Biomedical Engineering, vol. 68, no. 10, pp. 2986–2996, 2021. doi: 10.1109/TBME.2021.3058424 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0315452.ref024] 24.Wang X., Yang S., Tang M., Yin H., Huang H., and He L., "HypernasalityNet: Deep recurrent neural network for automatic hypernasality detection," International Journal of Medical Informatics, vol. 129, pp. 1–12, 2019. doi: 10.1016/j.ijmedinf.2019.05.023 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref025] 25.Bongiovanni S., "Acoustic investigation of anticipatory vowel nasalization in a Caribbean and a non-Caribbean dialect of Spanish," Linguistics Vanguard, vol. 7, no. 1, p. 20200008, 2021. [Google Scholar]

[pone.0315452.ref026] 26.Fukushima K., "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biological cybernetics, vol. 36, no. 4, pp. 193–202, 1980. doi: 10.1007/BF00344251 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref027] 27.Elvin J., Williams D., and Escudero P., "Dynamic acoustic properties of monophthongs and diphthongs in Western Sydney Australian English," The Journal of the acoustical society of America, vol. 140, no. 1, pp. 576–581, 2016. doi: 10.1121/1.4952387 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref028] 28.Zatorre R. J. and Belin P., "Spectral and temporal processing in human auditory cortex," Cerebral cortex, vol. 11, no. 10, pp. 946–953, 2001. doi: 10.1093/cercor/11.10.946 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref029] 29.He K., Zhang X., Ren S., and Sun J., "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. [Google Scholar]

[pone.0315452.ref030] 30.Szegedy C. et al., "Going deeper with convolutions," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9. [Google Scholar]

[pone.0315452.ref031] 31.Boersma P., "Praat, a system for doing phonetics by computer," Glot. Int., vol. 5, no. 9, pp. 341–345, 2001. [Google Scholar]

[pone.0315452.ref032] 32.McAuliffe M., Socolof M., Mihuc S., Wagner M., and Sonderegger M., "Montreal forced aligner: Trainable text-speech alignment using kaldi," in Interspeech, 2017, vol. 2017, pp. 498–502. [Google Scholar]

[pone.0315452.ref033] 33.Bruyninckx M., Harmegnies B., Llisterri J., and Poch-Oiivé D., "Language-induced voice quality variability in bilinguals," Journal of Phonetics, vol. 22, no. 1, pp. 19–31, 1994. [Google Scholar]

[pone.0315452.ref034] 34.Martínez-Celdrán E., Fernández-Planas A. M., and Carrera-Sabaté J., "Castilian spanish," Journal of the International Phonetic Association, vol. 33, no. 2, pp. 255–259, 2003. [Google Scholar]

[pone.0315452.ref035] 35.Ortega-Garcia J., Gonzalez-Rodriguez J., and Marrero-Aguiar V., "AHUMADA: A large speech corpus in Spanish for speaker characterization and identification," Speech communication, vol. 31, no. 2–3, pp. 255–264, 2000. [Google Scholar]

[pone.0315452.ref036] 36.Sweeney C., The perceptual and instrumental assessment of nasality and nasal airflow errors associated with velopharyngeal dysfunction. Trinity College Dublin, 2000. [Google Scholar]

[pone.0315452.ref037] 37.Liu Y., Lee S. A. S., and Chen W., "The correlation between perceptual ratings and nasalance scores in resonance disorders: A systematic review," Journal of Speech, Language, and Hearing Research, vol. 65, no. 6, pp. 2215–2234, 2022. doi: 10.1044/2022_JSLHR-21-00588 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref038] 38.Moreno-Torres I., Lozano A., Bermúdez R., Pino J., Méndez M. D. G., and Nava E., "Unmasking Nasality to Assess Hypernasality," Applied Sciences, vol. 13, no. 23, p. 12606, 2023. [Google Scholar]

[pone.0315452.ref039] 39.Xu L. and Zheng Y., "Spectral and temporal cues for phoneme recognition in noise," The Journal of the Acoustical Society of America, vol. 122, no. 3, pp. 1758–1764, 2007. doi: 10.1121/1.2767000 [DOI] [PubMed] [Google Scholar]

[pone.0315452.ref040] 40.Hickok G. and Poeppel D., "Neural basis of speech perception," Handbook of clinical neurology, vol. 129, pp. 149–160, 2015. doi: 10.1016/B978-0-444-62630-1.00008-1 [DOI] [PubMed] [Google Scholar]

PERMALINK

Computing nasalance with MFCCs and Convolutional Neural Networks

Andrés Lozano

Enrique Nava

María Dolores García Méndez

Ignacio Moreno-Torres

Roles

Abstract

Introduction

Previous ML proposals

Our approach

Overview of the mfccNasalance model

Fig 1. Overview of the proposed CNN nasalance model for hypernasality prediction.

Audio input and preprocessing

Basic architecture of the neural networks

Fig 2. Convolutional Neural Network model for hypernasality prediction.

Kernels and phonetic information

Fig 3. Kernel shapes and phonetic information.

Training

Fig 4. Train data annotation and classification.

Testing and hypernasality computation

Methods

Speech samples

Table 1. Utterances in test data.

Dialect conditions

Fig 5. Train data annotation and classification.

Perceptual ratings

eNasalance ratings

Statistical analyses

Results

mfccNasalance vs. eNasalance: Same-dialect condition

Fig 6. Correlation between e-Nasalance and perceptual scores (orange rectangle), and mfccNasalance and perceptual scores (blue) in the same-dialect condition (Spain).

Kernel combinations and utterance types in the same-dialect condition

Table 2. Two-way ANOVA for Spain-trained models and optimal configuration per utterance type.

mfccNasalance vs. eNasalance: Different-dialect condition

Fig 7. Correlation between e-Nasalance and perceptual scores (orange rectangle), and mfccNasalance and perceptual scores (blue) in the different-dialect condition.

Table 3. Two-way ANOVA for Costa Rica (top) and Chile (down) trained models.

Discussion

mfccNasalance vs eNasalance

Optimal kernel configuration and utterance types

Dialect-effect

Future lines of research

Acknowledgments

Data Availability

Funding Statement

References

Decision Letter 0

Laura Morett

Roles

Author response to Decision Letter 0

Decision Letter 1

Laura Morett

Roles

Author response to Decision Letter 1

Decision Letter 2

Laura Morett

Roles

Author response to Decision Letter 2

Decision Letter 3

Laura Morett

Roles

Acceptance letter

Laura Morett

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases