Spectrotemporal Modulation Sensitivity in Speech and Melody Processing Among Mandarin Speakers

Abstract

Objectives:

Spectrotemporal acoustical markers enable humans to distinguish between speech and song across diverse cultures. Prior research revealed an asymmetric sensitivity in auditory processing: melody perception is more susceptible to spectral degradation, whereas speech sentence perception is more affected by temporal degradation. However, these findings have primarily been based on non-tonal languages, raising questions about how tonal languages, such as Mandarin, might influence these sensitivity patterns. This study investigates how Mandarin speakers process spectrotemporal features in speech and melody, addressing whether tonal language experience modulates the asymmetric spectrotemporal modulation sensitivity patterns observed in non-tonal languages.

Design:

Twenty-five Mandarin-speaking participants were recruited in the main experiment to discriminate speech or melody content of Mandarin songs under conditions of varying spectral or temporal degradation. An additional 25 participants in the control experiment listened to degraded hummed melodies or spoken sentences under similar degradation conditions.

Results:

Consistent with findings in non-tonal language speakers, Mandarin listeners were more sensitive to temporal degradation in speech and spectral degradation in melody. In addition, they exhibited heightened susceptibility to spectral degradation around 2 cyc/kHz in melody perception, indicating enhanced pitch sensitivity in music, possibly due to tonal language experience. Control analyses revealed reduced sensitivity when hummed melodies or spoken sentences were presented in a single domain, suggesting the influence of cognitive load on auditory processing.

Conclusions:

These findings underscore the quasi-universality of spectrotemporal cues in speech and melody perception across languages. However, tonal language experience subtly enhances spectral processing in music, providing new insights into the interaction between speech and music perception across different linguistic backgrounds.

INTRODUCTION

Speech and music are fundamental domains of auditory perception, sharing multiple acoustic cues but serving different communicative functions. On one hand, speech and song are both produced by the vocal tract, and have been proposed to have a common evolutionary origin (Patel 2007; Peretz et al. 2015). On the other hand, they have very distinct acoustic properties. Speech is primarily characterized by rapidly changing temporal features, while music relies heavily on spectral (frequency-based) information (Albouy et al. 2020). Speech rhythms typically range from 2 to 8 Hz, remaining relatively stable across languages, while music tempos generally fall between 0.5 and 3 Hz, corresponding to 30 and 180 beats per minute (Ding et al. 2017; Poeppel & Assaneo 2020). These distinct rhythm patterns lead to specific peaks in temporal modulation power, with speech centered around 5 Hz and music around 2 Hz (Ding et al. 2017). While both domains engage spectral cues, each emphasizes distinct acoustic features under the spectral domain. Speech utilizes suprasegmental features such as prosody to convey both lexical (Cutler et al. 1997) and emotional information (Scherer 1986), while music relies on pitch, pitch contours and timbre to construct melody and harmony (Schmuckler 2008).

Cognitive and neural evidence reveals both overlap and distinction in the perception of speech and music (Peretz et al. 2015; Te Rietmolen et al. 2024). Some studies propose a shared origin for the two domains (Nayak et al. 2022; Alagöz et al. 2024), while others highlight domain-specific processing within the auditory system (Norman-Haignere et al. 2015; Chang et al. 2024). Beyond its clinical application as a psychoacoustic test for assessing hearing loss (Bernstein et al. 2013; Zaar et al. 2023 2024), the spectrotemporal modulation (STM) framework has emerged as a powerful analytical approach for quantifying and manipulating acoustic features in both speech and music. It offers a mathematically rigorous representation of acoustic information across spectral and temporal dimensions, aligning with known neurophysiological properties of auditory processing (Singh & Theunissen 2003; Chi et al. 2005; Elliott & Theunissen 2009). STM analysis has successfully differentiated various natural sounds, including speech (Elhilali et al. 2003; Elliott & Theunissen 2009; Flinker et al. 2019), music (Thoret et al. 2016, 2020), alarm signals (Arnal et al. 2015, 2019), environmental sounds (Lemaitre et al. 2018; Robert et al. 2024), and animal vocalizations (Kowalski et al. 1996; Woolley et al. 2005), demonstrating its utility in characterizing the perceptual mechanisms underlying auditory perception, with different sound categories representing complementary acoustic niches (Krause 1987) and different neural networks. Crucially, an asymmetry has been observed in how spectral and temporal degradations affect speech and music perception. Albouy et al. (2020) found that French and English listeners are more sensitive to temporal degradation when processing the sentence content in songs, while melody discrimination is more susceptible to spectral degradation. This asymmetry also maps onto differential neural responses, with temporal degradation mainly affecting the left auditory cortex during sentence processing, and spectral degradation affecting the right auditory cortex during melody processing.

However, tonal languages, like Mandarin, may make different demands on language processing than non-tonal languages, yet their impact on STM sensitivity remains largely unexplored. Mandarin uses pitch variation to convey lexical meaning, a feature that may affect how its speakers process acoustic signals in both speech and music. Studies have shown distinct sensitivity to lexical tones in tonal language speakers compared with non-tonal language speakers (Li et al. 2018; Liang & Du 2018; Best 2019). Tonal language speakers show stronger perceptual weight toward pitch compared with other acoustic cues when perceiving English stress (Liu & Samuel 2004; Yu & Andruski 2010; Choi et al. 2018; Jasmin et al. 2021) and producing English stress (Nguyễn et al. 2008; Zhang et al. 2008). This enhanced sensitivity extends beyond speech, with superior performance in pure pitch or melody perception in tonal language speakers than non-tonal language speakers (Pfordresher & Brown 2009; Krishnan et al. 2010, 2019; Bidelman et al. 2011; Zheng & Samuel 2018). Neural evidence further supports this enhanced sensitivity, showing stronger neural responses, such as mismatch negativity, in tonal language speakers compared with non-tonal language speakers when perceiving speech or music (Giuliano et al. 2011; Hutka et al. 2015). However, this heightened pitch sensitivity does not always translate to better performance across auditory tasks (Peretz et al. 2011). Research by Albouy et al. (2024) has further demonstrated that STM profiles alone are sufficient to effectively differentiate speech from song across 21 languages including Mandarin and other tonal languages, raising the question of whether Mandarin speakers exhibit similar asymmetrical sensitivity to acoustic degradation as observed in non-tonal languages.

Building on these findings, the present study examined whether Mandarin speakers exhibit the same asymmetric effects of temporal and spectral degradation as speakers of non-tonal languages. We adapted the STM paradigm from Albouy et al. (2020) to Mandarin by using melodies sung in Mandarin, and investigated whether temporal degradation more strongly affects speech perception, while spectral degradation more strongly affects melody perception in a tonal language context. We expected to observe similar asymmetric effects in Mandarin speakers as previously reported for English and French, with an interaction of behavior performance in discriminating melodies or sentences under temporal and spectral degradations. We also hypothesized that Mandarin speakers might exhibit differentiated sensitivity for melody perception. While the temporal structures of speech were likely to show consistent sensitivity across languages (Ding et al. 2017; Varnet et al. 2017; Poeppel & Assaneo 2020), Mandarin speakers’ enhanced sensitivity to pitch might also result in heightened susceptibility to spectral degradation in both speech and melody perception. In the main experiment, participants listened to songs, processing both melody and speech content (dual domain) simultaneously. To test whether the asymmetrical sensitivity observed under the STM framework would persist when participants attended to a single domain, we conducted a control experiment. In this control task, participants listened to single-domain spoken sentences or hummed melodies—isolated from the dual-domain songs—and subjected to the same spectral and temporal degradations as in the main experiment. This allowed us to assess whether single-domain stimuli show reduced sensitivity to STM degradation, possibly due to less cognitive load. The present study thus aims to offer insights into the extent to which STM sensitivity in speech and music perception is similar across tonal and non-tonal languages, or is modulated by tonal language experience.

MATERIALS AND METHODS

Participants

Two groups of native Mandarin speakers were recruited for the study with a comparable sample size as in Albouy et al. (2020). The main experiment included 25 participants (13 females, 12 males; mean age = 23.4 years, age range = 19 to 29), while the control experiment included 25 participants (14 females, 11 males; mean age = 23.6 years, age range = 18 to 31). All participants were right-handed, had normal hearing (binaural pure tone thresholds were below 25 dB at all tested frequencies between 250 and 8000 Hz) and reported no neurological disorder. The study was approved by the Ethical Committee of the local Institutional Review Board. All participants provided written informed consent and received a monetary compensation of 60 to 80 CNY for their participation.

Stimuli

A total of 100 a cappella songs (10 melodies × 10 sentences) were constructed according to the same orthogonal paradigm used in Albouy et al. (2020), utilizing the same 10 melodies created by a professional composer. To adapt the paradigm to Mandarin, each sentence included 10 Chinese syllables, with pairs of sentences sharing identical syntax structure and rhyming on the final vowel (see Supplemental Digital Content, https://links.lww.com/EANDH/B765).

The songs in the main experiment were recorded from a professional Mandarin-speaking singer in a soundproof recording room, with a sampling rate of 44.1 kHz and 24-bit depth. Each song lasted 4.1 ± 0.21 sec. All sound stimuli were normalized to a consistent level based on average root-mean-square amplitude (excluding silent portions below −44 dB) using GoldWave volume match batch processing (https://www.goldwavechina.cn/). This normalization resulted in a volume level of approximately −18 dB (root-mean-square ≈ 0.1252) across stimuli. Fundamental frequency (F0) comparisons between the Mandarin songs and those in English and French (Albouy et al. 2020) revealed no significant differences (Ps > 0.05, two-sample t test, see Supplemental Digital Content, https://links.lww.com/EANDH/B765). For the control experiment materials, the songs were decomposed into single-domain sentences and melodies: 10 spoken sentences without melodies and 10 hummed melodies without any speech content. The stimuli were recorded from the same singer and underwent the same sound preprocessing as in the main experiment.

Spectral and temporal degradations were applied to the songs following the STM framework by Elliott and Theunissen (2009). The audio signals were first converted into STM modulograms using the ModFilter algorithm (Elliott & Theunissen 2009). Briefly, a two-dimensional Fast Fourier Transform was applied on the autocorrelation matrix derived from the spectrographic representation of the sound signal, along both the spectral and temporal dimensions, yielding a matrix representing the STM domain (i.e., the STM modulogram). To implement degradation, a mask was then applied to the modulation domain to remove spectral or temporal modulations beyond specific cutoff frequencies. For spectral degradation, the lowpass filter cutoffs were at [0.6, 1.5, 1.8, 2.0, 3.0] cyc/kHz; and for the temporal degradation, the cutoffs were at [1.0, 1.5, 2.0, 2.5, 3.5] Hz. This resulted in 1000 (10 sentences × 10 melodies × 10 degradations) stimuli in total.

Procedure

The experiments were carried out in a sound-attenuated room, with auditory stimuli presented binaurally through Sennheiser HD380 Pro headphones driven by a Dell desktop computer. Target stimuli were presented at 65 dB sound pressure level, calibrated using a sound level meter (Larson-Davis Model 831, Depew, NY) paired with a flat-plate ear simulator (Larson-Davis Model AEC201-A). The headphones were firmly positioned on the ear simulator to ensure proper acoustic sealing during calibration.

The experiments were controlled using the Presentation software (Neurobehavioral Systems, Berkeley, CA, USA), following a similar paradigm as in Albouy et al. (2020). In each trial, participants listened to a pair of a cappella songs and were then cued to judge whether the paired songs matched in either melody or sentence content. The task cue appeared after the songs, requiring participants to process both melodic and speech information before knowing which domain to compare (Fig. 2A). They responded “same” or “different” by pressing keys 1 and 2 with their right index and middle fingers, respectively. Trials were self-paced with no time limit and no feedback. For melody trials, participants were prompted to compare the melody, ignoring the sentence content of the previously heard songs, while for sentence trials, participants were asked to do the opposite (Fig. 2A). Half of the song pairs were different in the domain of interest, while all song pairs were different in the domain of non-interest. The degradation levels were always the same between songs in one pair. Participants responded “same” or “different” by pressing keys 1 and 2 with their right index and middle fingers, respectively.

Fig. 2.

Task illustration and raw accuracy scores under the most degraded conditions for the melody and sentence tasks in four language groups. The same task paradigm (A) in Albouy et al. (2020) was adapted in Mandarin songs. In each trial, participants listened to a pair of songs and then a visual cue indicated which domain of interest (sentences or melodies) they should discriminate by key pressing. Raw performance scores in (B) Mandarin main experiment, (C) Mandarin control experiment, (D) French group, and (E) English group (data from Albouy et al. 2020). Scores are presented as percentages across task domains (melodies, sentences) and conditions (temporal degradation, baseline, spectral degradation), showing only the most degraded conditions labeled with bold fonts: “T1” for the temporal cutoff at 1 Hz, “S0.6” for the spectral cutoff at 0.6 cyc/kHz, and “B” for baseline (see Materials and Methods). Bar plots show mean percent correct responses, with white circles indicating individual data. Pairwise comparison of scores between degradation types within task conditions were conducted using two-tailed Wilcoxon signed rank tests. Ns, not significant. *p < 0.05, **p < 0.01, ***p < 0.001, Bonferroni corrected.

In the main experiment, trials were organized into five blocks, lasting 75 min. The first four blocks consisted of 50 trials each, pseudo-randomized and balanced by task domain (melody/sentence), degradation cutoff, and song pair type (same/different). Across four blocks, the 100 songs were presented twice, balanced for first/second presentation and degradation type (temporal/spectral). The final block included 40 trials with non-degraded stimuli, serving as a baseline to normalize behavioral performance.

In the control experiment, the procedure was similar in comparing paired stimuli, except that the stimuli were in one of the two domains (i.e., degraded sentences or hummed melodies) instead of songs. Degraded stimuli were in the first four blocks and non-degraded ones were in the final block.

Data Analysis

To identify the differences in STM modulogram pattern between Mandarin songs in the present study and English/French songs from Albouy et al. (2020), acoustic analyses were done in MATLAB (The Mathworks) following the procedure outlined in Albouy et al. (2024). Sound stimuli were first normalized by dividing each waveform by the maximum absolute amplitude, and then downsampled to 16 kHz. STM modulograms were extracted using the ModFilter algorithm (Elliott & Theunissen 2009) from the normalized and downsampled digital signals of each song, using the default parameters with a Gaussian window of 10 msec in the temporal domain and 32 Hz in the frequency domain. Then we averaged the STM modulograms across 100 songs and smoothed the averaged STM modulogram with a 2D Gaussian kernel (filter size: 3 × 3 matrix cells) along both the spectral and temporal modulation axes. The resulting STM pattern was compared between different languages (i.e., Mandarin versus English and Mandarin versus French) using non-parametric permutation test statistics. To do this, we did 1000 iterations by randomly flipping the signs of the true difference between two language conditions to create a null distribution. Then, a two-sided p value was computed by determining the proportion of permutations that exceeded the true mean difference on either tail of the null distribution. The test was conducted for each cell in the STM modulogram. Because our STM modulogram has 32,040 cells (267 temporal bands × 120 spectral bands), it was corrected at p < 0.05 with Benjamini-Hochberg false discovery rate correction (Benjamini & Hochberg 1995). We also obtained the average STM modulograms for 10 hummed melodies and 10 spoken sentences using the same procedure as for the songs. To compare the Mandarin songs with hummed melodies, we averaged the STM modulograms of songs sharing the same melody but differing in sentence content. Similarly, to compare with spoken sentences, we averaged the STM modulograms of songs sharing the same sentence but differing in melody.

Behavioral analyses were conducted in R (R Core Team 2021). One participant in the main experiment and one in the control experiment did not reach 70% accuracy in the baseline condition in at least one domain and were excluded from the analyses, resulting in 24 participants in each group. Raw scores were the average accuracies for each degraded song version across trials, while the baseline was the average accuracies for the non-degraded songs. We first selected the most degraded conditions (i.e., 1 Hz in temporal degradation and 0.6 cyc/kHz in spectral degradation) and compared the raw accuracy scores across three language groups using a 2 (task: melody versus sentence) × 3 (degradation type: temporal degradation, baseline, spectral degradation) × 3 (language: Mandarin, English, French) mixed-effect analysis of variance (ANOVA). The raw accuracy score data for English and French speakers were taken from Albouy et al. (2020). We then compared performance between the Mandarin main experiment and the control experiment using a 2 (task) × 3 (degradation type) × 2 (group: experimental versus control) mixed-effect ANOVA. The scores within each group and conditions were compared using Wilcoxon sign ranked tests (Bonferroni corrected), unless otherwise stated.

Next, we normalized accuracy scores for each participant based on their baseline performance using the same equation as in Albouy et al. (2020) and Haiduk et al. (2024):

$${\displaystyle \begin{array}{l}{\displaystyle Normalizedscore=\frac{Rawscore-Chance}{Baseline-Chance}-1}\end{array}}$$

The chance level is 50%. A normalized score of zero indicates performance equal to baseline, whereas a positive or negative score means better or worse performance compared with the baseline, respectively. Similarly as in Albouy et al. (2020), individual normalized performance was then modeled using a linear regression model, separately for each degradation type, task domain and group, to examine how performance changed with increasing levels of degradation. The slopes of linear regression models were extracted and ANOVA analyses (ez package in R, Lawrence 2016) were conducted on the slopes to investigate the effects of task (melody versus sentence), degradation type (spectral versus temporal), and group (experimental versus control). The statistical significance for F-test in ANOVA were implemented using non-parametric permutation using ezPerm. Next, the slopes were compared between the two task types within each degradation type and group using paired Wilcoxon signed ranked tests (two-tailed, Bonferroni corrected across conditions) and the slopes were compared with 0 using paired Wilcoxon signed ranked tests (two-tailed, Bonferroni corrected within conditions). In addition, for the melody task under spectral degradation, we also compared model fitting between the linear regression and sigmoid models for each participant, as some of them apparently exhibited a nonlinear pattern.

Considering that the same-different paradigms are subject to response bias (Macmillan & Creelman 2004), we complemented our accuracy analyses with two additional signal detection theory metrics: perceptual sensitivity index (d’) and response bias criterion (c). Hit rates (H) were operationalized as the proportion of “same” responses on same trials, while false alarm rates (F) represented the proportion of “same” responses on different trials. Although reversing the definitions (i.e., treating “different” responses on different trials as hits) would produce mathematically equivalent d’ values, we followed the convention by using the “same” responses to reflect sensitivity to stimulus identity (Macmillan & Creelman 2004). To prevent extreme values (perfect hit or false alarm rates) that would yield infinite d’ values, we implemented the log-linear correction (Hautus 1995). Specifically, perfect hit rates (H = 1.0) were adjusted to with $\frac{N_{+0.5}}{N_{sametrials}+1}$; perfect false alarm rates (FAR = 0) with $\frac{N_{falsealarm+0.5}}{N_{differenttrials}+1}$, where N denotes trial counts. This correction preserves the interpretability of sensitivity estimates while minimizing bias (Stanislaw & Todorov 1999).

First, the perceptual sensitivity (d’) was computed as:

$${\displaystyle \begin{array}{l}{\displaystyle d^{\prime }=z\left(H\right)-z\left(F\right)}\end{array}}$$

where z(H) and z(F) correspond to z-transformed hit and false alarm rates respectively. Next, the response bias criterion (c) was computed as:

$${\displaystyle \begin{array}{l}{\displaystyle c=-\frac{z\left(H\right)+z\left(F\right)}{2}}\end{array}}$$

A negative c indicates a liberal bias (tendency to respond “same”), while a positive c indicates a conservative bias (tendency to respond “different”). These signal detection metrics were subjected to the same analytical framework used for raw accuracy and normalized scores.

RESULTS

STM Patterns in Mandarin Songs

To examine how similar or different the STM characteristics were between the tonal and non-tonal languages, we analyzed the STM patterns of undegraded songs in Mandarin, English and French. The STM patterns of songs in each language and the pairwise contrasts using permutation t tests were shown in Figure 1. While the patterns are similar overall, we observed differences in Mandarin songs compared with the songs in other languages (Fig. 1A). Mandarin songs showed significantly stronger power between 1 and 6 cyc/kHz of spectral modulation and in high temporal ranges (>2 Hz) of temporal modulation compared with English and French songs (Fig. 1B). The English and French songs only differed from each other in very limited areas in low spectral modulation field (<2 cyc/kHz, Fig. 1B). We also extracted the STM patterns of the 10 hummed melodies and 10 spoken sentences (Fig. 1C). Further analysis revealed that Mandarin songs exhibited stronger spectral modulation power than the hummed melodies sung by the same Mandarin speaker below 2 cyc/kHz across temporal modulation rates except between −0.5 and 0.5 Hz, and stronger spectral modulation power than spoken sentences around 2 to 3 cyc/kHz across temporal modulation rates (Fig. 1D).

Fig. 1.

Acoustic STM patterns of songs, sentences and melodies. STM patterns were extracted for songs in (A) three languages and (C) hummed melodies and spoken sentences in Mandarin. Paired comparisons were conducted (B) between the STM patterns of three languages and (D) between songs and speech/melodies using permutation paired two-tailed t tests. The nonparametric t values were thresholded at p < 0.05, corrected for multiple comparisons using the false discovery rate (FDR). Black lines in B and D indicated the boundaries of significant effect. STM indicates spectrotemporal modulation.

Performance Comparisons Across Languages and Between the Main and Control Experiments

To examine whether the spectral-temporal sensitivity can be generalized across different languages, we first compared Mandarin speakers’ raw accuracy scores at the most degraded conditions (i.e., 1 Hz in temporal degradation and 0.6 cyc/kHz in spectral degradation) in the present study with those of French (N = 27) and English (N = 22) speakers from Albouy et al. (2020). The raw accuracy scores of different degradations type and tasks were shown in the separate panels for each participant group in Figure 2. A 2 (task) × 3 (degradation type: temporal degradation, baseline, spectral degradation) × 3 (group: Mandarin, English, French) three-way mixed ANOVA showed significant main effects of task [F(2,70) = 120.55, p_perm < 0.001, η² = 0.25] and degradation type [F(2,140) = 97.66, p_perm < 0.001, η² = 0.27], along with an interaction between the two [F(2,140) = 317.37, p_perm < 0.001, η² = 0.54], indicative of the harder difficulty in the melody task and of the STM sensitivity asymmetry in speech and melody processing. However, no significant main effect of group [F(2,70) = 0.85, p_perm = 0.457, η² = 0.01] or interaction between group and task [F(2,70) = 2.08, p_perm = 0.123, η² = 0.01] or group and degradation type [F(4,140) = 0.85, p_perm =.457, η² = 0.01] emerged. This finding suggests comparable patterns across languages. A significant three-way interaction [F(4,140) = 3.42, p_perm = 0.032, η² = 0.03) however suggests slight language-based variations. This interaction effect differed notably in the French group, which showed less impairment in the melody task under spectral degradation (raw scores: French = 62.3%; Mandarin = 54.2%; English = 55.0%) but more impairment in the sentence task under temporal degradation (raw score: French = 60.9%; Mandarin = 71.0%; English = 64.5%), compared with the Mandarin and English groups.

We then compared the raw accuracy scores under the most degraded conditions between the Mandarin main experiment and the control experiment (Figs. 2B, C). A 2 (task) × 3 (degradation type) × 2 (group) mixed-effect ANOVA showed a significant main effect of group [F(1,46) = 30.43, p_perm < 0.001, η² = 0.15], with overall better performance in the control experiment. There was also a main effect of task [F(1,46) = 71.59, p_perm < 0.001, η² = 0.20], with the melody task being more challenging than the sentence task. Moreover, a significant interaction between group and task [F(2,92) = 8.82, p_perm < 0.001, η² = 0.03) suggested that the group in the main experiment performed worse than the control experiment in the melody task (p < 0.001) but not in the sentence task (p = 0.20). These findings suggest an unbalanced cross-domain interference, with sentence presence increasing melody task difficulty but not vice versa. The degradation type also had a main effect [F(2,92) = 32.59, p_perm = 0.002, η² = 0.15), and more importantly, the task and degradation type had a significant interaction [F(2,92) = 74.52, p_perm < 0.001, η² = 0.34). Pairwise comparisons within task showed that performance decreased only under spectral degradations in the melody task and only under temporal degradations in the sentence task, further validating the acoustical cue-specific STM sensitivities observed across groups. Last, the three-way interaction had a weak but significant effect [F(2,92) = 12.92, p_perm < 0.001, η² = 0.08]. Similar but less pronounced asymmetric patterns were observed for raw accuracy scores averaged across all degradation levels for each degradation type, task and group (Figure S1 in Supplemental Digital Content, https://links.lww.com/EANDH/B765).

In addition, perceptual sensitivity (d’) were also compared across tasks and degradation types in both the main and control experiments (Figure S2 in Supplemental Digital Content, https://links.lww.com/EANDH/B765). Consistent with the accuracy pattern in the main experiment, d’ decreased the most under spectral degradations in the melody task and decreased most under temporal degradations in the sentence task (Figure S2a in Supplemental Digital Content, https://links.lww.com/EANDH/B765), further confirming the STM asymmetric influences. The control experiment exhibited a similar but weaker double dissociation pattern in d’ (Figure S2b in Supplemental Digital Content, https://links.lww.com/EANDH/B765). While no response bias was observed in the control experiment, some response bias emerged in the main experiment, especially under spectral degradation in the melodic task (Figure S2c in Supplemental Digital Content, https://links.lww.com/EANDH/B765). See statistical details in Supplemental Digital Content, https://links.lww.com/EANDH/B765.

Distinct STM Sensitivities in Mandarin Speech and Melody Processing

To further evaluate sensitivity to degradation in Mandarin speakers, we analyzed accuracy scores across five degradation cutoffs, separately for different tasks (discriminating melodies versus sentences) and degradation types (spectral versus temporal) (left panels in Fig. 3). The individual accuracies under each condition were then fitted with linear regressions (average slopes and 95% confidence interval shown in each mid-panel in Fig. 3) and the linear coefficients were compared between the sentence and melody task under each degradation type and group (each right panel in Fig. 3). Two-way repeated ANOVAs on task and degradation type were conducted for the fitted linear coefficients for the main and control experiments, respectively. In the main experiment the fitted linear coefficients showed no main effect of task [F(1,23) = 0.13, p_perm = 0.720, η² = 0.001], but a significant main effect of degradation type [F(1,23) = 4.3, p_perm = 0.044, η² = 0.05] and task-by-degradation interaction [F(1,23) = 44.23, p_perm < 0.001, η² = 0.36]. Paired comparisons using Wilcoxon signed rank tests (two-tailed, Bonferroni corrected in 2 degradation types × 2 groups) showed a double-dissociation, namely that sentence task performance was more sensitive to temporal degradations than melody task performance (mean slope differences = −0.33, Z = −3.82, p = 0.0054; Fig. 3A, right panel), while melody task performance was more sensitive to spectral degradations than sentence task performance (mean slope differences = 0.36, Z = 3.97, p = 0.0030; Fig. 3B, right panel). Further analyses of one sample Wilcoxon tests (two-tailed) showed a significant slope only for temporal degradation in the sentence task (slope = 0.210, Z = 4.23, p < 0.001) and spectral degradation in the melody task (slope = 0.334, Z = 4.23, p < 0.001), but not in the two other conditions (p > 0.05).

Fig. 3.

Normalized performance in melody and sentence tasks under temporal and spectral degradations. The normalized performance under (A) temporal degradations and (B) spectral degradations in the main experiment using post-stimulus cueing on songs; and the normalized performance under (C) temporal degradations and (D) spectral degradations in the control experiment using sound stimuli in single domain (hummed melodies or spoken sentences). Each panel from left to right shows: (left) average normalized scores across participants with 95% confidence intervals in shade at five degradation cutoffs, (mid) individual performance modelled by linear regression or sigmoid regression (for the melody task under spectral degradations only), and (right) individual linear regression coefficients with error bars of 95% confidence intervals for both melody task (orange) and sentence task (teal). Paired comparisons were conducted between the slopes of two tasks within each panel, and each slope within condition against 0, using two-tailed Wilcoxon signed rank tests. Ns, nonsignificant. *p < 0.05, ***p < 0.001, †p < 0.10 by Bonferroni correction.

In the control experiment, we observed no main effects of task (p_perm = 0.727) or degradation type (p_perm = 0.321), but a weak interaction effect [F(1, 23) = 4.23, p_perm < 0.013, η² = 0.36]. There was a weak but significant difference between slopes of melody and sentence tasks under temporal degradations (mean slope differences = −0.10, Z = −2.71, p = 0.048; Fig. 3C, right panel), but not under spectral degradations (p = 0.281).Further one sample Wilcoxon tests (two-tailed) showed significant slope for temporal degradation in the sentence task (slope = 0.067, Z = 1.71, p = 0.014). However, spectral degradation in the melody task did not survive after Bonferroni correction (slope = 0.139, Z = 2.00, p = 0.047, before correction), and no significant effects were found in the two other conditions (p > 0.05).

In addition, a nonlinear trend in the melody task under spectral degradations from 1.5 to 2 cyc/kHz was noted in the main experiment (Fig. 3B, mid-panel). To quantify this, we applied a sigmoid model to normalized performance scores as a function of spectral degradation cutoffs, using the nlsr package in R. While the sigmoid model (R²_sigmoid = 0.357, Akaike Information Criterion [AIC] = 145.473, solid red line) provided a significantly better fit [F(1,2) = 10.749, p < 0.001] than the linear model (R²_linear = 0.238, AIC = 161.876, dashed orange line), this result should be interpreted cautiously given the limited data points. We also included individual-level analyses (Figure S3 in Supplemental Digital Content, https://links.lww.com/EANDH/B765), where only 8 out of 24 participants showed convergence and improved fit with greater variances explained (R²) with the sigmoid model over a linear model, further emphasizing the tentative nature of these findings. Nonetheless, we extracted the individual slopes at the midpoint of the sigmoid curve for the 8 participants best fitted by this model and compared them to the linear slopes of the remaining 16 participants in the melody task. The independent two-tailed Wilcoxon test showed that sigmoid slopes (mean = 3.56) were significantly steeper than the linear slopes (mean = 0.334, Z = 3.40, p < 0.001, Fig. 3B, mid-panel inset). We also conducted the similar analysis for each participant in the English and French groups. The sigmoid model failed to converge for the group-level data, so we fit the model to each participants’ normalized performance under the same condition (i.e,. spectral degradations in the melody task). Most participants exhibited a linear sensitivity to spectral degradations when discriminating melodies, with only 4 participants (<20%) in each group (English: Figure S4 in Supplemental Digital Content, https://links.lww.com/EANDH/B765; French: Figure S5 in Supplemental Digital Content, https://links.lww.com/EANDH/B765) showing a nonlinear trend. We compared the linear modelling slopes of the melody task under spectral degradations across the three language groups using one way ANOVA and found no significant group differences, F(2, 70) = 1.28, p_perm = 0.274, η² = 0.04 (Chinese: mean slope = 0.334; English: mean slope = 0.402; French: mean slope = 0.259).

Following the same analytical approaches, we also fitted the degradation levels against the individual perceptual sensitivity (d’) within each task and degradation types using linear regression. In the main experiment, we observed consistent STM asymmetric effects: d’ in the sentence task showed significant impairments with increasing temporal degradations (slope = 0.463, Z = 4.11, p < 0.001) but remained unaffected by spectral degradations (slope = 0.016, Z = 0.286, p = 0.990, Figure S6A in Supplemental Digital Content, https://links.lww.com/EANDH/B765); in contrast, d’ in the melody task decreased significantly with increasing spectral degradations (slope = 0.457, Z = 3.97, p < 0.001) but not temporal degradations (slope = −0.187, Z = −2.23, p = 0.124, Figure S6B in Supplemental Digital Content, https://links.lww.com/EANDH/B765). In the control experiment, d’ in discriminating spoken sentences was not affected by either degradation (Figure S6C in Supplemental Digital Content, https://links.lww.com/EANDH/B765), whereas d’ in discriminating hummed melodies was only affected by spectral degradations (slope = 0.273, Z = −3.04, p = 0.009, Figure S6D in Supplemental Digital Content, https://links.lww.com/EANDH/B765). See statistical details in Supplemental Digital Content, https://links.lww.com/EANDH/B765.

DISCUSSION

This study investigated the sensitivity to temporal and spectral degradations in speech and music processing among Mandarin speakers using the STM framework. Consistent with previous findings in non-tonal language speakers (Albouy et al. 2020; Haiduk et al. 2024), Mandarin speakers demonstrated an asymmetric sensitivity: as evidenced by accuracy and d’, speech sentence perception was more susceptive to temporal degradation, while melody perception was more vulnerable to spectral degradation (Figs. 2 and 3). Although some group differences were observed—particularly in the extent of spectral degradation effects on melody processing between Mandarin and English/French participants (Fig. 3B and Discussion later)—the overall pattern support the conclusion that the STM framework generalizes to tonal languages. This result aligns with recent cross-cultural evidence showing that spectrotemporal features are both necessary and sufficient to classify speech from songs across 21 diverse societies (Albouy et al. 2024). Together, these results reinforce the notion of a universal acoustic model of auditory processing for speech and music that transcends linguistic and cultural boundaries.

Interestingly, the reliance on pitch variation for lexical meaning in Mandarin did not alter the susceptibility of speech perception to spectral degradation. This suggests that, at the sentence-level, suprasegmental pitch information may be less crucial than other cues, such as voice onset time, in distinguishing speech content (Massaro & Cohen 1977). Moreover, the redundancy of spectral information in speech perception likely supports this resilience (Tierney 2018), allowing Mandarin listeners to maintain performance under spectral degradation in sentence processing.

However, some Mandarin speakers exhibited a unique heightened sensitivity to spectral degradation within specific frequencies (1.5 to 2 cyc/kHz) during melody perception (see Fig. 3B and Figure S3 in Supplemental Digital Content https://links.lww.com/EANDH/B765), as revealed by the sigmoid model fits, which may reflect an enhanced pitch sensitivity shaped by tonal language experience. These findings suggest that while core perceptual mechanisms underlying speech and music processing are largely consistent across languages, language-specific experiences can subtly modulate auditory sensitivity within these domains. The heightened sensitivity to spectral degradation in melody processing aligns with prior research indicating that tonal language experience strengthens pitch processing capabilities (Pfordresher & Brown 2009; Zheng & Samuel 2018; Krishnan et al. 2019). This enhanced pitch sensitivity may reflect a heightened perceptual weighting on spectral cues, shaped by linguistic experience, that extends from speech into music perception. Mandarin songs also displayed more power around 2 cyc/kHz of spectral modulation and in high temporal ranges (>2 Hz) of temporal modulation compared with English and French songs (Fig. 2), possibly due to language-specific phonetic structures. The spectral feature may contribute to the observed enhanced sensitivity around this frequency range in melody perception. Notably, this spectral sensitivity shift did not manifest in sentence perception, underscoring a more fine-grained reliance on spectral cues in music than in speech. However, interpretation of the heightened sensitivity should be considered preliminary. The apparent nonlinear sensitivity to spectral degradation in melody processing (Fig. 3B) was based on five data points. Although the sigmoid model fit the data better than a linear model at the group level, this was not consistent across individuals. Moreover, using the linear model yielded no significant slope differences among the three language groups under this condition. This suggests that the heightened sensitivity observed in Mandarin speakers is restricted to a specific spectral modulation range, rather than across the full range. Future studies should incorporate more fine-grained degradation increments to better characterize potential nonlinear trends.

The control experiment provided additional insights into the effects of cognitive load on auditory processing. When participants listened to hummed melodies or spoken sentences, overall sensitivity to degradation decreased (Fig. 2), suggesting that single-domain stimuli reduce cognitive load, allowing for more efficient auditory processing. In contrast, the dual-domain nature of song stimuli likely imposes greater cognitive load, thereby amplifying sensitivity to degradation. These findings highlight how cognitive complexity modulates auditory perception. Moreover, the observed asymmetry in cross-domain interference—where melody discrimination was hindered by the presence of speech content but not vice versa—suggests that melodic processing may be more vulnerable to interference. This effect is particularly relevant for tonal languages, where pitch is integral to linguistic meaning, and may reflect the heightened cognitive demands of tracking pitch information across overlapping domains.

However, the present experimental paradigm had several limitations. First, an alternative explanation for the lack of sensitivity to spectral degradation in the speech task may be the simplicity of the task itself, where detecting a difference in one or two words between song pairs is sufficient for accurate discrimination. While using more granular tasks such as sentence or melody comprehension and reproduction could better assess sensitivity, such tasks would be confounded by individual differences in music expertise, especially in the perception and reproduction of melodies. Second, the Mandarin song materials differed from the English and French materials in terms of linguistic content and performer, which may introduce acoustic confounds. Nonetheless, we controlled for syllable count, used the same musical notes, and found no significant differences in F0 across languages. Third, individual differences—such as musical and cognitive abilities—were not comprehensively matched or controlled across participant groups. Although we collected self-reported musical training histories and found that most participants had limited or no recent training, this does not fully rule out the influence of individual variability, particularly in melody tasks. Last, our conclusion is specific to Mandarin. Future research should explore this effect in other tonal languages with more complex tone distinctions, such as Cantonese and Thai, to confirm its generalizability.

In conclusion, this study provides new insights into how tonal language experience shapes sensitivity to spectrotemporal cues in speech and music perception. We first provide evidence that the processing of auditory spectral and temporal cues shows similar asymmetric patterns in a tonal language as in non-tonal languages, consolidating that spectral-temporal features are quasi-universal features across cultures (Albouy et al. 2024) that map onto brain mechanisms in a systematic way (Flinker et al. 2019; Albouy et al. 2020; Haiduk et al. 2024). While the broad perceptual mechanisms governing speech and music processing appear stable across languages, linguistic background contributes to a more nuanced understanding of domain-specific sensitivity in auditory processing. Mandarin speakers’ increased susceptibility to spectral degradation in melody perception underscores the impact of tonal language experience on pitch sensitivity. Moreover, the weakened STM sensitivity when perceiving single-domain stimuli and the unbalanced cross-domain interference between melody and speech processing further highlight the role of cognitive processes (such as attentional or working memory load) in modulating STM sensitivity in the auditory domain. Future research may further explore how these auditory processing patterns manifest across different tonal languages and investigate the potential applications of STM sensitivity in assessing and training auditory skills across diverse linguistic contexts.

ACKNOWLEDGMENTS

The authors thank Hou Chen, Xintong Jiang, and Laiyi Yu for creating the Mandarin songs.

OPEN PRACTICES

This manuscript qualifies for an Open Data badge and an Open Materials badge. The materials and data have been made publically available at https://osf.io/hsyvx/. More information about the Open Practices Badges can be found at https://journals.lww.com/ear-hearing/pages/default.aspx.

Supplementary Material

aud-47-506-s001.pdf ^{(2.1MB, pdf)}