Reverse Hierarchical Processing of Speech in Talker Identification

ABSTRACT

Spoken language contains critical information related to “what” is being said as well as “who” is talking. Listeners use various cues from speech to identify talkers. Talkers can be identified primarily based on acoustic cues, but access to higher‐level cues depends on the extent of language familiarity. Do listeners have greater processing costs when there is a greater number of cues available? Or do listeners leverage cues in a reverse hierarchical manner, using higher‐level cues more strategically to reduce processing costs? Here, we leveraged multiple methodological approaches, including pupillometry, analysis of the behavioral error pattern, and drift diffusion models (DDMs), to examine the dynamics of how learners identify novel talkers and accrue sensory evidence, balance cue usage, and manage processing costs during talker identity learning. Native English‐speaking adults learned to identify talkers in English and Mandarin, then were tested on recognizing them speaking new sentences, all while their pupillary responses, a proxy for processing cost, were recorded. Talker identification was more accurate and induced less pupil dilation in English relative to Mandarin. Analysis of error patterns showed that listeners relied more heavily on low‐level acoustic features in Mandarin than English, and DDMs revealed that higher evidence accumulation rates were associated with smaller pupil dilation. These findings suggest that, whenever available, listeners primarily use higher‐level abstract representations to identify talkers from the beginning of the learning process while relying on less efficient, lower‐level features of speech sounds in unfamiliar languages, consistent with the predictions of the reverse hierarchical framework.

Beyond replicating the well‐established language familiarity effect—better talker identification in the native (English) versus unfamiliar (Mandarin) language—we incorporated drift‐diffusion model, error pattern analysis, and pupillometry to understand the time course dynamics of talker learning. Even early in learning, native‐language identification showed reduced pupil‐indexed effort, less reliance on between‐talker acoustic similarity, and higher evidence accumulation rates. The persistent gap between languages supports reverse‐hierarchy theory: learners bypass lower‐level representations when higher‐level, abstract representations are available.

Abbreviations

ANOVA

analysis of variance

ASM

acoustic similarity matrix

DDM

drift diffusion model

GCA

growth curve analysis

HL

hearing level

ITI

intertrial interval

LFE

language familiarity effect

RHT

reverse hierarchy theory

RMS

root mean square

SPL

sound pressure level

1. Introduction

Human speech conveys both linguistic content (what is being said) and indexical information (who is speaking). Listeners are remarkably efficient at extracting both types of information simultaneously from the speech signal; they can recognize talkers from nonlinguistic content and comprehend linguistic content despite talker variability. However, this does not imply that the “what” and “who” information in speech are entirely dissociable. On the contrary, they are interdependent. Variability across talkers imposes processing costs on speech perception (Choi et al. 2018; Mullennix and Pisoni 1990), and talker identification is less accurate when the speaker uses an unfamiliar language. Native listeners thus have an advantage in identifying talkers, a phenomenon known as the language familiarity effect (LFE; Goggin et al. 1991; Thompson 1987).

Prior work has used multiple behavioral approaches to reach a consistent conclusion that talkers are more easily identified in a familiar language. Voice lineup studies demonstrated that people were better at identifying a voice from a set of voices in a familiar language than in an unfamiliar language (e.g., Goggin et al. 1991; Thompson 1987; Köster and Schiller 1997) and that people identified voices better when they spoke a familiar accent (other‐accent effect; e.g., Stevenage et al. 2012; Yu et al. 2021). Training studies, which track learning over time, similarly found LFE (Orena et al. 2015; McLaughlin et al. 2019; Perrachione and Wong 2007; Xie and Myers 2015; Bregman and Creel 2014). Through these studies, LFE has been robustly established across a range of languages including English, Spanish, French, German, Korean, and Mandarin.

What levels of linguistic representation drive the LFE when we learn new voices? Some studies have shown that phonological familiarity alone is sufficient or at least the primary factor that drives the effect of language familiarity (Fleming et al. 2014; Zarate et al. 2015; Yu et al. 2021). However, emerging evidence points to higher‐level lexical and semantic cues as key contributors of LFE (Bregman and Creel 2014; Xie and Myers 2015). Collectively, these findings from previous studies suggest that (i) while talker identification is possible to some extent in the absence of linguistic comprehension, listeners preferentially leverage higher‐level linguistic representations whenever accessible, rather than relying solely on acoustic cues of talkers, and that (ii) the use of higher‐level cues makes talker identification more accurate.

Nevertheless, behavioral findings alone provide limited insight into the time course and dynamics of talker learning. Here, we revisit LFE using a combination of physiological measures and behavioral modeling to understand how cue usage and processing costs in talker learning evolve over time. Such understanding can also allow us to adjudicate between contrastive models of perceptual learning that could scaffold talker learning: the reverse hierarchy theory (RHT; Ahissar and Hochstein 2004; Ahissar et al. 2009; Nahum et al. 2008) and the two‐stage perceptual learning model (Jiang et al. 2018; Serre et al. 2007).

RHT proposes that perceptual judgements are guided primarily by high‐level abstract representations unless access to those representations is blocked or attention is explicitly directed to lower‐level sensory features (Ahissar et al. 2009; Ahissar and Hochstein 2004; Nahum et al. 2008). In this view, talker learning in both native and non‐native languages is supported by the same underlying mechanism, with differences arising from the level of representation available early in learning. Within this framework, listeners have immediate access to higher‐level cues for native speech but not for non‐native language. By contrast, in the two‐stage perceptual model, learning begins with bottom‐up extraction of lower‐level features, which form the early scaffolding for later‐emerging, higher‐level categorical representations (Jiang et al. 2018; Serre et al. 2007). Thus, both frameworks posit that native and non‐native talker learning shares a common mechanism, but they differ in their accounts of whether learning begins from higher‐level abstract representations or from lower‐level sensory analysis.

Both models share the view that the representational level that drives learning shifts with expertise and context, but they make different predictions about the time course of perceptual learning. RHT would treat native‐ and non‐native‐language talker learning as outcomes of the same learning mechanism while making different predictions of early learning profiles depending on whether abstract representations are readily accessible: native‐language talker learning should be relatively efficient from the beginning, less effortful, and less dependent on bottom‐up acoustic cues, whereas unfamiliar‐language talker learning should initially be harder, more effortful, and more acoustically driven. With training, RHT predicts improvement in both languages but with a persistent gap in effort and acoustic cue reliance because higher‐level scaffolding remains limited in unfamiliar language. The two‐stage feedforward model, however, predicts a different trajectory of talker learning. Early learning is effortful in both native and unfamiliar languages because of the reliance on bottom‐up feature extraction; over time, talker identification in both languages will become more efficient and less effortful as more abstract category representations emerge, with differences between languages reflecting the rate of tuning rather than a qualitatively different strategy early in learning. While the two models do make different predictions about talker learning trajectory, we cannot adjudicate between the two models by using behavioral accuracy and response time measures alone.

In this study, we aimed to better understand the dynamics of LFE in talker learning by incorporating behavioral error patterns, pupillometry, and drift diffusion modeling (DDM). Native English listeners with no prior experience in Mandarin Chinese were trained to identify a set of native English‐speaking talkers and native Mandarin‐speaking talkers while their pupillary response was recorded.

Behavioral accuracy across listeners' native and non‐native languages is sufficient to establish LFE, but error patterns in the behavioral task can provide additional important insight into how listeners weight the usage of acoustic vs. higher‐level linguistic cues over the time course of talker learning. For example, by comparing the (dis)similarity patterns in talkers' low‐level acoustic features with the listeners' behavioral confusion patterns, Xiong et al. (2025) demonstrated that reliance on acoustic cues decreases not only with native‐language experience but also as the listener gets more familiar with the talker's accent. Using a similar approach as Xiong et al. (2025), we aimed to examine how acoustic cue reliance changes as the listeners learned to identify new talkers.

Pupillometry indexes processing cost and effort over the course of learning. Pupil dilation reflects arousal and effort during cognitive tasks (Beatty 1982) and provides a reliable marker of processing costs and the encoding and retrieval of task‐relevant information (Papesh et al. 2012). It has been widely used to assess cognitive load and listening effort across different levels of task difficulties (Winn et al. 2015; McHaney et al. 2021, 2024; McGarrigle et al. 2017; Borghini and Hazan 2018; Zekveld et al. 2013; Kuchinsky et al. 2016). Pupillary response also varies with subjective interpretations of stimuli, demonstrating the influence of higher‐level control factors (Naber and Nakayama 2013; Strauch et al. 2022). Because talker identification may recruit different processing levels depending on language familiarity, pupillometry can reveal how cue use shifts across languages and how those shifts affect processing cost.

DDMs take both trial accuracies and response times to model decisional processes during perceptual tasks (Ratcliff 1978; Ratcliff and McKoon 2008; Smith and Vickers 1988), allowing us to integrate these measures in the context of speed‐accuracy trade‐off. DDMs assume that the decision‐maker accumulates evidence for multiple possible decision options at varying rates (evidence accumulation rate), and that a decision is made when the accumulated evidence reaches a particular threshold for one of the options (decision threshold). Thus, within the framework of DDM, the evidence accumulation rate reflects the efficiency with which a decision‐maker attends to task‐relevant features of the stimuli (Kiani et al. 2008; Pereira et al. 2021; Smith and Vickers 1988). In the context of our study, this parameter is particularly relevant as we are interested not only in how quickly or accurately listeners identify talkers, but also in the specific cues they use to identify talkers, and how different cue usage patterns contribute to their effectiveness in talker identification. Because listeners can rely on cues of differing efficiency (more efficient higher‐level vs. less efficient lower‐level cues) depending on language familiarity, a higher evidence accumulation rate would imply greater reliance on higher‐level, abstract representations. DDM has not only been successful at describing various aspects of human perception and behavior (reviewed in Ratcliff et al. 2016), but also relates to baseline and phasic pupil dilation (de Gee et al. 2014; Dix and Li 2020; McHaney et al. 2024). The association between pupil dilation and decisional parameters is likely reflective of shared neural mechanisms involving the locus coeruleus‐norepinephrine system (Aston‐Jones and Cohen 2005; Gold and Shadlen 2000).

Taken together, our multimodal design allows us to quantify changes in cue use via error patterns, processing cost via pupillary response, and efficiency of information extraction via evidence accumulation rates. By tracking these measures over the course of talker learning in native and unfamiliar languages, we test whether early learning reflects immediate access to abstract representations (RHT) or initial bottom‐up feature extraction that subsequently tunes higher‐level mappings (two‐stage feedforward model). RHT predicts that, even in the early stage of learning, native‐language talker learning will show lower pupil‐indexed effort, higher evidence accumulation rate, and fewer acoustically driven confusions than unfamiliar‐language talker learning—and that these gaps will remain persistent throughout the course of training. In contrast, the two‐stage model predicts minimal early differences between languages in pupil‐indexed effort, evidence accumulation rate, and acoustic cue reliance; with training, both languages should exhibit reduced effort, increased evidence accumulation rate, and decreased reliance on acoustic cues, but the native language should show a steeper improvement rate, yielding growing gaps in the rate of change across measures.

2. Materials and Methods

2.1. Participants

24 Native English‐speaking adults were recruited (15 female, 9 male; ages 19–33 years; mean = 25.75 years, SD = 6.24 years). All participants were self‐reported native speakers of English with no history of language or hearing disorders and no prior experience with Mandarin Chinese. All participants passed the hearing screening test with pure tones of 250 Hz to 8 kHz hearing thresholds being lower than 25 dB HL. All participants gave written informed consent, and all study procedures were approved by the Institutional Review Board at Northwestern University.

2.2. Stimuli

We used audio recordings of 10 English sentences and 10 Mandarin sentences, spoken by 4 native English speakers and 4 native Mandarin speakers respectively (Bradlow, n.d.). Stimuli were normalized for RMS amplitude to 70 dB SPL in Praat (Boersma and Weenink 2023).

2.3. Task Procedure

The study was divided into two sessions separated by at least two hours, each of which contained one of the two stimulus languages—English or Mandarin. The order of the two languages was counterbalanced across participants. Auditory stimuli were presented through Sennheiser HD 280 Pro over‐ear headphones.

Pupillometry was recorded while participants completed a talker identification task. Participants' left eye pupil size was recorded at a sampling rate of 1000 Hz, using an Eyelink 1000 Plus Desktop Mount (SR Research; Ottawa, Ontario, Canada) with a chin and forehead rest for stabilization. The room lighting was set at a fixed, dim level for all participants to ensure that the luminance of the visual field was consistent. A nine‐point eye tracker calibration was performed before each language session started.

Each language session was composed of a training phase and a test phase. The training phase was composed of four blocks, and the test phase was one block. In both phases, each block contained 40 trials. Because the pupillary response unfolds slowly over time, we included a delay after each trial event to allow for changes in pupil size that were not contaminated by subsequent trial events (Koelewijn et al. 2018; McHaney et al. 2021, 2023; Zekveld et al. 2013; Figure 1A). To control for the effects of saccades and minimize pupil foreshortening (Hayes and Petrov 2016; Knapen et al. 2016), participants fixated on a cross in the center of the screen for a minimum of 500 ms at the beginning of each trial. After this fixation criterion was met, the sentence stimulus was presented. There was a four‐second delay from the offset of the stimulus to the response prompt on the screen in the form of “Which speaker?”, and participants had unlimited time to respond by pressing the number key (1, 2, 3, or 4) that corresponds to each of the four talkers in the language. Following the response, there was a two‐second delay before corrective feedback was given in the form of the text “Correct” or “Incorrect” displayed on the screen for two seconds. After the feedback, there was an intertrial interval (ITI) of one second.

FIGURE 1.

(A) Task procedure. Participants performed talker identification task in English and Mandarin, with a 4‐s delay between stimuli and response and a 2‐s ITI. Procedure shown in the figure is the structure of one trial in training phase; the corrective feedback and the 2‐s delay before the feedback were removed in test phase. (B) Error pattern analysis. The relationship between participants' error patterns and stimuli acoustic similarities were analyzed using Spearman's rank correlation.

In all the four training blocks, participants heard the same 10 sentences spoken by the four talkers (40 total stimuli), with every sentence stimulus presented once per block. In the test phase, participants heard 10 novel sentences spoken by the same four talkers, with each stimulus presented once in the test block. The order of the auditory stimuli was randomized within each block.

Manual drift correction was performed between each block by the experimenter to ensure high quality tracking of the pupil. Between blocks, participants were given the option to take a short break to prevent fatigue. Pupillary data from the four‐second period starting from the onset of the stimulus for each trial were used for further analysis.

2.4. Data Analysis

2.4.1. Behavioral Data Analysis

2.4.1.1. Accuracy

Accuracy was coded as 1 for correct and 0 for incorrect responses for each trial. Data were analyzed in R (version 4.3.2) using generalized linear mixed‐effects models implemented in the package lme4 (Bates et al. 2015). Separate models were estimated for the training phase and test phase. The model for the training phase included fixed factors of language, block and their interaction, random within‐participant slopes for language and block, and random intercepts for participants (Barr et al. 2013). The model for the test phase included a fixed factor of language, a random within‐participant slope for language, and random intercepts for participants. Contrast for the language factor was treatment‐coded with English as the reference level. Significance of fixed factors was determined in a Type III analysis of variance (ANOVA). Significant effects from the ANOVA were followed by testing specified contrasts on the terms in the linear mixed‐effects model. We adopted a significance criterion of α = 0.05, with p values in the mixed‐effects linear models based on the Satterthwaite approximation of the degrees of freedom using the lmerTest package (Kuznetsova et al. 2017).

2.4.1.2. Response Time

Response times were measured from the onset of the response prompt being displayed on the screen and were log‐transformed to ensure normal distribution. Outlier trials deviating from each participant's mean log response time in each block by more than 3 standard deviations were excluded from the analysis (1.76% of total trials).

Data were analyzed using the same combination of software and packages as described in the section above (2.4.1.1), with the exception that linear mixed‐effects models were used instead of generalized linear mixed‐effects models. As done with accuracy data, separate models were estimated for training and test phases: the model for training phase included fixed factors of language and block while the model for text phase included the fixed factor of language only, and both models included random within‐participant slopes for language and random intercepts for participants.

2.4.2. Drift Diffusion Modeling

We used the Bayesian implementation of the drift‐diffusion model (DDM) developed by Paulon et al. (2021) to compare the dynamics of the decision processes in the two language tasks (English, Mandarin) as a function of training. For every combination of response (d) and stimulus category (s), the DDM estimates an evidence accumulation rate parameter (μ _d,s ) and a decision threshold parameter (b _d,s ) that were fit. Evidence accumulation rate reflects how efficiently the decision‐maker extracts relevant information from the stimulus. Lower evidence accumulation rates are common for more difficult tasks when extracting relevant information from the stimulus is more challenging. The decision threshold parameter represents the amount of evidence that the decision‐maker needs to accumulate in order to make the decision, hence reflecting the decision‐maker's response caution (Bogacz et al. 2010). Higher decision thresholds suggest that the listener was more cautious to make a response, favouring accuracy over speed when making decisions. Importantly, this DDM allowed for the evidence accumulation rate and decision threshold parameters to vary between individuals, to control for the substantial variability across participants. The DDM also allowed for evidence accumulation rate and decision threshold parameters to change slowly over time across learning blocks. This helps to account for the changes in decision‐making processes during learning. Additionally, for each stimulus category (talker), an offset parameter was fit, which represents the time taken by stimulus encoding and motor response—actions that are not considered to be directly relevant to the decision process.

We estimated the DDM parameters separately for the two language conditions (English and Mandarin), as the participants came in for two separate sessions to complete the two language conditions. DDMs were run using the lddmm package in R (Paulon and Sarkar 2023). We used a Bayesian framework that assigned priors to the parameters and relied on samples drawn from the posterior using a Markov chain Monte Carlo algorithm for estimation and inference. Prior to estimating the DDMs, the top and bottom 1% of trials based on response times were excluded as outliers to improve the model fit. Markov‐chain Monte Carlo simulations were used to run 6000 iterations, with the initial 2000 samples discarded as burn‐in. The remaining iterations were thinned in intervals of 5 to reduce autocorrelation.

Consistent with previous auditory learning studies which showed that gradual improvement in correct categorical decisions characterizes learning (Roark et al. 2021, 2024), we focused on the DDM parameters associated with correct trials only (i.e., μ _d,s and b _d,s where d = s). Also, we included data only from the training phase and not the test phase because we were specifically interested in how the process of learning may impact the decisional strategies underlying talker learning in the two languages. We examined the posterior probabilities of evidence accumulation rate and decision threshold parameters for each language and each block. Posterior means are reported as point estimates and 95% pointwise credible intervals were used to assess uncertainty. Differences across blocks were inferred when the 95% pointwise credible intervals were nonoverlapping.

2.4.3. Pupillometry Data Analysis

2.4.3.1. Preprocessing

We analyzed pupillary data from −500 to 4000 ms from the onset of the speech stimulus. These pupillary data were preprocessed to remove noise (Winn et al. 2015; Zekveld and Kramer 2014). First, the data were downsampled to 50 Hz (Wierda et al. 2012), and any trial with more than 15% of its samples detected as blinks was removed (12% of total trials rejected; McMahon et al. 2016). Missing samples due to blinks and saccades were linearly interpolated to 120 ms before and after the blink. For each trial, the average pupil size in the 500 ms prior to the onset of the stimulus was used as the baseline pupil size, and the following pupil size after the stimulus onset was normalized against the baseline (Peysakhovich et al. 2015); the outcome variable reported in the Results section is the proportion change in pupil size relative to the baseline.

2.4.3.2. Growth Curve Analysis

Pupil responses within the time window from 0 to 4000 ms time‐locked to the speech stimulus onset were averaged across trials for each block for each participant and were analyzed using growth curve analysis (GCA; Mirman 2014). We used GCA for analyzing the pupillary responses because it provides a statistical approach for modeling changes in the timing and shape of the pupillary response over time, which does not follow a linear trajectory (Winn et al. 2015, 2018).

A GCA was fit to model the interactions between first‐, second‐, third‐, and fourth‐order orthogonal polynomials and the independent variables of interest. The fourth‐order polynomial was chosen to model the pupillary response because it was consistent with the trajectory of the data and provided a better fit than a model that included only first‐, second‐, and third‐order polynomials. This fourth‐order model uses five parameters to map the trajectory of the pupillary response: the intercept reflects the overall change in the pupillary response over the entire time window and can be interpreted as the mean change in pupil size over time. The linear term (ot1) represents the slope of the pupillary response over time. If the linear term is positive, this reflects the rate of dilation, while a negative linear term reflects the rate of constriction. The quadratic term (ot2) reflects the curvature of the pupil response. A larger, negative quadratic term suggests a more parabolic shape and values closer to zero reflect a more linear shape. The cubic and quartic terms (ot3, ot4) reflect the extent to which secondary and tertiary inflection points occur in the pupil response, respectively (Kuchinsky et al. 2014; Mirman 2014). The absolute value of each term reflects the strength of the pupil response, while the polarity of the term reflects the direction of the response.

GCAs were conducted using the lme4 package (Bates et al. 2015) with log‐likelihood maximization using the BOBYQA optimizer for convergence (Mirman 2014), and p‐values were estimated using the lmerTest package (Kuznetsova et al. 2017). We first estimated a GCA to examine the effect of language and training block. The model included fixed effects of each growth curve time term (ot1‐ot4), language, block, and all two‐ and three‐way interactions between time terms, language, and block. The random effect structure included a random intercept of participant and random slopes of participant on each time term.

We also estimated separate GCAs to examine how the DDM parameters of decision‐making were reflected in the pupillary response during listening. For this analysis, separate models were estimated for English and Mandarin conditions, only for trials that were categorized correctly. The models included fixed effects of the DDM parameter (evidence accumulation rate) on all time terms and all two‐way interactions between the DDM parameter and time terms. The GCAs also included a random intercept of participant and random slopes of participant on each time term.

2.4.4. Error Pattern Analysis

We used an error pattern analysis method similar to the approach by Xiong et al. (2025). Behavioral confusion matrices were constructed for each participant, for each block in each language, such that each cell in the matrix represented the proportion of hearing a talker and identifying them as each talker (e.g., out of all the trials where the participant heard Talker 3, the proportion of identifying them as Talker 4). For the purpose of this analysis, we excluded the data on the diagonals of these matrices as we aimed to analyze the error patterns only.

Two separate acoustic similarity matrices (ASMs) were constructed for English talkers and Mandarin talkers. To compute the similarities, six different acoustic measures were taken into account: mean F0, range of F0, speech rate, jitter, harmonicity, and formant dispersion. These features were selected following a previous study on perceptual dissimilarity judgement of voices (Perrachione et al. 2019). Using these features, dissimilarity matrices were first computed based on the Euclidean distances between talkers using the Python rsatoolbox library (https://rsatoolbox.readthedocs.io). ASMs were created by multiplying −1 to the dissimilarity matrices.

To ensure that the acoustical variance was comparable between the English and Mandarin speech samples used in the study, we ran an analysis comparing the variance of speech samples between the two languages. A Mann–Whitney–Wilcoxon test on the Euclidean distances between talkers in each language showed that there was no significant difference in acoustical variance of the speech stimuli between the two languages (W = 17, p > 0.93).

After the behavioral confusion matrices and the stimuli ASMs were constructed, we ran a correlational analysis between them using Spearman's rank correlation (Figure 1B). The Spearman's correlation coefficients were compared between the two language conditions. Stronger positive correlations would indicate that the error patterns follow acoustic similarities of the voices more closely.

3. Results

3.1. Behavioral Results

3.1.1. More Accurate Talker Identification in English Than Mandarin

We examined whether talker identification accuracy was different in the English and Mandarin conditions and whether it improved over the blocks in the training phase (Figure 2A). There was a significant main effect of block (χ ² = 44.78, p < 0.001) showing that participants learned to identify talkers over the course of training. We observed a significant main effect of language such that participants identified talkers in English more accurately than in Mandarin (χ ² = 353.61, p < 0.001), replicating LFE shown by previous studies. We also observed a significant interaction between language and block (χ ² = 23.55, p < 0.001), suggesting different magnitudes of learning across training blocks between the two languages. The specified contrast terms in the model revealed that the effect of block on Mandarin was smaller than on English (β = −0.29, s.e. = 0.059, z = −4.85, p < 0.001), showing that the effect of training was greater in English than Mandarin. The pattern of higher accuracy in English compared to Mandarin persisted in the test phase (χ ² = 125.99, p < 0.001).

FIGURE 2.

(A) Accuracies and (B) log (response time (ms)) of talker identification in each block in each language. Points and error bars indicate mean ± SEM. (C) Evidence accumulation rates and (D) decision thresholds estimated for drift diffusion models, in arbitrary units (a.u.). Points and error bars indicate mean ±95% credible intervals. (E) Mean ± SEM Spearman's correlation rank coefficient between each participant's behavioral confusion matrix and stimuli acoustic similarity matrix in each language. Higher coefficient indicates higher reliance on acoustic features when identifying talkers. In all panels, thin, semi‐transparent lines show individual participant data.

3.1.2. No Llanguage Difference in Response Time

We observed a significant effect of block on response time in the training phase such that the response times became faster with more training (χ ² = 13.56, p < 0.001; Figure 2B). There was no significant effect of language on response times in the training phase (χ ² = 0.33, p = 0.56) or interaction between block and language (χ ² = 0.98, p = 0.32). The response times were not significantly different between English and Mandarin in the test phase (χ ² = 3.14, p = 0.076).

3.2. DDM Results

3.2.1. Evidence Accumulation Rate Is Higher in English Than Mandarin

Overall, participants had higher evidence accumulation rates across all blocks for English sentences relative to Mandarin sentences. In English, the evidence accumulation rate increased from block 1 to block 2 and plateaued for the rest of the training phase (Figure 2C). For Mandarin, the evidence accumulation rate increased from block 1 to block 3 and plateaued afterwards. This suggests that listeners became more efficient at extracting relevant information from stimuli for decision‐making faster in English than in Mandarin.

3.2.2. No Difference in Decision Thresholds Between Languages or Between Blocks

Decision thresholds remained stable throughout the training phase for both English and Mandarin and were not different between language conditions (Figure 2D). This indicates that listeners were just as cautious to make a response in the English condition as in the Mandarin condition.

3.3. Greater Reliance on Acoustic Features in Mandarin Than English

We analyzed the association between behavioral error patterns and the acoustic similarities between talkers' voices in order to assess the listeners' reliance on talker‐specific acoustic features when identifying talkers. We observed a stronger positive correlation between the behavioral error (e.g., likelihood of hearing Talker 1 and identifying them as Talker 3) and the acoustic similarity (e.g., how similar Talker 1 and Taker 3's acoustics are) in Mandarin than in English (Figure 2E). These results reveal that listeners' identification of talkers more closely matched the acoustic similarities between the talkers in the unknown language compared to the native language.

3.4. Pupillometry

3.4.1. Greater Pupil Dilation in Mandarin Than English

The average pattern of pupillary response in each language for each block is illustrated in Figure 3. Growth curve analysis (GCA) showed a significant effect of language on overall pupil dilation in the first block of the training phase, with increased pupil diameter in Mandarin than in English (p < 0.001; Table 1). Linear and quadratic polynomial terms contributed significantly to the difference between the two languages (ps < 0.001), such that the rate of overall dilation was faster in Mandarin than English, and the primary curvature was stronger in Mandarin than English. There was no significant effect of language on secondary and tertiary curvatures of pupil dilation (ps > 0.60). These results reveal the effect of language on the pupil dilation during stimulus presentation rather than post‐stimulus processing, as the quadratic polynomial term is more representative of the initial increase of pupil diameter.

FIGURE 3.

Changes in pupil dilation across time in each block, showing increased processing cost in Mandarin than English. Each participant's pupil diameter was normalized to their pre‐stimulus baseline pupil diameter. The shaded region indicates standard error.

TABLE 1.

Fixed effects estimated for growth curve models on pupillary responses to study the effect of language and block. ^a

Contrast	β	s.e.	t	p
(Intercept)	11.5	0.87	13.13	< 0.001***
ot1	5.67	6.34	8.94	< 0.001***
ot2	−82.3	5.12	−16.08	< 0.001***
ot3	−0.23	3.14	−0.072	0.94
ot4	24.5	2.36	10.39	< 0.001***
Language (Mandarin vs. English)	1.80	0.14	12.94	< 0.001***
ot1 × language	12.6	1.98	6.39	< 0.001***
ot2 × language	−8.56	1.98	−4.33	< 0.001***
ot3 × language	1.02	1.98	0.51	0.61
ot4 × language	0.64	1.98	0.32	0.75
block	0.19	0.036	5.31	< 0.001***
ot1 × block	−0.52	0.510	−1.01	0.31
ot2 × block	2.81	0.510	5.51	< 0.001***
ot3 × block	2.82	0.510	5.54	< 0.001***
ot4 × block	−1.47	0.510	−2.88	< 0.01**
language × block	−0.21	0.051	−4.05	< 0.001***
ot1 × language × block	−0.66	0.721	−0.91	0.36
ot2 × language × block	1.23	0.721	1.7	0.089
ot3 × language × block	−3.21	0.721	−4.45	< 0.001***
ot4 × language × block	0.0030	0.721	0.004	1.00
Test phase
(Intercept)	11.5	1.21	9.44	< 0.001***
ot1	33.0	6.57	5.02	< 0.001***
ot2	−71.9	5.97	−12.04	< 0.001***
ot3	17.1	3.13	5.46	< 0.001***
ot4	14.5	2.58	5.62	< 0.001***
language (Mandarin vs. English)	1.61	0.078	20.61	< 0.001***
ot1 × language	22.7	1.11	20.50	< 0.001***
ot2 × language	−0.0049	1.11	−0.004	1.0
ot3 × language	−11.45	1.11	−13.11	< 0.001***
ot4 × language	−0.090	1.11	−0.81	0.42

^a The GCA model in R notation: lmer(pupil ~ (ot1 + ot2 + ot3 + ot4)*language*block + (ot1 + ot2 + ot3 + ot4 | participant), control = lmerControl(optimizer = “bobyqa”), data = gca.data, REML = FALSE).

^** p < 0.01.

^*** p < 0.001.

There was a significant effect of block on the intercept, quadratic, cubic, and quartic polynomial terms in English (ps < 0.01), showing that more training led to a larger change in the pupillary response over time, flatter primary and tertiary inflection points, and a sharper secondary inflection point. There was also a significant interaction effect of language and block on the intercept and cubic polynomial term (p < 0.001), suggesting that over the course of the training, the effect of language difference on the overall pupillary response and the curvature at the peak of the pupillary response decreased.

3.4.2. Greater Evidence Accumulation Rate Associated With Reduced Pupil Dilation

We examined the extent to which the decisional processes were reflected in the pupillary response by estimating GCAs within each language condition separately on average pupillary responses as a function of evidence accumulation rates (Table 2). In both English and Mandarin, a greater evidence accumulation rate was associated with reduced overall pupil dilation, a slower rate of dilation, and flatter primary and tertiary curvatures (ps < 0.01). Regardless of the type of evidence being accumulated in the two languages, more efficient extraction of evidence from the auditory stimuli resulted in reduced processing costs.

TABLE 2.

Fixed effects estimated for growth curve models on pupillary responses fitted separately for English and Mandarin, to study the effect of evidence accumulation rates (EA Rates). ^a

Contrast	β	s.e.	t	p
English
(Intercept)	15.61	1.01	15.50	< 0.001***
ot1	73.76	8.14	9.06	< 0.001***
ot2	−95.47	5.41	−17.64	< 0.001***
ot3	7.55	3.89	1.94	0.056
ot4	26.27	3.01	8.74	< 0.001***
EA Rate	−2.46	0.12	−19.11	< 0.001***
ot1 × EA Rate	−14.20	1.76	−8.04	< 0.001***
ot2 × EA Rate	13.19	1.68	7.86	< 0.001***
ot3 × EA Rate	0.47	1.58	0.30	0.77
ot4 × EA Rate	−43.80	1.39	−2.74	< 0.01**
Mandarin
(Intercept)	13.64	1.16	11.80	< 0.001***
ot1	71.95	7.47	9.63	< 0.001***
ot2	−92.18	6.35	−14.52	< 0.001***
ot3	2.15	4.19	−0.51	0.61
ot4	26.15	3.08	8.49	< 0.001***
EA Rate	−0.66	0.15	−4.52	< 0.001***
ot1 × EA Rate	−12.81	2.05	−6.26	< 0.001***
ot2 × EA Rate	13.95	1.93	7.22	< 0.001***
ot3 × EA Rate	0.72	1.97	0.36	0.72
ot4 × EA Rate	−5.24	1.71	−3.06	< 0.01**

^a The GCA model for each {language} condition in R notation: lmer(pupil ~ (ot1 + ot2 + ot3 + ot4)*EA_Rate + (ot1 + ot2 + ot3 + ot4 | participant), control = lmerControl(optimizer = “bobyqa”), data = {language}.gca.ddm.data, REML = FALSE).

^** p < 0.01.

^*** p < 0.001.

4. Discussion

We investigated the dynamics of talker identification by using two language contexts that varied the richness and types of speech cues available to the listener. Extending upon the traditional approach of measuring talker identification accuracies under different language conditions, we derived decisional processes using drift diffusion models (DDMs), analyzed the listeners' reliance on acoustic features of speech via behavior‐acoustic correlation analysis, and measured listeners' pupillary response as they completed the talker identification task. Consistent with prior work, listeners identified talkers more accurately in their native language than in a language they did not know. In the last training block and the test block, their accuracy was better than the initial block of training in both English and Mandarin, but the difference between the two languages was still significant. We observed an overall increase in pupil dilation when listeners were identifying talkers in a non‐native language compared to the native language, both during the training phase and the testing phase. Combined with the behavioral‐acoustic analyses, the differences in processing cost and accuracy between identifying talkers in a native versus non‐native language likely arise from listeners relying more heavily on bottom‐up acoustic cues when processing speech in a non‐native language, whereas they depend more on top‐down language‐based cues in their native language.

4.1. Reliance on Bottom‐Up Processing of Acoustic Features in Talker Identification in an Unfamiliar Language

Our analysis of acoustic measures of talkers' voices and participants' error patterns reveals that listeners rely more heavily on talker‐specific acoustics in an unfamiliar language than in their native language, replicating the previous finding (Xiong et al. 2025). As listeners have only limited phonological cues and no higher‐order linguistic knowledge in an unfamiliar language, they can only process the low‐level acoustic features of the talkers who speak an unfamiliar language. As a result, the behavioral error patterns were more closely associated with the acoustic similarities of the stimuli in listeners' unfamiliar language than in their native language.

Even at the very beginning of the training, listeners already showed a significant difference in acoustic cue reliance between native and unfamiliar languages. This reveals that, as soon as the higher‐level abstract representations were available (i.e., native language), listeners were relying on them instead of extracting lower‐level acoustic features, consistent with the predictions of RHT (Ahissar et al. 2009).

When listeners hear different talkers and judge how dissimilar the talkers are, their perceptual dissimilarity judgement does not differ among listeners with different native languages (Perrachione et al. 2019). This shows that listeners' native language experience does not affect perceptual sensitivity to talker acoustics, enabling them to process and compare the acoustic features of different talkers regardless of the language spoken. However, when asked to learn to identify the talkers rather than to judge how dissimilar the pairs of talkers are, listeners in our study drew from more than just the acoustics of the stimuli when higher‐level linguistic knowledge was available.

A potential limitation of this analysis is that there are fewer errors in English than Mandarin, thus making the range of error proportions in English smaller than in Mandarin. However, the fact that the behavior‐acoustic correlation in Mandarin remained strong even as the accuracies improved in the last training block shows that listeners consistently relied heavily on the acoustic features of the voices in a language that they do not understand. In the absence of top‐down guidance, learning did not reach beyond the bottom‐up processing of acoustic features.

4.2. Greater Pupil Dilation and Lower Evidence Accumulation Rate When Identifying Talkers in an Unfamiliar Language

We observed greater pupil dilation in the non‐native than the native language at the beginning of the training phase, and the pupil dilation difference between the two languages decreased as the training progressed through later blocks.

Not only did we observe increased pupil dilation associated with talker identification in the unfamiliar language relative to the native language, but we also observed that evidence accumulation rates in the unfamiliar language were lower than in the native language and did not increase beyond the third block. Rather than a gradual increase in evidence accumulation rate, which would be predicted by the two‐stage feedforward model, the increase in efficiency of evidence extraction in the unfamiliar language seemed to plateau. Listeners were able to access more abstract levels of speech in their native language, so they readily gained efficiency in accumulating relevant information for talker identification, and the gap between the two languages persisted, more consistent with the predictions of RHT.

Our observations are in line with previous neuroimaging studies on processing native vs. non‐native languages. Performing tasks in one's non‐native language has been associated with increased processing demands (Berken et al. 2015; Borghini and Hazan 2018), and identifying talkers in the native language produced weaker response relative to non‐native languages in the superior temporal cortex, supramarginal gyrus, and inferior frontal gyrus (Meng et al. 2024). However, greater proficiency with the non‐native language has been associated with recruitment of a more “classical” language network (Chee et al. 2001; Leonard et al. 2011) and more left‐lateralized brain regions, such that the neural recruitment for non‐native language processing resembles native language processing. In our study, the pupillary response differences between the two languages decreased as the listeners became more familiar with identifying talkers in the non‐native language. Future neuroimaging studies focusing on the dynamic talker learning trajectory will be able to further adjudicate between RHT and two‐stage feedforward model, as the two models make different predictions on the primary brain region that would support talker learning: RHT would predict that higher‐order language area such as inferior frontal gyrus would drive talker learning in the native language, whereas the auditory cortex would drive non‐native talker learning; two‐stage model would predict auditory cortex tuning before higher‐order representations emerge in the brain for both languages but with better representational fidelity for the native language.

Our analyses, integrating pupillometry data with evidence accumulation rates, revealed that a higher rate of evidence accumulation corresponded to smaller overall pupil dilation in both languages. This suggests that greater efficiency in extracting relevant information from stimuli leads to reduced cognitive processing demands, as reflected by decreased pupil dilation. Nonetheless, as discussed above, in their native language, listeners accumulated information in a way that enables higher‐level processing of the speech stimuli; in their unfamiliar language, listeners could only primarily gather information on the acoustic features of each talker. While these talker‐specific acoustic features may help with learning outcomes, processing more acoustic features and maintaining the information in working memory incur more processing costs than using more abstract representations, which is reflected in the pupillary response difference between native and unfamiliar languages. Taken together, our findings indicate that although the underlying mechanism for talker identification is consistent across native and unfamiliar languages, differences in the type and quality of evidence obtained in non‐native language impede the efficiency of evidence accumulation, thereby increasing the processing cost in non‐native language contexts.

A parsimonious interpretation of the results is that talker identification can be primarily carried out operating on the higher‐level, abstract representations of the speech stimuli, rather than low‐level sensory features, in line with reverse hierarchy theory (Ahissar and Hochstein 2004). Although previous studies have identified some key acoustic features that enable voice similarity judgements (Baumann and Belin 2010; Perrachione et al. 2019), learning to identify talkers based on their speech production inevitably involves processing of the linguistic information, as long as the listener is very familiar with the language. Language familiarity affords listeners the ability to perceive speech signals in relation to higher‐level, more abstract representations rather than granular processing of acoustic features. Primarily accessing the high‐level representations makes perception more efficient, whereas impaired ability to understand the complex relations between features leads to increased processing cost and more pupil dilation (Naber and Nakayama 2013). Thus, the pupil dilation increase in non‐native talker identification may reflect disrupted high‐level processing of the unfamiliar‐language speech stimuli. Taken together, the results reveal that talker identification in one's native language does not require the sequential processing of phonological, lexical, and semantic levels of speech. Instead, listeners can directly access abstract higher‐level cues, thus not incurring additional processing costs that would be predicted if the listeners were processing all different levels in a hierarchical, sequential manner. In fact, this ability to process higher‐level information in one's native language allows for a more efficient cognitive resource allocation by minimizing the processing of lower‐level acoustic features. While there has been evidence of brain regions and mechanisms that are specifically tuned to human voice processing (Belin et al. 2000; Latinus and Belin 2011), our findings demonstrate a strong interdependence between speech and voice perception for vocal speech signal.

4.3. Limitations and Future Directions

Our findings suggest that native‐language talker learning was supported primarily by higher‐level abstract representations throughout the training period. However, the relatively short duration of the study limits our ability to infer the longer‐term trajectory of learning. Within the RHT framework, enhanced sensitivity to lower‐level sensory features may emerge only after perceptual mastery has been achieved (Hochstein and Ahissar 2002; Shamma 2008; Reetzke et al. 2018). Although reliance on acoustic cues, as indexed by behavioral error patterns in this study, becomes difficult to assess once every listener's performance reaches ceiling, we would expect overtraining to increase listeners' sensitivity to lower‐level stimulus features as indexed by other measures such as frequency‐following response (e.g., Reetzke et al. 2018).

By contrast, our findings suggest that listeners relied relatively heavily on lower‐level acoustic information during non‐native‐language talker learning, and performance had not reached ceiling by the end of training. Although reliance on acoustic cues appeared to plateau in the later blocks, it remains unclear whether this pattern reflects a true asymptote or simply the limited portion of the learning curve captured within the present training window. Under an RHT account, because higher‐level abstract representations remain relatively inaccessible in the non‐native language, further improvement may increasingly depend on greater sensitivity to fine‐grained, talker‐specific acoustic detail that generalizes across sentences, thereby leading to continued reliance on lower‐level cues. An alternative possibility is that, with continued training on a restricted set of sentences, listeners would increasingly learn item‐specific acoustic patterns tied to the trained materials, rather than cues that support generalization to novel utterances. Future work that tracks pupillary responses and learning trajectories over a longer training period would help clarify how behavioral and physiological indices of learning evolve over time and how different representational levels contribute to talker learning in native and unfamiliar languages.

5. Conclusion

Talker identification in an unfamiliar language is less accurate than in one's native language. The pattern of greater pupil dilation response and heavier reliance on lower‐level acoustic features in unfamiliar language reveals that listeners resort to different strategies when identifying talkers in native vs. unfamiliar language: in their native language, they access the more abstract linguistic representation to identify talkers, whereas in an unfamiliar language, they have to rely on bottom‐up processing of acoustic features of talkers. Our findings are suggestive of reverse hierarchical processing of speech in talker identification, further revealing how linguistic processing and voice processing in speech perception are intertwined despite serving functionally different purposes.

Author Contributions

Ja Young Choi: conceptualization, software, data curation, investigation, formal analysis, writing – original draft, writing – review and editing. Shengyue Xiong: data curation, investigation, formal analysis, writing – review and editing. Jacie R. McHaney: software, writing – review and editing. Bharath Chandrasekaran: funding, conceptualization, writing – review and editing.

Funding

This work was supported by the National Science Foundation (DRL2409652).

Conflicts of Interest

The authors declare no conflicts of interest.

Choi, J. Y. , Xiong S., McHaney J. R., and Chandrasekaran B.. 2026. “Reverse Hierarchical Processing of Speech in Talker Identification.” European Journal of Neuroscience 63, no. 8: e70520. 10.1111/ejn.70520.

Data Availability Statement

Data and analysis scripts are available at https://osf.io/2m8nx/?view_only=6f0fe21cf1614b1ba1338e341110eac9.