Early phonetic learning without phonetic categories: Insights from large-scale simulations on realistic input

Significance

Infants become attuned to the sounds of their native language(s) before they even speak. Hypotheses about what is being learned by infants have traditionally driven researchers’ attempts to understand this surprising phenomenon. Here, we propose to start, instead, from hypotheses about how infants might learn. To implement this mechanism-driven approach, we introduce a quantitative modeling framework based on large-scale simulation of the learning process on realistic input. It allows learning mechanisms to be systematically linked to testable predictions regarding infants’ attunement to their native language(s). Through this framework, we obtain evidence for an account of infants’ attunement that challenges established theories about what infants are learning.

Abstract

Before they even speak, infants become attuned to the sounds of the language(s) they hear, processing native phonetic contrasts more easily than nonnative ones. For example, between 6 to 8 mo and 10 to 12 mo, infants learning American English get better at distinguishing English and [l], as in “rock” vs. “lock,” relative to infants learning Japanese. Influential accounts of this early phonetic learning phenomenon initially proposed that infants group sounds into native vowel- and consonant-like phonetic categories—like and [l] in English—through a statistical clustering mechanism dubbed “distributional learning.” The feasibility of this mechanism for learning phonetic categories has been challenged, however. Here, we demonstrate that a distributional learning algorithm operating on naturalistic speech can predict early phonetic learning, as observed in Japanese and American English infants, suggesting that infants might learn through distributional learning after all. We further show, however, that, contrary to the original distributional learning proposal, our model learns units too brief and too fine-grained acoustically to correspond to phonetic categories. This challenges the influential idea that what infants learn are phonetic categories. More broadly, our work introduces a mechanism-driven approach to the study of early phonetic learning, together with a quantitative modeling framework that can handle realistic input. This allows accounts of early phonetic learning to be linked to concrete, systematic predictions regarding infants’ attunement.

Adults have difficulties perceiving consonants and vowels of foreign languages accurately (1). For example, native Japanese listeners often confuse American English and [l] (as in “rock” vs. “lock”) (2, 3), and native American English listeners often confuse French [u] and [y] (as in “roue,” wheel, vs. “rue,” street) (4). This phenomenon is pervasive (5) and persistent: Even extensive, dedicated training can fail to eradicate these difficulties (6–8). The main proposed explanations for this effect revolve around the idea that adult speech perception involves a “native filter”: an automatic, involuntary, and not very plastic mapping of each incoming sound, foreign or not, onto native phonetic categories—i.e., the vowels and consonants of the native language (9–13). American English and [l], for example, would be confused by Japanese listeners because their productions can be seen as possible realizations of the same Japanese consonant, giving rise to similar percepts after passing through the “native Japanese filter.”

Surprisingly, these patterns of perceptual confusion arise very early during language acquisition. Infants learning American English distinguish and [l] more easily than infants learning Japanese before they even utter their first word (14). Dozens of other instances of such early phonetic learning have been documented, whereby cross-linguistic confusion patterns matching those of adults emerge during the first year of life (15–17). These observations naturally led to the assumption that the same mechanism thought to be responsible for adults’ perception might be at work in infants—i.e., foreign sounds are being mapped onto native phonetic categories. This assumption—which we will refer to as the phonetic category hypothesis—is at the core of the most influential theoretical accounts of early phonetic learning (9, 18–21).

The notion of phonetic category plays an important role throughout the paper, and so requires further definition. It has been used in the literature exclusively to refer to vowel- or consonant-like units. What that means varies to some extent between authors, but there are at least two constant, defining characteristics (22). First, phonetic categories have the characteristic size/duration of a vowel or consonant, i.e., the size of a phoneme, the “smallest distinctive unit within the structure of a given language” (1, 23). This can be contrasted with larger units like syllables or words and smaller units like speech segments corresponding to a single period of vocal fold vibration in a vowel. Second, phonetic categories—although they may be less abstract than phonemes* —retain a degree of abstractness and never refer to a single acoustic exemplar. For example, we would expect a given vowel or consonant in the middle of a word repeated multiple times by the same speaker to be consistently realized as the same phonetic category, despite some acoustic variation across repetitions. Finally, an added characteristic in the context of early phonetic learning is that phonetic categories are defined relative to a language. What might count as exemplars from separate phonetic categories for one language might belong to the same category in another.

The phonetic category hypothesis—that infants learn to process speech in terms of the phonetic categories of their native language—raises a question. How can infants learn about these phonetic categories so early? The most influential proposal in the literature has been that infants form phonetic categories by grouping the sounds they hear on the basis of how they are distributed in a universal (i.e., language-independent) perceptual space, a statistical clustering process dubbed “distributional learning” (24–27).

Serious concerns have been raised regarding the feasibility of this proposal, however (28, 29). Existing phonetic category accounts of early phonetic learning assume that speech is being represented phonetic segment by phonetic segment—i.e., for each vowel and consonant separately—along a set of language-independent phonetic dimensions (9, 19, 20).^† Whether it is possible for infants to form such a representation in a way that would enable distributional learning of phonetic categories is questionable, for at least two reasons. First, there is a lack of acoustic–phonetic invariance (30–32): There is not a simple mapping from speech in an arbitrary language to an underlying set of universal phonetic dimensions that could act as reliable cues to phonetic categories. Second, phonetic category segmentation—finding reliable language-independent cues to boundaries between phonetic segments (i.e., individual vowels and consonants)—is a hard problem (30). It is clear that finding a solution to these problems for a given language is ultimately feasible, as literate adults readily solve them for their native language. Assuming that infants are able to solve them from birth in a language-universal fashion is a much stronger hypothesis, however, with little empirical support.

Evidence from modeling studies reinforces these concerns. Initial modeling work investigating the feasibility of learning phonetic categories through distributional learning sidestepped the lack-of-invariance and phonetic category segmentation problems by focusing on drastically simplified learning conditions (33–38), but subsequent studies considering more realistic variability have failed to learn phonetic categories accurately (29, 39–43) (SI Appendix, Discussion 1).

These results have largely been interpreted as a challenge to the idea that distributional learning is how infants learn phonetic categories. Additional learning mechanisms tapping into other sources of information plausibly available to infants have been proposed (26, 28, 29, 39–44), but existing feasibility results for such complementary mechanisms still assume that the phonetic category segmentation problem has somehow been solved and do not consider the full variability of natural speech (29, 36, 39–43, 45). Attempts to extend them to more realistic learning conditions have failed (46, 47) (SI Appendix, Discussion 1).

Here, we propose a different interpretation for the observed difficulty in forming phonetic categories through distributional learning: It might indicate that what infants learn are not phonetic categories. We are not aware of empirical results establishing that infants learn phonetic categories, and, indeed, the phonetic category hypothesis is not universally accepted. Some of the earliest accounts of early phonetic learning were based on syllable-level categories and/or on continuous representations without any explicit category representations^‡ (48–51). Although they appear to have largely fallen out of favor, we know of no empirical findings refuting them.

We present evidence in favor of this alternative interpretation, first by showing that a distributional learning mechanism applied to raw, unsegmented, unlabeled continuous speech signal predicts early phonetic learning as observed in American English and Japanese-learning infants—thereby providing a realistic proof of feasibility for the proposed account of early phonetic learning. We then show that the speech units learned through this mechanism are too brief and too acoustically variable to correspond to phonetic categories.

We rely on two key innovations. First, whereas previous studies followed an outcome-driven approach to the study of early phonetic learning—starting from assumptions about what was learned, before seeking plausible mechanisms to learn it—we adopt a mechanism-driven approach—focusing first on the question of how infants might plausibly learn from realistic input, and seeking to characterize what was learned only a posteriori. Second, we introduce a quantitative modeling framework suitable to implement this approach at scale using realistic input. This involves explicitly simulating both the ecological learning process taking place at home and the assessment of infants’ discrimination abilities in the laboratory.

Beyond the immediate results, the framework we introduce provides a feasible way of linking accounts of early phonetic learning to systematic predictions regarding the empirical phenomenon they seek to explain—i.e., the observed cross-linguistic differences in infants’ phonetic discrimination.

Approach

We start from a possible learning mechanism. We simulate the learning process in infants by implementing this mechanism computationally and training it on naturalistic speech recordings in a target language—either Japanese or American English. This yields a candidate model for the early phonetic knowledge of, say, a Japanese infant. Next, we assess the model’s ability to discriminate phonetic contrasts of American English and Japanese—for example, American English vs [l]—by simulating a discrimination task using speech stimuli corresponding to this contrast. We test whether the predicted discrimination patterns agree with the available empirical record on cross-linguistic differences between American English- and Japanese-learning infants. Finally, we investigate whether what has been learned by the model corresponds to the phonetic categories of the model’s “native” language (i.e., its training language).

To identify a promising learning mechanism, we build on recent advances in the field of machine learning and, more specifically, in unsupervised representation learning for speech technology, which have established that, given only raw, untranscribed, unsegmented speech recordings, it is possible to learn representations that accurately discriminate the phonetic categories of a language (52–69). The learning algorithms considered have been argued to be particularly relevant for modeling how infants learn in general, and learn language in particular (70). Among available learning algorithms, we select the one at the core of the winning entries in the Zerospeech 2015 and 2017 international competitions in unsupervised speech-representation learning (57, 58, 68). Remarkably, it is based on a Gaussian mixture clustering mechanism—illustrated in Fig. 1A—that can straightforwardly be interpreted as a form of distributional learning (24, 26). A different input representation to the Gaussian mixture is used than in previously proposed implementations of distributional learning, however (29, 33, 35, 37–39, 41). Simple descriptors of the shape of the speech signal’s short-term auditory spectrum sampled at regular points in time (every 10 ms) (71) are used instead of traditional phonetic measurements obtained separately for each vowel and consonant, such as formant frequencies or harmonic amplitudes.^§ This type of input representation only assumes basic auditory abilities from infants, which are known to be fully operational shortly after birth (74), and has been proposed previously as a potential way to get around both the lack-of-invariance and the phonetic category segmentation problems in the context of adult word recognition (30). A second difference from previous implementations of distributional learning is in the output representation. Test stimuli are represented as sequences of posterior probability vectors (posteriorgrams) over $K$ Gaussian components in the mixture (Fig. 1B), rather than simply being assigned to the most likely Gaussian component. These continuous representations have been shown to support accurate discrimination of native phonetic categories in the Zerospeech challenges.

Fig. 1.

Gaussian mixture model training and representation extraction, illustrated for a model with three Gaussian components. In practice, the number of Gaussian components is learned from the data and much higher. (A) Model training: The learning algorithm extracts moderate-dimensional (d = 39) descriptors of the local shape of the signal spectrum at time points regularly sampled every 10 ms (speech frames). These descriptors are then considered as having been generated by a mixture of Gaussian probability distributions, and parameters for this mixture that assign high probability to the observed descriptors are learned. (B) Model test: The sequence of spectral-shape descriptors for a test stimulus (possibly in a language different from the training language) are extracted, and the model representation for that stimulus is obtained as the sequence of posterior probability vectors resulting from mapping each descriptor to its probability of having been generated by each of the Gaussian components in the learned mixture.

To simulate the infants’ learning process, we expose the selected learning algorithm to a realistic model of the linguistic input to the child, in the form of raw, unsegmented, untranscribed, multispeaker continuous speech signal in a target language (either Japanese or American English). We select recordings of adult speech made with near-field, high-quality microphones in two speech registers, which cover the range of articulatory clarity that infants may encounter. On one end of the range, we use spontaneous adult-directed speech, and on the other, we use read speech; these two speaking registers are crossed with the language factor (English or Japanese), resulting in four corpora, each split into a training set and a test set (Table 1). We would have liked to use recordings made in infants’ naturalistic environments, but no such dataset of sufficient audio quality was available for this study. It is unclear whether or how using infant-directed speech would impact results: The issue of whether infant-directed speech is beneficial for phonetic learning has been debated, with arguments in both directions (75–82). We train a separate model for each of the four training sets, allowing us to check that our results hold across different speech registers and recording conditions. We also train separate models on 10 subsets of each training set for several choices of subset sizes, allowing us to assess the effects of varying the amount of input data and the variability due to the choice of training data for a given input size.

Table 1.

Language, speech register, duration, and number of speakers of training and test sets for our four corpora of speech recordings

Corpus	Language	Reg.	Duration		No. of speakers
			Train	Test	Train	Test
R-Eng (83)	Am. English	Read	19h30	9h39	96	47
R-Jap (84)	Japanese	Read	19h33	9h40	96	47
Sp-Eng (85)	Am. English	Spont.	9h13	9h01	20	20
Sp-Jap (86)	Japanese	Spont.	9h11	8h57	20	20

Am., American; reg., register; spont., spontaneous.

We next evaluate whether the trained “Japanese native” and “American-English native” models correctly predict early phonetic learning, as observed in Japanese-learning and American English-learning infants, respectively, and whether they make novel predictions regarding the differences in speech-discrimination abilities between these two populations. Because we do not assume that the outcome of infants’ learning is adult-like knowledge, we can only rely on infant data for evaluation. The absence of specific assumptions a priori about what is going to be learned and the sparsity of empirical data on infant discrimination make this challenging. The algorithm we consider outputs complex, high-dimensional representations (Fig. 1B) that are not easy to link to concrete predictions regarding infant discrimination abilities. Traditional signal-detection theory models of discrimination tasks (87) cannot handle high-dimensional perceptual representations, while more elaborate (Bayesian) probabilistic models (88) have too many free parameters given the scarcity of available data from infant experiments. We rely, instead, on the machine ABX approach that we previously developed (89, 90). It consists of a simple model of a discrimination task, which can handle any representation format, provided the user can provide a reasonable measure of (dis)similarity between representations (89, 90). This is not a detailed model of infant’s performance in a specific experiment, but, rather, a simple and effectively parameterless way to systematically link the complex speech representations produced by our models to predicted discrimination patterns. For each trained model and each phonetic contrast of interest, we obtain an “ABX error rate,” such that $0\%$ and $50\%$ error indicate perfect and chance-level discrimination, respectively. This allows us to evaluate the qualitative match between the model’s discrimination abilities and the available empirical record in infants (see SI Appendix, Discussion 3 for an extended discussion of our approach to interpreting the simulated discrimination errors and relating them to empirical observations, including why it would not be meaningful to seek a quantitative match at this point).

Finally, we investigate whether the learned Gaussian components correspond to phonetic categories. We first compare the number of Gaussians in a learned mixture to the number of phonemes in the training language (category number test): Although a phonetic category can be more concrete than a phoneme, the number of phonetic categories documented in typical linguistic analyses remains on the same order of magnitude as the number of phonemes. We then administer two diagnostic tests based on the two defining characteristics identified above that any representation corresponding to phonetic categories should pass.^¶ The first characteristic is size/duration: A phonetic category is a phoneme-sized unit (i.e., the size of a vowel or a consonant). Our duration test probes this by measuring the average duration of activation of the learned Gaussian components (a component is taken to be “active” when its posterior probability is higher than all other components), and comparing this to the average duration of activation of units in a baseline system trained to recognize phonemes with explicit supervision. The second characteristic is abstractness: Although phonetic categories can depend on phonetic context^∥ and on nonlinguistic properties of the speech signal—e.g., the speaker’s gender—at a minimum, the central phone in the same word repeated several times by the same speaker is expected to be consistently realized as the same phonetic category. Our acoustic (in)variance test probes this by counting the number of distinct representations needed by our model to represent 10 occurrences of the central frame of the central phone of the same word either repeated by the same speaker (within-speaker condition) or by different speakers (across-speaker condition). We use a generous correction to handle possible misalignment (Materials and Methods). The last two tests can be related to the phonetic category segmentation and lack-of-invariance problems: Solving the phonetic category segmentation problem involves finding units that would pass the duration test, while solving the lack-of-invariance problem involves finding units that would pass the acoustic (in)variance test. Given the laxity in the use of the concept of phonetic category in the literature, some might be tempted to challenge that even these diagnostic tests can be relied on. If they cannot, however, it is not clear to us how phonetic category accounts of early phonetic learning should be understood as scientifically refutable claims.

Results

Overall Discrimination.

After having trained a separate model for each of the four possible combinations of language and register, we tested whether the models’ overall discrimination abilities, like those of infants (15–17), are specific to their “native” (i.e., training) language. Specifically, for each corpus, we looked at overall discrimination errors averaged over all consonant and vowel contrasts available in a held-out test set from that corpus (Table 1). We tested each of the two American English-trained and each of the two Japanese-trained models on each of four test sets, yielding a total of 4$\times$4 discrimination errors. We tabulated the average errors in terms of four conditions, depending on the relation between the test set and the training background of the model: native vs. nonnative contrasts and same vs. different register. The results are reported in Fig. 2 (see also SI Appendix, Figs. S1 and S4 for nontabulated results). Fig. 2A shows that discrimination performance is higher, on average, in matched-language conditions (in blue) than in mismatched-language conditions (in red). In contrast, register mismatch has no discernible impact on discrimination performance. A comparison with a supervised phoneme-recognizer baseline (SI Appendix, Fig. S3) shows a similar pattern of results, but with a larger absolute cross-linguistic difference. If we interpret this supervised baseline as a proxy to the adult state, then our model suggests that infant’s phonetic representations, while already language-specific, remain “immature”.^** Fig. 2B shows the robustness of these results, with 81.7% of the 1,295 distinct phonetic contrasts tested proving easier to discriminate on the basis of representations from a model trained on the matching language. Taken together, these results suggest that, similar to infants, our models acquire language-specific representations, and that these representations generalize across register.

Fig. 2.

(A) Average ABX error rates over all consonant and vowel contrasts obtained with our models as a function of the match between the training-set and test-set language and register. Error bars correspond to plus and minus one SD of the errors across resampling of the test-stimuli speakers. The native (blue) conditions, with training and test in the same language, show fewer discrimination errors than the nonnative (red) conditions, whereas there is little difference in error rate within the native and within the nonnative conditions. This shows that the models learned native-language-specific representations that generalize across register. (B) Letter-value representation (94) of the distribution of native advantages across all tested phonetic contrasts (pooled over both languages). The native-language advantage is the increase in discrimination error for a contrast of language L1 between an “L1-native” model and a model trained on the other language for the same training register. The “native register” advantage is the increase in error for a contrast of register R1 between an “R1-native” model and a model trained on the other register for the same training language. A native language advantage is observed across contrasts (positive advantage for $81.7\%$ of all contrasts), and there is a weaker native register advantage (positive advantage for $60.1\%$ of all contrasts).

American English –[l] Discrimination.

Next, we focus on the specific case of American English –[l] discrimination, for which Japanese adults show a well-documented deficit (2, 3) and which has been studied empirically in American English and Japanese infants (14). While 6- to 8-mo-old infants from American English- and Japanese-language backgrounds performed similarly in discriminating this contrast, 10- to 12-mo-old American English infants outperformed their Japanese peers. We compare the discrimination errors obtained with each of our four models for American English –[l] and for two controls: the American English [w]–[j] contrast (as in “wet” vs. “yet”), for which we do not expect a gap in performance between American English and Japanese natives (95), and the average error over all of the other consonant contrasts of American English. For each contrast and for each of the four models, we averaged discrimination errors obtained on each of the two American English held-out test sets, yielding 3$\times$4 discrimination errors. We further averaged over models with the same native language to obtain 3$\times$2 discrimination errors. The results are shown in Fig. 3 (see also SI Appendix, Figs. S2 and S6 for untabulated results and a test confirming our results with the synthetic stimuli used in the original infant experiment, respectively). In Fig. 3A, we see that, similar to 10- to 12-mo old infants, American English native models (in blue) greatly outperform Japanese native models (in red) in discriminating American English –[l]. Here, again, a supervised phoneme-recognizer baseline yields a similar pattern of results, but with larger cross-linguistic differences (Fig. 3C; see also SI Appendix, Fig. S5), again suggesting that the representations learned by the unsupervised models—like those of infants—remain somewhat “immature.” In Fig. 3B, we see results obtained by training 10 different models on 10 different subsets of the training set of each corpus, varying the sizes of the subsets (see Materials and Methods for more details). It reveals that 1 h of input is sufficient for the divergence between the Japanese and English models to emerge robustly and that this divergence increases with exposure to the native language. While it is difficult to interpret this trajectory relative to absolute quantities of data or discrimination scores, the fact that the cross-linguistic difference increases with more data mirrors the empirical findings from infants (see also an extended discussion of our approach to interpreting the simulated discrimination errors and relating them to empirical data in SI Appendix, Discussion 3).

Fig. 3.

(A) ABX error rates for the American English –[l] contrast and two controls: American English [w]–[j] and average over all American English consonant contrasts (C–C). Error rates are reported for two conditions: average over models trained on American English and average over models trained on Japanese. Error bars correspond to plus and minus one SD of the errors across resampling of the test-stimuli speakers. Similar to infants, the Japanese native models exhibit a specific deficit for American English –[l] discrimination compared to the American English models. (B) The robustness of the effect observed in A to changes in the training stimuli and their dependence on the amount of input are assessed by training separate models on independent subsets of the training data of each corpus of varying duration (Materials and Methods). For each selected duration (except when using the full training set), 10 independent subsets are selected, and 10 independent models are trained. We report mean discrimination errors for American English –[l] and [w]–[j] as a function of the amount of input data, with error bands indicating plus or minus one SD. The results show that a deficit in American English –[l] discrimination for Japanese-native models robustly emerges with as little as 1 h of training data. (C) To give a sense of scale we compare the cross-linguistic difference obtained with the unsupervised Gaussian mixture models (GMM) on American English –[l] (Left) to the one obtained with supervised phoneme-recognizer baselines (hidden Markov model, HMM; Right). The larger cross-linguistic difference obtained with the supervised baselines suggests that the representations learned by our unsupervised models, similar to those observed in infants, remain somewhat immature.

Nature of the Learned Representations.

Finally, we considered the nature of the learned representations and tested whether what has been learned can be understood in terms of phonetic categories. Results are reported in Fig. 4 (see also SI Appendix, Fig. S7 for comparisons with a different supervised baseline). First, looking at the category number criterion in Fig. 4A, we see that our models learned more than 10 times as many categories as the number of phonemes in the corresponding languages. Even allowing for notions of phonetic categories more granular than phonemes, we are not aware of any phonetic analysis ever reporting that many allophones in these languages. Second, looking at the duration criterion in Fig. 4B, the learned Gaussian units appear to be activated, on average, for about a quarter the duration of a phoneme. This is shorter than any linguistically identified unit. It shows that the phonetic category segmentation problem has not been solved. Next, looking at the acoustic (in)variance criterion in Fig. 4 C and D—for the within- and across-speakers conditions, respectively—we see that our models require, on average, around two distinct representations to represent 10 tokens of the same phonetic category without speaker variability and three distinct representations across different speakers. The supervised phoneme-recognizer baseline establishes that our results cannot be explained by defective test stimuli. Instead, this result shows that the learned units are finer-grained than phonetic categories along the spectral axis and that the lack-of-invariance problem has not been solved. Based on these tests, we can conclude that the learned units do not correspond to phonetic categories in any meaningful sense of the term.

Fig. 4.

Diagnostic test results for our four unsupervised Gaussian mixture models (in beige) and phoneme-recognizer baselines trained with explicit supervision (in pink). (Upper) American English native models. (Lower) Japanese native models. Models are tested on read speech in their native language. (A) Number of units learned by the models. Gaussian mixtures discover 10 to 20 times more categories than there are phonemes in the training language, exceeding any reasonable count for phonetic categories. (B) Average duration of activation of the learned units. The average duration of activation of each unit is computed, and the average and SD of the resulting distribution over units are shown. Learned Gaussian units get activated, on average, for about the quarter of the duration of a phoneme. They are, thus, much too “short” to correspond to phonetic categories. (C) Average number of distinct representations for the central frame of the central phone for 10 repetitions of a same word by the same speaker, corrected for possible misalignment. The number of distinct representations is computed for each word type with sufficient repetitions in the test set, and the average and SD of the resulting distribution over word types are shown. The phoneme-recognizer baseline reliably identifies the 10 tokens as exemplars from a common phonetic category, whereas our Gaussian mixture models typically maintain on the order of two distinct representations, indicating representations too fine-grained to be phonetic categories. (D) As in C, but with repetitions of a same word by 10 speakers, showing that the learned Gaussian units are not speaker-independent. Spont., spontaneous.

Discussion

Through explicit simulation of the learning process under realistic learning conditions, we showed that several aspects of early phonetic learning, as observed in American English and Japanese infants, can be correctly predicted through a distributional learning (i.e., clustering) mechanism applied to simple spectrogram-like auditory features sampled at regular time intervals. This contrasts with previous attempts to show the feasibility of potential mechanisms for early phonetic learning, which only considered highly simplified learning conditions and/or failed (26, 28, 29, 33–44, 46–48). We further showed that the learned speech units are too brief and too acoustically variable to correspond to the vowel- and consonant-like phonetic categories posited in earlier accounts of early phonetic learning.

Distributional learning has been an influential hypothesis in language acquisition for over a decade (24, 26, 27). Previous modeling results questioning the feasibility of learning phonetic categories through distributional learning have traditionally been interpreted as challenging the learning mechanism (26, 28, 29, 39–44), but we have instead suggested that such results may be better interpreted as challenging the idea that phonetic categories are the outcome of early phonetic learning. Supporting this view, we showed that when the requirement to learn phonetic categories is abandoned, distributional learning on its own can be sufficient to explain early phonetic learning under realistic learning conditions—using unsegmented, untranscribed speech signal as input. Our results are still compatible with the idea that mechanisms tapping into other relevant sources of information might complement distributional learning—an idea supported by evidence that infants learn from some of these sources in the laboratory (96–102)—but they suggest that those other sources of information may not play a role as crucial as previously thought (26). Our findings also join recent accounts of “word segmentation” (103) and the “language familiarity effect” (104) in questioning whether we might have been overattributing linguistic knowledge to preverbal infants across the board.

An Account of Early Phonetic Learning without Phonetic Categories.

Our results suggest an account of phonetic learning that substantially differs from existing ones. Whereas previous proposals have been primarily motivated through an outcome-driven perspective—starting from assumptions about what it is about language that is learned—the motivation for the proposed account comes from a mechanism-driven perspective—starting from assumptions about how learning might proceed from the infant’s input. This contrast is readily apparent in the choice of the initial speech representation, upon which the early phonetic learning process operates (the input representation). Previous accounts assumed speech to be represented innately through a set of universal (i.e., language-independent) phonetic feature detectors (9, 18–21, 48–51). The influential phonetic category accounts, furthermore, assumed these features to be available phonetic segment by phonetic segment (i.e., for each vowel and consonant separately) (9, 18–21). While these assumptions are attractive from an outcome-driven perspective—they connect transparently to phonological theories in linguistics and theories of adult speech perception that assume a decomposition of speech into phoneme-sized segments defined in terms of abstract phonological features—from a mechanism-driven perspective, both assumptions are difficult to reconcile with the continuous speech signal that infants hear. The lack of acoustic–phonetic invariance problem challenges the idea of phonetic feature detectors, and the phonetic category segmentation problem challenges the idea that the relevant features are segment-based (30–32). The proposed account does not assume either problem to be solved by infants at birth. Instead, it relies on basic auditory abilities that are available to neonates (74), using simple auditory descriptors of the speech spectrum obtained regularly along the time axis. This type of spectrogram-like representation is effective in speech-technology applications (71) and can be seen as the output of a simple model of the peripheral auditory system (ref. 90, chap. 3), which is fully operational shortly after birth (74). Such representations have also been proposed before as an effective way to get around both the lack-of-invariance and the phonetic category segmentation problems in the context of adult word recognition (30) and can outperform representations based on traditional phonetic measurements (like formant frequencies) as predictors of adult speech perception (105–109).

While the input representation is different, the learning mechanism in the proposed account—distributional learning—is similar to what had originally been proposed in phonetic category accounts. Infants’ abilities, both in the laboratory (24, 27) and in ecological conditions (25), are consistent with such a learning mechanism. Moreover, when applied to the input representation considered in this paper, distributional learning is adaptive in that it yields speech representations that can support remarkably accurate discrimination of the phonetic categories of the training language, outperforming a number of alternatives that have been proposed for unsupervised speech representation learning (57, 58, 68).

As a consequence of our mechanism-driven approach, what has been learned needs to be determined a posteriori based on the outcomes of learning simulations. The speech units learned under the proposed account accurately model infants’ discrimination, but are too brief and acoustically variable to correspond to phonetic categories, failing, in particular, to provide a solution to the lack-of-invariance and phonetic category segmentation problems (30). Such brief units do not correspond to any identified linguistic unit (22) (see SI Appendix, Discussion 4 for a discussion of possible reasons why the language-acquisition process might involve the learning by infants of a representation with no established linguistic interpretation and a discussion of the biological and psychological plausibility of the learned representation), and it will be interesting to try to further understand their nature. However, since there is no guarantee that a simple characterization exists, we leave this issue for future work.

Phonetic categories are often assumed as precursors in accounts of phenomena occurring later in the course of language acquisition. Our account does not necessarily conflict with this view, as phonetic categories may be learned later in development, before phonological acquisition. Alternatively, the influential PRIMIR account of early language acquisition (“a developmental framework for Processing Rich Information from Multi-dimensional Interactive Representations”, ref. 20) proposes that infants learn in parallel about the phonetics, word forms, and phonology of their native language, but do not develop abstract phonemic representations until well into their second year of life. Although PRIMIR explicitly assumes phonetic learning to be phonetic category learning, other aspects of their proposed framework do not depend on that assumption, and our framework may be able to stand in for the phonetic learning process they assume.

To sum up, we introduced and motivated an account of early phonetic learning—according to which infants learn through distributional learning, but do not learn phonetic categories—and we showed that this account is feasible under realistic learning conditions, which cannot be said of any other account at this time. Importantly, this does not constitute decisive evidence for our account over alternatives. Our primary focus has been on modeling cross-linguistic differences in the perception of one contrast –[l]; further work is necessary to determine to what extent our results extend to other contrasts and languages (110). Furthermore, an absence of feasibility proof does not amount to a proof of infeasibility. While we have preliminary evidence that simply forcing the model to learn fewer categories is unlikely to be sufficient (SI Appendix, Figs. S9 and S10), recently proposed partial solutions to the phonetic category segmentation problem (e.g., refs. 111–113) and to the lack-of-invariance problem (114) (see also SI Appendix, Discussion 2 regarding the choice of model initialization) might yet lead to a feasible phonetic category-based account, for example. In addition, a number of other representation learning algorithms proposed in the context of unsupervised speech technologies and building on recent developments in the field of machine learning have yet to be investigated (52–69). They might provide concrete implementations of previously proposed accounts of early phonetic learning or suggest new ones altogether. This leaves us with a large space of appealing theoretical possibilities, making it premature to commit to a particular account. Candidate accounts should instead be evaluated on their ability to predict empirical data on early phonetic learning, which brings us to the second main contribution of this article.

Toward Predictive Theories of Early Phonetic Learning.

Almost since the original empirical observation of early phonetic learning (115), a number of theoretical accounts of the phenomenon have coexisted (9, 19, 48, 49). This theoretical underdetermination has typically been thought to result from the scarcity of empirical data from infant experiments. We argue instead that the main limiting factor on our understanding of early phonetic learning might have been the lack—on the theory side—of a practical method to link proposed accounts of phonetic learning with concrete, systematic predictions regarding the empirical discrimination data they seek to explain. Establishing such a systematic link has been challenging due to the necessity of dealing with the actual speech signal, with all its associated complexity. The modeling framework we introduce provides a practical and scalable way to overcome these challenges and obtain the desired link for phonetic learning theories—a major methodological advance, given the fundamental epistemological importance of linking explanandum and explanans in scientific theories (116).

Our mechanism-driven approach to obtaining predictions—which can be applied to any phonetic learning model implemented in our framework—consists first of explicitly simulating the early phonetic learning process as it happens outside of the laboratory, which results in a trained model capable of mapping any speech input to a model representation for that input. The measurement of infants’ perceptual abilities in laboratory settings—including their discrimination of any phonetic contrast—can then be simulated on the basis of the model’s representations of the relevant experimental stimuli. Finally, phonetic contrasts for which a significant cross-linguistic difference is robustly predicted can be identified through a careful statistical analysis of the simulated discrimination judgments (SI Appendix, Materials and Methods 4). As an illustration of how such predictions can be generated, we report specific predictions made by our distributional learning model in SI Appendix, Table S1 (see also SI Appendix, Discussion 5).

Although explicit simulations of the phonetic learning process have been carried out before (29, 33–43, 45, 48, 72, 73), those have typically been evaluated based on whether they learned phonetic categories, and have not been directly used to make predictions regarding infants’ discrimination abilities. An outcome-driven approach to making predictions regarding discrimination has typically been adopted instead, starting from the assumption that phonetic categories are the outcome of learning. To the best of our knowledge, this has never resulted in the kind of systematic predictions we report here, however (see SI Appendix, Discussion 6 for a discussion of the limits of previous approaches and of the key innovations underlying the success of our framework).

Our framework readily generates empirically testable predictions regarding infants’ discrimination, yet further computational modeling is called for before we return to experiments. Indeed, existing data—collected over more than three decades of research (5, 15–17)—might already suffice to distinguish between different learning mechanisms. To make that determination, and to decide which contrasts would be most useful to test next, in case more data are needed, many more learning mechanisms and training/test language pairs will need to be studied. Even for a specified learning mechanism and training/test datasets, multiple implementations should ideally be compared (e.g., testing different parameter settings for the input representations or the clustering algorithm), as implementational choices that weren’t initially considered to be important might, nevertheless, have an effect on the resulting predictions and, thus, need to be included in our theories. Conversely, features of the model that may seem important a priori (e.g., the type of clustering algorithm used) might turn out to have little effect on the learning outcomes in practice.

Cognitive science has not traditionally made use of such large-scale modeling, but recent advances in computing power, large datasets, and machine-learning algorithms make this approach more feasible than ever before (70). Together with ongoing efforts in the field to collect empirical data on a large scale—such as large-scale recordings of infants’ learning environments at home (117) and large-scale assessment of infants’ learning outcomes (118, 119)—our modeling approach opens the path toward a much deeper understanding of early language acquisition.

Materials and Methods

Datasets.

We used speech recordings from four corpora: two corpora of read news articles—a subset of the Wall Street Journal corpus of American English (83) (WSJ) and the Globalphone corpus of Japanese (84) (GPJ)—and two corpora of spontaneous speech—the Buckeye corpus of American English (85) (BUC) and a subset of the corpus of spontaneous Japanese (86) (CSJ). As we are primarily interested in the effect of training language on discrimination abilities, we sought to remove possibly confounding differences between the two read corpora and between the two spontaneous corpora. Specifically, we randomly sampled subcorpora while matching total duration, number, and gender of speakers and amount of speech per speaker. We made no effort to match corpora within a language, as the differences (for example, in the total duration and number of speakers) only serve to reinforce the generality of any result holding true for both registers. Each of the sampled subsets was further randomly divided into a training and a test set (Table 1), satisfying three conditions: The test set lasts approximately 10 h; no speaker is present in both the training and test set; and the training and test sets for the two read corpora, and separately for the two spontaneous corpora, remain matched on overall duration, number of speakers of each gender, and distribution of duration per speaker of each gender. To carry out analyses taking into account the effect of input size and of the choice of input data, we further divided each training set in 10 with each 1/10th subset containing an equal proportion of the speech samples from each speaker in the original training set. We then divided each of the 1/10th subsets in 10 again following the same procedure and selected the first subset to obtain 10 1/100th subsets. Finally, we iterated the procedure one more time to obtain 10 1/1,000th subsets. See SI Appendix, Materials and Methods 1 for additional information.

Signal Processing, Models, and Inference.

The raw speech signal was decomposed into a sequence of overlapping 25-ms-long frames sampled every 10 ms, and moderate-dimensional (d = 39) descriptors of the spectral shape of each frame were then extracted, describing how energy in the signal spreads across different frequency channels. The descriptors comprised 13 mel-frequency cepstral coefficients with their first and second time derivatives. These coefficients correspond approximately to the principal components of spectral slices in a log-spectrogram of the signal, where the spectrogram frequency channels were selected on a mel-frequency scale (linear for lower frequency and logarithmic for higher frequencies, matching the frequency selectivity of the human ear).

For each corpus, the set of all spectral-shape descriptors for the corpus’ training set was modeled as a large independent and identically distributed sample from a probabilistic generative model. The generative model is a Gaussian mixture model with no restrictions on the form of covariance matrices and with a Dirichlet process prior over its parameters with normal-inverse-Wishart base measure. The generative model is depicted as a graphical model in plate notation in Fig. 5, where $n$ is the number of input descriptors, $\left(X_1,X_2,\dots ,X_n\right)$ are the random variables from which the observed descriptors are assumed to be sampled, and the other elements are latent variables and hyperparameters. The depicted variables have the following conditional distributions:

$$\begin{array}{ccccc}{X}_i\hfill & |\hfill & z_i,\left({\mu }_1,{\mu }_2,\dots \right),\left({\mathrm{\Lambda }}_1,{\mathrm{\Lambda }}_2,\dots \right)\hfill & \sim \hfill & \mathcal{N}\left({\mu }_{z_i},{\mathrm{\Lambda }}_{z_i}^{-1}\right)\hfill \\ {\mu }_k\hfill & |\hfill & {\mathrm{\Lambda }}_k,{\mu }_0,\lambda \hfill & \sim \hfill & \mathcal{N}\left({\mu }_0,{\left(\lambda {\mathrm{\Lambda }}_k\right)}^{-1}\right)\hfill \\ {\mathrm{\Lambda }}_k\hfill & |\hfill & {\mathrm{\Lambda }}_0,\nu \hfill & \sim \hfill & \mathcal{W}\left({\mathrm{\Lambda }}_0,\nu \right)\hfill \\ z_i\hfill & |\hfill & \pi \hfill & \sim \hfill & \mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\left(\pi \right)\hfill \\ \pi \hfill & |\hfill & \alpha \hfill & \sim \hfill & \mathrm{S}\mathrm{B}\left(\alpha \right)\hfill \end{array},$$

for any $1\le i\le n$, for any $k\in \left\{\mathrm{1,2},\dots \right\}$, with $\mathcal{N}$ the multivariate Gaussian distribution, $\mathcal{W}$ the Wishart distribution, Multi the generalization of the usual multinomial probability distribution to an infinite discrete support, and SB the mixing weights generating distribution from the stick-breaking representation of Dirichlet processes (120). Mixture parameters with high posterior probability given the observed input features vectors and the prior were found by using an efficient parallel Markov chain Monte Carlo sampler (121). Following previous work (60, 65), model initialization was performed by partitioning training points uniformly at random into 10 clusters, and the hyperparameters were set as follows: $\alpha$ to 1, ${\mu }_0$ to the average of all input features vectors, $\lambda$ to 1, ${\lambda }_0$ to the inverse of the covariance of all input feature vectors, and $\nu$ to 42 (i.e., the spectral shape descriptors dimension plus 3). We additionally trained a model on each of the 10 1/10th, 1/100th, and 1/1,000th training subsets of each of the four corpora, following the same procedure.

Fig. 5.

Generative Gaussian mixture model with Dirichlet process prior with normal-inverse-Wishart base measure, represented as a graphical model in plate notation based on the stick-breaking construction of Dirichlet processes.

Given a trained Gaussian mixture with $K$ components, mixing weights $\left({\pi }_1,{\pi }_2,\dots ,{\pi }_K\right)$, means $\left({\mu }_1,{\mu }_2,\dots ,{\mu }_K\right)$, and covariance matrices $\left({\mathrm{\Sigma }}_1,{\mathrm{\Sigma }}_2,\dots ,{\mathrm{\Sigma }}_K\right)$, we extracted a test stimulus representation from the sequence $\left(x_1,x_2,\dots ,x_m\right)$ of spectral-shape descriptors for that stimulus, as the sequence of posterior probability vectors $\left(p_1,p_2,\dots ,p_m\right)$, where for any frame $i$, $1\le i\le m$, $p_i=\left(p_{i1},p_{i2},\dots ,p_{iK}\right)$, with, for any $1\le k\le K$:

$$p_{ik}=\frac{{\pi }_k\mathcal{N}\left(x_i|{\mu }_k,{\mathrm{\Sigma }}_k\right)}{{\sum }_{j=1}^K{\pi }_j\mathcal{N}\left(x_i|{\mu }_j,{\mathrm{\Sigma }}_j\right)}.$$

As a baseline, we also trained a phoneme recognizer on the training set of each corpus, with explicit supervision (i.e., phonemic transcriptions of the training stimuli). We extracted frame-level posterior probabilities at two granularity levels: actual phonemes—the phoneme-recognizer baseline—and individual states of the contextual hidden Markov models—the ASR phone-state baseline. See SI Appendix, Materials and Methods 2 for additional information.

Discrimination Tests.

Discriminability between model representations for phonetic contrasts of interest was assessed by using machine ABX discrimination errors (89, 90). Discrimination was assessed in context, defined as the preceding and following sound and the identity of the speaker. For example, discrimination of American English [u] vs. [i] was assessed in each available context independently, yielding—for instance—a separate discrimination-error rate for test stimuli in [b]_[t] phonetic context, as in “boot” vs. “beet,” as spoken by a specified speaker. Other possible factors of variability, such as word boundaries or syllable position, were not controlled. For each model, each test corpus, and each phonemic contrast in that test corpus (as specified by the corpus’ phonemic transcriptions), we obtained a discrimination error for each context in which the contrasted phonemes occurred at least twice in the test corpus’ test set. To avoid combinatorial explosion in the number of ABX triplets to be considered, a randomly selected subset of five occurrences was used to compute discrimination errors when a phoneme occurred more than five times in a given context. An aggregated ABX error rate was obtained for each combination of model, test corpus, and phonemic contrast, by averaging the context-specific error rates over speakers and phonetic contexts, in that order.

Model representations were extracted for the whole test sets, and the part corresponding to a specific occurrence of a phonetic category was then obtained by selecting representation frames centered on time points located between the start and end times for that occurrence, as specified by the test set’s forced aligned phonemic transcriptions. Given model representations $\mathrm{\Delta }=\left({\delta }_1,{\delta }_2,\dots ,{\delta }_{n_{\delta }}\right)$ and $\mathrm{\Xi }=\left({\xi }_1,{\xi }_2,\dots ,{\xi }_{n_{\xi }}\right)$ for $n_{\delta }$ tokens of phonetic category $\delta$ and $n_{\xi }$ tokens of phonetic category $\xi$, the nonsymmetrized machine ABX discrimination error between $\delta$ and $\xi$ was then estimated as the proportion of representation triplets $a$, $b$, $x$, with $a$ and $x$ taken from $\mathrm{\Delta }$ and $b$ taken from $\mathrm{\Xi }$, such that $x$ is closer to $b$ than to $a$, i.e.,

$$\begin{array}{ll}\hfill ê\left(\mathrm{\Delta },\mathrm{\Xi }\right)≔\frac{1}{n_{\delta }\left(n_{\delta }-1\right)n_{\xi }}\sum _{a=1}^{n_{\delta }}\sum _{b=1}^{n_{\xi }}\sum _{\begin{array}{c}x=1\\ x\ne a\end{array}}^{n_{\delta }}& [{\mathbb{1}}_{d\left({\xi }_b,{\delta }_x\right)<d\left({\delta }_a,{\delta }_x\right)}\hfill \\ \hfill & +\frac{1}{2}{\mathbb{1}}_{d\left({\xi }_b,{\delta }_x\right)=d\left({\delta }_a,{\delta }_x\right)}],\hfill \end{array}$$

where $\mathbb{1}$ is the indicator function returning one when its predicate is true and zero otherwise, and $d$ is a dissimilarity function taking a pair of model representations as input and returning a real number (with higher values indicating more dissimilar representations). The (symmetric) machine ABX discrimination error between $\delta$ and $\xi$ was then obtained as:

$$\widehat{ϵ}\left(\mathrm{\Delta },\mathrm{\Xi }\right)=\widehat{ϵ}\left(\mathrm{\Xi },\mathrm{\Delta }\right)≔\frac{1}{2}\left[ê\left(\mathrm{\Delta },\mathrm{\Xi }\right)+ê\left(\mathrm{\Xi },\mathrm{\Delta }\right)\right].$$

As realizations of phonetic categories vary in duration, we need a dissimilarity function $d$ that can handle model representations with variable length. This was done, following established practice (12, 13, 55, 57, 68), by measuring the average dissimilarity along a time alignment of the two representations obtained through dynamic time warping (122), where the dissimilarity between model representations for individual frames was measured with the symmetrized Kullback–Leibler divergence for posterior probability vectors and with the angular distance for spectral shape descriptors.

Analysis of Learned Representations.

Learned units were taken to be the Gaussian components for the Gaussian mixture models, the phoneme models for the phoneme-recognizer baseline, and the phone-state models for the ASR phone-state baseline. Since experimental studies of phonetic categories are typically performed with citation form stimuli, we studied how each model represents stimuli from the matched-language read speech corpus’ test set.

To study average durations of activation, we excluded any utterance-initial or utterance-final silence from the analysis, as well as any utterance for which utterance-medial silence was detected during the forced alignment. The average duration of activation for a given unit was computed by averaging over all episodes in the test utterances during which that unit becomes dominant, i.e., has the highest posterior probability among all units. Each of these episodes was defined as a continuous sequence of speech frames, during which the unit remains dominant without interruptions, with duration equal to that number of speech frames times 10 ms.

The acoustic (in)variance of the learned units was probed by looking at multiple repetitions of a single word and testing whether the dominant unit at the central frame of the central phone of the word remained the same for all repetitions. Specifically, we counted the number of distinct dominant units occurring at the central frame of the central phone for 10 repetitions of the same word. To compensate for possible misalignment of the central phones’ central frames (e.g., due to slightly different time courses in the acoustic realization of the phonetic segment and/or small errors in the forced alignment), we allowed the dominant unit at the central frame to be replaced by any unit that is dominant at some point within the previous or following 46 ms (thus covering a 92-ms slice of time corresponding to the average duration of a phoneme in our read-speech test sets), provided it could bring down the overall count of distinct dominant units for the 10 occurrences (see SI Appendix, Materials and Methods 3 for more information). We considered two conditions: In the within-speaker condition, the test stimuli were uttered by the same speaker 10 times; in the across-speaker condition, they were uttered by 10 different speakers one time. See SI Appendix, Materials and Methods 3 for more information on the stimulus-selection procedure.

Data and Code Availability.

The datasets analyzed in this study are publicly available from the commercial vendors and research institutions holding their copyrights (83–86). Datasets generated during the course of the study that do not include proprietary information are available at https://osf.io/d2fpb/. Code to reproduce the results is available at https://github.com/Thomas-Schatz/perceptual-tuning-pnas.