Input Variability Facilitates Unguided Subcategory Learning in Adults

Abstract

Purpose

This experiment investigated whether input variability would affect initial learning of noun gender subcategories in an unfamiliar, natural language (Russian), as it is known to assist learning of other grammatical forms.

Method

Forty adults (20 men, 20 women) were familiarized with examples of masculine and feminine Russian words. Half of the participants were familiarized with 32 different root words in a high-variability condition. The other half were familiarized with 16 different root words, each repeated twice for a total of 32 presentations in a high-repetition condition. Participants were tested on untrained members of the category to assess generalization. Familiarization and testing was completed 2 additional times.

Results

Only participants in the high-variability group showed evidence of learning after an initial period of familiarization. Participants in the high-repetition group were able to learn after additional input. Both groups benefited when words included 2 cues to gender compared to a single cue.

Conclusions

The results demonstrate that the degree of input variability can influence learners' ability to generalize a grammatical subcategory (noun gender) from a natural language. In addition, the presence of multiple cues to linguistic subcategory facilitated learning independent of variability condition.

A major theoretical issue concerning language learning is how to structure the input in ways that best facilitate learning. In the present experiment, we investigated whether input variability might facilitate the learning of grammatical word forms in an unfamiliar language. A growing body of research has found that increasing the variability in the input to learners is beneficial for the learning of various linguistic skills, including word learning (Perry, Samuelson, Malloy, & Schiffer, 2010), speech sound categorization (Maye, Werker, & Gerken, 2002; Sadakata & McQueen, 2013), and reading (Apfelbaum, Hazeltine, & McMurray, 2013). Variability in spoken input has also been found to facilitate grammar learning and promote generalization of the grammar (Gómez, 2002; Torkildsen, Dailey, Aguilar, Gómez, & Plante, 2013). The effect of variability on learning grammatical structures was demonstrated in a seminal study by Gómez (2002), who examined the unguided learning of nonadjacent dependencies using an artificial language. A nonadjacent dependency describes a relation between two elements separated by one or more intervening elements. This type of dependency has many counterparts in natural language. For example, the English present progressive verb tense (e.g., is singing) involves a nonadjacent dependency between is and ing (Santelmann & Jusczyk, 1998). Gómez (2002) used similar aXb grammatical forms in which the element a (a nonword) predicts the element b (a different nonword) with any number of nonwords (X) occurring between the a and b elements. The variability of input was manipulated by drawing the middle element (X) from a pool of two, six, 12, or 24 different X elements. The results showed that the high-variability stimulus set facilitated learning in both typical adults and infants, as learning only occurred for those provided with the most diverse set of X elements (24 X exemplars). Grunow, Spaulding, Gómez, and Plante (2006) replicated this effect for typical adults contrasting 12 versus 24 different X elements in using the same aXb grammar. R. Gómez and Maye (2005) documented the learning trajectory of nonadjacent dependencies and found that the benefit of variability extended to even younger infants. Their study found that a minimum of 18 different X elements was needed to promote learning by infants.

As illustrated by these studies, the beneficial effect of variability has been repeatedly demonstrated for nonadjacent dependency learning. Moreover, a recent study by Torkildsen et al. (2013) investigated whether the variability effect seen for nonadjacent dependency learning would generalize to a grammar that is defined by dependencies between adjacent elements. They found that the positive effect of variability did in fact extend to the two grammatical forms used in their experiment. Their study used aX and Yb grammatical forms, in which a single a element preceded either three or 24 different X elements and a single b element followed either three or 24 Y elements. The study showed that typical learners could acquire this simple grammar even with low input variability, but that learners with a language-based learning disability only learned under the high-variability condition (Torkildsen et al., 2013). Torkildsen et al. (2013) suggested that the high-variability input facilitated a shift in learning strategy in this group, from focusing on exemplar-specific features to noticing the grammatical structure of the language.

One explanation for the variability effect is that when input variability is high, both adult and infant learners seek the invariant structure in the input (Gómez, 2002). Despite surface differences, the grammars used in Gómez (2002) and Torkildsen et al. (2013) have some important similarities in terms of their variable and invariant elements. In both studies, the grammatical form was defined by the distributional relations between the invariant elements and the variable elements, and learning required recognizing the positional relationship between these elements. However, there are other types of distributional relations in language input that differ conceptually from the artificial grammars used by Gómez (2002) and Torkildsen et al. (2013).

One such example is the noun gender marking systems that are present in many natural languages. In this article, we focus on gender categorization in Russian. Gender categorization of nouns in Russian, as in other languages, requires the coordination of lexical information, phonological characteristics, and grammatical morphemes (Corbett, 1982). Different theories have sought to explain how learners acquire syntactic category information. Some stress the importance of referential information (Pinker, 1984), whereas others point to the fact that natural languages include multiple cues to category membership (e.g., nouns) and subcategory membership (e.g., gendered nouns), which are both referential and nonreferential (Braine, 1992). There is growing evidence that distributional information plays a significant role in category and subcategory formation, and studies have shown that nonreferential cues alone are sufficient for learning to occur (Frigo & McDonald, 1998; McDonald & Plauché, 1995; Redington, Chater, & Finch, 1998). This is also true for the noun gender subcategories present in Russian language (Akhutina, Kurgansky, Polinsky, & Bates, 1999; Brooks, Kempe, & Donachie, 2011; Gerken, Wilson, & Lewis, 2005; Richardson, Harris, Plante, & Gerken, 2006; Rodina & Rodina, 2012). Because gender subcategories can be learned from distributional, nonreferential information alone, it seems that this learning problem might be served by a statistical learning mechanism (Saffran, 2003), which is particularly sensitive to the distributional patterns of the input. If so, then a high degree of variability in the input should facilitate subcategory learning, as it does the learning of grammatical strings (e.g., aX or aXb grammars).

In the present experiment, we investigate the learning of noun gender subcategories in Russian language using two contrasting variability conditions, while holding frequency of presentation constant. We ask if the beneficial effect of input variability demonstrated by Gómez (2002) and Torkildsen et al. (2013) extends to the learning of noun gender subcategories in a natural language. A subset of the Russian gender marking system serves as the stimuli for this experiment. Similar noun gender paradigms have been used by Gerken et al. (2005) and Richardson et al. (2006). The paradigm in the present experiment consists of lexical word stems that are paired with either single- or double-marked suffixes that are gender specific. For single-marked words, if a root word can take the masculine suffix (e.g., –ya), it can also take the alternate masculine suffix (–yem) and vice versa. Likewise, single-marked feminine nouns take either –oj or –u. The double-marked words are gender marked by two suffixes that always co-occur (e.g., –tel + –ya). If a root word takes the masculine double marking –telya, it can also take the alternate marking –telyem. Feminine double-marked words combine the phoneme –k with the feminine suffixes (i.e., –koj and –ku).

Learning the Russian gender categorization requires the detection of these invariant features, that is, the relationship between the two alternate markings (e.g., –yem and –ya vs. –u and –oj) and the set of root words that take one or the other set of suffixes. When words are single-marked, inferring this relationship is critical to learning. When words are double marked, there are two sets of distributional cues available. One is the relationship between the two suffixes that constitute the double-marked form (i.e., –tel is followed by either –ya or –yem, but –k is followed by either –u or –oj). This relationship between elements can be considered an adjacent dependency, which is considered easier to learn than are nonadjacent dependencies (Newport & Aslin, 2004). In addition to this relationship between suffixes, the combination of suffixes also shares the same relationship between the root word and its two alternate markings (i.e., –telyem and –telya vs. –ku and –koj). Therefore, by testing learning of both the single-marked and double-marked gender subcategories, we can test sensitivity to the word + suffix-class cue alone or in combination with the additional cue offered by the adjacent dependency represented by the double markings.

Studies on category learning have found that the presence of multiple cues facilitates the learning process (Frigo & McDonald, 1998; McDonald & Plauché, 1995; Mintz, 2002). In their artificial grammar learning experiment, McDonald and Plauché (1995) found that correlated pairs of cues assisted learning as long as both cues were fairly transparent as markers. The effect of multiple cues has also been demonstrated in subcategory learning in Russian. In Gerken et al. (2005) and Richardson et al. (2006), double-marked words were easier to learn relative to single-marked words. Double-marked words include correlated cues by including two grammatical morphemes that both signal subcategory membership. In addition, one of these cues (the within-word pairing of the two gendered suffixes) represents a form that is relatively easy to learn by typical learners (e.g., Torkildsen et al., 2013). Therefore, double-marked words convey more grammatical information concerning gender subcategory than do single-marked words, and should be easier to learn than single-marked words.

High-variability input is thought to promote attention to the invariant features of the input (in this case, the relationships between gendered suffixes) and make the distributional relationships that define the grammar more salient to the learner. Torkildsen et al. (2013) suggested that the learners with language-impairments in their study tended to use inefficient learning strategies, but that high-variability input facilitated successful statistical learning in this group. In contrast, the learners without impairments in Torkildsen et al. (2013) managed the same learning task in a low-variability condition, with as few as three exemplars repeated multiple times. Therefore, a large number of different exemplars might not be necessary to detect the pattern that the alternate suffixes (e.g., –ya and –yem) are interchangeable, particularly given that the participants in our experiment all have normal language skills. If these learners correctly generate a hypothesis or rule for the assignment of gendered suffixes on the basis of the input, they should show an effect of learning with relatively few unique exemplars. Two previous studies have demonstrated that both infants and adults can form these associations between Russian nouns and their alternate gender markers with relatively few exemplars (Gerken et al., 2005; Richardson et al., 2006). In these two studies, participants were exposed to just 12 lexical root words, which could be considered low variability relative to the levels associated with learning other grammatical strings (Gómez, 2002; Grunow et al., 2006). However, within the present experiment, we expect the previously demonstrated variability effect to generate stronger learning in the high-variability condition relative to a low-variability condition in which the exemplars are repeated to equate frequency across learning conditions.

In contrast to previous variability studies, here we use natural language stimuli instead of an artificial grammar, in order to increase the ecological validity of the experiment. Our main research objective is to investigate whether input variability facilitates subcategory learning in natural language as it does for the other types of grammatical forms studied to date (cf. Gómez, 2002; Torkildsen et al., 2013). To do so, we conducted a two-part experiment. The first part (Part A) tested early learning under high- and low-variability conditions. This was followed immediately by Part B in which additional opportunities for learning were provided. We hypothesized that presentation of a larger number of lexical items, with their gendered suffixes, would result in better learning compared to presentation of fewer words, each of which was repeated during the learning period. We expected the high-variability group to learn rapidly and to achieve better early learning than the high-repetition group. This was tested in Part A of the present experiment. To explore the effect of learning over time, our experimental design included three successive cycles of familiarization and testing for both groups. We propose that learning with low-variability input is not impossible, but requires more time relative to learning from high-variability input. This was tested in Part B of the present experiment.

Our final hypothesis is consistent with the findings of Gerken et al. (2005) and Richardson et al. (2006) in which double-marked words were easier to learn relative to single-marked words. In accordance with these results, we hypothesized that both the high-repetition group and the high-variability group would show a better test performance for double-marked words compared to single-marked words, due to the presence of correlated cues to subcategory membership. This should be true at all stages of learning, and therefore should be apparent in both Part A and Part B of the present experiment.

Part A

Method

Participants

Forty graduate students at the University of Bergen, Norway, were included in the experiment, 20 men (M = 23;6 [years;months]; SD = 3;3) and 20 women (M = 24;0; SD = 7;7). All participants were required to have no knowledge of Russian or to languages similar to Russian (e.g., other Slavic languages), to have no self-reported difficulties with language learning in general, no known hearing loss, and no self-reported developmental or acquired neurological disorders.

The participants were assigned to the two conditions (high variability and high repetition) by pseudorandomization such that equal numbers of men and women were assigned to each condition. Two additional potential participants were excluded. One was excluded due to significant exposure to a Slavic language, and another was excluded due to oversampling of female participants. The latter participant was randomly selected from the female participant pool in order to obtain gender-balanced groups. This resulted in a final participant set of 40 students.

The language background of our participants may influence the relative ease in learning a gender marking system, even though no participant spoke Russian or another Slavic language. It is important to know that Norwegian marks three subcategories of noun gender (masculine, feminine, and neuter) by using a gendered article (en lastebil, a truck), a noun suffix (lastebilen, the truck), or both (den lastebilen, that truck). In addition, knowledge of other languages may make learners more sensitive to the variety of ways distributional cues occur across languages in general. It is common for Norwegians to be multilingual. Norwegian, Swedish, and Danish are generally mutually understandable by speakers of any one of these languages. English is introduced in first grade as an obligatory second language, and in addition, a majority also chose an elective third language during elementary school. We surveyed participants concerning their language backgrounds beyond these three languages. All but two participants (both assigned to the high-repetition group) spoke Norwegian natively. The majority of the participants (20 in the high-repetition group and 19 in the high-variability group) spoke two or more languages in addition to their native language. The highest number of additional languages spoken was four. One participant (in the high-variability group) spoke only one language in addition to the native language.

Stimulus Material and Experimental Design

The present experiment was designed to manipulate variability in a Russian gender paradigm by increasing the number of root words elements presented in the high-variability condition (32 root words) relative to the high-repetition condition (16 root words repeated twice each). Note that despite the differences in the number of root words in the high-variability and high-repetition conditions, participants in each condition heard the same total number of stimulus presentations during the experiment. Even though the Russian noun gender system in full includes three subcategories (masculine, feminine, and neuter; Corbett, 1991), only the subsets of masculine and feminine gender markings were included in the present experiment.

Similar Russian stimuli have been used by Gerken et al. (2005) and Richardson et al. (2006). In the present experiment, however, we included more words in the initial familiarization phase compared to previous studies. This was necessary in order to make the difference between the high-repetition condition and the high-variability condition more prominent. During pilot work, we discovered that the increase in the total number of words relative to previous studies increased the overall difficulty of the task. Therefore, we broke the familiarization phases of both the high-variability and the high-repetition listening conditions into two components (see the Appendix for stimulus sets). In Phase I, we familiarized listeners with the general pattern of gender marking. In Phase II, we familiarized listeners with the specific lexical items they would be tested on, but not with the gender markers used during testing. This was followed immediately by a test phase, which used the lexical roots heard during Phase II paired either with correct or incorrect gendered suffixes. In order to perform well on the test phase, the participant had to detect the relationship between gendered suffixes and how they are paired with root words in Phase I, and then note the specific word–suffix pairings in Phase II in order to predict the alternate pairings during the Test phase.

Phase I Familiarization Stimuli

The items presented in Phase I familiarization were all grammatically correct words presented by either a male or female native speaker of Russian. These are listed in Appendix Table 1A. Previous studies have demonstrated that multiple talkers compared to single talkers facilitate word learning (Richtsmeier, Gerken, Goffman, & Hogan, 2009; Rost & McMurray, 2009, 2010). Motivated by these findings, the auditory material in the two familiarization phases were recorded by two different talkers, both of whom were native speakers of Russian. The sound files for these recordings were edited to correspond to the actual length of the word (silent intervals were removed) and to produce approximately equal loudness across words.

High-variability stimuli. The high-variability stimuli consisted of 32 different root words and eight different combinations of gender markings (suffixes). Each root word was combined with two possible suffixes, resulting in a total of 64 unique combinations of root words and gendered suffixes. The suffixes could be either single or double, and served to mark words as either masculine or feminine. There were two variations of feminine–single markings (–oj or –u), two feminine–double markings (–k + –oj or –k + –u), two masculine–single markings (–ya or –yem) and two masculine–double markings (–tel + –ya or –tel + –yem). As illustrated, double-marked words were gender marked by two morphophonological units. Any root word that could take one of the two possible masculine endings (e.g., konya) could also take the other masculine ending (e.g., konyem). This was true for both masculine and feminine, and for single- and double-marked words. Due to a coding error, an extra masculine single-marked word (male talker) was included and a masculine double-marked word was excluded (male talker).

High-repetition stimuli. The stimulus set consists of a subset of words used in the high-variability condition. It contains half the number of root words (16 root words), making up a total of 32 unique items. Each unique item was repeated twice for a total of 64 stimulus presentations.

Phase II Familiarization Stimuli

The Phase II familiarization stimuli, listed in Appendix Table A2, were identical for both the high-variability and the high-repetition conditions. All items were grammatically correct Russian words. The stimuli consisted of 12 root words that had not been heard during Phase I familiarization. These root words were combined with only one of each pair of single or double gender-marked suffixes. The Phase II familiarization items were equally divided between masculine/feminine and single/double markings. All items were recorded by the native Russian speakers heard during Phase I.

Test phase. The stimulus set for the test phase consisted of 24 items (see Appendix Table A3). Half of the items were grammatically correct (i.e., grammatical items), but the other half violated the grammatical pattern that the participants were exposed to during Phase I and Phase II familiarizations (i.e., ungrammatical items). The root words were the same as those in Phase II familiarization. However, the correct items in the test phase used the alternate gender markings. The test items were equally split between masculine/feminine and single-/double-marked forms. The words were recorded by a female native speaker of Russian. This talker was a different woman than was heard in Phase I, and she was only heard during the test phase.

Procedures

The experiment took place in a research lab at the University of Bergen, and the research procedures were approved by the Norwegian Social Science Data Services. Participants were tested individually. After signing the informed consent form, they completed the experimental tasks. Participants were seated in front of a laptop, asked to put on headphones, and given instructions on how to adjust the volume. The researcher then stepped out of the room, but remained ready to aid with technical issues if necessary. The experiment consisted of two familiarization phases (Phase I and Phase II) and a test phase. Participants received written instructions on the computer screen in advance of each phase. The experiment lasted for about 7 min. Participants were given monetary compensation of NOK 150 (approximately $25 US) for taking part in the experiment. Stimulus presentation and data collection were performed with E-Prime 2.0 Professional (Schneider, Eschmann, & Zuccolotto, 2002).

Phase I Familiarization

Phase I familiarization stimuli for both the high-variability and the high-repetition condition consisted of an equal number of items presented in a pseudorandomized order (block randomization). Each block consisted of 16 items with a 300–580-ms interstimulus interval. In both high-variability and high-repetition conditions, each block included four root words presented with each of the two possible suffixes. The intention of this blocked arrangement was to highlight the fact that a root word heard with the ending –oj also could take the ending –u, and vice versa. Likewise, both possible masculine–double markings –telyem and –telya were heard with the same root word during the same block. This also applied to the two alternate feminine–double markings –ku and –koj. Not more than two of the same root words appeared consecutively.

In the high-repetition condition, two consecutive blocks included the same root words. That meant that the root word (e.g., dushi) was presented eight times within two consecutive blocks. In the high-variability condition, each root word only appeared within one block. For both conditions, the Phase I familiarization set included a total of 256 items (eight blocks of stimuli heard twice). This was intended to provide significant initial exposure to promote early learning.

Phase II Familiarization

In Phase II familiarization, the participants heard 12 unique root words with one of the possible gender markings. Each root and suffix pairing was heard twice in a pseudorandomized order, for a total of 24 items.

Test phase. During the test phase, the participants heard another 24 individual items and were asked to judge each item as a grammatically correct or incorrect Russian word. Participants were asked to click on the symbol of a smiley face on the computer screen to mark a word as correct or a frowny face for incorrect in order to proceed to the next test item.

Results

The dependent variables for the learning conditions were the rate at which test items were accepted as correct and reaction time for test items. Acceptance rate is typically used in studies of this type in which learning is defined as significantly higher acceptance rates of grammatical items versus ungrammatical items. This method of analysis controls for response bias (i.e., an underlying bias towards accepting or rejecting items overall). Acceptance rate data for each item type and condition are presented in Table 1. To determine if the overall pattern of response for grammatical and ungrammatical test items was nonrandom, we tested each item type against chance performance (one-sample t test). Acceptance of grammatical items was above chance for the high-variability group, M = 8.7, t(1, 39) = 6.90, p = .000001, d = 1.54, and for the high-repetition group, M = 7.25, t(1, 39) = 3.26, p < .00205, d = 0.73. In contrast, acceptance rates for ungrammatical items were not significantly below chance for either group (high-variability group: M = 5.35, t(1, 39) = 1.55, p < .068, d = 0.35; high-repetition group: M = 6.55, t(1, 39) = 1.47, p < .921, d = −0.33).

Table 1.

Descriptive statistics for acceptance rate data (accepted grammatical vs. ungrammatical items) for single- and double-marked nouns for Part A for both the high-variability condition (n = 20) and the high-repetition condition (n = 20).

Test item type	High variability		High repetition
	M	SD	M	SD
Grammatical, double-marked	4.60	(1.05)	4.30	(1.17)
Ungrammatical, double-marked	2.70	(1.30)	3.05	(1.39)
Grammatical, single-marked	4.10	(1.37)	2.95	(1.28)
Ungrammatical, single-marked	2.65	(1.57)	3.50	(1.10)

The pattern of responses across item types was analyzed statistically with a mixed analysis of variance (ANOVA) with planned comparisons for specific hypotheses. A 2 × 2 × 2 × 2 mixed ANOVA was conducted with Group Condition (high repetition vs. high variability) and Participant Sex (male vs. female) as between-groups factors. Noun Marking (single vs. double) and Grammaticality (grammatical vs. ungrammatical test items) were within-group factors.

We found a significant main effect for Grammaticality, in which participants in general had more accepted grammatical items than ungrammatical items, and hence showed an effect of learning, F(1, 36) = 27.554, p = .000007, η²_p = .433. The effect of Grammaticality also interacted signficantly with Noun Marking (single- vs. double-marked), F(1, 36) = 7.305, p = .010, η²_p = .169. However, the main effect for Noun Marking was not significant F(1, 36) = 2.586, p = .1166, η²_p = .067.

There was also no main effect of Group Condition (high variability vs. high repetition), F(1, 36) = 0.107, p = .745, η² = .003, but we found a statistically significant interaction effect of Group Condition (high variability vs. high repetition) × Grammaticality (accepted grammatical vs. ungrammatical items), F(1, 36) = 11.797, p = .002, η²_p = .247. The three-way interaction of Group Condition × Grammaticality × Noun Marking was nonsignificant, F(1, 36) = 2.63, p = .1136, η²_p = .068.

The ANOVA revealed no main effect or interaction effects of participant sex (male vs. female), F(1, 36) = 1.48, p = .231, η² = .04, and none of the remaining interaction effects reached the level of statistical significance (Sex × Group Condition: F[1, 36] = 2.69, p = .110, η²_p = .069; Sex × Noun Marking: F[1, 36] = 0.003, p = .956, η²_p < .001; Sex × Grammaticality: F[1, 36] = 3.062, p = .089, η²_p = .078; Sex × Noun Marking × Grammaticality: F[1, 36] = 0.90, p = .766, η²_p = .002).

To explain the significant two-way Group × Grammaticality interaction in light of our hypotheses that high variability would facilitate learning, we conducted a series of planned comparisons. Recall that learning is defined as significantly higher acceptance rates for grammatical versus ungrammatical test items. A dependent-samples t test confirmed the hypothesis that participants in the high-variability condition accepted grammatical items significantly more frequently than ungrammatical items, and hence showed an effect of learning, t(1, 19) = 5.55, p = .00002, d = 1.79. For the high-repetition group, however, grammatical items were not accepted significantly more frequently than ungrammatical items, and hence this group did not show an effect of learning, t(1, 19) = 1.41, p = .176, d = 0.41. These effects are illustrated in Figure 1.

Figure 1.

Mean number of items accepted as grammatical versus ungrammatical for the high-variability group (n = 20) and the high-repetition group (n = 20) for Part A.

Previous research suggested that gender categories were easier to learn for double-marked nouns compared to single-marked nouns. To address this further we performed planned comparisons related to the significant Grammaticality × Noun Marking interaction. As hypothesized, a dependent-samples t test showed that there were significantly more accepted grammatical items than ungrammatical items for double-marked nouns, t(1, 39) = 5.73, p = .0000005 one-tailed, d = 1.17. The difference between the number of accepted grammatical versus ungrammatical items for the single-marked nouns did not reach the level of statistical significance, t(1, 39) = 1.37, p = .089 one-tailed, d = 0.32 (see Figure 2).

Figure 2.

Mean number of items accepted as grammatical versus ungrammatical for double-marked versus single-marked words for Part A. Both group conditions (high variability and high repetition) are included (n = 40).

Brief Discussion

The findings in Part A revealed that participants in the high-variability group accepted more grammatical items than ungrammatical items, and hence showed an effect of learning. In contrast, participants in the high-repetition condition did not accept more grammatical items than ungrammatical items. Therefore, the results showed that this input did not facilitate learning, and that the relatively few exemplars provided were not sufficient for participants to generalize the grammatical structure. In other words, the findings confirmed our hypothesis by revealing a substantial difference in learning outcome depending on the structure of the input. Our hypothesis regarding single- and double-marked nouns was also supported, in that participants learned the double-marked nouns (marking included two morphological cues) relative to the single-marked nouns (only one morphological cue).

Consistent with Gómez (2002), we suggested that the high number of different elements that were free to vary (root words) in the high-variability condition would make the invariant grammatical structure more salient and therefore easier to detect. Compared to the high-variability group, the high-repetition group heard half the number of root words with twice as many exposures to each exemplar. We suggest that this way of structuring the input, favoring repetition over variability, might have directed the participants' attention towards other properties of the input than the underlying distributional patterns. As a result, participants in the high-repetition condition did not generalize the grammatical structure.

In order to explore whether the two groups would benefit from more exposure to the same stimuli, we re-exposed both groups to the Russian stimuli (referred to as Part B). Our main hypothesis in Part B was that the high-repetition group would eventually learn the grammar categories if given more exposure to the input, even if variability remained relatively low.

There are other possible outcomes, which could apply to either the high-repetition or the high-variability group, or both. In standard statistical learning studies, adults are familiarized with input and then tested. They are not typically re-exposed to the stimuli or retested. This, in part, reflects a concern that the act of testing participants could change how they listen to subsequent input. Having been exposed to test items, for which individuals explicitly consider the structure of the input, may change the participants' approach to learning during re-exposure to the familiarization input. Therefore, more input could promote more learning, but it may not be the same manner of learning that occurred during the initial familiarization phase.

It is also the case that exposure to the test phase itself might change the participants' representation of the grammar as they proceed into a subsequent re-exposure to the input. Romberg and Saffran (2013) showed that statistical learning improved with length of exposure to the input. However, these authors also showed that the act of testing biased the participants' representation of the grammatical structures learned. Their participants were exposed to an artificial language that contained both an adjacent dependency (aX) and a nonadjacent dependency (aXb). If participants were tested on the adjacent dependency first, their representation of the nonadjacent dependency appeared to have been eroded (Romberg & Saffran, 2013). It may also be the case that the incorrect items heard during the test phase could erode the participants' implicit representations. For example, Gómez and Lakusta (2004) showed that infant learners could only tolerate 33% counterexamples to a grammar before they no longer showed evidence of learning. Our test phase contained 50% counterexamples (i.e., ungrammatical items). Although the total exposure to ungrammatical items was quite low overall in the first cycle (4% of the total input), the concentrated block of these items during testing could potentially erode an implicit representation. If the participants are engaged in implicit, statistical learning, then input is input, regardless of whether it occurred during the familiarization phases or the test phase. The testing block with the high proportion of counterexamples, heard just before the next round of input, could be particularly problematic. If so, we could see a reduction in learning in subsequent cycles of familiarization and test phases.

In order to gauge the extent to which participants thought they were using implicit versus explicit processes during Parts A and B, we administered a brief self-rating questionnaire concerning the basis of their judgments during the test phases. If one group was relying more on explicit learning processes, they should be more likely to report information concerning the grammatical regularities that define Russian gender marking.

Part B

Methods

Participants

Part B was conducted immediately following Part A. The same 40 graduate students (20 men, 20 women) who completed Part A also participated in Part B. The participants were assigned to the same conditions (high variability and high repetition) as previously.

Materials and Procedures

The stimulus set used in Part A was also implemented in Part B. The procedure was generally the same as in Part A, consisting of two familiarization phases (Phase I and Phase II) and a test phase. In order to measure a possible effect of learning after several exposures, the procedure used in Part A was completed two more times in Part B (referred to as Cycle 2 and Cycle 3). In Part A, Phase I familiarization consisted of 256 items (128 items in eight blocks heard twice). This was intended to provide significant initial exposure to promote early learning. However, pilot work indicated that this amount of input was not required in all three cycles to promote learning, and participants tended to fatigue or become bored with this amount of input. Therefore, Cycles 2 and 3 contained only one presentation of Phase I familiarization items (128 items total in eight blocks of stimuli).

After the final cycle, a self-rating questionnaire was administered. The primary objective was to obtain a self-reported measure of the participants' awareness of the rules for gender marking at the conclusion of the experiment. Participants were asked to rate how random they perceived their responses to be (1 = not random, 4 = random to a very large extent), the degree to which the stimuli followed rules (1 = followed rules, 4 = did not follow rules), their own degree of awareness of patterns in the input (1 = aware to a very large extent, 4 = not aware), and the degree to which they thought that any patterns they found might be correct (1 = very sure, 4 = not at all sure). They were also offered an opportunity to provide qualitative information concerning their perceptions. In addition, the questionnaire was used to gather information about the participants' second language skills reported in Part A, above. In particular, we asked which languages and how many languages they knew. We were interested in whether greater experience with different languages in general might facilitate learning.

Results

Acceptance rate data for each condition for Cycles 2 and 3 are presented in Table 2. As in Cycle 1, acceptance of grammatical items was above chance for the high-variability group in both Cycle 2, M = 7.95, t(1, 39) = 4.72, p = .0002, d = 1.05, and Cycle 3, M = 8.55, t(1, 39) = 5.05, p = .00007, d = 1.13. Likewise, the high-repetition group accepted grammatical test items more frequently than chance in Cycle 2, M = 8.00, t(1, 39) = 5.31, p = .00004, d = 1.19, and Cycle 3, M = 8.05, t(1, 39) = 4.81, p = .0001, d = 1.08. In contrast, acceptance rates for ungrammatical items were not significantly below chance for either group in Cycle 2 (high-variability group: M = 6.35, t(1, 39) = −0.84, p < .5866, d = −0.81; high-repetition group: M = 6.40, t(1, 39) = −0.89, p < .921, d = −0.20) or in Cycle 3 (high-variability group: M = 6.95, t(1, 39) = −2.37, p < .9716, d = −0.53; high-repetition group: M = 6.95, t(1, 39) = −1.96, p < .9357, d = −0.44).

Table 2.

Descriptive statistics for acceptance rate data (accepted grammatical vs. ungrammatical items) for single- and double-marked nouns for Part B for both the high-variability condition (n = 20) and the high-repetition condition (n = 20).

Test item type	High variability		High repetition
	M	SD	M	SD
Cycle 2
Grammatical, double-marked	3.85	(2.00)	4.40	(0.94)
Ungrammatical, double-marked	2.95	(1.00)	2.90	(1.33)
Grammatical, single-marked	4.10	(2.00)	3.60	(1.64)
Ungrammatical, single-marked	3.40	(0.00)	3.50	(1.32)
Cycle 3
Grammatical, double-marked	4.50	(2.00)	4.40	(1.05)
Ungrammatical, double-marked	3.15	(1.00)	3.25	(1.52)
Grammatical, single-marked	4.05	(2.00)	3.65	(1.14)
Ungrammatical, single-marked	3.80	(2.00)	3.70	(1.22)

The pattern of responses across item types was analyzed statistically with an ANOVA with planned comparisons for specific hypotheses. A 2 × 2 × 2 mixed ANOVA was conducted with Group Condition (high repetition vs. high variability) as a between-groups factor. Noun Marking (single- vs. double-marked) and Grammaticality (grammatical vs. ungrammatical test items) and Cycle (Cycle 2 vs. 3) were within-group factors. Because of the lack of any significant main or interaction effects for sex, we dropped this from the analyses of Cycles 2 and 3.

We found a significant main effect for Grammaticality, in which participants in general had more accepted grammatical items than ungrammatical items, and hence showed evidence of learning, F(1, 38) = 19.761, p = .00007, η²_p = .3421. As in Cycle 1, the effect of Grammaticality also interacted significantly with Noun Marking (single- vs. double-marked), F(1, 38) = 12.186, p = .0012, η²_p = .243. However, the main effect for Noun Marking was not significant F(1, 38) = 0.125, p = .7257, η² = .0033.

As in Cycle 1, there was no main effect of Group Condition (high variability vs. high repetition), F(1, 38) = 0.051, p = .823, η² = .0013. Unlike Cycle 1, we did not find a statistically significant interaction effect of Group Condition (high variability vs. high repetition) × Grammaticality (accepted grammatical vs. ungrammatical items), F(1, 38) = 1.53, p = .2237, η²_p = .0387. The three-way interaction of Group Condition × Grammaticality × Noun Marking was also nonsignificant, F(1, 38) = 1.352, p = .2522, η²_p = .0344.

The ANOVA revealed a main effect for Cycle (Cycle 2 vs. Cycle 3), F(1, 38) = 6.282, p = .01659, η²_p = .1419, with participants generally accepting more items in Cycle 3 compared to Cycle 2. None of the remaining interaction effects involving Cycle were statistically significant, Cycle × Group: F(1, 38) = 0.698, p = .409, η²_p = .0180; Cycle × Noun Marking: F(1, 38) = 0.315, p = .5781, η²_p < 0.0082; Cycle × Grammaticality: F(1, 38) = 0.378, p = .5420, η²_p = .0099; Cycle × Noun Marking × Grammaticality: F(1, 38) = 0.615, p = .4380, η²_p = .0159; Cycle × Noun Marking × Group Condition: F(1, 38) = 0.140, p = .7105, η²_p = .0037; Cycle × Grammaticality × Group Condition: F(1, 38) = 0.378, p = .5420, η²_p = .0099; and Cycle × Noun Marking × Grammaticality × Group Condition: F(1, 38) = 1.517, p = .0.2256, η²_p = .0384.

Our primary hypothesis was that the high-repetition group would show learning if provided with additional exposure to the Russian stimuli. We addressed this hypothesis with a series of planned comparisons. We conducted dependent-samples t tests for grammatical versus ungrammatical test items for Cycles 2 and 3. The tests revealed that participants in the high-repetition group showed an effect of learning for both Cycle 2, t(1, 19) = 3.24, p = .004, d = 0.80, and maintained it during Cycle 3, t(1, 19) = 2.29, p = .034, d = 0.51. To confirm that the high-variability group also maintained their learning, we repeated these comparisons with the high-variability group. This group accepted more grammatical than ungrammatical items for Cycle 2, t(1, 19) = 2.96, p = .008, d = 0.86, and Cycle 3, t(1, 19) = 2.40, p = .027, d = 0.71. These results are graphed in Figure 3.

Figure 3.

Mean number of items accepted as grammatical versus ungrammatical for the high-variability group (n = 20) and the high-repetition group (n = 20) for Part B.

Correlations between behavioral performance and survey responses are presented in Table 3. The degree to which participants felt their answers were random correlated negatively with acceptance of grammatical items in Cycle 1 (r = −.44, p < .005) and in Cycle 2 (r = −.45, p < .005), but not Cycle 3. The degree to which participants reported searching for regularities in the input correlated negatively with acceptance of ungrammatical items in Cycle 1 (r = −.32, p < .05) and in Cycle 2 (r = −.33, p < .05), but not Cycle 3. There were no significant correlations between the degree to which participants reported that they became aware of patterns in the input and their response accuracy.

Table 3.

Correlational analysis for the self-rating questionnaire and acceptance rate (number of accepted grammatical vs. ungrammatical items) for both the high-variability group and the high-repetition group combined (n = 40).

Questionnaire items	Cycle 1		Cycle 2		Cycle 3
	Grammatical	Ungrammatical	Grammatical	Ungrammatical	Grammatical	Ungrammatical
Age	.15	.04	.03	.10	−.12	−.03
No. of languages	.05	.08	.19	.18	.02	.29
1. Degree of randomness	−.44**	.13	−.45**	−.06	−.28	.10
2. Searched for regularities	.26	−.32*	.05	−.33*	.02	−.13
3. Awareness of regularities	.07	.07	.00	−.16	.13	−.07
4. Correct patterns?	−.23	.13	−.31	−.12	−.14	.01

Note. The questions were as follows: 1. How random were your responses? 2. Did you search for regularities? 3. Did you find any regularities? 4. Do you think that the patterns you might have found are correct? The answers were graded from 1 to 4 (1 = aware/not random/followed rules, 4 = not aware/random to a very large extent/did not follow rules).

^*p < .05;

^**p < .005.

Participants also provided qualitative input on their experiences. Twenty-nine of the 40 participants (15 from the high-repetition group and 14 from the high variability group) reported that their attention had been drawn towards the suffixes, but they did not explain this any further. Another four participants (three from the high-variability group and one from the high-repetition group) could identify the patterns they had recognized. One participant in the high-variability group thought that suffixes such as –koj were correct, another recognized words that ended on –u and –oj, and the third participant recognized suffixes such as –koj and –ilyem [sic]. The participant in the high-repetition group reported recognizing endings such as –a [sic], and –m [sic], and –ei [sic], and –u. None of the participants could explicitly explain the rules governing the gender suffixes. Furthermore, we found that the participants who reported that they had searched for and found patterns in the stimuli, reported that they did not start doing so until Cycle 2 (e.g., “in the first round … I made choices based on what sounded right or not. In the later rounds, I listened for specific details in words” and “I first began to look for patterns in the second round”). This suggests that at least some participants were applying explicit strategies during Part B. The majority of participants in both groups reported that they believed that they performed best in the third round of testing and that this was because they then remembered or recognized the words better. Two participants in the high-variability condition commented on the issue of test contamination. One reported that the repeated exposures made it hard to distinguish the incorrect words heard during previous testing from the correct stimuli heard during familiarization. Another participant reported that after a while he/she wondered whether all the test items were correct.

Brief Discussion

Participants in the high-repetition group showed an effect of learning by accepting more grammatical items than ungrammatical items in Part B. This confirmed our hypothesis that the high-repetition participants would learn after prolonged exposure to the low variability input. Participants in the high-variability group also maintained learning for Cycles 2 and 3, accepting more grammatical items than ungrammatical items.

Results from the questionnaire revealed that at least some participants began to have an explicit awareness of the rules after the repeated exposures. Moreover, participants who reported some level of awareness of patterns were also stronger performers. However, it is important to recall that participants did not provide this information until after the final cycle. Therefore, some caution is warranted concerning the direction of this relation. Although it may be that explicit awareness drove better learning, it is also possible that those who were initially strong implicit learners were more likely to later develop an explicit awareness of the patterns in the input. Indeed, the fact that most participants indicated that any awareness of patterns did not occur until after Cycle 1 is consistent with the latter explanation.

General Discussion

The main objective of this experiment was to investigate whether high-variability input would facilitate learning of the Russian gender marking system. The results from Part A clearly show that increasing lexical variability facilitates rapid implicit learning of subcategory membership. This finding is in accordance with previous studies demonstrating the beneficial effect of variability on learning other artificial grammatical constructions (Gómez, 2002; Torkildsen et al., 2013). Likewise, we replicated an effect of noun marking, with better performance on double-marked words relative to single-marked words.

Part B extended the learning period to determine whether performance would change with additional input. We expected that the high-repetition group would demonstrate learning with additional input. However, we raised the issue that the test phase presented during Part A might alter the participants' implicit representation of the categories due to exposure to incorrect test items (cf. Gómez & Lakusta, 2004; Romberg & Saffran, 2013). Despite this concern, participants in the high-variability group also maintained their learning for Cycles 2 and 3. Moreover, the high-repetition group, which did not show significant learning in Part A, was able to learn with additional input in Part B of this experiment.

Variability and the Presentation of Multiple Cues for Learning

In previous studies, researchers have suggested possible explanations of the effect of variability. Gómez (2002) first suggested that learners seek invariant structure in the input, and that increasing the variability of noninformative elements might make the underlying invariant structure more salient to the learner. In the Russian gender marking system there are two important invariant features. One invariant feature is the relationship between the two alternate markings (e.g., given olenya, the probability that olenyem is also correct is 1.0). In order to discriminate grammatical from ungrammatical test items, the participants have to detect this relationship. In double-marked words there is an additional cue to grammaticality in the relationship between the two suffixes that make up the double marking (e.g., given the suffix –tel, the probability that either –ya or –yem will follow is 1.0). This relationship is independent from the root word–suffix relationship. In the present experiment, we suggest that the high-variability condition served to promote attention to these invariant features of the input and highlighted the distributional relationships that defined the grammar.

Poletiek and van Schijndel (2009) pointed out that the extent to which a set of exemplars fully represents the underlying structure or grammar might be more important to learning than the number of different exemplars presented overall. This would predict that coverage, rather than input variability, should predict learning efficiency. However, in the present experiment, the underlying regularities of the grammar can be fully represented by even fewer unique exemplars than were presented to the high-repetition group. Indeed, the pattern demonstrating that one root word takes each of two correct alternate markings was present in the input to both groups.

Recall also that stimuli provided to both groups were structured in blocks to highlight the pattern of suffix use. Taraban (2004) found that blocking of learning trials was one way of drawing learners' attention to the defining morphemes, and that this facilitated gender-like category induction in an artificial language. Although he also manipulated the size of the lexicon (22 vs. eight noun-like words), this did not have a statistically significant effect on learning the artificial grammar, although the trend was in favor of the higher variability input. In this particular grammar, subcategory was indicated by an unmarked noun-like word that was followed by a gendered preposition. It is possible that the particular method of gender marking or the total number of exemplars used by Taraban (2004) made learning more difficult in general compared with the Russian gender system studied in the present experiment.

The only thing that differed between the high-repetition stimuli and the high-variability stimuli in Phase I familiarization was the number of different root words heard. Both conditions (high variability and high repetition) included the same number of presentations of each specific single- and double-marking (e.g., –oj and –telyem). For the high-repetition input, the ratio of the presentation of root words compared to gender markings was 4:1. For the high-variability input this ratio was 8:1. Plante, Vance, Moody, and Gerken (2013) have suggested that the ratio of invariant to variant features may drive which aspects of the input are learned (e.g., lexical units vs. grammatical patterns). The lower ratio may have directed the high-repetition participants' attention to noninformative elements of the input (e.g., lexical items, which repeated), making learning the distributional aspects of the gender markings less efficient.

The slower learning by the high-repetition group suggests that the two conditions may promote different learning mechanisms. It has previously been proposed that learners might apply different strategies to different types of learning problems (Marcus, 2000; Peña, Bonatti, Nespor, & Mehler, 2002). The present results suggest that the way the input is structured influences whether or not an efficient or robust statistical learning mechanism is initiated, and may even dictate how performance proceeds over time. Although this idea is theoretically intriguing and may even have practical implications for how language interventions might be structured, this idea is in need of further exploration. An important limitation of the present experiment is that, although it highlights differences in the learning trajectory on the basis of input, it is not possible to isolate the source of these differences. Further research in this area might identify both the nature of the learning strategy or strategies adopted when conditions are not optimal for statistical learning, and may even identify conditions under which those alternate strategies are indeed optimal.

Single- Versus Double-Marked Words

Consistent with our hypothesis, performance on double-marked nouns was stronger than on single-marked nouns. This is consistent with the findings in previous studies, where double markings were necessary for generalization to occur in both infants (Gerken et al., 2005) and in adults with a history of language-based learning disabilities (Richardson et al., 2006). Compared with previous studies, we were able to more thoroughly test generalization of the gender marking “rules” to previously unheard word and suffix combinations. Both Gerken et al. (2005) and Richardson et al. (2006) were only able to test generalization to a total of four novel items, compared with 12 novel items tested in the present experiment. Therefore, the present results provide a more stable representation of generalization to untrained items than was previously available.

Several authors have argued that multiple correlated cues assist the learner in successfully utilizing distributional information for category acquisition (Braine, 1966; Frigo & McDonald, 1998; McDonald & Plauché, 1995; Mintz, 2002; Reeder, Newport, & Aslin, 2013). This occurred for double-marked words in that both a specific lexical item and a gendered affix both predicted the final suffix (dushi –tel + –yem). Gerken et al. (2005) and Richardson et al. (2006) suggested that the double set of morphemes that signals the gender subcategory for double-marked words provides redundant cues to category membership. This redundancy facilitates learning of double-marked words. Consistent with this idea, McDonald and Plauché (1995) postulated the idea that if two cues are presented, one highly transparent and one less transparent, the learner might rely solely on the highly transparent cue, and not even notice the other cue. In this sense, the distinct cues provided by the double markings in our experiment might have overshadowed the less transparent cues for the single markings. The association between a single root word and suffix pairs that defines the single-marked subcategory requires learners to attend to patterns across words. This likely makes the single-marked subcategory patterns more difficult to learn than the adjacent dependency that characterized double-marked nouns. This difference may have contributed to the difference in learning single- and double-marked nouns as well.

Due to a coding error, the number of presentations of single- and double-marked words was not entirely equal in that participants heard one extra single-marked and one less double-marked noun in the familiarization phase. This was the same for both the high-repetition condition and the high-variability condition. The uneven number of markings was unfortunate. However, we believe that it did not affect the results. Even though the participants were exposed to slightly more single-marked than double-marked words, they still showed a learning advantage with double-marked words.

Natural Language Stimuli and Natural Language Experience

Studies investigating the implicit learning of grammatical structures have largely used artificial language stimuli. The obvious advantage of using artificial learning material is that it prevents the learners from benefiting from prior experience with the language itself, or closely related languages (Gómez & Gerken, 1999). In the present experiment, we presented individual words, along with their gender markings, in isolation so that other distributional information inherent to Russian was not present in the input to our participants. This is highly similar to the degree of control available with the use of an artificial language. In addition, the use of grammatical constructions from a natural language and stimuli recorded by native Russian talkers increased the ecological validity of our experiment.

We also controlled for previous linguistic experience by excluding participants with prior knowledge of Russian or other Slavic languages. However, we should note that Norwegian, the native language of the participants, also marks nouns for gender by adding a suffix (e.g., bil + –en, hus + –et), which shares properties with the Russian single-marked words in the present experiment. Similar to Russian, the Norwegian gender system also includes correlated cues to subcategory membership. However, in Norwegian, these correlated cues are reflected in a nonadjacent relationship between a gendered article and a gendered suffix (e.g., den bil + –en, det hus + –et). Because of the presence of gender marking in both languages, it is perhaps more remarkable that input variability still had a pronounced effect on learning for these Norwegian participants. Also, it is worth mentioning that Norwegians in general are exposed to a high variety of different languages in school, and this broad exposure might lead to a better learning outcome in experiments such as ours. Instead, as with studies of native English speakers (Gómez, 2002; Gómez & Maye, 2004; Grunow et al., 2006; Torkildsen et al., 2013), our findings revealed that Norwegian speakers benefitted from the high-variability stimuli relative to the high-repetition stimuli. Moreover, the number of languages spoken by the participants did not correlate with performance.

Summary and Conclusions

In summary, the results indicate that a high degree of stimuli variability facilitates rapid learning of gender subcategories. This extends the known benefits of high-variability input to grammatical features that are different in form than those previously studied. In the present experiment, learning was defined by measuring the participants' acceptance rate for generalization items, which indicated that performance was not due to verbatim memory for items previously heard. Our findings reveal that learners recognized the distributional patterns reflected in the gender markings and were able to apply this information to evaluate novel test items. Moreover, we found that learning was facilitated for double-marked nouns (including multiple morphological cues) compared to single-marked nouns.

Appendix

Table A1.

Phase I familiarization stimuli set for the high-repetition and high-variability conditions.

Stimulus item type	Masculine words	Feminine words
Single-marked, female voice
High repetition	Kon –ya/yem	Tysjacha –oj/u
	Olen –ya/yem	Gazeta –oj/u
High variability	Kon –ya/yem	Tysjacha –oj/u
	Olen –ya/yem	Gazeta –oj/u
	Den –ya/yem	Korzh –oj/u
	Korabl –ya/yem	Komnat –oj/u
Single-marked, male voice
High repetition	Aprel –ya/yem	Vistot –oj/u
	Fonar –ya/yem	Luzhits –oj/u
	Glukhar –ya/yem
High variability	Aprel –ya/yem	Vistot –oj/u
	Fonar –ya/yem	Luzhits –oj/u
	Ijul –ya/yem	Vysot –oj/u
	Bezdar –ya/yem	Ulits –oj/u
	Glukhar –ya/yem
Double-marked, female voice
High repetition	Vodi –telya/telyem	Blondin –koj/ku
	Dviga –telya/telyem	Rubash –koj/ku
High variability	Vodi –telya/telyem	Blondin –koj/ku
	Dviga –telya/telyem	Rubash –koj/ku
	Muchi –telya/telyem	Telnjash –koj/ku
	Osnova –telya/telyem	Skovord –koj/ku
Double-marked, male voice
High repetition	Dushi –telya/telyem	Karmel –koj/ku
		Petrush –koj/ku
High variability	Dushi –telya/telyem	Karmel –koj/ku
	Pokori –telya/telyem	Petrush –koj/ku
	Blagode –telya/telyem	Oblav –koj/ku
		Makus –koj/ku

Note. Due to a coding error, one extra masculine single-marked root word (male voice) was included and one masculine double-marked root word was excluded from the stimulus material.

Table A2.

Stimulus set Phase II familiarization for both high-repetition and high-variability groups.

Stimulus item type	Masculine words	Feminine words
Single-marked	Kartofel –yem	Chistota –oj
	Koren –ya	Voron –u
Double-marked	Uroven –yem	Zabav –oj
	Osvezhi –telya	Brjunet –ku
	Pokupa –telya	Devush –koj
	Ljubi –telyem	Maka –koj

Table A3.

Stimulus set for the test phase for both high-repetition and high-variability groups.

Test item type	Grammatical/correct words
	Masculine	Feminine
Single-marked	Kartofel –ya	Chistot –u
	Koren –yem	Voron –oj
	Urovn –ya	Zabav –u
Double-marked	Osvezhi –telyem	Brjunet –koj
	Pokupa –telyem	Devush –ku
	Ljubi –telya	Maka –ku
Test item type	Ungrammatical/incorrect words
	Masculine	Feminine
Single-marked	Kartofel –oj	Chistot –yem
	Koren –u	Voron –ya
	Urovn –oj	Zabav –yem
Double-marked	Osvezhi –telu	Brjunet –kya
	Pokupa –telu	Devuch –kyem
	Ljubi –teloj	Maka –kyem

Funding Statement

The experiment was supported in part by National Institute on Deafness and Other Communication Disorders Grant 1R01DC011276. Technical support was given by Bjørn Sætrevik, a postdoctoral student at the University of Bergen.