Optimizing word learning in post-secondary students with Developmental Language Disorder: The roles of retrieval difficulty and retrieval success during training

Abstract

Purpose:

Learning words to the level that they can be readily retrieved and produced can be challenging. The primary aim of the current study is to determine how retrieval difficulty, based on the level of cuing provided, and retrieval success during training relate to the phonological precision with which words are produced after a delay.

Method:

We performed additional analyses on data from McGregor, Gordon, et al., (2017) in which post-secondary students with Developmental Language Disorder (DLD, n=23) and typical development (n = 25) were trained on words via free and cued recall practice and tested 24-hours later.

Results:

Training via free recall led to more precise productions after the delay than training via cued recall for both groups. Additionally, the number of successful retrievals during training positively predicted retrieval after the delay. Furthermore, the precision of participants’ last production and worst production of each word were the best predictors of production precision after the delay.

Conclusion:

To optimally support encoding and delayed retrieval, students with and without DLD should utilize free recall practice. Additionally, words should be studied until they are successfully retrieved multiple times at a high level of phonological precision to support delayed retrieval.

Introduction

A common challenge for high school, vocational, and college students is to learn a large number of discipline-specific words. For an individual to add a new word to her lexicon, multiple aspects of the word must be learned (Gupta & Tisdale, 2009). These include but are not limited to, the phonological form of the word, the meaning of the word, and the link between the form and the meaning. For example, when studying for a biology class a student may correctly indicate that entomology is the study of insects and ethology is the study of animal behaviour. Thus, these words have been added to the student’s receptive vocabulary. However, without a precise representation of the phonological features of the word forms, she will struggle to use these words expressively. To the question, “What is the study of animal behaviour called?”, she might respond, “etology”. Although her representation of this form lacks phonological precision, she may know the word well enough to pass her biology class. However, passing the class is not the only goal. Students are training to become professionals and professionals are expected to have both receptive and expressive mastery of field-specific vocabulary. Expressive vocabulary is essential for effective communication and, thus, is related to academic and occupational outcomes (Martinussen & Mackenzie, 2015; Stothard, Snowling, Chipchase, & Kaplan, 1998).

Learning words to a high level of phonological precision such that they can be readily retrieved and produced is often challenging (Munro, Baker, McGregor, Docking, & Arciuli, 2012). For both children and adults, receptive vocabulary, the number of words that an individual understands; usually exceeds expressive vocabulary, the number of words that that individual can readily produce (Gershkoff-Stowe & Hahn, 2013). As an example of the challenge of adding words to one’s expressive vocabulary, consider the typical adult participants in Storkel (2015). At the end of a training session with novel word-referent pairs, these participants correctly named an average of 70% of the referents. However, after a week’s delay they only named 30% of the referents correctly. At this point, we would only consider words that had been successfully retained and retrieved to be a part of the participants’ expressive vocabularies. The representations of some of the forms, although produced successfully at the end of the first training session, were not robust enough to support successful retrieval and production after the delay. Notably, the words included in Storkel (2015) all consisted of one syllable whereas many discipline specific words are much longer (e.g. deoxyribonucleic acid). Therefore, these results may underestimate the extent of the problem.

Developmental Language Disorder

Although adding words to one’s expressive vocabulary can be challenging for learners with typical development (TD), other learners face even greater struggles with word learning. Developmental Language Disorder (DLD) ¹ is a life-long condition that affects an individual’s ability to learn and use language (Leonard, 2014) and includes approximately 7% of the population (Tomblin et al., 1997). Learning word forms such that they can be readily retrieved and produced is a particular area of weakness for this population (Gray, 2005; McGregor, Arbisi-Kelm, & Eden, 2017) Specifically, when given the same experience with unfamiliar words in experimental studies, individuals with DLD name fewer referents correctly than their typically developing peers (McGregor, Arbisi-Kelm, et al., 2017). Furthermore, individuals with DLD produce forms with less phonological precision (McGregor, Gordon, Eden, Arbisi-Kelm, & Oleson, 2017) and exhibit greater variability in their pronunciations of a given form than their peers (Benham, Goffman, & Schweickert, 2018) demonstrating weaker phonological representations.

Word Learning and Memory

When a learner first encounters a new word, he may encode some information about the word. This involves forming temporary representations of the word form, its meaning, and the link between the two. These representations are maintained in working memory and are associated with hippocampal activity (Davis & Gaskell, 2009). As demonstrated in Storkel (2015), some encoded information about words will be forgotten and other encoded information will be retained over post-training delays. To be retained, information about words must become associated with cortical activity through a process of consolidation (Davis & Gaskell, 2009). To use a word that has been retained in long-term memory, one must retrieve it. Not all words in long-term memory are equally easy to retrieve. For example, words that occur frequently in the language are retrieved more efficiently and accurately than words that are rare, even if both reside in long-term memory (Jescheniak & Levelt, 1994). Furthermore, the presence or absence of cues during the retrieval attempt can influence whether a word is successfully retrieved (McGregor, Gordon, et al., 2017; Munro et al., 2012) Such cues can include words that are semantically related to the target word or phonological cues such as the first phoneme of the form.

Once retrieved, the speaker must enact a motor plan to produce the word. Discrepancies between the retrieved representation and verbal production of a form is likely for children who are still developing their production abilities and for individuals with motor conditions that limit the accurate production of phonemes (Swingley & Aslin, 2000). However, it is much less likely that verbal productions will mismatch retrieved representations for adult learners who have robust production skills in their native language. Adding a word to one’s expressive vocabulary includes strengthening the representation, the retrieval routes, and the motor pathways such that the word can be retrieved and produced accurately when desired.

Retrieval-Based Learning

When an individual is unable to use a word that he has been taught, the root cause may be the process of encoding, consolidation, retrieval, or production. Individuals with DLD demonstrate marked weaknesses in their ability to encode the phonological features of forms from input (McGregor, Arbisi-Kelm, et al., 2017). In contrast, their ability to consolidate and later retrieve and produce word forms is an area of relative strength (McGregor, Gordon, et al., 2017). Difficulty with encoding word forms may be ascribed, in part, to weaker verbal working memory abilities which is a common characteristic of individuals with DLD (Archibald & Gathercole, 2006). Because of these weaknesses in encoding verbal information from input, individuals with DLD need more exposures to words than their typically developing peers before they develop robust enough representations to readily retrieve and produce them (Gray, 2005; McGregor, Arbisi-Kelm, Eden, & Oleson, 2020).

Are there ways to strengthen the encoding of word forms instead of or in addition to providing more exposures? One possibility is to introduce desirable difficulties. Desirable difficulties are components of learning that make the initial learning experience more challenging but lead to better outcomes (Bjork & Bjork, 2011). For example, retrieval-based learning, also known as learning through testing, is highly effective in supporting encoding, consolidation, and later retrieval of information in contrast to more passive study strategies such as re-reading class textbooks or notes (see Adesope, Trevisan, & Sundararajan, 2017; Roediger & Karpicke, 2006 for reviews). The benefits of retrieval-based learning have been robustly demonstrated for post-secondary students with typical development when they study arbitrary stimuli in laboratory settings (Karpicke, 2017). Recently researchers have extended this work to educationally relevant content including word learning. For example, retrieval practice supports the learning of discipline specific terms and their definitions (Pan et al., 2019; Rawson & Dunlosky, 2011) and the learning of translation pairs in a known and foreign language (Ariel & Karpicke, 2018). This strategy also supports word learning in children with DLD (Haebig et al., 2019; Leonard, Karpicke, Deevy, Weber, & Christ, 2019) and adults with DLD enrolled in post-secondary education (McGregor, Gordon, et al., 2017).

Although retrieval practice is more effective than passive study, not all variations of retrieval practice equally support encoding, consolidation, and delayed retrieval of target information (see Adesope et al., 2017; Karpicke, 2017). Notably, there are two key characteristics of retrieval practice that increase the effectiveness of this strategy. First, items that are successfully retrieved during practice are more likely to be retrieved after a delay than items that are not successfully retrieved during practice (Kornell, Bjork, & Garcia, 2011). Relatedly, items that are successfully retrieved more times as opposed to fewer times during practice are more likely to be retrieved after a delay (Ariel & Karpicke, 2018; Storkel, 2015). It is hypothesized that this occurs because when an individual successfully retrieves target information during practice, she builds and strengthens the retrieval pathway to that information (Karpicke, 2017).

A second key characteristic of retrieval practice is that learning is enhanced when retrieval is more effortful than when it is less effortful (Carpenter & DeLosh, 2006; Karpicke, 2017; Pyc & Rawson, 2009). In the current paper, we focus on how retrieval effort can be increased by providing fewer retrieval cues during practice. There are multiple hypotheses about why more effortful retrieval during practice increases the likelihood of retrieval after a delay (Karpicke, 2017). Notably, retrieving information with fewer cues may promote the development of retrieval routes that do not rely on the presence of specific cues (Carpenter & DeLosh, 2006). Thus, when an individual practices retrieval without cuing the individual will be more likely to retrieve the information after a delay when those specific cues are not present.

Given this evidence, it appears that engaging in retrieval practice that is maximally effortful but also likely to lead to successful retrieval is the best way to increase the likelihood of delayed retrieval. In this way, retrieval routes that are the least dependent on cues can be built and strengthened. However, it is important to note that learners benefit from unsuccessful as well as successful retrieval attempts when feedback is provided (Rowland & DeLosh, 2015). When the retrieval is successful, the learner receives an exposure to the correct information, their own retrieval, and the learner strengthens retrieval routes to that information. When the retrieval is unsuccessful and the learner is provided with feedback, she still receives exposure to the target information. Critically, her attention to and effort to encode this target information is enhanced when it is preceded by an unsuccessful retrieval attempt as opposed to an exposure not preceded by a retrieval attempt (Arnold & McDermott, 2013; Kornell & Vaughn, 2016). For example, when learners are presented with a pretest before being presented with target information, they demonstrate better encoding of that information than conditions that do not include a pretest.

Current Study

Overall, retrieval practice can enhance learning in multiple ways. Retrieval routes can be formed and strengthened when cuing is minimized and retrieval is successful. Additionally, encoding of subsequent presentations can be enhanced when retrieval is unsuccessful. Given these various principles, it is not readily apparent which type of retrieval practice will lead to optimal word form learning for individuals with DLD. On the one hand, individuals with DLD may successfully retrieve little information during retrieval practice without cuing. By responding to cued recall prompts, such as the first syllable of the target form, learners with DLD may receive more practice successfully retrieving and producing forms. This practice is likely to strengthen the retrieval routes to the forms and the motor pathways necessary for accurate productions. Furthermore, cued recall practice may be optimal because many failed retrieval attempts when few cues are provided could lead to discouragement for individuals with DLD. Thus, they might not demonstrate the same enhanced encoding from feedback as individuals with TD under this more effortful retrieval condition. Notably, cueing retrieval responses is a common practice in language intervention (Ebbels et al., 2012).

On the other hand, to build an expressive vocabulary the target skill is to be able to retrieve and produce word forms when cuing is not provided. Thus, practice retrieving forms without phonological cues, may help individuals with DLD build retrieval routes that are not cue dependent. In this way, they can optimize their ability to retrieve forms after delays. Furthermore, this more challenging retrieval condition may enhance encoding from feedback. When given a cue to the form, such as the first syllable, participants are likely to retrieve and produce more of the target phonemes correctly. Thus, they may pay less attention to the phonemes of the correct form when it is presented. However, without a cue, they are likely to produce fewer of the target phonemes correctly. Thus, their attention to and attempt to encode the phonemes of the target form when given as feedback may be enhanced.

To identify a blend of effort and success to optimally support word form learning in individuals with DLD, we conducted additional analyses on the data from the second study of McGregor, Gordon, et al., (2017). In this study, we trained post-secondary students with and without DLD on three sets of word-referent pairs that were all disyllabic. One set was trained via free recall prompts with feedback; one set was trained via cued recall prompts, in which participants were given the first syllable of the target form, with feedback; and one set was trained via passive exposures.² After a 24-hour delay, participants were asked to name all 27 objects via a free recall test. Assessing responses in a dichotomous manner (correct/incorrect) was not sensitive enough to reveal differences in the effectiveness of these training strategies. The total number of words produced with 100% phonological accuracy after the delay was near floor for both groups. However, when we coded responses based on the percentage of phonetic features produced correctly, we found differences between the training conditions. Specifically, students with and without DLD produced more precise phonological forms of words trained in the conditions with retrieval practice (free recall, cued recall) than of words trained in the passive exposure condition. For the conditions with retrieval practice, there was not a significant difference between groups (DLD, TD) or conditions (free recall, cued recall).

In McGregor, Gordon, et al., (2017) we determined how productions of forms throughout training and after the 24-hour delay differed based on DLD status and training condition. Through item-level analyses we can determine how retrieval effort and success during training relate to the probability of successful retrieval after the delay. In these analyses, we operationalize retrieval effort based on the level of cuing provided during training. Notably, each syllable of forms trained in the cued and free recall conditions included different levels of cuing to support retrieval. During each trial in the cued recall condition, participants were given the first syllable of the target form and were asked to produce the entire form. Thus, for the first syllable in the cued recall condition participants simply had to repeat the syllable immediately after they heard it, inducing low retrieval effort. Retrieving and producing the second syllable in the cued recall condition likely required more effort than producing the first syllable. However, participants had just been given the first syllable which served as an important cue to retrieve the second. In contrast, for the free recall condition participants had to retrieve and produce both the first and second syllables without any cues beyond seeing the visual referent. Thus, retrieving each syllable in the free recall condition was likely more effortful than retrieving the first or second syllables in the cued recall condition. Through comparing performance at the 24-hour delay for each syllable of forms trained in each condition, we can determine how the level of cuing provided during training affected retrieval success after the delay and whether this relationship differed between individuals with and without DLD.

Through the current analyses we can also determine how retrieval success during training related to retrieval success after the delay. We assessed retrieval success in two ways. First, we replicated coding strategies common in past research (Storkel, 2015) by determining how the number of times each syllable of each word was successfully retrieved during training. However, given that participants retrieved and produced few forms with 100% phonological precision during training, we also coded retrieval success based on the phonological precision of productions. To quantify phonological precision we utilized the same coding strategy as McGregor, Gordon, et al., (2017) which was originally developed by Edwards, Beckman, and Munson (2004). In this coding strategy, the phonetic features produced at each retrieval attempt are compared to the phonetic features of the target form to determine the percentage of features produced correctly. This coding strategy provides a high level of sensitivity to determine the relationship between the phonological precision of productions during training and the phonological precision of productions after a delay.

Given that individuals with DLD are likely to demonstrate a high level of variability in their productions of forms while they are learning them (Benham et al., 2018), variability of productions during training is another marker of retrieval success. Specifically, although an individual with DLD may produce a form with a high level of phonological precision one time during training, this does not necessarily indicate that she will retrieve and produce the form with a high level of phonological precision after the delay. In the current study, we determined for each syllable of each word the percentage of phonetic features produced correctly during the final training trial; the highest percentage of features produced correctly at any point during training; and the lowest percentage of features produced correctly at any point during training. By assessing how the variability of productions during training related to the phonological precision of productions after the delay, we can determine which aspects of training success are most indicative of delayed retrieval success and whether this relationship differs across individuals with and without DLD.

In summary, for the current study we conducted additional analyses on data collected for McGregor, Gordon, et al., (2017) in which post-secondary students with and without DLD studied novel forms linked to referents via free or cued recall probes. We asked the following key questions, not addressed in McGregor, Gordon, et al., (2017):

1. How does the level of cuing provided during training, which differed across training condition and syllable, affect retrieval after the 24-hour delay?

2. How does the number of successful retrievals of each syllable of each word during training relate to the probability of successful retrieval after the 24-hour delay?

3. How does the phonological precision of productions of each syllable of each word during training relate to the phonological precision at which those syllables are produced after the 24-hour delay?

Through these analyses, we can determine how retrieval effort, addressed in the first question; and retrieval success, addressed in the second and third questions, during training related to retrieval after the delay. Critically, through addressing these questions we can identify a blend of retrieval effort and success that is likely to support encoding and later retrieval of word forms for individuals with DLD.

Method

Participants

For the current study we analysed the data from 48 college students who participated in the second study of McGregor, Gordon, et al. (2017). These participants were all students enrolled in post-secondary education in the Midwest and had an age range of 18 to 24 years. The study protocol was approved by the University of Iowa’s institutional review board and all participants completed informed consent. Twenty-three of the participants qualified as having DLD and 25 qualified as TD.³ DLD status was independently verified in the lab via performance on a 15-word spelling test and the Modified Token Test for testing syntactic comprehension (Fidler, Plante, & Vance, 2011). These scores were weighted according to the procedure used by Fidler et al., (2011). All participants scored within the typical range for non-verbal IQ on the Kaufman Brief Intelligence Test (Kaufman & Kaufman, 2004) and all passed a pure-tone audiometric screening at 0.5, 1, 2, and 4 kHz at 25 dB bilaterally. More information about participant demographics and performance on standardized tests is provided in McGregor, Gordon, et al., (2017).

Stimuli

Stimuli included three sets of nine novel word forms (A/B/C), for a total of 27 words (Table 1). All words were disyllabic with primary stress on the first syllable and contained one of three prosodic shapes (CV.CVC/CVC.CVC/CCV.CVC). All words were based on real English words but differed from the real word in one or two final consonants. For each set, all of the words contained different onset segments and feature distribution of word onsets was proportionally balanced across each set (i.e. the proportion of manner, place, and voicing features were distributed similarly across the three sets). The three prosodic word shapes were also equally represented within each word set. All forms had similar phonotactic probabilities, with a mean segmental phonotactic probability of 0.05 and standard deviation of 0.01 (calcuated via Vitevitch & Luce, 2004), and the phonotactic probability of forms was balanced across sets. All forms had a relatively low neighborhood density with a mean of 1.74 neighbors and a range of 0 to 7 neighbors (calculated via Vitevitch & Luce, 2004). Auditory presentations of novel word forms were pre-recorded and produced by a female native speaker of Standard American English (see McGregor, Gordon, et al., 2017 for details about recording). Each novel word form was paired with a photograph of an unusual object that has no commonly known English name. Each set of nine word-referent pairs included photographs of three plants, three animals, and three inanimate objects.

Table 1.

Target words in each set. The real word from which the target word was derived is in parentheses.

Set A	Set B	Set C
nɛkləʤ (necklace)	dʌnʤəp (dungeon)	baɪsət (bison)
blɑvəd (blossom)	faʊntɪk (fountain)	tfɪpmʌz (chipmunk)
drӕməs (dragon)	ɡɑrlɪd (garlic)	dɛzɚɡ (desert)
fɔsɪb (faucet)	ʤӕkɪz (jacket)	ɡlɪtəf (glitter)
kɪʧɪv (kitchen)	kɑtəf (cottage)	kӕktəb (cactus)
klotəɡ (clover)	mɛləɡ (melon)	nudək (noodle)
mӕɡnəf (magnet)	plӕsəv (planet)	peɪntəʧ (painter)
sӕləp (salad)	stʌvəb (stomach)	skunəv (scooter)
sɪmbək (cymbal)	trӕɡɪʧ (traffic)	spaɪləp (spider)

Procedure

Participants completed four sessions. During the first session, they completed the consent process and standardized testing. During the second session they completed training on all word-referent pairs. During the third session, which occurred 24-hours after the second session, participants were asked to name all photographs via a free-recall test with sets intermixed in random order. The fourth session was conducted one week after training and included a visual-world paradigm test to assess whether participants’ memory for the new words competed with the processing of known words. For the current analysis, we only include data from the second (i.e. training session) and third (i.e. free recall test after a 24-hour delay) session.

During the second session each participant completed three types of training, one for each word set: passive study, learning via free recall testing, and learning via cued recall testing. All participants completed four training blocks for each word set, which were completed before moving onto the next word set. Order of training type and word set assigned to training type were counterbalanced across participants. During the first training block, they were shown each photograph one at a time and heard its name. For the learning via free recall condition, during the second, third, and fourth training blocks they were shown each photograph and were asked to name it. Thus, they were asked to name each referent three times during training. If they responded correctly, they heard a chime. If they responded incorrectly or produced no response, they heard the audio production of the target word. The cued recall condition was the same as the free recall condition with the exception that participants were given the first syllable of the word as a cue during the second, third, and fourth training blocks. They were prompted to say the whole word after they were given the cue. In the passive study condition, all training blocks were administered in the same way as the first training block in that they saw each photograph and heard the word. To answer the questions of interest for the current paper, we only analysed responses in the free recall and cued recall training conditions.

Coding

Participants’ productions were coded following the schema developed by Edwards, Beckman, and Munson (2004). In this coding schema, the phonological features produced at each assessment point are compared to the phonological features of the target word form to determine the percentage of features produced correctly. Each consonant includes three phonological features, place, manner, and voicing; and each vowel includes three phonological features, backness, height, and tenseness. By coding phonemes at the feature level, phonemes that are more similar to the target phoneme receive a higher score than phonemes that are less similar. Thus, if a participant were trying to produce the name “echidna” for a spiny anteater, “echiba” would receive a higher score than “echifa” because “b” shares two features with “d” (voicing and manner) while “f” does not share any major consonantal features with “d”.

Analyses and Results

Our analyses include three main parts to address our three questions. For all analyses, we assessed data at the syllable level, not the word level. First, we focused only on responses after the 24-hour delay and determined how training condition affected the probability of the production of each syllable (correct/incorrect) and the phonological precision of the production of each syllable. Second, we determined the extent to which the number of accurate syllable productions during training predicted the probability of an accurate production after the 24-hour delay. Third, we determined the extent to which the phonological precision of productions during training, coded as the percentage of phonetic features produced correctly for each syllable of each word, predicted the percentage of phonetic features produced correctly after the 24-hour delay.

Performance after the Delay Based on Cuing During Training

To address the first question, we conducted two mixed-effects models which both included responses after the 24-hour delay only. The first model included the probability of a correct production (i.e., 100% phonological accuracy) as the outcome variable and the second included the percentage of phonetic features produced correctly as the outcome variable. Both models included syllable (1,2), training condition (free, cued), and group (TD, DLD) and their interactions as fixed effects. Recall that for performance after the 24-hour delay, all words are produced following a free recall probe. We included intercepts for participant and words as random effects. For both models, eliminating either of these random effects reduced model fit so both were retained.

When the probability of a correct response was the outcome variable, the minimal model that converged and supported best model fit included a fixed effect for syllable only; the first syllable had a higher probability of being produced correctly than the second syllable (Table 2). When the percentage of phonetic features produced correctly was the outcome variable, the minimal model that converged and supported best model fit included fixed effects for training condition, syllable, group, and an interaction between syllable and group (Table 3). After the 24-hour delay, participants produced more phonetic features correctly for syllables of words that were trained via free recall than cued recall; and they produced a higher percentage of phonetic features correctly for first syllables than second syllables (Table 4). The nature of the group by syllable interaction is that there was a larger difference between productions of the first and second syllable for individuals with DLD (mean first syllable for both conditions combined = 0.70, mean second syllable = 0.64) than for individuals with TD (mean first syllable = 0.72, mean second syllable = 0.71).⁴

Table 2.

Model predicting the percentage of phonetic features produced correctly after the 24-hour delay.

	Estimate	Standard error	z-value	Pr(>∣z∣)
Intercept	−0.38	0.25	−1.50	0.13
Syllable a	−1.22	0.12	−9.76	<.0001

^aReference group was the first syllable.

Table 3.

Model predicting the percentage of phonetic features produced correctly after the 24-hour delay.

	Estimate	Standard error	t-value	Pr(>∣z∣)
Intercept	0.71	0.03	23.38	< .0001
Training Conditiona	−0.03	0.01	−2.36	<.05
Syllable b	−0.06	0.02	−3.27	<.05
Group c	0.02	0.04	0.56	0.57
Syllable x Group	0.05	0.02	2.18	0.03

^aReference group was free recall.

^bReference group was the first syllable.

^cReference group was DLD.

Table 4.

Average percentage of phonetic features correctly produced per syllable after the 24-hour delay.

	DLD		TD
	Syllable 1	Syllable 2	Syllable 1	Syllable 2
Free	0.72 (0.31)	0.65 (0.26)	0.74 (0.31)	0.72 (0.26)
Cued	0.67 (0.32)	0.63 (0.27)	0.70 (0.32)	0.71 (0.24)
Average across conditions	0.70 (0.32)	0.64 (0.26)	0.72 (0.31)	0.71 (0.25)

Performance after the Delay Based on the Number of Successful Retrievals During Training

We conducted a generalized mixed-effects model to determine how the number of correct productions of each syllable of each word during training related to the probability of a correct production 24-hours after training. The production of each syllable of each word during training was coded as correct if the target phonemes were produced with 100% accuracy. The percentage of correct productions during training was then calculated based on the number of correct productions out of the three naming opportunities (e.g. one correct production during training would be coded as 33% correct). For this analysis, we considered the percentage of correct productions in the following conditions: 1) free recall, first syllable; 2) free recall, second syllable; 3) cued recall, first syllable; and 4) cued recall, second syllable. As previously stated, each reflect different retrieval demands during training. As expected, participants were highly likely to produce the first syllable correctly immediately after they heard it in the cued recall condition (training trial 1, mean probability of correct production = 0.90, sd = 0.11; training trial 2, mean = 0.91, sd = 0.10; training trial 3, mean = 0.92, sd = 0.11). These responses were skewed towards ceiling performance which violates model assumptions, thus, responses to the first syllable in the cued recall condition were not included. Therefore, for this model fixed effects included group (TD, DLD); training condition and syllable (free recall, syllable 1; free recall, syllable 2; cued recall, syllable 2); and the percentage of correct productions of each syllable of each word during training. The model also included interactions of these fixed effects. We included random intercepts for participant and word. Eliminating either of these random effects reduced model fit so both were retained. The minimal model that converged and best supported model fit included fixed effects for the percentage of correct productions during training, condition, group (DLD, TD), and an interaction between group (DLD, TD) and the percentage of correct productions during training (Table 5).

Table 5.

Model predicting the probability of a correct production of each syllable after the 24-hour delay.

	Estimate	Standard error	z-value	Pr(>∣z∣)
Intercept	−2.16	0.28	−7.75	<.0001
Group a	0.24	0.32	0.73	0.46
Percentage of correct productions during training b	5.40	0.49	10.93	<.0001
Condition Free Recall, Syllable 2c	−0.32	0.20	−1.60	0.11
Condition Cued Recall, Syllable 2 c	−0.80	0.20	−3.94	<.0001
Group x Percentage of correct productions	−1.34	0.60	−2.24	0.03

^aReference group was DLD.

^bCoded as a percentage correct out of three training trials.

^cReference group was Free Recall, Syllable 1.

In Figure 1, we illustrate how the percentage of correct productions during training related to the probability of a correct production after the 24-hour delay based on training condition. Productions of the first syllable in the cued-recall condition are included in this figure for illustrative purposes but were not included in the model. Not surprisingly, as the percentage of correct productions during training increased, the probability of a correct production after the 24-hour delay also increased. The probability of a correct production of the second syllable of words trained via cued recall differed significantly from the other two conditions: Free Recall, Syllable 1 (z = −3.94, p <.0001) and Free Recall, Syllable 2 (z = −2.32, p = .02). Specifically, a similar percentage of correct productions during training lead to a higher probability of a correct production after the delay for the two free recall conditions (free, syllable 1; free, syllable 2) than for the cued recall, syllable 2 condition. The probability of a correct production after the 24-hour delay did not differ for the first and second syllables in the free recall condition (z = −1.60, p = 0.11).

Figure 1.

The relationship between the number of successful retrievals, coded as a percentage, of each syllable of each word during training and the probability of a successful retrieval after the 24-hour delay. Responses for each syllable in each condition (free, cued) are displayed separately.

With regards to the results for group (DLD, TD), the main effect was not significant. However, there was a significant interaction between group and percentage of correct productions during training. We include descriptive statistics for the average number of words produced with 0%, 33%, 66%, and 100% accuracy for each group for each syllable and condition (Table 6). The same level of accuracy during training lead to a higher probability of a correct production after the delay for the DLD group than the TD group. For example, producing a syllable of a given word correctly 100% of the time during training resulted in a 95% probability of producing that syllable correctly after the delay for participants with DLD and an 87% probability for participants with TD (Table 7).

Table 6.

Average number of words, broken down by syllable, condition, and percentage of correct productions during training in each condition and for each group.

0% correct	Free Recall, First Syllable	Free Recall, Second Syllable	Cued Recall, Second Syllable	Mean Across Conditions
DLD	3.74 (2.16)	6.83 (1.44)	6.65 (1.80)	5.74 (2.29)
TD	2.84 (2.44)	5.00 (2.48)	4.16 (1.97)	4.00 (2.45)
33% correct
DLD	2.35 (1.43)	1.39 (0.99)	1.26 (1.18)	1.67 (1.29)
TD	2.08 (1.12)	2.12 (1.48)	2.48 (1.16)	2.23 (1.26)
66% correct
DLD	1.61 (1.50)	0.65 (0.83)	0.83 (0.98)	1.02 (1.20)
TD	2.24 (1.39)	1.48 (1.26)	1.56 (1.00)	1.76 (1.26)
100% correct
DLD	1.30 (1.26)	0.13 (0.34)	0.26 (0.54)	0.57 (0.96)
TD	1.84 (1.52)	0.40 (0.71)	0.80 (1.12)	1.01 (1.30)

Table 7.

The probability of a correct production of a syllable of a given word after the 24-hour delay based on group (DLD, TD) and percentage of correct productions of that syllable of that word (out of 3) during training.

	DLD		TD
Percentage of times correct during training	Log-odds	Probability	Log-odds	Probability
0%	−2.57	0.07	−2.36	0.09
33%	−0.77	0.32	−0.95	0.28
66%	1.02	0.73	0.47	0.62
100%	2.86	0.95	1.94	0.87

Phonological Precision of Productions after the Delay Based on Phonological Precision of Productions During Training

We conducted a mixed-effects model to determine how the percentage of phonological features produced correctly for each syllable of each word during training related to the percentage of phonological features produced correctly for that syllable of that word after the delay. For this analysis, we were interested in how the variability of productions during training affected the phonological precision of productions after 24 hours. To capture this variability, we included participants’ productions of each syllable of each word during the last training trial, their best production of each syllable of each word (i.e., the response with the highest percentage of correct features during training), and their worst production of each syllable of each word (i.e., the response with the lowest percentage of correct features during training) as well as interactions between these fixed effects in the analyses (Table 8). Participants were highly accurate in the percentage of phonetic features they produced correctly for the first syllable of words trained in the cued recall condition as they were only required to repeat them immediately after they heard them. Thus, responses to the first syllable in the cued recall condition were excluded from this analysis as they were for the previous analysis.

Table 8.

Average percentage of phonetic features produced during training in the last training trial, best training trial, and worst training trial based on syllable and condition.

Last Training Trial
	DLD		TD
	Syllable 1	Syllable 2	Syllable 1	Syllable 2
Free	0.76 (0.30)	0.66 (0.27)	0.84 (0.27)	0.77 (0.24)
Cued	0.97 (0.12)	0.69 (0.25)	0.99 (0.05)	0.83 (0.21)
Best Training Trial
	DLD		TD
	Syllable 1	Syllable 2	Syllable 1	Syllable 2
Free	0.84 (0.22)	0.74 (0.22)	0.88 (0.20)	0.84 (0.18)
Cued	0.98 (0.07)	0.77 (0.19)	0.99 (0.02)	0.88 (0.17)
Worst Training Trial
	DLD		TD
	Syllable 1	Syllable 2	Syllable 1	Syllable 2
Free	0.42 (0.32)	0.41 (0.24)	0.47 (0.34)	0.48 (0.24)
Cued	0.91 (0.19)	0.45 (0.24)	0.96 (0.13)	0.57 (0.24)

Participants’ best production of each syllable of each word and their production during the last training trial were highly correlated (r = 0.85). In many, but not all cases, their best production was their last production. Thus, we ran two separate linear mixed effects models for the last production and the best production and compared the two models to see which one more accurately predicted responses after the 24-hour delay. Both models included group (TD, DLD), training condition/syllable (free recall, syllable 1; free recall, syllable 2; cued recall, syllable 2), and percentage of features from each participant’s worst production of each syllable of each word as fixed effects, interactions between these fixed effects, and intercepts for participant and word as random effects. An analysis comparing the two models revealed that the fit of the models were significantly different (Chi square = 19.94, p < .0001). The model with productions during the last training trial provided a better fit (AIC = −78.15) than the model with the best production (AIC = −58.21). Thus, we retained the model with the last production for further analyses. The version of this model that converged and provided the best fit included random intercepts for participant and word, and fixed effects for condition, percentage features correct during the worst training trial, and percentage features correct during the last training trial. There were two interactions: an interaction between condition and the last training trial and an interaction between the last training trial and the worst training trial (Table 9).

Table 9.

Model predicting the percentage of features produced correctly after the delay.

	Estimate	Standard error	t-value	Pr(>∣z∣)
Intercept	0.30	0.04	7.65	<.0001
Condition Free Recall, Syllable 2a	0.08	0.04	1.74	0.08
Condition Cued Recall, Syllable 2a	0.16	0.05	3.15	<.01
Percentage features from the worst training trial	−0.09	0.10	−0.91	0.37
Percentage features from the last training trial	0.44	0.05	9.28	<.0001
Condition Free Recall, Syllable 2 x last training trial	−0.11	0.05	−1.96	<.05
Condition Cued Recall, Syllable 2 x last training trial	−0.26	0.06	−4.43	<.0001
Last Training trial x worst training trial	0.31	0.10	3.03	<.01

^aReference group was Free Recall, Syllable 1.

In Figure 2, we illustrate the nature of the condition by last training trial interaction. Responses to the first syllable in the cued recall condition are included in this figure for illustrative purposes but were not included in the model. Not surprisingly, as the percentage of features produced correctly during the last training trial increased, the percentage of features produced correctly after the delay also increased. However, the nature of this relationship differed significantly between the first syllable in the free recall condition and the second syllable in the cued recall condition (t = 0.16, p <.01). Specifically, responses to the first syllable in the free recall condition were more predictive (i.e. steepest slope) of responses after the 24-hour delay than the second syllable in the cued recall condition. Responses to the second syllable in the free recall condition did not significantly differ from either of the two other conditions (Free recall, first syllable, t = 0.08, p = .08; Cued recall, second syllable, t = 0.08, p = .10).

Figure 2.

The relationship between the percentage of correct phonetic features produced during the last training trial for each syllable of each word and the percentage of correct phonetic features produced after the 24-hour delay. Responses for each syllable in each condition (free, cued) are displayed separately.

To investigate the nature of the last training trial by worst training trial interaction, we calculated the percentage of phonological features that were produced correctly after the delay based on different combinations of the worst trial and last trial productions during training. When the production accuracy of a given syllable of a given word during the last training trial was low (e.g. one standard deviation below the mean), the production accuracy during the worst training trial made little difference on the phonological precision after the delay. However, when the production accuracy during the last training trial was high (e.g. one standard deviation above the mean), the worst production of that syllable of that word made a difference in the phonological precision at which that syllable of that word was retrieved after the delay. Thus, for example, a syllable of a word that was produced with 100% accuracy during the last training trial and 75% accuracy during the worst training trial was likely to be produced with 86% accuracy after the delay. In contrast, a syllable of a word that was produced with 100% accuracy during the last training trial and only 19% accuracy during the worst training trial, was likely to be produced with 73% accuracy after the delay.

Discussion

Learning word forms to the level that they can be readily retrieved and produced accurately is challenging, especially for individuals with DLD. However, learning words to this level is important for students preparing to become professionals within their given fields. In McGregor, Gordon, et al., (2017) we demonstrated that retrieval-based practice is an effective study strategy to support learning word forms for post-secondary students with and without DLD. In the current study, we expanded upon these findings by determining how retrieval effort, based on the level of cuing provided during training, related to retrieval success after the 24-hour delay. We also determined how retrieval success during training related to retrieval success after the 24-hour delay. These analyses revealed three key findings. First, engaging in more effortful retrieval practice, in this case response to free recall prompts, led to more successful retrieval of word forms after the 24-hour delay. Second, retrieval success when defined as the number of times each syllable was successfully retrieved during practice positively predicted retrieval success after the delay. Third, retrieval success when defined as the phonological precision of productions during practice positively predicted the phonological precision of productions after the delay. We discuss each of these key findings in turn.

First, we found that training via free recall prompts supports better performance after the delay than training via cued recall prompts for individuals with and without DLD. In the original study (McGregor, Gordon, et al., 2017), we did not discern this difference when we compared performance after 24-hours for words trained in the various conditions (free recall, cued recall, passive exposures) within diagnostic groups (DLD,TD). However, the effect of training via free as opposed to cued recall was revealed when we combined the data across groups. This finding supports past work in which retrieval practice with fewer cues during training supports better delayed retrieval than retrieval practice with more cues (Adesope et al., 2017; Karpicke, 2017). This finding extends past work in two critical ways. First, we demonstrated that in the case of word learning, fewer cues during training contributed to more phonologically precise productions of forms after the delay. Second, fewer cues during training supported better retrieval after the delay for post-secondary students with and without DLD. This is the case even though students with DLD exhibited fewer successful productions and less phonologically precise productions than students with TD in response to free recall prompts during training (see McGregor, Gordon, et al., 2017). This finding supports the hypothesis that an unsuccessful retrieval attempt followed by feedback enhances encoding of target information (Kornell & Vaughn, 2016). Notably, across both free and cued recall conditions, participants received the same number of exposures to target forms. Yet, individuals demonstrated more phonologically precise productions after the 24-hour delay for the words that had been trained via free recall. Given that individuals with DLD demonstrate considerable difficulty encoding word forms from input, this is an educationally and clinically relevant finding. By administering free recall prompts with feedback during learning, individuals with DLD can enhance encoding from input and potentially decrease the total number of exposures typically required to add the word to their expressive vocabularies.

Second, we found that the probability of successfully producing each syllable of each word after the delay steadily increased as the number of successful retrievals during training increased. This finding is consistent with previous studies of word learning in typical adults (Ariel & Karpicke, 2018; Rawson & Dunlosky, 2011; Storkel, 2015). Notably, when guiding their own learning post-secondary students tend to follow a one-and-done strategy in that they discontinue studying an item after they successfully retrieve it one time (Dunlosky & Rawson, 2015). However, the current study and past research indicates that learners should continue studying an item until they successfully retrieve it multiple times to increase the probability of successful retrieval after a delay.

An additional finding from the current study was that when learners successfully retrieved syllables more times during training and the training involved the desirable difficulty of free recall, the probability of successful retrieval after the delay was even higher. Thus, maximizing successful retrievals during learning by reducing retrieval demands is not the ideal way to support encoding and delayed retrieval of word forms for individuals with and without DLD. Instead, learners should engage in effortful retrieval practice, but should continue to study words until they successfully retrieve them multiple times. Through engaging in continued practice that includes successful retrieval without cues, learners are more likely to build and strengthen retrieval routes. Thus, they increase the probability of successful retrieval after a delay when cues are not provided. Notably, Leonard and colleagues (2020) trained preschool-age children with and without DLD on novel word-referent pairs until they achieved a specific criterion. At that point, the children were provided with additional retrieval practice trials or additional presentations of the word-referent pairs. Both at the end of training and after a one-week delay, children with and without DLD produced more forms correctly when they had received extra retrieval practice as opposed to extra presentations of the words.

Third, we found that the phonological precision at which syllables of words were retrieved during training significantly predicted the phonological precision of post-delay productions. Thus, retrieval attempts that were ‘better errors’ in that they were closer to the target form were a sign of stronger learning than those that were ‘worse errors’ in that they were further from the target. Once again, retrieval difficulty played a significant role in that better errors made during free recall practice were associated with more precise productions after the delay than better errors made during cued recall practice. Additionally, the variability of productions during training, not just the last production, predicted the phonological precision at which syllables of words were produced after the delay. These findings suggest that practice retrieving and producing forms with a higher accurately multiple times under the more effortful retrieval condition does increase the phonological precision of productions after a delay. It is unrealistic for students to track the exact phonological precision of productions during study. However, they can be mindful of the general phonological precision at which they are retrieving and producing word forms during study. If they have not retrieved and produced a form with a high level of phonological precision multiple times, they may need additional practice with that form if they want to support successful retrieval and production at a later time point.

The primary of aim of the current study was to identify an optimal blend of effort and success to support word form learning in individuals with DLD. We determined that engaging in free recall practice with feedback, especially when that practice involved highly accurate productions over multiple attempts, was optimal. Based on findings from this study, there are several key questions that should be addressed through future research that could further elucidate how students can best support their academic and professional word learning goals. The first question is the number of study sessions to engage in and the ideal learning criterion needed for each session to maximize study efficiency. For example, Rawson and Dunlosky (2012) found that retrieving each item three times as opposed to one time correctly during an initial study session led to better retrieval after a delay but that this advantage diminished as the number of spaced study sessions increased. Given the robust findings on the advantages of spaced study sessions in supporting learning and retention (Cepeda, Pashler, Vul, Wixted, & Rohrer, 2006) it would be beneficial to investigate similar trade-offs for students who want to learn word forms to a high level of phonological precision.

An additional question is how to encourage students, especially students with DLD, to use effective study strategies to learn words to a high level of phonological precision. Current research demonstrates that students are more likely to implement retrieval-based learning strategies when they are given direct instruction about the benefits of this strategy (Ariel & Karpicke, 2018). In addition to training students on effective strategies, instructors can improve pedagogical practice by integrating retrieval-based practice and spaced practice in classroom instruction, assignments, and exams. For example, administering more frequent, low-stakes, assessments of learning throughout a course has been shown to be more effective at supporting learning and retention than administering several large exams (Roediger & Karpicke, 2006). Instructors should also consider how to implement assessments that not only support recognition of word forms, such as multiple-choice tests, but that support the students’ ability to encode and retain word forms at a high level of phonological precision. For example, students may benefit from attempting to retrieve word forms in fill-in-the-blank tests in which they must write the correct word form when they see its definition. Implementing these evidence-based practices in the classroom has the advantage of supporting the learning and retention of both students with typical development and students with language learning challenges.

In conclusion, retrieval-based practice is an effective study strategy that supports word learning and, in particular, may support learning word forms to a high level of phonological precision. While engaging in retrieval-based practice, students should not only consider how many times they retrieve each word successfully but should also consider the phonological precision at which they retrieve the words. Additionally, although practicing with fewer retrieval cues may lead to lower in-the-moment performance for students with and without DLD, this strategy can maximize their encoding and later retrieval of word forms. Given that the encoding of word forms is a particular weakness among individuals with DLD, this finding is especially important for students with DLD and the clinicians who support them. Overall, learning word forms to the level that they can be readily retrieved and produced with a high level of phonological precision is important for effective communication in educational and professional settings. Through engaging in effective study strategies that support learning words to a high level of phonological precision, students with DLD can support their larger educational and occupational goals.