Consistency and inconsistency in caregiver reporting of vocabulary

Abstract

Vocabulary checklists completed by caregivers are a common way of measuring children’s vocabulary knowledge. We provide evidence from checklist data from 31 children with and without autism spectrum disorder. When asked to report twice about whether or not their child produces a particular word, caregivers are largely consistent in their responses, but where they are inconsistent, these inconsistencies affect verbs more than nouns. This difference holds both for caregivers of children with autism spectrum disorder and caregivers of typically-developing children. We suggest that caregivers may be less sure of their child’s knowledge about verbs than nouns. This data converges with prior evidence comparing language samples of words children produce in a recorded interaction with checklist data, and it has implications for how researchers use checklist data in cases where the reliability of estimates of verb knowledge is critical.

Caregiver report of children’s vocabulary is a widely-used method in research. It is cheap, easy, and fast, and instruments such as the MacArthur-Bates Communicative Development Inventories (MCDI) have good concurrent and predictive validity (e.g., Feldman et al., 2005; Fenson et al., 1994). Nevertheless, researchers have queried the accuracy of caregiver report, at least for some types of words (Tardif, Gelman, & Xu, 1999; Tomasello & Mervis, 1994). Tardif et al. (1999) compared the words children produced in a language sample to checklist data, and found that English-speaking children produced verbs that their caregivers did not endorse more often than they produced nouns their caregivers did not endorse; the results did not hold for Mandarin-speaking mothers reporting on their Mandarin-learning children. Based on these results, Tardif and colleagues (1999) argued that caregiver report provides an underestimate of expressive verb knowledge for English learners. Similarly, Kover, Choi, and Naigles (2018) compared children’s productions in a language sample to caregiver checklist data and report that caregivers of children with and without autism spectrum disorder underreported verbs as compared to nouns. These two studies point to a potentially important pattern with respect to verbs versus nouns in caregiver report, but beyond these two studies there is scant empirical data.

Systematic discrepancies in the accuracy of caregiver report for verbs versus nouns are of both practical and theoretical interest. Practically, if verbs are indeed relatively poorly estimated as compared to nouns, it may be important to develop better assessments that specifically tap into children’s verb knowledge (e.g., Valleau, Konishi, Golinkoff, Hirsh-Pasek, & Arunachalam, 2018). The development of verb knowledge may be a better predictor of later grammatical skills than noun knowledge (Hadley, Rispoli, & Hsu, 2016), and therefore verb assessments may be particularly valuable.

Theoretically, our understanding of lexical development might be enhanced if we understand why caregiver report might differ for different kinds of words. One possibility is the one that Tardif et al. (1999) raised—that North American English-speaking parents are less attuned to their child’s acquisition of verbs than they are of nouns, perhaps because of a cultural emphasis on ensuring that children acquire nouns. For example, Chan, Brandone, and Tardif (2009) have proposed that children’s vocabulary development is embedded in a larger cultural difference in which people from individualistic cultures (e.g., North American English-speaking cultures) focus more on objects and agents as compared to people from collectivistic cultures, who tend to focus more on relations among agents. If this is true, it is likely that caregiver report will systematically underestimate children’s verb knowledge. It could also be that because verbs are hard to learn, children’s representations of the verbs are less robust than they are for nouns (e.g., Gleitman et al., 2005)—that is, caregivers’ difficulty in reporting on children’s knowledge of verbs reflects children’s own uncertainty about the exact meanings of verbs. In this case, caregivers may both underestimate and overestimate knowledge.

In the current study, we do not address all of these issues, but we provide evidence on the prerequisite issue: Taken together with previous studies comparing language sample data to checklist data, our study sheds light on how a word’s grammatical category—in particular, whether it is a noun or verb—affects caregiver reporting. Specifically, we examine inconsistency, or whether a caregiver provides different responses about the same word when completing two different vocabulary checklists at the same time.

We note from the outset that this study was not planned. Rather, we had access to a dataset of vocabulary checklists from caregivers of children with and without autism spectrum disorder (ASD) (Lord, Luyster, Guthrie, & Pickles, 2012) that we were examining for a different purpose, and we noted some inconsistencies in how caregivers reported on the same word on different checklists. Although the data includes an atypical population, we did not begin with the hypothesis that caregivers of children with ASD would show a noun/verb difference in vocabulary reporting relative to caregivers of children without the disorder. Kover, Choi, and Naigles (2018), in their comparison of language sample data to checklist data, found the aforementioned difference between nouns and verbs (with verbs underreported on the MCDI as compared to nouns) but no difference between typically developing children and children with ASD. Moreover, on most measures of caregiver language input such as mean length of utterance or quantity and diversity of language input, caregivers of children with ASD are similar to caregivers of typically developing children (e.g., Bang & Nadig, 2015); we do not therefore think that these caregivers would be especially different in how they evaluate their child’s verb knowledge. Note that our suggestion is not that children with ASD have the same vocabulary knowledge as typically developing children, but rather that caregivers of children with ASD report on their child’s knowledge equally successfully (or unsuccessfully) as caregivers of TD children. Therefore, we used this unique dataset both to address a question about caregiver reporting of vocabulary in general, and to see if this pattern held for ASD in particular.

Based on the above synthesis of prior literature and theories about differences between nouns and verbs, we hypothesized that caregivers might be more inconsistent when reporting on verbs than on nouns, regardless of whether or not their children have ASD. The dataset to which we had access included two different MCDI forms (Fenson et al., 1993) completed by the same caregivers at the same time point. Caregivers completed both the MCDI: Words and Gestures (MCDI 1), which asks caregivers whether their child “understands” and/or “says” each word on the checklist, and the MCDI: Words and Sentences (MCDI 2), which only asks caregivers to report about whether the child “says” each word. The MCDI 1 is intended for ages 8 to 18 months, and the MCDI 2 for ages 16 to 30 months. There is substantial overlap in the words listed on each form; they share 208 nouns and 55 verbs.

As a measure of inconsistency in caregiver report, we asked whether, for the same word, caregivers provided a different response for the “says” measure on the MCDI 1 than the MCDI 2. This difference could take multiple forms. A caregiver could report that the child “understands” a word on the MCDI 1 but that the child “says” the word on the MCDI 2; the report could be “neither” on the MCDI 1 but “says” on the MCDI 2; or, the report could be “doesn’t say” on the MCDI 2 and “says” on the MCDI 1. See Table 1. Note that this measure is similar to but different from test-retest reliability on the MCDI, which was determined by the test developers, who sent a second identical form to families a short interval after the initial completion of the form (Fenson et al., 2007). The authors reported that test-retest reliability was high for the production measure on the MCDI 1 (0.95, at an average interval of 1.35 months) and the MCDI 2 (0.95, at an average interval of 1.38 months) (though item-by-item stability may be lower for both nouns and verbs; Yoder, Warren, & Biggar, 1997). Because we are measuring reliability across two different forms, instead of the same form, it is less likely that caregivers will simply remember their responses from the previous administration. Further, by dividing the checklist into nouns and verbs, we can examine reliability for verbs specifically, which may be overshadowed in a combined measure because nouns outnumber verbs on the CDI as a whole.

Table 1.

Possible types of inconsistency by which column(s) the caregiver did or did not check for each word on each form. The fourth logical possibility, in which the word was unchecked on the MCDI 2, and only “understands” was checked on the MCDI 1, is not available because it is assumed that is a child is reported to say the word, they also understand it.

MCDI 1		MCDI 2
Understands	Says	Says
checked	unchecked	checked
unchecked	unchecked	checked
checked	checked	unchecked

This is not the only potential measure of accuracy in caregivers’ assessment of children’s knowledge but to the extent that it converges with previous work using language samples (Kover, Choi, & Naigles, 2018; Tardif et al., 1999), we will support the claim that checklist data is less accurate for verbs than for nouns. We think that measuring inconsistency using vocabulary checklists, as opposed to language samples as in prior work (Kover et al., 2018; Tardif et al., 1999), is important because language samples only capture a small portion of a child’s language production, dependent on the context in which the sample was obtained. Tardif et al. (1999) note that for both nouns and verbs, during the 30-minute language sample children produced approximately 10–15% of the words that were checked off on the CDI. It could in principle be the case that the remaining 85–90% is biased toward nouns; that is, that there were many more nouns than verbs that caregivers reported that the child can say but the child didn’t produce them in the language sample because the context didn’t support it. If this were true, their finding of more underreporting for nouns and verbs would be a function of the context of language sampling rather than true underreporting. Moreover, an additional benefit of using caregiver report—the MCDI in particular—in addition to a language sample is that it includes a measure of receptive knowledge, providing a broader measure of word knowledge than expressive skills alone. In sum, then, converging evidence using a different measure will make a stronger case than evidence from one approach alone.

Methods

Participants

The data set consists of 83 pairs of MCDI 1 and MCDI 2 checklists completed by one of the child’s primary caregivers at the same point. Because some of the checklist pairs were from the same child at different lab visits at different ages, these 83 pairs of checklists come from 31 different children. The children ranged in age from 14 to 43 months (mean age = 27 months, SD = 4.4 months) at the time of data collection.

Although the MCDI forms each specify an age range, the publishers note that they “may be used with older, developmentally-delayed children” (CDI Advisory Board, n.d.). When studying groups with developmental and language delays, researchers may provide caregivers with both forms, because they may not know which one is appropriate for the child’s level. This applies to our dataset; 21 of the 31 children had received a diagnosis of ASD, which is generally associated with concomitant language delays, and the other 10 typically developing (TD) children had never received an ASD diagnosis.

Coding

For each overlapping item on each pair of checklists, we examined whether the caregiver was consistent in reporting that the child either did say the word (i.e., the “says” column was checked on both forms) or did not say the word (i.e., the “says” column was not checked on either form, whether or not the “understands” column was checked on the MCDI 1). If it was inconsistent, we also recorded the direction of the inconsistency; that is, if it was the MCDI 1 or 2 that indicated the child said the word.

We then coded for whether the words were nouns or verbs; other categories were excluded. In English, many words on the MCDI are members of multiple grammatical categories; for example, the word “nap” can be either a noun or a verb. However, because the vocabulary forms contain subsections with headers for different categories of words, it is likely that caregivers completing the form correctly identify which is intended by referring to the category name and the other words it appears with in the same category. For this reason, we followed Tardif et al. (1999) in defining the list of nouns and verbs: nouns were all words in the categories Animals (Real or Toy), Vehicles (Real or Toy), Toys, Food and Drink, Clothing, Body Parts, Small Household Items, Furniture and Rooms, Outside Things and Places To Go; verbs were only those words in the Actions category. Some nouns on the MCDI were excluded from our analysis, including all of the words from the People category and from the Games and Routines category. Several of these are relational nouns that take an argument (e.g., friend of mine) or could function as proper nouns for young children (e.g., daddy). Several others (e.g., breakfast, lunch) can denote events rather than objects and may more closely resemble verbs than nouns with respect to how they are acquired (Arunachalam & He, 2018).

Results and Discussion

Caregivers were largely consistent: 90% of responses were consistent across the two forms (ASD group: 89%, TD group: 90%). However, there were more inconsistencies with verbs (14%; ASD group: 15%, TD group: 12%) than with nouns (10%; ASD group: 10%, TD group: 9%). Statistical analyses were conducted using the lme4 package (Bates, Mächler, Bolker, & Walker, 2015) in R version 3.5.1 (R Core Team, 2018). A binomial mixed-effects regression with grammatical category (noun vs. verb), group (ASD vs. TD), and their interaction as fixed factors; and with word, child, and lab visit nested within child (because some children visited the lab more than once) as random factors; revealed that this difference between noun and verb consistency was statistically significant (β = −0.47, p < .0001). There was no significant effect of group (ASD vs. TD; β = 0.12, p = .72) and no significant interaction between group and grammatical category (β = −0.21, p = .10). See Table 2.

Table 2.

Model parameter estimates for mixed-effects regression testing the roles of ASD diagnosis and grammatical category on inconsistent vocabulary reporting.

Parameter	Estimate	Standard Error	z-value	p-value
Intercept	−2.84	0.38	−7.53	<.001
Group (ASD, TD)	0.17	0.45	0.36	0.72
Grammatical Category	−0.32	0.11	−2.79	<.01
Group × Grammatical Category	−0.21	0.13	−1.68	0.094

We then conducted post-hoc tests on each group separately (ASD, TD) to ensure that the finding held for each. It did (ASD: β for grammatical category = −0.53, p < .0001; TD: β for grammatical category = −0.32, p < .01).

The inconsistencies were more likely to take the form of reporting more knowledge on MCDI 1 than on MCDI 2 (χ²(1) = 94.5, p < .0001): 60% of the inconsistencies were such that children were reported to “say” the word on the MCDI 1 form and left blank (indicating the child does not say the word) on the MCDI 2 form, with the remaining 40% having the child reported as either only “understanding” the word on the MCDI 1 form or left blank (indicating the child does not say the word) and checked as an indicator of saying the word on the MCDI 2 form. This seems reasonable if we imagine that caregivers fell into a pattern of assuming more knowledge from their child if the words on the checklist are easy, and assuming less knowledge if the words on the checklist are more difficult.

Role of age and vocabulary

These results suggest that caregivers have more difficulty reporting on their child’s verb knowledge than noun knowledge. However, one possibility that might weaken this conclusion is that older children, or children with higher vocabulary scores, are driving the effect. Older children and children with larger vocabularies will also typically have more verbs in their vocabularies, and the sheer increase in words that caregivers have to report on may lead to more inconsistencies. Therefore, we conducted a follow-up analysis in which we added the child’s age in months and proportion of CDI words produced (calculated only from words that were reported consistently across forms) as fixed factors in the model, each centered around their mean value, as well as interactions between grammatical category and age and grammatical category and vocabulary size. This analysis, too, yielded a main effect of grammatical category, but also a main effect of vocabulary size (but not of age) and interactions between grammatical category and age and grammatical category and vocabulary size. As in the previous analysis, there was no effect of diagnosis group (ASD vs. TD). See Table 3. The interaction between grammatical category and vocabulary size reveals that as vocabulary size increased, so did the proportion of inconsistent reports for verbs more so than for nouns. However, this interaction should be interpreted cautiously because vocabulary size was only calculated for the consistently reported data, and as we have seen, inconsistent reporting affected verbs more than nouns. Although there was no main effect of age, the trend was, as expected, toward more inconsistency for older children than younger children, and the interaction between grammatical category and age indicates that this increase in inconsistency with age was stronger for nouns than for verbs. Overall, we conclude that the difference between noun and verb reporting holds even when age and vocabulary size are taken into account.

Table 3.

Model parameter estimates for mixed-effects regression testing the roles of ASD diagnosis, grammatical category, age, and vocabulary size in inconsistent vocabulary reporting.

Parameter	Estimate	Standard Error	z-value	p-value
Intercept	−2.87	0.38	−7.55	<.001
Group (ASD, TD)	0.18	0.47	0.34	0.73
Grammatical Category	−0.38	0.070	−5.43	<.001
Age	0.031	0.034	0.88	0.38
Vocabulary	1.27	0.62	2.065	0.039
Grammatical Category × Age	0.044	0.011	4.025	<.001
Grammatical Category × Vocabulary Size	−1.61	0.21	−7.77	<.001

Role of knowledge of the words

Why are caregivers less consistent in reporting on verb knowledge than noun knowledge? One possibility is that because verbs tend to be acquired later and with more difficulty than nouns (e.g., Fenson et al., 1994; Gentner, 1982; Gleitman et al., 2005), caregivers have less experience hearing their child’s use of verbs and less certainty about their abilities with this category. To explore this possibility, we asked whether there was an association between how well known a word was across the sample—that is, excluding all inconsistent reporting, how often it was reported as “said”—and how often it was reported inconsistently. See Figure 1. Note that at the upper end of the y-axis, where there are a lot of inconsistencies, there is a fair amount of missing data because inconsistencies could not be included in the measure of whether the word was known. Nevertheless, the pattern is suggestive of the opposite of our predictions. It appears that better-known verbs are more, not less, likely to be reported inconsistently (r(53) = 0.32, p = 0.016); no such association holds for nouns (r(206) = −0.023, p = 0.75). It is unlikely, then that the rarity of hearing any given verb is a major driving force in caregivers’ abilities to report on it consistently. Consider as illustration the words that appear under the Furniture and Rooms heading on the CDI, including words like “bed,” “door,” and “garage.” Excluding inconsistencies, this category is relatively poorly represented in children’s productive vocabularies; the mean percentage of words “said” in this category, averaging across children, is 30%, while for the Action words (i.e., verbs) it is 33%. Nevertheless, the Action category has more inconsistencies (14%) than the Furniture category (9%), or indeed any other category of nouns. See Table 4. Appendix A lists all of the included words and the proportion of time they were reported inconsistently.

Figure 1.

Scatterplot illustrating for each word the association between the proportion of times it was reported inconsistently and the proportion of times it was reported as known of those for which it was reported consistently.

Table 4.

Proportion reported as known (only including consistent responses) and proportion inconsistent reporting across categories of words as listed on the MCDI.

Category	Proportion Known	Proportion Inconsistent
Action Words	0.34	0.14
Animals	0.49	0.097
Body Parts	0.53	0.10
Clothing	0.40	0.10
Food and Drink	0.49	0.095
Furniture and Rooms	0.30	0.093
Household Items	0.38	0.10
Outside Things	0.40	0.11
Toys	0.61	0.078
Vehicles	0.62	0.072

Role of characteristics of the word

Another possibility is that it has something to do with the referents of the words. On average, early-acquired verbs tend to be less imageable than early-acquired nouns (e.g., Gillette et al., 1999)—that is, it is easier to call up a visual image for noun concepts than verb concepts—and there is evidence that imageability plays a role in how early a word is learned, with nouns privileged over verbs (e.g., Hansen, 2017; Ma, Golinkoff, Hirsh-Pasek, McDonough, & Tardif, 2009; McDonough, Song, Hirsh-Pasek, Golinkoff, & Lannon, 2011). This may be further exacerbated by the fact that we limited analysis to concrete nouns listed on the CDI, whose referents are generally all imageable objects.

A similar, potentially relevant measure is “babiness” (Perry, Perlman, & Lupyan, 2015), or how much adults associated the word with babies. This measure is associated with early acquisition (Braginsky et al., 2019). Considering again as an example the Furniture category of nouns in comparison to the Action category of verbs, we examined those words on the CDI that are both included in our analysis (i.e., those that occur on both the CDI 1 and CDI 2) and that Perry et al. (2015) provided babiness ratings for. The Furniture words have a lower mean babiness rating of 2.91 than the Action word rating of 4.49 (t(66) = 2.97, p < .005), despite showing less inconsistency; it is unlikely, then, that insufficient “babiness” is what makes it difficult for caregivers to attend to children’s use of verbs.

The last possibility, which we do not have positive evidence for but which is compatible with the evidence thus far, is that it is something categorical about nouns vs. verbs, or object-denoting vs. event-denoting words. Future work will be needed to better understand what causes inconsistencies in caregiver report, and whether it is related more to the word’s grammatical properties (e.g., noun vs. verb) or to conceptual properties (e.g., what the word means or the contexts in which it is used).

General Discussion

Evaluating children’s vocabulary knowledge is one of the most fundamental tasks in understanding their language abilities. Caregiver report of vocabulary knowledge is a widely- used strategy and the MCDI has high validity and reliability (Fenson et al., 1994). Nevertheless, caregiver report has some drawbacks (e.g., Pan, Rowe, Spier, & Tamis-LeMonda, 2004; Robinson & Mervis, 1999), and verbs may be more affected than nouns (Tardif, Gelman, & Xu, 1999; Tomasello & Mervis, 1994). The current study demonstrates that when asked to report twice, on two different checklist forms, about whether or not their child produces a particular word, caregivers of children with and without ASD are more likely to produce inconsistent responses for verbs than for nouns. We take this as a measure of caregivers’ confidence in or certainty about their child’s knowledge of the word.

This is of course not the only potential measure of the accuracy or reliability of caregiver report. Language sample data, in which child speech is recorded and transcribed to determine what words children actually say, has been compared to checklist data and also shows that verbs are less accurately reported on by caregivers than nouns—that is, children produce more verbs that their caregivers do not check than nouns that their caregivers do not check (Kover et al., 2018; Tardif et al., 1999). The main advantage of language sample data is that it is obtained directly from the child, and therefore less likely to be affected by reporter bias (bias is possible in the transcription and coding processes, but researchers have developed methods to minimize these). However, as we noted earlier, language samples are restricted to small portions of the child’s language production; this is especially problematic for those language samples that are collected within specific contexts. Kover et al. examined caregiver-child play interactions, while Tardif et al. examined both toy play and book-reading. The context in which samples are taken strongly affects the type of language that caregivers use (e.g., Tamis-LeMonda, Kuchirko, Luo, Escobar, & Bornstein, 2017) (and even the type of toy during toy play has a substantial effect, e.g., Sosa, 2016). It is possible that the results reported by Kover et al. and Tardif et al. would have differed had the language samples been collected in different contexts.

An advantage of our method of measuring inconsistency within the CDI data is that it controls for context by looking within the checklist itself. The data in both cases is intended to represent the child’s productions overall, not just within a particular time. But there are limitations of our approach too. For example, when caregivers are inconsistent, we cannot know from our study which response was accurate. Language sample analysis can only expose underestimation (unless nearly every utterance the child produces is recorded, e.g., Roy et al., 2006), and so both methods share this limitation that we do not know the child’s true knowledge.

The current study’s finding of a difference between nouns and verbs thus adds a new dimension to our understanding of the previous language sample results. Underestimation of verbs could be because caregivers of English learners believe their child does not produce verbs that the child does in fact produce. Our results, however, instead suggest that caregivers are uncertain—sometimes they report that the child produces a verb and other times they report that the child does not. This nuance may have implications for our understanding of why verbs are relatively difficult to acquire. Research has revealed several differences between nouns and verbs that may impact their acquisition, several of which may be relevant in the current context. One is that caregivers do not often use verbs ostensively to point out ongoing actions (e.g., Gleitman & Gleitman, 1992; Tomasello & Kruger, 1992). Because children’s language production resembles their caregivers’ in many ways (e.g., Tomasello, 1992), children, too, may not use verbs ostensively as often as they do nouns. In turn, caregivers may have difficulty noticing children’s use of verbs, as they are not time-locked to the event being labeled; and in fact, past-tense forms seem particularly challenging for some caregivers to report accurately (Jyotishi, Fein & Naigles, 2017). If true, this suggests a common underlying cause for the difficulty for children of acquiring verbs and the difficulty for caregivers of reporting on children’s knowledge of verbs.

A second difference between nouns and verbs that may be relevant for this discussion is that early-acquired verbs in English are rated substantially lower on imageability than early-acquired nouns (e.g., Gillette et al., 1999; Ma, Golinkoff, Hirsh-Pasek, McDonough, & Tardif, 2009; McDonough, Song, Hirsh-Pasek, Golinkoff, & Lannon, 2011). This is thought to be one reason why verb acquisition is challenging; to determine the correct referent event in any given situation, as well as the verb’s correct extension to other events in the same category, children must abstract over irrelevant characters of the event and categorize together events that may look very different (e.g., a soccer player kicking and a baby kicking). It may be that the relative abstractness of the concepts denoted by English verbs also means that caregivers have difficulty calling to mind instances in which children have produced a particular verb; while it may be easy to think of times a child was in the vicinity of balls and had the opportunity to utter the word “ball,” it may be more difficult to think of times the child was in the vicinity of kicking events and had the opportunity to utter the word “kick.” Again, these considerations suggest that the same properties of lexical items are at play both in thinking about the acquisition process and in thinking about caregiver report.

Both of these considerations would lead to the prediction that caregivers would systematically underestimate verb knowledge, because caregivers do not often hear, or cannot call to mind occasions on which they do hear, their child use a particular verb. Although caregivers may indeed underestimate verb knowledge for these reasons, as suggested by Tardif et al. (1999) and Kover et al. (2018), our results indicate that in addition, caregivers sometimes have conflicting or uncertain judgments about their child’s knowledge. This could reflect something about the nature of the child’s knowledge about verbs. Words like verbs that are considered relatively difficult to acquire may require a more extended period of “slow mapping” (e.g., Swingley, 2010) than labels for object categories, and during this extended period children may display only emergent knowledge of the verb (e.g., Gleitman et al., 2005). For example, in novel verb learning studies, children’s initial representations for a new verb are often restricted to the particular objects with which the verb was initially encountered (e.g., Arunachalam & Waxman, 2015; Imai et al., 2008). In some cases, at least, an extended learning period may be required before children have identified the precise category of events and associated objects denoted by the verb. If a child initially understands the verb “kick” as applying only to balls, requests to kick other objects may be met with confusion or non-compliance, even as the child successfully comprehends and/or produces “kick” correctly in the context of balls. This may lead the caregiver to be unsure of whether the child knows the word. Thus, a relative difficulty in acquiring the meanings of verbs may translate to less robust or less specific representation of the verb for the child, which in turn results in less confidence from the parent about whether the child can understand and/or produce that verb.

One potential interpretive issue bears mention. Pine, Lieven, and Rowland (1996) argue that checklist data is not a reliable indicator of absolute differences in knowledge of different grammatical categories such as nouns vs. verbs. They argue that checklist data should not be interpreted as a measure of absolute language knowledge, or even absolute knowledge of one word type as compared to another. This is in part because the constructors of the checklist selectively sampled from the lexicon to decide which words appear on the list; and there is no reason to believe that, for example, a child who is reported to know 50% of the nouns on the checklist should also know 50% of the verbs. Instead, they suggest that the value of checklist data is in evaluating individual differences. Our claim, by contrast, is that some words may be more accurately reported on than others. Our data is particularly relevant for cases in which one wants to look at individual differences in some other measure—say, grammar—and ask whether they are predicted by the percent of nouns children know on the checklist and/or the percent of verbs children know. We suggest that converging evidence from multiple methods supports the idea that verbs may be less reliably reported on than nouns on caregiver report checklists.

A limitation of the current study is that we do not have access to detailed information about how the checklists were completed by caregivers. That is, we do not know in which order the caregiver completed the MCDI 1 and MCDI 2 or whether more than one caregiver was involved in completing each checklist. However, while this may affect the reliability of absolute estimates of consistency, we are unaware of any reason why these factors would differentially affect verbs versus nouns. Additionally, both forms were filled out during the same day, at the same lab visit, so it is unlikely that there was wide variation in completion patterns. Therefore, our primary question of whether verbs are reported on less consistently than nouns remains addressable from the current data set.

Although overall, caregivers were largely consistent in reporting vocabulary knowledge, our results converge with prior work (Kover et al., 2018; Tardif et al., 1999), suggesting that for a precise measure of verbs in particular, direct assessments of children in addition to caregiver report measures may be valuable. It will be an important step for future studies to directly and carefully evaluate child verb knowledge and compare it to parental report. However, as of yet, we do not have an assessment that focuses on verbs, is appropriate for young children, and depicts verb referents in more representative ways than the static pictures of actions shown in assessments such as the PPVT (Peabody Picture Vocabulary Test; Dunn & Dunn, 2007). Some recent studies have focused on this issue using eye gaze measures (e.g., Bergelson & Swingley, 2013; Nomikou, Rohlfing, Cimiano, & Mandler, 2019; Valleau et al., 2018) or pointing tasks (Konishi et al., 2016). We hope that the current study provides additional support for the idea that a verb-focused assessment, for example, using eye-tracking to evaluate children’s gaze as they view dynamic scenes depicting verb referents, may improve our precision of children’s verb knowledge as compared to caregiver report alone.

Appendix A.

List of words and the proportion of time they were reported inconsistently, ordered by the latter.

Nouns		Verbs
Word	Proportion Inconsistent	Word	Proportion Inconsistent
House	0.22	Go	0.22
Chicken (food)	0.19	Hug	0.22
Slide	0.19	Throw	0.22
Plate	0.18	Walk	0.20
Fish (animal)	0.17	Close	0.19
Bathtub	0.17	Touch	0.19
Lamp	0.17	Jump	0.18
Pants	0.17	Read	0.18
Store	0.17	Ride	0.18
Cow	0.16	Wipe	0.18
Noodles	0.16	Swing	0.17
Plant	0.16	Watch	0.17
Toast	0.16	Blow	0.17
Trash	0.16	Look	0.17
Bed	0.15	Play	0.17
Chair	0.15	Push	0.17
Dish	0.15	Help	0.16
Dress	0.15	Kick	0.16
Flower	0.15	Run	0.16
Hammer	0.15	Sing	0.16
Home	0.15	Take	0.16
Jacket	0.15	Wash	0.16
Lamb	0.15	Draw	0.14
Outside	0.15	Eat	0.14
Sun	0.15	Put	0.14
Brush	0.13	Sleep	0.14
Cake	0.13	Drink	0.13
Finger	0.13	Write	0.13
Swing	0.13	Bump	0.13
Owl	0.13	Kiss	0.13
Rock	0.13	Stop	0.13
School	0.13	Tickle	0.13
Sweater	0.13	Dance	0.12
Tongue	0.13	Bite	0.12
TV	0.13	Cry	0.12
Zipper	0.13	Fall	0.12
Bird	0.12	Feed	0.12
Bowl	0.12	Get	0.12
Box	0.12	Pull	0.12
Cat	0.12	See	0.12
Cup	0.12	Clean	0.11
Diaper	0.12	Give	0.11
Doll	0.12	Bring	0.11
Frog	0.12	Splash	0.11
Hand	0.12	Swim	0.11
Hat	0.12	Break	0.10
Keys	0.12	Love	0.10
Knee	0.12	Say	0.10
Leg	0.12	Smile	0.10
Meat	0.12	Drive	0.084
Milk	0.12	Hurry	0.084
Moon	0.12	Open	0.084
Mouse	0.12	Show	0.084
Mouth	0.12	Hit	0.072
Work	0.12	Finish	0.060
Party	0.12
Shovel	0.12
Sky	0.12
Soap	0.12
Sock	0.12
Spoon	0.12
Stroller	0.12
Table	0.12
Telephone	0.12
Toothbrush	0.12
Toy	0.12
Chicken (animal)	0.11
Broom	0.11
Button	0.11
Candy	0.11
Cracker	0.11
Crib	0.11
Donkey	0.11
Duck	0.11
Ear	0.11
Face	0.11
Drink	0.11
Fish (food)	0.11
Food	0.11
Glass	0.11
Jeans	0.11
Kitchen	0.11
Light	0.11
Nose	0.11
Paper	0.11
Pig	0.11
Pool	0.11
Raisin	0.11
Scissors	0.11
Stove	0.11
Toe	0.11
Tooth	0.11
Towel	0.11
Window	0.11
Block	0.096
Booboo	0.096
Boots	0.096
Bunny	0.096
Butterfly	0.096
Cheese	0.096
Egg	0.096
Giraffe	0.096
Glasses	0.096
Hair	0.096
Head	0.096
Horse	0.096
Watch	0.096
Kitty	0.096
Monkey	0.096
Peas	0.096
Pen	0.096
Puppy	0.096
Shirt	0.096
Stairs	0.096
Star	0.096
Tree	0.096
Tummy	0.096
Vacuum	0.096
Animal	0.084
Bear	0.084
Bib	0.084
Bicycle	0.084
Butter	0.084
Cheerios	0.084
Comb	0.084
Couch	0.084
Dog	0.084
Door	0.084
Eye	0.084
Orange	0.084
Fork	0.084
Garage	0.084
Goose	0.084
Highchair	0.084
Juice	0.084
Lion	0.084
Motorcycle	0.084
Water (outside)	0.084
Oven	0.084
Pajamas	0.084
Park	0.084
Picture	0.084
Pony	0.084
Potty	0.084
Refrigerator	0.084
Sheep	0.084
Sink	0.084
Spaghetti	0.084
Turkey	0.084
Bee	0.072
Bellybutton	0.072
Book	0.072
Carrots	0.072
Coffee	0.072
Drawer	0.072
Firetruck	0.072
Money	0.072
Penguin	0.072
Pizza	0.072
Rain	0.072
Shoe	0.072
Shorts	0.072
Snow	0.072
Squirrel	0.072
Tiger	0.072
Train	0.072
Turtle	0.072
Zoo	0.072
Airplane	0.060
Arm	0.060
Balloon	0.060
Bathroom	0.060
Beach	0.060
Bedroom	0.060
Bottle	0.060
Bug	0.060
Car	0.060
Cereal	0.060
Coat	0.060
Deer	0.060
Ice cream	0.060
Pillow	0.060
Purse	0.060
Rocking chair	0.060
Teddy bear	0.060
Truck	0.060
Banana	0.048
Beads	0.048
Bread	0.048
Bubbles	0.048
Cheek	0.048
Clock	0.048
Cookie	0.048
Elephant	0.048
Garden	0.048
Medicine	0.048
Penny	0.048
Apple	0.036
Backyard	0.036
Ball	0.036
Blanket	0.036
Bus	0.036
Living room	0.036
Play pen	0.036
Radio	0.024
Water (beverage)	0.012
Necklace	0.012
Church	0.00