Psychometric Evaluation of Lexical Diversity Indices: Assessing Length Effects

Abstract

Purpose

Several novel techniques have been developed recently to assess the breadth of a speaker's vocabulary exhibited in a language sample. The specific aim of this study was to increase our understanding of the validity of the scores generated by different lexical diversity (LD) estimation techniques. Four techniques were explored: D, Maas, measure of textual lexical diversity, and moving-average type–token ratio.

Method

Four LD indices were estimated for language samples on 4 discourse tasks (procedures, eventcasts, story retell, and recounts) from 442 adults who are neurologically intact. The resulting data were analyzed using structural equation modeling.

Results

The scores for measure of textual lexical diversity and moving-average type–token ratio were stronger indicators of the LD of the language samples. The results for the other 2 techniques were consistent with the presence of method factors representing construct-irrelevant sources.

Conclusion

These findings offer a deeper understanding of the relative validity of the 4 estimation techniques and should assist clinicians and researchers in the selection of LD measures of language samples that minimize construct-irrelevant sources.

Lexical diversity (LD) has been used in a wide range of areas, producing a rich history in speech-language pathology. For example, it has been used to understand the relationship between phonological processing and the development of reading skills (e.g., Smith, 2009); to differentiate children who are developing typically from children with specific language impairment (e.g., Thordardottir & Namazi, 2007); to screen and identify bilingual children with language deficits (Kapantzoglou, Fergadiotis, & Restrepo, 2010); to evaluate the progress of children after cochlear implantation (Ertmer, Strong, & Sadagopan, 2003); to differentiate individuals with aphasia from those who are neurologically intact (e.g., Lind, Kristoffersen, Moen, & Simonsen, 2009); to capture the differences between individuals with fluent and nonfluent aphasia (Fergadiotis & Wright, 2011); to inform models of language processing in aphasia (e.g., Gordon, 2008); to assess the efficiency of therapeutic approaches (Rider, Wright, Marshall, & Page, 2008); and to use as a possible early marker of dementia (Bucks, Singh, Cuerden, & Wilcock, 2000).

Both researchers and clinicians have utilized a variety of measures to capture the LD of language samples. However, identifying a robust approach to measuring LD has been challenging, because typical measures of LD reflect not only the construct of interest (i.e., LD) but also construct-irrelevant variation, such as the length of samples. As a result, the score interpretation of these measures is problematic. For example, the type–token ratio (TTR; Chotlos, 1944; Templin, 1957), which is one of the most used indices in speech-language pathology, is inherently flawed because it varies as a function of sample length (Heaps, 1978). Therefore, comparisons across language samples of different speakers, or even across different samples produced by the same speaker, may be confounded by sample length. Standardizing sample size is not a satisfactory solution to this problem because researchers would have to agree on the number of tokens to estimate TTR—an unrealistic expectation given the multitude of applications. Further, discarding text may reduce a language sample's integrity and lead to spurious results, because the introduction of novel vocabulary in the truncated language sample would not necessarily be uniformly spread throughout the sample (Youmans, 1991).

Several tools have emerged from the field of computational linguistics that can be used to measure the LD of a language sample. All four measures described here claim to be relatively robust to length variation, a feature that allows for comparisons of discourse samples within and between participants, as well as across studies, without requiring truncation of language samples. Further, because they are estimated from the whole transcript, they should be less prone to measurement error due to clustering of novel content words. With a few exceptions, these measures have been used in limited applications in the field of speech-language pathology. The purpose of the current study is to explore the validity of these four LD measures. Before delineating the goals of our study in greater detail, we describe how these four measures are computed and the literature addressing their validity.

Computation of Four LD Measures

Measure D

The measure D (Malvern & Richards, 1997; McKee, Malvern, & Richards, 2000) reflects the LD of a language sample by capturing how fast TTR decreases in the sample. Computationally, estimating D involves a series of random text samplings to plot an empirical curve of TTR versus number of tokens for a sample. Initially, 35 tokens are randomly drawn from the text without replacement, and the TTR is estimated. This process is repeated 100 times, and the average TTR for 35 tokens is estimated and plotted. The same routine is then repeated for subsamples from 36 to 50 tokens. The average TTR for each subsample of increasing token size is subsequently plotted to form the empirical curve. Then a theoretical curve is produced that maximizes its fit to the empirical TTR curve using the least-squares approach, with TTR calculated as follows (McKee et al., 2000):

$$TTR=\frac{D}{N}\left[{\left(1+2\frac{N}{D}\right)}^{\frac{1}{2}}-1\right]\text{.}$$ (1)

Lower D values reflect steeper theoretical curves. Because D is the product of a stochastic process, its value varies each time the program is run. For that reason, the whole process is repeated three times; the final D value is the average of the three runs.

Maas Index

Another approach researchers have used to minimize the effect of sample length is linearizing. Conceptually, this approach is based on the assumption that the TTR curve can be fitted relatively well by a logarithmic curve. In theory, if one could transform the relationship between N and TTR to achieve linearity, it would be straightforward to use regression analysis to estimate the slope of that linear relationship. This approach, recently advocated by McCarthy and Jarvis (2007), was first described by Maas (1972; a²):

$$a^2=\frac{\left(\mathit{logTokens}-\mathit{logTypes}\right)}{lo{g}^2\mathit{Tokens}}\text{.}$$ (2)

Measure of Textual Lexical Diversity (MTLD)

Another tool that has been proposed for estimating LD is the MTLD (McCarthy, 2005). Conceptually, for any given sample, MTLD reflects the average number of consecutive words for which a certain TTR is maintained. For language samples with high LD, multiple words may be produced before TTR drops below a certain cutoff score, because there is a lower propensity of the same words being repeated. During the estimation process, TTR is estimated after each word. For example, “I” (TTR = 1.00) “am” (TTR = 1.00) “tired” (TTR = 1.00) “but” (TTR = 1.00) “I” (TTR = .800) “am” (TTR = .667) “also” (TTR = .714) and so forth. However, when a default TTR value (e.g., .720) is reached, the factor count increases by a value of 1, and the TTR evaluations are reset. The same process is repeated until the last token of the language sample has been added and the TTR has been estimated. Then the total number of words in the text is divided by the total factor count. For example, if the text has 300 tokens and the factor count is 4.1, then the MTLD value is 73.17. Subsequently, the whole text in the language sample is reversed, and another score of MTLD is estimated. The forward and reversed MTLD scores are averaged to provide the final MTLD estimate.

Moving-Average Type–Token Ratio (MATTR)

Conceptually, the moving-average type–token ratio MATTR (Covington & McFall, 2010) calculates the LD of a sample using a moving window that estimates TTRs for each successive window of fixed length. Initially, a window length is selected—for example, 10 words—and the TTR for words 1–10 is estimated. Then the TTR is estimated for words 2–11, then 3–12, and so on to the end of the text. For the final score, the estimated TTRs are averaged.

Validity of LD Score Interpretations

The validity of LD scores would be threatened to the extent that construct-irrelevant sources of variance were found to influence the observed scores. In other words, in the face of evidence that observed scores varied systematically as a function of a variable other than the construct of interest, the viability of the claims could be compromised (e.g., Kane, 2009). That would be the case if, for example, scores obtained from a specific approach were found to reflect more than just LD. In that case, it would be difficult to argue in favor of such an approach and the conclusions reached when using that approach.

The validity of D has been investigated in several studies (e.g., Durán, Malvern, Richards, & Chipere, 2004; Malvern & Richards, 1997; Richards & Malvern, 1997), in which estimates of D were found to correlate strongly with well-validated measures of language and also with developmental and demographic variables. However, other studies have suggested problems with this technique. For example, Owen and Leonard (2002) found that mean LD as estimated by D varied as a function of sample length. Samples that were truncated to 250 words had a significantly lower mean D score compared to samples that were truncated to 500 words. On the basis of these findings, Owen and Leonard concluded, “It appears that D does not entirely avoid the problem of sample size influence” (p. 935). In addition, Fergadiotis, Wright, and West (2013) evaluated four measures of LD to determine how effective the measures were at measuring LD for people with aphasia. They applied MTLD, MATTR, D, and the hypergeometric distribution–based D (McCarthy & Jarvis, 2007). Results from a confirmatory factor analysis suggested that MTLD and MATTR were relatively pure measures of LD, whereas D and the hypergeometric distribution–based D covaried with TTR, even after accounting for the variance that was attributed to LD. This relationship suggested that the hypergeometric distribution–based D and D were affected by language sample size. Finally, Koizumi and In'nami (2012) investigated D and MTLD in Japanese second language learners of English and reached similar conclusions regarding the validity of these two indices.

With respect to MTLD, McCarthy and Jarvis (2010) explored its convergent and discriminant validity. The authors used three indices of LD—Maas, D, and Yule's K (a probabilistic index; Yule, 1944)—to investigate its convergent validity. They found that MTLD was moderately to strongly correlated with all three indices (.84, .69, and .85, respectively). The high coefficients were evidence of its convergent validity. Further, MTLD did not correlate strongly with TTR (r = .32), a measure known to be influenced by length. This finding was evidence of its discriminant validity. Koizumi and In'nami (2012) reported additional supporting evidence for the validity of MTLD in their study. They found that MTLD was minimally related to length, especially in samples that were longer than 100 tokens.

Goals of the Study

D, Maas index, MTLD, and MATTR are four novel techniques that have been developed to assess the breadth of a speaker's vocabulary exhibited in a language sample. Though all of the techniques claim to measure LD, each takes a distinctly different computational approach on the basis of its own set of theoretical assumptions. Therefore, it is unclear whether these techniques measure the same construct and to what extent they produce valid and reliable scores. The current study explored how the scores of different techniques for estimating LD relate to each other. Further, the research questions of this study were cast in a modern psychometric framework and investigated using structural equation modeling (SEM). The specific aim of this study was to increase our understanding of the validity of the scores generated by different LD estimation techniques. The following research questions were addressed:

• Do all the techniques generate scores that are manifestations of the same latent variable (i.e., LD)?

• Is there a single latent variable determining performance for each estimation technique or are there specific method factors that jointly determine the scores?

• What are the magnitude and the nature of the relationships among the observed scores, the LD, and the method factors?

• Is there evidence that supports the use of a specific estimation technique?

Method

Participants

Four-hundred and forty-two adults met the following inclusion criteria for participation in the study: (a) no history of stroke, head injury, or neurogenic disorder according to self-report; (b) aided or unaided hearing acuity within normal limits; (c) normal or corrected visual acuity; (d) monolingual use of English; (e) normal cognitive functioning, as indicated by a score of 24 (out of 30) or higher on the Mini–Mental State Examination (Folstein, Folstein, & McHugh, 1975; Lopez, Charter, Mostafavi, Nibut, & Smith, 2005); and (f) no signs of depression, as indicated by a passing score (0–4) on the Geriatric Depression Scale (Brink et al., 1982). Participants' demographic information can be seen in Table 1.

Table 1.

Participants' demographic information.

Characteristic	Men (n = 186)		Women (n = 256)
	n	%	n	%
Ethnicity
African American	14	7.53%	20	7.81%
Hispanic	7	3.76%	7	2.73%
Other	7	3.76%	8	3.13%
White	158	84.95%	221	86.33%
Education level completed
Some high school	3	1.61%	0	0.00%
12th grade	25	13.44%	30	11.72%
Some college	52	27.96%	80	31.25%
Bachelor's degree or higher	106	56.99%	146	57.03%
Age
M (SD)	52.09 (20.19)		52.13 (18.60)
Range	20–89		20–87
MMSE score	54.38		54.72
GDS score	1.36		1.11

Note. MMSE = Mini–Mental State Examination (Folstein, Folstein, & McHugh, 1975); GDS = Geriatric Depression Scale (Brink et al., 1982).

Discourse Elicitation: Stimuli and Instructions

Tasks were designed to elicit four discourse types from participants. The first task was developed to elicit procedural discourse. The other three tasks were created to elicit three types of narrative discourse: eventcasts, storytelling, and recounts. Each discourse task was introduced with a warm-up task.

For the procedural discourse task, the examiner demonstrated the steps required to make a pot of coffee. Then, to obtain two procedural language samples, the examiner asked participants to provide the steps (a) to make a peanut butter and jelly sandwich and (b) to plant a flower in a garden.

For the eventcasts discourse task, the warm-up task included a brief narrative provided by the examiner describing the picnic scene from the Western Aphasia Battery–Revised (Kertesz, 2007). Next, the participants were asked to produce a story for the cookie-theft picture from the Boston Diagnostic Aphasia Examination–Third Edition (Goodglass, Kaplan, & Barresi, 2001). As part of the warm-up task, feedback was provided to avoid eliciting a simple description of objects, characters, or their physical characteristics. Then, to elicit the eventcast language sample, participants were presented with Nicholas and Brookshire's (1993) single and sequential pictures and were asked to produce a story that was based on temporal sequencing (“Take a minute to look at this picture; when you are ready, tell me a story that has a beginning, middle, and end”).

For the storytelling discourse task, the warm-up task involved the examiner providing an example of how to tell a story using a wordless picture book (The Great Ape; Krahn, 1978). To obtain two storytelling language samples, participants were given an unlimited amount of time to view and become familiar with the stories depicted in the wordless picture books Picnic (McCully, 1984) and Good Dog, Carl (Day, 1985). They were then asked to “tell a story that goes with the pictures” for each of the books.

For the recounts discourse task, the examiner initially modeled the task by providing a brief personal narrative about a trip to San Diego, California. To elicit the recounts language sample, participants were asked to recall and share three past experiences: what they did (a) last weekend, (b) during their last holiday, and (c) during their last vacation. If participants stopped after 15 s or more, they were prompted with “Can you tell me more?”

Transcription

Samples were digitally recorded and then orthographically transcribed in the computerized language analysis format (MacWhinney, 2000) by trained research assistants. Approximately 10% of the samples were randomly selected and transcribed again for reliability purposes. Intra- and interrater word-by-word transcription reliability were both above 85%. Nonwords, hesitations, revisions, repetitions, and onomatopoeia were coded via transcription codes in the format and were excluded from further analysis. Next, for each participant, four separate files were created that represented the four discourse types. Procedures included statements about how to make a peanut butter and jelly sandwich and how to plant a flower in a garden; eventcasts involved the four picture descriptions; stories included narratives about Picnic and Good Dog, Carl; and recounts comprised the participant's accounts of three previous experiences.

Estimating LD

For each of the four input files created for a participant, LD was estimated using each estimation technique. D was estimated using the voc-D program in the computerized language analysis format. Maas and MTLD were computed using Gramulator 5 (McCarthy, Watanabe, & Lamkin, 2012). Finally, MATTR was calculated using the computer software developed by Covington (2007). On the basis of the results of these programs, a data set was created that included 16 LD variables (4 discourse types × 4 estimation methods) for each individual.

Modeling Approach

We investigated the structure of the 16 variables using a multitrait–multimethod (MTMM; Campbell & Fiske, 1959; Eid & Diener, 2006) approach with SEM. The MTMM approach refers to the analysis of relationships among measures that are a function of multiple traits assessed using multiple methods (Geiser, 2008). Casting the MTMM approach within an SEM framework allows researchers to build SEM models that can decompose the variance that is due to traits, the variance that is due to methods, and the unique variance associated with each observed indicator. The MTMM matrices were analyzed using the correlated traits–correlated methods parameterization (Lance, Noble, & Scullen, 2002; Widaman, 1985).

In the context of our study, the multiple traits of the MTMM approach were represented by the four types of discourse, and the methods were represented by the four LD estimation methods. The discourse types were not viewed as traits in the traditional psychological sense, but were in recognition that the LD of language samples is likely to differ as a function of these types. In other words, variance due to discourse type is conceptually meaningful. On the other hand, for a particular discourse type, LD should not differ among samples as a function of estimation method. Thus, variance due to estimation method is viewed as problematic.

To answer the research questions of this study using SEM, a series of nested models was specified on an a priori basis. Each model was evaluated in terms of a variety of global fit indices. Overall, the model was evaluated using the robust χ² statistic (Muthén & Muthén, 2010), which allows for nonnormality of the data and for missing data. However, it is important to note that this χ² test is sensitive to sample size, and with large enough sizes it is likely to lead to rejection of the null hypothesis even when the model fits well (Kline, 2005).We also used χ² difference tests to evaluate differences between nested models. Three additional fit indices were computed to evaluate models: comparative fit index (Bentler, 1990), root-mean-square error of approximation (Steiger & Lind, 1980), and standardized root-mean-squared residual (Hu & Bentler, 1999). On the basis of published guidelines, good fit is indicated by a comparative fit index close to or higher than .95, a root-mean-square error of approximation value below .08 with the upper bound of the 95% confidence interval below .10, and a standardized root-mean-squared residual less than or close to .08, although cutoffs are somewhat model dependent (Brown, 2006; Hu & Bentler, 1999; Kline, 2005).

We used the following steps in our model evaluation process:

Step 1

The first model (Model 1) specified a single factor to underlie all 16 combinations of discourse types and LD estimation techniques, with no covariances among the residual scores. This factor could be interpreted as general LD characterizing speakers.

Step 2

The second model (Model 2) specified four factors differentiating the four types of discourses: procedural, eventcasts, storytelling, and recounts (i.e., LD_Pro, LD_Eve, LD_Sto, and LD_Rec, respectively). LD scores for these different discourse types were seen to be independent of each other. In other words, LD for one discourse type did not predict LD for other discourse types. For this model, there was an assumption that the results for the different methods converged (i.e., the orderings of the language samples were comparable across estimation methods for a particular discourse type).

Step 3

Model 3 was the same as Model 2 except the discourse type factors were allowed to be correlated. This model was probably more realistic in that individuals who have higher scores in one discourse type are likely to have higher scores in other discourse types. Again, comparable to Model 2, the assumption was that the various methods converged.

Step 4

Model 4 included not only the four correlated factors representing the discourse types (LD_Pro, LD_Eve, LD_Sto, and LD_Rec), as did Model 3, but also four additional factors representing the four estimation techniques or methods (i.e., M_D, M_Maas, M_MTLD, and M_MATTR). Accordingly, each variable was a function of one discourse type factor and one estimation method factor (as well as a residual source). The method factors were specified as uncorrelated. If Model 4 increased fit over and above Model 3, the inclusion of method factors improved fit, suggesting that the estimation methods produced divergent results for the various discourse types.

Step 5

The last step included an exploration of the relationship between the method factors. Specifically, it allowed for the method factors to be correlated to indicate that they were similar in nature.

Preliminary Analyses

Descriptive statistics for the number of types, tokens, and TTRs can be found in Table 2. Overall, these descriptive analyses are within expected ranges.

Table 2.

Descriptive statistics of the number of types, tokens, and type–token ratio (TTR) for each type of discourse.

Lexical diversity index	n	M	SD	Range	Skewness	Kurtosis
Procedures
Types	434	81.21	35.58	30–404	2.82	17.46
Tokens	434	192.43	109.33	53–1,224	3.25	21.12
TTR	434	0.45	0.08	0.25–0.66	0.3	0.01
Eventcasts
Types	441	185.96	54.55	74–507	1.17	3.26
Tokens	441	448.56	192.5	118–2,044	2.2	11.76
TTR	441	0.44	0.06	0.25–0.63	0.35	0.41
Storytelling
Types	441	310.18	80.83	115–639	0.87	1.19
Tokens	441	1,048.45	398.97	295–2,997	1.42	3.2
TTR	441	0.31	0.04	0.17–0.47	0.09	0.27
Recounts
Types	434	180.82	101.36	53–875	2.42	9.32
Tokens	434	472.4	432	85–3,775	3.73	18.74
TTR	434	0.44	0.09	0.2–0.71	0.02	0.01

The amount of missing data was minimal for the variables that corresponded to the 16 combinations of discourse types and LD estimation techniques. The percentage of missing data ranged from 0.23% to 2.71%, with the mean being 1.02%. Reasons for missing data included recording equipment failures, video format conversion difficulties, and tester oversight during task administration and data collection.

Data were screened for outliers univariately (scores > 3.3 SDs beyond the mean) and multivariately with Mahalanobis distance (ps < .001) using SPSS. No scores were identified as outliers multivariately. Univariately, 0.39% of the scores across all variables were identified as outliers and were removed from the data set. The language sample of each outlier was inspected, and for the majority of individuals, outliers were due to participants not following task instructions. For example, some participants produced a mixture of expository, narrative, and procedural discourse when asked to produce procedural discourse.

After the outliers were eliminated, descriptive statistics were computed, as shown in Table 3. The assumption of normality of the linguistic complexity measures was evaluated by examining skewness and kurtosis statistics in Table 3 and by visually inspecting graphs of them. Overall, none of the variables demonstrated extreme skewness or kurtosis. However, to be prudent, the models were estimated with Mplus 6.1 (Muthén & Muthén, 2010) using the MLR estimator, which estimates parameters using maximum likelihood with standard errors and a χ² test statistic that are robust to nonnormality. MLR accommodates missing data in the analyses under the assumption of missingness at random (cf. Enders, 2010).

Table 3.

Descriptive statistics of the untransformed major study variables after the removal of outliers.

Lexical diversity index	n	M	SD	Range	Skewness	Kurtosis
Procedures
D	430	37.03	11.70	14.82–75.41	0.71	0.29
Maas	434	60.91	10.05	37.05–89.34	0.16	−0.23
MTLD	432	33.15	8.75	18.27–62.94	0.86	0.50
MATTR	434	168.63	13.51	133–202.75	−0.03	−0.29
Eventcasts
D	439	61.91	11.71	31.5–100.34	0.5	0.33
Maas	440	106.70	12.40	68.04–141.49	−0.25	0.15
MTLD	438	50.30	10.72	25.45–86.17	0.64	0.40
MATTR	441	300.02	11.78	264–331.2	−0.09	0.08
Storytelling
D	439	57.53	13.36	22.78–101.63	0.77	0.62
Maas	437	133.72	11.06	101.3–161.12	0.11	−0.36
MTLD	437	42.76	9.72	21.95–75.55	0.68	0.56
MATTR	441	292.85	13.95	252.4–330.8	−0.10	0.01
Recounts
D	433	64.40	14.54	28.44–112.15	0.30	0.07
Maas	430	105.25	16.60	56.06–155.77	0.04	−0.09
MTLD	430	53.14	12.30	21.09–98.98	0.68	0.61
MATTR	437	306.92	13.00	265.2–344.4	−0.03	0.04

Note.  MTLD = measure of textual lexical diversity; MATTR = moving-average type–token ratio.

Results

Main SEM Analyses

A series of hypotheses using nested model comparisons were tested, followed by a substantive evaluation of model parameters if the model fit adequately. To perform the nested model comparisons, the scaled difference χ² test statistic (Muthén & Muthén, 2010) was used. A summary of the fit statistics for the various fitted models can be seen in Table 4.

Table 4.

Goodness-of-fit indices for structural equation models.

Model	Model description	χ2	df	CFI	RMSEA (90% CI)	SRMR
Model 1	Single latent variable	3,888.62*	104	.4	.29 (.28–.29)	.21
Model 2	Four uncorrelated content factors	1,112.63*	104	.84	.15 (.14–.16)	.26
Model 3	Four correlated content factors	757.53*	98	.89	.13 (.12–.14)	.08
Model 4a	Four correlated content factors; four uncorrelated method factors	302.63*	82	.97	.08 (.07–.09)	.07
Model 4a	Four correlated content factors; two uncorrelated method factors (MD, MMaas)	327.69*	90	.96	.08 (.07–.09)	.07
Model 5	Four correlated content factors; two correlated method factors (MD, MMaas)	274.86*	89	.97	.07 (.06–.08)	.07
Model 3′	Four correlated content factors; truncated language samples	300.56*	98	.98	.07 (.06–.08)	.03
Model 5b′a	Four correlated content factors; two uncorrelated method factors (MD, MMaas); truncated language samples	282.69*	89	.98	.07 (.06–.08)	.03

Note. CFI = comparative fit index; RMSEA = root-mean-square error of approximation; CI = confidence interval; SRMR = standardized root-mean-squared residual; M = Method.

^aModel converged to an inadmissible solution.

^*p < .001.

Steps 1 and 2

The parsimonious one-factor model demonstrated very poor fit to the data and failed to satisfy any of the published criteria for acceptable fit discussed earlier. Model 2, which specified four uncorrelated discourse-type factors, resulted in a large increase in model fit compared to Model 1. However, despite the considerable improvement, Model 2 also did not demonstrate acceptable fit to the data.

Step 3

Model 3 was comparable to Model 2 but specified freely estimated covariances among the discourse type factors. Model 3 resulted in further improvement of model fit compared to Model 2 on the basis of the indices reported in Table 4. Further, using the scaled difference χ² test statistic, Model 3 was found to fit the data significantly better than Model 2, Δχ²(6) = 370.29, p < .001. However, despite the improvement in overall fit, Model 3 failed to demonstrate acceptable fit to the data.

Step 4

Model 4 included not only correlated discourse type factors but also orthogonal estimation method factors. In addition, the discourse type factors were specified to be uncorrelated with the estimation method factors. Although the global fit indices generally indicated relatively adequate fit for Model 4, two parameters yielded out-of-bound estimates (i.e., Heywood cases). Specifically, the unstandardized residual variance of the observed variable corresponding to MATTR scores for storytelling was negative (−.418), and the standardized loading on this measure with the MATTR method factor, which should be a correlation coefficient, was estimated to be larger than 1 (1.16).

Further inspection of estimates for the solution suggested that the MATTR method factor was ill defined, as evidenced by the very low standardized loadings with the remaining MATTR estimated variables (i.e., all loadings were less than .01 in absolute value). Also, the standardized loadings for the MTLD method factor were very low (i.e., all standardized loadings were less than .10 in absolute value).

Given that these two method factors appeared to be poorly defined with Model 4, a new model (Model 4a) was specified with only two orthogonal method factors: D and Maas. Model 4a is consistent with the hypothesis that only the LD variables estimated using the D and Maas indices were jointly determined by both content and method factors. The model converged to a proper solution and yielded good fit (see Table 4). It should be noted that the standardized parameter values for this model were similar in value for those of Model 4.

Step 5

The estimation method factors were specified to be uncorrelated in Step 4 to avoid improper solutions (Kenny & Kashy, 1992). In this step, Model 5 was specified that allowed the D and Maas method factors (i.e., M_D and M_Maas) to be correlated. The model converged to a solution, yielded no out-of-bound estimates, and demonstrated good fit to the data. Models 4a and 5 were compared using a χ² test statistic to explore whether freeing the covariance between the two method factors resulted in improved model fit. Allowing for a correlation between the two method factors did show improved fit, Δχ²(1) = 55.49, p < .001. The path diagram for Model 5 is presented in Figure 1, and the variances of the LD measures can be seen in Table 5. In Figure 1, consistent with SEM conventions for graphical model representation, squares indicate observed variables (e.g., D from the procedural discourse language samples). Circles represent unobserved latent variables that are estimated as part of the model (e.g., LD_Pro that represents an error- and bias-free estimate of the LD of a procedural sample). In this model, straight arrows are standardized factor loadings and represent the effect of latent variables on observed variables. The curved double-headed arrows between latent variables represent the correlations between these variables. If latent variables are not connected by double-headed arrows, they are uncorrelated. Because any observed variable is affected only by uncorrelated latent variables, the standardized factor loadings are correlations between the observed and latent variables.

Figure 1.

Model 5. Lexical diversity (LD) factors are based on procedures, eventcasts, stories, and recounts. Method factors are based on D and Maas. Pro = Procedural; Eve = Eventcasts; Sto = Storytelling; Rec = Recounts; MTLD = measure of textual lexical diversity; MATTR = moving-average type–token ratio; M = Method.

Table 5.

Standardized factor loadings and residual variances for Model 5.

Lexical diversity index	Factor loadings						Residual variance
	LDPro	LDEve	LDSto	LDRec	MD	MMaas
Procedures
D	.91				.19		.14
Maas	−.75					.27	.37
MTLD	.92						.16
MATTR	.97						.06
Eventcasts
D		.84			.38		.16
Maas		−.51				.62	.35
MTLD		.94					.12
MATTR		.94					.12
Storytelling
D			.86		.38		.12
Maas			−.60			.57	.32
MTLD			.98				.05
MATTR			.98				.04
Recounts
D				.73	.31		.37
Maas				−.46		.36	.66
MTLD				.95			.10
MATTR				.96			.08

Note. For all factor loadings, p < .001. Pro = Procedural; Eve = Eventcasts; Sto = Storytelling; Rec = Recounts; M = Method; MTLD = measure of textual lexical diversity; MATTR = moving-average type–token ratio.

As shown in Figure 1 and Table 5, a number of interesting patterns emerged across the LD measures. The factor loadings for the discourse-type factors (i.e., LD_Pro, LD_Eve, LD_Sto, and LD_Rec) generally were large in magnitude (Mdn = .91; range of absolute values = .46–.98). In comparison, the loadings for the two estimation method factors were relatively small but nevertheless substantial. For the D factor (M_D), the loadings ranged from .19 to .38 in absolute value, with a median value of .35. For the Maas factor (M_Maas), the loadings tended to be larger, ranging in absolute value from .27 to .62, with a median value of .45. It is interesting to note that the factor loadings for the discourse type factors were highest for the storytelling language samples when they were evaluated using MTLD and MATTR, but they were highest for the procedural language samples when they were evaluated using D and Maas.

In Table 6 we present the decomposition of the variances associated with all combinations of discourse types and LD estimation techniques into the proportions attributed to the various factors and the residual source. Provocatively, the LD measures assessed using D and Maas had not only a portion of their variances account for by method factors but also greater proportions of their variances unaccounted for by all factors in the model relative to other measures.

Table 6.

Proportion of variance for each factor and the residuals for Model 5.

Lexical diversity index	Variance due to each factor						Residual variance
	LDPro	LDEve	LDSto	LDRec	MD	MMaas
Procedures
D	.83				.04		.14
Maas	.56					.07	.37
MTLD	.85						.16
MATTR	.94						.06
Eventcasts
D		.71			.14		.16
Maas		.26				.38	.35
MTLD		.88					.12
MATTR		.88					.12
Storytelling
D			.74		.14		.12
Maas			.36			.32	.32
MTLD			.96				.05
MATTR			.96				.04
Recounts
D				.53	.10		.37
Maas				.21		.13	.66
MTLD				.90			.10
MATTR				.92			.08

Note. For each combination of type of discourse and estimation technique, the sum of the variance attributed to different sources sums up to 1 (within rounding error). Pro = Procedural; Eve = Eventcasts; Sto = Storytelling; Rec = Recounts; M = Method; MTLD = measure of textual lexical diversity; MATTR = moving-average type–token ratio.

Follow-Up Analyses

A series of follow-up analyses was performed to investigate the nature of the method factors that were found previously. In the previous analyses, complete language samples were analyzed that varied naturally in length. On the basis of prior literature, it was hypothesized that these method factors were related to length effects. To assess whether different lengths of text produce different outcomes on measures of lexical complexity, we manipulated the language samples to equate them in terms of length. If the method factors in the previous analyses were due to length, then experimentally removing the length variation should eliminate the need to include method factors in the models.

The language samples were truncated to 75 words, which was the length of the shortest language sample in the data set. By truncating to 75 words, we were able to retain the language samples for all 442 participants. Two models were fitted to the data for the truncated samples: Model 3′, which specified four correlated discourse type factors but no method factors, and Model 5b′, which specified two correlated method factors in addition to four correlated discourse type factors. These models were identical to Models 3 and 5, but they were estimated using the truncated data set.

The fit for Models 3′ and 5b′ are presented at the bottom of Table 4. Model 3′ yielded very good fit without specifying additional method factors. On the other hand, Model 5b′ had similar fit but yielded an inappropriate solution (i.e., Heywood cases). In other words, when language samples were equated in terms of length, method factors were no longer necessary to model the data. Overall, these results, in combination with the previous findings, suggest that the method factors were required in the analyses for the language samples that varied in length because D and Maas were sensitive to sample length.

Discussion

The goal of this study was to investigate the validity of four computational tools for quantifying LD. We examined a corpus of four types of discourse (procedures, eventcasts, storytelling, recounts) from 442 adults. Language samples varied with respect to their length. Using four techniques (D, Maas, MTLD, and MATTR), LD scores were estimated for each discourse type. Subsequently, data were modeled using SEM to uncover their latent structure. The modeling yielded several important findings.

LD Indices

D and Maas

Regarding D, the parameters of Model 5 suggest that the validity of its score interpretations with respect to LD was not as strong as for MTLD and MATTR. First, the average loading of the content factors was .83 for D-based scores across discourse types. In addition, the average proportion of variance attributed to the content factors was approximately 70%. Even though these estimates are high and suggest a strong relationship with appropriate factors, they are considerably lower than the respective parameter values of MTLD (i.e., 90%) and MATTR (i.e., 93%). Further, when language samples were measured using D, scores reflected the effects of a D-specific method factor in addition to the content factors. Even though on average the proportion of variance that was accounted for by this method factor (11%) was not excessive, it constitutes a second dimension along which D scores varied systematically.

The modeling in this study provides a coherent explanation for the contradictory conclusions reached by McCarthy and Jarvis (2010) regarding D, Maas, and MTLD. In their article, they reported high correlations among the three tasks, which they interpreted as evidence of convergent validity. However, they also reported a discriminant analysis in which scores generated by these three techniques were used to differentiate among the types of discourse. The results of this analysis suggested that D, Maas, and MTLD each contributed unique information to the prediction model. On the basis of these results, the authors concluded that “at least three of the sophisticated LD indices used in this study do not appear to assess exactly the same latent trait” (pp. 390–391). From Model 5, these results could be explained on the basis of the different sources of covariance in the data. Observed LD scores on the basis of D and Maas are influenced by secondary method factors that “enrich” their scores and allow them to contribute additional explanatory information in the prediction model of the discriminant analysis.

The interpretation of a Maas score as a clear indication of the LD may not be warranted. Maas demonstrated the lowest average loading across discourse types (−.56), and it was influenced heavily by a method factor. Maas loadings were estimated to be negative because unlike the other three indices, for which higher scores indicate higher levels of the latent trait, higher Maas scores are associated with lower levels of the latent trait because of how Maas is estimated. This inverse relationship is reflected in the negative sign. The average proportion of variance that was attributed to the method factor was approximately 22%, more than twice as large as for D. In fact, when Maas scores were derived for eventcasts, the influence of the method factor was stronger than the influence of the content factor. Given that Maas scores were so strongly influenced by a variable other than the construct of interest, they should not be interpreted as valid LD indicators.

MTLD and MATTR

Variables associated with MTLD and MATTR were found to be very strong indicators of the LD of a language sample in absolute terms, and stronger in comparison with D and Maas. Content factors (i.e., discourse type factors) accounted for, on average, 91% of the variances of the variables associated with these techniques across all discourse types. Importantly, MATTR and MTLD differed in another way from the other two indices in the study: Scores that were generated using MATTR and MTLD were influenced only by the content factors and did not demonstrate systematic effects from construct-irrelevant sources. Therefore, holding all else constant, MTLD and MATTR provide a more accurate reflection of the LD for the samples of the discourse type elicited in this study.

The highest average proportion of variance shared between measures and content factors across discourse types was associated with MATTR-generated variables. Content factors accounted for, on average, 93% of the variances of these variables across all discourse types. This finding highlights the potential of this new measure to produce estimates that are valid and reliable indicators of the LD of language samples. Further, this finding is important because, with the exception of Fergadiotis et al. (2013), currently there is very limited research on this measure.

Regarding MTLD, our findings lend support for its use and add to the results of previous studies reported in the literature. MTLD has been found to correlate strongly with a number of LD indices, including D and Maas, leading researchers to argue in favor of its validity (e.g., McCarthy, 2005). In our study, variables that were estimated using D, Maas, and MTLD strongly loaded factors that represented discourse-specific LD (Model 5). This model may provide a more accurate account of why variables that are generated on the basis of different estimated techniques relate to each other and can be used to explain the high correlations that have been observed across other studies in the literature.

The Nature of the Method Factors

The results from the follow-up analyses provide evidence that both method factors were associated directly or indirectly with length. In the first set of analyses, language samples varied with respect to length. Subsequently, using truncated language samples, a comparison of Models 3, 3′, 5b, and 5b′ revealed that when the observed scores were based on language samples that varied in length, the inclusion of method factors was necessary to achieve adequate model fit. However, when data were truncated, the method factors were no longer required to obtain acceptable fit. This pattern of results suggests that relating the method factors to length was justified.

With respect to D, the nature of the method factor might not be directly related to length but rather to the number of topics or themes a speaker produces. Owen and Leonard (2002) found that D scores were higher for 500-word samples than for 250-word samples. These results were similar both for children who were developing typically and for children with language impairment. Further analysis showed that the distributions of D scores (for each length size) overlapped by approximately 70%. This finding was attributed to the possibility of new themes appearing in the second half of the samples that had not appeared in the first half. Owen and Leonard argued that the introduction of new themes “would, by necessity, introduce new content words, altering the D scores of the children towards a slightly higher score” (p. 936).

If this was indeed the case, it could also explain why the correlation between the Maas and D method factors in our study was moderately high (i.e., .50). If D and Maas were directly related to length, one might have expected a higher correlation between these method factors. However, if what increases D estimates were only indirectly related to length and were mediated by the number of themes, elaborations, or episodes, it would be expected that the two method factors would still be correlated but not as strongly.

Even though it is not clear how length relates to D, the results from Owen and Leonard's (2002) study are consistent with the configuration and the parameter estimates of Model 5. For example, if the loadings from discourse-specific LD factors to D-estimated variables were equal to 1, this would suggest that D reflected purely the LD of the language samples and nothing else. In that hypothetical scenario, one would expect the two distributions from Owen and Leonard's study to overlap completely, because manipulating other factors (such as length) would have no direct effects on the scores. However, results of the current study indicate that the loadings were not perfect, and a second factor, which could be associated directly or indirectly with length, was found to influence D scores in Model 5. Therefore, relating these findings back to Owen and Leonard's results, scores for the 500-word language samples would be expected to differ from those of language samples of 250 words. Also, whereas TTR would be expected to drop systematically as a function of sample size, D would be expected to increase for both groups when longer samples were analyzed.

With respect to Maas, similar findings have been reported in the literature. For example, Tweedie and Baayen (1998) used Monte Carlo simulation and demonstrated that a notational variant of the Maas index was monotonically influenced by length. Cossette (1994) reached similar conclusions and argued that Maas was not length invariant.

However, our findings are not consistent with the conclusions reached by McCarthy and Jarvis (2007), who argued that influence of length on Maas was minimal. In their study, they used data from 23 previously published corpora; 16 were written corpora, and seven were spoken. To examine the effects of length, they partitioned the texts to assess whether the smaller parts could project the score of the whole text when reconstituted. The main idea was that if a text is divided into two parts, the average LD of those two parts should approximate the LD of the original text. Through partitioning of the text into smaller parts, the trend of the average LD scores would reveal whether there was a relationship between the LD measurement and text size. McCarthy and Jarvis argued that “the more the mean LD score of the section sizes correlates with the mean token size of the section size, the less the LD measure is able to satisfactorily project” (p. 478). Using this approach, they concluded that all LD measures were a function of length, but Maas shared only 2% of variance with the mean token size across all corpora.

One possible explanation for the strong method effects in our data set is that Maas may correlate with length much more in spoken tasks than written tasks. McCarthy and Jarvis (2007) reported the percentage of variance Maas shared with the mean token size across all corpora. However, an examination of their Table 7 (p. 480) indicates that the correlations between Maas and mean token size for written and spoken corpora were .12 and .32, respectively. The squared average correlation across all corpora would indeed suggest that Maas and mean token size share only 2% of variance. However, the squared correlations of Maas scores and the mean token size for written and spoken discourse tasks separately would be equal to 1% and 10%, respectively. This would suggest that one would expect a much stronger length effect in data that are based on spoken discourse.

Clinical and Research Implications

There are several situations in which the technique one uses to estimate LD might play a significant role in the interpretation of the LD scores. For example, it is often the case in the field of speech-language pathology that within clinical populations there are subpopulations that demonstrate unique clusters of symptoms. For example, individuals with Broca's and Wernicke's aphasia differ characteristically in terms of the volume of their verbal output. Comparisons across these two groups could be quite misleading, depending on the estimation technique used. For example, given that individuals with Wernicke's aphasia produce longer language samples, D would probably overestimate their LD. That, in turn, could result in artificially inflating the possibility of detecting differences between the language samples of the two groups. In contrast, using Maas could potentially lead to reaching the opposite conclusion; that is, given that people with Wernicke's aphasia produce longer samples, their Maas scores would suggest lower LD because of the length effects. Subsequently, the use of Maas scores could minimize or mask any differences between the language samples of the two groups.

LD has also been used as an indicator of lexical retrieval improvement during discourse production before and after treatment. Again, the decision to use a problematic technique to estimate the LD of a sample could lead to erroneous conclusions due to length effects. Longer samples could be produced as a result of testers establishing rapport with patients or patients getting familiar with the task demands. Therefore, it is possible that clinicians could be making judgments about the efficacy of a treatment by misinterpreting length effects as changes in LD.

From the four techniques examined in this study, D has been employed most often in the field of speech-language pathology. Several studies have reported results and reached conclusions on the basis of D scores. The interpretation of these findings has relied on the assumption that D is not a function of length and therefore that language samples of different size could be compared meaningfully. However, the findings of the current study suggest that when a language sample is evaluated using D, scores cannot be interpreted unequivocally as reflections of the language sample's LD without taking into account the sample's length and the possibility that the production of new themes might be contributing to score differences.

Given that the estimation of LD can be contaminated by sample length, it would be very helpful to publish language sample characteristics, such as mean and range of types and tokens, when reporting results. Sharing this kind of information would allow readers an opportunity to critically evaluate the findings of the study. This is the case even if authors decide to estimate LD using one of the methods that were found to be relatively free of systematic length effects in this article. The first reason is that techniques such as MTLD and MATTR might perform differently in different types of discourse that have not been studied yet. Further, it is possible that what might be considered an unbiased technique today might be proven biased in the future. The long history of LD indices that claimed to be free of length effects, only to be found problematic later, makes a strong argument for this conclusion.

Overall, applying MTLD and MATTR to the language samples of 442 individuals provided evidence suggesting that these techniques generated scores that were strong indicators of the LD of language samples. However, an advantage of MTLD over MATTR is that, at this point, the former is actively researched and more evidence has been accumulated that favors its validity. By contrast, very little is known about MATTR, and future studies should be designed to explore its performance under different conditions. An advantage of MATTR is that it is equivalent to TTR and thus fairly straightforward to grasp and explain. It does not require an understanding of frequency distributions, curve fitting, or the nature of stochastic processes in order to convey its meaning. Its simplicity increases the face validity of the technique (i.e., its potential to measure what it is supposed to measure at face value). Face validity is a very desirable property, especially for professionals who work with adults or children with speech and language disorders in clinical settings (e.g., Gordon, 2008; Lind et al., 2009). If the involved parties believe that an index is nonsensical, absurd, or a waste of time, they will be reluctant to use it or trust it. However, if the meaning of the scores is easily conveyed, the index enables more meaningful communication between clinicians, patients, and their families.

Funding Statement

This research was partially supported by National Institute on Aging Grant R01AG029476. We are especially grateful to the study participants.