Average One Year Change in Lexical Measures of Written Narratives for School Age Students

Today’s society places particularly high importance on written language, with writing skills among the top educational priorities for students (Fallon & Katz, 2011; Nippold, Ward-Lonergan, & Fanning, 2005). Evidence of writing as an educational priority is apparent in the universal standards which emphasize writing skills in the expectations for each grade level (National Governors Association Center for Better Practices and Council of Chief State of School Officers, 2010a). As such, written language skills have a strong influence on ratings of academic achievement and performance. Further, high stakes testing often requires that students convey ideas and information clearly through written text (e.g. Common Core State Standard, ELA-Literacy.W.5.2). Among expected competencies, students are expected to link ideas within and across categories of information and use precise language to inform or explain a topic (e.g. Common Core State Standard W.5.2.b and c, and d).

Despite these expectations, research findings suggest that writing is among vulnerable aspects of language and literacy achievement (Katusic, Colligan, Weaver, & Barbaresi, 2009; Mayes & Calhoun, 2007). Almost 75% of 8^th and 12^th grade students struggle to achieve proficiency in writing (National Center for Education Statistics, 2012), evidenced by the fact that only one quarter of students in 8^th and 12^th grade write at a proficient level (NCES, 2012). Similarly, only 28% of fourth grade students in the United States meet the expectation for proficiency in writing (Persky, Daane, & Jin, 2003). These trends reiterate the need for additional research on writing to better understand the components and development of writing across grades (Graham, Gillespie, & McKeown, 2013).

Language Components of Writing

One contributing factor to students’ writing performance is oral language, recognizing that generating and conveying ideas or a message in writing relies heavily on a student’s underlying language skills (Juel et al., 1986). A positive correlation between oral language skills and writing performance is well recognized (Beringer, & Abbott, 2010; Berninger, Abbott, Abbott, Graham, & Richards, 2002; Gillam & Johnston, 1992; Kim, Al Otaiba, Silder & Greulich, 2013; Macki, Dockrell & Lindsay, 2013; Olinghouse & Leaird, 2009; Shananhan, 2006). From this perspective, the process of writing is sometimes viewed as the integration of numerous interconnected language skills (e.g. spelling, vocabulary, grammar). Although oral language is a recognized contributor to writing quality, the precise nature of the relationship, between aspects of oral language and dimensions of writing, remains unclear. Researchers have nominated numerous component skills that are related to overall writing quality, including handwriting fluency, spelling, reading, and oral language (Kent & Wanzek, 2016).

Among influential oral language components of writing quality, there is general agreement that oral vocabulary skills contribute to an essential component of writing performance (Kent & Wansik, 2016). In a recent meta-analysis of writing quality outcomes, vocabulary or word choice was the most frequently occurring component that was evaluated (Kent & Wanzek, 2016). In previous studies, writing productivity demonstrated positive relationships to writing quality for children in the elementary grades (e.g., Abbott & Berninger, 1993; Wagner et al., 2011). Of specific interest, diversity of vocabulary in writing (number of different words or NDW) and productivity (total number of words or TNW), have been shown to demonstrate developmental differences by grade (Wood, Schatschneider & Hart, 2017).

Additionally, investigators have reported a significant relationship between lexical diversity (NDW) and students’ performance on standardized measures of vocabulary knowledge and reading achievement (Berninger et al., 2002; Miller, Andriacchi, & Nockerts, 2015; Wood, Bustamante, Schatschneider, & Hart, 2018). In a cross-sectional study (Wood et al., 2017), researchers described written personal narratives of 907 children in first through eighth grade in terms of lexical diversity, productivity, and accuracy (grammatical, spelling, and conventions) by grade. Children’s performance on written language sample measures was significantly correlated to performance on standardized language and literacy assessments. Further, in a subsequent study of students in first through fifth grade (Wood et al., 2018), lexical diversity in written language sample measures was significantly correlated to students’ performance on assessments of reading vocabulary (r = .50, p <.0001) as measured by the Gates-MacGinitie Reading Test (GMRT-4, MacGinitie & MacGinitie, 2006). In contrast, other studies, have reported relationships between reading and writing in grades first to fourth grade (e.g. Ahmed et al., 2014) but not for students above fourth grade (Kent & Wanzek, 2016).

Given the potentially positive relationship between students’ writing and performance on standardized measures of language and literacy, written narrative tasks hold promise for use as a progress monitoring tool; however, additional studies are needed of expected school year growth, or average change in a one-year period. There are numerous cross-sectional studies substantiating the utility of lexical diversity and grade level expectations (Malvern, Richards, Chipere, & Duran, 2004, Puranik et al., 2008); however, most existing research on grade level lexical diversity is cross-sectional rather than longitudinal (e.g. Wood et al., 2017).

Although there is a basic assumption that lexical measures in written narratives are generally sensitive to developmental progression (Berman & Verhoeven, 2002), there have been mixed findings in the literature regarding the typical growth trajectory over time on specific microstructural measures. A handful of studies have examined average change in written language with a focus on measures of lexical diversity (e.g., number of different words or NDW) (e.g. Fey, Catts, Proctor-Williams, Tomblin, & Zhang, 2004), grammatical complexity (e.g. Brimo & Hall-Mills, 2018) and language in writing (e.g. Abbott, Berninger & Fayol, 2010).

Previous findings support the expectation of growth or significant change in writing productivity (total number of words) in writing samples across the school year (Dockrell, Connelly, Walter, & Critten, 2015; Malecki & Jewell, 2003). In one such study (Malecki & Jewell, 2003), investigators examined writing productivity on a 3 minute writing task for 946 students in first through eighth grade with writing measures at two time points (fall and spring). The authors reported that older students outperformed younger students on all scoring indices. Students demonstrated significant increases in writing productivity and improvements on accuracy indices from fall to spring time points, supporting the utility of such writing measures. Similarly, in a more recent study by Dockrell et al (2015), investigators examined written texts of 192 students who were in 3^rd, 4^th, and 5^th grade. The authors reported significant effects of time across a 5 month period for total words produced in students’ expository and narrative writing samples.

Findings of a several studies demonstrate children’s progression in number of different words (NDW), also referred to as lexical diversity, across development (Fey et al., 2004; Wagner et al., 2011). In one such longitudinal study (Fey et al., 2004), children made gains in lexical diversity from 2^nd to 4^th grade, measured by number of different words (NDW). Very few studies have examined average change in lexical diversity for students in later grades, beyond fifth grade. The general paucity of studies of written language that consider the developmental spectrum of lexical measures in written language samples supports the need for additional research on the impact of various components of writing across the developmental spectrum (Graham et al., 2012).

Purpose/Aims

The current study aimed to examine longitudinal growth in typically developing school age children on a written narrative task. Specific research questions included:

1. What is the average change in children’s written language performance in one year? Specifically, does lexical diversity and productivity show significant mean change across school years and if so, for which grades?

2. Does accuracy of written language change across school years? Specifically, do students decrease in the number of grammar and spelling errors across grades?

3. Is there a significant relationship between students’ lexical diversity in written samples and their performance on standardized measures of reading achievement?

Method

Participants

To examine children’s performance language use in written personal narrative language samples, the investigators used existent data from the Florida Twin Project on Reading, Behavior and Environment, a cross-sequential twin study focusing on correlates of reading development (Taylor, Hart, Mikolajewski & Schatschneider, 2013). The available database included written language samples from 1,023 school age children 1^st – 9^th grade. Because there were only two students in ninth grade, they were considered outliers and removed. Among the available pool of participants, teachers had identified 116 children with diagnosed disorders using the state’s categorization of exceptionalities (speech impaired, language impaired, deaf or hard of hearing, visually impaired, specific learning disabled, profoundly mentally handicapped, dual-sensory impaired, autistic, developmentally delayed, and other health impaired) which constituted approximately 11% of the pool. Children with exceptionalities were then excluded from the database of interest, because it was not feasible to confirm correct diagnosis or acquire additional information about severity or services. There were 907 children without identified disorders in elementary and middle school (1^st through 8^th grade) in the database. Of these 907 children, the final database included 749 students who participated in a second time point one year later.

The final dataset of 749 students was comprised of 420 girls and 329 boys. Table 1 provides the specific number of students per grade which ranged from 52 to 146 students in Grade 1 through Grade 8 (subsequently in year 2 students were in 2^nd – 9^th grade). Participants’ race and ethnicity demographics included: 55% White, 23% Hispanic, 15% Black or African American, 4% Mixed (two or more races) and 3% Asian. The racial distribution of the sample was similar to the proportions reported by the Florida Department of Education (FDOE, 2018–2019) for children in Florida schools (37% White; 34% Hispanic; 22% Black; 4% Mixed; 3% Asian). Spanish was spoken in the home for a few of the participating households; although, students with limited English proficiency were not recruited for participation students and all academic instruction in English-only and. Parents reported their household income with the largest percentages of income falling within three categories: 37% reported income between $10,000 and $49,000; 38% between $50,000 and $109,000; and 25% between $110,000 and $209,000 or greater. Parents reported their highest level of education to be: 4.6% without high school graduation; high school (12.1%), some college (19.1%), associate’s degree (12%), bachelor’s (23%), attended graduate school without obtaining masters (3%), and completed graduate school (25.5%).

Table 1.

Participants’ Performance on Standardized Language and Literacy Measures

Assessment	Description	N	Mean	SD
FAIR Word Analysis	Measures spelling abilities	209*	98.35	14.78
FAIR RC	Measures reading comprehension	221*	103.05	14.56
FAIR Maze	Measures reading fluency	218*	109.69	14.58
FCAT Reading	Assesses reading achievement	625	237.95	25.09
GMRT	Measures reading comprehension	739	538.14	47.57

Note.

^*For FAIR subtests the number is less than the total sample size because only the age appropriate subtests were administered by grade. FAIR = Florida Assessments for Instruction in Reading Aligned to the Language Arts Florida Standards (FAIR-FS; Foorman, Torgesen, Crawford, & Petscher, 2009). FAIR RC = reading comprehension; FCAT = Florida Comprehensive Assessment Test (FCAT) developmental scaled score in Reading (administered to participants in 3^rd-9^th grade) (Florida Department of Education, 2014); GMRT =Gates-MacGinitie Reading Tests (4^th ed.) (GMRT) (MacGinitie & MacGinitie, 2006).

Measures

Writing task.

A personal narrative task was used to elicit samples of students’ writing in English. The typed prompt, One day when I got home from school…was provided at the top of a double-sided lined piece of paper, which was similar to written prompts used in other studies (Connelly, Dockrell, Walter, & Critten, 2012; Dockrell, Ricketts, Charman, & Lindsay, 2014; McMaster & Espin, 2007). The writing task was not constrained by time or length. The instructions informed parents that the activity was expected to take 10–15 minutes and students could continue up to 15 minutes.

Lexical Diversity and Productivity of Writing Samples.

Measures of vocabulary, such as lexical productivity and diversity or the “range of vocabulary in a language sample” (Malvern, Richards, Chipere, & Durán, 2004, p.16) have been widely used as indicators of compositional productivity in writing (e.g., Abbott & Berninger, 1993; Berman & Verhoevan, 2002; Mackie & Dockrell, 2004; Puranik et al., 2008; Scott & Windsor, 2000; Wagner et al., 2011). To measure lexical diversity and productivity, researchers used Systematic Analysis of Language Transcripts (SALT) to calculate the number of different words (NDW), total number of words (TNW), and the ratio of NDW to TNW, also referred to as type token ratio (TTR). A measure of total words has been used as a standard measure of fluency and productivity in curriculum based writing measures for decades (Martson, 1989) with reported correlations with the Test of Oral and Written Language as high as .84 (e.g. Deno, Marston, & Mirkin, 1982). Other studies have reported weaker relationships depending on the assessment measure compared (e.g. Gansle, Noell, VanDerHeyden, Naquin, & Slider, 2002). To date, NDW and TNW are among the most commonly used metrics of vocabulary in writing samples (Danzak, 2011; Hall-Mills & Apel, 2013; Koutsoftas & Gray, 2012; Price & Jackson, 2015); however TTR was also included to take into account the diversity in relation to the length in an effort to minimize potential limitations of NDW (Duran, Malvern, Richards & Chipere, 2004).

Accuracy of writing sample.

In addition to lexical measures, the existing literature supports the importance of including measures of accuracy (Dockrell et al., 2015; Malecki & Jewell, 2003; Scott & Windsor, 2000). In response, investigators calculated the rate and proportion of errors within two broad categories of error types, including errors of spelling and errors of grammar. The decision for the inclusion of a measure of spelling accuracy was based on the wide recognition of spelling as an essential component of writing (Berninger, Abbott, Nagy, & Carlisle, 2010; Devonshire & Fluck, 2010). Previous studies have reported sensitivity of spelling accuracy to developmental change in students’ narrative texts across 5-month periods (Dockrell et al., 2015). Additionally, spelling accuracy has been reported to predict text composition in students in 1^st grade through 7^th grade (Abbot, Berninger, & Fayol, 2010) and differentiate children with specific language impairment (Broc, Bernicot, Olive, Favart, Reilly, Quémart, & Uzé, 2013).

Similarly, the decision to include a measure of grammatical accuracy was based on a number of previous findings suggesting that measures of correct writing sequences were sensitive to student progress over time (Dockrell et al., 2015; Malecki & Jewwell, 2003). Further, grammaticality, or proportion of grammatical errors, has been found to be sensitive to achievement differences and differentiates between children who are typically developing and children with language impairments (Eisenberg, & Guo, 2013; Scott & Windsor, 2000). In a study by Scott and Windsor (2000), the frequency of grammatical errors was the only measure that distinguished children with language learning disabilities from their peers.

Notably, there are various levels of coding that could be used for a finer description of types of errors, such as categorizing types of spelling errors (e.g. Bahr, Silliman, Berninger, & Dow, 2012; Quick & Erickson, 2018; Masterson & Apel, 2013) and types of grammar errors (Eisenberg & Guo, 2013); however, the specific types of errors were not within the focus of the current research questions. In response, a broad measure of accuracy based on the proportion of errors was utilized. Use of such an accuracy measure (proportion of errors) is consistent with other sources that have reported spelling errors as a proportion to total words (Broc et al., 2013; Dockrell et al., 2015) and grammaticality as the proportion of utterances with grammatical errors (Eisenberg & Guo, 2013).

General Descriptive Measures

In order to provide general descriptive information on the participating students’ language and literacy skills, performance on the Florida Assessments for Instruction in Reading (FAIR), a standardized measure of language and literacy, is provided in Table 1. A detailed description of each subtest follows in a subsequent section. Results suggested that students in the participant pool demonstrated performance in the average range (a standard score of 85–115) on the FAIR subtests. The average performance on standardized language and literacy measures provided further evidence that the participant pool was considered to be typical in their development.

Florida Assessments for Instruction in Reading.

Children’s oral language and literacy performance was assessed through statewide testing using the Florida Assessments for Instruction in Reading Aligned to the Language Arts Florida Standards (FAIR-FS; Foorman, Torgesen, Crawford, & Petscher, 2009). Students’ performance on the FAIR-FS was used to describe global language and literacy skills. The FAIR is a norm-referenced computer-adaptive screening and diagnostic battery of reading component skills. The FAIR has been scaled to measure reading skills in grades K-12. The FAIR was normed on students in eight counties of Florida. The sampling strategy for the norming of the tasks matched the state achievement and demographic distribution across all grade levels (Foorman et al., 2009). Task descriptions and psychometric information are provided by the authors (Foorman, Petscher, & Schatschneider, 2015) and latent profiles and their association with standardized reading outcomes have been derived from the normative sample (Foorman, Petscher, Stanley, & Truckenmiller, 2016). The FAIR subtests vary across grades. As such, subtests are described below according to grade level.

FAIR- Grades 1–2.

For students in 1^st grade and 2^nd grade, the vocabulary task and word reading and fluency subtests were included in the current study. The Vocabulary Pairs task is comprised of 24 items. Students are instructed to look at pictures on the screen as they are named (e.g. cow, orange, pig) and then asked to pick the two that go together. Internal consistency was reported to be .92 for word reading, and .89 for the vocabulary subtest. Based on the technical manual (Foorman et al., 2015), evidence for this task’s validity is provided by concurrent correlations with the Peabody Picture Vocabulary Task–4^th Edition (Dunn and Dunn, 2007) ranging from .47 to .67. Marginal reliability across grades 3–10 is .91, .89, and .90 for fall, winter, and spring performance.

The word reading and fluency subtest is a teacher-administered subtest of the FAIR, designed to task the student’s decoding, rate, accuracy, and expression. For this subtest, the students are asked to read a passage aloud to the teacher who scores accuracy after one minute of reading. Examples of errors include mispronunciations, omissions, substitutions, reversals, and hesitations longer than 3 seconds. Distractors represent orthographic patterns known to predict reading comprehension (Garcia & Cain, 2014). Concurrent correlations range from .30 to .46 with Torgesen, Wagner, and Rashotte’s (2012) Test of Word Reading Efficiency (Foorman, Koon, Petscher, Mitchell, & Truckenmiller, 2015).

FAIR- Grades 3–8.

For students in 3^rd grade through 8^th grade, the reading comprehension, word analysis (spelling), and reading efficiency and fluency (maze) subtests were utilized in the current project. The reading comprehension subtest is designed to take approximately 15 minutes, although it is untimed. During this subtest, the student is asked to silently read a passage of 200–1300 words and answer 7–9 multiple choice questions. Students are allowed to refer back to the passage when responding to questions. Concurrent validity (r = 0.67–0.74) with the Standford Achievement Test- 10^th Edition is reported in the FAIR technical manual (Foorman et al., 2015).

The word analysis subtest, which is untimed, is thought to measure phonological awareness and spelling. For this task, the computer provides an audio recorded word and a sentence using the word. Students are asked to respond by typing the word. Predictive validity is reported with the word reading and phonological awareness tasks predicting to the Stanford Achievement Test-10 (r= 0.40 – 0.60).

The Maze task was included to assess text reading efficiency. The subtest requires approximately 6 minutes. It is designed to task the student’s decoding, rate, accuracy, and expression. Marginal reliability across grades 3–10 is reported to be .86, .88, and .93 for fall, winter, and spring administrations (Foorman, Koon, Petscher, Mitchell, & Truckenmiller, 2015) and concurrent validity (r =0.30–0.46) was reported with the Test of Word Reading Efficiency (Torgesen, Wagner, and Rashotte, 2012).

Academic Achievement in Reading.

Administrative data was available for participating children from school records. In order to describe the reading achievement of participants and examine performance as it relates to change in lexical measures, we obtained participants’ performance on the Florida Comprehensive Assessment Test (FCAT). The FCAT was administered to students in 3^rd-10^th grade throughout the state (Florida Department of Education, 2014). The FCAT is designed to measure students’ achievement related to Sunshine State Standards in reading, math, and science. We obtained developmental scaled scores (DSS) for participants in 3^rd – 9^th grade with a reported range of 140–307. To interpret the scale of the score, we examined the established criteria for passing which was identified by the Department of Education for each grade level. The criteria considered to be passing varied from 198–255 depending on the grade (e.g 198–209 for Grade 3; 208–220 for Grade 4; 216–229 for Grade 5; and 222–236 for Grade 6). Table 1 displays the average performance of the participant pool.

Reading Vocabulary.

An additional test was administered to assess reading vocabulary, the Gates-MacGinitie Reading Test- Fourth Edition (GMRT-4) (MacGinitie & MacGinitie, 2006) which involved two separate subtests depending upon grade. For children in first and second grade, Word Decoding subtest (GMRT-4, MacGinitie & MacGinitie, 2006) was administered. The Word Decoding test is a measure of word reading and vocabulary knowledge, in which the child is shown a picture (e.g. practice item: pig) and a list of four similar written words (e.g. big, fig, pig, dig). The child must pick the correct word that corresponds to the picture.

Children in third grade and above completed the Gates-MacGinitie Reading Test- Fourth Edition Vocabulary subtest (GMRT) (MacGinitie & MacGinitie, 2006). For this subtest, the child is shown a sentence with one word underlined (e.g. practice item: She felt happy). The student is asked to choose from a list of four options (e.g. sleepy, hot, ready, glad), the correct option that represents a synonym or short definition of the underlined word. The task is intended to draw upon children’s vocabulary knowledge as well as other general literacy skills. The 2006 test manual reports construct validity estimates of .79 to .81; test-retest reliability of between .85 and .90; and internal reliability of .96.

Procedures

Investigators mailed a packet of materials in the mail to parents of twins, including an informed consent to sign, questionnaires and an ‘achievement pack’ which included a writing task and a reading vocabulary test. The achievement pack included specific instructions to the caregiver on how to administer the tests, including verbatim instructions to read aloud to the twins, time constraints (e.g. 15 minutes for writing task), as well as a place for any testing errors to be noted. Caregivers mailed the packet back to the research team along with the signed consent and other questionnaire materials. The written transcripts were entered electronically by undergraduate students in speech-language pathology. A graduate research assistant reviewed the paper copies against the electronic file to check accuracy resulting in 99% reliability score in electronic transcription.

Investigators typed children’s written samples into electronic transcript files and formatted the files for SALT analysis (Systematic Analysis of Language Transcripts) following standard SALT formatting (Miller & Iglesias, 2012). Transcriptionists maintained children’s use of capitalization, misspellings, and original punctuation without correction. Research assistants inserted codes proximal to each error or deviation from Standard English. In an attempt to describe errors, children’s written language errors were categorized into broad types of errors including grammatical errors and spelling errors. Examples of commonly occurring grammatical errors included: omission of past tense and possessive markers, the lack of verb-tense agreement, and the lack of singular and plural subject-verb agreement. Additionally, errors of syntax such as the omission of conjunctions, incomplete sentences, and run on sentences were also noted as grammatical errors. All misspelled words in the writing sample were identified by inserting a SALT code in the writing sample. Spelling errors included any deviation from the spelling conventions of Standard English.

Coding rules were established to distinguish potentially overlapping categories. For example, if the possessive marker s was present but the apostrophe was missing (e.g. principals office), it was considered a punctuation omission which was a writing convention error. In contrast, if the possessive marker s was not present (e.g. principal office) is was counted as a grammatical error. However, one lexical item was allowed to have more than one error attributed. In the above example, if principal was mis-spelled and lacking a possessive marker (e.g. prinsipal office), the spelling error code was additionally assigned with the grammatical error code.

Research assistants trained on the codes entered the error codes as described above for grammatical errors and spelling errors. The SALT software was also utilized to aggregate the occurrence of each type of error code. The first author reviewed one of every ten transcripts to double score the error codes. It was expected that coding errors would occur at a rate of between 3%−6% during sample analysis based on what is commonly reported in the literature (e.g. Fey et al., 2004; Gillam & Johnston, 1992; Sun & Nippold, 2011; Windsor, Scott & Street, 2000). Any disagreements in error assignments were discussed to resolve errors as coding continued. Agreement was 97% and 95% for spelling and grammar errors.

Analyses

The study design has both a cross-sectional and a repeated measures component. The cross-sectional component consists of students from Grade 1 to Grade 8. Then those students were assessed a second time one year later. To analyze both the within-year and across-year data on NDW (lexical diversity), TNW (lexical productivity), errors in spelling and errors in grammar, and the proportion of these data elements to total number of words, we fit a mixed-effects model that treated the one-year within-subjects data as a repeated measures factor, while also fitting a curve to the between-subjects means over grade. This model was able to simultaneously provide estimates of within-subjects change in measures as well as overall mean changes across grade. A quadratic model of mean change across grade was fit to best represent the mean changes in these measures.

Results

To address the first research aim, examining the average change in children’s written language performance in one year, descriptive means and standard deviations are reported at each time point (Table 2 and Table 3). Descriptive analyses of microstructural measures of writing (total number of words, lexical diversity, and accuracy) demonstrated large variability in performance within and across time points. Means and standard deviations on variables of interest for participants in 1^st - 4^th grade are provided in Table 2. Table 3 provides means and standard deviations for students in 5^th – 8^th grade.

Table 2.

Means and Standard Deviations for Microstructural Measures of Writing by Grade (1^st – 4^th Grade)

	1st grade				2nd grade				3rd grade				4th grade
	Time 1 n = 75		Time 2 n = 75		Time 1 n = 52		Time 2 n = 52		Time 1 n = 55		Time 2 n = 55		Time 1 n = 97		Time 2 n = 97
	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
NDW	32	18	59	28	57	33	61	27	63	30	84	38	93	36	91	44
TNW	51	35	103	62	105	72	104	55	117	71	151	82	168	77	163	101
Spelling Errors	7	6	6	5	6	6	4	5	6	7	6	5	7	9	4	5
Grammar Errors	2	2	5	4	3	3	5	5	3	3	6	6	4	4	4	5

Note. TNW = total number of words. NDW = number of different words. Time 1 represents performance in the fall of the listed grade. Time 2 reflects the same students one year later. Spelling errors and grammar errors refers to the rate of errors per sample.

Table 3.

Means and Standard Deviations for Microstructural Measures of Writing by Grade (5^th – 8^th Grade)

	5th grade				6th grade				7th grade				8th grade
	Time 1 n = 111		Time 2 n = 111		Time 1 n = 142		Time 2 n = 142		Time 1 n = 146		Time 2 n = 146		Time 1 n = 71		Time 2 n = 71
	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
NDW	91	42	89	38	106	58	102	44	106	51	98	45	108	53	105	45
TNW	168	92	160	78	195	125	184	95	195	111	172	94	195	106	182	90
Spelling Errors	4	4	3	4	4	4	3	3	3	3	3	3	3	3	3	3
Grammar Errors	3	3	4	5	3	4	4	4	3	4	4	5	2	3	4	4

Note. TNW = total number of words. NDW = number of different words. Time 1 represents performance in the fall of the listed grade. Time 2 reflects the same students one year later. Spelling errors and grammar errors refers to the rate of errors per sample.

Lexical Change

To examine average within-subjects one-year change for students at different grade levels, we examined change in written language measures across time points by grades. The average change by grade level in lexical diversity (NDW) between time 1 and one year later for each grade is depicted in Figure 1. At the initial time point there were large grade level differences in performance, particularly during 1^st −4^th grade. Individual change between time points was most apparent in the earlier grades. Students in Grade 1 showed the most dramatic change in mean NDW when tested one year later, with an average increase of 26 different words. Children in second grade demonstrated slightly greater lexical diversity when reassessed one year later, and third grade students showed a significant increase of 21 different words on average one year later. In contrast, for students in upper grades (5^th-9^th grade), there was no significant change in lexical diversity between time points, with less than a 7 word difference between Time 1 and Time 2.

Figure 1.

Lexical diversity (NDW) by grade at time 1 and 1 year later.

Note: Lexical diversity is represented by number of different words (NDW). The horizontal axis represents time and grade (e.g. G1= Grade 1). The end points of the line segments reflect repeated measures of NDW for students at Time 1 and Time 2 one year later.

The average change in NDW was tested for significant differences between time points. When average change was considered collectively (regardless of grade level), students’ average gains were not significant (F (1,1483.4) = 3.31, p = .069). The resulting curve fit of change estimated across grades change is provided in Table 4. Despite the overall lack of significant change, the average change in mean NDW for individual grade levels was significant for students in Grade 1 (F (1, 1482.9) = 16.6 p <.0001), Grade 2 (F (1, 1482.7) = 15.5, p<.001), and Grade 3 (F (1, 1483.3) = 5.53 p = .01) as seen in Table 3 and depicted in Figure 1. Similarly, lexical productivity as measured by TNW showed similar patterns of increases in Grade 1 and Grade 3 but not a substantial change for other grades.

Table 4.

Model Parameters: Testing Curve Fit of Microstructural Writing Measures Across Time

Dependent Variables	Parameter	Estimate	Standard Error	t value	p value
NDW	Intercept	32.61	4.49	7.27	<.0001
	Slope	22.90	6.06	8.34	<.0001
	Quadratic	−1.75	0.38	−4.62	<.0001
	Wave	24.65	6.06	4.07	<.0001
TNW	Intercept	53.35	9.64	5.53	<.0001
	Slope	45.72	5.90	3.47	<.0001
	Quadratic	−3.65	.81	−4.50	<.0001
	Wave	44.89	9.64	3.47	.0010
TTR	Intercept	0.64	0.02	28.00	<0.0001
	Slope	−0.01	.01	−1.03	0.302
	Quadratic	0.001	0.001	0.69	0.493
	Wave	0.04	0.025	1.77	0.077
Grammar Errors	Intercept	1.77	.34	5.13	<.0001
	Slope	.751	.21	3.56	<.0001
	Quadratic	−.093	.03	−3.21	.0010
	Wave	3.30	.58	5.68	<.0001
Spelling Errors	Intercept	7.04	.54	13.10	<.0001
	Slope	−.54	.33	−1.63	.103
	Quadratic	−.019	.045	−.06	.679
	Wave	−1.49	.67	−2.23	.026
Proportion of Grammar Errors	Intercept	0.058	.004	14.01	<.0001
	Slope	−0.089	.002	−3.53	<.0001
	Quadratic	0.0005	.0003	1.47	0.141
	Wave	−0.024	.004	−4.74	<.0001
Proportion of Spelling Errors	Intercept	0.058	.003	17.22	<.0001
	Slope	−0.012	.002	−5.66	<.0001
	Quadratic	0.008	.0003	2.82	<.005
	Wave	0.071	.006	12.10	<.0001

Note. NDW = number of different words. TNW = total number of words. TTR = NDW/TNW. Proportion of errors refers to number of errors divided by TNW. Wave refers to the within-subjects change in one year segments, displaying within-subjects change for Grade 1 to Grade 2.

To further examine the significant differences in lexical change in writing samples, we also tested the proportion of different words to total words, referred to as type-token ratio (NDW/NTW) for significant differences between time points. When average change was considered collectively, students’ average gains in the proportion of different words to total words was not significant (F(1,1058.9) = 3.1, p = .077). The resulting curve fit of change estimated across grades change is provided in Table 4. There was an interaction between grade and wave. The average change in mean ratio of different words to total words for individual grade levels was significant for Grade 5 students (F (1, 1058.1) = 5.67 p <0.018) and Grade 6 (F (1, 1058.02) = 6.77, p=0.009).

Accuracy

To address the second research aim, we examined students’ rates of errors in spelling and grammar. Although there was significant mean change across grades as demonstrated in Table 4, there was large variability within and between time points as noted in the descriptive means and standard deviations presented in Table 2 and 3. There were inconsistencies in increases and decreases across time as demonstrated in Table 4. Spelling errors occurred with higher frequency in the lower grades (e.g. 1^st – 3^rd grade), with a mean of 6–7 spelling errors per student. In the upper grades the average number of spelling errors decreased to 3–4 errors on average.

To further examine the proportion of errors, we subsequently examined the frequency of errors relative to the total number of words. Overall, there was significant change in the proportion of grammatical errors and spelling errors (refer to Table 4). There was also an interaction between grades and proportion of errors. For errors of spelling, the proportion of errors significantly decreased from one grade to the next in 1^st through 4^th grade but stabilized in 5^th grade with no significant change from 5^th to 6^th or from 6^th grade to upper grades. The proportion of grammatical errors showed significant increases from 1^st through 6^th grade. Conversely there was no significant change in 7^th or 8^th grade.

Relationships

To address the third research question regarding the relationship between written lexical measures (lexical diversity and productivity) and performance on standardized assessments, correlational analyses were conducted to examine the relationship between the GMRT-4, FCAT, and the lexical written narrative measures. There was a moderately strong positive relationship between lexical measures from writing samples and students’ performance on the GMRT-4 reading assessment as demonstrated in Table 5. Similarly, there was a small to moderate positive significant relationship between lexical measures (e.g. NDW, TNW) and students’ general reading achievement on the comprehensive state assessment. TNW was highly correlated with NDW and therefore distinctive relationships with reading achievement cannot be presumed.

Table 5.

Correlations between Lexical Diversity and Productivity and Achievement Measures

	N	M	SD	1	2	3	4
1. NDW (Lexical Diversity)	750	89.63	43.17	---	.97**	.39**	.44**
2. TNW (Lexical Productivity)	750	158.75	90.41		---	−.31**	.38*
3. FCAT Reading	625	237.95	25.09			---	.75**
4. GMRT-4 SS	739	538.14	47.57				--

Note.

^*p<0.05

^**p<0.01

All values were rounded to the hundredth place.

FCAT DSS = Florida Comprehensive Assessment Test Developmental Scaled Score in Reading (administered to participants in 3^rd–9^th grade)(Florida Department of Education, 2014)

GMRT-4 = Gates-McGinitie Reading Test- Fourth Edition Standard Scores (GMRT-4, MacGinitie & MacGinitie, 2006)

Discussion

Key Findings

Based on the average observed growth in lexical diversity, it appeared that students in Grade 1 and Grade 3 showed significant growth in lexical diversity and productivity from one year to the next. Students in later grades did not show significant change in lexical diversity or productivity; however significant change occurred in lexical diversity relative to total number of words. Additionally, the proportion of spelling errors decreased from one year to the next, for students in the elementary grades but not upper grades. The proportion of grammatical errors to total words increased across grades. Finally, there was a modest positive significant relationship between lexical diversity in writing samples and students’ performance on a standardized measures of reading achievement and reading comprehension.

Lexical Change

The finding that students in early elementary (1^st grade through 3^rd grade) showed gains in lexical diversity substantiates the utility of writing measures such as lexical diversity (NDW) and productivity (TNW) for progress monitoring as they appear to be sensitive to growth in early elementary grades. This finding is similar to those reported in previous studies such as Fey et al. (2004) and Bitetti and Hammer (2016). The average change amounts in the current findings are similar to those reported by Fey et al (2004), which reported average gains of 15.83 NDW in written responses as children progress from 2^nd to 4^th grade. The current results for NDW are also somewhat similar to those reported by Bitetti and Hammer (2016) despite different modalities of the samples. The current finding that NDW grew by 26 in one year for first graders and 21 for 3^rd graders is more than the average change reported by the average of 5.74 NDW growth reported in the Bitetti and Hammer study. The current findings substantiate the conclusions of Fey et al. (2004) that written composition tasks are educationally relevant tools for children in early elementary grade levels.

The lack of change on average in lexical diversity and productivity by students in upper grades was an unexpected finding; however, it should be taken together with the finding that students in the upper grades showed changes in the ratio of diversity relative to total number of words. The fact that students in the upper grades appeared to diversify the words in their writing samples relative to the total number of words may suggest that students were becoming more succinct in their responses. The significant change in the ratio of different words to total words, reiterates the importance of considering measures of lexical diversity or productivity in relation to each other for students in upper grades. Measures of diversity and productivity in isolation of each other, may not capture important change for students in the upper grades. Additionally, other lexical measures may be necessary to capture lexical change in the upper grades beyond diversity and productivity. We suspect that vocabulary growth in middle and high school may be apparent in the type of lexical items. As such, follow-up studies that assess other aspects of lexical sophistication (e.g. abstract nouns, tier two words, academic vocabulary) may be useful in illuminating other lexical changes in the upper grades.

Despite the lack of expected change for students in the upper grades, the current findings substantially add to our understanding of important considerations for assessing written lexical performance of students in the upper grades. The current findings offer important additions the literature base, particularly since there are few other written language studies available to compare findings that have focused on written vocabulary skills of students above Grade 7 (Kent & Wanzek, 2016). Available studies that have focused on students’ writing performance in upper grades have not examined lexical diversity and productivity specifically (e.g. Berninger, Abbott, Thomson, & Raskind, 2001; Bishop & Clarkson, 2003).

Accuracy

The finding that students showed improved accuracy in spelling from one grade level to the next through 4^th grade was not surprising. The fact that there was not significant change in the proportion of spelling errors (relative to total words) after Grade 5 may support the assumption that spelling skills are established in late elementary grades and stabilize. Additionally, these findings substantiate that writing samples are sensitive growth in spelling of students in elementary school which supports the usefulness of writing samples for progress monitoring of spelling skills. This is considered an appealing option for potential consideration, given the efficiency and feasibility of gathering classroom-wide administered written sample to provide a snapshot of spelling as a measure of progress across a school year.

In contrast, students in the current study demonstrated increases in the proportion of grammar errors relative to the total number of words between time points and grades for Grade 1 through Grade 6. This finding may reflect that the written samples of students in younger grades were primarily comprised of simple sentence structures. Further, it is considered likely that students attempt more complex sentence structures and sophisticated morphosyntactic forms as they progress through grades; however, the coding system in the current design did not allow for in-depth examination of sentence types. As noted by Crossley and McNamar (2010), sentences with difficult syntactic constructions include embedded phrases and as such, are often structurally dense, ambiguous, and more likely to be ungrammatical (Graesser, McNamara, Louwerse, & Cali., 2004). Although assessing grammatical complexity was not a primary aim of the current study and additional tiers of coding was not feasible, the findings warrant further exploration of students’ attempted complexity with additional categorization of grammar errors to illuminate the types of change occurring in grammatical skills. In light of the current findings, further exploration of grammatical structures including subcategories of error coding for finer distinction are recommended and planned for a future study.

Relationships between Language and Literacy Assessments

The finding that lexical measures from writing samples (i.e. NDW, TNW) were significantly related to reading achievement lends further support for the importance of written vocabulary skills on academic achievement. Further, the fact that there was a moderately strong relationship between lexical diversity in written samples and students’ performance on a standardized assessment of achievement substantiates the utility of NDW and TNW in writing samples as estimates of students’ language and literacy performance specifically. The moderately strong relationship observed in the current study was consistent with earlier reported findings when examined at the initial time point (Wood et al., 2018) and supported by numerous other studies in the literature substantiating a relationship between oral language and writing quality (e.g. Kent & Wanzek, 2016). Taken together, the findings substantiate the educational relevance of students’ writing samples as being a good predictor of performance on standardized measures of achievement.

Limitations

Results should be interpreted cautiously, given that only a single written sample was obtained from each participant at each time point. It cannot be assumed that the sample is representative of the students’ typical language diversity, productivity, complexity, or accuracy. The authors acknowledge that a single sample can be influenced by uncontrolled variables such as communicative interactions that preceded the sample, time of day, motivation, fatigue, and interest in the topic or prompt. Also, the same writing prompt was used at both time points which could be argued provides a familiarity advantage at the second time point; however, in the current dataset it appears more likely to have negatively affected growth. Perhaps students were bored or disinterested in the task when asked to write on the same topic again a year later.

Frequency, type, rate, and proportion of errors is admittedly a gross estimate of accuracy and has notable limitations that should be taken into consideration when interpreting results. It can be argued that given a narrative task students can chose words they know how to spell well and therefore the task may overestimate their general spelling abilities. Although this argument has been made previously, counterarguments in the literature, such as authors Broc et al., have argued this to be unlikely given the restricted topic which was not freely chosen and the lack of overuse of words observed. Although examining types of spelling or grammatical errors was not within the specific aims of the current study, examining error types is planned as a follow-up analyses, recognizing that spelling errors are not equal in type or severity (Bahr, Silliman, Berninger, & Dow, 2012; Quick & Erickson, 2018; Masterson & Apel, 2013). Future studies may consider the use of an accuracy metric coding system for error types based on the level of linguistic knowledge evidenced by the spelling error (e.g. Masterson & Apel, 2013; Quick & Ericson) and type of grammatical errors with consideration for syntactic complexity of the response (e.g. Eisenberg & Guo, 2013). We anticipate that a score for each morpheme and/or a scaled score for proximity to the Standard English spelling or other spelling classification systems for writing samples (Quick & Erickson, 2018) may provide additional insight into developmental changes.

Additionally, it should be noted when interpreting the results that lexical diversity and productivity may have been influenced by other skills not measured in the current study, such as speed of writing, fatigue, and spelling accuracy. Findings of previous studies suggest that children avoid writing words that they cannot spell (Sumner, Connelly, & Barnett, 2916). As a result, measures of lexical diversity may underestimate children’s ability to produce a variety of words, particularly for children who experience spelling difficulties. Although examining the relationship between speed of written text generation, spelling, and NDW was not a research aim of the current paper, findings of burgeoning studies suggest students generate longer responses when given electronic formats. A few studies have reported higher productivity and more revision by students in 5^th grade and upper grades when a keyboarding option was provided for writing (Berninger & Winn, 2008; Min, Guo, Higginbotham, & Qui, 2018). As such, we cannot presume the current findings are comparable to other studies reporting written samples generated with technology. In a future study it would be interesting to compare average performance on NDW between hand generated compositions and a keyboarding condition with electronic spelling assistance or correction.

Conclusions and Implications

Students in Grade 1 and Grade 3 showed significant growth in lexical diversity and productivity from one year to the next. Students in later grades did not show significant growth in lexical diversity as measured by diversity or productivity but demonstrated change in lexical diversity proportional to total number of words. The proportion of spelling errors decreased from one year to the next for students in 1^st grade through 4^th grade. Changes in the proportion of grammatical errors are considered inconclusive and warrant further exploration using more detailed coding of error types and sentence complexity. There was a modest positive significant relationship between lexical diversity in writing samples and students’ performance on standardized measures of reading achievement and reading comprehension.

The current findings add to the literature base that supports our understanding of component skills across the developmental spectrum. Results of this study could be useful for teachers, speech-language pathologists, and other educational personnel in gauging reasonable levels of expected change in students’ narrative writing in one year. Adding to the knowledge base of average change provides information to support teachers in evaluating expected change in students’ writing performance. Additionally, these findings may be useful for educational personnel in providing evidence-based feedback to parents regarding their students’ rate of development in comparison to expectations for their grade.