Comparing the Effects of Reading Intervention Versus Reading and Mindset Intervention for Upper Elementary Students with Reading Difficulties

Abstract

The primary purpose of this study was to examine the effects of providing mindset intervention in addition to reading intervention compared to only reading intervention for fourth graders with reading difficulties. Reading intervention was provided daily in 45 min sessions throughout the school year. Mindset intervention occurred in small groups for 24–30 min lessons. Multilevel structural equation modeling via n-level SEM was used to account for the latent variable representation of constructs, and the complex nesting and cross-classification structure of the data. Students in the reading intervention plus mindset condition significantly outperformed the business as usual condition on nonword reading (d = 0.35) as did students in the reading intervention condition (d = 0.20), who also outperformed the business as usual condition on phonological processing (d = 0.28). There were no significant differences among students in the three conditions on non-word reading, word reading, phonological processing, reading comprehension, or growth mindset. Initial reading achievement, mindset, and problem behavior did not generally moderate these findings.

Analogous to multi-level interventions in the field of medicine that target high-risk populations (e.g., Clauser, Taplin, Foster, Fagan, & Kaluzny, 2012; Stevens et al., 2016), multi-tiered early reading interventions provide instruction that is progressively more intensive in organization and delivery of instruction. Multi-tiered intervention frameworks are promoted in federal education legislation (Every Student Succeeds Act, 2015; Individuals with Disabilities Education Act, 2004) and have demonstrated promise for assisting students with early reading difficulties in improving their reading achievement at the earliest grades (Gersten et al., 2008; O’Connor et al., 2005; Wanzek et al., 2018) as well as reducing the incidence of reading disability (Carney & Stiefel, 2008; O’Connor et al., 2013; Wanzek & Vaughn, 2011). Multi-tiered instruction is practiced in all 50 states (Zirkel & Thomas, 2010). However, data from the National Assessment of Educational Progress (NAEP) indicate that, on average, 65% of fourth grade students do not read grade level text proficiently, and 70% of fourth grade students with disabilities do not achieve basic levels of reading (National Center for Education Statistics, 2019). Furthermore, it is concerning that reading achievement on the NAEP assessment dropped significantly from 2015 to 2017, and again in 2019, for students at the 25th and 10th percentiles in reading (this includes students with and without disabilities). Given that learning disabilities in reading occur along a continuum of severity rather than a clear cutpoint of achievement (Fletcher, Lyon, Fuchs, & Barnes, 2018), students who struggle with reading beyond the early elementary years are at risk for identification with disabilities.

Disability and Intervention

Reading achievement data suggests a critical need for upper elementary students with reading difficulties or disabilities (RD) to receive more intensive interventions in reading to assure that they make adequate progress. As would be expected, the focus of interventions for students with RD has been their academic deficits. However, planning intensive intervention for these students may include both academic and psychosocial aspects of disability. In fact, across disabilities, a biopsychosocial model of planning intervention is recommended in the International Classification of Functioning, Disability and Health (World Health Organization, 2002).

In terms of academics, intensive interventions require explicit and systematic instruction, ample opportunities to respond, and immediate, process-oriented feedback related to the identified academic challenges (Vaughn, Wanzek, Murray, & Roberts, 2012). However, the psychosocial aspects of support are rarely integrated in these academic interventions, which is problematic given students with RD are also at risk for increased levels of anxiety (Nelson & Harwood, 2011), behavioral issues (Ackerman, Izard, Kobak, Brown, & Smith, 2007; Nelson, Benner, Lane, & Smith, 2004), as well as lower academic self-concept and depression, (Bear, Minke, & Manning, 2002; Elbaum, 2002; Maag & Reid, 2006).

There is also a direct correlation between students’ self-determination (i.e., ability to persevere through intentional, self-initiated actions) and their academic achievement (Zheng et al., 2014), with students who demonstrate higher levels of self-determination better able to sustain academic engagement in challenging academic content. This is important because, the ability to intentionally engage in deliberate, effortful practice of challenging tasks is the highest predictor of skill acquisition (Ericsson, Krampe, & Tesch-Römer, 1993). Some theories of intelligence and some research suggests that students who believe that intelligence and academic ability are dynamic and can be developed through effortful practice of challenging work (i.e., growth mindset) demonstrate higher academic achievement than students who believe academic ability is innate and not malleable (i.e., fixed mindset; Blackwell, Trzesniewski, & Dweck, 2007; Dweck, 2006; Yeager & Dweck, 2012). This may be particularly relevant to students with the most significant reading problems who may need the most effort or practice to accelerate their learning, or for students who use problematic behavior to escape effortful practice in academic tasks. The primary purpose of the current study was to examine the implementation of an intervention with the academic component of reading instruction along with the psychosocial component of mindset for fourth-grade students with RD. A secondary, exploratory purpose was to examine whether student characteristics, such as initial reading achievement, level of fixed mindset, or problem behavior, moderated the effect of intervention.

Reading Intervention for Upper Elementary Students

Research on reading interventions in the upper elementary grades has increased in the past decade. In 2010, Wanzek, Wexler, Vaughn, and Ciullo (2010) identified only 24 studies of reading intervention for upper elementary students. A little more than half of these studies (n = 13) were experimental or quasi-experimental studies. The authors noted positive effects for interventions focused on word recognition or reading comprehension. However, they reported several limitations to the literature including the large majority of studies examined a single reading component (e.g., main idea strategy instruction), most utilized only researcher-developed measures to report effects, likely overestimating intervention efficacy, and most interventions were short in length (15 min sessions) or brief in duration (less than six weeks of instruction).

A more recent meta-analysis found an additional 25 experimental and quasi-experimental studies on reading interventions published since the Wanzek et al. synthesis (Donegan & Wanzek, submitted). Of the 37 study group comparisons, 32 were between a treatment intervention and a no treatment control group (i.e. students didn’t receive an intervention). Overall, small significant effects were reported for foundational (e.g., decoding, word reading, fluency) outcomes (g = 0.20) and comprehension outcomes (g = 0.09). However, there were no significant effects for upper elementary reading interventions when only standardized measures of reading were examined (g = 0.04 for foundational outcomes; g = 0.05 for comprehension outcomes).

Thus, the current research for upper elementary students demonstrated smaller effects than research on reading interventions for early elementary students with reading difficulties (Gersten et al., 2008; Wanzek et al., 2016), a finding in line with meta-analyses reporting lower overall effect sizes for reading interventions provided after Grade 3 (Scammacca et al., 2015; Wanzek et al, 2013). The higher effects reported in the primary grades may demonstrate benefits for intervening early or may also be an indicator of more false positives identified with reading difficulties in the early grades when all students are still acquiring beginning rreading abilities. However, one meta-analysis has reported higher effect sizes for older students such as 3^rd and 4^th graders and students with reading difficulties than students in younger grades or students at risk for reading difficulties (Suggate, 2010). Examination of ways to intensify reading interventions is needed for these students who continue to experience reading difficulties into their fifth year of elementary school. Although research from the early elementary level found it was effective to decrease instructional group size or increase amount of intervention (Gersten et al., 2008; Vaughn et al., 2012), individual studies of older students (Vaughn et al., 2010) as well as the results of several meta-analyses of interventions for older students provide no evidence that intervention effectiveness differs by typical variations in instructional group size or relative number of hours of intervention (Flynn, Zheng, & Swanson, 2012; Scammacca, Roberts, Vaughn, & Stuebing, 2015; Wanzek et al., 2013).

Psychosocial Elements in Intervention for Students with RD

Recent research supports the promise for intensifying academic interventions by embedding a psychosocial aspect of learning (Cassidy, 2014; Fuchs & Fuchs, 2015). The study of mindset is built on previous work examining effort, goal-setting, and how effort is linked to outcomes. For years, research studies on acquiring expertise have noted motivation to attend to and exert effort in practicing a task (deliberate practice) was the best predictor of improved performance (Ericsson, 2008; Ericsson et al., 1993). The highest levels of performance occurred when the task level was appropriate for the learner, corrective performance feedback was immediately provided, and repeated practice in the same or in a similar task was performed. Sustained effort in deliberate practice, rather than innate talent or experience, had the highest correlation with acquisition of expertise in a variety of disciplines, including academics (Ericsson, 2008). Thus, a person’s perseverance and willingness to work towards growth is one of the highest predictors of success, leading to onc of the key mechanisms of a growth mindset.

However, students with learning difficulties often consider their successes or failures to be the outcome of sources out of their control (e.g., luck or innate ability) and may expend less effort as a result (Borkowski, Estrada, Milstead, & Hale, 1989; Schunk, 2003; Torgesen & Licht, 1983). In the area of disability, psychosocial work in the related concepts of goal-setting, increasing awareness of progress towards goals, and linking positive behaviors such as effort to outcomes have been associated with improvements in academic achievement for students with learning difficulties and disabilities (Berkeley, Scruggs, & Mastropieri, 2010; Okolo, 1992; Page-Voth & Graham, 1999; Robertson, 2000). However, Cirino and colleagues (Cirino et al., 2017) note the inherent goal setting and monitoring of progress that is in well-designed evidence-based reading instruction, finding that the addition of a specific component to address these areas did not add to student achievement.

Recent research on mindset extends this work, but has not yet been specifically applied to the vulnerable population of students with RD at the upper elementary level. As mentioned earlier, some research suggests implicit theories of one’s own intelligence, or their mindset, are correlated with their academic ability. In academics, mindset refers to the attitudes, beliefs, and dispositions about learning that are most associated with successful academic outcomes (Dweck 1999; Dweck, Chiu, & Hong, 1995). A fixed mindset is a belief that individuals are born with a certain, finite ability that doesn’t change (Dweck, 2006). By contrast, a growth mindset is the belief that “your basic qualities are things you can cultivate through your efforts” (Dweck, 2006, p. 7), and is defined as students understanding their intelligence grows when they work through challenges. Hence, the students who value their efforts in achieving academic goals are able to fully engage in deliberate practice to learn. Several studies have noted the correlational relationship between growth mindset and academic tasks, each demonstrating that people with a growth mindset are more likely to improve their grades, recover their grade after failure on a test/assignment, study to learn, and stay motivated (Blackwell et al., 2007; Henderson & Dweck, 1990).

Importantly, mindset appears to be malleable; initial intervention work has shown some success in teaching a growth mindset, although the effects have not been consistent across studies and rarely have included standardized test scores. Additionally, the research has focused on older students rather than elementary students. For example, an experimental study with seventh graders examined a computer-based workshop providing instruction about brain anatomy, the brain’s malleability, the effect of learning on the brain, and the notion that intelligence is not static (Blackwell et al., 2007). Authors reported the grade point averages (GPA) of students in the mindset treatment were 0.3 points higher than students in the control who received study skills only.

A few studies have demonstrated that even very brief mindset training can have positive results with middle school, high school, or college students (Aronson et al., 2002; Good, Aronson, & Inzlicht, 2003; Paunesku et al., 2015). Trainings in these experimental studies involved reading materials about the malleability of intelligence and either mentoring or writing a letter to a struggling younger student about the importance of having a growth mindset. One recent study using a nationally representative sample of ninth graders with GPAs below their school’s median (n = 6,230) provided online training in growth mindset for less than one hour (Yeager et al., 2019). Across these studies, relative to controls, participating students had significantly greater increases in GPA. One study did note students receiving either incremental theory of intelligence and/or attribution training significantly outperformed a control condition that focused on the dangers of drug use, on state reading assessments with effect sizes ranging from 0.52 to 0.71 (Good et al., 2003).

The connection between student mindset and academic outcomes reached the popular media and led many schools and districts to consider ways to implement growth mindset instruction (Blad, 2016; Kirp, 2016). However, recent large meta-analyses examining the relation of mindset and academic outcomes have shown smaller and more mixed effects. One meta-analysis explored the effects of teaching growth mindset, but identified only two studies examining the effect on reading performance, with an overall effect of g = 0.65 (Sarrasin et al., 2018). Sarrasin et al. pointed out that the results were largely affected by one study examining reading fluency and suggested the effects may differ if a higher-level reading ability such as reading comprehension had been assessed. Another meta-analysis (Sisk, Burgoyne, Sun, Butler, & Macnamara, 2018) examined differences in academic achievement for adolescents and adults who received a growth mindset intervention relative to a control condition. Academic achievement was broadly defined (e.g., standardized tests, exams, GPA). The average effect size of mindset on academics was very small (d = 0.08), but moderator analyses revealed a stronger effect for the high-risk group (d = 0.19) and also for individuals from low socioeconomic backgrounds (d = 0.34). No disaggregated analyses of the effect on reading specifically were provided. The authors also considered whether studies measured mindset at both pre- and posttreatment time frames as a manipulation check. Notably, only 28 of the 43 effect sizes provided this information within the studies; the effect of mindset training on academics was not significant even when mindset was changed after intervention.

A recent study of a reading intervention incorporated a self-regulation component that included activities for positive self-talk, goal-setting, monitoring of strategy use, and growth mindset for students with RD in grades 2 to 4 (Denton, Montroy, Zucker, & Cannon, in press). Students participating in the intervention performed similarly to students remaining in typical school intervention services. Sisk et al. emphasized the need for future research with younger students andsubgroup analyses such as students with learning difficulties. They also recommended researchersincorporate manipulation checks, or pre-posttreatment assessment of mindset, to learn whether mindset interventions led to changes in mindset.

Study Purpose

The purpose of this study was to expand previous research by examining the effects of providing both reading and mindset intervention to address reading and psychosocial aspects of RD at the fourth-grade level. We were also interested in exploring the extent to which these interventions differentially benefited students. Specifically, we addressed two research questions:

1. What are the effects of intensive reading intervention with mindset training (reading intervention plus mindset) relative to reading intervention alone and business-as-usual (BAU) comparison on the academic outcomes of fourth-grade students with or at risk for reading disabilities?

2. What characteristics are related to student response to intervention?

We hypothesized that students in the reading intervention plus mindset condition would improve their reading outcomes more than students in the reading intervention only and BAU conditions. We predicted students in the reading intervention plus mindset condition woulddevelop a growth mindset to better work through the challenges in their reading development and, thus, would be able to progress more efficiently in the reading intervention. We also hypothesized that students with lower initial reading achievement, higher initial levels of fixed mindset, or higher levels of problem behavior would benefit more from the addition of the mindset intervention as these students may have more significant barriers to accelerating their reading achievement.

Method

Participants

Screening.

We recruited two cohorts of fourth-grade students from twelve schools in two school districts across two urban areas of the United States. One site was a mid-sized city in the southeastern part of the United States. The other site was a large, metropolitan area in the south. To identify our sample with RD, we sought to include students with identified disabilities and those at risk for identification with disability due to low reading achievement in early grade foundational skills. The classroom teachers identified students reading below grade level, including those receiving supplemental reading instruction or identified with dyslexia or reading disabilities. We excluded students identified with vision, hearing, or intellectual disabilities to maximize alignment of the study’s interventions to student needs. We screened these students whose parents consented using the Test of Word Reading Efficiency - Second Edition (TOWRE-2; Torgesen, Wagner, & Rashotte, 2012). Students who scored below the 30th percentile on the TOWRE-2 total word reading composite score, indicating word reading accuracy and/or fluency difficulties, were eligible for the study.

Assignment to condition.

Students who screened into the study completed a battery of pretest measures prior to being assigned to one of the three study conditions: (a) reading intervention plus mindset, (b) reading intervention, or (c) BAU comparison (e.g., typical school services). Using stratified random assignment (Shadish, Cook, & Campbell, 2002), we rank ordered students’ scores on the screening measure within each school, created triads of students with similar scores, and randomly assigned students within triads to condition.

Participant demographics.

Across the two years, a total of 361 fourth-grade students with or at risk for reading disabilities (RD) qualified for the study. Data provided by the school districts indicated key demographic characteristics of the sample, but 6% of data provided was missing. Of the 254 students with complete gender data, 51% (n = 184) were female. In terms of ethnicity, 46% were identified as Hispanic. In terms of race, 42% of the students identified as Black, 18% as White, 5% as American Indian, and 1% as Asian with 34% not reported (31% provided only an ethnicity, but no race information). Of those who reported language status, 14% of students were English language learners; all received instruction in English within their fourth grade classrooms… Seventy-four percent of the sample qualified for the free or reduced lunch program. Additionally, data provided by the districtss documented that 14% of students in the sample were identified with a disability. Of these students, 50% were identified as having a specific learning disability, 14% with a speech or language impairment, 10% with other health impairment, 2% with autism, 2% with emotional/behavior impairment; however 22% did not have a specific disability category identified in the data provided.

Attrition.

Overall and differential attrition was low (What Works Clearinghouse, 2014). After completing the screener, a total of 21 students (5.8% of the total sample) withdrew from the study due to families moving, school-scheduling conflicts, or parent-initiated withdrawal. Attrition by condition was comparable; six students (5%) in the reading intervention condition, seven (5.8%) in the reading intervention plus mindset condition, and eight (6.7%) in the BAU comparison withdrew from the study. The relation between overall attrition (i.e., 5.8%) and differential attrition (i.e., 0.8%, 1.7%, and 0.9%) was such that whether a conservative or liberal attrition standard is applied the expected bias is low (Institute of Education Sciences, 2019).

Study Procedures

All consented fourth graders were screened in September, and eligible students were pretested in September/early October. Students participated in the reading intervention from October to April. Those students assigned to the reading intervention plus mindset condition received the mindset intervention (a 12-week program) from December to April.

The reading intervention was provided in small groups of three to five students for 45 min daily during the school’s designated intervention time. Whenever possible within school schedules, we created homogenous instructional groups based on student pretest scores. Depending on attendance, individual students received 66–89 sessions of intervention (M = 73.5, SD = 3.7). The mindset intervention was provided in groups of four to seven students for 30 min twice weekly at a time designated by the school, usually a class that rotated so students didn’t miss the same class each time (e.g., music, science/social studies) Each student in the reading intervention plus mindset condition received 24 mindset lessons.

We conducted fidelity observations of reading interventionists once monthly and mindset interventionists once per intervention unit for five units. To establish inter-rater reliability each of the observers established 90% or higher coding accuracy with the team lead (i.e., gold standard approach, Gwet, 2007) for the reading intervention and for the mindset intervention. In the reading intervention, we rated each intervention component on a 4-point scale (3 = Excellent implementation [all checklist items implemented accurately]; 2 = Adequate implementation; 1= Weak implementation; 0 = Not completed [interventionist planned to complete component but did not]). We averaged each component’s ratings for an overall implementation rating. Reading intervention fidelity also included a global instructional quality rating and student engagement rating. Both of these global ratings used a 3-point scale. Quality checklist items included individualization, guided practice, pacing/wait time, monitoring, explicit and specific feedback, time management, and behavior management.

For the mindset intervention, we rated fidelity to each activity on a scale of 0–2. A score of 2 indicated the interventionist completed two thirds to all of the elements in the lesson activity; by contrast a score of 0 indicated the interventionist completed less than one third of the elements in the lesson activity. The activity ratings were averaged for an overall implementation rating. The mindset intervention fidelity also assessed active student engagement using the 3-point rating scale.

In addition, to document the instructional components, duration, and quality of reading instruction, as well as student engagement across all study conditions, we observed all reading interventions provided to students during their school day using the Instructional Content Emphasis Instrument-Revised (ICE-R; Edmonds & Briggs, 2003). Only the main content area of reading, instructional grouping, instructional quality rating, and student engagement rating were used on the ICE-R. Thus, observers recorded the amount of time spent in each reading component during the observation (e.g., phonics and word recognition, comprehension) as well as the type of instructional groupings (e.g., small group, individual) used throughout the session.. A 4-point global instructional quality rating based on the delivery of instruction (e.g., specific feedback, modeling, pacing, scaffolding) and a global 3-point student engagement rating were recorded for each observation. Observations of all reading interventions students received (treatment and BAU) were conducted in the fall, winter, and spring. Observers met reliability on the ICE-R by obtaining at least 90% reliability with the lead coder using a gold standard.

Description of Interventions

Reading intervention.

We implemented the Lindamood Phoneme Sequencing Program (LiPs; Lindamood & Lindamood, 2011) with all students assigned to the reading intervention and reading intervention plus mindset treatment conditions. LiPS is a reading intervention designed to support students with or at risk for RD through explicit, systematic instruction in phonological awareness, phonics, and text reading.

Each LiPS lesson included five components; within the 45 min lesson, the amount of time for each component across lessons varies slightly depending on the scope and sequence. In the first two components, students received explicit instruction in letter-sound relations. During initial LiPS lessons, students learned to feel the oral motor movements required to produce each sound (i.e., phoneme) in words. Second, we introduced new content (e.g., new letter sounds, orthographic patterns, and multisyllabic concepts) for the lesson and connected the new content to previously mastered concepts.

In the final three components of LiPS lessons, students applied their letter-sound knowledge through different activities. First to focus on development of phonemic awareness, students used colored chips to track changes in phonemes they heard across chains of real and pseudo words. For multisyllabic words, students tracked syllable changes and, if relevant, changes in phonemes within the accented syllable (e.g., con/TENT to con/TEND to con/TEND/er). Next, students read and spelled real and pseudo words that targeted the lesson’s concepts. In the final component of the LiPS lessons, students practiced individualized high frequency words lists and read connected texts. During text reading, interventionists supported fluent reading of the text, asked text-specific comprehension questions, and provided explicit instruction in formulating main idea statements.

Reading interventionists received three full days of training before beginning intervention. The training included a focus on letter-sound relationships, phonemic awareness activities, reading and spelling single-syllable words, and comprehension procedures. Reading interventionists received an additional day of training after approximately 6 weeks of implementation that focused on the more advanced content of multisyllabic word reading and spelling procedures. Throughout the training, instructional concepts and procedures were modeled and interventionists were provided time to practice and received feedback from the trainers. During implementation, interventionists were observed by project coaches every 1–2 weeks, and received feedback on implementation after each observation.. Mean overall LiPS implementation fidelity ratings were high (2.7 out of 3 points possible) with individual interventionists’ means ranging from 2.18 to 2.98. Additionally, mean global instructional quality ratings were high (2.73) with individual interventionists means ranging from 2 to 3. Mean student engagement ratings were also high (2.82) with individual interventionists’ means ranging from 2.2 to 3.

We also observed the reading intervention using the ICE-R to record the amount and type of reading instruction. Sessions occurred for an average of 43 min (SD = 5). Phonics and word reading instruction comprised the most amount of time (M = 24.35 min [57%] of time; SD = 12.6). We observed that, on average, our intervention reading instruction also included phonological awareness instruction (M = 6.19 min [14%] of time, SD = 4.2), comprehension instruction (M = 5.02 min [12%] of time, SD = 3.5), text reading with no other instruction occurring (M = 3.7 min [9%] of time, SD = 2.5), fluency instruction (M = 0.6 min [1%] of time, SD = 1.1), and spelling instruction (M = 0.53 min [1%] of time, SD=1.7). The other reading component instruction (i.e., vocabulary) accounted for less than 1% of instructional time. On average, differentiated instruction (students working on differentiated assignments and/or working with the teacher one-on-one) was provided to students for 1.42 min or 3% of time and students worked independently for 2.84 min or 7% of time. Non-instructional time was relatively brief (M = 2.6 min [6%] of time, SD = 2.1). The average global instructional quality rating was 3.2, indicating high-average to excellent quality. The average global student engagement quality rating was 2.6 indicating a medium to high level of student engagement.

Mindset intervention.

We implemented Brainology^® (Mindset Works, Inc., 2016) with all students assigned to the reading intervention plus mindset treatment condition. Brainology^® is a stand-alone global mindset training program that includes both independent online and teacher mediated activities for students. Students complete online modules that are followed by teacher-led activities and discussions to apply the information in the online modules. The modules and activities emphasize the ways that the brain functions and changes with skill practice. Through Brainology^® lessons, students learn the difference between growth and fixed mindsets. Students also learn strategies for applying a growth mindset to tackle academic challenges in any area. Brainology^® includes) with approximately 2.5 hr of instruction online and 10 hr of follow-up activities led by the teacher.

Mindset interventionists received one half day of training for Brainology^® in October of each year. Training consisted of reading and discussing articles about growth mindset, practicing instruction and reviewing the implementation guide for Brainology^®. Similar to the reading interventionists, mindset interventionists received feedback on implementation every 1–2 weeks. Mean overall mindset implementation fidelity ratings were high (1.8) with individual interventionists means ranging from 1 to 2. Mean active student engagement ratings were also high (2.7) with individual interventionists’ means ranging from 1 to 3.

Intervention teachers.

We hired and trained 20 reading interventionists and five mindset interventionists. To avoid carryover across conditions and control for teacher effects in the reading intervention across conditions, we trained interventionists in only reading or mindset intervention. The reading interventionists provided reading instruction to all treatment students regardless of the treatment condition The majority of the interventionists were female (n = 23; 96%). Interventionists’ highest degree earned included bachelor’s degree (n = 3; 12.5%) and master’s degree (n = 21; 87.5%). A total of 22 interventionists had degrees in education-related careers (certified teachers, counseling/psychology, speech/language, social work). The remaining two interventionists’ degrees were in non-education areas. The interventionists were black (n = 1; 4%), white (n = 20; 83%), and american indian (n = 3, 13%). Twenty percent of the interventionists reported their ethnicity as hispanic.

School-provided supplemental instruction.

A total of 147 students (n = 53 BAU [44%]; n = 94 treatment [39%]) received school-provided supplemental reading instruction throughout the study. Teacher reports indicated that this supplemental reading intervention was most often delivered by classroom teachers (49% of students) or other certified teachers (17% of students) with six interventions (11%) delivered by a paraprofessional, student teacher, or teaching assistant, and three interventions (2.7%) delivered by speech-language pathologists. Teacher reports indicated that none of the teachers had formal training in growth mindset instruction nor did they report any formal activities around growth mindset in their classrooms.

We also used the ICE-R to record the amounts and types of supplemental reading instruction, which occurred for an average of 35 min (SD = 11) per observation. Comprehension instruction comprised the most amount of time in the supplemental reading instruction (M = 12.63 min [36%] of time; SD = 10.5). Supplemental reading instruction also included phonics and word reading instruction (M = 7.6 min [22%] of time, SD = 8.7), other academic content instruction (e.g. writing or grammar instruction; M = 4.98 min [14%] of time, SD = 6), text reading with no other instruction occurring (M = 4.34 min [12%] of time, SD = 4.7), non-instructional time (M = 3.35 min [10%] of time, SD = 4.1), vocabulary (M = 2.03 min [6%] of time, SD = 4.6), spelling (M = 0.69 min [2%] of time, SD = 2.7) and fluency (M = 0.55 min [2%] of time, SD = 1.9). The other reading component instruction (i.e., phonological awareness) accounted for less than 1% of instructional time. On average, differentiated instruction (students working on differentiated assignments and/or working with the teacher one-on-one) was provided to students for 4.4 min or 13% of time and students worked independently for 8.71 min or 25% of time. The average global instructional quality rating was 2.8, indicating low-average to high-average quality. The average global student engagement quality rating was 2.5 indicating a medium to high level of student engagement.

Measures

Students were assessed on the reading achievement and mindset measures at pretest (September) and posttest (April). Research assessment staff, who were blind to student condition, were trained to administer and score each assessment with 100% agreement reliability prior to each assessment period. The teacher-completed problem behavior scale used for moderation was administered in January/February after teachers were familiar with students. Two research assistants also blind to condition independently double-scored and double-entered all data to ensure scoring and entering reliability.

Real and nonword reading.

The TOWRE-2 (Torgesen et al., 2012) is an individually-administered test of word reading fluency that includes two, 45 s subtests. For the sight word efficiency (SWE) subtest, students read a list of real words. For the phonemic decoding efficiency (PDE) subtest, students read a list of decodable pseudowords (i.e., nonwords). Assessors report the number of words read correctly in the allotted time. The SWE and PDE subtest scores create a composite score of total word reading efficiency. We used the composite scores to determine student eligibility for this study. The TOWRE-2’s average test-retest reliability is estimated at .87.

To assess students’ untimed word reading ability, we administered the letter word identification (LWID) and word attack (WA) subtests of the Woodcock Johnson Tests of Achievement - 4^th Edition (WJ-IV; Schrank, Mather, & McGrew, 2014). The WA subtest measures students’ ability to decode pseudowords of increasing difficulty. The LWID subtest assesses students’ ability to read real words. Internal consistency estimates range from .90 to .99.

Phonological processing.

We measured students’ phonological awareness using the blending and elision subtests of the Comprehensive Test of Phonological Processing - 2^nd Edition (CTOPP; Wagner, Torgesen, & Rashotte, 1999). On the blending subtests, test administers speak a series of segmented real words. Students blend the segmented word together, receiving a point for each word correctly blended. In contrast, on the elision subtest, students listen to a segmented word and orally provide the word with a specific segment removed. Each subtest has internal consistency estimates greater than .80 and test-retest coefficient estimates between .70 and .92.

Fluency.

We measured students’ oral reading fluency (ORF) using the Dynamic Indicators of Basic Early Literacy Skills - 6^th Edition (DIBELS; Good & Kaminski, 2002). Assessment staff individually-administered the ORF subtest of DIBELS with students. On this subtest, students read a series of three passages for one minute each. Test administrators score the total words read and number of errors, calculating the number of words read correctly in the minute. The ORF subtest of DIBELS has test-retest reliability estimates of .92–.97 for elementary students and alternate-form reliability estimates of .89–.94 across equated grade level passages.

Reading comprehension.

We measured students’ reading comprehension using the Gates-MacGinitie Reading Tests - 4^th Edition (GMRT; MacGinitie, MacGinitie, Maria, Dreyer, & Hughes, 2006) and the passage comprehension (PC) subtest of the WJ-IV (Schrank et al., 2014). The GMRT is group-administered, norm-referenced assessment. For the comprehension subtest, students read expository and narrative passages of increasing length and answer related multiple-choice comprehension questions. Internal consistency reliability estimates of the GMRT for fourth grade range from .91 to .96. On the PC subtest of the WJ-IV, students read cloze passages, identifying words that would fit blank spaces within the passage. The PC subtest has split-half reliability estimates of .83 to .96 for elementary age students (5–11 years).

Mindset.

We used an adapted version of the Student Mindset Survey (Blackwell et al., 2007) from Brainology^®. Petscher, Al Otaiba, Wanzek, Rivas, and Jones (2017) revised the survey (intended for older students) to align with fourth-grade student needs. Revisions included word substitutions to increase comprehensibility for some items (e.g., substituting “smart” for “intelligent” as in “You can learn new things, but you can’t really change how smart you are.”) and the removal of redundant items. Students rated these items on a likert scale ranging from 1 = disagree a lot, 2 = disagree, 3 = disagree a little, 4 = agree a little, 5 = agree, and 6 = agree a lot. Petscher et al. assessed the technical adequacy of the measure, which led to a final, adapted Student Mindset Survey that included eight general mindset items (α =.76) and a possible range of raw scores from 8 to 48, with lower scores indicating a more fixed mindset and higher scores indicating a more growth mindset. In the present study, the set of 8 items resulted in a sample internal consistency of only α = .57. However, removing two of the items resulted in improved reliability (α = .68).

Behavior.

The Social Skills Improvement System Rating Scale (SSIS; Gresham & Elliott, 2008) is a standardized, norm-referenced assessment of students’ behavior, social skills, and academic competency. We had teachers complete the problem behavior scale, measuring students’ externalizing/internalizing behaviors and hyperactivity/attention, based on a 4-point scale of the frequency with which students engage in each item (never, seldom, often, or almost always). We used these teacher ratings as a possible moderator of treatment effect. The problem behavior scale has test-retest reliability estimates of .75–.85 for children ages 3–18 and internal consistency estimates of .78–.95 for children ages 5–12.

Data Analysis

Primary and exploratory impact analyses were estimated using a longitudinal, multilevel structural equation modeling (ML-SEM) framework. Two primary benefits to this approach are that an SEM approach minimizes the impact of measurement error in the observed measurements on the estimation of effects and effect sizes in randomized controlled trials (RCTs); moreover the multilevel modeling approach accounts for the nested structure of the data (Goddard, Goddard, Kim, & Miller, 2015), which is critical for the proper estimation of the standard errors.

Although ML-SEM in the context of RCTs has benefits and is increasing in its utility for RCTs, they have been primarily used when subjects are fully nested in the RCT (i.e., FN-RCT). Researchers have increasingly noted the limitations conventional modeling techniques when the RCT employs a partially nested design (PN-RCT; Baldwin, Bauer, Stice, & Rohde, 2011; Lohr, Schochet, & Sanders, 2014). In a FN-RCT, all individuals are nested within a higher-level unit, such as students nested within classrooms. In a PN-RCT designs, only some individuals are nested within a higher-level unit whereas others are not. Baldwin and colleagues noted that this can frequently happen in education where individuals in a treatment condition are nested in a small group and the BAU control group individuals are not. The design of the current study was such that all individuals were nested within classrooms and schools; however, students in the reading intervention condition were nested in reading intervention small groups. Students in the reading intervention plus mindset condition were nested in both reading intervention and growth mindset intervention small groups. Control students were only nested in classrooms and schools. A further design complexity was that students in the reading intervention plus mindset condition were cross-classified across reading and growth mindset intervention small groups.

Multiple studies have proposed analytic approaches that take into account the partially nested data within the ML-SEM context (Lohr et al., 2014; Sterba et al., 2014; Wanzek et. al., 2017). We modeled PN-RCT data using n-level SEM (nSEM; Mehta & Neale, 2005; Petscher & Schatschneider, 2019). nSEM is a broad framework of multilevel SEM where the definition of level is far more flexible than convention multilevel modeling. A level in nSEM can refer to data where the nested structure is defined, such as individuals nested in a higher level, or it could also mean the level of fixed-effects factors such as sex (i.e., male and female), disability status (e.g., speech impaired, language impaired, or no disability), or treatment group (e.g., treatment and control). Petscher and Schatschneider (2019) showed how the inherent flexibility of nSEM can be used for the direct estimation of effects under many complex data structures, including PN-RCT with cross-classification. In the present study, our design in the nSEM framework included seven levels to account for one fixed effect factor of condition with three levels (i.e., students in each of the three conditions), and four random effect factors (i.e., reading intervention small groups, growth mindset intervention small groups, classrooms, and schools).

Our model comparison process for each outcome in the PN-RCT nSEM main effects analysis occurred in two step process. In the first step, the child-level latent post-test factors were constrained to have equivalent intercepts and for the loadings, we chose one loading from each of the latent factors to be constrained to equality both across groups and across time (pretest and posttest). Building the models this way allows for the latent factors to be on the same metric and allows the difference between latent means for each group to be interpretable. The second step released the constrained intercepts between pairs of child-level groups to estimate whether allowing freely estimated post-test means resulted in improved fit. A statistically significant improvement in the log likelihood difference between the fixed intercept and freed intercept models provides an indication of a significant condition effect. Within all main effect models, latent pretest variables were used as covariates in the model with constrained slopes between groups for the purpose of testing primary impacts.

Two sets of exploratory models tested for the moderation of treatment effects. In the first model set, the constrained pretest-posttest slope was included as was a constrained slope of growth mindset to the post-test. This model assumed the homogeneity of regression for the former constraint and an equality of prediction of growth mindset between conditions. A second model in this set relaxed the two constrained slopes to test whether pretest and growth mindset moderated the relation between condition and the latent post-test. The second set of exploratory models included (1) pretest-posttest slope as well as a slope of initial word reading (i.e., WJ-LWID) and (2) pretest-posttest slope as well as slope of initial problem behavior in the prediction of the latent post-test outcomes. The two-step process of fixed and freed modeling was conducted for the exploratory models and the log likelihood difference test was used for hypothesis testing.

Results

Descriptives

A preliminary review of the data showed missing data rates varied from 0% to 6.4%. We conducted an intent to treat analysis so these numbers include student attrition from the study described earlier, as well as any missed assessments from an individual student. Little’s missing completely at random (MCAR) test suggested that the missing data did not meet the reasonable assumptions for MCAR (χ²₍₁₅₈₎ = 224.36, p < .001). The missing data did not appear to be related to the missing values themselves, thus, using full information maximum likelihood for model estimation was appropriate. The missing data between the BAU and reading intervention ranged from 0% to 3.3% compared to 0% to 1.7% between BAU and reading intervention plus mindset, and 0% to 3.4% between reading intervention and reading intervention plus mindset. The relation between the overall and differential missing data was such that there was a tolerable threat of bias under the optimistic and cautious assumptions for attrition bias.

Descriptive statistics by study condition are reported in Table 1. Students’ standardized fall word reading (i.e., WJ-IV LWID and TOWRE SWE) were generally one standard deviation below the mean (i.e., less than 85), as were their nonword reading (i.e., WJ-IV WA and TOWRE PDE) and reading comprehension scores (i.e., WJ-IV PC and GMRT RC). This is consistent with selection criteria for participation in the study (i.e., scores below the 30th percentile on the TOWRE-2 composite score). Spring scores demonstrated the same approximate pattern as fall indicating that students were improving over time in order to maintain the same level of standard score. Due to the phenomenon that time-normed standard scores essentially wash out important developmental variance, the standard scores are reported for descriptive contextualization purposes but raw scores were used for the longitudinal nSEMs.

Table 1.

Descriptive Statistics of Measures by Condition.

		BAU (n = 121)		Reading (n =121)		Reading/Mindset (n =119)
Time Point	Measure	M	SD	M	SD	M	SD
Fall	WJ-IV LWID SS	85.55	12.46	85.39	10.52	84.58	12.86
	TOWRE-2 SWE SS	78.51	11.09	77.51	10.14	78.11	11.09
	WJ-IV WA SS	87.13	13.89	85.19	12.63	86.10	12.74
	TOWRE-2 PDE SS	74.73	10.08	74.79	10.82	73.25	10.95
	ORF P1	70.40	24.72	66.53	24.64	69.13	26.43
	ORF P2	67.50	28.34	65.06	28.46	66.36	28.22
	ORF P3	59.82	21.94	60.33	20.80	59.80	22.72
	WJ-IV PC SS	78.78	10.78	77.94	10.66	78.97	10.28
	GMRT RC SS	436.49	24.14	441.33	23.45	440.27	25.11
	CTOPP Blending SS	6.23	2.72	6.33	2.17	5.92	2.18
	CTOPP Elision SS	6.25	2.37	6.46	2.41	6.18	2.32
	MINDSET Raw	18.72	8.46	19.89	8.53	19.85	9.11
Spring	WJ-IV LWID SS	83.83	11.79	85.55	10.74	83.66	12.57
	TOWRE-2 SWE SS	83.12	12.46	81.70	11.44	81.94	11.43
	WJ-IV WA SS	85.50	13.77	89.84	12.73	88.25	14.98
	TOWRE-2 PDE SS	79.81	11.80	81.18	12.33	80.86	12.44
	ORF P1	82.63	25.65	82.58	26.27	81.26	28.14
	ORF P2	73.12	24.20	76.10	25.38	71.49	25.64
	ORF P3	67.21	25.42	68.78	27.15	68.05	26.49
	WJ-IV PC SS	79.49	11.24	80.29	10.74	80.69	12.25
	GMRT RC SS	446.90	24.98	448.54	30.10	450.91	24.99
	CTOPP BLENDING SS	7.19	2.86	8.30	3.31	7.41	2.62
	CTOPP ELISION SS	6.23	2.27	6.79	2.49	6.64	2.66
	MINDSET Raw	28.04	5.94	28.17	6.76	28.31	6.53

Note. BAU = business-as-usual condition; Reading = reading intervention condition; Reading/Mindset = reading intervention plus mindset condition. WJ-IV LWID = WJ-IV Letter Word Identification; TOWRE-2 SWE = Test of Word Reading Efficiency: Sight Word Efficiency subtest; WJ-IV WA = WJ-IV Word Attack; TOWRE-2 PDE = Test of Word Reading Efficiency: Phonemic Decoding Efficiency subtest; ORF = DIBELS Oral Reading Fluency – Passages 1, 2, and 3; WJ-IV PC = WJ-IV Passage Comprehension; GMRT RC = Gates-MacGinitie Reading Tests: Reading Comprehension subtest; CTOPP Blending = Comprehensive Test of Phonological Processing, Blending; CTOPP Elision = Comprehensive Test of Phonological Processing, Elision; Mindset = Growth Mindset Raw Scores; SS= Standard Score.

Correlations (Table 2) across measures within each time point showed generally moderate to strong associations among the reading measures. By contrast, the growth mindset total score was relatively uncorrelated with the reading measures; in fall the correlations ranged from −0.07 to 0.15 and in the spring the correlations ranged from 0.03 to 0.13.

Table 2.

Fall (Lower Diagonal) and Spring (Upper Diagonal) Correlations among Observed measures.

Measure		1	2	3	4	5	6	7	8	9	10	11	12
1.	WJ-IV LWID SS	1.00	0.64	0.68	0.69	0.71	0.68	0.70	0.67	0.40	0.14	0.49	0.03
2.	TOWRE-2 SWE SS	0.65	1.00	0.41	0.69	0.77	0.75	0.76	0.47	0.33	0.10	0.29	0.13
3.	WJ-IV WA W	0.69	0.47	1.00	0.65	0.46	0.44	0.46	0.45	0.28	0.25	0.55	0.06
4.	TOWRE-2 PDE SS	0.58	0.59	0.57	1.00	0.65	0.67	0.65	0.46	0.27	0.22	0.43	0.08
5.	ORF P1	0.77	0.80	0.54	0.60	1.00	0.88	0.86	0.59	0.45	0.10	0.34	0.03
6.	ORF P2	0.72	0.78	0.50	0.57	0.92	1.00	0.83	0.55	0.43	0.13	0.32	0.03
7.	ORF P3	0.71	0.75	0.49	0.60	0.88	0.90	1.00	0.54	0.46	0.06	0.34	0.04
8.	WJ-IV PC W	0.66	0.58	0.46	0.34	0.59	0.59	0.55	1.00	0.42	0.16	0.40	0.08
9.	GMRT RC SS	0.30	0.33	0.14	0.18	0.39	0.43	0.43	0.32	1.00	0.12	0.25	0.06
10.	CTOPP BLENDING SS	0.37	0.18	0.40	0.28	0.22	0.20	0.21	0.30	0.13	1.00	0.29	0.04
11.	CTOPP ELISION SS	0.41	0.26	0.46	0.31	0.27	0.26	0.25	0.30	0.13	0.39	1.00	0.04
12.	MINDSET RAW	−0.06	0.06	−0.04	0.15	−0.02	0.00	−0.02	−0.06	−0.07	−0.04	−0.01	1.00

Note. WJ-IV LWID = WJ-IV Letter Word Identification; TOWRE-2 SWE = Test of Word Reading Efficiency: Sight Word Efficiency subtest; WJ-IV WA = WJ-IV Word Attack; TOWRE-2 PDE = Test of Word Reading Efficiency: Phonemic Decoding Efficiency subtest; ORF = DIBELS Oral Reading Fluency – Passages 1, 2, and 3; WJ-IV PC = WJ-IV Passage Comprehension; GMRT RC = Gates-MacGinitie Reading Tests: Reading Comprehension subtest; CTOPP Blending = Comprehensive Test of Phonological Processing, Blending; CTOPP Elision = Comprehensive Test of Phonological Processing, Elision; Mindset = Growth Mindset Raw Scores; SS= Standard Score. Values in bold are not statistically significant, p < .05.

Effects of Reading Interventions

The first step in fitting these nSEM models was to identify which observed variables to load onto which latent variables. In these models, the WJ-IV word identification and TOWRE-2 sight word efficiency subtests comprised the word reading factor, the WJ-IV word attack and TOWRE −2 phonemic decoding efficiency comprised the non-word reading factor, the three ORF passages represented the fluency factor, the GMRT reading comprehension and WJ-IV passage comprehension subtest were loaded onto the reading comprehension factor, and the CTOPP measures were loaded onto the phonological processing factor. These factors were fit for the pretest data and the posttest data. For each of the substantive factors, a multi-group model was fit where the posttest factor was regressed onto the pretest factor for the treatment groups and the comparison group. For the BAU condition, we modeled the nesting of students in classrooms and schools, in the reading intervention condition we modeled the nesting of students in the reading intervention small group, classrooms, and schools, and in the reading intervention plus mindset condition we modeled the cross-classification of students in the reading intervention and growth mindset intervention small groups and nesting in classrooms and schools.

The factor loadings in each of the substantive models were constrained such that for each observed variable, the pretest-posttest and the across-group loadings were constrained to equality, as were the error variances. Baseline equivalence was tested across the latent pretest means with all standardized latent means in pairwise comparisons across groups as g < .10. The regression line from the pretest to the post-test means were also constrained to be equal across the two groups. To address the first research question related to relative treatment effects on each of the latent outcomes and the second research question related to characteristics, or moderators, of response to the treatments, we tested the difference between the latent posttest means for pairwise comparisons across the three groups. In the initial, unconditional means model (i.e., no predictors) the school variance for each of the outcomes was < .01; thus, it was removed to improve parsimony of model specification (i.e., 3 student levels and 3 random effects).

The results of the main effect analyses are reported in Table 3. The comparison of reading intervention to BAU only showed a statistically significant effect in favor of the individuals in the reading intervention condition for the nonword reading (p = .015; d = 0.29) and phonological processing (p = .012; d = 0.28) outcomes. Both effects were statistically significant after applying a Benjamini-Hochberg correction and the remaining tests of efficacy were not statistically significant. Effect sizes across the remaining outcomes were: d = −0.02 (growth mindset), d = 0.08 (word reading), and d = 0.13 (oral reading fluency), d = 0.19 (reading comprehension).

Table 3.

nSEM Impact Model Results for Latent Word Reading, Non-Word Reading, Fluency, Reading Comprehension, Phonological Processing, and Observed Growth Mindset.

Comparison	Outcome	Model	-2LL	nParm	AIC	BIC	Δ−2LL	Δdf	p	d
Reading vs. BAU	Word Reading	Fixed	7108.29	16	7140.29	7217.84
		Freed	7106.82	17	7140.82	7223.22	1.47	1	.225	0.08
	Non Word Reading	Fixed	6775.14	16	6807.14	6884.69
		Freed	6769.28	17	6803.28	6885.68	5.86	1	.015*	0.29
	Fluency	Fixed	7706.40	16	7738.40	7815.93
		Freed	7702.55	17	7736.55	7818.93	3.85	1	.050	0.13
	Reading Comp.	Fixed	8025.15	16	8057.15	8134.65
		Freed	8023.53	17	8057.53	8139.87	1.62	1	.203	0.19
	Phon. Processing	Fixed	5465.70	16	5497.20	5575.20
		Freed	5459.42	17	5493.42	5575.76	6.28	1	.012*	0.28
	Mindset	Fixed	1423.04	5	1433.04	1449.96
		Freed	1422.89	6	1434.89	1455.20	0.15	1	.699	−0.02
Reading/Mindset vs. BAU	Word Reading	Fixed	7077.22	17	7111.22	7193.38
		Freed	7077.13	18	7113.13	7200.13	0.09	1	.762	0.02
	Non Word Reading	Fixed	6708.16	17	6742.16	6824.32
		Freed	6701.47	18	6737.47	6824.46	6.70	1	.010*	0.35
	Fluency	Fixed	7616.84	17	7650.84	7733.01
		Freed	7616.73	18	7652.73	7739.72	0.12	1	.732	0.02
	Reading Comp.	Fixed	7887.34	17	7921.34	8003.44
		Freed	7884.92	18	7920.92	8007.86	2.42	1	.120	0.23
	Phon. Processing	Fixed	5459.27	17	5493.27	5575.36
		Freed	5456.61	18	5492.61	5579.52	2.66	1	.103	0.2
	Mindset	Fixed	1403.91	6	1415.91	1436.16
		Freed	1403.90	7	1417.90	1441.52	0.01	1	.917	0.06
Reading vs. Reading/Mindset	Word Reading	Fixed	7004.37	17	7038.37	7120.62
		Freed	7003.94	18	7039.94	7127.03	0.43	1	.513	0.06
	Non Word Reading	Fixed	6692.68	17	6726.68	6808.94
		Freed	6692.53	18	6728.53	6815.62	0.15	1	.697	−0.05
	Fluency	Fixed	7726.93	17	7760.93	7843.16
		Freed	7714.84	18	7750.84	7837.91	12.09	1	.001*	0.15
	Reading Comp.	Fixed	7783.43	17	7817.43	7899.65
		Freed	7783.36	18	7819.36	7906.41	0.07	1	.784	−0.03
	Phon. Processing	Fixed	5490.04	17	5524.04	5606.24
		Freed	5489.22	18	5525.22	5612.26	0.82	1	.367	0.12
	Mindset	Fixed	1439.71	7	1453.71	1477.40
		Freed	1436.40	8	1452.40	1479.47	3.32	1	.069	−0.04

Note. BAU = business-as-usual condition; Reading = reading intervention condition; Reading/Mindset = reading intervention plus mindset condition. −2LL = −2*log likelihood, AIC =Akaike Information Criteria, BIC = Bayes Information Criteria. Effect size d computed by computing the difference between the two latent posttest means and dividing by the pooled observed posttest standard deviations. Fixed is a model where the post-test latent means were constrained to equality. Freed is a model where the post-test latent means were free to vary.

^*Effect remains significant after applying Benjamini-Hochberg correct for the false discovery rate.

The comparison of reading intervention plus mindset to BAU showed a statistically significant effect (with Benjamini-Hochberg adjustment) of the reading intervention plus mindset on the nonword reading outcome (p = .010; d = 0.35). The effect sizes across the remaining outcomes were: d = 0.02 (word reading and oral reading fluency), d = 0.06 (growth mindset), d = 0.20 (phonological processing), and d = 0.23 (reading comprehension). The comparison of the two intervention conditions only showed a statistically significant effect for reading intervention compared to reading intervention plus mindset on oral reading fluency (p = .001; d = 0.15). Effect sizes for the remaining outcomes where no significant effects were observed are as follows: d = −0.05 (nonword reading), d = −0.04 (growth mindset), d = −0.03 (reading comprehension), d = 0.06 (word reading), and d = 0.12 (phonological processing).

Characteristics of Student Response to Intervention

Our moderator analysis for pretest and growth mindset (Table 4) showed no statistically significant improvements in the model by allowing the relation of pre-test to post-test to vary between groups or the relation of growth mindset to post-test to vary between groups. For selected outcomes, the freely estimated regressions between groups resulted in worse model fit (i.e., an increase in the log likelihood). A similar pattern of results was observed for the moderator analyses that included pre-test and word reading as moderators (Table 5). The next set of exploratory moderator analyses indicated that higher initial growth mindset and initial phonological processing was associated with higher phonological processing scores for reading intervention compared to the BAU (p < .001). Further, moderator analyses for problem behavior (Table 6) showed that higher initial reading comprehension and scores on problem behavior (i.e., high standard scores, indicating fewer problem behaviors) were associated with higher latent reading comprehension posttest scores for reading intervention compared to the BAU (p = .043) and that higher initial phonological processing and scores on problem behaviors were related to higher latent phonological processing posttest scores (p < .001).

Table 4.

nSEM Exploratory Moderation Model Results for Latent Word Reading, Non-Word Reading, Fluency, Reading Comprehension, Phonological Processing, and Observed Growth Mindset with Baseline Pretest and Growth Mindset as Moderators.

Comparison	Outcome	Model	-2LL	nParm	AIC	BIC	Δ−2LL	Δdf	p
Reading vs. BAU	Word Reading	Fixed	8726.57	25	8776.57	8903.17
		Freed	8849.46	27	8903.46	9040.19	−122.89	2	-
	Non Word Reading	Fixed	8381.05	25	8431.05	8557.65
		Freed	8382.38	27	8436.38	8573.10	−1.33	2	-
	Fluency	Fixed	9320.47	25	9370.47	9497.05
		Freed	9319.94	27	9373.94	9510.64	0.54	2	.765
	Reading Comp.	Fixed	9642.90	25	9692.90	9819.43
		Freed	9687.43	27	9741.43	9878.08	−44.53	2	-
	Phon. Processing	Fixed	7077.66	25	7127.66	7254.19
		Freed	7075.75	27	7129.75	7266.40	1.91	2	.384
Reading/Mindset vs. BAU	Word Reading	Fixed	8747.37	26	8799.37	8930.86
		Freed	8747.19	28	8803.19	8944.79	0.18	2	.913
	Non Word Reading	Fixed	8363.04	26	8415.04	8546.52
		Freed	8362.69	28	8418.69	8560.28	0.35	2	.838
	Fluency	Fixed	9285.78	26	9337.78	9469.26
		Freed	9285.40	28	9341.40	9483.00	0.38	2	.829
	Reading Comp.	Fixed	9554.29	26	9606.29	9737.71
		Freed	9554.05	28	9610.05	9751.58	0.24	2	.887
	Phon. Processing	Fixed	7126.53	26	7178.53	7309.92
		Freed	7125.84	28	7181.84	7323.34	0.68	2	.710
Reading vs. Reading/Mindset	Word Reading	Fixed	8646.57	26	8698.57	8830.07
		Freed	8644.82	28	8700.82	8842.44	1.74	2	.418
	Non Word Reading	Fixed	8326.52	26	8378.52	8510.03
		Freed	8332.43	28	8388.43	8530.05	−5.90	2	-
	Fluency	Fixed	9355.82	26	9407.82	9539.30
		Freed	9355.25	28	9411.25	9552.85	0.56	2	.755
	Reading Comp.	Fixed	9426.96	26	9478.96	9610.42
		Freed	9424.59	28	9480.59	9622.16	2.37	2	.306
	Phon. Processing	Fixed	7206.07	26	7258.07	7389.51
		Freed	7206.07	28	7262.07	7403.62	0.00	2	.999

Note. BAU = business-as-usual condition; Reading = reading intervention condition; Reading Plus = reading intervention plus mindset condition. −2LL = −2*log likelihood, AIC =Akaike Information Criteria, BIC = Bayes Information Criteria. Fixed is a model where the post-test latent means were constrained to equality. Freed is a model where the post-test latent means were free to vary.

Table 5.

nSEM Exploratory Moderation Model Results for Latent Word Reading, Non-Word Reading, Fluency, Reading Comprehension, Phonological Processing, and Observed Growth Mindset with Baseline Pretest and WJ LWID as Moderators.

Comparison	Outcome	Model	-2LL	nParm	AIC	BIC	Δ−2LL	Δdf	p
Reading vs. BAU	Non Word Reading	Fixed	8522.35	25	8572.35	8699.25
		Freed	8521.67	27	8575.67	8712.71	0.69	2	.710
	Fluency	Fixed	9364.98	25	9414.98	9541.85
		Freed	9365.04	27	9419.04	9556.07	−0.06	2	-
	Reading Comp.	Fixed	9774.91	25	9824.91	9951.74
		Freed	9774.20	27	9828.20	9965.18	0.71	2	.701
	Phon. Processing	Fixed	7331.25	25	7381.25	7508.08
		Freed	7317.15	27	7371.15	7508.13	14.09	2	.001
Reading/Mindset vs. BAU	Non Word Reading	Fixed	8433.42	26	8485.42	8617.05
		Freed	8432.87	28	8488.87	8630.64	0.54	2	.762
	Reading Comp.	Fixed	9605.29	26	9657.29	9788.86
		Freed	9605.28	28	9661.28	9802.97	0.01	2	.996
	Fluency	Fixed	9270.11	26	9322.11	9453.75
		Freed	9270.10	28	9326.10	9467.86	0.02	2	.992
	Phon. Processing	Fixed	7346.35	26	7398.35	7529.90
		Freed	7346.12	28	7402.12	7543.79	0.23	2	.892
Reading vs. Reading/Mindset	Non Word Reading	Fixed	8415.46	26	8467.46	8599.21
		Freed	8415.07	28	8471.07	8612.95	0.39	2	.823
	Fluency	Fixed	9319.74	26	9371.74	9503.46
		Freed	9319.74	28	9375.74	9517.60	0.00	2	-
	Reading Comp.	Fixed	9466.52	26	9518.52	9650.23
		Freed	9462.60	28	9518.60	9660.44	3.92	2	.141
	Phon. Processing	Fixed	7331.04	26	7383.04	7514.72
		Freed	7330.39	28	7386.39	7528.20	0.65	2	.724

Note. BAU = business-as-usual condition; Reading = reading intervention condition; Reading/Mindset = reading intervention plus mindset condition. −2LL = −2*log likelihood, AIC =Akaike Information Criteria, BIC = Bayes Information Criteria. Fixed is a model where the post-test latent means were constrained to equality. Freed is a model where the post-test latent means were free to vary.

Table 6.

nSEM Exploratory Moderation Model Results for Latent Word Reading, Non-Word Reading, Fluency, and Reading Comprehension, Phonological Processing, with Baseline Pretest and Problem Behavior as Moderators.

Comparison	Outcome	Model	2LL	nParm	AIC	BIC	Δ−2LL	Δdf	p
Reading vs. BAU	Word Reading	Fixed	9140.07	25	9190.07	9316.88
		Freed	9140.02	27	9194.02	9330.98	0.04	2	-
	Non Word Reading	Fixed	8798.23	25	8848.23	8975.04
		Freed	8796.33	27	8850.33	8987.29	1.89	2	-
	Fluency	Fixed	9734.89	25	9784.89	9911.68
		Freed	9734.66	27	9788.66	9925.59	0.24	2	0.888
	RC	Fixed	10104.8	25	10154.8	10281.5
		Freed	10098.5	27	10152.5	10289.4	6.30	2	0.043
	Phon. Processing	Fixed	7602.17	25	7652.17	7778.92
		Freed	7493.48	27	7547.48	7684.37	108.69	2	<.001
Reading/Mindset vs. BAU	Word Reading	Fixed	9123.44	26	9175.44	9307.01
		Freed	9123.54	28	9179.54	9321.23	−0.10	2	0.878
	Non Word Reading	Fixed	8748.59	26	8800.59	8932.17
		Freed	8748.35	28	8804.35	8946.04	0.25	2	0.883
	Fluency	Fixed	9662.87	26	9714.87	9846.44
		Freed	10175.8	28	10231.8	10373.5	−512.91	2	-
	RC	Fixed	9932.91	26	9984.91	10116.4
		Freed	9995.38	28	10051.4	10193	−62.47	2	-
	Phon. Processing	Fixed	7508.73	26	7560.73	7692.21
		Freed	7552.12	28	7608.12	7749.71	−43.39	2	-
Reading vs. Reading/Mindset	Word Reading	Fixed	9034.15	26	9086.15	9217.83
		Freed	9033.26	28	9089.26	9231.07	0.89	2	0.640
	Non Word Reading	Fixed	8717.89	26	8769.89	8901.58
		Freed	8818.76	28	8874.76	9016.57	−100.87	2	-
	Fluency	Fixed	9744.64	26	9796.64	9928.3
		Freed	9743.82	28	9799.82	9941.61	0.82	2	0.663
	RC	Fixed	9812.57	26	9864.57	9996.21
		Freed	9812.56	28	9868.56	10010.3	0.01	2	0.994
	Phon. Processing	Fixed	7594.97	26	7646.97	7778.59
		Freed	7576	28	7632	7773.74	18.97	2	<.001

Note. BAU = business-as-usual condition; Reading = reading intervention condition; Reading/Mindset = reading intervention plus mindset condition. −2LL = −2*log likelihood, AIC =Akaike Information Criteria, BIC = Bayes Information Criteria. Fixed is a model where the post-test latent means were constrained to equality. Freed is a model where the post-test latent means were free to vary.

Discussion

In this study, our primary purpose was to examine the effects of reading intervention plus mindset training relative to reading intervention and to typical school services. A secondary, and more exploratory purpose was to examine whether student characteristics may have moderated student response to interventions. We hypothesized that students receiving mindset intervention in addition to reading intervention would demonstrate accelerated reading outcomes compared to students receiving the reading intervention only or to students receiving typical school services. We also hypothesized that the intervention would benefit students with lower initial reading achievement, higher initial levels of fixed mindset, or higher levels of problem behavior that may be interfering with their reading progress.

Related to our first research question, we did not find a value added of adding the psychosocial aspects of mindset relative to the reading intervention. Effect sizes were similar across both reading groups when compared to BAU, generally showing small effects for nonword reading (d = 0.29 to 0.35), phonological processing (d = 0.20 to 0.28), and reading comprehension (d = 0.19 to 0.23). Generally, we found no effects for word reading and oral reading fluency (d = 0.02 to 0.13). Although there was a significant effect for oral reading fluency in favor of the reading intervention group, the effect was very small (d = 0.15) and the statistical significance likely an artifact of a latent variable created with different passages of the same measure, increasing the latent construct’s reliability coefficient. It is encouraging that students with RD who received either of the intensive reading interventions accelerated their learning in phonological areas, a key deficit for many students with disabilities (Trainin & Swanson, 2005). Recall that to qualify as RD for the study, students scored below the 30th percentile on a standardized test of word and non-word reading.

These positive effects in phonological areas likely reflect the reading intervention focus on helping students with RD hear and manipulate sounds in words. The observations verified that a majority of reading intervention time focused on word reading instruction (57%) and phonological awareness (14%), and relatively less time on comprehension (12%) and text reading (9%). By contrast, in our observations of school provided intervention, relatively higher proportions of time were spent on comprehension (36%) and less time on phonics and word reading (22%) or phonological awareness (less than 1%). However, despite the focus of the treatment on word reading instruction and the accelerated learning on phonological tasks, we noted similar growth among the treatment and BAU conditions in word reading.

These findings add to previous research suggesting that students with RD benefit from intensive reading intervention, with stronger effects on foundational skills than on reading fluency or reading comprehension (Miciak et al., 2017; Wanzek et al., in press). It is important to consider that even after intervention, students generally ended the year with relatively weak standard scores, and with very poor oral reading fluency. Relative to benchmarks for risk at the end of fourth grade (e.g., DIBELS score of 124 words correct per minute), the median oral reading fluency scores were much lower, ranging from 71.44 to 76.10 words correct per minute, for students in each of the study conditions. Notably, although students in the treatment made more gains in phonological areas like nonword reading, their gains in word reading were similar to the comparison group. We also did not see the level of gains in word reading or comprehension from fall to spring that were noted in a previous study implementing the same reading intervention (Torgesen et al., 2001). The sample in the Torgesen study averaged more significant reading difficulties and received an average of 10 hours more intervention in a shortened (6 week), clinic setting with one-on-one instruction. Thus, there may have been ways in which the Torgesen implementation was more intensive than the implementation in our study. A closer examination of the reading program implementation that examines the actual amount of word reading and text reading practice (e.g., number of words) could provide valuable information for the true dosage of reading components students receive across interventions.

We found no differential benefit of the reading intervention plus mindset intervention on student mindset. Thus, any trend toward a stronger growth mindset was similar across study conditions and may have been developmental or an artifact of students with reading difficulties receiving reading intervention from the school or research team. In other words, simply receiving reading intervention and/or experiencing gains in reading achievement may also result in movement towards a growth mindset. Sisk et al. (2018) reported a similarly small effect (d = 0.08) of mindset intervention on academic outcomes for adolescents and adults, but a stronger effect (d = 0.18) for students at academic risk which we did not note in this study of students with reading difficulties. The additional mindset intervention for these elementary students with reading difficulties did not help students change their mindset over other reading interventions. Moreover, the mindset total score was relatively uncorrelated with the reading measures suggesting the improved reading outcomes that did occur were part of the reading interventions and not a change in mindset.

The biopsychosocial model of intervention may require more deliberate integration of mindset into the reading intervention to adequately intensify the intervention. There is some evidence that mindset may be multi-dimensional, with underlying factors specific to a content area like reading (Petscher et al., 2017). Thus, a mindset intervention intended to address the general factor of mindset across academic areas may not be intensive enough for students with reading difficulties in the upper elementary grades. These students have particular difficulties in the area of reading and may need the mindset work to be explicitly embedded to their area of need to specifically address a mindset they may have in the content area of reading.

Our secondary focus for this study was exploratory, to learn more about for whom these interventions might be more or less effective. To that end, we explored several characteristics including initial reading status, mindset, and problem behavior as potential moderators. Generally, these factors did not moderate the effects in the study conditions. We did note that higher initial growth mindset scores and phonological processing skills significantly predicted higher phonological processing outcomes at posttest for students in the reading intervention when compared to BAU. Higher initial reading comprehension scores along with lower behavior problems also predicted higher reading comprehension outcomes for students in the reading intervention when compared to BAU. However, it is likely that these significant moderation effects, though reliable, are quite small and not practically significant (e.g., Ridgeway, 2016) given that the effect was not found for other students receiving the same reading intervention.

Limitations and Conclusions

Despite the merits of the rigor of our study, there are important limitations. Our study was conducted in samples of schools with relatively high percentages of students from low socioeconomic backgrounds. Thus, our findings might not generalize to schools serving students from higher socioeconomic backgrounds, or to students with stronger reading skills. In addition, our team hired and trained the interventionists and so future research is needed to replicate such work within typical school practice where it might be more challenging to attain such high levels of fidelity.

In summary, results from the present study extend prior research that upper elementary students with severe reading difficulties may accelerate learning in some foundational reading skills over typical intervention, but, similar to previous research, overall reading achievement effects are weak. Results also indicate that a stand-alone growth mindset focused intervention, did not appear to improve reading outcomes, beyond reading intervention for fourth-grade struggling readers. These students likely require interventions with even greater intensity to accelerate their reading to allow them to read on grade level in order to read for content area instruction, and to be prepared for success in middle and high school. The current study focused on global mindset. Future research examining content-specific growth mindset by integrating mindset intervention specifically in reading intervention would provide important information to the field. Perhaps students with reading difficulties can benefit more from mindset instruction when it is more closely aligned with their specific area of difficulty.