Advanced (Measurement) Applications of Curriculum-Based Measurement in Reading

Abstract

The purpose of this article is to provide a commentary on the current state of several measurement issues pertaining to curriculum-based measures of reading (R-CBM¹). We begin by providing an overview of the utility of R-CBM, followed by a presentation of five specific measurements considerations: 1) the reliability of R-CBM oral reading fluency, 2) issues pertaining to form effects, 3) the generalizability of scores from R-CBM, 4) measurement error, and 5) linearity of growth in R-CBM. We then conclude with a presentation of the purpose for this issue and broadly introduce the articles in the special issue. Because oral reading fluency is one of the most common measures of R-CBM, much of the review is focused on this particular type of assessment; however, the issues presented extend to other assessments of R-CBM.

Oral reading fluency (ORF) is one of many curriculum-based measures of reading (RCBM) that serves as an indicator of overall reading achievement (Fuchs, Fuchs, Hosp, & Jenkins, 2001). Research has demonstrated the importance of fluently reading connected text as a theoretical construct (Hudson, Pullen, Lane, & Torgesen, 2009; LaBerge & Samuels, 1974) and scores from fluency measures have been validated as proxies for measuring reading comprehension (Buck & Torgesen, 2003; Ridel, 2007; Roehrig et al., 2008; Shapiro, Solari, & Petscher, 2008). As a result, measures of ORF have widespread utility in their capability as screeners of future reading comprehension difficulties, and as tools to monitor student progress in overall reading skills. Along with a decade of attention to accountability, the recent spread of Response to Intervention (RtI) models as a means of identifying and monitoring students at risk for reading failure has made R-CBM and ORF in particular more popular than ever (Cummings, Atkins, Allison, & Cole, 2008).

Whereas ORF measures used to be idiographic, and truly based in local curricula (e.g., Hintze, Shapiro, Conte, & Basile, 1997; Shinn, Gleason, & Tindal, 1989), a number of standardized measurement systems with common sets of reading passages have since been developed; for example, AIMSweb, DIBELS, easyCBM, and Edcheckup are all widely used RCBM measurement systems that include a measure of ORF. Although the procedures, prompts, and passages can vary among measurement systems, certain practices have become standard across systems. Typical R-CBM scoring conventions include (a) a count of the total words read correctly, (b) mispronunciations, substitutions, omissions, and transpositions are counted as errors, (c) inserted or repeated words are ignored, (d) students are allowed to “self-correct” a previously mispronounced word if they do so within three seconds, and (e) if students hesitate or struggle with a word for three seconds, an examiner provides the word but marks it as an error on the test form (Hosp, Hosp, & Howell, 2007; Stecker et al., 2005). In addition, most systems offer schools benchmark passages for occasional status checks (i.e., screening purposes) and for more frequent checks of response to intervention (i.e., progress monitoring). R-CBM data may be aggregated at the classroom, grade, or school level to provide quarterly indicators of school health that can be used to allocate resources (Cummings, et al., 2008).

Regardless of the system used, R-CBM research across the last 25 years has focused on investigating (a) the technical features of the static score (i.e., baseline or aggregate score(s)), (b) the technical features of the slope (i.e., time series performance), and (c) the instructional utility of the measure (Deno, 2003; Fuchs, 2004). However, the increasingly widespread and novel use of ORF for higher stakes accountability decisions has brought closer scrutiny to an expanded set of technical adequacy and validity information for ORF. Many of these concerns hinge on psychometric issues, such as reliability and error of measurement for estimates of students' status and growth.

Reliability of R-CBM ORF

Generally speaking, reliability estimates for R-CBM ORF measures are uniformly high. For example, when estimated using three or more passages administered concurrently, alternate form reliability for a single passage is usually estimated at .80 and above and for the set of passages is approximately .90 and above (Baker et al., 2008; Biancarosa, Bryk, & Dexter, 2010; Howe & Shinn, 2002; Roberts, Good, & Corcoran, 2005). As a result, and perhaps due to historical reasons (e.g., Marston, 1989) most, though not all, R-CBM systems take the median or mean score from three passages administered concurrently. In a 2011 review of National Center for Response to Intervention (NCRTI) website, the authors identified 11 unique R-CBM screening tools. Of those, six required that two or more reading passages to be administered for screening assessment purposes. For progress-monitoring purposes, five R-CBM tools required the administration of multiple passages per assessment period. Although the administration of multiple passages per assessment period improves the static reliability of the average score, it is not yet known if this practice sufficiently addresses some of the other concerns about ORF validity.

Form Effects and R-CBM ORF

As is evident from the lower alternate form reliability for single passage test administrations compared to multiple passage administrations, reliability suffers due to passage-specific differences that cannot be eliminated entirely through careful text authoring (e.g., using readability formulae) nor the typical ORF measure development process (Ardoin, Suldo, Witt, Aldrich, & McDonald, 2005; Ardoin, Williams, Christ, Klubnik, & Wellborn, 2010; Betts, Pickart, & Heistad, 2009; Christ & Ardoin, 2009). Indeed, recent research has demonstrated that from a progress-monitoring perspective, these passage-specific, or form, effects can have a profound influence on the trajectory of monitored students (Ardoin & Christ, 2009; Francis, Santi, Barr, Fletcher, Varisco et al., 2008; Hintze & Christ, 2004) as well as diagnostic accuracy (Petscher & Kim, 2011). As a result, there are now calls for equating of ORF probe passages. In fact, researchers have begun to use equated ORF scores in their studies (e.g., Barth, Stuebing, Fletcher, Cirino, Francis et al., 2012; Denton, Barth, Fletcher, Wexler, Vaughn et al., 2011), but the trade-offs among various methods of equating and the costs associated with them have only begun to be investigated (Albano & Rodriguez, 2012; Christ & Ardoin, 2009; Francis et al., 2008; Griffiths, VanDerHeyden, Skokut, & Lilles, 2009).

Generalizability and Dependability of R-CBM ORF

Studies examining ORF progress-monitoring measures from a Generalizability (G) theory perspective have demonstrated that although relative generalizability is very good, absolute generalizability is much lower. The advantage of G theory studies is that they account for multiple sources of unreliability (or error) simultaneously and also allow for projection of reliability under different circumstances. In the first such G theory study (Hintze, Owen, Shapiro, & Daly, 2000), researchers accounted for variance due to the measurement materials (i.e., passages based on a long-term versus a short-term goal level of difficulty), the grade level of students (second- through fifth-grade), and assessment occasions (twice per week over either eight or ten weeks) and found that the reliability of ORF for assessing growth for an individual student was .90 when using long-term goal materials (i.e., end of grade level) and .80 when using both instructional level and long-term goal materials. Projecting the reliability when only a single passage source is used for two grade levels, dependability dropped to .82 when using long-term goal materials and .67 when using both instructional level and long-term goal materials (Hintze et al., 2000). To date, the only other G theory study obtained reliability of slope estimates using 20 grade-level passages (range of coefficients =.81 - .97; Poncy, Skinner, & Axtell, 2005). In contrast to Hintze et al., Poncy and colleagues (2005) examined the influence of passages alone on reliability. Together these studies suggest that alternate form reliability may overestimate true reliability because traditional psychometric calculations for reliability (i.e., correlation coefficients) do not take into account the possibility of absolute score differences across forms (e.g., Albano & Rodriguez, 2012).

Error of Measurement for ORF

Related to the issue of reliability, is the consideration of the standard error of measurement (SEM) associated with ORF assessment. SEM provides a transparent metric for evaluating the degree of measurement error associated with a student's particular raw score. Poncy and colleagues (2005) offered the first peer-reviewed publication to address the SEM associated with R-CBM, and these results have since been replicated by other researchers (e.g., Christ & Ardoin, 2006; Christ & Silberglitt, 2007). The SEM associated with a single ORF score has been found to range from five to 15 wcpm, with a median SEM of 10 wcpm (Christ & Silberglitt, 2007). Even when passage effects are controlled through a Generlizability (G) theory approach, the SEM can range from three or four to as many as 12 wcpm (Poncy et al., 2005). Furthermore, a line of research by Christ and colleagues has demonstrated a profound impact of SEM on estimates of slope and, ultimately, decisions about adequate student progress (Christ, 2006; Ardoin & Christ, 2009; Christ, Zopluoglu, Long, & Manghen, 2012). The work of Christ and colleagues reveals that when the SEM is small (i.e., < 6), slope estimates show little error as long as ORF is measured six or more times. However, when the SEM is larger, variation in slope estimates can also be quite large; with unsatisfactory psychometric properties until at least 10 or more ORF measures have accumulated. It is worth noting that both (a) small SEMs (< 6) are rarely observed in practice and (b) that waiting for a satisfactory number of data points to address unreliability in a student data set has the potential to either delay or misalign intervention support with student need. Ensuring that R-CBM is validated for the intended use of important educational decisions requires a renewed focus on methods used to evaluate test administration.

Linearity of R-CBM ORF Growth

Adding to the complexity of measurement modeling, is the problem that change in ORF over time may not follow a linear trajectory nor be the same for readers of all abilities, which most technical manuals and research, including the aforementioned work, assume (see also, Speece & Ritchey, 2005). Research over the past decade suggests that ORF growth may in fact be curvilinear with faster growth observed earlier in the academic year and slower growth later (e.g., Al Otaiba, et al., 2009; Christ, Silberglitt, Yeo, & Cormier, 2010; Crowe, Connor, & Petscher, 2009; Deno, Fuchs, Marston, & Shin, 2001; Logan & Petscher, 2010). In addition, ORF slopes vary depending on the skill level of readers at the beginning of the school year, receipt of special education services, the varying type and quality of instructional support that is available to students across the school year, and a number of other individual characteristics (Puranik, Petscher, Al Otaiba, Catts, & Lonigan, 2008; Silberglitt & Hintze, 2007; Wang, Algozzine, Ma, & Porfeli, 2011). The results of this line of research imply that applying the same standard for expected growth, and a linear standard at that, may be a flawed practice. Considering that research has demonstrated that growth in ORF uniquely explains individual differences in reading comprehension (Kim, Petscher, Schatschneider, & Foorman, 2010), it is critically important that the functional form of growth is correctly specified.

Purpose of the Special Issue

Deno, Mirkin, and Chiang (1982) identified R-CBM as the critical CBM task and a robust indicator of overall reading competence, and four decades of research have demonstrated R-CBM's value for providing reliable and valid screening and monitoring information for first grade and beyond (e.g., Wayman, Wallace, Wiley, Tichá, & Espin; 2007). However, research on R-CBM for new purposes is not yet complete. The purpose of this special issue is to bring together a set of articles that highlight at least some of the novel measurement issues facing RCBM researchers in the 21st century. Issues of form equivalency, methodologies for estimating linear and non-linear change over time, and using R-CBM tools as indicators of teacher effects are the three key areas on which we will focus.

In the first article, Stoolmiller, Biancarosa, and Fien explore passage equivalence from a robust and novel measurement perspective. The authors apply a set of linear and non-linear factor analytic models to scores from oral reading fluency passages, and discuss the appropriate methods for equating given the type of measurement model applied. Cummings, Park, and Bauer follow this paper by examining the persistence of form effects in the DIBELS Next Oral Reading Fluency measures in first through sixth grades; evaluating the value of different equating strategies to control for the differences in average fluency rates across the passages. The third article in this series by Christ, Monaghen, Zopluoglu, and Van Norman evaluate growth and diagnostic accuracy of R-CBM scores using pre-post assessment data ranging from six week to 20 week intervals. The fourth article, by Kamata, Nese, Patarapichayathan, and Lai, use scores from passage reading fluency to demonstrate how piecewise growth models and piecewise growth mixture models may be applied to data where only three time points exist per individual. The fifth article, by Hart, Taylor, and Schatschneider, presents a unique methodology using ORF scores to demonstrate how classrooms may causally affect later performance in fluency tasks. Though this design has traditionally been used in medical research, this paper represents the first application in education to account for school effects above the student level.

Conclusion

The articles represented here begin to delve more deeply into recent calls to comprehensively explore issues of the reliability and validity of fluency scores. Because RCBM, especially oral reading fluency, will continue to be used by practitioners and researchers, it behooves us to continually study methodologies and practices for improving the psychometric quality of the scores from such assessments. Advances in educational measurement have been occurring at a rapid pace, yet even some relatively older innovations developed in the last ten to fifteen years [e.g., item response models with time parameters (Verhelst, Verstralen, & Jansen, 1997), latent change scores models (McArdle, 2001)], which hold some great promise for understanding individual differences, have seen little application in the field of assessment research. As accessible software packages and programs are developed for educational researchers, the feasibility of implementing such models is greatly enhanced. The potential promise of these advances in measurement is the ability to estimate more reliable student scores and remove multiple sources of bias. Subsequently, this augments the capacity to use R-CBM scores to correctly identify students' strengths and weaknesses, the need for additional services or accommodations, as well as monitoring progress.