The articles in this Special Topic issue present a range of assessment models and challenges for improving the identification and early intervention of students in need of additional supports. Although each article targets a unique aspect of student learning (learning behaviors, math skills, reading comprehension, behavioral functioning, and ratings of engaged and disruptive behavior), collectively they highlight the importance of assessment practices in effective problem solving.
In our commentary, we use prevention science as a framework for considering the contributions of the articles in this special topic with a particular focus on the role of assessment. A recent report from the National Research Council (NRC) and Institute of Medicine (IOM) (2009) attributed much of the progress in advancing knowledge about prevention of emotional and behavior problems over the past 2 decades to the relatively young field of prevention science. As an interdisciplinary field, prevention science provides a step-by-step model for solving public health problems, including educational underachievement. Specifically, prevention science is a systematic method for identifying, monitoring, and altering meaningful targets that have been demonstrated to be associated with critical youth outcomes. Accurate and efficient assessment tools are essential at each step of the prevention science research cycle.
Core Elements of Prevention Science
In their seminal paper, Kellam, Koretz, and Moscicki (1999) traced the development of prevention science to the integration of three related fields: epidemiology, life course development, and intervention trials technology. Epidemiology is the foundation for prevention science. It refers to the study of the distribution of disease or health-related behaviors/events and is a core element of any public health approach, like prevention science. The purpose of epidemiology is to provide real-time data about health events so as to identify intervention targets and inform intervention policies and practices.
Within the field of epidemiology, surveillance is the strategy for continuous, systematic collection, analysis, and interpretation of health-related processes over time to guide planning, implementation, and evaluation of practices (World Health Organization, 2012). Surveillance practices can help identify emerging public health crises, determine the impact of public health interventions, and provide ongoing information about population health. Below we discuss epidemiology and surveillance, life course development, and intervention trial technology as the bases for prevention science.
Epidemiology and Surveillance
Public health officials use surveillance data to monitor the prevalence and incidence of diseases across world populations. Surveillance is most widely understood as an element of public health approaches to somatic disease prevention. For instance, the general public may be aware that public health officials have ongoing surveillance systems for tracking infectious diseases such as the flu. These systems allow officials to monitor outbreaks, identity causes, and prevent the spread of diseases, but they can also be applied to select appropriate response strategies and then monitor the effects of interventions.
Surveillance methods have been extended to include public health approaches to preventing emotional and behavior problems. For instance, surveillance systems have been established for monitoring the prevalence and incidence of crimes, substance abuse, and mental disorders (Biglan, Mrazek, Carnine, & Flay, 2003). Although behavioral surveillance systems have tended to lag behind systems for more traditional somatic diseases, emerging technology has led to exciting advances in many of these systems (Biglan et al., 2003; Wagner et al., 2012). For instance, community-level violence prevention scientists now are able to use real-time crime reports to assess both the need for intervention and also the impact of tried interventions (Wagner et al., 2012). These data provide information about the type and location of crimes on a daily basis. As an example, several reports have evaluated the effects of CeaseFire Chicago (and other programs like Safe Streets Baltimore) on violence rates using daily crime data (Skogin et al., 2008; Wagner et al., 2012). The intervention is implemented in neighborhoods or small communities, so precise crime location data were used to document reductions in homicides and nonfatal shootings within specific boundaries of neighborhoods implementing versus those not implementing the intervention.
In addition to health behaviors, researchers have recently called for the surveillance of environments known to promote or interfere with health behaviors. For instance, Biglan, Flay, Embry and Sandler (2012) reviewed relevant literature and found that most major emotional and behavior problems stem from a similar set of environmental conditions, which can be prevented by increasing the prevalence of nurturing environments. Measuring and monitoring nurturing environments as well as aversive conditions can provide preventionists with information needed for effective interventions. Many tools for tracking toxic environments have already been developed. For instance, juvenile arrest rate, homicide rate, percentage of families in poverty, number of vacant buildings, can be aggregated within communities to indicate neighborhood risk for children (Bass & Lambert, 2004).
An excellent example of school-based research that fits in the surveillance category is the article by Volpe and Briesh (this issue) which extends the development of Direct Behavior Ratings (DBR). DBR was developed as a unique class of behavior assessment to provide a formative approach to measuring behavior for both screening and progress monitoring applications (Chafouleas, Riley-Tillman, & Christ, 2009; www.directbehaviorrating.com). In this article, the authors extend the development of DBR which to date has been primarily focused on the DBR Single Item Scale (DBR-SIS) to develop a DBR Multiple Item Scale (DBR-MIS). The continued development of this class of assessment should widen the application of DBR and allow for prevention oriented educational professionals to more accurately identify students in need of intervention and subsequently monitor the effectiveness of intervention.
Life Course Development
Prevention science integrated epidemiological methods with a life course perspective for understanding and monitoring social and behavior processes (Kellam et al., 1999; Kellam & Van Horn, 1997). Thus, it is a central concept in prevention science that human development is a critical context for understanding appropriate assessment and intervention targets (risk and protective factors), especially during key life transitional periods.
Seeing human development as the context for assessment and intervention during key life transitional periods is consistent with the concept of a behavioral cusp within the behavior analytic community (Cooper, Heron, & Heward, 2007). Behavior cusps are behaviors that bring the learner into contact with a new collection of contingencies once they are learned. Focusing on target behaviors that open up a new world of opportunity allows interventionists to maximize their impact. For example, reading at a fluent level allows a child to independently learn a vast amount of information in books. This construct is the basic idea behind targeting general outcome measures (GOMs) to provide ongoing data about risk and response to intervention. Another excellent example is assessing parenting behaviors across child development. This is critical for understanding potential behavioral dysfunctions to target for intervention that occur in early childhood and how developmental appropriate parenting contexts evolve through adolescence. For instance, research has shown that monitoring is a critical aspect of effective parenting during adolescence, a time when peers play a salient role in development (Patterson, Reid, & Dishion, 1992).
A life course epidemiological perspective is what led to the important advances in understanding developmental pathways to antisocial behaviors (Patterson et al., 1992). Patterson and colleagues described the developmental research that led to the important discovery of risk and protective factors at each life stage associated with antisocial behaviors. As their work documents, the pathway to serious antisocial behavior is set in motion during the toddler years with coercive parent-child interactions. Rejecting and nonsupportive responses from teachers at school entry exacerbate the problems of aggressive children (Arnold, Griffith, Ortiz, & Stowe, 1998). Mainstream peers are equally reluctant to interact with aggressive children, which precludes opportunities for them to learn the necessary emotion regulation, social problem-solving, and conflict resolution skills that are essential for integration into the mainstream peer group (see Reinke & Herman, 2002). Eventually, aggressive youth are likely to drift into a deviant peer group where antisocial behaviors and academic failure are reinforced (Patterson et al.,1992). Common co-occurring consequences of these patterns in adolescence include academic failure, depression, and criminal behaviors. The insights produced by this elegant line of research has clear implications for assessment and intervention strategies that can be implemented at each stage of development to interrupt these pathways.
Intervention Trials Technology
In addition to using surveillance data and contextualizing assessment data within a developmental perspective, prevention science relies on intervention trials technology to determine the causal processes involved in social and behavioral dysfunction and as the tool for evaluating whether interventions were effective (Kellam et al, 1999; Kellam & Langevin, 2003). The research cycles for intervention trials includes several stages that need not occur in a sequence but collectively yield a strong knowledge-base. Stages included the following: (1) defining the target problem and conceptual framework, (2) testing the conceptual framework, (3) designing and testing interventions intended to manipulate hypothesized causal processes, (4) extending successful interventions to be implemented in field trials, and (5) disseminating the findings. The prevention research cycle has clear implications for assessment as described below.
Stages of the Prevention Research Cycle
Defining and Detecting the Problem Behavior
A foundational concept in all problem solving models is the idea that it is critical to clearly define the problem before effective solutions can be identified. Prevention science adds to this the idea that problem solving is strengthened within the context of a sound conceptual model that specifies not only the targeted problem but also malleable causes of it (Kellam et al., 1999). These include risk and protective factors relevant to particular developmental periods. Further, each construct within the conceptual model must have measures that result in valid decisions and data that are reliable for the relevant populations in order to assess if the model is functioning as expected and to determine if the malleable constructs identified for intervention are altered as a result of the intervention (Kellam et al., 1999; Kellam & Langevin, 2003).
Screening and the Triangle
For convenience sake, we often talk about the three-tiered prevention model, and schools use these tiers to identify students in need of additional services. On the one hand, this tiered-triangle analogy is useful. Screening measures serve the purpose of identifying who is not responding to universal supports (Burns, Riley-Tillman, VanDerHeyden, 2012). On the other hand, in practice, the demarcations of who is in and out of each tier can be unclear. Of course, the three-tiered model is an oversimplification of reality. While servicing those in Tiers 1 and 3 is somewhat clear, Tier 2 can result in a massive catch-all category that runs the gamut from students with minor to significant needs.
All of the articles in the special issue have implications for defining target behaviors, including risk and protective factors. In the Shapiro and Gebhard (this issue) article, they determined that both curriculum-based measurement (CBM) and computer adapted testing (CAT) measures provided information about math competence. The CAT has the advantage of predicting future state achievement performance. Given the importance of these tests to local stakeholders in their own definitions of the problem, this evidence suggests CATs may be the preferred tool for defining math competence. Other elements that would be important in deciding on tools include the ease and cost of administration and as well as the tool’s sensitivity to repeated measurements and responsiveness to intervention.
On the other hand, Baxter and Ardoin (this issue) found that Maze assessments may not provide functional information about reading comprehension skills. Essentially this paper serves as a validation study and indicated that further work is needed toward finding a useful measure of reading comprehension skills. Such work is critical in identifying relevant and psychometrically sound assessment tools. Considering the impact of the development of CBM in relation to measuring reading fluency, the focus on developing a similar method to measuring reading comprehension is logical. Despite some success, there seem to be limitations to this approach to measuring reading comprehension (Paris, Carpenter, Paris, & Hamilton, 2005). One lesson to take from this line of research is that different target behaviors are likely to need unique approaches to capturing core elements.
Rikoon, McDermott, and Fantuzzo (this issue) represents a study focused on the evaluation of proximal risk and protective factors that may impact an outcome rather than the particular outcome trying to be altered. As the authors note, learning behaviors are associated with positive reading and math skill development. Although not articulated in their paper, presumably their conceptual model would identify learning behaviors as proximal causes of math and reading skills (or perhaps interaction terms with instruction). Thus, within a prevention research cycle, measures of learning behaviors could serve as tools for testing mediation models proposed by their conceptual framework. Ultimately, these tools could also be used if validated as measures of manipulation checks in intervention studies (e.g., if learning behaviors play a causal role in math and reading skills, interventions could be developed to improve learning behaviors). Intervention trials could then assess whether the intervention improved learning behaviors and in turn academic competencies.
Stability of Measurement
Methodological sophistication is a hallmark of prevention science. Because prevention science concerns itself with measurement of processes over the life course, the field will continually develop and refine methods for assessing and measuring processes over time (Kellam et al., 1999). For instance, growth mixture modeling allows researchers to capture the development of a process through repeated measurement. Rather than assuming a static time-point adequately captures a construct, these methods allow researchers to consider growth and change on processes through repeated measurements, which provides another perspective on what it means for a measure to be reliable.
Aside from the Volpe and Briesch article (this issue), the other papers in the special topic did not consider the issue of measurement stability (multi-stage, multi-level growth modeling). The papers treated each of their constructs as static processes. One implication of this approach is measurement error over time is left unexamined.
Theory, Risk/Protective Factors, and Intervention
Prevention science is focused on solving public health problems. Thus, ultimately measurement and theory are used to develop effective interventions that alter targeted behaviors. Although researchers cannot assign individuals to risk conditions (e.g., classrooms with ineffective instruction, poor parenting), prevention science researchers use experimental method to isolate causal processes by assigning individuals to interventions designed to reduce risk factors and increase protective factors. If these factors are indeed causally related to outcomes, altering these factors should produce predictable changes in outcomes.Kellam et al. (1999) referred to this process as “malleability through experimental manipulation” (p. 470).
Developing, Testing, and Refining Conceptual Models
A conceptual model is needed to define putative mechanisms believed to be involved in the targeted behavior outcome. Conceptual models are developed based on existing theories and evidence, and ongoing research then contributes to validating the model or refining it. Potential mediator and moderator variables are defined by the conceptual model. Proximal and distal outcomes are considered as well given that some manipulations may produce more immediate changes in behavior while other changes may be delayed. For instance, increasing effective classroom management practices may produce immediate or proximal effects on teacher behaviors and student time on task but its distal effects on student achievement may not be observed until much later.
Contextualism is a commonly used theoretical foundation for prevention science work (Biglan, 2004). A core tenet of contextualism is that identified targets should be malleable. This is vital if interventions are to be socially valid and impact societal problems. Contextualism contrasts with a mechanist approach which concerns itself with all variables regardless of whether they are potentially responsive to intervention (Biglan, 2004).
Although none of the papers in the special issue clearly articulates the conceptual framework guiding their work, each has elements that could be consistent with contextualism. For instance, abundant research suggests that reading and math skills can be improved. A key question for the CATs and Maze approach is whether the skills being measured by these methods are equally malleable as skills being tapped by other approaches (like CBM). Measures that tap behaviors or skills that can be altered are preferred in a prevention and contextualist framework. Measurement of static skills and behaviors is not functional from a prevention perspective (by definition static characteristics cannot be prevented) and are more in line with a common past criticism of school psychology assessment methods that could be characterized as admiring the problem. Similarly, although not studied by the authors, learning behaviors as conceptualized in that paper may indeed be teachable and capable of being altered. Making decisions about a construct’s malleability at the earliest stages of research is a critical step to determine whether time and effort needed to study the construct will be worth the investment based on likelihood of serving as a useful construct for improving public health.
From a contextualist perspective, learning behavior assessment (Rikoon et al., this issue) might be considered a protective factor that subsequent researchers may view as the target of intervention. In this model, the researchers might develop an intervention to promote learning behaviors, use the measure described in the paper to ensure the manipulation occurred, and then measure hypothesized proximal and distal outcomes of this manipulation. For instance, the researchers might hypothesize increasing learning behaviors would have a proximal effect on student attendance and engagement in learning as well as more distal effects on achievement.
Measuring Implementation Fidelity
A key consideration in prevention and implementation science is implementation fidelity. Measurement strategies to assess fidelity are essential to ensure the manipulation actually occurs. In addition, it is critical to develop both a wide variety of integrity measures, and more dynamic methods that consider what alterations are commonly made in the field (Power et al., 2006; Webster-Stratton, Reinke, Herman, & Newcomer, 2011), and if those alterations are in fact an improvement over core program. It must be understood that educational professionals may not adhere to a standard protocol but rather adapt the program to their classroom, and the process for natural adaptation should be researched and refined. A focus on understanding critical components or principles of evidence-based practice (EBP) so that they can be adapted within natural settings is exciting, but it will only be fully realized when we embrace that practitioners may have contextual knowledge to improve EBP within context. Therefore, the implementation process should be assessed with psychometrically sound measures of fidelity. Fidelity can then be evaluated as a mediator or moderator to intervention outcomes. Additionally, measuring and understanding how practitioners adapt EBP within varying contexts can further inform intervention development and implementation science.
Linking Assessment and Intervention
Data-based decision making has been a catch-phrase for applying a problem-solving approach in schools. The data-based decision framework is entirely compatible with the surveillance systems that guide public health decisions. As noted earlier, improvements in surveillance systems for behavior patterns and environments have occurred in recent years and corresponding improvements in integrating behavioral surveillance with interventions will occur. Consider the exciting advances that have occurred in allowing individuals to monitor their food consumption and activity levels as a tool for changing their health-related behavior (e.g., Shuger et al., 2011). For instance, one company has developed a tool for users to connect their scales to a system that communicates their weight each day to a group of peers and professionals (Freedman, 2012). In turn, this information not only gives the individual personalized feedback about their weight and health behaviors but also a supportive audience who can then reinforce and shape behavior change. In a similar manner, daily performance feedback has been used to improve teacher classroom management skills (Reinke, Lewis-Palmer, & Martins, 2007; Reinke, Lewis-Palmer, & Merrell, 2008). Technology is also leading to emerging sophisticated methods for measuring social networks within schools to provide a more nuanced understanding of peer influences on development and guidance for strategically arranging peer interactions (Kindermann & Gest, 2009).
The Science of Dissemination
One misrule sometimes applied to understanding prevention science is that the field does not concern itself with community partnership (Miller & Shinn, 2005; Wandersman, 2003). In his seminal papers, Kellam consistently wrote that field trials and community participant involvement were essential in all phases of the research cycle (Kellam et al., 1999; Kellam Van Horn, 1997; Kellam & Langevin, 2003). For instance, he noted that, “developmental epidemiologically based prevention research is built within the social and political structure of the population” (p. 477; Kellam et al., 1999). Moreover, “Broad ownership is critical for implementing rigorous research and for sustaining program fidelity” (p. 137; Kellam & Langevin, 2003).
Ultimately, prevention science is focused on changing the public health on a societal level. So while single trial effects are important in building a knowledge base, they are only useful to the extent that this knowledge is applied to population behaviors. Dissemination science is concerned with the methods and strategies needed to implement interventions found effective in settings outside the research context (NRC and IOM, 2009). Here the focus is on the skills and supports needed for community practitioners to implement the interventions to produce practical and meaningful change.
Prevention to dissemination science provides a context on which the future of educational research must focus. Complex system models (e.g. response to intervention or positive behavior supports and interventions) must combine a variety of evidence based programs and assessments in real world settings. As such, it is critical that educational professionals are also trained in evidence-based practices. While we have made a great deal of development in terms of the programs, more attention needs to be applied to the variety of assessment methods necessary. For example, a wider variety of surveillance methods will be required. The development of DBR as illustrated in the Volpe and Briesh article in this issue provides an excellent example of the continued development of prevention-oriented surveillance methods. As noted above, dissemination will also rely on effective dynamic methods of measuring implementation of both critical components and adaptations.
Summary and Conclusions
Prevention science provides a context for considering the quality and importance of emerging assessment systems. Measures that fit within the prevention research cycle are likely to prove more functional than those that do not. By considering the role of measures and their validation within the research cycle, researchers are more likely to develop a research agenda that is linked to public health priorities and positive student outcomes.
Acknowledgments
Preparation of this manuscript was supported in part by the Johns Hopkins Center for Prevention and Early Intervention under a grant funded by the National Institute of Mental Health (P30 MH08643) awarded to Dr. Nicholas Ialongo and by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A100342 to Dr. Wendy Reinke. The opinions expressed are those of the authors and do not represent views of the Institutes or the U.S. Department of Education.
References
- Biglan A, Mrazek PJ, Carnine D, Flay BR. The integration of research and practice in the prevention of youth problem behaviors. American Psychologist. 2003;58:433–440. doi: 10.1037/0003-066x.58.6-7.433. [DOI] [PubMed] [Google Scholar]
- Burns MK, Riley-Tillman TC, VanDerHeyden A. Response To Intervention Applications Volume 1: Academic and Behavioral Interventions. New York, NY: The Guilford Press; (In Press) [Google Scholar]
- Chafouleas SM, Riley-Tillman TC, Christ TJ. Direct Behavior Rating (DBR): An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention. 2009;34:195–200. [Google Scholar]
- Cooper JO, Heron TE, Heward WL. Applied Behavior Analysis. Upper Saddle River, NJ: Pearson Education, Inc.; 2007. [Google Scholar]
- Biglan A, Flay B, Embry D, Sandler I. The critical role of nurturing environments in promoting human well-being. American Psychologist. 2012;67:257–271. doi: 10.1037/a0026796. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Freedman DH. The perfected self. The Atlantic. 2012 Jun [Google Scholar]
- Kindermann T, Gest SD. Assessment of the peer group: Identifying naturally occurring social networks and capturing their effects. In: Rubin K, Bukowski W, Laursen B, editors. Handbook of Peer Interactions, Relationships and Groups. New York: Guilford Press; 2009. pp. 100–117. [Google Scholar]
- Kellam S, Langevin D. A framework for understanding “evidence” in prevention research and programs. Prevention Science. 2003;4:137–153. doi: 10.1023/a:1024693321963. [DOI] [PubMed] [Google Scholar]
- Kellam SG, Koretz D, Moscicki E. Core elements of developmental epidemiologically based prevention research. American Journal of Community Psychology, 1999;27:463–483. doi: 10.1023/A:1022129127298. [DOI] [PubMed] [Google Scholar]
- Kellam SG, Van Horn V. Life course development, community epidemiology, and preventive trials: A scientific structure for prevention research. American Journal of Community Psychology. 1997;25:177–188. doi: 10.1023/a:1024610211625. [DOI] [PubMed] [Google Scholar]
- Miller RL, Shinn M. Learning from communities: Overcoming difficulties in dissemination of prevention and promotion efforts. American Journal of Community Psychology. 2005;35:165–177. doi: 10.1007/s10464-005-3395-1. [DOI] [PubMed] [Google Scholar]
- National Research Council and Institute of Medicine. Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: National Academies Press; 2009. [PubMed] [Google Scholar]
- Patterson G, Reid J, Dishion T. Antisocial boys. Eugene, OR: Castalia Publishing Company; 1992. [Google Scholar]
- Power TJ, Blom-Hoffman J, Clark AT, Riley-Tillman TC, Kelleher C, Manz PH. Reconceptualizing treatment integrity: A partnership-based framework for linking research with practice. Psychology in the Schools. 2005;42:495–507. [Google Scholar]
- Reinke WM, Herman KC. Creating school environments that deter antisocial behaviors in youth. Psychology in the Schools. 2002;39:549–559. [Google Scholar]
- Reinke WM, Lewis-Palmer T, Martin E. The effect of visual performance feedback on teacher behavior-specific praise. Behavior Modification. 2007;31:247–263. doi: 10.1177/0145445506288967. [DOI] [PubMed] [Google Scholar]
- Reinke WM, Herman KC, Stormont M, Brooks C, Darney D. Training the next generation of school professionals to be prevention scientists: The Missouri Prevention Center Model. Psychology in the Schools. 2010;47:101–110. [Google Scholar]
- Reinke WM, Lewis-Palmer T, Merrell K. The Classroom Check-up: A classwide consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review. 2008;37:315–332. [PMC free article] [PubMed] [Google Scholar]
- Skogan WG, Hartnett SM, Bump N, Dubois J. Evaluation of CeaseFire Chicago. Northwestern: Institute for Policy Research; 2008. [Retrieved on June 14, 2012]. from, http://www.ipr.northwestern.edu/publications/ceasefire_papers/mainreport.pdf. [Google Scholar]
- Shuger S, Barry V, Sui X, McClain A, Hand G, Wilcox S, Meriwether R, Hardin J, Blair S. Electronic feedback in a diet- and physical activity-based lifestyle intervention for weight loss: a randomized controlled trial. International Journal of Behavioral Nutrition and Physical Activity. 2011;8:41–49. doi: 10.1186/1479-5868-8-41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wagner DW, Whitehill JM, Vernick JS, Parker EM. Evaluation of Baltimore’s Safe Streets Program. Baltimore, MD: Johns Hopkins School of Public Health; 2012. [Retrieved on June 14, 2012]. from, http://www.rwjf.org/files/research/73802.safestreetsevaluationfinal.pdf. [Google Scholar]
- Wandersman A. Community Science: Bridging the gap between science and practice with community-centered models. American Journal of Community Psychology. 2003;31:227–237. doi: 10.1023/a:1023954503247. [DOI] [PubMed] [Google Scholar]
- Webster-Stratton C, Reinke WM, Herman KC, Newcomer L. The Incredible Years Teacher Classroom Management Training: The Methods and Principles that Support Fidelity of Training Delivery. School Psychology Review. 2011;40:509–529. [Google Scholar]
- World Health Organization. [Retrieved on June 14, 2013];Public health surveillance. 2012 from http://www.who.int/topics/public_health_surveillance/en/