The Feasibility of First Step to Success with Preschoolers

Abstract

The primary purpose of this study was to examine feasibility of the preschool version of the First Step to Success (FSS) intervention. Toward this end, the following four research questions were addressed: (1) To what extent was the intervention implemented with integrity? (2) To what extent do teachers and parents perceive the intervention to be socially valid? (3) To what extent were teachers and parents satisfied with the intervention? and (4) To what extent was the intervention effective in reducing problem behavior and improving social skills? Twelve students participated in the study. Treatment integrity, social validity, and satisfaction results were analyzed at the aggregate level, and a reliable change index was calculated at the case level for primary outcome measures to assess the potential efficacy of the intervention. Fidelity data suggest the preschool version of the intervention can be implemented with acceptable integrity by coaches and teachers in preschool settings. Social validity outcomes suggest parents’ perceptions of the program’s goals, procedures, and outcomes were extremely favorable, and social validity from the teacher perspective was acceptable. The results provide initial evidence that participating in the preschool version of the FSS intervention improves children’s social skills and decreases problem behavior.

In the past 20 years, more and more professionals have expressed concern about the number of children entering school who have been exposed to abuse, neglect, and harsh or inconsistent parenting practices. These risks frequently result in a lack of school readiness and the display of disruptive behavior patterns that educators find highly objectionable (Evans, Weist, & Serpell, 2007). Upon entering school, all students are required to make two essential social-behavioral adjustments; that is, they must satisfactorily negotiate the demands of teachers, who control instructional settings, and they must learn to cope with peer group dynamics that play out primarily within free-play settings. Failure in either of these critically important adjustment areas is problematic, but failure in both puts a child’s school career and, in some cases, his or her life chances at risk.

Within the past decade, many preschool and early childhood care settings have been overwhelmed by the increasing numbers of children who are unable to perform satisfactorily in either peer- or teacher-related domains (McCabe, Hernandez, Lara, & Brooks-Gunn, 2000; Qi & Kaiser, 2003). These students are unresponsive to the minimal but necessary demands of schooling (for example, listening, sharing, cooperating, accepting adult direction, focusing on assigned tasks, and so forth), as evidenced by national survey results indicating that preschool-age children are expelled three times the rate of K–12 students (Gilliam, 2005). In addition, they tend to display challenging behavior patterns at a much higher rate than normal, which severely stresses the management skills of most teachers (Karr-Morse & Wiley, 1997). Powell, Fixsen, Dunlap, Smith, and Fox (2007), for example, reported that the prevalence of challenging behavior is on the rise generally within preschool settings, with teachers citing difficulties involving 10 percent to 15 percent of their students.

Considerable progress has been made over the past several decades in developing and disseminating evidence-based interventions for children in preschool and the primary elementary grades designed to promote child social—emotional development (see Detrich, Keyworth, & States, 2007; Hoagwood, 2004; Hoagwood et al., 2007). A systematic review of the early childhood literature by Dunlap and Conroy (2003) found the following four general categories of behavioral interventions that have replicated bodies of scientific evidence supporting their use: (1) positive behavior support, (2) applied behavior analysis interventions to promote social interaction, (3) classroom prevention practices, and (4) social—emotional learning programs. The last category, social—emotional learning programs, refers to comprehensive, manualized interventions designed to promote school success that generally focus on friendship skills, emotional recognition, problem solving, and a variety of social skills. Finally, Joseph and Strain (2003) reviewed 10 curriculums designed to promote the social—emotional development of young children using nine criteria. Only two of these reviewed programs—First Step to Success (FSS) (Walker et al., 1998) and The Incredible Years: Dinosaur School (Webster-Stratton, Reid, & Hammond, 2001) received a high confidence rating. FSS failed to meet only the acceptability of interventions and replications across settings criteria. Since publication of this review, the developers of FSS have addressed these remaining two criteria via systematic research reported in a range of published studies (see Walker, Severson, & Seeley, 2010). In addition, the FSS intervention has been adapted to the developmental level of preschoolers and to fit within the context of early education settings.

Studies that attend to treatment integrity, social validity, and satisfaction are particularly important in the early stages of research for new interventions because they have implications for the feasibility of the intervention being applied within different contexts (for example, by school personnel rather than research staff, with fewer resources). However, a comprehensive review of positive behavioral intervention research with young children having challenging behavior, conducted by Conroy, Dunlap, Clarke, and Alter (2005), found that only 26 percent of the studies conducted between 1984 and 2003 reported any social validity measures. Social validity has been described by Wolf (1978) as the extent to which the participants perceive the goals of the intervention as important, the intervention procedures as acceptable, and the intervention’s effects as meaningful. Although there are many conceptualizations of treatment integrity, Hagermoser Sanetti, and Kratochwill (2009) suggested the dimensions proposed by Dane and Schneider (1998) are particularly strong, both because they are validated by other models and because the dimensions—adherence, exposure (that is, dosage), quality, participant responsiveness, and program differentiation—can be assessed independently and differentially predict student outcomes (Dusenbury, Brannigan, Hansen, Walsh, & Falco, 2005; Hirschstein, Edstrom, Frey, Snell, & MacKenzie, 2007). This comprehensive undemanding of treatment integrity is imperative to our ability to infer intervention effectiveness and related to social validity insofar as interventions that are socially valid are more likely to be implemented with integrity, particularly when applied within the context of educational settings (Sheridan, Swanger-Gagne, Welch, Kwon, & Garbacz, 2009).

To date, no evaluations of feasibility, efficacy, or effectiveness have been published on the preschool version of the FSS intervention. This article describes the specific adaptations made to the FSS intervention supporting its use in early childhood settings and examines feasibility through an assessment of treatment integrity, social validity, and satisfaction. In addition, outcome data are evaluated to determine the potential efficacy of the FSS program.

METHOD

In this section, the components of the FSS intervention, as well as adaptations for preschool settings, are described. Next, the research questions addressed, implementation sites, coach recruitment and training procedures, participant identification procedures, and study measures are detailed.

FSS

FSS is an early intervention program designed for at-risk primary level, elementary school children who show clear signs of emerging externalizing behavior problems (for example, aggression toward others, oppositional-defiant behavior, tantrumming, rule infractions, escalating confrontations with peers and adults, and so forth) (Walker et al., 1997). The at-risk child is the primary focus of the intervention; however, teachers, peers, and parents or other caregivers participate in the intervention as implementation agents under the direction and supervision of a school-based behavioral coach, frequently a school social worker, who has overall responsibility for coordinating the intervention. The FSS intervention requires two to three months, from start to finish, per application, and is applied to only one child at a time in a regular classroom setting. The program was developed through a four-year, federal grant (1992 to 1996) to Hill Walker and associates from the Office of Special Education Programs of the U.S. Department of Education.

FSS consists of three modules designed to be applied in concert with each other: (1) universal screening (Walker, Severson, & Feil, 1995), (2) the school module (Hops & Walker, 1988), and (3) the home module. The three modules of FSS are based on extensive research on school and home intervention procedures with children having challenging behavior and over a decade of work related to the universal, early screening of children at risk of school failure (see McCord, 1993; Patterson, Reid, & Dishion, 1992; Walker et ah, 1988).

School Module (CLASS)

The school intervention module of FSS is an adapted version of the Contingencies for Learning Academic and Social Skills (CLASS) program developed by Hops and Walker (1988). CLASS is divided into three successive phases: coach, teacher, and maintenance. The behavioral coach phase (program days 1 to 5) is the responsibility of an adult, often a school social worker or early childhood interventionist, who coordinates the implementation process. The role requires someone who can directly implement the program for brief portions of the school day and monitor, supervise, and support participating teachers as they assume control of the program. The CLASS program begins with two, 20-minute periods daily and is eventually extended to the entire school day. Initially, the coach, in close proximity to the target child, monitors her or his classroom behavior using a red and green card on which one point is awarded every 30 seconds. If the child’s behavior is appropriate when the point award interval occurs, the point goes on the green side of the card. To meet the criterion, 80 percent or more of the available points during the 20-minute period have to be awarded. A brief, interactive reward activity involving the target child and peen is made available immediately following the intervention period. If the reward criterion is met, the child also earns an interactive, play-based home reward designed to both reinforce the child’s behavior and enhance the child—parent relationship.

Over the course of the program, use of the red/green card is faded out, ideally completely by program day 20, and the interval in which points and praise can be earned is gradually extended. In the later stages of the program, the target student works in blocks of multiple days to earn class and home rewards of higher magnitude.

The teacher phase (program days 6 to 20) is operated by the classroom teacher in whose room the CLASS program is initially implemented. The teacher assumes control of the program’s operation on program day six but with close supervision and support provided by the behavioral coach. The behavioral coach provides monitoring and technical assistance as needed for the regular teacher throughout the remainder of the teacher phase. Teacher phase implementation tasks include the following: operating the program daily, awarding praise and points according to program guidelines and contingent on child performance, supervising delivery of group activity and school rewards, and communicating with parents on a regular basis regarding the target child’s performance. The teacher works closely with the behavioral coach, child, parents, and peers throughout the total implementation period.

The maintenance phase of the CLASS program lasts from program day 21 to 30, after which the school intervention is terminated. In this final program phase, the target child is rewarded primarily with praise and expressions of approval and recognition from the teacher at school and the parents at home. An attempt is made during this phase to reduce the child’s dependence on the program by substituting adult praise for points, reducing the amount of daily feedback given and making occasional rewards available contingent on exemplary performance.

Home Module (homeBase)

The homeBase component of FSS consists of a series of six lessons designed to enable parents and caregivers to build child competencies and skills in six areas that affect school adjustment and performance. The six target skills that parents are asked to teach their children are as follows: (1) sharing school, (2) cooperation, (3) limit setting, (4) problem solving, (5) friendship making, and (6) developing confidence. HomeBase contains lessons, instructional guidelines, and parent-child games and activities for directly teaching these skills. HomeBase requires six weeks for implementation and begins after the target child has completed program day 10 of the CLASS program.

The coach visits the parents’ home on a weekly basis and conducts the homeBase lessons in that setting. Following each session, materials are left with the parents that facilitate daily review and practice of each skill with the target child. The homeBase lessons require approximately one hour each. Parents are encouraged to work with their child 10 to 15 minutes daily and to focus on practicing the homeBase skills being taught.

An important, shared goal of homeBase is to build a strong, positive link between home and school. HomeBase is designed to strengthen parenting skills in developing child competence in key performance areas related to school success. Parents and caregivers are enlisted as partners, with the school, in helping the child get off to the best possible start in his or her school career. Its ultimate goal is to unite educators and parents-caregivers in helping vulnerable children experience early school success.

HomeBase content is based on over 25 years of research at the Oregon Social Learning Center (OSLC) involving hundreds of families who have contributed to current knowledge of the family-based factors related to children’s competent social adjustment (see Patterson, 1982; Patterson et al., 1992). The approach used in teaching parents how to improve their child’s school success in homeBase reflects numerous OSLC clinical trials and research efforts to study the processes inherent in family based, behavior change processes (Dishion, Patterson, & Kavanagh, 1992; Patterson, 1982). It also stresses the importance of developing a collaborative relationship with parents and tailoring the delivery and implementation of the target skills to meet the family’s existing skill level(s) in applying them. The OSLC knowledge base on parent training and intervention is derived from families of diverse socioeconomic conditions and social and emotional resources.

FSS Preschool Adaptations

As noted, the FSS early intervention program was developed originally for application with behaviorally challenged students enrolled in the primary grades. The adaptations of FSS that were judged necessary for developing the preschool version were dictated by the following three factors: (1) differences in the nature and dynamics of preschool and primary grade settings, (2) skill level differentials favoring primary grade teachers, and (3) developmental differences in maturation between preschoolers and primary grade students. A description of these adaptations is presented in Table 1.

Table 1.

Preschool Adaptations to the First Step to Success Intervention

Adaptation	Description and Rationale
Differences in preschool and primary settings	In general, preschool classrooms do not have universal behavioral principles in place (for example, defined clear expectations, directly taught behavioral expectations, positive reinforcement, and clear consequences for unacceptable behaviors). When these universal principles are not in place, it is very difficult to successfully change the behavior of a few students with challenging behaviors. The preschool adaptation includes staff development training sessions for all the participating FSS teachers and assistants to establish a positive, proactive environment for all students.
Skill differentials between preschool and primary grade level teachers	As a rule, primary grade teachers are better trained in both instructional and behavior management strategies than preschool teachers. Furthermore, certification standards for the former are more rigorous than for the latter. We have found considerable skill differentials between these two groups of teachers that can mediate the efficacy of the FSS program’s application. As a result, we find it useful to supplement FSS preschool programs with group training in behavior management strategies whenever possible. Our experience is that, in general, preschool teachers were quite surprised that simple expectations such as hanging up coats, walking into the classroom, asking for help appropriately, sitting in a circle, or putting away toys had to be specifically taught and reinforced until automatic.
Accommodating preschool and primary maturational differences	The preschool years have often been called the “magic years” because they are a time of significant changes within brain development which, in turn, influence changes in cognitive, language, and social skills. Preschool children’s developmental skills are emerging, but this is a unique developmental phase wherein they lack the skills mastery of those in the primary grades. Preschool children tend to rely on information derived from their senses, so it is easier for them to understand new learning when they can see, touch, or hear it. They learn better from an experiential rather than a “sit still and listen” approach. They need concrete directions and cannot follow complex, multistep directions such as “clean your room.” Memory is often context specific, so they may not generalize rules from one situation to another. For example, knowing how to put away toys at home may not generalize to school. As a result, they may get anxious when they do not know what the adult expectations are, often resulting in overactive behavior. They learn well from routines and from environments that explicitly indicate the required expectations (for example, where to hang coats, where to engage in pretend play), from clear examples, and from imitation of peers. Preschool children may be egocentric, and have difficulty taking another person’s point of view, but they can learn well when adults use teachable moments to help them understand.
Specific preschool FSS adaptations	During feasibility testing of the FSS intervention in preschool settings, the following adaptations were made in the program and implementation procedures to address the previously mentioned developmental differences. These features and changes were built into the final version of the preschool FSS program to increase its efficacy with preschool target populations. • The coach role-played with the child each day before the implementation session. • The coach phase was extended from six to 10 days. • If the Head Start program was an all-day program, the intervention was conducted twice a day for about 10 minutes during the coach phase. • If the Head Start program was half a day, the intervention was implemented once a day. • The coach problem-solved more with the child during the intervention than is recommended with the regular program. • For several students, individual as well as class rewards are implemented for meeting criteria during the intervention. • During the first few days of the teacher phase, the coach was in very close contact with the teacher and the assistant. The coach was often in the room for several days to monitor the program and provide feedback. • The coach was also encouraged to write e-mail messages to the teacher’s supervisor to communicate positive changes. • To remind the adults in the classroom to notice the target child doing the right thing, a “Green Button” pin was created. When the teacher phase started, the target student was present with a “Green Smiley-Face Button.” Each day, the child would wear this button and leave it at school each afternoon. The green button helped the adults to notice the target student and praise the child when things were going well. In some cases, the target student would select another student each day who had been especially kind and helpful and would present that student with a green button. Eventually the entire class could be part of the “Green Button” club!
	The following adaptations were made during the homeBase component: • Coaches were encouraged to conduct the homeBase meetings while the child was present. The coach modeled positive interactions with the child and demonstrated for parents how to do the homeBase activities. • If for some reason the child could not be present, the coach was encouraged to role-play with the parent as if the parent were the child and the coach were the parent. We found that often parents do not know how to play games with their children. This type of role play teaches parents in a respectful, nonembarrassing way how to positively interact and play with their children.

Note: FSS = First Step to Success.

Research Questions

The primary purpose of this study was to examine feasibility of the intervention by investigating its treatment integrity, social validity, and satisfaction. The potential efficacy of the preschool version of FSS was also assessed by examining student outcomes (parent- and teacher-reported problem behavior and social skills). Specifically, this study addressed the following four research questions: (1) To what extent was the intervention implemented with integrity? (2) To what extent do teachers and parents perceive the intervention to be socially valid? (3) To what extent were teachers and parents satisfied with the intervention? and (4) To what extent was the intervention effective in reducing problem behavior and improving social skills?

Study Sites

This study was conducted simultaneously within two sites—one in Salem, Oregon, and the other in Louisville, Kentucky. The Community Action Agency (CAA), the host agency for Oregon, serves preschool and Head Start populations within Oregon’s Marion and Polk counties. CAA Head Start has 23 classrooms serving 820 children and their families across 17 sites. Of the children enrolled in Head Start, 53 percent were Hispanic, 40 percent had Spanish as the family’s dominant language, and 25 percent were white.

The early childhood program in Louisville is housed within the Jefferson County Public School (JCPS) system, which serves approximately 100,000 students. The system’s early childhood program contains Head Start/Early Start, prekindergarten, tuition-based preschool, and early childhood special services. Through these programming options, JCPS serves over 5,000 children in approximately 250 classrooms in 53 buildings. Of the children enrolled in the early childhood program, approximately 20 percent were white, 10 percent were Latino, and 65 percent were black.

Coach Recruitment and Training Procedures

In Oregon, teachers from seven of eight classrooms agreed to participate and signed informed consent letters. In Kentucky six of the 10 teachers invited to participate consented to take part in the study. Coaches in Oregon and Kentucky were hired with grant funds, and were therefore employees of the Oregon Research Institute and the University of Louisville, respectively. Coaches were hired as temporary employees, each serving two to four teachers/classrooms. Coaches ranged from bachelor’s- to master’s-level professionals representing education and social work disciplines. None of the interventionists in Kentucky had prior experience with the FSS intervention; all were social workers having experience with school-based interventions.

Participating teachers attended a one-day workshop in which they received training in the FSS preschool intervention and training in primary prevention strategies to reduce challenging behavior and promote social competency within each classroom. All coaches attended both the teacher FSS training session and a one-day, coaches -only training session.

Participant Identification Procedures

After teachers agreed to participate, we provided them with waiver-of-consent letters and forms, which they sent to the parents of each student in the classroom. The letter described and notified parents of the classroom-wide screening procedure, detailed the proposed study, and explained steps for declining participation in the screening process.

We asked teachers to complete stages one and two of the Early Screening Project (ESP) (Feil & Becker, 1993; Feil, Severson, & Walker, 1998; Walker et al., 1995). The ESP is a multiple-gating screening system used in early childhood settings to identify children three to five years old who are at elevated risk for problem behaviors. The ESP contains three linked screening gates of increasing intensity; the first two were utilized in this study. Stage one is based on teacher nomination and ranking of students with externalizing and internalizing problems, respectively. The highest ranked students in stage one move to screening stage two, where they are evaluated using teacher ratings on scales that measure adaptive behavior, maladaptive behavior, and aggression. For this pilot study, only children demonstrating externalizing behaviors were identified for participation. Children who were at high risk on at least one of the screening stage-two scales, had no more than four absences in the past month, and were assessed as being able to understand cause-and-effect relationships were rank ordered by classroom according to severity and were considered prospective participants. We invited parents of the highest ranked child in each classroom to participate in the study. If their parents declined, the parents of the next highest ranked child were invited until we obtained permission from one parent of an identified child from each participating classroom.

Across 14 classrooms, 62 students were screened. Forty-six students (74%) met eligibility criteria for participation, 20 students from Kentucky and 26 students from the Oregon site. In accordance with our study design, we recruited one student per classroom, starting with the first-ranked student in each classroom. From the seven Kentucky classrooms, we received consent from the parents of four first-ranked students and three second-ranked students. In Oregon, the parents of four first-ranked, two second-ranked, and one third-ranked student consented. In total, 14 students participated in the intervention phase of the study, seven from each site. Two from the Kentucky site did not complete the intervention (attrition = 14 percent). Analyses were completed for the 12 students who completed the intervention only. Students ranged in age from three to five years old (M = 4.0, SD = 0.6) and were predominantly male (83 percent). Based on teacher reports of student ethnicity, seven of the 12 students (58 percent) were Hispanic, three (25 percent) were black, and two were white (17 percent). The majority of the students lived in English-speaking households (58 percent); the remaining students lived in bilingual, English- and Spanish-speaking households (33 percent) or Spanish-only households (8 percent). Four of the 12 children (33 percent) were on an individualized education plan at the beginning of the study. Six of the 12 participating students (50 percent) were ranked first on the basis of ESP stage 2 screening criteria. The remaining six students were ranked either second (25 percent) or third (25 percent).

The percentage of participants who were at risk and the level of risk for each of the four ESP stage 2 scales used for this study are described in Table 2. Although fewer students were within the risk range on critical events, all students participating in this study were within the risk range, at least one standard deviation from the mean, on at least one of the four scales. The majority of the sample was more than two standard deviations below the mean on adaptive behavior and three-fourths of the sample was more than two standard deviations above the mean on both maladaptive and aggressive behavior.

Table 2.

Rates of Teacher-reported Student Risk for ESP Scales at Baseline

Scale	Within Risk Rangea (%)	At Risk (%)	High Risk (%)	Extreme Risk (%)
Critical Events	58.3	8.3	16.7	33.3
Adaptive Behavior	100.0	16.7	33.3	50.0
Maladaptive Behavior	91.7	8.3	8.3	75.0
Aggressive Behavior	83.3	0.0	8.3	75.0

^aIncludes at risk (1.0 SD), high risk (1.5 SD), and extreme risk (2.0 SD) levels.

The mothers of each participating child in the FSS program implemented homeBase with assistance from the FSS coach. Parents ranged in age from 22 to 43 years (M = 30, SD = 6.2); 50 percent of the participating parents were Hispanic, 42 percent were black, and 8 percent were neither Hispanic nor black. One participating parent had completed a four-year university degree, three (25 percent) completed some college, five (42 percent) completed high school or received a GED, and two (17 percent) did not complete high school.

All participating teachers were female. Five teachers were white and four were black. Reported ethnicity for the remaining three teachers was Hispanic and Asian. The reported number of years teaching was fairly variable among participating teachers, ranging from 1 to 17 years (M = 5.7, SD = 5.1). Most teachers reported having an associate’s (58 percent) degree. The remaining respondents indicated having received a high school diploma (25 percent) or bachelor’s degree (8 percent).

Measures

We collected treatment integrity, social validity, and satisfaction assessments to assess feasibility, along with outcome measures, to evaluate the potential efficacy of the intervention. Direct observations of coaches and teachers were used to assess treatment integrity for the school component, and coaches reported on the completion rates and quality of parents’ participation in homeBase. Focus group discussions and parent interviews were completed at posttest to assess social validity, and a satisfaction survey was administered to teachers and parents, also at posttest, to evaluate satisfaction. Finally, teacher- and parent-reported measures of child social skills and problem behaviors were assessed before and after the intervention to determine the potential effectiveness of the intervention.

Treatment Integrity

We collected data to assess the coach’s and teacher’s fidelity of program implementation. These data were collected an average of one time during the coach phase (M = 1.33) and two times during the teacher phase (M = 1.83) of the program, which has been a common practice in research related to the FSS intervention. The fidelity measure, adapted to the preschool context, includes 16 items that assess the implementer’s delivery of core program components. For each component, the measure assessed adherence, whether the component was implemented (yes/no), and quality of implementation using a five-point scale. For the adherence score, the proportion of program components implemented was computed from the 16 dichotomously scored (0 = No, 1 = Yes) items (items 1 through 16). The quality rating was computed for the 16 implementation quality scores rated on a five-point scale. Quality items were recoded from a scale ranging from 1 to 5 to a scale ranging from 0 to 1, with 0 = very poor, 0.25 = poor, 0.50 = okay, 0.75 = good, and 1.0 = excellent. The items were averaged, resulting in a composite score for each.

Social Validity

Project staff facilitated focus groups with participating teachers and early childhood administrators to collect formative feedback and assess the social validity of the intervention. The discussions in Kentucky and Oregon consisted of 12 and eight participants, respectively. Teachers were specifically asked whether they believed goals of the intervention were important, the intervention procedures were acceptable, and the intervention’s effects were meaningful. Each focus group lasted from 45 to 60 minutes and was led by two facilitators. Research staff took detailed notes during the group discussion and transcribed them within 24 hours.

Project staff conducted semistructured phone interviews with parents (N = 10) to assess the social validity of the FSS intervention. The substance of the questions was identical to the school staff focus group questions, but modified slightly for parents.

Satisfaction

Teachers and parents completed satisfaction surveys as part of the post-intervention data collection process to assess their satisfaction with the FSS program. Teachers responded to 13 items designed to assess the use, compatibility, effectiveness, training, and support of the intervention in the classroom setting. Parents responded to 12 items to assess satisfaction focused on ease of use, behavior change, program support, and program impact on the family. Item responses were made on a five-point Likert scale ranging from 1 = strongly disagree to 5 = strongly agree (Rowe, 2001).

Outcome Measures

The Social Skills Improvement System (SSiS), a revised and renormed version of the Social Skills Rating System (SSRS) (Gresham & Elliott, 1990), taps three important domains: prosocial behavior, problem behavior, and academic competence. The academic competence scale, intended for reporting on kindergarten through grade 12 students, was not administered as part of this study, given the age of the children. The 46-item social skills scale assesses the core prosocial skills of communication, cooperation, assertion, responsibility, empathy, engagement, and self-control as reported by the teacher’s and parent’s perceived frequency on a four-point scale in which 1 = N (never), 2 = S (seldom), 3 = 0 (often), and 4 = A (almost always). The SSiS targets learned behaviors that promote prosocial interactions with adults and peers (Gresham & Elliott, 2008). The 30-item teacher-reported and 33-item parent-reported problem behavior scales assess internalizing and externalizing problem behavior that may interfere with the acquisition or performance of prosocial behaviors. Like the social skills scale, problem behavior items are assessed on a four-point scale assessing perceived frequency. Problem behavior subscales include externalizing, bullying, hyperactivity/inattention, internalizing, and autism spectrum. The national standardization sample of the SSiS-RS included 4,550 children ages three to 18. The SSiS-RS demonstrates appropriate levels of research integrity based on the widespread use and substantial body of scholarly research on the SSRS (Gresham & Elliott, 1990). The psychometric properties of the SSiS-RS have been compared with those of the SSRS (Gresham, Elliott, Cook, Vance, & Kettler, 2010; Gresham, Elliott, Kettler, 2010; Gresham, Elliott, Vance, & Cook, 2011). Results of this systematic comparison indicate that the SSiS-RS offers a broader conceptualization of important social behaviors and is psychometrically superior to its predecessor. The social skills and problem behavior standard scores were our target outcomes for this pilot study.

Data Analysis

Qualitative Analyses Procedures

We used a mixed-methods triangulation design (McMillan & Schumacher, 2006), including a thematic analysis procedure (Braun & Clarke, 2006) and the General Inductive Approach of Evaluation (Thomas, 2006) to analyze and classify results of interviews and survey responses. Thematic analysis identifies themes and provides a framework for analysis, interpretation, and reporting of qualitative data. Further, we used a constructionist approach for interpreting the data in which codes were assigned to explicit statements made by participants during interviews.

We also used two coders to evaluate the interrater reliability of our coding scheme. The two coders first independently reviewed focus group and interview transcripts and identified themes within broad categories. QSR International (2008) NVivo 9 qualitative software was used to code interviews and conduct thematic analyses. The coders met and discussed possible themes and subthemes in each category to develop a preliminary coding scheme. These themes represented a pattern of responses related to stakeholder perceptions about goals, procedures, and outcomes of the FSS preschool intervention. The coders engaged in two waves of independent coding, comparison, and revision of the coding scheme.

Quantitative Analysis Procedures

To identify responders and nonresponders to the FSS program, we computed a measure of clinical significance for each student. Examination of clinical significance facilitates the evaluation of whether a program has a practical and beneficial impact on student functioning (Martinovich, Saunders, & Howard, 1996). Following the Jacobson-Truax method (Jacobson & Truax, 1991), we applied a two-step criterion that accounts for each student’s level of functioning and the magnitude of change on each teacher- and parent-reported outcome measure. To assess whether each student’s level of functioning had improved, we calculated a cutoff score between the functional sample (the SSiS normative sample) and our sample. Then, to determine whether the observed change in scores from pre- to postintervention was statistically reliable, we calculated a reliable change index (RCI) for each student outcome by computing the difference between observed baseline and postintervention scores and dividing by the standard error of measurement.

RESULTS

Feasibility (that is, treatment integrity, social validity, and satisfaction) results were analyzed at the aggregate level. RCIs were calculated for each individual case for the purpose of identifying responders and nonresponders to assess the potential efficacy of the intervention. The Results section is organized around the research questions.

To What Extent Was the Intervention Implemented With Integrity?

Treatment adherence and quality, which could range from 0 = very poor through 1.0 = excellent, indicated that both coaches and teachers were implementing a majority of the observed core program components for the school component, though coaches implemented a slightly higher proportion of program components (M = .88, SD = .06) than did teachers (M = .83, SD =.16). Also, implementation quality was higher for coaches (M = .86, SD = .09) than for teachers (M = .74, SD = .13).

To What Extent Do Teachers and Parents Perceive the Intervention to be Socially Valid?

We examined social validity via the qualitative analysis of parent, teacher, and early childhood administrator interview data collected via phone interviews and focus groups. We analyzed the qualitative data to identify themes that could improve future program support and implementation.

A description of each theme and subtheme identified through the qualitative analysis, as well as the frequency of data related to each theme, is provided in Table 3. Stakeholders were specifically asked whether they believed the goals of the FSS intervention, defined as following directions, completing school tasks, and getting along with others, were important. Responses were coded as either important or not important. Teachers appeared reluctant to answer this question or did not understand the question. However, participating parents were quick to offer support for the goals of the FSS intervention, with all four who were interviewed suggesting it was important for their child to learn to follow directions because they served as role models for younger siblings in the home.

Table 3.

Focus Group Interview Findings: Themes, Findings, and Prevalence of Response

Theme	Relation to Study Purpose	Description Based on Findings	Illustrative Quote	# Sources Referencing Theme	# Times Theme Referenced
Importance of the goals	Social validity	Stakeholders were specifically asked whether they believed the goals of the intervention, defined as following directions and completing school tasks and getting along with others, were important. Responses were coded as either important or not important.	“It is important for my child to learn to follow directions because they are a role model for my younger children.”	Important = 2 Not important = 1	Important = 5 Not important = 1
Procedures	Social validity	Responses coded in the procedures theme relate to the feasibility of the procedures. These comments were further divided into training (formal or informal) and nontraining-related comments, with formal training related to the full-day workshop that preceded the intervention phase and informal training related to the support provided by the coach during the intervention.	“Sometimes had to give reward before end of day because they would leave early, but child hadn’t completed day, felt like it was taking a shortcut. Sometimes the teaching assistant didn’t like pausing during classroom routine, wanted to keep things flowing.”	Training = 2 Nontraining = 4	Training = 9 Nontraining = 27
Effectiveness	Social validity	Respondents shared their perceptions as to whether the goals of the intervention had been met, and these responses were coded in the effectiveness theme. If the respondent indicated the child, the teacher, or other children in the classroom benefited from the intervention, the comment was coded as effective. Similarly, if a respondent expressed belief that the intervention was ineffective, or made statements leading one to believe the effectiveness of the intervention was questionable, the comment was coded in the ineffective subtheme.	“This helped the child to be more conscientious about their behavior, the visual cues were very helpful, the prompter gave him cues also to stay on task, the green card also helped the other children’s behavior. The targeted child became a model for the other children to follow.” “When he was going to have a behavior problem, he would see red and think about what he was doing.” “The personal philosophy doesn’t match mine. In Early Childhood Education training, a child needs “more encouragement not ‘prizes’ more encouragement. Saying ‘good job, you’re doing well,’ it’s too general.”	Effective = 4 Ineffective = 2	Effective = 39 Ineffective = 13
Satisfaction	Satisfaction	Although participants’ satisfaction with the intervention procedures or effectiveness was implied in many comments in the previously described themes, there were a few comments related to satisfaction that could not be coded in the pre-established categories. These comments were coded under this theme.	“The coach was wonderful; Julie helped to explain any confusion in the process and recognized frustration with me and the computer and was helpful in explaining things. She also gave good insight on why steps were done in a certain way.” “We will be advocating for parents and liaisons to use this program at home. We will use this program next fall with the new students.”	High = 2 Low = 1	High = 6 Low = 7
Stigmatization		Researchers were particularly interested in whether teachers or parents believed participating children were singled out, stigmatized, or labeled as a result of participating in the intervention.	“The whole class got involved in helping the targeted child succeed; they also celebrated with him when he was successful.” “The child didn’t seem to have any negative experiences. All the kids wanted Ramie to work with them.”	2	8
Recommendations	Implementation improvement	Not surprisingly, respondents offered specific suggestions to improve satisfaction, fidelity, or outcomes. In addition, some participants of the interviews communicated strategies they used that were not systematically endorsed by the training procedures, and these were also coded in the recommendations theme. Recommendations fell into two general categories: (1) formal trainings or informal trainings (that is, coach support) and (2) nontrainings.	“I think you should use a stopwatch rather than the class prompter or prompter CD because it was more reliable and mobile.”	Training = 3 Nontraining = 4	Training = 13 Nontraining = 19

Responses were coded under the procedures theme relative to this feasibility. These comments were further divided into training (formal or informal) and nontraining related comments, with formal training referring to the full-day workshop that preceded the intervention phase and informal training referring to the support provided by the coach during the intervention.

Three times as many references addressed nontraining aspects of the procedures, such as the transition between the coach and teacher phases, use of the reinforcement (beeper) system, delivery of verbal reinforcement, and Green/Red Card game protocol. Teachers appear to have commented on the procedural aspects that were most problematic for them. With regard to transition between the coach and teacher phases, one teacher requested that the coach be physically present for a few days before the teacher takes over. In addition, teachers from both sites reported technical difficulties with the prompter systems (CDs and class prompter); even if functioning properly, several teachers commented that these systems do not allow them to travel out of the classroom during the Green/Red Card game.

The practice of providing attention for desirable behavior was raised as a concern by teachers in Oregon. Specifically, some Oregon teachers objected philosophically to acknowledging desirable child behavior, citing concern that doing so would make children dependent on praise in the future. Another issue that was only mentioned by Oregon teachers involved use of the timers to guide reinforcement. Specifically, these teachers struggled to balance the demands of implementing the program and attending to their typical responsibilities. In Kentucky, some teachers struggled to modify delivery of the end of the game reinforcement procedure so that it fit within the context of the school day. Although teachers from both sites commented that the training was a little overwhelming, comments in this regard were not very detailed. It is clear from teacher comments that the teachers’ perception of the helpfulness of the coach was critical to their overall satisfaction with the FSS intervention.

Respondents shared their perceptions as to whether the goals of the intervention had been met. These responses were coded in the effectiveness theme. If the respondent indicated the child, the teacher, or other children in the classroom benefitted from the intervention, the comment was coded as effective. Similarly, if a respondent expressed belief that the intervention was ineffective, or made statements leading one to believe the effectiveness of the intervention was questionable, the comment was coded in the ineffective subtheme. There were nearly four times as many references suggesting the intervention was effective versus ineffective (or questionable). All four parents who were interviewed expressed belief that the intervention was effective. Comments suggesting the intervention was not effective were only recorded by Oregon teachers, and many of these comments related to the belief that external motivators only produce short-term change.

We were particularly interested in whether teachers or parents believed participating children were singled out, stigmatized, or labeled as a result of participating in the intervention. Six of the eight references from Kentucky teachers and parents suggested the target child was not stigmatized while participating in the program. Three of the four parents participating in the interviews stated explicitly that the child was not singled out; however, one stated, “One time the teacher said, ‘He was horrible today and he didn’t get the green card.’”

It was not surprising that respondents offered specific suggestions to improve satisfaction, fidelity, and outcomes. In addition, some interview participants communicated strategies they used that were not systematically endorsed by the training procedures. These strategies were also coded in the recommendations theme. Recommendations fell into two general categories: (1) formal trainings or informal trainings (that is, coach support); and (2) nontrainings. Perhaps the most frequent suggestion from teachers was to have all staff (for example, assistant teacher, bus monitor, resource teachers) participate in trainings. Other suggestions related to training were as follows: schedule them closer to the time the intervention starts, have a refresher session, and have regularly scheduled (that is, monthly) meetings with the teachers and trainer/coaches. Another teacher requested more attention be spent on the content of the homeBase component.

One teacher from Kentucky and several teachers from Oregon thought it would be better if the coach implemented the Green/Red Card game during all 20 program days, and another suggested both teachers wear a card; two others recommended the transition from the coach phase to the teacher phase simply be more gradual. Finally, a teacher commented that it was helpful to have the coach talk to her at night by phone. Many teachers, particularly in Kentucky, made recommendations related to the systems in place to remind teachers to acknowledge positive behavior and track points. For example, one teacher set a stopwatch as a visual prompt to remind herself to acknowledge desirable child behavior. Several teachers reported using a stopwatch instead of the class prompter or prompter CD because it was more reliable and mobile. A number of recommendations were related to the reward structure. For example, teachers recommended that all children be given a choice of rewards and that more reward choices be made available. Another teacher recommended the reward activity be implemented so as to not interfere with the daily routine. Finally, one teacher suggested that the teacher accompany the coach on a home visit.

To What Extent Were Teachers and Parents Satisfied with the Intervention?

We examined descriptive statistics of the satisfaction scales to assess the extent to which participants were pleased with FSS program implementation, support, and outcomes. Teacher-reported satisfaction scores were highly variable. Mean scores ranged from 2.85 to 4.62 on a five-point Likert- type scale, with 5 indicating high satisfaction. Item-specific analyses indicated that teachers (Ν =11) were least satisfied with the extent to which the program interfered with their other teaching activities (M = 2.60) and most satisfied with the extent to which they received ongoing support or help while using the program (M =4.00). Although the sample size was small (N = 6), parent satisfaction scores were very high, ranging from 4.25 to 5.00 on the same five-point Likert scale. A within-item analysis suggests parents were highly satisfied with all aspects of the program, with item means ranging from 4.50 to 5.00.

To What Extent Was the Intervention Effective in Reducing Problem Behavior and Improving Social Skills?

We reviewed teacher and parent reports of social skills and problem behaviors using the SSiS to target responders and nonresponders. Changes in the teacher-reported social skills scale produced the largest percentage of program responders. Nine of 12 students (75 percent) demonstrated clinically meaningful improvements in social skills at postintervention. Six of the 12 students met both criteria of the Jacobson-Truax method. These students’ postintervention social skills scores moved to within the specified cutoff equal to a standard score of 82 and had an RCI greater than 1.96. The remaining three students’ teacher-reported social skills score remained unchanged from baseline to postintervention.

Three of 12 students (25 percent) demonstrated clinically meaningful reductions in teacher-reported problem behaviors at postintervention, meeting both criteria. The remaining nine student’s teacher- reported problem behavior scores did not significantly change from baseline to postintervention based on the RCI. Based on parent-reported data, two of eight students improved on social skills and three of eight students had meaningful reductions in problem behavior in the home setting. Overall, across parent- and teacher-reported SSiS social skills and problem behavior scales, nine of 12 students recovered or improved, based on the Jacobson- Truax method, on at least one of the four outcomes. Student-level pre- and postintervention standard scores on the SSiS teacher- and parent-reported social skills and problem behavior scales and the corresponding RCI for each student are summarized in Table 4.

Table 4.

Standard Scores and Reliable Change Index for SSiS Teacher- and Parent-reported Social Skills and Problem Behavior Scales

	Teacher Report						Parent Report
	Social Skills			Problem Behavior			Social Skills			Problem Behavior
Student	Base	Post	RCI	Base	Post	RCI	Base	Post	RCI	Base	Post	RCI
Oregon
9601	72	99	5.95	112	101	−1.42	—	—	—	—	—	—
9602	84	99	3.40	105	105	0.00	94	108	1.50	109	105	−0.42
9603	62	111	11.20	112	95	−2.13	72	117	4.84	118	86	−4.05
9604	91	94	0.71	112	123	1.28	126	124	−0.21	118	110	−0.98
9605	81	87	1.28	137	130	−0.85	—	—	—	—	—
9606	66	77	2.13	148	130	−1.85	—	—	—
9607	62	76	2.55	145	141	−0.43	—	—	—	—	—	—
Kentucky
9501	84	99	3.54	122	102	−2.41	87	104	1.78	108	93	−1.81
9502	79	83	0.85	132	137	0.57	95	89	−0.64	133	131	−0.28
9503	75	105	6.80	143	99	−5.54	95	74	−2.28	141	112	−3.63
9504	74	85	2.41	101	111	1.28	114	106	−0.85	95	96	0.14
9505	73	90	3.83	122	109	−1.56	83	110	2.92	138	99	−5.02

Note: SSiS = Social Skills Improvement System; RCI = reliable change index.

DISCUSSION

This study demonstrates the promise of the FSS intervention in preschool settings and contributes to the growing literature base that provides preschool teachers and administrators with viable options to improve their students’ apostrophe chances of later school success (for example, Gresham, Cook, Crew, & Kern, 2004; Walker & Gresham, 2003; Walker, Ramsey, & Gresham, 2004). Fidelity data suggest the preschool version of the FSS intervention can be implemented with acceptable integrity by coaches and teachers in preschool settings when implemented with adequate resources and supported by the program’s developers. Further, program completion rates indicate teachers’ and parents’ engagement in the program was at acceptable levels. Social validity outcomes suggest parents’ perceptions of the program’s goals, procedures, and outcomes were extremely favorable, and social validity from the teacher perspective was acceptable. It should be noted that several of the teacher suggestions were valuable and can easily be incorporated into the FSS intervention materials or procedures. For example, it is clear that the beeper system notifying teachers when to administer points requires modification. Also, the transition between the coach and teacher phases should be slower. Finally, preschool programs that do not support the use of positive attention and the use of behavioral principles for young children should weigh the pros and cons of the intervention carefully before committing resources to the program. Although this study design limits definitive statements about causality, the observational data and consistency of the findings across cases and sites is promising, and the measurement and data collection procedures can be refined for a large- scale efficacy study. The pattem of results was also noteworthy. Specifically, the magnitude of parent- reported change in social skills was substantially lower than teacher reports of social skills improvement. These results are not surprising given existing literature indicating that children’s social maladaptation and emotional distress can vary across home and school settings and across informants (for example, parent and teacher) (Dishion & Kavanagh, 2003). The poor concordance between parent and teacher reports must be interpreted with caution because the variance could be due to differences in informant perception or behavior differences within the home and school settings. Differences may also be attributable to the differential scores at baseline, resulting in change being more difficult to document. For example, one child’s social skills were classified by parent reports in the above-average range and four were classified in the average range at baseline. It is important to note that the two children rated by parents as having below-average social skills were classified in the normative range at posttest. Likewise, parent reports of problem behaviors were also notably more favorable than teacher reports, with five and two children being classified as above average and average, respectively, at baseline. The ability of the intervention to replicate findings across research sites is particularly impressive. Further study of preschoolers’ responsiveness to the FSS program is clearly warranted by results of this study.

There are several limitations of the study that should be noted. The primary limitations are that the design does not adequately control threats to internal validity and that the sample size limits power to detect changes. Another limitation involves the failure to collect treatment integrity data related to the home component. It is important to note that the lack of parent-related data from the Oregon site was the result of the parents not being asked for it, rather than the parents being asked, but not responding. The current study was also limited to evaluating only immediate postintervention effects. Lastly, parent attrition from the study was high at the Oregon site, which may have biased the formative feedback received. Future research should control for threats to internal validity by using randomized controlled designs with sufficient statistical power, attending to treatment integrity related to the home component, and including a follow-up assessment to determine whether the intervention effects are maintained over time and have an impact on elementary school readiness. There are also a number of strengths in the current study. Notably, the fidelity and outcome results were replicated across sites, which considerably improves the external validity of the study.

Additional research is needed to advance our understanding of the effects of this intervention. Specifically, evaluating the impact of the study in the context of a randomized control trial would control for threats to internal validity. In addition, a larger sample would increase the study’s power and, therefore, increase the likelihood that true changes in child outcomes would be detected. Future research should also examine whether initial gains are maintained when the intervention procedures are withdrawn, and in subsequent years. Adding direct observations of child behavior in the home and school settings and measures of academic functioning would be useful in future research efforts. Finally, conducting moderation and mediation analysis to better understand for whom the intervention works best, and under what conditions, would be extremely beneficial.

IMPLICATIONS FOR SCHOOL SOCIAL WORK

There are also implications specific to school social work. With regard to practice, there are few empirically supported intervention practices that fit the theoretical orientation and skill set of social workers better than FSS. Specifically, the focus on the home and school setting (that is, ecological focus) as well as the strengths-based nature of the curriculum are aspects of the intervention most social workers will find appealing. It is important to note that the majority of the interventionists in this study were social workers. This study also represents some methodological components that school social work researchers and practitioners may find appealing, and could serve as a model for other studies. Specifically, evaluating the feasibility of interventions in educational settings is an area of research that has been overlooked yet is critically important. Feasibility studies examining treatment integrity, social validity, and satisfaction not only contribute to the literature base once disseminated, but also provide valuable information locally for school-based response to intervention or positive behavior support teams charged with making significant placement decisions on the basis of a student’s response to an intervention. The case-level analysis, particularly the RCI, may also be of interest to social work researchers, many of whom are conducting research with few resources and, therefore, small sample sizes.

CONCLUSION

Although the primary purpose of this pilot study was to prepare for a large-scale randomized controlled trial, the results provide preliminary support for the feasibility and potential efficacy of the preschool version of the FSS early intervention program. The early childhood adaptations to the FSS intervention appear to be a superb option for support service staff charged with removing barriers to learning, and the home-school components are an ideal fit with school social work practice. Future efforts will systematically address the limitations of the current study, particularly with regard to threats to internal validity and evaluation of external validity of the FSS preschool intervention.