The Summer Treatment Program Meets the South Side of Chicago: Bridging Science and Service in Urban After-School Programs

Abstract

Background

This paper describes efforts to apply the principles and strategies of an empirically-supported treatment for children with disruptive behaviour problems to a park after-school program serving children in urban poverty.

Method

Collaboration with staff proceeded in stages: (1) relationship building, needs assessment, and resource mapping; (2) intervention adaptation and implementation; and (3) implementation support, problem-solving, and sustainability.

Results

Four tools capitalised on inherent strengths of the parks, accommodated child and staff needs, and emerged as feasible and effective: Group Discussion, Good Behaviour Game, Peers as Leaders, and Good News Notes.

Conclusions

Recreational settings offer opportunities for mental health promotion for children in urban poverty.

Introduction

The past decade has witnessed increased attention to the need for mental health research that bridges science and service (e.g. Hoagwood, Burns, & Weisz, 2002). Despite a growing literature that better represents the patients, providers, and settings of routine care, transporting evidence-based interventions to community care settings is challenging (e.g. Schoenwald, Sheidow, & Letourneau, 2004), and the widely acknowledged research to practice gap remains (e.g. Fixsen et al., 2009). This paper describes the iterative process and inherent challenges of adapting the principles and strategies of the empirically-supported Summer Treatment Program (STP; Pelham, Greiner, & Gnagy, 1997) for use by after-school staff in a large, urban park district.

Research aims

Project NAFASI (Nurturing All Families through After School Improvement; Frazier, Cappella, & Atkins, 2007) represents an ongoing partnership between the investigative team and Chicago Park District to strengthen the benefits of after-school programs for children in urban, high-poverty communities. The partnership began with a federally-funded study examining a model for mental health consultation, training and support for park staff. The study proceeded in two phases. In Phase One, we collaborated for one year with after-school staff at one park to adapt an efficacy-based, manualised intervention – the Summer Treatment Program (STP; Pelham et al., 1997) - focused on facilitating positive peer socialisation, reducing disruptive behaviours, increasing prosocial behaviours, and improving academic performance. In accordance with Step 1 of the Clinic/Community Intervention Development Model (Hoagwood et al., 2002), the goal was to apply the principles and simplify the strategies of the STP to make them feasible, effective, and sustainable for staff in publicly-funded after-school programs serving urban, poor communities. The subsequent year, in Phase Two, we pilot tested the adapted intervention in three park after-school sites, to study its impact on children's mental health outcomes, compared to three demographically similar sites that received no intervention. This paper describes Phase One of the research, in particular the process and challenges associated with moving the STP from university-based efficacy trials to a community-based practice setting.

Method

This study was conducted in accordance with American Psychological Association Ethical Guidelines and approval from the Institutional Review Board for recruitment, informed consent, and data collection procedures.

Intervention for adaptation: Summer Treatment Program

The Summer Treatment Program (STP; Pelham et al., 1997) was originally designed as a camp for children with attention-deficit hyperactivity disorder (ADHD). Numerous iterations have been implemented and examined with nearly 2500 children across three decades. The STP integrates social emotional learning and behaviour supports into recreational and classroom activities. It is a highly structured and standardised response cost system in which children receive points for following rules and lose points for breaking them. The STP is designed to facilitate positive peer socialisation (e.g. good sportsmanship), reduce disruptive behaviours (e.g. noncompliance), increase prosocial behaviours (e.g. rule following), and improve academic performance. Recent evaluations include one large, between group trial of youth with ADHD (Pelham et al., 2000) and several within-group crossover (e.g. Chronis et al., 2004), and well-controlled, single-subject (e.g. Coles et al., 2005) studies. Collectively, findings demonstrate symptom reduction and improved functioning across multiple domains. The focus on reducing impairment in recreational contexts, flexibility of intervention components, and data on impact and consumer satisfaction made the STP an ideal intervention for adaptation to after-school programs.

Setting for adaptation: Chicago Park District Park Kids program

The Chicago Park District’s Park Kids after-school program offers homework, sports, and recreation to support children’s academic, physical, and social competencies. The program operates in approximately 80 field houses serving nearly 8000 youth in grades kindergarten to eight from 3:00 to 6:00 p.m. via three 10-week seasonal sessions. Park Kids began as an organised, comprehensive, citywide after-school program in 1995 with substantial funding for healthy snacks, auxiliary homework staff, and professional development. Over time, funding dissipated leaving parks to rely on nominal enrollment fees insufficient to support staff training or program implementation. Successful programs rely on strong local leadership and the commitment and capacity of park staff to do their best with limited resources.

Method of adaptation: STP meets Park Kids

The goal of this work was not to implement the STP but rather to apply its principles and adapt its strategies to enhance Park Kids. The Project NAFASI intervention team was comprised of project staff, including the principal investigator (1st author), 3 clinical psychology interns (one intern, 2nd author, had prior experience as a counselor in the STP), and 2 research assistants, and park staff, including the supervisor and 3 after-school staff. The complexity of the STP, coupled with several notable differences in setting, staff, and families, required the team to adapt core features while adhering to underlying principles that reflect an extensive empirical literature on social learning and reinforcement theory.

Program differences

The STP is intensive in terms of cost, staff-to-child ratio, staff characteristics, initial and ongoing training, and frequency and quality of supervision. The STP costs $3,000 for an 8-week full-day camp session, compared to Park Kids which relies on enrollment fees ranging in cost from $20 to $100 for a 10-week (daily, 3:00–6:00) session. Reflecting this difference in fees, the STP typically employs 4–5 staff for every 12 to 16 children. In contrast, the Park Kids staff-to-child ratio varies widely. Among those participating in both phases of this work, some had no more than two full-time staff for upwards of 30 youth.

STP staff is comprised of advanced undergraduate students with substantial experience in child psychology. They receive 80 hours of initial training on principles of behaviour management, including intensive modeling and role-plays of STP procedures, ongoing performance feedback, and daily fidelity checks and supervision by senior staff. Park Kids front-line staff includes full-time physical instructors and supervisors with college degrees and part-time recreation leaders with high school diplomas. The park district offers seasonal workshops but funding for staff development or curriculum planning is limited, as the majority of funds are allocated to direct services for children.

Participant differences

Although there is increasing demographic variability among families participating in STP camps, most have the financial resources to pay associated costs. In contrast, families who utilise Park Kids reside in urban, often high-poverty communities. In our sample, 96% of children received free/reduced lunch and 26% of parents were unemployed. Children who attend STP camps tend to arrive on time and remain all day, whereas Park Kids youth often arrive late or leave early. Parents typically have limited involvement, as children often walk to and from the park themselves or with friends or siblings. In contrast, the STP requires significant involvement from parents, beginning with a clinical intake session and group orientation and continuing with daily contact at departure around children’s individual behaviour goals and a weekly, group-based behaviour parent training session.

One notable exception to the differences between the STP and Park Kids is the comparability of behaviour problems exhibited by children themselves. Although STP enrolls a clinical sample of children and Park Kids enrolls a community sample, both groups exceed national norms for disruptive behaviours. In fact, children in the Park Kids sample exhibited more parent-reported peer problems (M=2.2 versus 1.4, t₁₀₇=4.4, p<.001, Cohen’s d=.46), hyperactivity-inattention (M=3.9 versus 2.8, t₁₀₇=4.4, p<.001, Cohen’s d=.40), conduct problems (M=1.9 versus 1.3, t₁₀₇=2.8, p=.006, Cohen’s d=.27) and Total difficulties (M=9.8 versus 7.1, t₁₀₇=4.3, p<.001, Cohen’s d=.42) on the Strengths and Difficulties Questionnaire (mental health screening form), compared to national norms (Frazier et al., submitted).

Goal for adaptation: feasible, effective, and sustainable intervention

Drawing on recommendations by Connor-Smith and Weisz (2003), The Conduct Problems Prevention Research Group (CPPRG, 2002), and by Pelham and colleagues, our goal was to apply the principles and components of the STP in a way that: (1) capitalised on the inherent strengths, structures, and mission of Park Kids; (2) accommodated the needs of children and staff; and (3) was feasible, effective, and sustainable within the constraints of their program. We avoided a priori decisions regarding the number of changes permitted or specific endpoints in order not to inhibit or disrupt the collaborative process. We kept comprehensive minutes and detailed field notes of team meetings, and careful records of changes to intervention tools. Collaboration proceeded in three stages (Frazier et al., 2007): (1) relationship building, needs assessment, and resource mapping; (2) intervention adaptation and implementation; and (3) implementation support, problem-solving, and sustainability. We relied on several clinical principles and strategies to guide discussions at each stage, described below with specific excerpts from our field notes to illustrate their application.

Results

Stage 1: Relationship building, needs assessment, and resource mapping

This first stage emphasized relationship building via planned meetings, informal dialogue, and participant observation. The intervention team met weekly for lunch, and project staff spent an average of an additional 6 hours each week at the park during Park Kids. We borrowed tools from community-based participatory research and motivational interviewing to ensure that the service model would reflect the goals of our park colleagues. Consistent with models of collaboration rooted in a spirit of mutual trust and respect (Garland, Plemmons, & Koontz, 2006), maintaining open communication was crucial. A park administrator summarised it best:

The university team is not coming to facilitate their own program, but to support the park staff. As we start to implement things, they will be present to help things work out … The learning goes both ways. We must diffuse the fear that anyone is being judged or evaluated by talking about this fear openly.

Park staff shared examples of strengths and challenges within their program, experiences with children, training opportunities, activity rotations, rules and routines of the park. Through this dialogue and extensive participant observation (2 to 3 hours, 3 days each week), project staff acquired a better understanding of park procedures and enhanced rapport with staff and kids. Project staff provided direct child support when needed (e.g. during homework), thereby demonstrating their understanding of roles and expertise in working with youth. Maintaining ongoing, open dialogue and a consistent presence at the park communicated that we were committed to supporting the goals and priorities of the Park Kids program.

Inviting park staff to identify the strengths and needs of their program and contribute to intervention design accomplished several goals. First, it facilitated a collaborative foundation that enabled park staff to share in a candid and productive way what they found most challenging about youth care work. Second, the dialogue produced a list of facilitators and barriers to quality service delivery and recommendations for change that increased their investment in the intervention. Third, it ensured that the emerging service model would be responsive to the unique and most urgent needs of their setting. Toward the end of stage one, the intervention team coalesced around the importance of clear, specific, and well-defined rules and expectations.

Project staff: If we asked the kids what the rules are, what do you think they’d tell us?

Park supervisor: They would probably say, “We don’t know.” Each instructor has their own set of rules. This is a public park. The only thing we ask is that you respect it.

Project staff: What do you think a third grader thinks that means?

Park staff: Following directions; When there are apparatus, don’t play on it; Respecting others’ space and belongings; Ask permission to leave.

Park supervisor: Maybe we should come up with general house rules with the kids?

Initially park staff expressed concern that too much emphasis on rules would create rigidity and contradict the social and recreational goals of the park setting. Project staff guided discussion by way of relevant analogies to illustrate that rules were neither peripheral nor disruptive to sports and games – to the contrary, they were fundamental to both.

Project staff: We found that kids in the STP were getting left out of sports because they didn’t understand the rules. Do you get a sense that the kids here understand the rules?

Park staff: I find that lots of kids do know the rules to basketball. I try to make sure all the kids know the rules because we have beaten other teams because we knew the rules. Teaching the rules will help them understand the game better. Without rules it would be total chaos.

Extensive discussion over several weeks led to consensus around four park rules, mirroring those of the STP, and relevant to all rotations and activities: (1) Be in your assigned area; (2) Follow directions; (3) Respect people, place, and things; and (4) Participate. Park staff decided to formally introduce and operationalise the new park rules with participating children via games and discussion, and to review them often. The STP requires counsellors to facilitate a group discussion with campers at the start of every activity and transition, during which rules, including specific definitions and examples, are reviewed.

Project staff: Sometimes it is easier for kids to remember a smaller number of rules and they will learn when you say them repeatedly. In the summer program we also have the kids tell us what the rules mean and what the consequences are for breaking the rules.

After-school program staff renamed the group discussion a “rap session” and began each afternoon – but not each activity – with a similar review of rules and routines for the day. This introduction of the new rules and plan for a daily rap session launched the winter session of Park Kids and, simultaneously, Stage Two of the collaboration.

Stage 2: Intervention adaptation and implementation

During Stage Two project staff offered clear and specific empirically-supported STP recommendations designed to increase rule-following and decrease rule-breaking.

Project staff: How do you encourage the behaviour that you want to see and discourage the behaviour you don’t want to see? The STP has rewards for following rules and consequences for breaking them. If everyone is comfortable with the rules we can spend some time thinking about a reward system because without these the rules are meaningless.

Stage Two involved didactic instruction including a combination of anecdotal examples from the STP, visual illustrations, and live demonstrations. Often, the result was a blend of an STP strategy with a park staff’s own idea for how to address a specific need. Ongoing, weekly one-hour meetings were used to address questions or concerns about particular intervention tools, and to invite further dialogue or modifications.

Park supervisor: I don’t want the kids to get the wrong idea being rewarded for things they should do. Then kids can be slick too. They will act a certain way if they know they will get a reward.

Project staff: Rewarding kids for things they should be doing is a concern that many people raise. Grown-ups have rules we follow. Like I follow the rules when I drive and I don’t I get a ticket. And when we go to work we get paid for it. Kids are going to create free time whether we give it to them or not. We want to use free time as a reward because otherwise they are going to take time during activities. So do what you have to now and then have free time later… If they do what they are supposed to do and they do it well, then what’s wrong with free time?

Project staff maintained a consistent presence several times each week for an average of 6 hours during the after-school program to offer real-time, in vivo support to park staff around intervention implementation. Project staff demonstrated strategies, encouraged park staff to practise, and provided performance feedback. For example, when launching the Good Behaviour Game (GBG), a group-based, contingency-based system of rewards and consequences for rule-following behaviour, park staff requested that project staff introduce the strategy to the children and model how rule-breaking would be addressed.

Project staff: If all the kids are starting in the homework room then there can be somewhere posted a series of smiley faces. Every time a kid breaks a rule you take away a smiley face.… At the end of the day they need one smiley face left to get their reward… Our staff is going to support you as much as possible in the first couple of weeks… We can put the rules on big sheets of paper so the kids can see them. We could label the rule breaking and praise the rule following …

After several weeks, park staff facilitated the GBG themselves, with coaching and feedback from project staff. Whereas every child in the STP has their own individualised Daily Report Card (DRC), park staff relied on the group-based GBG to enforce rule-following at the park. They utilised the DRC only for individual children whose more severe behavioural difficulties resulted in losing too many points for the group.

In addition to introducing new intervention tools, the intervention team also built upon strategies that park staff were already using. For example, some staff relied on older youth (grades six to eight) to help with younger kids (grades kindergarten to five), in particular during sports.

Park staff: There are some kids who can tumble. They are too old to compete, but they can teach what they have learned to others.

Project staff: Have you done anything like that?

Park staff: Yeah, I find that to be very helpful. It is better to have those kids who have experience to help those less experienced.

Peers as Leaders (PALS) represented one attempt to build on natural routines and simultaneously respond to concerns raised by staff. Specifically, homework time presented numerous challenges. Although some children completed their work quickly and independently, others came in without homework, and still others brought assignments that exceeded their instructional levels. Often children became playful or loud, thereby disrupting their peers. PALS provided an opportunity for children to move into a peer-supervised game room after completing their work. Drawing on the STP junior counsellor program, we offered training for older students in peer-assisted learning, managing stress and frustration, conflict resolution, facilitating games and activities, and positive reinforcement and praise. We created a manual, contract, and record keeping procedures for them.

Park staff: The kids can’t go to the game room when they complete their homework because they can’t be unsupervised.

Project staff: What if some older kids – peer leaders – supervise the younger kids?

Park staff: The younger kids are not there the same time as the older kids and the older kids are supposed to be in their activities.

Park supervisor: What if some of the peer leaders go to the classes with the little kids and assist the instructor? Everyone can’t be peer leader on the same day.

Project staff: Let’s say each kid has two name cards. When they come in on Monday they place their name cards on the two days they will peer lead for the week. We want two peer leaders each day.

Access to the game room provided an incentive for students to complete their work and maintained a quiet homework space. Moreover, older youth had the opportunity to develop age-appropriate skills related to leadership and responsibility. By involving all of the instructors and recreation leaders in the intervention planning, modification, and implementation, project staff tried to ensure that, like the rules, PALS had a structured role in each activity rotation.

Stage 3: Implementation support, problem-solving, and sustainability

For the remainder of the year, each project staff was linked to a single park staff with whom they met weekly to provide ongoing support for intervention implementation, respond to individual questions, and help problem-solve challenges unique to each rotation. Ongoing weekly team meetings revealed that some intervention tools had become substantively integrated into natural park routines:

Park supervisor: [Name] does a Rap Session with the kids every day. She goes over the rules and has them give examples and she says she feels like that is second nature now and the kids are really engaged.

Others, though, presented more of a challenge:

Park supervisor: With the Good Behaviour Game she doesn’t feel like that is second nature. Just things come up. Like, before, if something was annoying she might or might not have said anything, but now anytime that kind of behaviour comes up she has to label it and take off a mark at that moment. She says she needs more time with that.

Project staff acknowledged openly and often that the evidence-based recommendations they were drawing from (a) would need to be simplified further or otherwise adapted to accommodate the strengths and capacities of park staff and (b) may be necessary but not sufficient to meet the needs and challenges of the staff and kids. The intervention team tried to think creatively and expansively during remaining weekly meetings about community-based applications of the STP manual, tools, and training procedures. Project staff reiterated often that the goal was not to replace Park Kids with the STP, but rather to integrate specific principles and intervention tools into natural program routines.

As the spring session drew to a close, dialogue shifted to emphasize sustainability of recommended tools for this community practice setting.

Project staff: The things that we do here – we want you to be able to keep them going if you think they are meaningful. We would like to think with you about this peer leader piece, for example.

Park supervisor: Will you be coming up with a peer leader manual draft?

Project staff: Yes we will. And we will share it with you. I don’t want us to do stuff that you guys like and find helpful, but when we leave would disintegrate.

This final stage empowered park staff to modify and implement recommended tools and strategies, but with ample opportunities to problem-solve barriers and trouble-shoot anticipated challenges with project staff. The goal was for Park Kids staff to feel comfortable, competent, and effective using the strategies beyond research funding and in the absence of the project staff, in anticipation of the subsequent academic year.

Discussion

The multiple and extensive differences between the STP and Park Kids demanded a flexible approach to intervention implementation. The intervention team, comprised of project and park staff, found that social-learning principles emphasizing clear rules and routines most effectively built on the strengths, addressed the needs, and accommodated the constraints of the park program and setting. Four intervention tools emerged as most acceptable, feasible, and effective, and became the primary tools introduced to new park staff during Phase Two of the research: Group Discussion (GD; Pelham et al., 1997), Good Behaviour Game (GBG; Barrish, Saunders, & Wolf, 1969; Embry, 2002), Peers as Leaders, and Good News Notes (GNN, Rubenstein et al., 2000).

Intervention tools

Group discussion

The GD provides a structured format through which park staff and youth label and define each park rule. For example, ‘Stay in your Assigned Area’ may be defined differently during homework (e.g. a particular seat at a table) versus sports (e.g. the dugout, your position on the field). The GD offers an opportunity to review incentives and transition into afternoon recreation. Although STP counsellors utilise a GD at the start of every new activity or transition, park staff found that complicated given the flow of children in and out of park activities (i.e. children arriving late, others leaving early) and used a ‘rap session’ only at the start of each afternoon instead.

Good Behaviour Game

The GBG is a group-based, contingency-based behaviour management program designed to reward rule following and minimise rule breaking. The GBG has a strong evidence base drawn from decades of school-based research (e.g. Tingstrom, Sterling-Turner, & Wilczynski, 2006). The game begins at the start of an activity with clear activity rules and a bank of points. Rule breaking by any individual child results in the loss of a point for the entire group. If at least one point remains at the end of the activity, children earn small group rewards (e.g. free time). Small activity rewards can accumulate toward larger weekly or monthly rewards. STP counsellors utilise individual behaviour programs (i.e. Daily Report Cards) for each participating camper. Although high staff-to-child ratios made individualised point systems too cumbersome in the park setting, the group format of the GBG was well received.

Peers as leaders

PALS was developed to address two challenges identified by park staff: (1) a high staff-to-child ratio and (2) concerns that older youth in Park Kids (grades 6 to 8) were disengaged. Project staff trained older youth to assist younger peers academically and recreationally while supporting prosocial behaviour. Peer leaders received a 5-hour training related to peer-assisted learning strategies for reading (Fuchs, Fuchs, & Burish, 2000), how to facilitate games and activities for younger children (Pelham et al., 1997), and how to model and encourage prosocial behaviours through praise and social reinforcement. The investigative team is currently examining an expanded PALS intervention for the parks through which middle school youth with conduct problems, paired with socially competent peers, train after school for summer roles as junior camp counsellors.

Good news notes

GNN are small certificates used to reinforce youth for improvement toward specific goals. GNN were especially appealing to park staff because they could be utilised flexibly – daily or weekly; for all students or for targeted students; and to reinforce work habits (e.g. homework completion), emotional regulation (e.g. frustration tolerance), rule following (e.g. following directions), athletic skills (e.g. tumbling) or social skills (e.g. helping a peer; problem solving). GNN provided an alternative to individual point systems for reinforcing individual student effort and communicating progress to parents.

Further developments

Upon conclusion of Phase One at the end of the academic year, project staff utilised the summer months to prepare for Phase Two, which was scheduled to begin the following September. For Phase Two, park district leadership selected six new demographically comparable Park Kids locations within the same region of the city where needs were high and resources low. The project team implemented the four intervention strategies and the service model described above at three randomly assigned intervention sites and evaluated the impact on children’s social and behavioural outcomes compared to three no-intervention comparison parks (Frazier et al., submitted). Staff reports of acceptability and feasibility of implementation revealed high satisfaction and consistent use and perceived utility of the interventions, but low maintenance at follow-up (Lyon, Frazier, Mehta, Atkins, & Weisbach, in press).

Limitations

Two limitations to this work warrant mention. First, as noted in the introduction, this paper describes Phase One in a study examining the application of an empirically-supported treatment for children with disruptive behaviour problems to a park after-school program serving children in urban poverty. We did not systematically collect fidelity or outcome data until Phase Two the subsequent year, and therefore have relied on our meeting minutes, field notes, and narrative records to tell this story. Second, we elected to build our collaboration and intervention around the Summer Treatment Program, which perhaps imposed some artificial constraints on our efforts to support park staff and children after school. The alternative would have been to design a new intervention, building on the strengths and addressing the needs of the park. In fact, we considered both options but decided that the STP provided a useful framework – an evidence-informed set of principles – that we could apply flexibly and that we expect will generalise to other parks and recreational settings providing services to children during out-of-school time.

Summary

This work responds to the need for mental health reform in urban poor communities (Kataoka, Zhang, & Wells, 2002) by integrating mental health promotion into the natural routines and resources of after-school care. Unlike many transportability studies, the goal of this work was not to transport the STP in its entirety to the Chicago Park District or to replace Park Kids with the STP. Instead, our goal was to apply social learning principles and integrate and adapt STP strategies to make them acceptable to and useful for indigenous service providers in the community park setting. In turn, we strived to support park staff to meet the challenges presented by their enrolled youth and enhance the quality and capacity of their after-school program. The intensive collaboration described herein helped both to facilitate those goals and to familiarise the research team with this setting, in which we see tremendous opportunities for mental health promotion for children in urban poverty.

This work formed the foundation for what has since become a long-standing collaboration. Our intensive experience has enhanced our appreciation for the needs and capacities of recreational settings and after school programs, and we are continuing to build a service model in collaboration with the Chicago Park District that embraces the organisation’s mission and program goals, responds to the needs of enrolled youth, builds on strengths of staff and setting, and respects the constraints of resources and funding. We believe the foundation for this ongoing work was laid by the relationships and principles described here.

Key Practitioner Message.

• Despite a growing literature that better represents the patients, providers, and settings of routine care, transporting evidence-based interventions to community care settings is challenging

• Social-learning principles emphasizing clear rules and routines most effectively built on the strengths, addressed the needs, and accommodated the constraints of the park program and setting

• Recreational settings offer opportunities for mental health promotion for children in urban poverty