Using the Interactive Systems Framework in Understanding the Relation between Program Capacity and Implementation in Afterschool Settings

Abstract

The present study uses the Interactive Systems Framework (ISF) to understand how capacity influences the implementation of prevention programs in afterschool settings. Eight afterschool sites received the Good Behavior Game (GBG) intervention, a program designed to foster supportive behavioral management and positive youth behavior. In line with the Prevention Support System component of ISF, the intervention afterschool staff was trained and received weekly on-site support from coaches in implementing the GBG. It was found that GBG implementation was greatest in afterschool programs that rated high on both organizational and community levels of capacity; high scores on only one level of capacity resulted in lower implementation scores. Thus, afterschool sites that were more organized, maintained adequate facilities, and developed strong linkages to individuals or organizations in the community scored higher in implementation fidelity and quality. This study highlights the importance of considering interactions among multiple levels of general capacity in efforts to promote evidence-based practices in afterschool settings.

Afterschool programs have been linked to a variety of positive outcomes for children including improved academic scores, lower rates of behavioral problems and drug use, increased confidence and social development, and increased civic engagement (Durlak, Mahoney, Bohnert, & Parente, 2010; Gottfredson, Gerstenblith, Soulé, Womer, & Lu, 2004; Grossman, Campbell, & Raley, 2007; Tebes, et al., 2007). Findings, however, have not been consistent (Durlak & Weissberg, 2007). The inconsistency of these outcomes has shifted the focus of research to understanding the characteristics of afterschool programs that promote positive child outcomes and to bridging this research with practice in afterschool settings (Hynes, Smith, & Perkins, 2009). Using the Interactive Systems Framework as a heuristic, this study seeks to understand how program capacity influences implementation and the use of evidence-based practices among practitioners in afterschool settings.

The literature on characteristics of effective afterschool programs has emphasized the importance of organizational structure and supportive relationships. Grossman et al. (2007), for example, found that group management and adult support were the two most important characteristics of afterschool programs that shaped positive child outcomes. Group management referred to fair ground rules, ongoing encouragement, and consistent expectations, and was associated with higher levels of engagement and learning. Similarly, youth interpreted adult support and praise as indicators that adults cared about their achievement and wanted them to be successful. Pierce, Bolt, and Vandell (2010) also found that positive staff-child relations in afterschool programs were consistently beneficial for the elementary-aged students, highlighting the importance of adult support in shaping positive behavior outcomes. Lastly, in their review of research, Durlak and Weissberg (2007) found that children who attended afterschool programs that set goals, prepared specific lesson plans, and allotted time for children to develop and practice new social skills had greater self-esteem, more positive social behaviors, and less problem behaviors and drug usage than children who did not attend afterschool programs using these evidence-based practices.

Considering the research on the importance of structure and support in afterschool programs, it is clear that interventions seeking to improve these two core areas are necessary. What is less understood, however, is why some afterschool programs are able to implement evidence-based interventions that can lead to improved child outcomes while other programs may not. The ISF seeks to examine approaches to fostering the broader-based use of empirically-based practices (Wandersman et al., 2008). The goal of the current study is to examine the role of capacity in afterschool programs and its relation to implementation of an evidence-based intervention, the Good Behavior Game (GBG; Kellam et al., 2008) that seeks to increase structure and support in educational settings.

GBG

GBG is an empirically-supported behavior management system that has found to have long-term positive effects for children. In longitudinal research, high-risk youth exposed to GBG in first grade were less likely to use drugs or be aggressive in middle school and in early adulthood (Ialongo et al., 1999; Kellam et al., 2008; Kellam et al., 1994). A unique aspect of this program is that both staff and children work together to set and support high-standards of behavior.

Once behavior standards have been agreed upon, the staff assign children to work in cooperative teams and encourage them to exhibit their best behavior. Throughout the game, staff use praise, encouragement, and the unemotional recording of “spleems” (misbehaviors) to determine how well the children are doing and whether or not they win the game. Initially, games are relatively short in duration and last three to five minutes. Over the course of the implementation period, staff are coached to know when to increase the frequency and duration of the games and eventually introduce an unannounced “secret game” which is meant to help children stay on their best behavior. All teams have the opportunity to “win” the game, and the children receive intangible “prizes” for their good behavior. These prizes consist of privileges and activities that are appropriate for use in afterschool programs (i.e., extra outside time, 30 seconds to make jungle noises, staff have to carry the children’s book bags, etc.). The selection of prizes can be changed on a regular basis to maintain the enthusiasm of both the staff and the children for the game.

Interactive Systems Framework

The Interactive Systems Framework (ISF) is an important tool for bridging research-to-practice in the field of prevention or intervention science. The goal of the ISF is to foster the broad implementation of intervention programs, and because organizational and staffing structures of afterschool are often varied and unique, the ISF is particularly relevant for guiding interventions in these contexts.

The ISF model includes three interactive, bi-directional systems: the Prevention Synthesis and Translation System, the Prevention Support System, and the Prevention Delivery System (Wandersman et al., 2008). The Prevention Synthesis and Translation System focuses on how to make prevention research accessible for practitioners. The Prevention Support System, considered “a key element of the framework” (Wandersman et al., 2008, p. 175), centers on how to facilitate and support implementation through technical assistance and other support. The Prevention Delivery System addresses how interventions are conveyed to and used by a broader audience. The ISF also acknowledges two influential elements in implementing a new program of science in the field: Individual Factors and Organizational Factors. That is, the three systems detailed above could be executed, but the quality of implementation will still depend on several key individual variables such as the education, experience, and attitude of the implementer (Kam, Greenberg, & Walls, 2003), as well as the organization’s capacity for leadership, commitment, structure, and climate (Wandersman et al., 2008).

The concept of capacity and its relation to implementation has been further delineated by Flaspohler, Duffy, Wandersman, Stillman, and Maras (2008). In particular, they distinguish between types (general and innovation-specific) and levels (individual, organizational, and community) of capacity. General capacity encompasses infrastructure, skills, and motivation of an organization, whereas innovation-specific capacity refers to forms of capacity that are necessary in order to adopt and sustain a specific implementation in a novel setting. The authors make a distinction between these two types of capacity; yet, they also acknowledge that there may be instances of overlap. In addition, Flashpohler et al. suggest that within both innovation-specific and general capacity there are three levels: (1) individual level includes staff capability, openness, and background; (2) organizational level includes organizational structure and resource availability; and (3) community level includes linkages to the community and connections among people, organizations, and outside communities and organizations. While we acknowledge that the individual level of general capacity is critical to the implementation of evidence-based practices, this study will focus on the latter two levels, organizational and community, in its relation to GBG implementation. See Figure 1 for the conceptual model used for implementation in this study.

Figure 1. Conceptual model of program capacity used for implementation of GBG in afterschool settings.

*Types (general, innovation-specific) and levels (individual, organizational and community) also apply to the Prevention Synthesis and Translation System and the Prevention Delivery System but were not the focus of this study.

Purpose of Study

The present study uses the ISF to understand how capacity influences the implementation of GBG in afterschool settings. In line with the Prevention Support System of the ISF, staff training and technical assistance were the primary focus of this study’s implementation efforts. Two levels of general capacity were examined: organizational and community. It was hypothesized that afterschool sites with high scores on both levels would demonstrate higher quality implementation of GBG (see Figure 1).

Method

Sample

This pilot study involved 12 elementary-level, school-based afterschool programs located in rural and urban areas of the northeastern United States. The original sample consisted of eight randomly assigned programs. Prior to launching the pilot study, it became clear that four additional, matched, treatment sites were necessary in order to adequately pilot the implementation. Due to budget limitations, however, no additional control groups were added. As such, eight of the twelve programs were assigned as treatment sites and received training and coaching in GBG, and the four remaining sites were assigned as control sites and continued “business as usual.”

The afterschool programs that participated in this pilot study served a total of 677 children: 533 in the eight treatment sites and 144 in the four control sites. Of those children, 61% were Latino, 33% African American, 5% White, and 1% Other. These students attended schools in which 60% of the student population was economically disadvantaged. Table 1 provides the demographics on the children in the treatment and control sites.

Table 1.

Enrollment Demographics (%) for Treatment and Control Sites

Demographics		Treatment	Control
% Grade level
	1st	22%	16%
	2nd	19%	18%
	3rd	18%	22%
	4th	19%	25%
	5th	16%	8%
	Other	6%	11%
% Race/Ethnicity
	African American	33%	34%
	Latino	61%	59%
	White	5%	7%
	Asian American/Pacific Islander & American Indian	1%	0%
	Other	0%	0%

Procedure

Staff Training and Technical Assistance

For the eight treatment-sites only, GBG coaches provided on-site technical assistance to the afterschool staff on a weekly basis for a total of eight weeks. The study’s GBG implementation team included a GBG supervisor who was trained by the developer of PAX-GBG and two GBG coaches who were carefully selected based on their experience working with children and staff in afterschool settings. The GBG supervisor and research team provided a special training for the coaches that included the components of GBG, background readings on GBG and its constructs, and strategies on how to engage and motivate adult learners. The coaches communicated regularly with the GBG supervisor and research team throughout the implementation via bi-weekly conference calls and/or face-to-face meetings to address challenges they were encountering in the field, to learn strategies to address those challenges, and to enhance their overall coaching skills.

Immediately prior to the start of the implementation period, the supervisor and GBG coaches conducted a half-day workshop for the staff in the treatment sites. Following the workshop, the GBG supervisor and coaches were assigned up to three sites each and visited their sites weekly to provide technical assistance for the staff by modeling GBG strategies or simply providing support and consultation to facilitate the implementation of GBG.

Data Collection

Throughout the eight-week implementation period, the GBG supervisor and coaches completed a weekly implementation report for each of their assigned treatment sites. The reports described the frequency and duration of game strategies, the staff’s level of comfort and familiarity with GBG, whether or not prizes were awarded immediately, and how well the staff incorporated advanced aspects of GBG (i.e., youth leadership opportunities).

Measures

General Capacity Measure

Capacity in the afterschool sites was measured by a self-report director survey (Wisconsin Center for Education Research, 2007). The director survey was divided into eight sections: general program information, space and materials, planning time, program-school collaborations, parent communication, community interactions, professional development, and a section on the director’s additional job positions. Each survey took directors approximately 45 minutes to complete and were collected prior to the start of implementation.

Organizational capacity (α = .79) was assessed by summing scores from items of the director’s survey. Items addressed areas such as average number of days/week of operation, average daily hours of operation, care provider/child ratio, the use of a structured curriculum, staff management, professional development, available space, and available materials. Directors indicated the average number of days per week of operation and the average daily hours of operation in two open-ended response questions. The care provider/child ratio item asked directors to choose from the following responses: “1 provider for 5 or fewer children,” “1 provider for 10,” “1 provider for 15,” “1 provider for 20,” “1 provider for 25,” “1 provider for more than 25 children.” Care provider/child ratio responses were reverse-scored and summed. One item inquired whether programs implemented a structured curriculum. Response categories were “yes” or “no.”

Professional development items assessed whether staff were offered opportunities for trainings, workshops, or in-service professional development programs. Possible responses included “yes” or “no.” Available materials items assessed whether programs had materials such as calculators, books, games, a photocopier, reference materials for staff, and physical education equipment. Possible responses to these items were “not available,” “limited,” and “adequate.” Available space items assessed whether programs had access to an art room, science lab, music room, cafeteria, kitchen, gym, storage, staff planning, auditorium, library, parent meetings, general classroom space, and playground. Possible responses were “not available,” “limited,” and “adequate.” Staff management items assessed the frequency of meetings with staff to discuss program details, plan program activities, discuss individual student needs, and other discussions. Possible responses were “bi-monthly,” “monthly,” “once each semester,” or “once each year.” See Table 2 for means, standard deviations, and ranges for organizational capacity by treatment (N=8) and control (N=4) groups.

Table 2.

Means, Standard Deviations, and Ranges of Organizational- and Community-Level Capacity for Treatment, Control, and Treatment and Control Groups

IV=Level of Capacity		Treatment (N=8)	Control (N=4)	Treatment & Control (N=12)
Organizational
	M	62.3	59.3	60.8
	SD	6.6	7.1	6.4
	Actual Range	55-72	49-65	49-71
Community
	M	56.5	56.4	55.0
	SD	12.1	11.9	11.2
	Actual Range	36-72	45-73	35-73

Community capacity (α = .76) was assessed by summing scores from items of the director’s survey. Items addressed areas such as community collaboration, program-school relations, and parent communication. Community collaboration items assessed the number of community collaborators that provided resources such as special programs for children on or off site, special programs for parents on or off site, volunteers for program, regular mentors for children, materials or supplies, funding, referred students to program, and other supports. Possible response choices were “none,” “1,’ “2-4,” and “5 or more.” Program-school relation items assessed the frequency of afterschool staff meetings with the school’s principal or teachers to discuss the following: program-related issues; planning of program content; classroom or shared space; program enrollment; student discipline issues; program staffing; curriculum concepts; homework; and individual student needs. Possible response choices included “never,” “sometimes,” and “regularly.” Parent communication items assessed the frequency with which the afterschool staff met with parents, either individually or on the phone. Possible response choices included “never,” “1-2 times a year,” “at least 2-3 times a year,” “monthly,” “bimonthly,” and “weekly.” See Table 2 for the means, standard deviations, and ranges for communication capacity by treatment (N=8) and control (N=4) groups.

Implementation Measure

The implementation measure was completed by GBG coaches on a weekly basis for eight weeks and consisted of two components: quantity and quality of implementation. Quantity of implementation (α = .50) consisted of nine items and asked the frequency with which staff used important GBG strategies such as behavioral tally sheets, timers, teams, prizes for good behavior, and secret games. Response choices included “yes” or “no.” The quantity of implementation subscale also asked coaches to report the number GBG games played and winning teams observed. Quality of implementation (α = .56) consisted of three questions which asked coaches to report on the program staff’s collective mastery of GBG, openness to learning GBG, and purposeful learning of GBG. Coaches rated program staff according to seven-point Likert scale that ranged from (1) low to (7) high. The low-to-moderate reliabilities for these two implementation subscales may be due to the staggered and sequenced nature of the implementation training schedule and the vast differences in staff’s initial ability to implement innovative strategies.

Results

The following results are divided into two sections. First, we describe the organizational and community capacity of the 12 afterschool programs participating in the study (i.e., 4 control and 8 treatment sites). Then, using the eight treatment sites only, we examine the relationship between capacity and GBG implementation.

General Capacity

Among the 12 afterschool programs in the study, 66.7% of the programs operated five days/week; the remaining programs operated three days/week. Hours per day of operation ranged from two-to-three, where 33% percent of the programs operated three hours/day, 25% programs operated two-hours/day, and the remaining programs operated between two to three hours per day. The care provider/child ratio ranged from one provider per 20 children to one provider per 5 children. In this sample, 66.7% reported a 1:10 caregiver/child ratio, 16.7% reported a 1:15 caregiver/child ratio, 8.3% reported a 1:20 caregiver/child ratio, and 8.3% reported a 1:5 caregiver/child ratio. Approximately 92% (i.e., 11 out of 12) of the afterschool program sites reported the use of a structured curriculum such as GBG or other programs designed to promote socioemotional learning. See Table 2 for the means, standard deviations, and ranges on organizational and community capacity variables for the full sample (N=12).

Implementation

The eight treatment sites in this study were assessed on their level of quantity and quality of GBG implementation. Out of nine possible GBG strategies, treatment sites used on average M= 6.4 (SD= 1.6). Quality of implementation was also assessed across the eight intervention sites (M= 13.1, SD= 2.3, Range = 3-21). Table 3 presents means and ranges of quantity and quality of implementation by site as rated by GBG coaches.

Table 3.

Average Quantity and Quality of Implementation Strategies -Treatment Sites Only (N=8)

Intervention Sites	Quantity of Implementation Mean (Potential Range)	Quality of Implementation Mean (Potential Range)
Site 1	6.0 (0-9)	13.3 (3-21)
Site 2	6.1 (0-9)	9.7 (3-21)
Site 3	6.3 (0-9)	13.0 (3-21)
Site 4	6.5 (0-9)	16.0 (3-21)
Site 5	3.0 (0-9)	12.0 (3-21)
Site 6	7.5 (0-9)	15.5 ( 3-21)
Site 7	7.8 (0-9)	14.2 (3-21)
Site 8	8.0 (0-9)	16.8 (3-21)

Capacity and Implementation

Among the eight treatment sites in this study, we examined the relationship between levels of afterschool program general capacity (i.e., organizational and community) and the implementation of GBG. Mean splits on the organizational and community capacity variables were conducted in order to create two sets of groups: (a) high vs. low organizational capacity and (b) high vs. low community capacity. Dependent variables consisted of coaches’ reports of quantity and quality of GBG implementation. Baseline levels of capacity, including training or technical assistance, were not controlled for in the following analyses.

Two-way analyses of variance were conducted using organizational-level capacity (low or high) and community-level capacity (low or high) as the independent variables and quantity of GBG implementation as the dependent variable. A significant two-way interaction (see Table 4) was found (F(3, 8)= 145.7, p<.001). Programs that rated highly on both organizational and community capacity used more GBG strategies than programs who rated high on organizational capacity and low on community capacity (see Figure 2). Table 5 presents the results for means, standard deviations, effect sizes, and sample sizes for the two-way interaction.

Table 4.

Results from Two-Way Analysis of Variance in which Quantity of Implementation is the Dependent Variable-Treatment Sites Only (N=8)

	Effect
Main Effects:		F	DF	P value	Effect Size
	Organizational	14.6	3,8	.019*	.78
	Community	121.1	3,8	.000***	.97
Interaction:
	Organizational* Community	145.7	3,8	.000***	.97

^*p<.05,

^***p<.001

Figure 2.

Two-Way Analysis of Variance in which Quantity of Implementation is the Dependent Variable (Y Axis) - Treatment Sites Only (N=8)

Table 5.

Means, Standard Deviations, and Sample Size for Two-Way ANOVA: Organizational and Community-Level Capacity × Quantity of Implementation-Treatment Sites Only (N=8)

TYPE OF CAPACITY	LEVEL OF CAPACITY	MEAN	SD	Actual Range	N
Organizational
	Low	6.2	0.2	6.0-6.5	4
	High	6.6	2.4	3.0-8.0	4
Community
	Low	5.5	1.7	3.0-6.5	4
	High	7.3	0.9	6.1-8.0	4

Two-way analyses of variance were conducted using organizational-level capacity (low or high) and community-level capacity (low or high) as the independent variables and quality of GBG implementation as the dependent variable. A significant two-way interaction (see Table 6) was found (F(3, 8)= 10.8, p<.05). Programs that rated highly on both organizational-level and community-level capacity were more likely to be rated higher on their quality of implementation of GBG than programs that rated low on organizational capacity and high on community-level capacity (see Figure 3). Table 7 presents the results for means, standard deviations, effect sizes, and sample sizes for the two-way interaction.

Table 6.

Results from Two-Way Analysis of Variance where Quality of Implementation is the Dependent Variable-Treatment Sites Only (N=8)

	Effect
Main Effects:		F	DF	P value	Effect Size
	Organizational	2.4	3,8	.20	.37
	Community	.15	3,8	.72	.04
Interaction:
	Organizational * Community	10.8	3,8	.03*	.73

^*p<.05

Figure 3.

Two-Way Analysis of Variance in which Quality of Implementation is the Dependent Variable (Y Axis) - Treatment Sites Only (N=8)

Table 7.

Means, Standard Deviations, and Sample Size for Two-Way ANOVA: Organizational and Community-Level Capacity × Quality of Implementation-Treatment Sites Only (N=8)

LEVEL OF CAPACITY	LEVEL OF CAPACITY	MEAN	SD	Actual Range	N
Organizational
	Low	13.0	2.6	9.7-16.0	4
	High	14.6	2.0	12.0-16.8	4
Community
	Low	13.6	1.7	12.0-16.0	4
	High	14.0	3.1	9.7-16.8	4

Discussion

In accordance with the ISF, results from this study suggest that capacity is fundamental for strong implementation of interventions in afterschool programs. Specifically, it was found that GBG implementation was greatest in afterschool programs that rated highly on both organizational and community levels of capacity. High scores on only one level of capacity resulted in lower GBG implementation scores in afterschool programs. Therefore, it may be important for research to assess the interplay among levels of capacity when seeking to increase the use of empirically-based practices in afterschool settings. The findings from this study suggest that considering only one level of general capacity may not be sufficient.

Unlike past research, this study sought to operationalize an often overlooked level of capacity: community capacity. Community capacity involves the ability of afterschool programs to develop the trust and commitment of stakeholders who are an integral part of the program’s ability to maximize positive and desired outcomes. These stakeholders include parents, teachers and principals, local volunteers, and other members of the community. Because positive child outcomes depend on relationships when various stakeholders, honest, authentic multi-level communication is essential and begins when program directors and staff are able to dialogue with parents, teachers, and community leaders about students’ needs (Flaspohler et al., 2008).

Another strength of this study was in its measurement of implementation. GBG implementation was assessed according to quantity and quality of implementation. This allowed for us to examine whether capacity influenced the number of evidence based practices used in programs or the quality with which these interventions were used by staff. Findings suggest that quality and quantity of implementation peak when there is high capacity at both the organizational and community level. Quantity of GBG intervention was lowest when programs rated high on organizational-level capacity and low on community-level capacity. Therefore, high-resourced programs that put forth low effort in connecting with outside entities may not be receptive to investigators or in learning new practices. Quality of GBG implementation was lowest when programs rated low on organizational-level capacity and high on community-level capacity. This finding may be due to the stress associated with a lack of resources such as space and materials that precludes staff from being fully receptive to innovation.

The present study is not without its limitations. First, the amount of time the intervention was introduced to the program sites was limited to an eight-week period. The small sample size (N=8) of implementation sites also may have also limited the power to detect significant relationships. Further phases of the larger project will gather data from additional sites, and the larger total sample will enable stronger conclusions. Second, our measures of capacity and implementation could be strengthened. For example, quality of implementation was assessed through the coaches’ ratings of staff GBG mastery for a particular amount of time. GBG staff members may have been observed at a time when their level of GBG implementation was low, when in fact, their GBG implementation skills are normally high. Also, the quality of implementation measure used in this study did not account for individual differences across staff members in their quality of GBG implementation. In the full study that is undergoing, coaches and afterschool staff will provide qualitative information to assess implementation. Lastly, the individual level of general capacity such as attitude, education, and experience of afterschool program staff clearly play a critical role in the delivery of innovative strategies with children during the afterschool hours and should be considered in future research (Flaspohler et al., 2008; Wandersman et al., 2008).

The current study marks one of the first attempts to introduce the GBG intervention, an evidence-based program that seeks to strengthen structure and support, to afterschool programs. Since the majority of research to this point has examined GBG in school classrooms (Kellam et al., 2008), the ISF was a particularly useful heuristic in our efforts of increasing empirically-based practices in the novel setting of afterschool. In line with the Prevention Support System of the ISF, GBG implementation efforts were focused on providing ample and regular technical assistance to staff. However, as indicated by the ISF, the next step in the research-to-practice model is to extend generalizability through delivery and greater practitioner utilization.