After-School Programs and Children’s Mental Health: Organizational Social Context, Program Quality, and Children’s Social Behavior

Abstract

Objective.

The current study examined associations among organizational social context, after-school program (ASP) quality, and children’s social behavior in a large urban park district.

Method.

Thirty-two park-based ASP are included in the final sample, including 141 staff and 593 children. Staff reported on organizational culture (rigidity, proficiency, resistance) and climate (engagement, functionality, stress), and children’s social skills and problem behaviors. Youth and their parents reported on program quality indicators (e.g., activities, routines, relationships). Parents also completed a children’s mental health screener. A series of Hierarchical Linear

Results.

Models revealed that proficiency and stress were the only organizational predictors of program quality; associations between stress and program quality were moderated by program enrollment and aggregated children’s mental health need. Higher youth and parent perceived program quality related to fewer staff-reported problem behaviors, while overall higher enrollment and higher aggregated mental health need were associated with fewer staff-reported social skills.

Conclusions.

Data are informing ongoing efforts to improve organizational capacity of urban after-school programs to support children’s positive social and behavior trajectories.

An estimated 10.2 million US children (18%) in grades K-12 utilized after-school programs (ASP) in 2014 (Afterschool Alliance, 2014), with low-income children spending more hours per week than peers from higher income families (Capizzano, Tout, & Adams, 2000). School-based programs mostly provide tutoring, while neighborhood programs offer recreation, sports, arts and cultural activities. Despite wide variability in enrollment, activities, staff and training (Bouffard & Little, 2003), fostering healthy development remains central to the mission of most ASPs, where routines and activities are designed to promote social skills, facilitate peer relations, and enhance social emotional learning (Frazier, Cappella, & Atkins, 2007; Gottfredson, Gerstenblith, Soulé, Womer, & Lu, 2004). ASP are especially critical in communities of concentrated urban poverty, where needs are daunting, costs of unproductive time are profound, and benefits of ASP participation are especially strong. Decades of research reveal robust psychosocial and academic benefits of organized ASP, particularly for low-income urban youth (e.g., Durlak, Weissberg, & Pachan, 2010; Mahoney, Lord, & Carryl, 2005). The current study is part of collaboration with urban park district ASP focused on mitigating risks and supporting healthy youth trajectories.

Benefits to Children Vary with Program Quality

Benefits to youth of ASP participation rely on high quality developmental experiences that: (1) are characterized by a supportive environment, (2) occur within structured interaction between adults and youth, and (3) incorporate opportunities for meaningful and reflective engagement by participating youth (see Smith, Peck, Denault, Blazevski, & Akiva, 2010). Early research in low-income, urban communities illustrated the importance of staff-child relationships for both young children and adolescents (Roffman, Pagano, & Hirsch, 2001; Rosenthal & Vandell, 1996), and more recent “best practice” guidelines highlight the foundation that positive relationships create for subsequent skills-building opportunities and development of social competencies (see Ollhoff & Ollhoff, 2012). Recent longitudinal analysis from the NICHD Study of Early Child Care and Youth Development highlights that children’s positive experiences with ASP staff were associated with parental reports of fewer externalizing behaviors, as well as increased assertion and self-control skills, even after controlling for child and family level covariates (Wade, 2015).

Quality of ASP experience also has been linked to emotional, behavioral and academic outcomes. Among early elementary school samples, both staff-child and peer relationships contributed to children’s program satisfaction, academic performance (Pierce, Bolt, & Vandell, 2010), behavioral functioning, and social skills (e.g., Shernoff, 2010; Vandell et al., 2005). This effect holds true for older youth as well, with positive program experiences – including peer relationships, emotional support from adults, opportunities for autonomy – related to gains in work habits, task persistence and prosocial behavior reported by classroom teachers (Kataoka & Vandell, 2013). There is some evidence that boys demonstrate more social-emotional gains than girls (Pierce et al., 2010; Wade, 2015). Programs emphasizing social skills and character development have greater impact on reducing delinquent behavior among middle school students than programs without social development goals (Gottfredson et al., 2004). Durlak et al.’s (2010) meta-analysis of 68 ASP revealed that higher-quality programs with sequenced, active and explicit skills training, led to improvements in behavior and school achievement.

Other studies have examined staff and setting characteristics that influence program delivery and children’s outcomes. Most compelling, Wade (2015) identified four effects on child-level outcomes related to staff-level variables (based on staff-reported social skills and parent-reported behavior): (1) fifth grade children had higher self-control ratings in programs staffed by those with more training in child development; (2) fourth graders had more positive relationships with after-school caregivers and higher assertion ratings when staff had more experience; (3) first graders had lower levels of parent-reported internalizing and externalizing behaviors when staff received higher wages – this relationship held at third grade for externalizing behaviors only; and (4) lower child-to-staff ratios were associated with more positive relationships with after-school caregivers and higher assertion scores.

Potential benefits of program participation may be compromised in underrepresented and underserved communities, where human and material resource limitations (space, staffing, equipment) and elevated health and safety concerns (for staff and youth) may interfere with reliable program offerings and high-quality implementation. A small but growing literature has examined efforts to train and support after school staff around academic enrichment, coaching behaviors, activity engagement and behavior management (e.g., Frazier, Chacko, Van Gessel, O’Boyle, & Pelham, 2012; Helseth & Frazier, 2018; Smith et al., 2014), in particular for programs serving communities characterized by poverty and violence. Studies have revealed strong evidence for feasibility of intervention, high staff satisfaction with consultation, and encouraging though modest improvements in children’s adaptive functioning ( Frazier, Mehta, Atkins, Hur, & Rusch, 2013). Sustaining such practices in the absence on ongoing support has proven challenging ( Lyon, Frazier, Mehta, Atkins, & Weisbach, 2011), however, supporting mounting data that suggests a direct focus on setting-level social processes may be necessary to impact the effectiveness and sustainability of evidence-based interventions by front-line staff.

Organizational Influences on Service Quality

Pivotal work in child welfare and juvenile justice indicates that the social context of an organization contributes uniquely to delivery of high quality services and outcomes for children (Glisson & James, 2002). A rich literature demonstrates that an organization’s effectiveness (i.e., service quality and client outcomes) depends on how well its organizational social context (OSC) – in particular organizational culture and climate – supports the objectives and implementation of the core service technology (Hemmelgarn, Glisson, & James, 2006).

Organizational culture is defined as behavioral norms and expectations, shared by members of a work unit, and used to guide work performance and socialization of new staff in priorities of the organization (e.g., the way we do things). Rigid cultures are characterized by expectations that employees have little flexibility, provide limited input into key management decisions, and carefully follow a host of bureaucratic rules and regulations. Proficient cultures expect that workers behave competently, maintain up-to-date knowledge, and place the wellbeing of clients first. Resistant cultures have workers who show little interest in adopting new practices, and suppress efforts to change, improve, or integrate innovative technologies.

Psychological climate is an individual’s perception of the psychological impact of the work environment on his or her own functioning and well-being (e.g., stress; James & James, 1989). When members of a work unit agree on the extent to which their work environment is stressful, their shared perceptions may be aggregated to describe an organizational climate (e.g., the work environment is characterized as healthy or unhealthy; Jones & James, 1979; Joyce & Slocum, 1984). Engaged climates are formed when employees perceive that they are able to personally accomplish many worthwhile tasks and remain personally involved with their work and their clients. Functional climates are formed when employees perceive they have the help and cooperation needed from coworkers and administrators to do a good job and have a clear understanding of how they can work successfully within the organization. Stressful climates are formed when employees perceive that they are emotionally exhausted from or overloaded in their work and are unable to complete expected tasks associated with their role and function.

Data from several studies and settings document associations among readiness to change, climate, staff workload, and client engagement (e.g., Landrum, Knight, & Flynn, 2012). A national survey of 100 community mental health clinics revealed that proficient cultures were associated with sustaining new service technologies and positive climates were associated with lower staff turnover as compared to clinics with rigid or resistant cultures, or negative climates, respectively (Glisson, Schoenwald, et al., 2008). In other data, proficient cultures and positive climates both predicted higher job commitment and satisfaction (Glisson, Landsverk, et al., 2008). Indeed, studies of human service organizations robustly demonstrate that organizational support for adopting evidence-based recommendations for service delivery has been associated with positive staff reports of climate and culture (e.g., Aarons & Sawitzky, 2006), including a stronger sense of the organization’s mission and perceived opportunities for professional development (Fuller et al., 2007).

Accumulating data suggest that improvements in organizational climate and staff commitment to organizational mission are necessary for sustaining high-quality service delivery (i.e., improvement in work attitudes creates a more effective workforce). To this end, several models for organizational intervention in human services demonstrate the malleability of organizational characteristics and opportunity to improve service delivery via improvements to climate and culture (e.g., Aarons, Ehrhart, Moullin, Torres, & Green, 2017; Glisson & Schoenwald, 2005). The current study sought to identify organizational influences on program quality and outcomes for youth, and in turn levers for intervention by which to enhance the mental health promoting benefits of children’s after-school program participation.

What We Know and What We Don’t Know

To summarize, we know that ASP can contribute to promoting life skills development and positive adjustment outcomes, in particular for children from low-income communities. We know that benefits to children rely on high quality program indicators including positive relationships between youth and with staff, consistent routines, and activities including homework assistance and recreation that facilitate skills such as task persistence, frustration tolerance, and problem solving. We also know that in historically disenfranchised and underserved communities characterized by resource poverty, food and housing insecurity, and unemployment, potential benefits of program participation may be compromised by inadequate resources, frequent staff turnover, and extensive mental health needs among enrolled children. What we don’t know, however, is the extent to which organizational social context, namely organizational climate and culture, influence program delivery and outcomes in urban ASP as has been demonstrated in specialty mental health settings such as child welfare and juvenile justice. To our knowledge, the current study is the first examination of organizational social context in a large urban after-school setting, and the first to examine associations among organizational influences, child and family experiences, and children’s outcomes in ASP.

The Current Study

The current study examined associations among staff-reported organizational social context, child and caregiver perceptions of program quality, and children’s social behavior. A series of Hierarchical Linear Models (Raudenbush & Bryk, 2002) were used to test the following a priori hypotheses: Hypothesis 1. Organizational culture and climate (predictors) will be significantly and positively associated with parent- and child-perceived program quality (outcomes), such that programs characterized more by constructive cultures (i.e., flexible, proficient, open to innovation) and positive climates (i.e., engaged, functional, healthy) will be perceived as higher quality as compared to programs characterized more by destructive cultures or negative climates and Hypothesis 2. Parent- and child-perceived program quality (predictors) will be significantly associated with children’s staff-reported social behaviors (outcomes), in a positive direction with social skills and in a negative direction with problem behaviors. That is, staff will report higher social skills and fewer problem behaviors for children enrolled in programs perceived as higher quality by parents and children. We also examined program enrollment and aggregated mental health need of participating children as potential moderators.

Method

Setting

This research reflects a decade-long collaboration with a large Midwestern urban park district that offers daily after-school (3:00 – 6:00) recreational programming (including homework time and physical education) in 94 parks around the city for children in grades K-8. Children rotate through homework, physical education, and recreation activities according to grade-level groups (e.g., K-2, 3–5, 6–8). There is no particular academic assistance, recreation, physical activity, or life skills curriculum implemented. Instead, programs routinely divide time equivalently across supervised homework (variability across programs in regard to resources and assistance available), sports (variability reflects the season, intramural opportunities, expertise of instructors and preferences of enrolled youth), and recreation (including a combination of indoor and outdoor competitive and cooperative games). Programming is publicly funded, though families also pay a modest fee on a sliding scale for each 10-week session. Enrollment varies substantially across parks (range 10 to 100 children).

Sample

All 94 parks offering daily afternoon recreational programming were eligible to enroll, and forty-four (47%) participated. Programs that declined cited several reasons, most often that they had too few staff or program activities, corresponding to a small enrollment. Indeed, size differentiated participating and non-participating programs; there were no other salient differences (e.g., demographics of neighborhoods or families served). All children attending participating programs were eligible to participate. Total study enrollment included N=768 children (52% of approximately 1,470 who were enrolled in eligible programs; non-participating families either did not attend recruitment events because they were unavailable or disinterested, attended recruitment events but declined to participate, or attended recruitment events and accepted information about the study but never returned to participate). All staff contributing to overseeing or delivering daily afternoon homework or recreation programming were eligible to participate, including park supervisors, physical coaches, recreation leaders, and auxiliary staff. Total study enrollment included N=178 staff (75% of approximately 237 employed and eligible; staff cited several reasons for non-participation, mostly related to time, burden, or disinterest in research). We also recruited children attending therapeutic recreation ASP from three park sites (n=4 in final sample). Ten children declined assent to complete research measures and nine children were omitted from the final sample due to ineligibility, concerns about consent, or active participant withdrawal.

Most (53.8%) of parents enrolled more than one child in the study. We received partial or complete data for N=728 children who were 52.2% female, 51.9% African American, 18.5% Latino, 6.2% White, 2.2% Asian or Pacific Islander, and 5.6% Other race/ethnicity (15.5% not reported). Children ranged in age from 4 to 14 years old (M = 8.95, SD = 2.19) with 36.5% in grades K-2 (n=266), 47.0% in grades 3–5 (n=342), and 16.5% in grades 6–8 (n=120). A total of 37.2% lived in single-parent households and 19.1% of parents reported being unemployed. The average household size was 4.26 people with 29.1% of families earning less than $25,000 per year and 58.5% of children receiving free or reduced-price school lunch. In addition, 15.8% of children had at least one foreign-born parent and 22.3% spoke languages other than English at home. A total of n=63 parents elected to complete research measures in Spanish. We obtained partial or completed data for a total of N=163 staff, including 40 park supervisors, 52 instructors (full-time physical education coaches or arts and crafts instructors), and 48 recreation leaders (part-time staff), and 23 auxiliary staff (e.g., special arts/cultural staff, attendants). Staff self-identified as 58.4% female, 41.3% Black/African American, 25.6% White, and 23.1% Latino.

Measures

Family Survey.

Parents reported on parent (e.g., education, employment), family (e.g., income, language, country of origin), and child (e.g., age, grade, school) demographics.

Program Staff Survey.

Staff reported basic demographic information (e.g., race, ethnicity, gender), level of education, professional training and work experience. Park supervisors provided additional information related to program structure, enrollment, funding, and staff employment as well as program space and materials (27 items), team meetings (3 items), relationships with administrators and teachers at partner schools (26 items), relationships with parents of participating children (4 items), and linkages with other organizations (9 items). Supervisor-reported Park Enrollment ranged from 11 to 102 children (M = 33.88, SD = 19.25).

Organizational Social Context.

The Organizational Social Context (OSC) Measurement System, based on Glisson’s (2002) theoretical model, assesses staff perceptions of key dimensions of their organization. The measure consists of 105 items to which staff respond utilizing a 5-point Likert scale (1=not at all to 5=to a very large extent). The OSC has been used to assess organizational culture and climate in mental health, child welfare, juvenile justice and other human service systems nationwide. Confirmatory factor analysis on two U.S. national samples of social and mental health service organizations respectively (Glisson, Landsverk, et al., 2008; Glisson, Green, & Williams, 2012) revealed 16 first order factors (e.g., Formalization, Emotional Exhaustion, Role Clarity, Organizational Commitment; αs range from .71 to .92), 7 second order factors (e.g., Rigidity, Resistance, Engagement, Morale; αs range from .78 to .94) and fit indices of .96 for both CFI and IFI, .95 for NFI, and an RMSEA of .051, indicating acceptable fit. These samples are the basis for national OSC norms used in developing culture and climate profiles of organizations. Organizational culture and climate measured by the OSC have been associated with a variety of staff attitudes and behavior, service quality and service outcome criteria in both survey and experimental research (e.g., Aarons, Glisson, Green, Hoagwood, Kelleher, & Landsverk, 2012; Glisson, Schoenwald, et al., 2008).

The number of staff per park ranged from 1 to 6. A total of 163 OSCs were received from 42 parks. Of these, 22 OSCs were discarded due to response inconsistencies or insufficient site-level data (e.g., only 1–2 staff per park or missing data exceeded 10%), resulting in 12 parks being discarded because they did not have the minimum number of participants (n = 3 or more) required to compute aggregated culture and climate scores. Organizational culture and climate scores were computed by aggregating across front-line staff within each park on six primary dimensions (Culture: rigidity, proficiency, resistance; and Climate: stress, functionality, and engagement). Acceptable response agreement was calculated using the r_wg statistic (James, Demaree, & Wolf, 1984). This resulted in 14 OSC dimension scores within parks with r_wg values below the acceptable cutoff of .7 excluded from the analysis. Individual dimension scores were discarded based on this cutoff; acceptable OSC dimension scores within parks were retained.

Parent-Reported Program Quality.

Parent perceptions of program quality were assessed with 20 items related to satisfaction with program activities, environment, and staff-child relationships (1 = strongly disagree to 5 = strongly agree). Six items come from a child care quality scale (O’Connor, 1991), three items from a peer network characteristics measure, Kids with My Kid (Vandell et al., 2005), and 10 items from a parent satisfaction scale (Rosenthal & Vandell, 1996). Principal factor analysis (n = 468) revealed a one-factor solution (α =.89), and the current sample revealed high internal reliability with (n = 499 parents; α =.89).

Child-Reported Program Quality.

Children in 3^rd - 8^th grade reported perceptions of program quality (1 = never to 4 = always) on the 36-item After-school Environment Scale (ASES; Rosenthal & Vandell, 1996). Three factors include: Emotional Support (19 items, α =.95), Autonomy/Privacy (6 items, α =.79), and Peer Affiliation (6 items, α = .80). Scales were moderately correlated (r = .33 to .57) and test-retest reliability ranged from .70 to .91. The current sample (n = 314) yielded high internal reliability for the total scale (α =.88) and maintained the 3-factor structure (Emotional Support α =.88; Peer Affiliation α =.74; Autonomy/Privacy α =.63). The Total Score was used in current analyses.

Aggregate Mental Health Need.

Parents reported on child mental health risk using the Strength and Difficulties Questionnaire (SDQ), a screening tool designed for children ages 4–10 and 11–17 which consists of 25-items rated along a 3-point scale (0=not true to 2=certainly true; Goodman, 2001; Mathai, Anderson, & Bourne, 2002). A Total Difficulties score was computed as the sum of scores across all items on four clinical problem scales: hyperactivity/inattention, emotional problems, conduct problems, and peer relationship problems. Psychometric properties reflect strong internal consistency (α = .82 parent version for Total Difficulties) and test-retest reliability (4 to 6 months mean r = .62; Goodman, 2001). For the current sample, internal consistency for Total Difficulties for 4 – 10-year olds was α = .82 and Total Difficulties for 11–17-year olds was α = .84. The average percentage of children with borderline or higher Total Difficulties scores across all parks in the sample was 7.3% (SD = 5.7, range 0% to 22.5%).

We computed a cumulative impact score of children’s mental health need to represent a potential (but understudied) influence on program quality and likely source of variability across programs. Specifically, we calculated the percentage of children in our sample at each park with Total Difficulties scores at or above the borderline range (i.e., >14), relative to total Park Enrollment, and included it as a park-level covariate in all analyses.

Child Social Behavior.

Staff identified the physical instructor or recreation leader at their park that knew each participating child best to report on observed social behaviors. The designated staff member completed the Social Skills Improvement System (Gresham & Elliott, 2008), reporting on children’s Social Skills (46 items related to cooperation, communication, assertiveness, responsibility, empathy, and engagement, and self-control; α = .98) and Problem Behaviors (30 items related to externalizing and internalizing behaviors, bullying, and hyperactivity/inattention; α = .97) along a 4-point scale (0=never to 3=always). The SSIS has high internal consistency, test-retest and inter-rater reliability, and external validity with established measures of social, emotional and behavioral functioning.

Procedure

This study was conducted in accordance with APA Ethical Guidelines and approval from the university IRB for recruitment, informed consent, and data collection procedures.

Participants were recruited from ASP offered by the partnering urban park district. Recruitment began with meetings of park leadership (e.g., superintendent, program directors), regional managers and area managers to discuss goals, objectives, and procedures of the planned work, followed by meetings within areas among park supervisors to introduce the study, answer questions, and plan for staff and family recruitment. We hosted informational meetings at all but one of 17 areas across the three regions of the city from 2009 to 2011. Following meetings, the project coordinator followed up with individual park supervisors regarding their interest in participating, which most often depended on their size and enrollment, corresponding availability of after-school program offerings at the time of recruitment, and to a smaller degree, the extent to which they perceived families may be interested and available for recruitment events and study procedures. Research staff spent one week at each participating park to recruit participants and collect data during the park district’s fall, winter, or spring program sessions.

Families were invited to attend Family Night informational sessions, during which they received detailed information about the study, an opportunity to ask questions, and an invitation to enroll in the research. Consented parents completed research measures at Family Night or returned their packets to research staff during the week-long data collection. Parents received $20 for each child data packet they returned. Children assented separately and completed measures (with assistance from research staff as needed) during a week of data collection at their park. All program staff members were eligible to consent; they completed research measures either at a pre-arranged staff meeting designated by the park supervisor (with lunch in lieu of compensation, at request of the organization) or individually during or outside of program hours.

Data Analytic Plan

Data were cross-sectional and clustered, such that child data (Level 1) was nested within park ASP (Level 2), indicating the need for an analytic approach that considers the increased correlations among nested individuals (see Raudenbush & Bryk, 2002). A series of hierarchical linear models (HLMs) was used to examine how organizational culture (rigidity, proficiency, resistance) and climate (stress, functionality, and engagement) dimensions (predictors) each related to parent- and child-perceived program quality (outcomes); an additional series of HLMs was used to examine how perceived program quality (predictors) related to staff-reported child social skills and problem behaviors (outcomes). HLM was chosen because it explicitly models clustered data by partitioning variance attributable to clusters (here, parks) and allows for examination of cluster-specific effects (Raudenbush & Bryk, 2002), thereby providing a richer context for our conclusions. Notably, while we refer to “predictors” and “outcomes” per HLM conventions (Cohen, Cohen, West & Aiken, 2003), analyses yield information about associations between constructs, but do not imply directionality or causality. All analyses were conducted in SPSS version 21 (IBM Corp., 2015).

As noted above, 12 parks were excluded due to missing or insufficient site-level data. Of the remaining 32 parks, missing data on the remaining predictor and outcome variables were extensive, ranging from 1.2 % to 43.1%, reflecting conventional challenges associated with collecting community-based data from multiple sources. Missing value analysis was conducted for all variables to be included in analyses and all other child, parent, and park level variables in the dataset. Missing values were not related to other variables in the dataset; thus, we proceeded under the assumption that missing values were missing at random for all analyses. Due to widespread missingness at the park, parent, and child levels – attributed primarily to staff and child absences during data collection and limited opportunities to follow-up with parents) – we were unable to utilize widely-cited recommendations (e.g., multiple imputation) for multilevel data (Mistler, 2013). Expectation maximization is robust to this amount of missing data and was used to impute missing values (Graham, 2009). Park-level variables (i.e., Aggregate MH Need, Park Enrollment, OSC dimensions) were homogenized across parks following imputation.

Several park-level variables were included as covariates to account for observed heterogeneity across parks. Specifically, we controlled for Park Enrollment, Aggregate MH Need score, and dummy codes for park district Region (i.e., Central versus North = Region 1, and South versus North = Region 2). Note that although parks were nested within regions, we did not include region as a level 3 predictor because there were only three; recommendations vary from 10 to 50 for the minimum number of clusters in a level (Hox, 1998; Kreft, 1996).

Hypothesis 1: Organizational culture and climate (predictors) will be significantly and positive associated with parent- and child-perceived program quality (outcomes).

For models examining associations between OSC dimensions and parent- or child-reported Program Quality, OSC predictor variables were grand mean centered (Peugh & Enders, 2005), allowing for unbiased estimates of level 2 effects. These analyses focused on the effect of level 2 (park) OSC predictors. Each HLM utilized a random intercept and fixed slope, reflecting mean differences between parks on the outcome but no differences between parks in the relationship between predictor and outcome. Each OSC dimension (i.e., Rigidity, Proficiency, Resistance, Engagement, Functionality, and Stress) was examined separately for Parent and Child-reported Program Quality. Park Enrollment, Aggregate MH Need, and Regions 1 and 2 were level 1 covariates. OSC dimension was the level 2 predictor. Cross-level interactions between each OSC dimension and Park Enrollment and Aggregate MH Need were included in each model.

Hypothesis 2: Parent- and child-perceived program quality (predictors) will be significantly associated with children’s staff-reported social behaviors (outcomes), in a positive direction with social skills and in a negative direction with problem behaviors.

For models examining the relationship between Parent- or Child-reported Program Quality and child social behavior (i.e., Social Skills and Problem Behavior subscales on the SSIS), the perceived program quality predictor variables were cluster mean centered; cluster mean centering (Peugh & Enders, 2005) allows for unbiased estimates of level 1 effects. The primary interest in this subset of the analyses was the effect of level 1 (Parent- or Child-reported Program Quality) predictors. Each HLM utilized a random intercept and a fixed slope, reflecting mean differences between parks on the outcome but no differences between parks in the relationship between the predictor and the outcome. Parent- and Child-reported Program Quality were examined separately for the Social Skills and Problem Behavior Scales of the SSIS. Again, Park Enrollment, Aggregate MH Need, and dummy codes for Region were level 1 covariates. Parent- or Child-reported Program Quality was the level 1 predictor. Cross-level interactions between program quality and Park Enrollment and Aggregate MH Need were also included in each model. Significant interactions were probed using methods described in Bauer & Curran (2005) and Preacher, Curran, & Bauer (2006) at covariate values of −1 standard deviation (i.e., Low), Mean, and +1 standard deviation (i.e., High), except where noted. Park Enrollment values were 14.63 children (Low), 33.88 children (Mean), and 53.13 children (High), while Aggregate MH Need values were 1.6% (Low), 7.3% (Mean), and 13% (High) of children at the park with borderline or higher SDQ scores.

Results

Intraclass correlations (ICCs) and design effects were calculated for all outcome variables. ICCs reflect the proportion of variance attributable to level 2 (park); values range from 0 to 1, with higher values indicating that more variance is attributable to the park level. ICCs ranged from 4% to 99.9%. Specifically, more than 99.9% of the variability in both the Child- and Parent-reported Program Quality measures were due to differences between parks. In contrast, 4% and 5% of the variability in SSIS Social Skills and Problem Behaviors, respectively, were due to differences between parks, meaning that more than 95% of the variability on both subscales was accounted for by differences between children enrolled at the same park. Design effects reflect the degree to which standard errors would be underestimated if clustering were ignored; values greater than 1 indicate that ignoring clustering could result in spuriously significant results. All design effects for the aforementioned measures were greater than 1, thereby indicating the need to account for clustering in the analyses (Child-reported Program Quality was 18.5, Parent-reported Program Quality was 18.5, SSIS Social Skills was 1.70, and SSIS Problem Behaviors was 1.91). The magnitude of these design effects justifies our use of mixed models to correctly and separately account for child (level 1) and park (level 2) variance.

Hypothesis 1: Organizational culture and climate (predictors) will be significantly and positive associated with parent- and child-perceived program quality (outcomes).

The results of mixed models relating OSC dimensions to program quality are summarized in Table 1. Across the six dimensions, OSC Proficiency and Stress were the only significant predictors of perceived program quality. There was a positive relationship between OSC Proficiency and Parent-reported Program Quality (b = 0.17, p < .05). For both Parent- and Child-reported Program Quality, the effect of OSC Stress was conditional; the effect of OSC Stress on Parent-reported Program Quality was moderated by Aggregate MH Need, and the effect of OSC Stress on Child-reported Program Quality was moderated by both Aggregate MH Need and Park Enrollment.

Table 1.

Mixed model regression coefficients for the relationship between OSC dimensions (predictor) and reported program quality.

	OSC				Park SDQ	Park Enrollment	OSC x Aggregate MH Need	OSC x Park Enrollment
	Intercept	dimension	Region 1	Region 2
OSC Predictors of Parent-reported Program Quality:
OSC Rigidity	79.55**	0.02	−1.48	2.05	−0.19	−0.06	0.01	0.01
OSC Proficiency	80.42**	0.17*	−2.21	1.59	−0.17	−0.02	−0.01	0.00
OSC Resistance	79.83**	0.17†	−2.04	1.26	−0.28†	−0.09†	0.00	0.01†
OSC Engagement	81.05**	0.06	−1.31	0.45	−0.12	0.03	−0.04†	0.00
OSC Functionality	80.01**	0.15	−2.70	2.52	−0.18	−0.04	−0.01	0.00
OSC Stress	79.21**	−0.02	−1.60	3.21	−0.26	−0.05	0.02*	0.00
OSC Predictors of Child-reported Program Quality:
OSC Rigidity	100.24**	−0.07	−2.72	3.79	−0.26	−0.16*	0.03	0.00
OSC Proficiency	100.19**	−0.15	−1.97	2.80	−0.32	−0.14*	−0.04	0.00
OSC Resistance	101.35**	0.08	−3.58	−0.07	−0.19	−0.15*	−0.02	0.01
OSC Engagement	102.06**	−0.22†	−3.73	−1.08	−0.14	−0.13†	−0.05	0.00
OSC Functionality	100.24**	0.01	−2.87	2.72	−0.34	−0.16*	−0.03	0.01
OSC Stress	99.78**	−0.01	−1.83	4.42†	−0.41*	−0.20**	0.03*	−0.01*

Note: Rows correspond to separate mixed models for each dimension of the OSC as a predictor of Parent- or Child-reported Program Quality, with Region (to account for clustering), Aggregate MH Need, and Park Enrollment included as level 1 covariates. All level 1 (individual) and level 2 (park) residual variance values were significant at p < .05. OSC = Organizational Social Context measurement system, Region 1 = Central versus North, Region 2 = South versus North, Parent-reported Program Quality = Parent Satisfaction with ASP, Child-reported Program Quality = After-school Environment Scale (ASES) total score.

^†p < .10

^*p < .05

^**p < .01

Aggregate MH Need moderated the relationship between OSC Stress and Parent-reported Program Quality. At parks with mean and high Aggregate MH Need values (7.3% and 13% or more youth meeting SDQ cutoffs, respectively), there was no relationship between OSC Stress and Parent-reported Program Quality; simple slopes were 0.11 and −0.02, respectively, all ps >.10. However, for parks with low Aggregate MH Need values (1.6% or less youth meeting SDQ cutoffs), there was a marginally significant negative relationship between OSC Stress and Parent-reported Program Quality; simple slope was b = −0.14, p < .10. At the lower bound for Aggregate MH Need, where 0% of youth met SDQ cutoffs (i.e., 4 parks), the negative slope was b = −0.18, p < .05.

The relationship between OSC Stress and Child-reported Program Quality was moderated by both Aggregate MH Need and Park Enrollment. For sites with low Aggregate MH Need (1.6% or less youth meeting SDQ cutoffs) and high Park Enrollment (≥53 youth), the relationship between OSC stress and Child-reported Program Quality was significantly negative (b = −0.48, p < .05). For sites with mean or high Aggregate MH Need (7.3% and 13% or more youth meeting SDQ cutoffs, respectively) and low Park Enrollment (≤14.5 youths), the relationship between OSC Stress and Child-reported Program Quality was significantly positive (b = 0.27, p < .05 and b = 0.45, p < .01, respectively). All other slopes (from 3 levels of enrollment x 3 levels of SDQ) were not significantly different from 0.

Hypothesis 2: Parent- and child-perceived program quality (predictors) will be significantly associated with children’s staff-reported social behaviors (outcomes), in a positive direction with social skills and in a negative direction with problem behaviors.

The results of the mixed models relating program quality to children’s social behavior are summarized in Table 2. Both Parent- and Child-reported Program Quality predicted SSIS Social Skills and Problem Behaviors; the association was moderated by both Aggregate MH Need and Park Enrollment. The relationship between Program Quality and SSIS Problem Behaviors was significantly negative for both Child- (b = −0.24, p < .01) and Parent-reported Program Quality (b = −0.35, p < .01), indicating that higher perceived program quality was related to fewer problem behaviors.

Table 2.

Mixed model regression coefficients for associations between parent- and child-perceived program quality (predictors) and children’s staff-reported social behaviors (outcomes)

Outcome	Intercept	PQ	Region 1	Region 2	Park SDQ	Park Enrollment	PQ x Park SDQ	PQ x Park Enrollment
SSIS Social Skills
Child-reported PQ	102.83**	0.49**	−4.41	−2.94	−0.18	−0.03	0.02	0.00
Parent-reported PQ	102.83**	0.66**	−4.41	−2.95	−0.18	−0.03	−0.04*	−0.01**
SSIS Problem Behavior
Child-reported PQ	16.75**	−0.24**	−1.67	−2.64†	0.44**	0.03	0.00	0.00
Parent-reported PQ	16.75**	−0.35**	−1.67	−2.62†	0.44**	0.03	0.00	0.00

Note: Each row corresponds to separate mixed models for Parent- and Child-reported Program Quality as a predictor of SSIS Social Skills and SSIS Problem Behavior on the Social Skills Improvement System (SSIS), with Region (to account for clustering), Aggregated MH Need, and Park Enrollment included as level 1 covariates. All level 1 (individual) residual variance values were significant at p < .05, while level 2 (park) residual variance values were not. PQ = Program Quality Region 1 = Central versus North, Region 2 = South versus North, Aggregated MH Need = percentage of child participants above the SDQ cutoff, Park Enrollment = children enrolled per park, Parent-reported Program Quality = Parent Satisfaction with ASP, Child-reported Program Quality = After School Environment Scale (ASES) total score.

^†p < .10

^*p < .05

^**p < .01

The relationship between Parent-reported Program Quality and SSIS Social Skills was moderated by both Aggregate MH Need and Park Enrollment (Figure 1). For parks with lower enrollment (≤14.5 youths), the relationship between Parent-reported Program Quality and SSIS Social Skills was significantly positive, regardless of Aggregate MH Need level (b = 1.53, p < .01 for low Aggregate MH Need; b = 1.48, p < .01 for mean Aggregate MH Need; b = 1.42, p<.01 for high Aggregate MH Need). For parks with mean enrollment (34 youths), the relationship of Parent-reported Program Quality to SSIS Social Skills was smaller, but still significant and positive, regardless of Aggregate MH Need (b = 0.80, p < .01 for low Aggregate MH Need; b = 0.74, p < .01 for mean Aggregate MH Need; b = 0.68, p < .01 for high Aggregate MH Need). Among high enrollment parks (≥53 or more youth), there was no relationship between Parent-reported Program Quality and SSIS Social Skills (b = 0.06, NS for low Aggregate MH Need, b = 0.00, NS for mean Aggregate MH Need, b = −0.05, NS for high Aggregate MH Need). Higher Aggregate MH Need related to lower SSIS Social Skills overall.

Figure 1.

Association between Parent-reported Program Quality and SSIS Social Skills, Moderated by Park Enrollment and Aggregate MH Need.

Note. Low and High values are ±1 SD from the mean for Aggregate MH Need (M = 7.3%, SD = 5.7%) and Park Enrollment (M = 33.88, SD = 19.25).

Discussion

We examined associations among staff-reported OSC, parent and child perceptions of ASP quality, and children’s staff-reported social skills and problem behaviors within urban programs serving predominantly families of color. To our knowledge, this study is the first test of organizational influences, program quality and children’s social behavior in urban ASP. We examined six dimensions of staff-reported and -aggregated culture (rigidity, proficiency, resistance) and climate (functionality, engagement, stress). Only proficiency and stress related to program quality, and the nature of associations for stress were qualified by park enrollment and overall aggregated mental health risk. Higher youth and parent perceived program quality related to fewer staff-reported problem behaviors; overall higher enrollment and higher aggregated mental health need were associated with fewer staff-reported social skills.

Proficiency, Stress, and Program Quality

A proficient organizational culture reflects that staff are knowledgeable and competent; are responsive to children’s needs; and place the well-being of enrolled children above other priorities. Programs with more proficient cultures were perceived by parents as higher quality, lending partial support to findings from mental health that show clinics with cultures embracing high levels of proficiency, and low levels of rigidity (management decisions; social structures; rules, regulations, and procedures) and resistance (openness to change / innovation), are more successful in sustaining new evidence-based treatments (Glisson, Schoenwald et al., 2008). In the present study, though, proficiency emerged as the only dimension of culture to influence parent perceptions of program quality perhaps reflecting that its operational characteristics are more observable and meaningful to families, as compared to rigidity and resistance.

A stressful organizational climate reflects that staff feel exhausted and burned out; experience work overload or work conflict; report having insufficient time, colleagues or resources to perform their jobs effectively; and worry that the quality of their service suffers because of rules, regulations, or bureaucracy. The emergence of stress as the only organizational climate dimension meaningfully related to perceived program quality mirrors findings of several youth service organizations. Indeed, stress impacts service continuity and responsiveness in child welfare and juvenile justice (Hemmelgarn et al., 2006) and teacher relationships and classroom effectiveness (e.g., Kokkinos, 2007). Organizational climate dimensions engagement (personal involvement and accomplishment) and functionality (role clarity, cooperation, growth and advancement) may be less relevant for after-school staff – as compared to caseworkers or teachers – given the transient nature of the job which may translate to lower organizational commitment but manifest in behaviors less observable to children and their families.

The association of stress to youth reported program quality varied under different conditions of enrollment and aggregated mental health risk. In parks characterized by high enrollment but low risk, stressful climates were associated with lower youth satisfaction; in parks with low enrollment but high risk, stressful climates were associated with higher youth satisfaction. Although counter-intuitive at first look, closer consideration of these relationships points to several possible explanations. The impact of a stressful climate on programming may be mitigated to some extent by supportive colleagues or supervisors, or stress may influence full-time physical instructors differently from part-time recreation leaders (reflecting a difference in organizational commitment) or seasoned staff differently from newer staff. Staff in some parks with stressful climates may become disengaged resulting in lack of structure and elevated chaos that may be perceived positively by some youth (e.g., “I get to do what I want”) but negatively by youth who prefer a consistent routine, competitive sports, and opportunities for learning. Staff in parks with stressful climates may be less tolerant of disruptive behaviors leading to stricter and more rigid interactions, thereby reducing chaos but perhaps damaging relationships as well.

A stressful organizational climate may influence programs differently in large versus small enrollment parks due to corresponding differences in staff-student ratios or staff relationships with one another. ASP staff work fluidly across concurrent youth programs and activities, and youth attendance fluctuates significantly day to day; therefore, ratios are variable both within and between parks and over time as seasonal sports change. Although we considered utilizing staff-to-student ratios (instead of enrollment) in analyses, ultimately, we determined that Park Enrollment was more reliable despite its limitations. In fact, even enrollment numbers are tenuous, as they may overestimate (in cases where registered youth attend infrequently, such as once or twice / week, when soccer practice is cancelled) or underestimate (in cases where park supervisors allow unregistered youth to join activities instead of sending them back to the street) the number of youth served. Notably, though, enrollment size may carry significant implications when it comes to mitigating staff stress. Under conditions of high enrollment, staff are stretched thin in terms of delivering curriculum and engaging children, even if there are fewer significant mental health needs. With fewer children, even if significant mental health need is present, 1:1 interactions may allow for more directed management of both program delivery and behavioral engagement. Large enrollments though may influence quality of relationships, for instance via more opportunities for conflict or provocation, that in turn influence the overall program climate.

Program Quality Relates to Youth Social Behavior

We also examined associations among youth and parent perceptions of program quality with staff-reported youth social behavior. As expected, higher perceived quality related to fewer problem behaviors and higher social skills. Specifically, when youth and parents were satisfied overall with program experiences, staff reported fewer problem behaviors. Parent-reported program quality also related to higher staff-reported social skills, such that satisfied parents had kids that were perceived by staff to exhibit skills such as good communication, cooperation, empathy, and self-control. Associations were strongest in parks with small to average enrollment, lending support to a rich literature on after-school program quality that suggests positive relationships, consistent routines, and opportunities for learning facilitate children’s engagement and social emotional competence (e.g., Pierce et al., 2010).

On the other hand, high enrollment eliminated the association between program quality and social behavior, such that even when children and parents reported positive experiences and high satisfaction, staff in high enrollment parks rated children’s social skills lower (compared to staff ratings in low-to-average enrollment parks). Regardless of enrollment, staff in parks with high rates of aggregated mental health need also rated children’s social skills as lower, perhaps reflecting an observation or reporting bias. High staff-to-child ratios (i.e., high enrollment) and disengaged or disruptive behaviors (i.e., aggregated mental health need) may contribute to elevated stress among staff; in turn, stressed staff may be less aware of or attentive to positive behaviors (e.g., cooperation, assertiveness) and more likely to observe and report problem behaviors than social skills. These findings highlight challenges facing after-school staff, and bring to mind conclusions in the literature that staff proficiency may be the strongest indicator of ASP success (e.g., Cross, Gottfredson, Wilson, Rorie, & Connell, 2010).

Organizational Levers for Change

Intervention at an organizational level provides a unique opportunity to leverage the inherent, universal mental health promoting capacity of a large and complex youth service organization. In response to the rich literature documenting associations among social context, service quality and client outcomes, several organizational intervention models have emerged (e.g., Glisson & Schoenwald, 2005; Aarons et al. 2017) to target service provider, organization, and community level variables that influence service quality and outcomes. Theoretically they are guided by principles of effective public service organizations (Osborne & Gaebler, 1992) and designed to reduce social, technical, and strategic barriers to organizational effectiveness and to facilitate an infrastructure, and healthy culture and climate, that supports effective implementation of core technology (Glisson & Schoenwald, 2005).

We have been working over many years with park collaborators to apply organizational principles and intervention components to systematically remove barriers to high quality service delivery and introduce sustainable structures that support innovation. Emphasis on relationships, resource mapping, and sustainable supports (Frazier et al., 2007) offers a platform for training, consultation and problem-solving toward adoption of intervention tools such as Peer-Assisted Learning (Wright, 2004) for supporting academic enrichment and the Good Behavior Game (Barrish, Saunders, & Wolf, 1969) for managing disruptive behaviors. Systematic network development ensued by linking parks with local schools and libraries to help reduce isolation, support children’s homework completion, and improve resources, strategies that may be particularly helpful in reducing stress experienced by staff in high enrollment parks.

Preliminary findings toward the end of the study informed a week-long series of organizational intervention activities that included meetings with the Chief Programming Officer and leadership team, regional and area level meetings, and a two-day, city-wide conference for line-level instructors and park supervisors to generate excitement for the park’s commitment to building an infrastructure to support ongoing quality improvement. During the conference, staff generated ideas for improving youth skills building (e.g., what life skills can kids best acquire through park activities; what interferes with kids learning these skills; how do skills differ for children versus adolescents) and staff training (e.g., what competencies do staff need in order to support youth development). The goal was to plan monthly workgroups that could use data to generate structures that build on strengths and remove barriers and solutions to enhance program quality, youth engagement, and youth skills development. Although district-level transitions ultimately led to different priorities, the focus on improving staff proficiency and reducing stress highlights pathways toward supporting youth development in urban ASP.

Limitations

A number of limitations warrant consideration while interpreting findings. First, 44 of 94 eligible park programs participated; thus, generalizability of the findings to ASP within or beyond the region should be considered with caution. Importantly, continual changes in structure and activities of after school time impacted the number of parks eligible to participate in the study (planned 84 parks at the time of grant submission, reduced to 75 parks at the time of initial funding, increased to 94 parks by the end of recruitment procedures). Programs that declined participation most often indicated their staff was too small, child enrollment too low, or activity offerings too few to justify the time and burden of study enrollment for them and their families. Second, many families enroll multiple children in community ASP; as described, approximately 50% of participating families enrolled more than one child in the current study. Clustering within family is most relevant to analyses involving parent report of program quality. We did not account for clustering due to low ICCs for all outcomes. Third, culture and climate represent aggregated scores among staff within each park. Aggregating scores requires three or more completed OSC’s per park; thus, 12 parks in our sample were excluded from analyses. Loss of data from excluded parks reduced power associated with other analyses; however, t-tests revealed that excluded parks were not significantly different from parks that remained in the sample on other variables of interest. Additional missingness-at-random at the level of child-, parent-, and staff-reported data required imputation. Fourth, data were cross-sectional, and while they offer a meaningful snapshot of associations among organizational social context, perceived program quality, and children’s social behavior, caution interpreting findings is warranted as we are unable to draw causal conclusions about longitudinal associations or influences of organizational culture and climate over time.

Lastly, additional variables may influence associations among social context, program quality, and child social behavior; to this end we examined enrollment and aggregated mental health need as covariates. There is no precedent in the literature for aggregating SDQ scores across children to generate an overall setting-level mental health risk score; in fact, we were unable to identify an alternative method by which to capture the overall level of mental health need among youth in a classroom or program. Our measure may be imprecise (as the impact of aggregated need probably varies with enrollment and daily attendance, which themselves may be under-reported, over-reported or otherwise inconsistent) or underestimate the overall mental health need (because children with the greatest impairment may not have enrolled in the study). Nevertheless, our extensive and intensive experience with urban ASP in communities characterized by resource poverty suggest that the high frequency of mental health problems among enrolled youth (Frazier et al., 2013) interferes daily with after-school program relationships, routines, activities and interventions to a high degree – in part because staff receive minimal training in child development or mental health and thus have not the comfort, confidence, nor competence to meet those needs or effectively engage those youth. This may contribute meaningfully to the experience of reported stress represented by the organizational climate data.

Clinical Implications and Conclusions

We are continuing to examine the capacity for ASP to facilitate healthy developmental trajectories and the opportunities of both organizational and line-level interventions to reduce staff stress and support effective programming. The work presented here is part of a long-standing academic-community collaboration devoted to a shared vision for leveraging the unique opportunity of ASP to mitigate risks and support healthy developmental trajectories for urban youth. Professional development and workforce support are focused on equipping staff with knowledge, skills, and strategies to increase student engagement, reduce disruptive behaviors, and create and leverage teachable moments for social-emotional learning. Overall the goal is to improve staff proficiency and reduce stress to improve overall program quality as well as individual children’s adaptive functioning.

Enhancing the quality and capacities of ASP via organizational intervention and workforce support represents an alternative use of community mental health resources. In particular, providing consultation to organized programs would enable mental health clinicians to more efficiently support their communities, with a flexible range of universal, targeted, and intensive supports for staff, children, and families. The work herein is consistent with a contextually-relevant understanding of children’s mental health, and responds to mental health disparities in under-served communities.