Evaluation of a Community-based Program for Young Boys At-Risk of Antisocial Behaviour: Results and Issues

Abstract

Objective

We assess the impact of a community-based intervention program for boys 6–11 years old at-risk of antisocial behaviour, and compare changes in behaviour and competence pre-post for intervention and wait-list comparison group.

Method

Interested parents called for enrolment. Inclusion required police contact and/or clinical scores (T>69) on Child Behaviour Checklist (CBCL) or Teacher Report Form (TRF), no developmental delay and English speaking. The program included two core 12-week groups (children’s, parents’) and optional additional services. Twelve sessions (February 2002–December 2005) provide pre-post intervention data, boys waiting at least 6 months formed a comparison group (starting April 2005). Outcomes included CBCL and TRF behaviour scales (rule-breaking, aggression, conduct, total problems) and competence. Repeated measures analysis of variance was done.

Results

Pre-post outcome comparisons indicated improvements among all boys, with significant differences favouring intervention boys on CBCL behaviour scales, but not TRF outcomes. Effect sizes were small to medium. Persisting high post-behaviour levels, unmeasured variation in additional services, and other design and sampling issues are noted.

Conclusions

More rigorously designed program evaluation is required.

Introduction

Aggressive and antisocial behaviours such as fighting and stealing are common among boys (Offord & Lipman, 1996). These behaviours commonly co-occur with other emotional and behavioural problems, and academic, peer and authority difficulties during childhood. Problems often persist, with later school dropout, substance use, and unemployment (Fergusson et al, 2005; Loeber, 1991). Younger age of onset (10 or less) is considered higher risk for persistent problems (Lahey et al, 1998).

The associated burden of suffering is great, with impact on the children, families and victims, and costs in academic, health and judicial systems (Connor et al, 2006). By adulthood, young children with antisocial behaviours may cost society ten times that of children without these behaviours (Scott et al, 2001). While some programs exist to assist young children and families with these difficulties (e.g., Webster-Stratton & Taylor, 2001), continued efforts aimed at early intervention are needed.

We report on an intervention program for high-risk children: boys from 6- to 11- years old who have been in trouble with the law, or are deemed to be at risk of same. The SNAP™ Under 12 Outreach Program (ORP) was developed in Toronto (Hrykiw-Augimeri et al, 1993), based on the concept of crime prevention through social development and accounting for risk factors at various ecological levels that can lead to criminal activity (Bronfenbrenner, 1979). Similar approaches have been demonstrated to improve and prevent antisocial behaviours (e.g., Henggeler et al, 1999), though not consistently (Cunningham, 2002). There are core ORP child and parent groups, and families can access additional activities/services offered as part of the program. Previous program evaluation has been positive but limited methodologically (e.g., Augimeri et al, 2007).

We report on the Banyan SNAP™ Under 12 Outreach study, a program replication. Our objectives are to (i) compare changes in offending behaviour and social competence pre-post among ORP boys and a waitlist comparison group, and (ii) report on issues relevant to evaluation of community-based programs.

Method

Participants were recruited through community advertisement (newspaper, radio, local cable television) and suggestions to families by police, child welfare, school personnel and children’s mental health services. To be eligible for the program, boys had to be 6-to-11 years of age, live in Hamilton, Ontario, and must have had police contact or be considered to be at risk of police contact. Interested parents/boys meeting these eligibility criteria during a telephone interview were interviewed face-to-face within two weeks. Boys accepted in the program had reported police contact and/or risk of this due to elevated scores (clinical range, T score >69) for “offending” behaviours (rule-breaking, aggressive and conduct) on the Child Behaviour Checklist (CBCL) or Teacher’s Report Form (TRF) (Achenbach & Rescorla, 2001). Boys with significant developmental delay and non-English speaking were excluded.

Sessions began in February 2002, and ran three times a year (Winter, Spring and Fall). Each session consisted of three children’s groups (SNAP™ Children’s Group) and parents’ groups (SNAP™ Parent Group). Each group included seven children (total n per session = 21, except session one = 17).

Participants were scheduled for the next available session on a first-come, first-served basis. Due to an extensive waiting list, boys/families waiting at least 6 months formed the comparison group (beginning April 2005).

Intervention:

The SNAP™ Under 12 Outreach Program (ORP) was developed in Toronto (Hrykiw-Augimeri et al, 1993). Each core component, the children’s group (SNAP™ Children’s Group) and parallel parents’ group (SNAP™ Parent Group), is manualized (Earlscourt Child and Family Centre, 2001a, 2001b) and runs 12 weeks, 1.5 hours per week. Children’s groups utilize cognitive-behavioural self-control and problem-solving techniques and are structured [arrival time/free play, teaching and reinforcement of self-control/problem-solving skill (Stop Now and Plan), recreation, snack, relaxation/end], with opportunities for unstructured/structured play, discussion, modeling, coaching and behavioural rehearsal. CBT-based strategies have been elements of successful treatments for similar boys (e.g., Henggeler et al, 1999; Kazdin et al, 1989). Parent groups teach child management techniques, and are structured (specific parent/child problem, applicable parenting skill, modeling, role-playing, discussion, homework, relaxation). Concurrent child and parent groups run on weekday evenings.

Optional additional services available to boys/families while waiting for and during the core components include academic tutoring, clinical services, community advocacy, community hook-up, crisis intervention, Hamilton Arson Prevention Program for Children, individual befriending, individualized parent intervention (Levene, 1998), individual parent counselling, siblings’ club, parent befriending and booster sessions, pre-group parent support, school advocacy, school support, pre-group children’s program and victim restitution. These optional services were available to all boys pre-ORP (i.e., both intervention and comparison). After the core components, services include post- programs for boys, parents and siblings.

Each of the core components was facilitated by two ORP staff trained by a developer of the Toronto ORP (LA). Group sessions were videotaped. Program fidelity checks were completed monthly by LA for the first 18 months of the children’s group program, with high fidelity (95–98%). Subsequently program fidelity checks were done quarterly moving to biyearly, with continued high fidelity.

Data Collection:

Baseline/pre- data were collected during the first telephone and face-to-face interviews. Follow-up/post- data collection occurred at 6-, 12-, 24-, 36- and 48- months after core component start. Comparison data were collected at pre- and 6-month post while still on the waiting list. We report on pre- and 6-month post-data for ORP and comparison families.

Measures:

Demographic data collected at pre-included child age at first CBCL (years), parent age (years), lone parent (no spouse or partner currently in the home), marital status (married, divorced, separated, widowed, never married, other), income source (wages and salaries, government assistance including unemployment insurance, other), low income [mean income of area of residence based on postal code (Statistics Canada, 2001) below median 2001 income of Hamilton ($63,031)], police contact (parent-report), and child welfare involvement (current and/or past, ward status).

The CBCL and TRF (Achenbach & Rescorla, 2001) were completed pre- and post. Offending behaviour was measured by rule-breaking, aggressive and conduct scales on CBCL and TRF. Social competence was operationalized by total problems (CBCL, TRF), total competence (CBCL) and adaptive functioning (TRF). Total problems include externalizing and internalizing problems. Total competence represents engagement in community activities, social skills and school. Adaptive functioning includes “works hard”, emotional problems, learning and “how happy compared to other students of same age”. Higher scores represent greater difficulties, except total competence and adaptive functioning (reversed).

Attendance at groups was also collected (number of sessions, and % completing 8 of 12 sessions or “graduating”).

Sample Collection:

Boys completing the ORP program February 2002–December 2005, and comparison boys are included. Pre- data are available for 223 ORP children [either pre-CBCL (223) and/or pre-TRF (202)] who were assigned to a session, and pre-post data for 132 CBCLs (59.2%) and 102 TRFs (50.5%). Pre- data are available for 116 comparison children [either pre-CBCL (114) and/or pre-TRF (90)], and pre-post data for 77 CBCLs (67.5%) and 67 TRFs (74.4%).

Analyses:

We used SPSS version 12 (SPSS, 2003). Descriptive statistics and pre-comparisons between ORP and comparison participants used two sample t-tests, and chi-square analyses. Changes in parent- and teacher outcomes (pre-post) were analyzed using repeated measures analyses of variance (ANOVA). Age, any child welfare and time between pre-post assessments (days) were included as covariates. Effect sizes were calculated (Cohen, 1992).

Results

Baseline / Pre- Characteristics (Table 1)

Table 1.

Baseline Characteristics of Children and Families

	ORP (n = 223)			Comparison (n = 116)
Characteristics	n	X(SD)	%, (No.)	n	X(D)	%, (No.)
Child Agea	223	9.8* (1.7)		115b	9.4 (1.6)
Parent Age	177	36.7 (6.8)		110	36.3 (7.3)
Lone Parent	218		52.8 (115)	115		47.0 (54)
Marital Status	218			115
Married			28.4 (62)			33.0 (38)
Divorced			15.6 (34)			7.8 (9)
Separated			15.1 (33)			15.7 (18)
Widowed			2.3 (5)			2.6 (3)
Never Married			24.3 (53)			26.1(30)
Other			14.2 (31)			14.8 (17)
Income source	214			109
Wages			60.7 (130)			67.9 (74)
Government Assistance			24.3 (52)			17.5 (19)
Other			15.0 (32)			14.7 (16)
Low Income	221		67.0 (148)	116		64.7 (75)
Police contact	182		40.7**(74)	108		25.0 (27)
Child Welfare
Any	223		51.1*(114)	75		36.0 (27)
Ward Status	131		6.9 (9)	82		3.7 (3)
CBCLc
Rulebreaking	223	72.9*(6.9)		114	71.1 (6.7)
Aggressive	223	79.4 (10.3)		114	78.0 (10.0)
Conduct Problems	223	77.2 (8.0)		114	76.0 (7.3)
Total Problems	223	73.2 (6.3)		114	72.2 (7.0)
Competence	207	36.3 (8.5)		107	36.4 (9.0)
TRFd
Rule Breaking	202	64.9 (8.3)		90	65.4 (7.8)
Aggressive	202	68.4 (11.1)		90	69.0 (9.9)
Conduct Problems	202	67.5 (11.0)		90	69.3 (11.2)
Total Problems	202	66.4 (9.8)		90	67.4 (8.6)
Adaptive Functioning	202	37.7 (5.0)		87	37.4 (5.0)

^aAt CBCL completion

^bAge calculated from TRF for one boy, other missing

^cParent-rated

^dTeacher-rated

^**p < 0.01

^*p < 0.05

At baseline/pre-, ORP boys are significantly older (t(df) = 2.06(336), p<0.05), have more police contact (chi square(df) = 7.32(1), p<0.01) and higher rates of any child welfare involvement (chi square(df) = 5.15(1), p<0.05) than comparison boys. There are no significant differences between ORP and comparison groups on CBCL or TRF scales, except CBCL rule-breaking behaviour (t(df) = 2.25(335), p<0.05, ORP worse).

Baseline characteristics of ORP and comparison boys providing pre-post data are similar to those providing pre-data only (not shown), without significant differences on age (t(df)= 1.01 (207), p=0.31), and police contact (chi square(df) = 1.76(1), p=0.18). Differences remained on any child welfare contact (chi square(df) = 4.42(1), p<0.05) and CBCL rule-breaking (t(df)=2.07 (257), p<0.05, ORP worse).

Comparison of participants providing pre-data only vs. those providing pre-post data showed significant loss of boys with reported police contact (chi square(df) = 7.35(1), p<0.01) but no other significant differences.

Treatment Effects (Table 2)

Table 2.

Pre-Post CBCL and TRF Scores for ORP and Comparison Groups

			(x, SD)
Outcome		n	Pre	Post
CBCLa
Rulebreakinga	I	132	73.2 (6.6) ***	67.5 (8.2)
	C	77	70.9 (6.9) ***	67.6 (7.2)
Aggressivea	I	132	80.3 (10.6) ***	72.0 (11.08)
	C	77	78.1 (9.6) ***	73.4 (10.7)
Conduct Problemsa	I	132	77.6 (8.0) ***	70.7 (9.6)
	C	77	75.8 (7.4) ***	72.0 (7.5)
Total Problemsa	I	132	73.9 (6.2) ***	67.8 (8.2)
	C	77	72.4 (6.3) ***	69.1 (7.3)
Total Competencea	I	118	37.7 (8.3)*	40.0 (9.1)
	C	67	37.2 (9.4)	38.5 (10.1)
TRFb
Rulebreakingb	I	102	64.2 (8.5)	63.5 (8.2)
	C	67	66.1 (8.4)	64.0 (9.2)
Aggressiveb	I	102	67.1 (11.0)	66.3 (11.0)
	C	67	69.1 (10.4)	67.7 (12.3)
Conduct Problemsb	I	102	66.7 (11.3)	65.1 (10.5)
	C	67	70.2 (12.0)	67.7 (12.8)
Total Problemsb	I	102	65.2 (9.8)	63.8 (10.1)
	C	67	67.6 (8.7)	66.0 (10.1)
Adaptive Functioningb	I	101	38.0 (5.8)*	39.4 (5.7)
	C	64	37.1 (5.3)	38.2 (4.9)

I = ORP or intervention, C = comparison

^aParent-rated

^bTeacher-rated

^***p < 0.001

^**p < 0.01

^*p < 0.05

ORP boys show significant improvements pre-post on all CBCL outcomes and TRF adaptive functioning. Comparison boys show significant improvements on all CBCL outcomes, except total competence, and on no TRF scales. Pre-CBCL “offending” behaviours (rule-breaking, aggressive, conduct) plus total problems were in the clinical range (T score >69) for ORP and comparison boys, and remained so at post for CBCL aggressive and conduct for ORP and comparison boys. No TRF “offending” behaviours were in the clinical range pre- or post, except pre-conduct problems for comparison boys.

Attendance was varied between sessions with average number of sessions attended ranging from 6.9 to 10.7 out of 12 in the children’s groups, and 5.6 to 9.5 out of 12 in the parents’ groups (data not shown). Graduation rates from the program (attendance at 8 or more sessions) ranged from 48% to 88% for the children’s groups and 44% to 80% for the parents’ groups (data not shown).

Multivariate analyses (Table 3) indicate significant differences in improvement between ORP and comparison boys on all CBCL outcomes except competence, and no TRF outcomes. Effect sizes were small to medium for CBCL “offending” behaviours and total problems. Models run including any child welfare as a covariate (reduced sample size) showed no change in results (not shown).

Table 3.

Repeated Measures Analysis of Variance^a Results

Outcome	df	F	p	Effect Size
CBCLb
Rulebreaking	1,202	5.54	0.02	0.36
Aggressive	1,202	6.43	0.01	0.38
Conduct Problems	1,202	6.89	0.01	0.40
Total Problems	1,202	6.85	0.01	0.41
Competence	1,178	0.93	0.34	−0.15
TRFc
Rulebreaking	1,164	1.75	0.19	−0.24
Aggressive	1,164	0.29	0.59	−0.10
Conduct Problems	1,164	0.44	0.51	−0.12
Total Problems	1,164	0.09	0.77	−0.05
Adaptive Functioning	1,160	0.02	0.88	−0.03

^aVariables included in ANOVA: Child age at pre-CBCL, days between pre-and post-evaluation, intervention

^bParent-rated

^cTeacher-rated

Discussion

We evaluated a community-based program for a high-risk group, young boys (6–11 years old) with behaviours leading to police contact or that put them at risk of same. Boys in ORP and comparison groups, and their families, were disadvantaged, many reporting child welfare involvement. Both groups improved from pre-post evaluation. ORP boys improved significantly more than comparison boys on parent-rated “offending” behaviours (rule-breaking, aggressive, conduct) and total problems, though not teacher-rated outcomes.

At pre-evaluation, ORP boys had significantly more police and child welfare contact than comparison boys. This may relate to timing of enrollment. At program initiation, specific community partners, including police and child welfare agencies, were aware of the ORP program and may have informed parents. Later the program was more widely known, and enrolling boys were more likely to become the comparison group.

Teacher-rated outcomes demonstrated no significant improvements pre-post for any ORP or comparison outcomes except adaptive functioning for ORP boys. Timing of pre-post evaluations meant data were often collected in different academic years, so different raters were likely used.

Parent ratings suggest ORP participation is associated with reduction in offending behaviours compared to waiting list boys. There was no convergence among parent and teacher raters. If program participation has the capacity to reduce these behaviours in a high-risk sample, it is encouraging since the young enrollment age suggests participants may have high risk for persistent antisocial behaviours.

There are several cautions. First, some boys completing the program continue with behaviours at the clinical level. Second, these results only measure behaviours short-term (pre-post program). Demonstration of persisting differences would strengthen the results, but comparison boys had no further follow-up before their ORP session began. Third, there was variable use of the additional broader program options (e.g., school involvement) by ORP and comparison families. Provision of intervention at multiple levels is a program strength (Bronfenbrenner, 1979). However, incomplete tracking of additional services used prevents investigation of their impact on outcomes. Fourth, a large number of participants who provided baseline data were not part of the pre-post data set. There are likely a range of reasons for this data loss (e.g., family move, drop-out from the program). Boys with police contact at pre- were significantly more likely to not be included in the pre-post data set, but otherwise no significant differences were found.

The Banyan ORP was a careful replication of the Toronto ORP. However, most agencies serving similar boys would be unable to provide all ORP components. The core cognitive behavioural children’s group is used as a stand-alone program more widely in Ontario and beyond, but impact has not been evaluated.

Strengths include a manual-based CBT-type program, leader training, fidelity checks, both parent and teacher ratings and sample size. Program evaluation was initiated by the program-delivery agencies and funder (National Crime Prevention Strategy), with our evaluation team joining later. This demonstrates forward thinking, as there are many programs that are run with no evaluation component. However, there are a number of limitations. In addition to the cautions above, some components essential to clinical intervention trials were missing (Moher et al, 2001). For example, the nature of the participant sample is not precisely understood. Police contact primarily included incidents that would have been chargeable had the boy been older, but further details are lacking on this entry criterion. Methodology was limited by use of a waiting list comparison group, gathered after the intervention arm of the study had begun, versus one gathered through random allocation. Detailed information on those completing pre- but not post-evaluation is lacking, though many similarities and one difference has been identified. Participants and assessors were not blinded, and parental expectancies may influence ratings (Boyle et al, 1999).

All community-based programs should include an evaluation component. The methodologic and practical issues discussed above illustrate the challenges of conducting rigorous program evaluation in typical community settings. Improvements to the design (e.g., use of a randomized controlled trial), sample maintenance (e.g., decreased sample loss between pre- and post-evaluations, with good description of those lost), monitoring of co-intervention (e.g., other additional program components used), and longer follow-up would increase the scientific rigor of this and similar community-based program evaluations.

From a clinical standpoint, boys and families represented in this program frequently seek, or are encouraged to seek, assistance through clinical services and community agencies. This program evaluation demonstrated that, while both ORP and comparison groups improved from pre- to post-evaluation, ORP boys improved significantly more than waiting list boys on parent-rated “offending” behaviours (rule-breaking, aggressive, conduct problems) and total problems, but not teacher-rated outcomes. The capacity to reduce these behaviours in a high-risk sample is important. However, persisting clinical behavioural levels post- ORP, short-term evaluation, unmeasured variation in non-core program utilization, and other design and sampling issues require cautious interpretation of our results. More rigorously designed program evaluation is needed to more clearly assess ORP impact for this high-risk group of boys and their families.