Randomised comparison of the effectiveness and costs of community and hospital based mental health services for children with behavioural disorders

Abstract

Objective

To test the hypothesis that a community based intervention by secondary child and adolescent mental health services would be significantly more effective and less costly than a hospital based intervention.

Design

Open study with two randomised parallel groups.

Setting

Two health districts in the north of England.

Participants

Parents of 3 to 10 year old children with behavioural disorder who had been referred to child and adolescent mental health services.

Intervention

Parental education groups.

Main outcome measures

Parents' and teachers' reports of the child's behaviour, parental depression, parental criticism of the child, impact of the child's behaviour on the family.

Results

141 subjects were randomised to community (n=72) or hospital (n=69) treatment. Primary outcome data were obtained on 115 (82%) cases a year later. Intention to treat analyses showed no significant differences between the community and hospital based groups on any of the outcome measures, or on costs. Parental depression was common and predicted the child's outcome.

Conclusions

Location of child mental health services may be less important than the range of services that they provide, which should include effective treatment for parents' mental health problems.

Introduction

Views about where to provide secondary mental health services for children and adolescents have changed repeatedly over the past 50 years. Child psychiatry started as a community discipline in child guidance clinics.¹ There were, however, many practical difficulties in the administration of these clinics,¹ and during the 1970s and '80s many clinics closed and were replaced by hospital based services.¹,² Over the past 15 years, however, hospital based services too have been criticised—on the grounds that they are inaccessible, stigmatising, expensive, poorly integrated with community services, and less likely to produce gains that generalise to other environments, such as school.³,⁴ Political pressure has also been applied to child mental health services to return to the community.²

The assumption that community based child mental health services lead to better outcomes than hospital based services has not been tested in a randomised trial in the United Kingdom. We conducted such a trial, whose main hypothesis was that for children with behavioural disorders a community based intervention would be significantly more effective and less costly than a hospital based intervention.

Participants and methods

Participants

The study was based on the parents of children with behavioural disorders. Parents were eligible if they were judged able to participate in groups—for example, if they did not have a major mental disorder—and had children who were (a) aged 3-10 years, (b) had a clinical diagnosis of oppositional disorder,⁵ (c) had normal intelligence (clinical judgment), and (d) had been referred to the child and adolescent mental health service in either of two health districts in the north of England.

Interventions

During the study each district child and adolescent mental health service provided the same intervention either in a children's hospital or in a community setting—for example, health centres and community resource centres. As this was a pragmatic trial of the interventions used in the NHS, each service used their routine interventions for behavioural disorder for the age group being studied. In one of the districts, this was the videotape modelling parental group education programme of Webster-Stratton and colleagues.⁶ The other district used a programme of parental education groups with parallel child groups. In both districts the interventions were provided by various professionals, including community psychiatric nurses, psychologists, social workers, and psychiatrists. Therapists who took part in the study were trained in at least two parental groups before leading a group.

Procedures for assignment and blinding

The unit of randomisation was the parent/index child pair. After written consent had been obtained, an independent statistician at a distant site randomly allocated participants to the community or hospital based interventions, stratified by health district. Group allocation was concealed from the outcome assessor, who was asked at the end of the study to guess which intervention had been given to which parent. These guesses were no better than chance (59/141 or 42% correct). As studies of psychosocial treatments can only ever be single blind, participants' expectancies of treatment could bias the results. The parents' expectancy of treatment after randomisation was therefore assessed with a 0 to 8 continuous scale, where 8 was a very high expectancy that treatment would help.⁷ The mean level of expectancy among parents was similar in both groups (community group: 5.7 (95% confidence interval 5.3 to 6.1); hospital group: 5.8 (5.4 to 6.3)).

Assessment of effectiveness

Measures were completed before treatment and at two follow up stages—three months after treatment started and at about one year. Because parental perception of a child's problems is one of the most important determinants of the use of services,⁸ the primary outcome was parental report of the child's behaviour.⁹ Secondary outcomes comprised the teacher's report of the child's behaviour,⁹ parental reports of the impact of the child's behaviour on the family,¹⁰ parental criticism of the child (assessed by counting the number of critical comments during a five minute speech sample¹¹), and parental perception of parenting problems.¹² Parental depression was assessed with the Beck questionnaire.¹³ In line with previous research in the United Kingdom with the parents of children with behavioural disorder,¹⁴ a score of 15 or more on the Beck scale was categorised as “high,” indicating a high level of maternal depression.

Assessment of costs

Information on the use of all services by both the children and their primary carer (usually the mother) during the trial was collected from the primary carer at the final follow up interview. A questionnaire was designed for the purpose of the study and based on data collection methods developed in a previous trial.¹⁵ The perspective of the trial was that of all service providers, including the NHS, social services departments, education departments, and voluntary and private sectors. This enabled the differential impact of the parenting skills groups on each sector to be clearly quantified. In addition, the cost of travel to attend the sessions and the cost of the crèche facilities were recorded. Unit costs were for the financial year 1998-9 and were collected from local service providers or national published unit costs^16–18 or calculated directly from relevant salary scales. All future costs were discounted at an annual rate of 5%. Further details on the costing of services are available on request.

Statistical analysis

We projected a sample size of about 40 cases in each group for two reasons. Firstly, before starting the study we had asked the purchaser (the health authority) and the provider of the existing child and adolescent mental health service in one of the districts (Salford) how large the difference in treatment effect between a community and hospital service would have to be to influence their plans for child and adolescent mental health services. These services were being reviewed at the time of the study, and both the purchaser and the provider made a commitment to implement the findings. They agreed that only a large difference (effect size of 0.8 for the mean difference in parental report of child behaviour between community and hospital treatment at the 3 month assessment) would lead to changes in the ways that services were delivered. Thirty four cases per group are needed for a 90% chance of detecting this difference with a two sided test at the conventional 5% significance level.¹⁹ Secondly, a Canadian study comparing community and clinic based child mental health services found a significant difference in parental reports of child behaviour with about 40 cases in each group.⁴

Just one intention to treat analysis—that is, on all participants who were randomised and who could be followed up regardless of whether they had started or completed the intervention—was conducted at the end of the study. The data were analysed with SPSS 9.0 for Windows. Changes from baseline were calculated for the outcomes, and t tests for independent samples were used to compare the community and hospital groups.

Although costs were not normally distributed, analyses compared the mean costs in the two groups with standard t test methods, with the validity of results confirmed by using the “bootstrapping” technique.²⁰ The advantage of this approach, rather than logarithmic transformation or conventional non-parametric tests, is the ability to make inferences about the arithmetic mean.²¹

Results

Participant flow and follow up

The figure shows the trial profile. In all, 187 eligible parents were referred to the trial. Of these, 25 failed to respond to an invitation to take part—for example, did not attend—and 21 refused randomisation. The remaining 141 parents were then randomly allocated to community (n=72) or hospital (n=69) treatment; 115/141 (82%) cases had complete data at baseline and at follow up one year later.

Participants' and children's characteristics

The median age of the children was 6.9 (range 3-10) years, and 112 (79%) were boys. The primary carer, who completed the parental questionnaires, was the mother in 136 out of 141 cases. In 94 out of 141 (67%) cases the parents were receiving state benefits, and in 61 out of 141 (43%) cases the parents were single. Seventy three out of 134 (54%) parents who completed the Beck depression questionnaire at baseline had a “high” score. The groups did not differ significantly in respect of these characteristics.

The mean behaviour intensity score (possible range 36-252, where the higher the score, the greater the intensity of behavioural problems)⁹ before treatment was 178.0 (95% confidence interval 172.8 to 183.1), higher than the mean score before treatment reported in another UK study (152.8; 144.3 to 161.4)²² and in studies of parental education in North America.²³

Effectiveness

The two groups did not differ significantly on any of the measures at baseline or at either of the two follow up assessments (table 1). There were no significant effects of location of treatment (community v hospital ) on changes in any of the outcomes (table 2). A high proportion of children (89/116 (77%)) had a high score (⩾127) on parental report of the intensity of behavioural problems at the three month assessment. The risk of children having a high score after treatment was greater (odds ratio 2.8; 1.3 to 6.1) for those whose primary carer had a high score on the Beck depression questionnaire at baseline (48/55) compared with those with a low score (⩽14) (36/56). Confounding by social class or behavioural problems at baseline did not account for this association.

Table 1.

Mean scores and standard deviations for primary and secondary outcomes at baseline, three months' follow up, and follow up at one year

	Before treatment	3 months' follow up	1 year follow up
Intensity of child's behavioural problems—parental report*
Community:
No of cases	68	61	62
Mean (SD) score	175.9 (31.6)	156.5 (41.2)	160.2 (42.7)
Hospital:
No of cases	67	55	56
Mean (SD) score	180.1 (29.4)	157.8 (43.0)	152.8 (40.7)
Intensity of child's behavioural problems—teacher's report*
Community:
No of cases	61	57	65
Mean (SD) score	123.1 (50.2)	114.0 (52.1)	110.5 (45.7)
Hospital:
No of cases	62	49	61
Mean (SD) score	117.8 (47.5)	104.3 (46.2)	111.9 (43.7)
Parental depression†
Community:
No of cases	67	60	61
Mean (SD) score	16.9 (10.5)	10.3 (9.3)	12.1 (10.4)
Hospital:
No of cases	67	53	56
Mean (SD) score	16.6 (9.4)	11.4 (10.3)	9.1 (9.2)
Impact of child's behaviour on family‡
Community:
No of cases	68	60	62
Mean (SD) score	11.5 (5.0)	8.8 (5.5)	9.1 (5.6)
Hospital:
No of cases	66	55	56
Mean (SD) score	12.0 (4.7)	9.7 (5.9)	8.6 (5.8)
Parental criticism of child¶
Community:
No of cases	66	55	59
Mean (SD) No	2.4 (2.2)	2.0 (2.3)	1.8 (2.1)
Hospital:
No of cases	60	52	53
Mean (SD) No	2.1 (2.2)	1.6 (2.4)	1.3 (2.5)
Parental report of parenting problems**
Community:
No of cases	59	61	62
Mean (SD) score	3.8 (0.8)	3.1 (0.9)	3.1 (0.9)
Hospital:
No of cases	54	55	56
Mean (SD) score	3.7 (0.9)	3.3 (0.9)	3.2 (1.0)

SD=standard deviation. 

The table includes all available data, regardless of whether data on a subject were missing at other time points. 

^*Minimum score 36, maximum 252; the higher the score the greater the intensity. 

^†Minimum score 0, maximum 63; the higher the score the greater the intensity. 

^‡Minimum score 0, maximum 22; the higher the score the greater the intensity. 

^¶Number of critical comments about the child during the five minute speech sample. 

^**Minimum score 1, maximum 7; the higher the score, the greater the perceived parenting problems. 

Table 2.

Change in scores for primary and secondary outcomes at three month follow up and at one year follow up, compared with baseline scores. Values in parentheses are 95% confidence intervals

	No in community group	No in hospital group	Change in score compared with baseline		Difference between community and hospital groups	P value
			Community group	Hospital group
Intensity of child's behavioural problems—parental report
At 3 months	58	54	18.8 (11.9 to 25.7)	21.3 (12.9 to 29.7)	2.5 (−13.2 to 8.2)	0.64
At 1 year	60	55	17.4 (10.5 to 24.4)	27.2 (19.3 to 35.1)	−9.8 (−20.2 to 0.6)	0.06
Intensity of child's behaviour—teacher's report
At 3 months	51	46	13.7 (4.9 to 22.5)	7.4 (−3.1 to 17.8)	6.3 (−7.1 to 19.7)	0.30
At 1 year	57	58	10.7 (1.2 to 20.1)	8.9 (−2.6 to 20.4)	1.8 (−12.9 to 16.6)	0.81
Parental depression
At 3 months	57	52	5.7 (3.8 to 7.6)	4.1 (1.7 to 6.5)	1.6 (−1.3 to 4.6)	0.28
At 1 year	59	55	4.6 (2.4 to 6.9)	6.2 (4.3 to 8.3)	−1.7 (−4.7 to 1.3)	0.28
Impact of child's behaviour on family
At 3 months	57	53	2.8 (1.5 to 4.1)	1.9 (0.8 to 3.1)	0.9 (−0.8 to 2.6)	0.30
At 1 year	60	55	2.6 (1.3 to 3.9)	3.1 (1.9 to 4.2)	−0.5 (−2.2 to 1.2)	0.57
Parental criticism of child
At 3 months	52	48	0.5 (−0.1 to 1.2)	0.3 (−0.3 to 1.0)	0.2 (−0.7 to 1.1)	0.69
At 1 year	56	50	0.6 (0.0 to 1.3)	0.9 (0.2 to 1.5)	−0.2 (−1.1 to 0.7)	0.64
Parental report of parenting problems
At 3 months	51	43	0.6 (0.4 to 0.9)	0.3 (0.1 to 0.6)	0.3 (0.0 to 0.6)	0.05
At 1 year	54	42	0.7 (0.5 to 0.9)	0.5 (0.3 to 0.8)	0.1 (−0.2 to 0.4)	0.38

For scoring systems see footnote to table 1. 

In line with other research with the families of children with behavioural disorders,²²,²⁴ there were high rates of non-attendance. Location did not influence attendance rate. For example, the proportion of participants who attended more than 50% of offered sessions was similar in the community (37/72 (51%)) and hospital (41/69 (59%)) groups (odds ratio 0.8; 0.6 to 1.2). Compliance with treatment tended to be worse (0.7; 0.5 to 1.1) among parents who were depressed at baseline (37/73 (51%)) than among those who were not (40/61 (66%)). The relation between parental depression at baseline and children's behavioural problems at the three month assessment was stronger among those who complied poorly with treatment (5.8; 1.5 to 23.2) than among those who complied well (1.8; 0.7 to 4.7).

This study followed up 82% of randomised participants more than a year later. Nevertheless, missing outcomes can violate the principles of intention to treat analysis.²⁵ We therefore conducted an extreme case analysis in which missing outcomes on community cases were assumed to have improved (to the mean value for the community group at three months), and in which missing outcomes in the hospital group were assumed not to have changed. This did not significantly affect the results.

Costs

Altogether, 118 out of 141 (84%) parents (61 in the community group, 57 in the hospital group) completed the resource questionnaire at the final follow up interview; these responses were included in the economic evaluation (table 3). The mean length of follow up was similar in both groups (65.7 weeks in the community group, 65.0 in the hospital group). To remove the influence of this small difference, however, costs per week (as well as overall cost) were calculated. No significant differences between the hospital and community groups were found in the mean overall cost per child (difference £904 (£−1254 to £3062); P=0.41), per primary carer (£611 (£−143 to £1365); P=0.11), or in total (£1515 (£−742 to £3772); P=0.19), or in terms of cost per week. To assess the robustness and generalisability of the results, several univariate sensitivity analyses were carried out. These analyses did not affect the results.

Table 3.

Total costs (£) over trial period, by cost sector

	Community (n=61)			Hospital (n=57)		Mean difference (95% CI)
	Mean (SD)	% of total		Mean (SD)	% of total
Children:
Intervention	374 (322)	10		488 (511)	17	−115 (−269 to 40)
NHS psychiatric services	1379 (4054)	37		997 (2158)	35	382 (−814 to 1578)
Other NHS services	321 (648)	9		285 (498)	10	36 (−176 to 247)
Education services	1513 (4958)	40		761 (2613)	27	752 (−707 to 2212)
Social services	120 (350)	3		290 (1200)	10	−170 (−488 to 148)
Voluntary and private sector	29 (133)	1		10 (50)	1	18 (−19 to 55)
Total cost per child	3736 (7210)	100		2831 (4091)	100	904 (−1254 to 3062)
Total cost per child per week	53 (99)			41 (49)		12 (−17 to 41)
Primary carer:
NHS psychiatric services	360 (2241)	30		22 (53)	4	339 (−250 to 927)
Other NHS services	791 (1316)	67		550 (1105)	96	241 (−204 to 686)
Social services	29 (221)	3		0 (0)	0	29 (−29 to 87)
Private sector services	3 (18)	0		0 (0)	0	3 (−2 to 8)
Total cost per carer	1183 (2721)	100		572 (1120)	100	611 (−158 to 1380)
Total cost per week per carer	18 (42)			9 (16)		9 (−3 to 20)
Total cost:	4919 (7668)			3403 (4332)		1515 (−777 to 3807)
Median (90% range)	2091 (279-19 794)			1875 (276-15 147)
Total cost per week:	71 (104)			50 (54)		21 (−10 to 52)
Median (90% range)	33 (4-330)			31 (5-187)

CI=confidence interval; SD=standard deviation. 

Discussion

The present study did not find that community based treatment was more effective than hospital based treatment. This finding contrasts with that of Cunningham and colleagues⁴ in Canada, who reported that children with behavioural problems treated in the community had better outcomes than those referred to a specialised clinic. It is, however, difficult to interpret the results of that study because outcome data were obtained on less than a third of randomised subjects.

Methodological issues

Three issues should be borne in mind when interpreting the results of the present study. Firstly, the trial was powered on the basis of a significant difference in clinical outcomes. The sample size may have been too small to detect a significant difference in costs. Indeed, the actual cost differences found between the two groups were large, with the hospital group costing 30% less overall than the community group. Secondly, there are several different models of community child and adolescent mental health services. This study modelled the common situation in which secondary services were located in just one or two community settings. Different results might have been obtained if we had studied primary care interventions, such as training health visitors in parental education methods,²⁶ or interventions in which mental health professionals support community schemes such as befriending.²⁷ These interventions have, however, seldom been evaluated in randomised trials in the United Kingdom. Thirdly, the results from this study may not apply to the treatment of other child psychiatric disorders or, indeed, to the treatment of behavioural problems using other methods.

Planning services for children with behavioural disorders

We conclude that in planning services for children with behavioural disorders, greater attention must be paid to factors other than the location. Our findings suggest, for example, that a service for children with behavioural problems must also be able to call not just on parental education groups but also on interventions such as effective treatments for parental depression. The finding that parental depression was more strongly associated with poor child outcomes in the presence of poor compliance with treatment suggests, however, that poor compliance might be partly responsible for the failure of treatment in the children of depressed parents. This group may therefore need extra help to attend child mental health services. As a substantial proportion of the costs associated with behavioural disorders in children are borne by the educational services, there may also be scope for further developing the role of mental health workers in schools.

What is already known on this topic

It is assumed that community based child mental health services lead to better outcomes than hospital based services, although this has not been tested in a randomised trial in the United Kingdom

What this study adds

Community based child mental health services are not necessarily more effective or cheaper than hospital based services

The outcomes of children's mental health problems are determined by many other factors, such as parental mental health

Child mental health services should provide effective treatment for parental mental health problems

The range of mental health services available is more important than where the service is given

Figure.

Trial profile of eligible participants