Children's acquisition and retention of safety skills: the Lifeskills program

Abstract

Objectives

Assessment of safety skills performance and knowledge, to evaluate the education offered by the Lifeskills “Learning for Living” village, Bristol, UK which emphasizes interactive learning‐by‐doing.

Design

Two quasi‐experimental matched control group studies. Study 1: knowledge and performance three months post‐intervention. Study 2: knowledge pre‐intervention and post‐intervention at three time points, to distinguish between immediate learning and longer term retention.

Setting

The Lifeskills training village, Bristol, UK; primary schools in four education authorities in the area.

Participants

Study 1: 145 children aged 10–11 years; 109 from the Lifeskills program, 36 control. Study 2: 671 children aged 10–11 years; 511 Lifeskills, 160 control.

Outcome measures

Three areas (road, home, and fire safety). Five performance tests: observation of children's safety skills. Five knowledge tests: pictorial quiz.

Results

Study 1: Lifeskills/intervention children did better than control children on performance and knowledge tests. The knowledge‐performance correlation was r = 0.51. Study 2: intervention children did better than control children immediately after the intervention and three months later on all five knowledge tests. On three tests the intervention group showed retention of knowledge from immediately post‐intervention to three months, but on two tests there was some loss. This loss was primarily among children from scholastically lower achieving schools. In all other respects the intervention was equally successful for boys and girls, and for children from higher and lower achieving schools.

Conclusions

The Lifeskills package improved both knowledge and performance but had shortcomings. Complexity of material did not affect knowledge acquisition but did affect its retention.

Lifeskills “Learning for Living” is a safety education “village” in Bristol, UK, designed for children aged 10–11 years. In a two hour school visit, children go round 10 realistic sets—for example, kitchen, bedroom, street—which contain hazards. In line with good teaching practice, one adult accompanies three or four children, encouraging them to spot hazards and discuss among themselves ways of eliminating or avoiding them.¹ The program recognizes that knowledge is necessary but not sufficient for successful performance of safety skills. Children therefore learn by doing. For example they open a door to a room “on fire” and learn to perform the correct escape routine.

Every year 8000 children experience the Lifeskills programme, a prototypical example of interactive safety training in the UK. Despite increasing emphasis on evidence based practice,²,³ little information about the impact of such programs is available. There was an evaluation of a programme like Lifeskills in Maryland, USA, but it had no matched control group, behavioral tests, or measure of knowledge retention over time.⁴ Other evaluations of safety education have not always distinguished immediate learning from longer term retention.⁵

In 2001 the Department of Health and the Health and Safety Executive commissioned an evaluation of the Lifeskills program. This paper reports two assessments of road, home and fire safety learning. The first investigated both knowledge and behavior; the second looked at immediate knowledge acquisition and longer term retention.

Study 1: Knowledge and performance of safety skills

Method

Participants

Children aged 10–11 years from seven coeducational state schools: n = 145 (67 boys, 78 girls). Schools were selected to represent a range of scholastic achievement on the 2001 National Key Stage 2 SATs. For the analyses schools were placed in two categories: “higher achieving” (SATs score equal to or above the national average) and “lower achieving” (SATs score less than the national average). The control and intervention groups were matched in terms of gender (girls = 54% intervention, 50% control), and also proportions of children from higher and lower achieving schools (see table 1).

Table 1 Control and intervention groups classified by schools' scholastic achievement level (Study 1).

School achievement level	Control		Intervention
	Higher	Lower	Higher	Lower
Schools (n)	1	1	3	2
Children (n)	22	14	75	34

To maximize comparability between intervention and control children, all participants were from schools booked into Lifeskills during a single school year. Testing took place during a two week period in spring. This provided a three month lapse between tuition and testing for the intervention group, while allowing time for the control group to make their Lifeskills visit after testing had finished. The advantages of this experimental design were felt to outweigh the disadvantage that it limited the number of schools available for recruitment, particularly in the control group.

Materials and procedure

Testing was carried out at an eight‐roomed building beside a private road. This setting provided real‐life equivalents of the Lifeskills training sets. There was a three minute time limit for each safety skill included. As each child attempted the skills, he or she was observed individually by two adults, one a Lifeskills guide, the other a trained assessor unaware of whether the child was in the intervention or control group. Each assessor independently recorded the child's performance, using a coding sheet previously piloted on 32 children. For example, for fire safety, the child opened a door to discover a “fire”. The guide said, “Do whatever you would do if it was a real fire”. The coding sheet listed: Shuts door to fire, Gets down, Shouts “Fire”, Leaves building, Uses phone outside. Without conferring, the guide and assessor checked off each action if the child performed it. Subsequent analysis showed that guides' and assessors' coding sheets were in complete concordance about the performance of 95% or more children for each of the five tests. Where a guide's and an assessor's sheets differed, the child was scored as unsuccessful on the task.

Children also completed pictorial tests developed for the evaluation. For example, for fire safety children were shown eight pictures from which they were to select depictions of the correct actions and place them in the right order. For room safety, children were required to circle each of seven potential hazards in a picture of a kitchen. Over 100 children participated in the development and piloting of these tests. Children were given one point for each task on which they were completely correct. Test‐retest reliability for total of tasks correct across a two hour interval was established with 52 children (r = 0.82).

There were five knowledge and performance tests:

• car stopping distance if the driver brakes hard at 30 mph

• what to do about a gas leak

• what must never be done in the event of a gas leak

• kitchen hazards (for example, hot cooking pan on the edge of a stove)

• what to do in the event of a fire.

As performance was the principal focus, observation preceded the pictorial tests to avoid cues from the pictures influencing how children performed.

Ethics approval was granted by the Social Sciences and Law Ethics Committee, Oxford Brookes University.

Results

Children received one point for each test on which they had all features correct. Performance and knowledge scores were separately summed giving two scores out of five for number of tests passed. Table 2 shows the mean scores and the results for the ANOVA: 2 (Control/Intervention) × 2 (Gender) × 2 (Lower/Higher achievement level of child's school). For both performance and knowledge, intervention children did significantly better than control children. According to conventions for evaluating effect sizes, the partial η² were “large”.⁶ There were no main or interaction effects for gender or school achievement level.

Table 2 Mean number of tests out of five passed by control and intervention group children (Study 1).

	Mean number of tests passed		F (df 1137)	p Value	Effect sizes, partial η2
	Control (n = 36)	Intervention (n = 109)
Performance
Mean	0.93	1.91	28.66	<0.001	0.17
SD	0.55	0.96
Knowledge
Mean	0.79	2.60	69.72	<0.001	0.34
SD	0.82	1.11

Table 3 covers the separate tests, showing percentages of children correct on each. Five loglinear analyses were conducted, one per test. The factors were 2 (control/intervention), 2 (gender), and 2 (schools' achievement level). Intervention children did better than control children except on the performance test of kitchen hazards. Over 80% of each group passed this.

Table 3 Percentage of control and intervention children correct on all features, test by test (Study 1).

	Performance			Knowledge
	Control	Intervention	Partial χ2	Control	Intervention	Partial χ2
n	36	109		36	109
Car stop distance	3%	37%	23.73***	8%	81%	65.01***
Gas‐should‐do	6%	27%	8.81**	3%	15%	4.09*
Gas‐never‐do	3%	22%	9.92**	17%	43%	9.71**
Kitchen hazards	81%	86%	0.57	50%	73%	6.41*
Fire escape routine	0%	11%	7.20**	11%	47%	15.91***

*p<0.05, **p<0.01, ***p<0.001 (df 1).

Achievement level of the children's schools had no main or interaction effects. Gender had a main effect on three tests. Across intervention and control groups, more boys than girls passed the car stopping distance performance test. More girls than boys passed the kitchen hazards performance test and knew what should be done on “gas‐should‐do”.

Knowledge and performance were significantly related. From the two five‐point scores summed across tests, the knowledge‐performance correlation was r = 0.51 (n = 145, p<0.001). The relation between knowledge and performance was highly significant for four of the five tests (car stopping distance, χ² = 32.73; gas‐should‐do, χ² = 11.41; gas‐never‐do, χ² = 12.88; fire escape routine, χ² = 12.39; df = 1, p<0.001 in each case). The relation approached significance for kitchen hazards (χ² = 2.43, df 1, p<0.06).

Discussion

The Lifeskills intervention successfully teaches children safety knowledge and skills. Table 2 shows that on average the intervention children passed twice as many performance tests and three times as many knowledge tests as control children.

The significant relation between knowledge and performance, both overall and for each safety skill, indicates that the greater the children's safety knowledge the better their performance. Knowledge and performance of safety skills have not always been found to relate.⁷ The Lifeskills programme emphasizes behavioral skill learning in addition to knowledge. The strong relation between knowledge and performance in the current study may not be found if an education program only has a knowledge focus.

The intervention was equally successful regardless of gender or school background. This is important in light of the well known relationships between childhood accidents and both gender and social deprivation.⁸,⁹ The intervention children's success is impressive, especially as the Lifeskills program covers many safety skills in two hours. Nevertheless the average number of intervention children passing each test was only 52% for knowledge and 37% for performance. As these tests were three months after the intervention it is possible that children had initially learned but subsequently forgotten. Study 2 investigates whether knowledge showed immediate post‐intervention improvement but was lost over time.

Study 2: Knowledge acquisition and retention

Introduction

Study 2 investigated acquisition and retention of safety knowledge by testing children at three time points: pre‐intervention, immediately post‐intervention, and three months post‐intervention. For practical reasons this study could not include a performance measure. To test the children before training, give them the training, and then test them again would have demanded an unreasonable six hours of their and their schools' time. In contrast knowledge tests could be incorporated at the beginning and end of a school's regular training visit with relatively little increase in the visit's duration.

Method

Participants

Children from 20 coeducational state schools: n = 671 (345 boys, 326 girls). The control and intervention groups were matched in terms of gender (girls = 47% intervention, 52% control), and also proportions of children from higher and lower achieving schools (see table 4).

Table 4 Control and intervention groups classified by schools' scholastic achievement level (Study 2).

School achievement level	Control		Intervention
	Higher	Lower	Higher	Lower
Schools (n)	3	3	8	6
Children (n)	105	55	328	183

To maximize comparability between intervention and control children all participants were from schools booked into Lifeskills during a single school year. The intervention group was drawn from schools making their Lifeskills training visit between December and February. They and the control group were tested during this period. All follow up testing occurred during March to May. Control group schools made their training visit to the Lifeskills centre after all testing was completed.

Materials

The same five pictorial tests of children's knowledge as in Study 1.

Procedure

Tests were administered twice to the intervention children at Lifeskills—once before and once after they were taken round. Control group children were tested at school. To mimic the test conditions of the intervention group, they completed the tests twice with an interval of two hours. All participants filled in the tests a third time at their schools three months later.

All testing, at Lifeskills or in school, was conducted by the same researcher to class sized groups in a classroom setting. Children were not allowed to confer.

Results

Overall impact of intervention

Each of the five tests was scored as correct only if all features were correct. There was acceptable test‐retest reliability: r = 0.73 between the control group's scores for total number of tests correct at time 1 and time 2, two hours apart.

Table 5 shows the mean number of tests passed pre‐intervention, immediately post‐intervention, and three months post‐intervention. A mixed design ANOVA was used: 2 (control/intervention) ×2 (gender) ×2 (lower/higher school achievement level) ×3 (time). There were main effects for intervention (F = 141.23, df 1/663, p<0.001) and time (F = 146.05, df 2/663, p<0.001). Crucially there was an interaction effect for intervention × time (F = 108.77, df 2/663, p<0.001, partial η² 0.14). This reveals that the intervention was successful. Both groups started at a similar level. Intervention group children's knowledge improved dramatically immediately post‐intervention, and was substantially maintained three months later. There were no group × time interaction effects with school achievement level or gender.

Table 5 Mean numbers of tests out of five passed by control (C) and intervention (I) group children (Study 2).

Time of test	Pre‐intervention		Immediate post‐intervention		Three months post‐intervention
	C	I	C	I	C	I
n	160	511	160	511	160	511
Score	0.86	0.76	1.04	2.93	0.89	2.09
SD	0.87	0.83	0.93	1.21	0.98	1.51

Test by test analysis

Table 6 shows the percentages of children who were correct on all features of each test. Five loglinear analyses were conducted, one per test. The factors were 2 (control/intervention), 2 (gender), 2 (schools' achievement level), and 3 (time).

Table 6 Percentage of control (C) and intervention (I) children correct on all features test by test† (Study 2).

			Time of test
			Pre‐intervention		Immediate post‐intervention		Three months post‐intervention
	n		C	I	C	I	C	I	Partial χ2
	C	I							Intervention× Time
Car stop distance	132	401	26%	25%	24%	94%	21%	88%	121.71***
Gas‐should‐do	121	376	6%	8%	4%	56%	2%	21%	16.26***
Gas‐never‐do	121	376	24%	16%	21%	64%	25%	52%	39.46***
Kitchen hazards	132	402	24%	22%	33%	51%	42%	66%	11.35**
Fire escape routine	132	404	13%	10%	20%	45%	21%	43%	13.48***

†Each row of this table contains the analysis of one test and is based only on children who completed that test.

**p<0.01, ***p<0.001 (df 1).

On all five tests intervention group children outperformed control group children at immediate post‐intervention and three months post‐intervention. On three tests intervention group children's knowledge remained almost identical between immediate post‐intervention and three months post‐intervention. It fell slightly on gas‐never‐do. On only one test (gas‐should‐do), did their knowledge decline dramatically over the three months. Of the intervention children 8% passed this pre‐intervention, 56% immediately post‐intervention, and 21% at three months (still significantly higher than the control group's 2%). A critical message in gas safety education is “Use a landline rather than a mobile phone to call assistance”. Intervention group use of the landline rose from 17% pre‐intervention to 60% immediately post‐intervention but fell back to 31% at three months.

There were no intervention × gender or intervention × time × gender interaction effects. Gender had only one main effect: across intervention and control groups girls did better than boys on kitchen hazards.

There were two main effects for school achievement level: children from higher achieving schools did better than children from lower achieving schools on fire escape routine and kitchen hazards.

On two tests interaction effects revealed a differential impact of the Lifeskills intervention according to school achievement level. These were gas‐should‐do and gas‐never‐do. Intervention children improved immediately post‐intervention on both, regardless of their school's achievement level. But by three months there was a greater decline in knowledge among intervention children from lower achieving schools: gas‐should‐do, partial χ² = 7.42, df 2, p<0.05; gas‐never‐do, partial χ² = 8.06, df 2, p<0.05.

Discussion

The Lifeskills intervention produced immediate learning which remained at a similar level three months later. The average number of intervention children passing each test rose from 16% pre‐intervention to 62% immediately post‐intervention, and only fell to 54% three months later. The lack of fall off in knowledge between immediate and three month post‐intervention suggests that failure on the knowledge tests at three months was primarily due to lack of initial learning rather than subsequent forgetting.

Nevertheless, on two of the five tests, there was a three month fall off in intervention children's knowledge. This suggests that children had subsequently forgotten information they had successfully learned.

Relative complexity of instructional elements may be responsible for the different pattern of results on different tests. Some elements of the Lifeskills programme involve simple instruction of one or two messages (car stopping distance). Others are slightly more complex, involving recognition of several items (kitchen hazards). The most complex involve learning a set of actions to take in an emergency (fire escape routine, gas‐should‐do, gas‐never‐do).

All elements of the program, including the most complex, showed immediate post‐intervention improvement. At three months, decrements in knowledge occurred for two of the three most complex elements. In the case of what should never be done in the event of a gas leak, there was a slight fall off from 64% immediately post‐intervention to 52% at three months. For what should be done in the event of a gas leak, there was a dramatic fall off from 56% to 21%. The idea that informational complexity accounts for this failure of retention is supported by the interactions between school achievement and intervention on both tests of gas emergency knowledge. Immediately post‐intervention, children's knowledge showed improvement regardless of school background. At three months however children from lower achieving schools showed significantly greater loss of knowledge than children from higher achieving schools.

Conclusion

The two studies demonstrated the success of the Lifeskills program, which places emphasis on learning by doing in small interactive groups. The package successfully taught safety knowledge and, more importantly, accurate performance of safety skills. It was equally successful for boys and girls.

This success was uneven. The message about car stopping distance was well learned and retained. This is important as road accidents are the principal cause of injury and death in the age group targeted by the intervention.¹⁰ In contrast fewer children passed the gas‐should‐do test immediately after the intervention and even fewer three months later. This was primarily due to confusion about whether to use a land line or a mobile phone, a potentially dangerous failure in Lifeskills instruction given the widespread and growing use of mobile phones in this age group.¹¹

The results can be viewed with confidence. The pictorial tests were reliable. They showed test‐retest reliability, and levels of knowledge on each test were almost identical in the two studies. The samples were large. A quasi‐experimental design was employed involving matched control groups and in Study 2, pre‐intervention and post‐intervention measures. The studies did have limitations. Slightly less than half the program was evaluated owing to time restrictions during a school visit. Practical difficulties prevented the inclusion of a behavior measure in Study 2. Furthermore although the testing was conducted within a real (but safe) environment, it inevitably involved role playing rather than reactions to real‐life emergencies.

Evaluations may suggest potential solutions to problems they have uncovered.¹² Study 2 tested the children across time and thereby revealed shortcomings in the Lifeskills programme which would not have been revealed by testing at a single time point. Message complexity did not inhibit knowledge acquisition but was deleterious to its longer term retention thus showing the desirability of follow up practice for complex routines.

Key points

• Child safety training schemes are increasingly popular but systematic evaluation of whether children acquire new knowledge and skills is rare.

• The current report describes a matched control longitudinal evaluation of Lifeskills, a UK nationally funded safety skills program for 10–11 year old children which emphasizes learning by doing.

• The Lifeskills intervention improved children's knowledge and performance of safety skills.

• Knowledge and performance were highly correlated.

• The evaluation distinguished between immediate acquisition and longer term retention of knowledge. It revealed that complexity of teaching material did not affect acquisition but did affect retention, especially among children from lower achieving schools.

• An evaluation of this kind looking at a variety of types of instruction across three time points is practically useful in showing what needs changing and how it can be changed.