Program Fidelity Measures Associated With an Effective Child Restraint Program: Buckle-Up Safely

Abstract

Objectives. We sought to identify the program fidelity factors associated with successful implementation of the Buckle-Up Safely program, targeting correct use of age-appropriate child car restraints.

Methods. In 2010, we conducted a cluster randomized controlled trial of 830 families with children attending preschools and long day care centers in South West Sydney, New South Wales, Australia. Families received the Buckle-Up Safely program in the intervention arm of the study (13 services). Independent observers assessed the type of restraint and whether it was used correctly.

Results. This detailed process evaluation showed that the multifaceted program was implemented with high fidelity. Program protocols were adhered to and messaging was consistently delivered. Results from multilevel and logistic regression analyses show that age-appropriate restraint use was associated with attendance at a parent information session hosted at the center (adjusted odd ratio [AOR] = 3.66; 95% confidence interval [CI] = 1.61, 8.29) and adversely affected by the child being aged 2 to 3 years (AOR = 0.14; 95% CI = 0.07, 0.30) or being from a family with more than 2 children (AOR = 0.34; 95% CI = 0.17, 0.67).

Conclusions. Findings highlight the importance of parents receiving hands-on education regarding the proper use of age-appropriate child restraints.

Road traffic injury remains a leading cause of death and injury worldwide.¹ Young children are particularly vulnerable.² The protective effects of using child car restraints have been known for decades.^3–9 Legislation introduced into New South Wales, Australia, in 2010 amended previous legislation and mandated the use of age-appropriate child car restraints.¹⁰ Under this legislation, children aged 0 to 6 months must be in a rear-facing child car restraint, children aged 6 months up to 4 years must travel in a rear-facing or forward-facing child restraint and children aged up to 7 years must travel in either a forward-facing child car restraint or a booster seat.¹¹ For the purposes of this article, child car restraints are defined as per the Australian Standards (AS/NZS: 1754 2010)¹² and appropriate use as per the legislation.

Researchers have implemented and evaluated programs targeting increased use of size- or age-appropriate child car restraints^13–16 and their correct use.^16,17 A 2006 Cochrane systematic review of interventions targeting booster seat use in children aged 4 to 8 years concluded that interventions should be multifaceted.¹⁸ Not all programs have been successful. A recent, large-scale trial based in child care centers in the United States found no improvement in child booster seat use among 1010 children aged 4 to 8 years following a multifaceted education program including distribution of free booster seats.¹³ The authors concluded that further work was needed to identify effective methods and messages. That study would have likely benefitted from a detailed process evaluation to identify effective elements of the program and to assess program fidelity.

Process evaluations of injury prevention and safety promotion strategies are not routinely reported beyond a presentation of the proportion of people exposed to the intervention and perhaps to report factors that negatively affected program implementation.¹⁹ More often more detailed process evaluations accompany articles reporting a null effect, in an attempt to explain the result.^19,20 In the pursuit of greater efficiency and more targeted effectiveness of interventions, there is increasing demand from research grant providers and program planners to ensure evaluations include a greater emphasis on process evaluations. In 2004 Bellg et al., as part of the Treatment Fidelity Workgroup of the National Institutes of Health Behavior Change Consortium, presented recommendations to build treatment (or program) fidelity into health behavior intervention research.¹⁹ Those recommendations addressed elements of study design, provider training, treatment delivery, treatment receipt, and treatment enactment.²¹

Earlier process evaluations focused on formative evaluations to guide program development. Recently there has been increasing interest in summative process evaluations. These draw on measures of program fidelity (such as adherence, dose, quality of program delivery, participant responsiveness, and program differentiation)^22,23 that are then combined with program outcomes to provide insight into how the program implementation affected the study outcome. Summative process evaluations can minimize chances of a Type II error.²⁰ An example of a Type II error would be one in which an evaluation finds no difference between the intervention and control groups when an effect may have been seen had the program been implemented as designed.

Examining program fidelity in pragmatic trials also gives an indication of how likely it is that the program could be implemented as intended beyond the study population.²² A robust process evaluation can assess factors that may affect the uptake of the program in a given setting, such as attitudes of service providers toward the program or consistent delivery of key messages. As such, this can then inform practitioners and policymakers when planning to disseminate programs more broadly.

In 2010 we conducted a cluster randomized controlled trial in South West Sydney, New South Wales, Australia, targeting increased use of age-appropriate child car restraints and decreased misuse in children aged 3 to 5 years. The intervention program, called Buckle-Up Safely, was shown to be highly effective, particularly among families in which English was not the main language spoken at home.¹⁶ In this article, we aim to identify the program fidelity factors associated with successful implementation of the Buckle-Up Safely program.

METHODS

We recruited 28 early learning centers (preschools and long day care centers) in South West Sydney to participate in the cluster randomized controlled trial targeting optimal restraint use in children aged 3 to 5 years enrolled at the centers.¹⁷ Through a process of balanced randomization, centers were allocated either to receive the Buckle-Up Safely program or to continue with their usual educational program. After the conduct of the baseline surveys and the allocation of centers to either intervention or control arms, and before the start of the Buckle-Up Safely program, 1 center withdrew from the study, leaving 13 centers in the intervention arm and 14 in the control arm. Refer to Keay et al. for the CONSORT statement and sample size calculation for this cluster randomized controlled trial.¹⁶

Program Description

The Buckle-Up Safely program is described in detail elsewhere^17,24 and the program fidelity measures are presented against the program’s core components in Table 1. Briefly, the program involved staff professional development, a supported road safety education program, a parent information session at each center, access to subsidized restraints and vouchers for free fitting checks. Each core component of the program was guided by a training manual. Resnick et al. recommended that all behavior change studies develop training manuals to ensure consistency of message across sites.²⁵

TABLE 1—

Program Fidelity Measures for Each Component of the Buckle-Up Safely Program: South West Sydney, New South Wales, Australia, 2010

Description	Teachers’ Professional Development Workshop (Children’s Education Program)	Parent Information Session	Subsidized Restraint Scheme and Free Restraint Checks
Design
Attendance at each program component	No. staff attending the sessions	No. of parents attending the session	No. restraint fitters who attended the restraint fitters’ training session for their participation in the Buckle-Up Safely program
Provider credentials	Delivered by early childhood road safety consultant (yes/no)	Delivered by early childhood road safety consultant (yes/no)	Restraint checks conducted by authorised restraint fitter (yes/no)
Successful training to ensure delivery of the program	Session observed by researcher (yes/no)	Session observed by researcher (yes/no)	All restraint fitters received the Buckle-Up Safely restraint fitters’ manual (yes/no)
Ensuring program has been delivered as intended	All elements of the manual (“Out and About: Safe Journeys with Kids in Cars – OAKIC”) covered during workshop (yes/no)	All elements of the manual covered during workshop (yes/no)	Bag with restraint check voucher distributed to parents (yes/no)
Assessing that the program was received by the participants	Parents reporting having had a conversation with service staff about using child car restraints Feedback from staff about workshop	Parents reported attending an information session	Total no. subsidized restraints purchased per center Total no. restraint checks per center Parent reporting recent restraint purchase
Assessing level of uptake of the desired behavior	No. child restraint learning experiences implemented at the center (≥3 vs ≤2)	Outcome measures—observed child restraint use (age-appropriate use, correct use, and a binary measure of age-appropriate and correct use vs incorrect or age-inappropriate use)	Outcome measures—observed child restraint use (age-appropriate use, correct use, and the binary measure combining age-appropriate and correct use)

The center staff and research team managed the subsidized restraint distribution scheme. Restraints were fitted by authorized restraint fitters. In New South Wales restraint fitters operate as commercial entities within a network managed by the state government. All participating restraint fitters took part in an information session and received a manual specifically updated for the purposes of the Buckle-Up Safely program.

Theoretical Framework

The Buckle-Up Safely program messages focused on supporting people to progress through the stages of behavior change and were guided by the Precaution Adoption Process Model (PAPM),^26,27 a theoretical framework previously used in an intervention targeting child safety promotion practices.¹⁵

Program Fidelity Measures

Data were collected from the parent survey conducted as part of the Buckle-Up Safely program outcome evaluation, observations of the delivery of the professional development workshops and parent information sessions (for which a checklist was used to assess if all program content was delivered), 1 site visit to each center during the implementation of the children’s educational program, and an interview with each center director before the program began and once the program was finished.

One early childhood road-safety education consultant from Kids and Traffic Early Childhood Road Safety Education Program, an organization funded by the state government’s road safety agency (http://www.kidsandtraffic.mq.edu.au) conducted all the professional development and parent information sessions. After the professional development workshops, all centers received follow-up phone calls and site visits from members of the research team.

Program fidelity measures were guided by the Treatment Fidelity Workgroup of the National Institutes of Health Behavior Change Consortium,^19,25,28 as well as previous work conducted by Dusenbury²² and Gearing et al.²¹ Each program component was assessed for its design, training, delivery, receipt and enactment (Table 1).

Program acceptability, and potential threats to program fidelity, such as high staff turnover or a center director’s perception of child restraint information being of low or no relevance to their usual educational program, were assessed through data collected during semi-structured interviews with center directors. In these interviews, center directors were asked about staff turnover, how relevant the program was to their work, and their perceptions of how the program had been received by parents and other staff members.

Program Outcome Measures

Program outcome measures were published in the evaluation of the cluster randomized controlled trial.¹⁶ Briefly, child restraint use was observed by trained observers as children arrived at the centers in the morning. These observers were blinded to center allocation. The primary outcome was a 4-point ordinal measure combining both age-appropriate restraint use and correct use. For the process evaluation, this was collapsed to a binary measure: optimal (correct use of an age-appropriate restraint) versus sub optimal (incorrect use or not an age-appropriate restraint) because the ordinal measure did not meet proportional odds assumptions in this analysis.

Furthermore, we surmised that correct use and age-appropriate restraint use could be affected by different program factors and child or parental characteristics so we also assessed these separately to examine which factors were associated with each behavior.

Analysis

To examine program fidelity measures against observed restraint use we applied multilevel logistic regression. Level 1 variables were parent responses to the program evaluation survey and level 2 variables were center characteristics (such as observed implementation of the education program, staff retention, number of children aged 3 to 5 years enrolled; for further information see Table A, available as a supplememnt to the online version of this article at http://www.ajph.org). Univariate cut-off entry into the models was at a P level of less than or equal to .10 for optimal restraint use (age-appropriate restraint and used correctly) and any significant errors and at a P level of less than or equal to .05 for age-appropriate restraint use as a result of small numbers (n = 58) in 1 of the categories. Measures in which all the scores were the same across all centers were not included in the analyses. For example, all professional development workshops and parent information sessions were observed by a member of the research team and all elements of those sessions were covered as per their respective manuals. Two measures were not included in the modeling process because of missing data: annual household income (52; 16% missing) and parents’ perception that their child had traveled to the center using the same sort of restraint as other children the same age (30; 9% missing). Least significant variables were progressively eliminated from each model until only those with a P level of less than .05 remained. Adjusted odds ratios (AOR) and their 95% confidence intervals (CIs) were estimated from the final models.

We performed all statistical analyses using Stata 12 (StataCorp LP, College Station, TX). Interviews with center directors were thematically analyzed to assess center directors’ responses to the program.

RESULTS

Program fidelity measures and observed restraint use are presented for the 328 participants in the intervention arm (Table B, available as a supplement to the online version of this article at http://www.ajph.org, presents the program fidelity measures at each center).

Program Fidelity

Professional development workshop and education program.

All components of the professional development workshops and parent information sessions were covered at each center as per the manual (the “Out and About, Safe Journeys with Kids in Cars” Workshop Training Manual). Overall, the professional development workshops were well attended. The proportion of early childhood educators from each center who attended the workshops ranged from 10% to 100% with more than 80% of staff attending from 5 centers. Directors reported sharing the information with those staff members unable to attend.

The level of implementation of the Buckle-Up Safely children’s education components were assessed during the site visits. All centers had implemented at least 1 education suggestion and 7 centers had implemented more than 3. All centers also displayed posters that had been specially developed for the Buckle-Up Safely program.

Parent information session.

At each session, all components of the parent information session were covered according to the manual (Parent Information Training Manual). Attendance at the parent information session varied greatly within each center. One center had a policy that parents were expected to attend a certain number of parent meetings each year and the information session was considered one of those meetings so was attended by 36 (60%) of parents of children aged 3 to 5 years. Conversely, there were 6 centers in which fewer than 10 parents attended the information session. Though parent attendance at the information sessions was not strong at all the centers, center directors reported that they believed the numbers were similar to, and in some instances better than, the numbers at other parent meetings they hosted.

Restraint distribution scheme.

Assessment of the restraint distribution scheme showed 163 restraints were purchased and 204 restraints checked by certified restraint fitters across all intervention centers, representing 20% and 25% of enrolled children aged 3 to 5 years respectively. We did not collect data on how effectively this manual was utilized by the fitters.

Observed restraint use.

Correct use of age-appropriate restraints was observed in 140 of 341 (43%) participants. In univariate analyses, 3 factors were significantly associated with optimal restraint use, and remained significant in multilevel multivariate logistic regression models (Table 2). Children whose parent reported knowing that a forward-facing restraint was better than a booster seat for a 3-year-old child had almost twice the odds of being in an age-appropriate restraint used correctly (adjusted odds ratio [AOR] = 1.82; 95% CI = 1.03, 3.26; P = .04). Parents who reported they had received brochures about child restraints from their early learning center had twice the odds of their child being in an age-appropriate restraint used correctly (AOR = 1.99; 95% CI = 1.12, 3.55; P = .02). Conversely, children aged 2 to 3 years had significantly lower odds than children aged 4 to 5 years of being in an age-appropriate restraint used correctly (AOR =  0.28; 95% CI = 0.16, 0.49; P < .001; Table 2).

TABLE 2—

Multilevel Logistic Regression of Characteristics Associated With Optimal Restraint Use: South West Sydney, New South Wales, Australia, 2010

Variable	Univariate OR (95% CI)	Multivariate AOR (95% CI)
Individual-level factors
Child < 4 y	0.29 (0.17, 0.49)	0.28 (0.16, 0.49)
Parent < 36 y	0.77 (0.49, 1.20)
Parent’s education beyond high school	1.36 (0.83, 2.23)
Household income < AUD $60 000	0.72 (0.44, 1.18)
Knows a forward-facing restraint offers a 3-year-old child greater benefit than a booster seat	1.86 (1.07, 3.23)	1.82 (1.03, 3.26)
Belief that child travels same way as other children	1.17 (0.79, 3.55)
Cost prevents parent buying their preferred restraint	0.88 (0.53, 1.45)
At least 3 children in family	0.87 (0.54, 1.39)
Individual restraint action
Restraint was bought within the previous 6 mo	1.71 (0.95, 3.07)
Restraint had been checked by restraint fitter in last 6 mo	1.08 (0.77, 1.78)
Parent reported having received restraint information at information session at center	1.17 (0.76, 1.86)
Parent reported having received information via brochures	1.90 (1.12, 3.25)	1.99 (1.12, 3.55)
Center-level characteristics
Children’s education well implemented	1.36 (0.87, 2.12)
Up to 1 new staff member	0.79 (0.50, 1.25)
Observer’s perceived measure of support from center staff	1.36 (0.97, 1.92)
Center restraint stats
Total no. restraints bought at center	1.02 (0.95, 1.09)
Total no. restraints fitted at center	1.01 (0.97, 1.06)
Service is run by local council	1.03 (0.65, 1.64)

Note. AOR = adjusted odds ratio; CI = confidence interval; OR = odds ratio. Evaluation of optimal restraint use was a binary measure for correct use of age-appropriate restraint versus incorrect or age-inappropriate restraint use. All models were adjusted for clustering by center. ORs were for optimal restraint use; AORs were adjusted for other variables in the column.

A majority of participants were using age-appropriate restraints (n = 270; 82%). In multivariate analyses, those significantly less likely to be in age-appropriate restraints were children aged 2 to 3 years compared with children aged 4 to 5 years (AOR = 0.14; 95% CI = 0.07, 0.30; P < .001) and children of families with at least 3 children in the family versus smaller families (AOR = 0.34; 95% CI = 0.17, 0.67; P = .002). Children of parents who reported having received information about child restraints from an information session at the center had 3.66 times the odds of being in an age-appropriate restraint (95% CI = 1.61, 8.29; P = .002; Table 3).

TABLE 3—

Multilevel Logistic Regression of Factors Associated With Age-Appropriate Restraint Use: South West Sydney, New South Wales, Australia, 2010

Variable	Univariate OR (95% CI)	Multivariate AOR (95% CI)
Individual-level factors
Child < 4 y	0.20 (0.10, 0.38)	0.14 (0.07, 0.30)
Parent < 36 y	0.56 (0.30, 1.05)
Household income < AUD $60 000	0.53 (0.29, 0.98)
Knows a forward-facing restraint offers a 3-year-old child greater benefit than a booster seat	2.20 (1.15, 4.21)
Belief that child travels same way as other children	2.12 (0.94, 4.77)
Cost prevents parent buying their preferred restraint	0.48 (0.26, 0.88)
At least 3 children in family	0.47 (0.26, 0.88)	0.34 (0.17, 0.68)
Restraint was bought within the previous 6 mo	1.36 (0.60, 3.06)
Restraint had been checked by restraint fitter in last 6 mo	0.98 (0.51, 1.87)
Parent report of restraint information
Parent reported having received restraint information at an information session at center	2.39 (1.20, 4.78)	3.66 (1.62, 8.29)
Parent reported having received information via brochures	2.46 (1.35, 4.48)
Parent reported having received information via DVD	1.79 (0.92, 3.48)
Center-level factors
Children’s education well implemented	1.35 (0.98, 1.88)
No new staff	0.78 (0.44, 1.38)
Observer’s perceived measure of support from center staff	1.54 (0.97, 2.43)
Total no. restraints bought at center	1.08 (0.99, 1.18)
Total no. restraints fitted at center	1.05 (0.99, 1.11)

Note. AOR = adjusted odds ratio; CI = confidence interval; OR = odds ratio. All models were adjusted for clustering by center. ORs were for optimal restraint use; AORs were adjusted for other variables in the column.

We observed correct use of restraints in 182 cases (56%). Restraint type was the only variable associated with correct use. Forward-facing restraints and rear-facing restraints combined had greater odds of being incorrectly used compared with booster seats (AOR = 2.57; 95 % CI = 1.62, 4.07; P < .001). No other individual level factors or program measures were significantly associated with correct use of restraints (Table 4).

TABLE 4—

Multilevel Logistic Regression of Factors Associated With Major Errors of Restraint Use: South West Sydney, New South Wales, Australia, 2010

Variable	Univariate OR (95% CI)
Individual-level factors
Child < 4 y	1.55 (0.97, 2.48)
Parent < 36 y	1.07 (0.69, 1.67)
Household income < AUD $60 000	1.16 (0.72, 1.90)
Using a rear- or forward-facing restraint vs a booster seat	2.57 (1.62, 4.07)
Knows a forward-facing restraint offers a 3-year-old child greater benefit than a booster seat	0.71 (0.42, 1.20)
Belief that child travels same way as other children	0.97 (0.48, 1.96)
Cost prevents parent buying their preferred restraint	0.86 (0.52, 1.42)
At least 3 children in family	0.80 (0.50, 1.23)
Restraint was bought within the previous 6 mo	0.71 (0.39, 1.27)
Restraint had been checked by restraint fitter in last 6 mo	0.91 (0.55, 1.50)
Parent reported having received restraint information at information session at center	1.37 (0.87, 2.16)
Parent reported having received information via brochures	0.80 (0.48, 1.32)
Parent reported having received information via DVD	0.91 (0.57, 1.45)
Center-level factors
Children’s education well implemented	0.89 (0.57, 1.38)
< 2 new staff members	1.07 (0.69, 1.68)
Observer’s perceived measure of support from center staff	0.84 (0.57, 1.17)
Total no. restraints bought at center	1.02 (0.96, 1.10)
Total no. restraints fitted at center	1.02 (0.98, 1.06)
Service is run by local council	1.01 (0.65, 1.56)

Note. CI = confidence interval; OR = odds ratio. All models were adjusted for clustering by center. ORs were for optimal restraint use.

Potential Threats to the Fidelity of the Program

Staff turnover was minimal across the centers. There were no teaching staff changes in 5 centers, only 1 new staff member in 6 centers, 2 in 1 center, and 1 center had 3 new staff members, representing 42% of that center’s teaching staff. Before the Buckle-Up Safely program began, 9 (69%) center directors said the delivery of road safety education to children was highly relevant to early childhood education.

Center Directors’ Response to the Buckle-Up Safely Program

The Buckle-Up Safely program was well received by each of the directors. Three centers identified a key person who was committed to driving the program implementation for their center. The professional development workshop was well regarded with a director stating: “Phenomenal program—the staff, children, and parents were all educated,” while another reported “All staff were so motivated by the workshop and felt it was the best one we’d attended.” Despite the parent information sessions not having large numbers of parents attending, center directors reported sharing information with parents who could not attend the session. As one director put it:

Parent information session was fantastic! The parents responded so well; would love to be able to do it each year at one of our parent meetings; the content was clear; parents were raving about it—telling other parents about it . . . I personally gave the bags out to the parents who couldn’t attend the session—I went through everything that was in the bags with each parent.

Directors identified several benefits for the children, parents, teachers, and the broader community. When asked for suggestions to improve the program, directors suggested more time to deliver the program, access to more resources and expansion of the program to children younger than 3 years. Overall, directors found the program extremely beneficial, with 1 director saying: “This has been a huge success; I’ve told all the other centers in [the region] to take it up if it’s ever offered to them; I thanked our children services manager for allowing us to have this,” while another said “Having an expert in the field to speak with families was fantastic; posters (especially the translated posters) resonated with parents; the information has spread through word of mouth.”

DISCUSSION

Buckle-Up Safely is a multifaceted education-based program delivered through early childhood centers, which has previously been shown to be effective in increasing correct use of age-appropriate restraints.¹⁶ This analysis has shown that the Buckle-Up Safely program was delivered with high fidelity and was well received by center directors who all felt the program was relevant to their education program for the children; more than half felt it was highly relevant.

To our knowledge, this is the first study targeting correct use of age-appropriate child car restraints to attempt to examine process measures in relation to program outcomes.

We identified factors that determined age-appropriate restraint use but it remains unknown what influences correct restraint use alone, despite an increase in correct restraint use occurring in the intervention centers.¹⁶ The age of the child was strongly associated with age-appropriate restraint use, with children aged 2 to 3 years less likely to be in an age-appropriate restraint than children aged 4 to 5 years. Similarly, having more than 2 children in the family was associated with a child not being in the right restraint for age. These findings are consistent with earlier research on the determinants of optimal restraint use.²⁹

Parents able to recognize the safety benefit for a 3-year-old child being restrained in a forward-facing restraint rather than a booster seat was significantly associated with a child being in a correctly used age-appropriate restraint, a finding that supports previous research.³⁰ This result supports earlier findings in meta-analysis that education can be effective in increasing age-appropriate use of restraints.¹⁸ Attendance at the parent information session was associated with increased use of age-appropriate restraints. Similarly parents reporting their center had given them brochures about restraints was associated with the combined measure of age-appropriate restraint and their correct use.

We had expected that the subsidized restraint scheme, as measured by the number of restraints recently purchased and number of restraint fitting checks recently conducted would significantly impact either correct use or use of age-appropriate restraints. Results from previous studies offering a restraint distribution scheme coupled with education, pooled in a meta-analysis, found the distribution scheme with education increased age-appropriate restraint use.¹⁸ Access to accredited restraint fitters has also been shown to produce increases in age-appropriate restraints and their correct use.³¹ Significantly more parents at intervention than control centers reported that their child’s restraint was purchased and had been checked by a restraint fitter in the previous 6 months. We powered the study to detect a difference in restraint use between the intervention and control groups in the clustered randomized controlled trial. Given a larger sample size, the restraint distribution scheme could possibly be found to be a significant component of the program.

Limitations

There are some limitations of this process evaluation. We did not assess how closely the restraint fitters adhered to the core Buckle-Up Safely messages, nor did we assess restraint fitters’ perception of and experience with the Buckle-Up Safely program. Without those insights, we cannot confidently report on the fidelity of that aspect of the program.

Conclusions

We conducted a detailed summative process evaluation guided by recommendations from the National Institutes of Health Treatment Fidelity Workgroup^19,28 and other authors.^21,22 We found preschools and long day care centers were viable settings to deliver programs targeting child road safety and that center directors were highly supportive of the program. Putting a child in the right restraint for their age and using the restraint correctly were associated with different program factors and different child or family characteristics. This evaluation supports the need to implement a multifaceted program when targeting both correct use and age-appropriate child car restraint use.