Controlled ecological evaluation of an implemented exercise training programme to prevent lower limb injuries in sport: differences in implementation activity

Abstract

Objective

The public health benefits of injury prevention programmes are maximised when programmes are widely adopted and adhered to. Therefore, these programmes require appropriate implementation support. This study evaluated implementation activity outcomes associated with the implementation of FootyFirst, an exercise training injury prevention programme for community Australian football, both with (FootyFirst+S) and without (FootyFirst+NS) implementation support.

Method

An evaluation plan based on the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) Sports Setting Matrix was applied in a controlled ecological evaluation of the implementation of FootyFirst. RE-AIM dimension-specific (range: 0–2) and total RE-AIM scores (range: 0–10) were derived by triangulating data from a number of sources (including surveys, interviews, direct observations and notes) describing FootyFirst implementation activities. The mean dimension-specific and total scores were compared for clubs in regions receiving FootyFirst+S and FootyFirst+NS, through analysis of variance.

Results

The mean total RE-AIM score forclubs in the FootyFirst+S regions was 2.4 times higher than for clubs in the FootyFirst+NS region (4.73 vs 1.94; 95% CI for the difference: 1.64 to 3.74). Similarly, all dimension-specific scores were significantly higher for clubs in the FootyFirst+S regions compared with clubs in the FootyFirst+NS region. In all regions, the dimension-specific scores were highest for reach and adoption, and lowest for implementation.

Conclusion

Implementing exercise training injury prevention programmes in community sport is challenging. Delivering programme content supported by a context-specific and evidence-informed implementation plan leads to greater implementation activity, which is an important precursor to injury reductions.

Background

There is considerable high-quality evidence that exercise training programmes can prevent lower limb injuries (LLI) in sport.^1–4 However, relatively few studies have investigated the implementation or effectiveness of such injury prevention programmes (IPPs) in real-world settings and contexts.^{5 6} Frameworks such as the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework provide a solid basis for designing and evaluating the implementation of IPPs.^{7 8} However, the specific RE-AIM components for the implementation of IPPs in team ball sports are rarely reported.⁶

Numerous cross-sectional studies have reported the level of awareness, adoption and implementation of LLI prevention programmes among sports participants and coaches.^9–13 In general, these have found moderate to high awareness, limited programme use and very limited programme fidelity. Several cross-sectional follow-up studies assessing coach awareness or uptake of, or player compliance with, LLI prevention programmes following an intervention (eg, coach training workshop, resource dissemination, programme delivery by an external agent and so on) have also reported high post-intervention awareness, moderate programme use and high programme modification.^14–18

Four published studies have evaluated the impact of different strategies on the implementation of sports injury prevention interventions by community sports coaches and participants. In a study of the impact of a squash eye protection promotion strategy, players exposed to the strategy were 2.4 times more likely to wear appropriate eyewear than players exposed to usual implementation practice.¹⁹ An examination of team and player adherence to the FIFA11+ LLI prevention programme showed that soccer teams with coaches exposed to an 11+ workshop performed a significantly higher number of 11+ exercises compared with teams with coaches who accessed the 11+ online.²⁰ Rugby union coaches exposed to a theory-informed, evidence-informed and context-informed diffusion plan to promote the uptake of a neck and spinal IPP had greater increases in programme knowledge, frequency of training players in the programme and perceived quality of programme delivery than control coaches.²¹ Finally, a study comparing an interactive mobile application to written materials to implement an ankle IPP showed equal compliance among athletes exposed to either implementation method.²²

The National Guidance for Australian Football Partnerships for Safety study is one of the first large-scale sports injury prevention implementation studies to use a controlled, ecological design to assess the effectiveness of an exercise training IPP at the population level. Its associated IPP, known as FootyFirst, is an exercise training programme developed to target LLI in community Australian football (community AF).²³ An evidence-informed and context-specific implementation plan was developed to raise awareness of FootyFirst and to support its uptake and delivery in community AF clubs in three specific geographic regions.²⁴

This paper evaluates the impact of the multistrategy implementation plan on FootyFirst-related implementation activities. Implementation activities are compared across community AF regions and clubs that received FootyFirst with implementation support (FootyFirst+S) and FootyFirst without implementation support (FootyFirst+NS). It was hypothesised that there would be greater FootyFirst implementation activity among clubs in the FootyFirst+S regions than in clubs in the FootyFirst+NS region.

Methods

A controlled ecological trial, underpinned by the principles of pragmatic health research,²⁵ was conducted with community AF clubs. Full details of the study design, including additional information about the choice and locations of the three geographical regions, are available elsewhere.²⁶ The evaluation design was underpinned by the RE-AIM Sports Setting Matrix.⁸ This paper reports the impact of the implementation plan on the FootyFirst implementation activity—reach, perceived effectiveness, adoption, implementation and maintenance—among the targeted clubs.

Participants

The study was conducted in community AF clubs from three well-defined geographically distinct and socioeconomically similar regions in Victoria, Australia. The regions were chosen because of their representative and strong microcosms of community AF, their comparable playing standards and administrative capacity. The characteristics of the clubs and leagues within the three regions are summarised in table 1.

Table 1.

The characteristics of the leagues participating in the FootyFirst evaluation according to the year and level of implementation support they received

	Region 1 FootyFirst+S* Year 1 and year 2	Region 2 FootyFirst+NS† Year 1 and year 2	Region 3 FootyFirst+S* Year 2 only
No. of governing/administrative bodies	1	1	1
No. of clubs and competitions	22 clubs in two competitions	25 clubs in two competitions	31 clubs in three divisions
	No promotion/relegation between competitions	No promotion/relegation between competitions	Promotion/relegation between divisions
No. of teams per club	2 (seniors and reserves)	2 (seniors and reserves)	2 (seniors and reserves)
Approximate no. of registered players‡	1100	1250	1510

*FootyFirst with implementation support.

†FootyFirst without implementation support.

‡Based on 25 registered players per team.

FootyFirst

The IPP offered to all clubs across the three regions was FootyFirst, an evidence-informed, expert-endorsed progressive exercise training programme to reduce LLIs (ankle, knee, hamstring and hip/groin injuries) in adult, male community AF players. The evidence underpinning FootyFirst,^{1 27 28} and the process used to develop it,^{23 24} are described elsewhere. In summary, FootyFirst consisted of a dynamic warm-up to prepare players to participate in the programme, followed by leg strengthening and conditioning exercises, and training to improve balance and landing and side-stepping/cutting/change of direction techniques (details at: http://www.aflcommunityclub.com.au). The programme was divided into four components (hamstring strength, groin strength, hip strength, and balance, landing and changing direction) with five levels of progression for each component. Progression was recommended when players had the strength, muscular endurance, flexibility and movement skills to complete a level with the correct technique. FootyFirst was developed as a replacement for traditional community AF team pre-training warm-up programmes and to take approximately 20 min to complete with minimal equipment required. It was designed to be included in all community AF team training sessions and to be completed at least twice a week. A comprehensive coaches’ manual (with contents also provided on a compact disc), instructional video and large posters (one for each programme level) were developed to support the programme content (details at: http://www.aflcommunityclub.com.au/index.php?id=1905).

The implementation support

An ecological approach, using step 5 of the Intervention Mapping health promotion programme planning protocol,²⁹ was taken to develop a FootyFirst implementation plan in the FootyFirst+S regions (regions 1 and 3). The development of the implementation plan for region 1, including the rationale for the specific strategies included in the plan, is described elsewhere,²⁴ and a similar process was used to develop the implementation plan for region 3. All implementation strategies were designed around achieving the five RE-AIM dimensions of reach, perceived effectiveness, adoption, implementation and maintenance adapted for community sport.⁸

The specific strategies to facilitate the implementation of FootyFirst in the FootyFirst+S regions, and the RE-AIM dimensions they were designed to influence, are summarised in online supplementary file 1. Region 2 received FootyFirst and no additional implementation support (FootyFirst+NS), reflecting usual implementation practice in community AF (see online supplementary file 1).

Evaluation data collection

Due to the multicomponent nature of the strategies incorporated in the implementation plan, a convergent parallel, mixed-method evaluation design³⁰ was used to facilitate a comprehensive evaluation with valid inferences. This involved integrating qualitative and quantitative data to triangulate multiple perspectives.³¹ Data sources (described fully in online supplementary file 2) included:

• preseason and postseason surveys with coaches and club administrators

• postseason semistructured interviews with coaches and club administrators

• club-reported weekly implementation data

• direct observation of training sessions

• internal programme documentation and records of club participation in implementation strategies.

Table 2 summarises the different sources of data and implementation evidence obtained from the clubs in each region.

Table 2.

Number (%) of clubs in each region for which data was available from the different sources of data/evidence for the RE-AIM scoresheet compilation

Source of evidence	Region 1 (n=22) FootyFirst+S	Region 2 (n=25) FootyFirst+NS	Region 3 (n=31) FootyFirst+S
FootyFirst survey	17 (77)	9 (36)	31 (100)
FootyFirst interview	11 (50)	0 (0)	8 (29)
FootyFirst-related active communication (telephone and email)	21 (95)	25 (100)	28 (90)
FootyFirst resources	20 (91)	25 (100)	26 (84)
FootyFirst-related meetings	19 (86)	3 (12)	22 (71)
Footy-First related passive communication (Twitter, autotext)	15 (68)	0 (0)	14 (45)
FootyFirst-related event (training, launch, expo or advisory group meeting)	19 (86)	0 (0)	29 (94)
Weekly FootyFirst implementation data	12 (56)	0 (0)	4 (13)
FootyFirst-related observation	10 (45)	0 (0)	8 (26)
Research assistants’ notes and recall	22 (100)	25 (100)	31 (100)
‘Intention to implement FootyFirst’ form	16 (73)	2 (8)	17 (55)

FootyFirst+NS, FootyFirst without implementation support; FootyFirst+S, FootyFirst with implementation support; RE-AIM, Reach Effectiveness Adoption Implementation Maintenance.

Development of a RE-AIM scoresheet

There is little guidance available on how to use the RE-AIM framework as an evaluation tool. Therefore, a novel evaluation process (detailed in online supplementary file 3) was developed by two of the authors (AD and CFF). A RE-AIM scoresheet summarising the evidence available from the various data sources under each of the five RE-AIM dimensions was created for each participating club. For all clubs, the scoresheet comprised 26 items as follows:

• 11 reach-related items (1 each from the postseason survey and the postseason interview and 9 from internal programme documentation and records)

• 2 perceived effectiveness-related items (1 each from the postseason survey and the postseason interview)

• 6 adoption-related items (2 from internal programme documentation and records and 1 each from the postseason survey, the postseason interview, observations and self-report weekly implementation data)

• 5 implementation-related items (1 each from the postseason survey, postseason interview, internal programme documentation and records, observations and self-report weekly implementation data)

• 2 maintenance-related items (1 each from the postseason survey and the postseason interview).

Three additional scoresheet items for region 3 clubs included: one item in each of the reach, perceived effectiveness and adoption dimensions based on evidence extracted from the 2013 coach and administrator preseason surveys.

RE-AIM scoresheet coding, scoring and analysis

The research team (AD and two research assistants) identified, collated and agreed on the available evidence from all data sources for each item for each club. This was then summarised in a RE-AIM scoresheet for each club (see online supplementary file 3). Two independent assessors then rated the evidence for each of the 29 items in each of the 78 RE-AIM scoresheets as: evidence of yes (ie, evidence supported achievement of the dimension aim), evidence of no (ie, evidence supported non-achievement of the dimension aim), no evidence or unsure. The assessors then assigned an overall rating of achievement for the aim for each RE-AIM dimension and assigned a score as: ‘Not Achieved’=0, ‘Partially Achieved’=1 and ‘Fully Achieved’=2. A total RE-AIM score (range 0–10) was generated for each club by summing the five RE-AIM dimension scores. Higher scores indicate more implementation activity.

RE-AIM dimension and total RE-AIM scores were compared across regions using one-way analysis of variance, after checking for normality of the data. Differences in the total RE-AIM scores for the clubs in the FootyFirst+S regions and the clubs in the FootyFirst+NS region are presented as a difference in the means, with 95% CIs. All analysis was performed with SPSS (V.22).

Results

The total RE-AIM scores were more than 2.1 times higher for clubs in the two FootyFirst+S regions than for clubs in the FootyFirst+NS region, after the first year of implementation of FootyFirst in each region (table 3). In addition, there was no significant difference between the total RE-AIM scores between the clubs in the two FootyFirst+S regions after the first year of implementation. This suggests that the addition of the implementation support to FootyFirst helped facilitate the desired implementation activity in the targeted clubs.

Table 3.

Mean RE-AIM dimension-specific (range 0–2) and total RE-AIM (range 0–10) scores for all regions after 1 year of FootyFirst implementation

	Reach		Perceived Effectiveness		Adoption		Implementation		Maintenance		Total RE-AIM
	Score	95% CI	Score	95% CI	Score	95% CI	Score	95% CI	Score	95% CI	Score	95% CI
FootyFirst+S regions
Region 1 (2012)	1.91	1.72 to 2.00	0.77	0.41 to 1.13	1.36	0.94 to 1.78	0.36	0.04 to 0.69	0.86	0.45 to 1.28	5.27	3.97 to 6.58
Region 3 (2013)	1.86	1.66 to 2.00	0.73	0.34 to 1.12	1.05	0.60 to 1.49	0.09	0.00 to 0.22	0.45	0.10 to 0.81	4.18	3.06 to 5.30
FootyFirst+NS region
Region 2 (2012)	1.36	1.13 to 1.59	0.16	0.01 to 0.31	0.24	0.00 to 0.51	0.00	0.00 to 0.00	0.20	0.00 to 0.44	1.92	1.21 to 2.63

FootyFirst+NS, FootyFirst without implementation support; FootyFirst+S, FootyFirst with implementation support; RE-AIM, Reach Effectiveness Adoption Implementation Maintenance.

There was no evidence of implementation of FootyFirst among clubs in the FootyFirst+NS region, and these clubs had lower dimension-specific scores for nearly all of the other RE-AIM dimensions, compared with clubs in the FootyFirst+S regions (table 3). There were non-zero dimension-specific scores for all RE-AIM dimensions for clubs in the FootyFirst+S regions.

When the RE-AIM dimension and total RE-AIM scores were compared after 2 years of FootyFirst implementation in region 1 (FootyFirst+S) and region 2 (FootyFirst+NS), clubs in region 1 scored significantly higher on every RE-AIM dimension than did clubs in region 2: reach: 1.89 versus 1.44; perceived effectiveness: 0.75 versus 0.18; adoption: 1.20 versus 0.22; implementation: 0.23 versus 0.00; maintenance: 0.66 versus 0.14 (figure 1).

Figure 1.

Comparison of mean RE-AIM dimension scores (range 0–2) and total RE-AIM scores (range 0–10) for clubs in region 1 (FootyFirst+S) and region 2 (FootyFirst+NS), after 2 years of FootyFirst implementation. FootyFirst+NS,  FootyFirst without implementation support; FootyFirst+S, FootyFirst with implementation support; RE-AIM, Reach Effectiveness Adoption Implementation Maintenance.

Discussion

There has been considerable attention given to preventing LLIs in sport over recent years, and several exercise training interventions have been developed and evaluated. However, the effectiveness of these interventions has been limited by the extent to which the target groups (eg, players or coaches) have implemented them.^{18 32} The majority of the evidence of the efficacy exercise training interventions comes from RCT trials, many of which have rarely considered implementation factors. When studies have considered implementation issues, this has typically been as a minor component, and most have reported limited aspects of intervention implementation.³³ The evaluation reported in this paper addresses this significant gap in the literature by demonstrating how to influence IPP implementation processes and outcomes.

There are many challenges involved in conducting implementation research in real-world settings, and a pragmatic approach is warranted to make best use of multiple and varied data sources. It is important to recognise that injury reductions will only occur when implementation activities, such as those that increase awareness leading to actual performance of the intervention, are achieved successful. The vast majority of previous studies related to exercise training IPPs in sport have not considered whether intervention target groups actually implemented the interventions as intended.³³ This is problematic because this information is needed to link injury reductions to the implemented intervention. Moreover, how an individual responds to implementation activities is significantly influenced by a range of factors such as how the intervention is implemented, who implements it and the broader ecological system in which it is implemented. Only one other study—an evaluation of an exercise training programme in junior soccer players exposed to coach-led programme delivery with or without additional physiotherapist involvement²⁰—has specifically compared programme implementation strategies. However, that evaluation was part of a highly controlled RCT and the measure of implementation activity was limited to adherence (defined as the proportion of all possible sessions where the 11+ was delivered, the number of team 11+ sessions/week and the mean number of team 11+ exercises/session).

The RE-AIM Sports Setting Matrix was developed to guide the design and evaluation of sports IPPs, especially those delivered through community sport.⁸ It achieves this through its extension of the RE-AIM framework³⁴ developed to incorporate the different ecological levels of the sports delivery system. The evaluation presented in this paper used the RE-AIM Sports Setting Matrix to evaluate multiple aspects of the implementation of an exercise training intervention at the team, club and regional league levels, in relation to different levels of implementation support. Overall, the mean total RE-AIM score was significantly higher for clubs in the FootyFirst+S regions than for clubs in FootyFirst+NS region, indicating that the implementation support had a major impact on FootyFirst implementation activity. Moreover, on every RE-AIM dimension, the clubs in the FootyFirst+S regions scored significantly higher than the clubs in FootyFirst+NS region.

The reach dimension score measured the awareness of FootyFirst among the intended target groups. This was the highest scoring RE-AIM dimension, approaching maximum scores for clubs in the FootyFirst+S regions. The reach dimension score was still high among clubs in the FootyFirst+NS region (although lower than for clubs in the FootyFirst+S regions) and the highest for any RE-AIM dimension in the FootyFirst+NS region clubs. This is not surprising as all clubs in all regions were provided with the FootyFirst resources, so awareness should have been high.

In this evaluation, the original RE-AIM effectiveness dimension³⁴ was modified to a measure of perceived effectiveness because, according to the Diffusion of Innovations theory,³⁵ the rate of translation of an innovation, in contrast to the public health impact, depends more on the end-users’ subjective perception of its effectiveness than it does on objective evidence of an innovation’s efficacy. There was a significantly higher score for the perceived effectiveness dimension for clubs in the FootyFirst+S regions than for clubs in the FootyFirst+NS region. This probably reflects the additional educational activities provided to the former, as part of the implementation plan. The fact that FootyFirst was perceived to be more effective when accompanied by the implementation plan is also demonstrated by lower injury counts in regions where clubs were exposed to FootyFirst+SF (Finch et al unpublished data, 2017).

The RE-AIM adoption dimension is best understood as a measure of the extent to which the intended target groups used FootyFirst. Among clubs in all regions, the adoption dimension scored second highest, and the mean scores were significantly higher for clubs in the FootyFirst+S regions than those in the FootyFirst+NS region. This suggests that clubs had difficulty transitioning from awareness of, to use of, FootyFirst without implementation support.

The lowest scoring RE-AIM dimension was implementation, though the score for this dimension was still significantly higher for clubs in the FootyFirst+S regions than for clubs in the FootyFirst+NS region. The implementation dimension is a measure of the extent to which the intended target groups used FootyFirst properly. We have previously shown that the fidelity of FootyFirst implementation was low in a sample of clubs because it was not delivered as intended.³⁶ The low implementation scores among clubs in the FootyFirst+S regions may be due to the high standard set to achieve the implementation dimension aim, with all players required to complete the correct number of all exercises, at least twice a week, and advance through appropriate progressions. It is also possible that the implementation support provided was insufficient to ensure improvements in the implementation activities measured for this dimension.

The last RE-AIM dimension, maintenance, was a measure of coach and club intentions to continue using FootyFirst. The mean maintenance score was significantly higher in clubs in the FootyFirst+S regions than for clubs in the FootyFirst+NS region. It is possible that the implementation support provided, and higher adoption and implementation scores among clubs in the FootyFirst+S regions, demonstrated to coaches and clubs in those regions that FootyFirst was feasible and beneficial, and this was reflected in opinions about its ongoing value.

This evaluation demonstrates that the additional implementation support provided to clubs in the FootyFirst+S regions facilitated higher levels of implementation activity among those clubs compared with clubs in the FootyFirst+NS region. However, the perceived effectiveness, implementation and maintenance mean scores for clubs in FootyFirst+S regions were all between 0 and 1 (ie, located between ‘Partially Achieved’ and ‘Not Achieved’), and the 95% CI for the total RE-AIM mean scores for clubs in FootyFirst+S regions included the midpoint value of 5.00. This may reflect the difficulty in gathering data related to these specific RE-AIM dimensions. Alternatively, the implementation support provided may not have been sufficient to facilitate high levels of programme implementation and maintenance among the targeted clubs. It is beyond the scope of this paper to report on the facilitators and barriers to FootyFirst implementation. Data related to these issues were gathered in the postseason surveys and interviews and will be reported in a subsequent publication. It is anticipated that barriers to implementing FootyFirst that emerge from this study will be similar to the barriers to the adoption and use of LLI prevention exercise programmes in other team ball sports including the: nature of the IPP, competency and confidence of coaches, attitudes and beliefs of coaches and players, support from external sources and the time and resources available to implement IPPs at training.^{33 37} There are some limitations to the approach adopted in this study. First, there was no single objective data source available to measure the achievement of the aims of the RE-AIM dimensions. Therefore, a pragmatic evaluation approach was adopted that relied on a convergent design, merging quantitative and qualitative data from a range of sources for an overall interpretation. However, the reliability of the coding approach developed for this study was assessed (see online supplementary file 3), and there were very high levels of agreement across independent assessors suggesting robustness in the findings.

Second, this evaluation relied on the availability and completeness of data from the multiple data sources. Evidence supporting the level of achievement of the aims of the five RE-AIM dimensions was available from the full gamut of sources—such as surveys, interviews, direct observations, notes from the research team and participation in implementation activities—for clubs in the FootyFirst+S regions. In contrast, the evidence for clubs in the FootyFirst+NS region came predominantly from active participation in FootyFirst-related communication (such as emails and telephone calls), being provided with FootyFirst resources (mainly by post), research assistant notes and recall of engagement with clubs and completion of FootyFirst surveys. In the FootyFirst+NS region, only one club participated in a postseason interview, and none participated in a direct observation or provided weekly implementation data. Therefore, the evidence to demonstrate the achievement of the aims of each of the RE-AIM dimensions for clubs in the FootyFirst+NS region was drawn from limited sources and may be less accurate than that for clubs in the FootyFirst+S regions. Finally, clubs in the FootyFirst+NS region did not have the opportunity to participate in FootyFirst-related passive communication activities (Twitter and autotext messaging) or attend FootyFirst-related events because these activities were directly related to the implementation strategies used in the FootyFirst+S regions.

In summary, this study evaluated the implementation of an exercise training programme in terms of all RE-AIM dimensions and the desired implementation activities underpinning them. By comparing these dimensions across two regions exposed to the same IPP content but with different implementation support, it demonstrates that successful implementation of an IPP requires more than just a good programme. The fact that both total and all dimension-specific RE-AIM scores were significantly higher for clubs in the FootyFirst+S regions, than for clubs in the FootyFirst+NS region, supports our unpublished findings that injury reductions were also greater in the former, as assessed in a controlled ecological evaluation. Together, this gives us some confidence in concluding that exercise training programmes will only be effective when the investment in designing and delivering evidence-based and context-specific implementation plans matches the investment currently made in developing the evidence-based programme content.

What is already known on the subject.

• The efficacy of exercise training programmes to prevent lower limb injuries in sport has been shown in highly controlled randomised controlled trials (up to 50% of lower limb injuries).

• The implementation of such programmes, usually quantified as adherence or compliance, has a major effect on injury reductions achieved in real-world settings.

• Implementation factors, other than adherence/compliance associated with exercise training programmes, have rarely been reported in the literature.

What this study adds.

• This is one of the first studies to evaluate the full range of implementation dimensions and activities associated with the implementation of an exercise training programme to prevent sports injuries.

• Exercise training programmes that are supported by context-specific and evidence-informed implementation plans are associated with greater desired implementation activities than are programmes without such implementation support.

• High-fidelity implementation of exercise training programmes in standard football training, and their ongoing maintenance, remains challenging.