A multi‐component evaluation framework of a state‐wide preventive health program: My health for life

Abstract

Issue

Chronic disease is a growing problem affecting approximately half of all Australian adults. In response to growing calls for action on chronic disease, the My health for life program was created, aimed at improving the health of individuals at high risk of developing preventable chronic disease. The preventive health program is multi‐modal, cross‐culturally tailored and contains complex social marketing, community engagement, risk assessment and health promotion components. Therefore, a multi‐component evaluation framework is essential to understand the effectiveness of the My health for life program. This brief report details the evaluation.

Methods

The evaluation design uses non‐randomised, longitudinal analysis using repeated measures, observational, program goal‐based and pretest‐posttest design features to assess the program, its specific modalities and its program adaptations. To ensure timely and credible evaluation, different evaluative implementation frameworks and methods are considered. Quantitative and qualitative methods collect an array of program data at differing levels to assess the processes, outcomes and impacts of My health for life.

Discussion

The implemented evaluation framework has allowed measurement of: (i) process impacts including uptake, retention and attrition, participant satisfaction, fidelity and program stakeholder engagement and (ii) outcomes relating to individual participant level changes in health behaviours.

So what?

This evaluation is an example of an integrated evaluation approach in a large successful preventive health program. Findings from the evaluation will ultimately inform the applicability and transferability of the program and inform policy makers, stakeholders and other health professionals in preventive health practice.

1. INTRODUCTION

The burden of chronic disease is increasing globally, ¹ and there is an urgent need to arrest and, over time, reverse this trend. ² Australian statistics are consistent with global figures, with chronic disease affecting around half of all Australians and account for 37% of all hospitalisations (2015‐16) and 61% of the total burden of disease in 2011. ³ The growing prevalence of chronic disease, as well as the increases in modifiable risk factors, amongst Australians highlight the magnitude of this public health issue. ³ To address these challenges in Australia, national ⁴ and state ⁵ government frameworks were prioritised to address the issues relating to chronic disease. In response, the My health for life program was developed using a co‐design process ⁶ and informed by chronic disease prevention programs, FIN‐D2D (Finland), and Life! (Australia). ⁷ My health for life is a state‐wide government funded initiative, with the aim of identifying individuals at high risk of developing chronic disease, in particular type 2 diabetes, stroke and heart disease and offering a program that supports them in adopting and maintaining positive lifestyle changes to manage their risk factors. ⁷ My health for life is a, multi‐component preventive health program, which involves integrated social marketing, community engagement and health promotion inputs and is further described elsewhere. ⁷

Evaluation is critical for developing an evidence base for complex interventions where there is seldom a clear casual chain. ⁸ This requires assessment of the intervention implementation, outcomes and effectiveness to inform processes to increase impact. ⁹ This report outlines the plans to evaluate the processes, outcomes and impacts of the My health for life program in one Australian setting representing diverse epidemiological and social contexts.

2. METHODS

2.1. My health for life evaluation design

The objectives of the evaluation are to (i) assess participant, service and system impact and outcomes, (ii) provide ongoing insight and feedback on key success indicators and key areas for improvement, (iii) provide evidence on chronic disease prevention programs to inform policy and practice in Queensland and (iv) contribute to the evidence based on implementation of large‐scale chronic disease prevention programs. A pragmatic mixed‐method (quantitative and qualitative) approach underpins the evaluation. Quantitative aspects include a non‐randomised design, longitudinal analysis using repeated measures, adopting observational, program goal‐based and pretest‐posttest design features. A goal‐based evaluation design with predetermined program goals set by the funding body are the standards of the evaluation. ¹⁰ Longitudinal, repeated measures are incorporated for measuring progress towards specific program objectives. ¹⁰ A qualitative approach describes the contexts in which the program is delivered and explores contextual factors that may influence the delivery or impact, and the outcomes.

The evaluation is underpinned by a whole‐system approach and examines the three systems levels of the program: (i) overall program at the macro system level; (ii) site/region at the meso system level and (iii) individual participant at the micro system level. Impact and outcome evaluation will investigate the overall program effectiveness, quality, community outcomes and potential for future scale‐up to advance system changes to processes and resource integration of the program to ensure its sustainability and institutionalisation as “business as usual” using an evidence‐informed approach. The program will be evaluated in terms of: effectiveness (health behaviour changes, benefits and barriers to behaviour change), implementation (dose delivered and received, fidelity, recruitment) (micro system level), program context (meso system level) and potential for program sustainability at the organisation and funder level (macro system level).

Process evaluation follows a continuous improvement model based around the circular process of planning, organising, implementing, evaluating and refining. The process evaluation is continuous and occurs throughout the lifetime of the program and its delivery, enabling feedback to the program team on key success indicators and pinpointing areas where improvements can be made. This feedback loop provides opportunities for innovative solutions to be developed and implemented.

2.2. Evaluation and implementation frameworks

The evaluation approach is guided by evaluation frameworks and conceptual models to strengthen the evaluation questions and analysis. The use of translational research frameworks and implementation models adds rigor to the evaluation of real‐world programs, while considering key factors of both implementation and impact. The My health for life evaluation is predominately guided by the RE‐AIM framework. ¹¹ The framework is used to ensure critical constructs for evaluating public health impact at an individual and service levels are included. Constructs include Reach (number, proportion, and representativeness of individuals who are willing to participate in a given initiative), Effectiveness (impact of the intervention), Adoption (number, proportion, and representativeness of settings and intervention agents who initiate the program), Implementation (consistency and cost of delivery of intervention) and Maintenance (individual impacts post completion of the program and the extent to which a program becomes institutionalised or part of the routine organisational practice. ¹⁰

Two additional models that enable a more comprehensive implementation evaluation have been included. The Conceptual Model of Implementation Research ¹² highlights that implementation efforts require both an empirically tested intervention and an implementation strategy or set of strategies to translate the evidence into practice. ¹² This model is used to assess implementation outcomes such as feasibility, fidelity, penetrations, acceptability, sustainability, uptake and costs. ¹²

The Consolidated Framework for Implementation Research (CFIR) ¹³ has five key elements of implementation: Intervention characteristics; Outer settings; Inner settings; Characteristics of individuals; and Process are used to ultimately evaluate implementation and sustainability of the intervention. ¹³ The use of the CFIR framework will enable implementation evaluation of the specific settings used to implement and deliver the program. Table 1 describes the key evaluative constructs of these frameworks and how each are applied to the evaluation in areas relating to process, impact and outcome evaluation. Instruments utilised for evaluating these areas are discussed in the following section.

TABLE 1.

Evaluation methodological frameworks and their application

Framework	Construct	Details	Evaluative area and query	Data collection source and method
RE‐AIM	Reach Meso level	Reach into the target population	Process: What marketing and communication strategies were used to reach the target group? To what extent did the program reach the target group? To what extent did the program reach the QLD population?	Program website, dashboard and marketing reports (secondary data)
	Effectiveness Micro level	Effectiveness or efficacy (impact) of an intervention on outcomes	Impact: To what extent was the My health for life program efficacious at changing health behaviours in participants?	Individual level surveys (secondary data)
	Adoption Meso level	Adoption of the intervention by target settings, institutions and staff	Process: How many and which organisations supported the program implementation?	Program dashboard (secondary data) Community engagement reports (secondary data)
	Implementation Meso level	Implementation consistency and cost of delivery of intervention	Process: How was the program developed and implemented at both project and program levels? What factors facilitated program implementation and what factors were barriers to implementation at a project level?	Program reports (secondary data) Evaluation team observations and program facilitator interviews (primary data)
	Maintenance Micro level Macro level	Maintenance of intervention effects in individuals and settings over time	Outcome: To what extent were primary behaviour change outcomes maintained by participants post the My health for life program? To what extent is the My health for life program part of organisational practices?	Individual level surveys (secondary data) Program reports (secondary data) Program stakeholder and facilitator interviews (primary data)
Conceptual model of implementation research – implementation outcomes	Feasibility Meso level	The extent to which the program or practice can be successfully used or carried out within a setting	Process: How feasible was the program perceived to be? What barriers regarding feasibility were encountered? What strategies were utilised to address/prevent any barriers regarding feasibility? How successful were implementation strategies in addressing barriers and facilitating implementation feasibility?	Program stakeholder and facilitator interviews (primary data)
	Fidelity Micro level	The degree to which a program or practice was delivered as intended	Process: What level of fidelity was observed in implementation? What barriers to fidelity were encountered? What strategies were utilised to address/prevent any barriers to fidelity? How successful were implementation strategies in addressing barriers and facilitating fidelity?	Evaluation team observations of program delivery Program implementation team and facilitator interviews (primary data)
	Penetration Meso level	The integration of an intervention within a service setting	Process: What level of penetration was observed? What barriers regarding penetration were encountered? What strategies were utilised to address/prevent any barriers regarding penetration? How successful were implementation strategies in addressing barriers and facilitating penetration?	Program dashboard (secondary data) Evaluation team observations and program facilitator and stakeholder interviews (primary data)
	Acceptability Meso level	The perception among stakeholders that a program or practice is agreeable, palatable or satisfactory	Process: How did stakeholders perceive the program? What barriers regarding acceptability were encountered? What strategies were utilised to address/prevent any barriers to acceptability? How successful were implementation strategies in addressing barriers and facilitating acceptability?	Program facilitator and stakeholder interviews (primary data)
	Sustainability Macro level	The extent to which a newly implemented intervention is maintained or institutionalised into “business as usual” for a service setting	Process: What level of sustainability was observed? What barriers regarding sustainability were encountered? What strategies were utilised to address/prevent any barriers to sustainability? How successful were implementation strategies in addressing barriers and facilitating sustainability?	Program facilitator and stakeholder interviews (primary data) Program Operations Management Committee Meeting minutes (secondary data) Steering committee meeting minutes (secondary data)
	Uptake Meso level	The intentional initial decision or action to take on or try an intervention/program	Process: What percentage of uptake was observed? What barriers regarding uptake were encountered? What strategies were utilised to address/prevent any barriers to uptake? How successful were implementation strategies in addressing barriers and facilitating uptake?	Program dashboard and reports (secondary data) Evaluation team observations and program facilitator and stakeholder interviews (primary data)
	Costs Macro level	The true cost of an implementation effort, which includes the costs of the intervention, the implementation strategy used, and the location of service delivery	Process: What costs for implementation were observed? What barriers regarding costs were encountered? What strategies were utilised to address/prevent any barriers to true costs of implementation? How successful were implementation strategies in addressing barriers and facilitating true costs of implementation?	Program dashboard and reports (secondary data) Program governance reports (secondary data) Evaluation team observations and program facilitator and stakeholder interviews (primary data)
Consolidated Framework for Implementation Research	Intervention characteristics Meso level	Complexity, cost, perceived quality	Process: What aspects of the intervention characteristics acted as facilitators and barriers to program implementation across different settings?	Evaluation team observations and program facilitator and stakeholder interviews (primary data)
	Outer settings Micro level	Participant needs and resources, cosmopolitanism, peer pressure, external policies and incentives	Process: What aspects of the outer settings intervention characteristics acted as facilitators and barriers to program implementation across different settings?	Evaluation team observations and program facilitator and stakeholder interviews (primary data) Environmental audit (food and physical activity environment) (primary data)
	Inner setting Macro level	Policies and incentives, population unmet needs	Process: What aspects of the inner setting intervention characteristics acted as facilitators and barriers to program implementation across different settings?	Evaluation team observations and program facilitator and stakeholder interviews (primary data)
	Characteristics of individuals Micro level	Leadership, readiness, learning climate, self‐efficacy, knowledge and beliefs, stage of change	Process: What aspects of characteristics of individuals acted as facilitators and barriers to program implementation across different settings?	Evaluation team observations and program facilitator and stakeholder interviews (primary data)
	Process Meso level	Engagement, planning, reflecting and evaluating	Process: What aspects of the process acted as facilitators and barriers to program implementation across different settings?	Evaluation team observations and program facilitator and stakeholder interviews (primary data)

2.3. Evaluation instruments and data

A risk assessment survey developed by the My health for life program is implemented prior to baseline to determine risk of chronic disease and establish program eligibility. The risk assessment survey is completed online via the My health for life website or in person by the risk assessment team. The survey includes the collection of basic socio‐demographic information including age, gender, location, Aboriginal or Torres Strait Islander status and ethnicity. The survey also collects data to establish risk of chronic disease, diabetes ¹⁴ and cardiovascular disease. ¹⁵ In addition to these items, a further survey is conducted with each participant at baseline (T1), sessions 5 (T2) and 6 (T3) ‐ and 12 months (T4) post program enabling longitudinal analysis (see Table 2). The program survey includes instruments which measure changes in behaviour, self‐efficacy and health quality of life (see Table 2). To ensure a rigorous evaluation, the tools used to assess individual level outcomes of My health for life are developed using previously validated instruments (see Table 2). The first 3 years of the program will collect data from a minimum of 10 000 participants at three time points (T1, T2 and T3).

TABLE 2.

Outcome measures, instruments and time points of the evaluation

Time point	Risk assessment (T0) (0 wk)	Session 1 (T1) (Baseline – 0‐6 wk)	Session 3 10 wk)	Session 5 (T2) (14 wk)	Session 6 (T3) (26 wk)	Maintenance (T4) (52 wk)
Measures
My health for life Risk Assessment (total risk score)	X
Risk of diabetesa	X					X
Risk of cardiovascular diseaseb	X
Socio‐demographics incl. age, gender, SES, CALD and Aboriginal and Torres Strait Islander status	X
Anthropometry incl. height, weight, waist circumferencec		X	X	X	X	X
Dietary habits incl. daily fruit/vegetable, sugar sweetened beverage and takeaway intaked		X		X	X	X
Alcohol and tobacco usee		X		X	X	X
Physical activityf		X		X	X	X
HRQoLg		X		X	X	X
SERVQUALh				X
Knowledge of health behaviours, intention and confidence, and support		X		X	X

Primary data collected by program facilitators (coaches); Measured using: ^aAdapted AUSDRISK items, ^{14 b}Absoloute CVD Risk, ^{15 c}WHO Steps Surveillance Manual protocol and standards, ^{16 d}Australian Bureau of Statistics National Health Survey items, ^{17 e}National Drug Survey items, ^{18 f}Active Australia, ^{19 g}CDC HRQoL measure ²⁰ and ^hSERVQUAL scale. ²¹

Abbreviations: SES, socio‐economic status; CALD, culturally and linguistically diverse; HRQoL, health‐related quality of life.

Survey instruments, administered by program facilitators, including in person using pen and paper by face‐to‐face group coaches, and via telephone by telephone health coaches. Participant pen and paper surveys are entered into the survey data tool (specifically developed for the program) by the telephone health coaches. For the purpose of the evaluation, all program participant data are de‐identified when provided as secondary data to the evaluation team by the program team.

While the program evaluation and data collection will be ongoing, longer term impacts will be assessed via the long‐term effects (6 months or more post last intervention contact) of the program on outcomes. Specifically, a sub‐sample of participants (n = 520) will undertake a follow‐up “Maintenance Survey” on diabetes risk, modifiable lifestyle behaviours and quality of life 6 months post program completion (T4). Data will be used to determine long‐term intervention effects on healthy behaviours and risk reduction.

The process evaluation determines whether program activities have been implemented as intended and is assessed using a combination of primary and secondary data. As shown above in Table 1, qualitative and quantitative data are collected across a variety of program areas and levels from participants, facilitators/telephone health coaches, service providers and stakeholders on program uptake, retention and attrition, satisfaction, community engagement and social marketing, networking and partnerships and program governance.

At the participant level, the evaluation team only have access to secondary, non‐identifiable data, provided by the My health for life program in accordance with program data transfer regulations. Due to the ongoing nature of the program, data collection and analysis is continual. Outcome and process evaluation reporting occurs annually. Ongoing outcome and process data analysis and reporting allows continual feedback to the program to enable continual improvements to program delivery. Overall, a broad mixed‐methods approach is undertaken for data analyses. Data are cleaned and entered into one master data set using the current version of SPSS software. Descriptive information is tabulated across evaluation domains and frequency counts and percentages are reported across the RE‐AIM indicators. Inferential statistics and repeated measures (within and between groups) are used when determining effectiveness.

3. DISCUSSION

This report provides a worked example of how to embed an evaluation in the design and implementation of a complex intervention or program, which are scarce throughout current literature. ²² This report responds to calls for guidance on how to proceed in evaluation studies and the decision‐making surrounding this. ²³ The main strength of a process evaluation design, as adopted in this study, is the efficient use of continuous data collection and analysis enabling ongoing feedback to the program team and ability for continuous quality improvement during the program's lifecycle. Further strengths of the design is the efficient use of extant data collection tools and processes that are embedded in program delivery and the use of several methods of data collection allow for enough flexibility to capture diverse perspectives related to the program implementation. Moreover, the qualitative approach provides access to nuanced information about processes, priorities and constraints of the program context, which are generally not elicited in a quantitative approach. Although the program evaluation has a number of strengths, it is not without limitations. Firstly, there is no comparison group, meaning that only contribution, rather than attribution can be claimed when discussing the effectiveness of the program. Second, the evaluation relies on self‐report data from participants rather than objective measures of behaviour. This program evaluation examines one context, which may limit its generalisablity into other contexts and settings.

Additionally, the evaluation will create a baseline that the program team and service providers can use to monitor progress in the future. The evaluation will contribute to building long term administrative data that can help to explain changes in outputs and outcomes over time including behaviour change. Examining all three levels of the system allows the evaluators to identify the mechanisms in the system which can be leveraged for greater engagement and program success.

The evaluation will provide vital insights into what contributes to the program outcomes and impacts, while generating new knowledge on the implementation and scale‐up of the program. On an individual level, this program will benefit by helping participants to understand chronic disease risk and available options for actions. On a community level, outcomes of this research have the potential to reduce individual risk factors for chronic disease. Found effective, the My health for life program can advance knowledge and understanding of the effect of multiple health behaviour change interventions in modifying lifestyle risk factors in adults. This preventive health program can potentially be adapted and tailored for other age groups or different at‐risk populations and can be applied in a variety of community or corporate settings to address the growing burden of chronic disease. This research will inform policy, practice and investment decisions regarding how to optimally meet the needs of Queenslanders at risk of chronic disease.

DISCLOSURES

The authors declare that they have no conflict of interests.

Funding information

The My health for life program is a funded initiative by the Queensland Government, through Health and Wellbeing Queensland. The evaluation of the program was awarded via a tender process where funding was provided to Griffith University by the My health for life program lead agency Diabetes Queensland. The My health for life program was developed and is delivered by the Healthier Queensland Alliance.

Parkinson J, McDonald N, Seib C, Moriarty S, Anderson D. A multi‐component evaluation framework of a state‐wide preventive health program: My health for life. Health Promot J Austral. 2022;33(S1):271–7. 10.1002/hpja.591