How is the sustainability of chronic disease health programmes empirically measured in hospital and related healthcare services?—a scoping review

Abstract

Objectives

Programmes to address chronic disease are a focus of governments worldwide. Despite growth in ‘implementation science’, there is a paucity of knowledge regarding the best means to measure sustainability. The aim of this review was to summarise current practice for measuring sustainability outcomes of chronic disease health programmes, providing guidance for programme planners and future directions for the academic field.

Settings

A scoping review of the literature spanning 1985–2015 was conducted using MEDLINE, CINAHL, PsychINFO and The Cochrane Library limited to English language and adults. Main search terms included chronic disease, acute care, sustainability, institutionalisation and health planning. A descriptive synthesis was required. Settings included primary care, hospitals, mental health centres and community health.

Participants

Programmes included preventing or managing chronic conditions including diabetes, heart disease, depression, respiratory disease, cancer, obesity, dental hygiene and multiple chronic diseases.

Primary and secondary outcome measures

Outcome measures included clarifying a sustainability definition, types of methodologies used, timelines for assessment, criteria levels to determine outcomes and how methodology varies between intervention types.

Results

Among 153 abstracts retrieved, 87 were retained for full article review and 42 included in the qualitative synthesis. Five definitions for sustainability outcome were identified with ‘maintenance of programme activities’ most frequent. Achieving sustainability was dependent on inter-relationships between various organisational and social contexts supporting a broad scale approach to evaluation. An increasing trend in use of mixed methods designs over multiple time points to determine sustainability outcomes was found.

Conclusions

Despite the importance and investment in chronic disease programmes, few studies are undertaken to measure sustainability. Methods to evaluate sustainability are diverse with some emerging patterns in measurement found. Use of mixed methods approaches over multiple time points may serve to better guide measurement of sustainability. Consensus on aspects of standardised measurement would promote the future possibility of meta-analytic syntheses.

Strengths and limitations of this study.

• The systematic identification and abstraction of eligible publications assessed by more than one author and the use of manual reference searching and forward citation tracking as well as the use of an appropriate quality assessment tool for classifying the eligible publications.

• The scoping method of this review has allowed us to map the heterogeneous body of literature in this field.

• The scoping method has also allowed us to include a greater range of study designs and methodologies, currently used in the chronic disease field, to allow for rich and in-depth discussion.

• The heterogeneous nature of the literature precludes a meta-analytic review making the results less generalisable.

• The systematic method applied in this scoping review means we may have missed some relevant reports if these have been published in the grey literature.

Background

The continuation of an effective health programme, beyond the initial implementation phase, is paramount to maintaining better outcomes for patients with chronic disease. There exists a paucity of knowledge regarding the best means to ensure sustainability of chronic disease health programmes.¹ The over-riding challenge is to sustain health programmes after initial programme support has been removed or has expired. However, research providing evidence for effective sustainability strategies for health programmes is underdeveloped,² and clear recommendations to promote sustainability are limited.³

Measurement of sustainability in the chronic disease field presents a challenge due to the scope of assessments utilised, interventions provided and the heterogeneity between and within illnesses.⁴ In addition, comorbidities are frequent for people with chronic disease and the subsequent interactions between diseases and treatments provide further challenges for measurement.⁵ Thus, the interventions developed and implemented in the chronic disease field are complex.

As such, in recent times, it has been suggested that a multifaceted approach to measuring sustainability is required to determine outcomes.^{6 7} The outcomes are determined by the aim of individual researchers and may vary from sustained health outcomes to continuation of programme activities. For example, Rowley et al⁸ were interested in evaluating the effectiveness of a new programme to prevent obesity, diabetes and heart disease in a remote indigenous community. Thus, the outcomes of interest to determine sustainability of an effective health programme were specific health measures (body mass index and impaired glucose tolerance).⁸ In contrast, Brand et al⁹ were interested in evaluating adherence to clinical practice guidelines for chronic obstructive pulmonary disease. The guidelines were evidence based and thus had previously proven to improve health outcomes. As such, the authors aimed to evaluate effectiveness of adherence to the guidelines since adherence had previously been proven to improve health outcomes.⁹ A different approach has also been to determine the level of community ownership of the programme. Koskan et al¹⁰ found that participants in the evaluation of an obesity prevention programme viewed sustainability as increased community ownership of the programme with less support from outside organisations. Indeed, these examples illustrate that sustainability outcomes often move beyond longitudinal programme outcomes and are reframed to include spread of the programme and community ownership. In this paper, sustainability of chronic disease health programmes focuses on programme processes as opposed to health outcomes.

Clearly, sustainability is a multidimensional concept encompassing a diversity of forms along a continuation process with indicators of success that fall into distinct categories.⁷ These include: (1) maintenance of the health benefits achieved through a programme; (2) maintenance of the core activities central to the programme; and (3) continued capacity of the community to build and deliver health programmes (the extent to which community members are educated and can access programme resources).⁷ Scheirer and Dearing⁶ have added to the list including: (1) programme diffusion (when the underlying concepts or innovations spread to new locations);⁶ (2) maintaining new organisational practices, procedures and policies;⁶ and (3) maintaining attention to the issues addressed by the programme.¹¹ Clearly, many variables need to be considered and clearly defined in studies investigating sustainability, and planning for, or measuring, sustainability must be included in the programme development stages.

It is acknowledged that it is not always necessary to sustain all original programme activities.^{1 7 12} Therefore, Greenhalgh et al¹³ contend that the more complex approach of drawing narratives from multiple interacting processes can offer unique and in-depth insights into measuring sustainability outcomes as a move away from simply measuring relationships between a set of dependent and independent variables.¹⁴ As such, in sustainability research, differing approaches based on either being intervention focused or making an assessment of sustainability as a part of complex systems have implications for the way research in this field is conducted and is influenced by the health discipline.¹⁵ Several authors have proposed frameworks for programme sustainability assessment to enable some standardisation and guidance within different disciplines.^16–19

A recent review of empirical studies used to investigate health programme sustainability revealed that 40–60% of programmes continued in some form; however, the studies’ designs were weak.²⁰ For example, key variables and definitions were not clearly defined.²⁰ Clearly, more evidence and clearer recommendations are needed for current sustainability research. A current understanding of the research base is needed to provide evidence-based recommendations and facilitate well-informed decision-making on programme sustainability methods in the health field. Without clear evidence, successfully sustaining health programmes may continue to be a challenge.¹⁴

Identifying the research question

Chronic diseases, in particular those related to cardiovascular disease and diabetes, are a growing concern to governments in developed and developing countries since they are a major source of health loss in society.²¹ Until now, the information on how to best sustain and measure the sustainability outcomes of chronic disease prevention and management programmes is rare. The primary aim of this paper was to conduct a review of the current literature describing the sustainability of chronic disease programmes. The second aim was to summarise the empirical methods used to measure sustainability of chronic disease health programmes. Health programmes in the field of chronic disease management within hospital, primary care and community health settings were the primary focus. For this review, outcomes measured for determining if a programme has been sustained are referred to as the ‘sustainability outcome’. The research questions for this scoping review included:

1. How are sustainability outcomes of health programmes defined for measurement in the field of chronic disease?

2. What methodologies are used to measure the defined sustainability outcomes including the types of study designs?

3. What is the typical timeline for assessing the sustainability of a programme? This is designed to enable a better understanding of when the sustainability phase generally begins.

4. What criteria levels are set to determine if sustainability has been achieved based on the sustainability outcome and are these predefined?

5. How does the methodology vary between intervention types? This question was designed to explore the relationship between the nature of the programme itself and type of study methodology used.

Methods

Ethics and dissemination

This is a scoping review of the literature, thus formal ethics was not required as patient data or meta-analysis was not conducted.

We conducted a scoping literature review. The aim of a scoping review is to map existing literature in a given field when the topic is of a heterogeneous nature and has not been extensively reviewed.²² Our decision to adopt this approach for our review was based on allowing us to provide an overview of the diverse body of sustainability literature, including the greater range of study designs and methodologies, currently being used in the chronic disease field.

In order to clarify the focus of this scoping review, the outcomes of interest need to be specified.²³ Therefore, in this review, a sustainability outcome was defined as the long-term survival of programme activities: health benefits or continued capacity of organisations to deliver and adapt programme activities.⁷

Identifying relevant studies

An electronic database search was conducted for studies, using MEDLINE, CINAHL, PsychINFO and The Cochrane Library, searching dates between 1 January 1985 and 1 June 2015 using a combination of MeSH and free text by Boolean operators (table 1). The combination of MeSH and free text operators included one of the key MeSH headings together with each of the key words (eg, Programme evaluation with chronic disease, acute care, continuation, institutionalisation, maintenance and programme development) for each of the databases. The reference lists of included papers were also manually searched to identify any potentially relevant studies not found in the electronic search. Forward citation tracking²⁴ was also used to identify additional papers.

Table 1.

Electronic database key word terms and search engines

MeSH terms	Key words	Search engines
Program sustainability Program evaluation Diffusion of innovation Organisational innovation	Chronic disease Acute care Continuation Institutionalisation Maintenance Program development	MEDLINE—1985–1 February 2013
Program sustainability Program evaluation Diffusion of innovation Organisational innovation	Chronic disease Acute care Continuation Institutionalisation Maintenance Program development	Scopus—1985–1 July 2013
Program sustainability Program evaluation Diffusion of innovation Organisational innovation	Chronic disease Acute care Continuation Institutionalisation Maintenance Program development	Cumulative Index to Nursing and Allied Health—1985–1 July 2013
Program sustainability Program evaluation Diffusion of innovation Organisational innovation	Chronic disease Acute care Continuation Institutionalisation Maintenance Program development	PsychINFO—1985–1 July 2013
	Maintenance	The Cochrane Library
	Health planning
	Program development

Study selection

Each potentially eligible publication was independently assessed for inclusion and quality. LF performed the initial review of publications with the following criteria set for inclusion of studies: (1) studies with a focus on health programme sustainability, including the various measures used to determine the sustainability outcome, as well as the reported factors, facilitators and barriers to sustainability, (2) no pre-selected criteria for assessment time periods were used, (3) settings included primary care, hospitals, mental health centres and community health, (4) studies that were randomised controlled, controlled or descriptive and had retrospective or prospectively collected data were included, and finally, (5) multicase and single-case studies without original data were included.

The following criteria were set for exclusion of the studies: (1) studies limited to implementation processes only, (2) programmes that were not related to chronic illness, (3) studies that did not specify clear measures for assessing programme sustainability, (4) studies were not included if the stated measures did not match the main study findings, and finally, (5) opinion pieces and conceptual studies were not included. The co-authors assessed the filtered abstracts or full articles following the above criteria.

After initial review, methodological quality of the studies was assessed by evaluating the design, methods, baseline data, interventions, assignment methods, statistical methods and explanation of results using the Transparent Reporting of Evaluations with Non-randomised Designs (TREND) guidelines.²⁵ Additional items from the Consolidated Standards of Reporting Trials (CONSORT) guidelines were included for the one RCT study included.²⁶ For example, studies were excluded if there was insufficient information on study sample, statement of objectives, methods of recruitment, unit of delivery, methods used to collect data, statement of results for stated primary and secondary outcomes, and clear discussion.

Charting the available evidence

An Excel-based (Microsoft Office version14.2.1) extraction checklist was designed and used to check data using the following descriptors: information on target population, eligibility criteria and methods of participant recruitment, details of intervention delivery, statement of specific objectives and hypotheses, methods used to collect data, sample size, assignment methods, unit of analysis, relevant results, theoretical relevance and generalisability. Since the broad nature of the included studies would preclude meta-analysis, a descriptive synthesis is provided. One study included in the analysis was a randomised controlled trial (RCT). This study was evaluated using the CONSORT guidelines.²⁷ Additional items were added to the excel extraction checklist for these studies including protocol and registration, eligibility criteria, risk of bias, study selection and summary measures.²⁶

Defined sustainability outcomes were extracted from the studies and categorised according to six intervention-based indicators from the literature: (1) maintenance of the health benefits achieved through a programme; (2) maintenance of the core activities central to the programme; (3) continued capacity of the community to build and deliver health programmes,⁷ (4) programme diffusion;⁶ (5) maintaining new organisational practices, procedures and policies;⁶ and (6) maintaining attention to the issues addressed by the programme.¹¹ The indicators were specifically selected since they cover a diverse range of possible outcomes that fit with the broad definition of sustainability outcomes chosen for this review. The indicators were adapted from authors Shediac-Rizkallah and Bone⁷ and Scheirer and Dearing.⁶ These authors have shaped different understandings of the concepts of sustainability and have emphasised that a multifaceted approach to measuring sustainability is now required to determine outcomes. Therefore, these indicators were used to provide the framework for describing the sustainability outcomes reported from the included studies. Each study included in this paper was also categorised for intervention type according to Scheirer's¹⁹ definitions of intervention type. Six intervention type categories were used: (1) interventions implemented by individual providers, (2) interventions requiring coordination by multiple staff, (3) new policies, procedures and technologies, (4) capacity or infrastructure building, (5) collaborative partnerships and (6) broad scale system change.¹⁹

Collating, summarising and reporting the results

Database searching yielded 130 studies (figure 1) for screening and 32 additional articles were identified through manual searching and forward citation tracking. After duplicates were removed, 153 articles were screened. Sixty-six were eliminated based on the eligibility criteria. A further 45 papers were excluded after full-text review because they did not conform to the criteria for chronic disease programme settings (eg, specialist infectious disease centres); did not meet methodological criteria because they included non-health-based programmes (were based on the education system and settings in public safety—housing, crime, etc), or did not state clear sustainability outcomes or related factors or determinants of sustainability. This left 42 articles meeting the review criteria.

Figure 1.

Literature search flow diagram.

The 42 eligible studies were focused on programmes for preventing or managing chronic conditions including diabetes (n=5), heart disease (n=4), depression (n=8), respiratory disease (n=3), cancer (n=3), obesity (n=5), dental hygiene (n=1), aged care (n=4) and multiple chronic diseases across primary care and/or remote settings (n=9). The qualitative synthesis of the included studies is presented according to each of the specific research questions as outlined below.

Defined sustainability outcomes

The defined outcomes are listed in table 2. The maintenance of programme activities as a defined outcome for measurement was used in the majority of studies (n=37). Specific programme components were identified as the measure for sustainability outcome in these studies. Specific components included: continued adherence to evidence-based recommendations,^{9 13 28–37} use of programme-specific tools,³⁸ patient referrals,^{31 39} regular measures of clinical indicators,^{40 41} and direct outcomes from involvement in programme activities such as the percentage of people attempting dietary change from the Rowley et al⁸ study.

Table 2.

Defined outcomes, intervention type, unit of analysis and methodology for all studies included in the final analysis

	Author	Health programme	Unit of analysis	Defined outcomes	Intervention type	Method and assessment period
1	Aitaoto28	Diabetes Today Initiative Centre for Disease Control and prevention funded empowerment programmes for diabetes management	Multiple community healthcare settings	Continuation of programme activities	Capacity building Broad scale system change	Case series—uncontrolled longitudinal study Qualitative case study Follow-up interviews with coalition representatives in each community Assessment Period 4 years postfunding
2	Ament43	Short stay after breast cancer surgery	Multisite hospital setting	Compliance with key recommendations of the programme Proportion of patients treated in short stay at follow-up	Intervention requiring coordination by multiple staff New policies, procedures and technology	Case–control study Before–after design Retrospective chart audit Assessment Period 5 years postimplementation
3	Barnett44	Falls prevention intervention: the Stay on Your Feet programme (SOYF)	Multiple community healthcare settings	For health professionals—Recall of SOYF; influence on practice; Use of SOYF resources; For elderly residents—Recall of SOYF strategies; behavioural changes attributed to SOYF	Intervention requiring coordination by multiple staff New policies, procedures and technologies Broad scale system change	Uncontrolled longitudinal case study Surveys with health professionals and focus groups with elderly community-dwelling residents Assessment Period 5 years postsupport
4	Bailie29	Improving delivery of preventive medical services through the implementation of locally developed best practice guidelines for disadvantaged populations in remote settings	Multiple community healthcare settings	Improvement in percentages of delivered services between baseline and follow-up audits	Broad scale system change	Interrupted time series with random sampling Follow-up with repeated audits over a 3-year-period Assessment Period 3 years from baseline
5	Bereznicki45	Community pharmacy intervention for asthma medication	Multiple community pharmacies	Average usage of medication	New policies, procedures and technology	Case–control study Before–after design Retrospective audit Assessment Period 5 years postimplementation
6	Blasinsky57	Project IMPACT: 7 site RCT on depression treatment in older adults	Multisite primary care setting	Continuation of all or part of the programme after funding ceased	Intervention requiring coordination by multiple staff Broad scale system change	Interrupted time series Qualitative study; evidence of continuation programme after funding ended; (1) review of grant proposals (2) site visits (3) semistructured telephone interviews with key players Data collected at 3 points; (1) baseline (2) One-year post to explore the implementation experience (3) One-year postcessation of clinical services to 5 explore sustainability Assessment Period 1 year postsupport
7	Boehm46	Slim without diet (SWD) Aim to change individuals' eating and diet habits	Multiple community healthcare settings	Maintain weight loss	Intervention requiring coordination by multiple staff New policies, procedures and technology	Prospective cohort study Quantitative time-series Questionnaire Assessment period 12 months postsupport
8	Bond47	National Implementing Evidence-Based Practices Project in mental healthcare settings	Multiple community healthcare settings	Fidelity to the EBP model Continuation of programme activities	Intervention requiring coordination by multiple staff Capacity building	Case–control study Before-after design Interview at 2 years and 6 years postimplementation Quantitative and Qualitative Assessment Period 6 years postimplementation
9	Bracht58	Improvement of cardiovascular health in Minnesota (USA) through a heart health programme involving establishment of local boards, community organisation, training and volunteers—The Minnesota Heart Health Program (MHHP)	Multiple community healthcare settings	Continuation of programmes postfunding	Broad scale system change Collaborative partnerships	Interrupted longitudinal time-series study Quantitative: Measurement of long-term programme maintenance occurred through annual surveys investigating continued incorporation of programme activities in community groups Assessment Period 6 years postimplementation
10	Brand9	Adherence to Chronic Obstructive Pulmonary Disease clinical practice guidelines	Single healthcare organisation	Adherence to COPD guidelines	Intervention requiring coordination by multiple staff	Case study Mixed methods evaluation Survey Interview Focus groups Assessment Period 2 years post baseline
11	Campbell48	Ottawa Model of Smoking Cessation Hospital-based inpatient smoking cessation programme	Multisite hospital-based setting	Improved performance of OMSC activities from baseline	New policies, procedures and technology Intervention requiring coordination by multiple staff	Multisite case study Qualitative Interviews Assessment Period 7 years from baseline
12	Carpenter56	Community clinical oncology programme	45 Community clinical oncology programmes	Treatment trial accrual Cancer prevention and control accrual Total trial accrual	Broad scale system change	Longitudinal quasi-experimental Data collected from progress reports and management systems from 2000–2007 Assessment Period 8 years from implementation
13	Chin42	Health Disparities Collaborative (HDC) to improve Diabetes care	Individual patient level Multiple community healthcare settings	Continued patient improvements in diabetes care	Capacity building Broad scale system change	RCT with embedded prospective longitudinal study Retrospective chart review of randomly selected patients Assessment period 4 years from implementation
14	Goodson38	Put Prevention into Practice Institutionalisation of tools for preventive services by primary care providers in USA	Multiple community healthcare settings	Use of tools	New policies, procedures and technologies	Interrupted time series study Interview and audit across three time points from implementation to follow-up Assessment Period 6 years postimplementation
15	Greenhalgh13	Three preventive services—Stroke, Kidney and Sexual health	Individual patient level Multiple healthcare organisations	Health benefits; programme activities; practices and procedures; capacity to undertake quality improvement; interorganisational partnerships	Broad scale system change Intervention requiring coordination by multiple staff	Case study design Mixed methods Quantitative document review Qualitative interviews Assessment Period 7 years postimplementation
16	Clinton59	Multi-intervention physical activity and nutritional health health promotion programs Two programs	Multiple community healthcare settings	Meetings KPIs Adaptation Degree of implementation Organisational development Progress Collaboration Sustainability Evaluation readiness	Broad scale system change Intervention requiring coordination by multiple staff	Interrupted time series study Interview, survey and case studies collected at various time points Assessment Period 4 years post implementation
17	Cramm60	Chronic care model—22 disease management programme targeting cardiovascular disease, chronic obstructive pulmonary disease, diabetes, heart failure, stroke, depression, psychotic disorders and comorbidity	Multiple community healthcare settings	Quality of chronic care delivery Routinisation of practice	New policies, procedures and technologies Intervention requiring coordination by multiple staff	Longitudinal prospective interrupted time-series study Quantitative administration of survey at three time points Assessment Period 2 years postimplementation
18	Gundim61	Telemedicine and telehealth centre	Single University site	Indicative factors of sustainability: Institutional Functional Economic-financial Renewal Academic-scientific Partnerships Social welfare	Broad scale system change	Retrospective longitudinal study Mixed methods—interviews, documents and reports. Data collected at 6 time points over 10 years Assessment period 10 years
19	Gruen62	Improvement of access to specialist services in remote Australian Aboriginal communities by specialist outreach visits	Multisite community health and hospital settings	Number of consultations	Broad scale system change Intervention requiring coordination by multiple staff New policies, procedures and technologies	Case study Process evaluation of outreach service Document review and semistructured interviews Assessment Period 3 years postimplementation
20	Hearld63	Aligning Forces for Quality Improving quality of healthcare for chronically ill people through multistakeholder healthcare alliances.	Multisite community health and hospital settings and government agencies	Organisational change	Broad scale system change	Interrupted longitudinal time-series study Quantitative: Measurement of long-term programme maintenance occurred through surveys at three time points investigating factors that promote change
21	Jansen30	Heartbeat 2 Health counselling programme for high-risk Cardiovascular patients	Individual patients Single healthcare organisation	Sustainability of health benefits; programme activities; capacity; commitment Role of external change agent	Capacity building Intervention requiring coordination among multiple staff	Case Study Data derived from registrations, reports and focus group interviews Assessment Period 3 years postimplementation
22	Koskan10	Promotoras de salud Community health-promotion projects for obesity-related lifestyle behaviours	Programme planners from multiple community sites	Community empowerment	Broad scale system change Intervention requiring coordination by multiple staff	Case Study Qualitative in-depth interviews Assessment Period Not stated
23	Lassen64	Increase consumption of fruit and veg intake at worksite canteens	Multiple community sites	Fruit and veg consumption	Intervention requiring coordination by multiple staff New policies, procedures and technologies	Interrupted time series study Mixed-methods Assessment Period 1 year postbaseline—3 time points
24	Lee31	Primary Care Treatment of Depression (RESPECT-D) Treatment of depression by care managers in primary care supervised by specialist—RCT	Individual providers across multiple healthcare organisations	Continued patient referrals; continued programme activities	Broad scale system change Intervention requiring coordination by multiple staff	Multisite case study Mixed methods Descriptive evaluation conducted at 2 time points. Assessment Period 1 and 3 years post support
25	Manning49	Community health networks to reduce cancer disparities in African-American people	Multiple community sites	Continued partner relationships	Collaborative partnerships	Longitudinal interrupted time series study Mixed methods study Interviews, surveys and reports Assessment Period 5 years from implementation
26	McDermott40	Improved Diabetes care in remote Australian Aboriginal communities by health worker-run registers, recall and reminder systems, care plans, and specialist outreach	Multiple community healthcare settings Individual patient level	Number of people on registers; Care processes; appropriate clinical interventions; patient outcome measures	Broad scale system change	Multisite case study Quantitative Three-year follow-up clinical audit Assessment Period 3 years post support
27	Meredith32	Depression in primary care	Multiple healthcare organisations	Improved delivery of services; Spread of collaborative efforts	Intervention requiring coordination by multiple staff	Interrupted time series study Process evaluation data and 18-month telephone interviews Semistructured telephone interviews Assessment Period Quantitative—from implementation Qualitative—18 months post support
28	Nease33	Improving Depression care collaborative; Implemented across 16 primary care practices. Both depression care and change management processes were taught to staff at implementation	Multiple healthcare organisations	Continued use of interventions Maintenance of change management processes	Capacity building Intervention requiring coordination among multiple staff	Case series study Qualitative follow-up telephone interview Assessment Period 2 years post support
29	O'Loughlin34	Community-based Cardio-vascular disease risk-factor reduction programs in Canada	Multiple community health settings	Permanence of the programme (self-report perception on Likert scale of permanence)	Intervention requiring coordination among multiple staff	Case Study Quantitative Telephone survey of key informants Assessment Period Up to 10 years postimplementation
30	Palinkas50	Multi-faceted Depression and Diabetes (MDDP) Programme	Individual provider level across multiple healthcare organisations Individual patient level	Improved patient outcomes; Improved access to services; Improved consumer satisfaction	Collaborative partnerships	Qualitative study took place in the context of an RCT Semistructured interviews and focus groups Examined patient and provider perceptions of implementation and sustainability of the programme Assessment Period 2 years postimplementation
31	Pluye35	Quebec Heart Health Demonstration Project	Multiple community health settings	Continuation of programme activities Specific routinisation events	Collaborative partnerships	Retrospective multiple-case study 5 cases—5 community health centres Documents an interviews Assessment Period 10 years postimplementation
32	Ramsay39	Educational reminder messages for knee and lumbar spine radiographs	Individual providers across multiple healthcare organisations	Number of referrals	Intervention implemented by individual providers New policies, procedures and technologies	Interrupted time series Quantitative Monthly total number of referrals over a 1 year period Assessment Period 1 year postbaseline
33	Reinschmidt51	Border Health Family Diabetes Programme Community capacity building intervention	Multiple community health settings	Programme adaptation to other communities	Collaborative partnerships Capacity building Broad scale system change	Case study Qualitative study Document review Face-to-face interviews with individuals from the adapted diabetes intervention programmes Assessment Period 4 years postimplementation
34	Rowley8	Chronic-disease prevention—Obesity, Diabetes and Cardiovascular disease—in a remote Australian Aboriginal community by education, regular physical activity, and cooking classes—a lifestyle improvement programme	Individual patient level	Health measures (body mass index and impaired glucose tolerance); percentage of people attempting dietary change; and physical activity	Intervention requiring coordination among multiple staff	Interrupted time series study Quantitative Assessment Period 2 years postimplementation
35	Scheirer65	School-based fluoride mouth rinse program (FMRP). Improving dental hygiene	Community health and education settings	Adoption and continuation of programme activities	Intervention requiring coordination among multiple staff New policies, procedures and technologies	Interrupted time series study Mixed-methods Telephone interviews following a structured questionnaire with superintendents of public school districts Assessment Period 6 years postimplementation
36	Sheaff52	Improving the future for older people—reducing emergency bed days for over 75s	Nine acute hospital sites	Emergency bed days	Broad scale system change Intervention requiring coordination among multiple staff	Realist case evaluation Mixed methods case study Quantitative content analysis documents and questionnaires Assessment Period 4 years from baseline
37	Slaghuis53	Care for Better Improving care in nursing homes	Multiple nursing homes and home care organisations for the elderly	Factors related to routinisation and institutionalisation to form a sustainability scale	Intervention requiring coordination among multiple staff New policies, procedures and technologies	Case study Mixed methods case study Questionnaire to team members Assessment Period 1 year postfunding for the programme
38	Steadman36	ACCESS Access to community care and effective services and supports for homeless people with mental illness	Multiple community health settings	Status of services; source of funding secured Systems integration activities	Collaborative partnerships Capacity building	Case study Qualitative Assessment Period 1–6 months post support
39	Swain37	The National Implementing Evidence-Based Practices Project for people with serious mental illness Examined the implementation of 5 psychosocial practices in routine mental healthcare setting in 8 states	Multiple healthcare organisations	Continuation of practice	Intervention requiring coordination among multiple staff	Multisite case study Mixed methods Telephone survey gathering qualitative and quantitative data from site representatives with programme leaders and trainers Assessment Period 2 years post implementation
40	Thorsen54	Worksite canteen intervention of serving more fruit and vegetables 6-month intervention to increase fruit and veg consumption	Multiple community health settings	Fruit and veg consumption	Intervention requiring coordination among multiple staff Collaborative partnerships	Multisite case study Quantitative Measurement of Fruit and Veg consumption over a 3-week period Assessment Period 5 years post support
41	Wakerman55	The Sharing Health Care Initiative (SHCI) demonstration Project Chronic disease management in remote Australian Aboriginal communities through community-based self-management education	Multiple community health settings	Community awareness of chronic disease; community perception of the programme; recording and follow-up activities; improved clinical markers	Broad scale system change	Multisite case study Mixed methods Clinical audit and interview Assessment Period 26 months postimplementation
42	Whitford41	Prevention of Diabetic complications in UK general-practice clinics by a multifaceted, diabetes service in primary and secondary care	Individual patient level Multiple healthcare organisations	Documentary of clinical data; clinical indicators	Intervention requiring coordination among multiple staff	Case–control study Before-after design Assessment Period 10 years postimplementation

Most often, the system was the unit of analysis in these studies to determine outcome. For example, Meredith et al,³² in their study of depression among 17 primary care centres, found that even though only 11 of the sites continued programme activities, 15 reported spread of the activities to other providers and patient groups, suggesting that the programme had reach into new areas. This provides a holistic overview of the continued performance of activities within the context of organisational culture and programme growth. In addition, Greenhalgh et al,¹³ in their follow-up study of three services focused on the prevention of stroke, kidney and sexual health in primary and secondary care centres, found that most programme activities continued at 7 years follow-up. In addition, they found that significant cultural changes had occurred within these organisations and that services had spread into new areas.

Summary of study design and methodologies

Varying study designs and methodologies were used in the studies eligible for this review. Designs included case study reports (n=21), through to randomised controlled trials (n=1), and participant numbers varied from 20 to 2000. Table 3 summarises the study designs for all included studies.

Table 3.

Study designs used to measure defined outcomes

Study design	Frequency
Interrupted time-series	14
Randomised controlled trial (RCT)	1
Single and multisite case study design	19
Longitudinal quasi-experimental	1
Longitudinal case study design	2
Case control study and Cohort designs	5

Nearly half of all the studies (n=19) were designed using a mixed methods approach to measure outcomes. The combination of methods included quantitative measures (audit data, document review and surveys) combined with qualitative methods (in-depth interviews and focus groups with organisational staff members). Table 4 illustrates the relationship between the various outcomes evaluated and the most common methods used to collect these data. The research methodology varied between studies according to the type of defined outcomes (see table 4). Data collection for studies designed to measure health outcomes mostly used quantitative methods (50%) such as document review and audit.^{9 13 29–31 35 38–40 42} Mixed methods were most commonly used to measure all other indicators.

Table 4.

Methods of data collection used across defined sustainability outcomes

	Defined outcomes
Data collection methods	Health benefits (%)	Programme activities (%)	Community capacity (%)	Program diffusion (%)	Policies and procedures (%)	Combined (%)
Mixed methods	33	51	83	60	66	71
Quantitative methods	55	32	–	–	33	14
Qualitative methods	11	16	16	40	–	14

Timing of data collection

The timing and duration of data collection varied between the studies with assessment periods ranging from 1 to 10 years. Just over half of the studies (n=24) had a single ‘snapshot’ time point to measure sustainability outcome indicators against baseline data.^{9 13 28 30 31 33–37 40 41 43–55} The other half used longitudinal data^{14 42 56} or multiple time points^{8 29 32 38 39 57–65} such as the Ramsey et al³⁹ study. Timing of data collection appeared to vary according to the way sustainability was defined and the indicators selected to measure outcomes. For example, Ramsay et al³⁹ were specifically interested in observing any variability in intervention effects over time. They used monthly quantitative audit data over a 12-month period from baseline to assess adherence to clinical guidelines.³⁹ In contrast, for O'Loughlin et al,³⁴ sustainability was defined according to the level of ‘institutionalisation’ a programme reached within an organisation. Institutionalisation refers to the integration of a programme into the normal routines and everyday practice of an organisation.³⁴ They focused on one single follow-up time point to assess the degree of ‘institutionalisation’ through specific survey questions relating to organisational characteristics and activities.³⁴

In just over half of the studies (n=25), the assessment of the sustainability outcome started immediately post the implementation period. In a quarter of the studies (n=10), data were collected from the time of withdrawal of programme support, while in several studies (n=6) the assessment of sustainability started from baseline. Baseline was commonly defined as the period before active implementation of interventions prior to the start of the programme. The time to the start of that assessment is relevant information to address the first aim of this review—how are sustainability outcomes defined for measurement in the field of chronic disease? We seek to understand perspectives around when sustainability is achieved, as opposed to ongoing programme activity, and when does this start? The results reveal that in just over half of the studies in the review researchers view sustainability as beginning from the implementation phase and beyond.

Criteria for judging sustainability outcome

None of the studies in this review had published preset or post hoc numerical criteria for judging if sustainability had been achieved. Programmes were defined as being sustained if there was evidence of continued improvements from baseline measures or if outcomes were maintained following the implementation phase or cessation of funding, regardless of the magnitude achieved.

Intervention type and study methodology

The most commonly used intervention types were broad scale system change (n=18), interventions requiring coordination of multiple staff (n=26) and new policies, procedures and technologies (n=12) (see table 2). Interventions requiring coordination of multiple staff predominantly used quantitative measures (n=22) and approximately two-thirds qualified the findings with qualitative methods.

The studies designed to assess broad scale system change interventions had longer assessment periods (3–7 years) and more commonly used longitudinal study designs with multiple time points to measure outcomes throughout the programme cycle. For example, Bailie et al²⁹ repeated an audit on five occasions over a 3-year period. Likewise, McDermott et al⁴⁰ evaluated retrospective patient data over time from implementation.⁴⁰

Discussion

The scope of care provided to people with chronic illness is heterogeneous, spanning multiple disease types and management interventions. This heterogeneity of care is reflected in the methods used for sustainability research in the chronic disease field. The results of this review reveal that indicators for sustainability, research methodology and timing and duration of data collection methods varied according to the defined outcomes and intervention type.

In most of the studies, the sustainability outcome was defined as the maintenance of programme activities. This fits well with the implementation cycle: the problem area is identified, and goals and activities are designed and implemented and evaluated over time to assess ongoing maintenance.^{66 67} They are intervention-focused with activity-based outcomes. However, this narrow perspective does not provide evidence on whether health benefits are being achieved based on this ongoing activity, and is often an indirect causal assumption that is implied.

Given that improved outcomes are what most programmes aim for, it was surprising that only about one-quarter of studies included the measurement of patient-level health outcomes as a primary outcome of sustainability. Nevertheless, most of the programme activity outcomes measured in the included studies could be considered reasonable surrogates for achieving health outcomes. For example, adherence to COPD evidence-based guidelines in the Brand et al⁹ study is likely to result in improved outcomes since the relationship between patient interventions delivered through a programme has an evidence base associated with achieving positive health benefits.

A trend towards defining the primary outcome as the maintenance of programme activities, as evidenced in this review, could be due to a large focus on self-management interventions to manage chronic disease.⁴ Self-management strategies are typically measured by patient self-report outcomes.⁴ It has previously been suggested that the effectiveness of self-management interventions are mostly derived from variable evaluation-based measures that carry a substantial amount of measurement error.⁴ This may contribute to a shift in focus towards activity-based outcomes focusing on the quality and continuation of delivered services rather than health-based outcomes. Clinical indices provide clear reference points for precomparison and postcomparison allowing for concise summaries and recommendations for health providers.

The majority of studies were characterised as broad scale system change interventions.¹⁹ These studies used clinical markers as a primary outcome in addition to programme activities, community capacity, programme diffusion and/or policy outcomes. The various outcomes together gave a bigger picture of change and transformation with the whole organisation itself. This method of using health outcome data with other outcomes may provide more robust evidence about the whole broad scale change. Consideration of intervention type may assist in the complex process of planning for sustainability research in the chronic disease field.

Our results reveal that the study methodology varied according to intervention type and this may go some way in explaining the variation that exists in research between programs. For example, evaluations of broad scale system change interventions often employ multiple time points to measure outcomes, demonstrating an interest in researching trends over time. These studies often evaluated multiple outcomes as well. A growing number of studies (one half from this review) are using multiple time points over time to measure indicators. This finding supports Scheirer et al's¹⁹ hypothesis that such interventions require ongoing evaluations.

This is an important consideration for future sustainability research on chronic disease management. The feasibility of continuous measurement should be explored through clinical registries or document review from the outset of programme planning.^{68 69} Longitudinal testing over multiple time points may serve to distinguish residual improvements from implementation to sustained persistence of improvement.⁷⁰

Measurement needs to encapsulate how programs evolve over time, adapt to changing contexts and transform to suit changing health system needs. Evaluations of such studies should also capture spread into new areas or any difficulties as a result of potential environmental changes. Importantly, none of the studies in this review defined predetermined or post hoc numerical criteria as an empirical measure of sustainability being achieved or what would be considered acceptable degrees of regression/attrition. Decision rules made around defining outcomes for dependent variables appeared to be a subjective process based on the objectives and theoretical underpinnings unique to each study. Moreover, since these programs are situated within complex systems with evolving contexts, it may be that setting rigid criteria for judging outcomes is not realistic or desirable.

Our review supports the conceptualisation of sustainability as a broad scale approach.^{14 2 13} The results reveal that most studies took a complex approach with mixed methods and defined outcomes that extend beyond continuation of programme activity. For example, Meredith et al³² quantitatively evaluated improved delivery of services and qualified their findings through in-depth interviews to determine the spread of programme into other areas. Consistent with the work of Gruen et al¹⁴ and Greenhalgh et al,² our findings reveal that the sustainability outcome is dependent on a complex set of inter-relationships between various organisational and social systems impacting broad scale change. It extends the work of Gruen et al's¹⁴ and Greenhalgh et al's² by providing a synthesis of the specific research methods used to measure sustainability in the area of chronic disease. It also extends Wiltsey-Stirman et al's¹⁵ review of methods used to measure sustainability outcomes by providing a description of intervention types in the chronic disease field. Finally, it supports and extends Scheirer et al's¹⁹ framework for evaluating interventions by reviewing intervention types and relating these to the study methodology currently used.

The strengths of our review include the systematic identification and abstraction of eligible publications assessed by more than one author and the use of manual reference searching and forward citation tracking as well as the use of an appropriate quality assessment tool for classifying the eligible publications. The limitations of our review include the fact that we may have missed relevant reports if these have been published in the grey literature. The lack of standardisation and reporting of the sustainability outcomes precluded our ability to perform meta-analytic synthesis. Consensus on aspects of standardised measurement and reporting would promote the future possibility of meta-analytic syntheses.

Conclusion

Overall, despite chronic diseases being a major focus of attention worldwide, owing to the growing financial impacts on health systems and associated disease burden imposed on society, publications on the sustainability of programmes within this field are limited and cover only a small range of conditions, such as diabetes. This review provides an important contribution to understanding the way sustainability is currently measured in the chronic disease field. Currently, research methods to evaluate health programme sustainability in the area of chronic disease management vary widely and very little work in this area has been published. Given the increasing burden of chronic disease in our society, it is also a matter of concern that there are no clear guidelines on the best way to measure sustainability in this field.

However, our review shows some emerging patterns in the research methods used to measure sustainability. First, there is a clear trend in the use of assessment through multiple time points from baseline to gain valuable information about the ongoing effectiveness of given programmes. Second, a clear relationship between intervention type and use of research methods is shown, with broad scale system change interventions using longitudinal designs with multiple time points. Finally, our results support recent recommendations that evaluation of sustainability requires a holistic approach capturing all elements with the use of mixed methods approaches.

Further development of current empirical methods used to measure sustainability, in the area of chronic disease, is needed to provide direction for programme planners and evaluators in designing, implementing and evaluating health programmes. More research into methodology and measurement is needed to provide a more rigorous science behind sustainability research and clearer guidelines for future programme planners and evaluators.