Abstract
Objectives
Long-term delivery, or sustainability, of evidence-based interventions is necessary for public health benefits to be realised. However, sustainment of effective interventions is poor. Understanding the evidence-base and identifying potential gaps is necessary to inform where future research efforts are most warranted.
Study design
We undertook a repeat cross-sectional bibliographic review of research published in 10 public health journals across three time periods (2010, 2015 and 2020/2021).
Methods
Studies were eligible if they were a data-based study or review article. Studies were assessed as to whether they focused on sustainability. The percentage of public health research studies assessing sustainability overall and by the three time periods was calculated. The association between time period and the proportion of sustainability articles was assessed using logistic regression. Descriptive statistics were used to summarise study characteristics overall and by time period.
Results
10,588 data-based articles were identified, of which 1.3 % (n = 136) focused on sustainability. There was a statistically significant association between time period and the proportion of sustainability research, with a slight increase across the three time periods: 0.3 % (95 % CI: 0.1 %, 0.7 %) in 2010, 1.4 % (95 % CI: 1.0 %, 1.9 %) in 2015 and 1.6 % (95 % CI: 1.3 %, 1.9 %) in 2020/2021. Most research was descriptive/epidemiological (n = 69, 51 %), few focused on measurement (n = 2, 1.5 %) and none on cost effectiveness. Only one intervention study assessed the effect of specific sustainability strategies.
Conclusions
This bibliographic review highlights the need for more public health research on sustainability, particularly in the areas of measurement, sustainability interventions, and cost effectiveness.
Keywords: Sustainability, Research output, Bibliographic review, Public health, Evidence-based interventions, Long-term implementation
1. Introduction
Governments internationally make considerable investment in the development and delivery of public health programs, although expenditure for public health is often less than other areas [[1], [2], [3], [4], [5], [6]]. While investment in public health interventions has significant potential to positively influence health and economic outcomes, the potential benefits of such programs can only be realised if they are implemented at a population level [6]. Accordingly, funding schemes have been developed to facilitate translational research and build an evidence base in this area [6]. Such investments have improved the production of implementation research, with implementation science output in the field of public health having more than tripled in the decade to 2017 [7]. The focus of such research has begun to transition from studies describing implementation models, frameworks, and determinants, to those developing and testing the effect of implementation strategies on implementation indicators like adoption, fidelity, and cost [8].
To date, implementation research has predominantly focused on initial implementation of evidence-based interventions (EBI) [[9], [10], [11]]. Sustainability is an emerging but important phase of the implementation process [[12], [13], [14]]. It has been defined as “after a defined period of time, a program, clinical intervention, and/or implementation strategies continue to be delivered and/or individual behaviour change (i.e., clinician, patient) is maintained; the program and individual behaviour change may evolve or adapt while continuing to produce benefits for individuals/systems” [15]. Despite the importance of sustained delivery of effective EBIs, the impacts of many EBIs cease or are reduced over time, particularly once active implementation support is removed [[16], [17], [18]]. For example, a comprehensive review of 125 empirical studies of public health and clinical interventions reported that less than half of EBIs were sustained to high levels of fidelity at least two years following initial implementation [16]. Furthermore, whilst limited to the school setting, a more recent review by Herlitz et al., in 2020 found that none of the 18 school-based interventions included in the review were sustained in their entirety [17].
To maximise the potential benefits to the community from investment in program implementation, public health policy makers and practitioners need a strong evidence base on which to inform their decisions to support the sustained delivery of EBIs. For example understanding what factors may impact sustainability or what strategies are effective in supporting sustainability, will help public health policy makers and practitioners to better plan for and support long term delivery of EBIs. Encouragingly there are examples of such informative work having recently been undertaken [17,[19], [20], [21], [22]]. To progress the field, a number of priority- and agenda-setting publications have called for the conduct of empirical research which will advance the science of sustainability, including its conceptual development as well as the development of robust measures of key constructs and outcomes, and strategies to improve program sustainability [10,11,16,[23], [24], [25]]. Despite repeated calls for action, it is unclear whether public health research (or funders) have prioritised this field and what type of sustainability research is being produced.
Bibliographic reviews examine the quantity and type of publications in a specific field over a defined time period [26]. Such reviews help describe research activity, characterise the evidence-base, identify potential gaps in the literature, and inform where future research efforts may be warranted [[27], [28], [29]]. Researchers have undertaken bibliographic reviews to describe research activity in specific public health content areas such as tobacco smoking, alcohol use, falls prevention and physical activity [[27], [28], [29], [30], [31]], as well as public health research more broadly [32]. However, we are not aware of any published bibliographic reviews that have described the research on sustainability in public health.
1.1. Aims
This bibliographic review assesses sustainability-related research in the area of public health across three time-periods (2010, 2015, 2020/2021). The specific aims were to:
-
1)
Describe the proportion of data-based public health research articles that focus on sustainability, that have been published in 10 leading public health journals across the three time-periods;
-
2)
Assess whether the proportion of public health research articles focused on sustainability has changed over time; and
-
3)
Describe the characteristics of public health research articles focused on sustainability across the three time-periods.
2. Methods
2.1. Study design and sample
A repeat cross-sectional bibliographic review was undertaken of public health research published within a purposive sample of leading public health journals across three time periods that is the years: 2010, 2015 and 2020/2021 (up to August/September 2021 when the search was completed).
The 2020 SCImago Journal & Country Rank [33] was used to produce a comprehensive sample of top-tiered public health journals. A list of journals indexed in the top 25 % (i.e., Q1) from the category of “Public, Environmental and Occupational Health” was created and reviewed by three experts in public health (NN, TM and AH). From the list of 73 available Q1 journals, the 10 top ranked journals (according to their Q1 rating), and which were considered the most relevant to public health based on assessment of the journals scope by the three experts, were selected for inclusion in this study. See Table 1 for an overview of included journals. Any article published in these selected journals was considered to be public health related.
Table 1.
Characteristics of journals used in sampling frame.
| Journal name | Impact factora | Impact factor year | First year of publication | Number of eligible articles |
|---|---|---|---|---|
| American Journal of Preventive Medicine | 6.604 [51] | 2021 | 1985 | 2010: 259 |
| 2015: 300 | ||||
| 2020/2021: 452 | ||||
| Total: 1011 | ||||
| American Journal of Public Health | 9.308 [52] | 2020 | 1911 | 2010: 364 |
| 2015: 525 | ||||
| 2020/2021: 299 | ||||
| Total: 1188 | ||||
| Annual Review of Public Health | 21.870 [53] | 2021 | 1980 | 2010: 28 |
| 2015: 28 | ||||
| 2020/2021: 53 | ||||
| Total: 109 | ||||
| BMC Public Health | 4.135 [54] | 2021 | 2001 | 2010: 786 |
| 2015: 1246 | ||||
| 2020/2021: 3359 | ||||
| Total: 5391 | ||||
| European Journal of Public Health | 4.424 [55] | 2021 | 1991 | 2010: 105 |
| 2015: 221 | ||||
| 2020/2021: 351 | ||||
| Total: 677 | ||||
| Implementation Science | 7.327 [56] | 2021 | 2006 | 2010: 95 |
| 2015: 209 | ||||
| 2020/2021: 158 | ||||
| Total: 462 | ||||
| International Journal of Public Health | 5.1 [57] | 2021 | 1974 | 2010: 77 |
| 2015: 102 | ||||
| 2020/2021: 154 | ||||
| Total: 333 | ||||
| Lancet Public Health | 72.427 [58] | 2021 | 2016 | 2010: NA |
| 2015: NA | ||||
| 2020/2021: 96 | ||||
| Total: 96 | ||||
| Prevention Science | 3.931 [59] | 2021 | 2000 | 2010: 36 |
| 2015: 106 | ||||
| 2020/2021: 165 | ||||
| Total: 307 | ||||
| Preventive Medicine | 4.637 [60] | 2021 | 1972 | 2010: 167 |
| 2015: 294 | ||||
| 2020/2021: 553 | ||||
| Total: 1014 |
Impact factor sourced from journal website.
2.2. Study identification and eligibility
All articles published in the 10 public health journals from each of the three selected time-periods were downloaded and assessed independently by one review author (BM, DL or KA) for eligibility. As a quality control measure, a random 10 % of all articles were independently assessed for eligibility by a second author (CG). All included studies were checked for eligibility by a second author (AH, NN or CG). Individual studies were considered eligible if they were a data-based study or review article and published in 2010, 2015 or 2020/2021 (up to August/September 2021 when the search was completed). We selected 2010 as the first time-period as this was five years after publication of an original agenda-setting paper for sustainability [34], and thus allowed time for research in this area to begin to accrue. Similar to other bibliographic reviews, studies were excluded if they did not detail data-based research (e.g., editorials, letters with no new data, corrections, conference abstracts and protocol papers) [28].
2.3. Data extraction and statistical analyses
One author (DL, BM, or KA) independently classified whether each eligible article focused on sustainability or not. Due to variation and inconsistencies in how sustainability has been defined [15], we employed a broad definition of whether a study focused on sustainability to ensure all relevant articles were included. We included all studies where the authors indicated a focus on sustainability, or used related terms to describe their objectives or results (i.e., maintenance, institutionalisation, routinisation, continued implementation, or long-term follow-up). The following characteristics were then extracted from these studies: type of research, study design, setting, and content area. Type of research articles were classified based on definitions used in previous bibliographic reviews [28,30,32], including: descriptive/epidemiological (i.e., focus on prevalence, pattern or predictors), measurement (i.e., development or psychometric evaluation of data collection instruments), intervention (i.e., testing the effect of interventions), or review. Study designs were classified according to categories used in previous bibliographic reviews [32]: systematic review/meta-analysis, non-systematic review, randomised controlled trial, non-randomised controlled trial, cohort study, cross-sectional study, cost effectiveness, qualitative, mixed methods or other. Each respective study aim was used to classify the research type and design. Data extracted from all eligible sustainability studies were checked by a second author (CG or AH).
All analyses were conducted in SAS version 9.3. Descriptive statistics were used to summarise the characteristics of included studies. The proportion and 95 % confidence interval (CI) of articles focusing on sustainability out of all eligible articles was calculated for each of the three time periods. A logistic regression was used to assess whether there was an association between time period and the proportion of sustainability research published. Due to the small number of eligible articles, assessment of the statistical differences between article characteristics (e.g., study design and type) was not feasible.
3. Results
A total of 12,526 articles were published across the three time-points and 10 public health journals, of which 10,588 were data-based articles eligible for inclusion in this study. The number of data-based articles published in each journal across the three years is presented in Table 1. Of the 10,588 data-based articles, 136 (1.3 %) focused on sustainability (see Fig. 1).
Fig. 1.
Flow diagram illustrating study selection.
3.1. Aim 1: Proportion of data-based research articles focused on sustainability
The percentage of original research articles that focused on sustainability was 0.3 % (95 % CI: 0.1 %, 0.7 %) in 2010, 1.4 % (95 % CI: 1.0 %, 1.9 %) in 2015 and 1.6 % (95 % CI: 1.3 %, 1.9 %) in 2020/2021.
3.2. Aim 2: Proportion of data-based research articles focused on sustainability over time
Compared to the earliest publication year of 2010, we observed a small increase in the proportion of data-based research articles focusing on sustainability in the more recent publication years (i.e., 0.3 % in 2010 vs. 1.4 % in 2015 and 1.6 % in 2020/2021); while we saw a similar proportion of articles focusing on sustainability in 2015 and 2020/2021 (1.4 % vs. 1.6 %). Logistic regression results indicated a statistically significant association between year of publication and whether data-based articles focused on sustainability (p = 0.0007). Specifically, compared to 2010 the odds of a sustainability study being published were approximately five times higher in both 2015 (odds ratio (OR): 4.5; 95 % CI: 1.9, 10.5) and 2020/2021 (OR: 5.0; 95 % CI: 2.2, 11.6).
3.3. Aim 3: Characteristics of sustainability research
Across all time periods the majority of sustainability articles were descriptive/epidemiological (n = 69, 51 %), followed by intervention (n = 46, 34 %) and literature reviews (n = 16, 12 %). Few articles focused on measurement (n = 2, 1.5 %). All but one of the 46 intervention studies assessed the longer-term outcomes of an intervention, including the sustained benefit (n = 22, 48 %), sustained delivery (n = 19, 41 %) or both sustained benefit and delivery of an EBI (n = 5, 11 %). Only one intervention study assessed the effect of specific strategies designed to sustain the delivery of an EBI. The most frequently used study design was mixed methods (n = 31, 23 %), followed by qualitative (n = 29, 21 %). No studies explored cost-effectiveness (see Table 2).
Table 2.
Characteristics of eligible articles focusing on sustainability.
| Characteristic | Year |
Total (n = 136) |
||
|---|---|---|---|---|
| 2010 n = 6 |
2015 n = 42 |
2020/2021 n = 88 |
||
| Number (%) | Number (%) | Number (%) | Number (%) | |
| Countrya | ||||
| Australia | 0 | 2 (6 %) | 6 (7 %) | 8 (6 %) |
| Canada | 0 | 1 (3 %) | 6 (7 %) | 7 (6 %) |
| United States | 3 (60 %) | 14 (39 %) | 21 (25 %) | 38 (30 %) |
| United Kingdom | 0 | 5 (14 %) | 5 (6 %) | 10 (8 %) |
| Multiple | 0 | 3 (8 %) | 5 (6 %) | 8 (6 %) |
| Otherb | 2 (40 %) | 11 (31 %) | 41 (49 %) | 54 (43 %) |
| Type of research | ||||
| Descriptive/epidemiological | 1 (17 %) | 27 (64 %) | 41 (47 %) | 69 (51 %) |
| Measurement | 0 | 0 | 2 (2 %) | 2 (1 %) |
| Interventionc | 4 (67 %) | 9 (21 %) | 33 (38 %) | 46 (34 %) |
| Review | 1 (17 %) | 6 (14 %) | 9 (10 %) | 16 (12 %) |
| Other | 0 | 0 | 3 (3 %) | 3 (2 %) |
| Research design | ||||
| Systematic review/meta-analysis | 0 | 4 (10 %) | 6 (7 %) | 10 (7 %) |
| Non-systematic review | 0 | 2 (5 %) | 4 (5 %) | 6 (4 %) |
| Randomised controlled trial | 3 (50 %) | 6 (14 %) | 13 (15 %) | 22 (16 %) |
| Non-randomised controlled trial | 1 (17 %) | 1 (2 %) | 2 (2 %) | 4 (3 %) |
| Cohort study | 1 (17 %) | 9 (21 %) | 12 (14 %) | 22 (16 %) |
| Cross-sectional study | 0 | 0 | 7 (8 %) | 7 (5 %) |
| Cost effectiveness | 0 | 0 | 0 | 0 |
| Mixed methods | 1 (17 %) | 9 (21 %) | 21 (24 %) | 31 (23 %) |
| Qualitative | 0 | 9 (21 %) | 20 (23 %) | 29 (21 %) |
| Other | 0 | 2 (5 %) | 3 (3 %) | 5 (4 %) |
| Setting | ||||
| Educational | 2 (33 %) | 14 (33 %) | 18 (20 %) | 34 (25 %) |
| Workplace | 0 | 1 (2 %) | 4 (5 %) | 5 (4 %) |
| Community (e.g., sporting clubs, churches, and community pharmacies) | 2 (33 %) | 18 (43 %) | 43 (49 %) | 63 (46 %) |
| Clinical | 2 (33 %) | 7 (17 %) | 21 (24 %) | 30 (22 %) |
| Other | 1 (17 %) | 2 (5 %) | 1 (1 %) | 17 (14 %) |
| Public Health focus | ||||
| Smoking | 0 | 3 (7 %) | 9 (10 %) | 12 (9 %) |
| Alcohol | 0 | 2 (5 %) | 2 (2 %) | 4 (3 %) |
| Nutrition | 0 | 6 (14 %) | 19 (22 %) | 25 (18 %) |
| Mental health | 2 (33 %) | 6 (14 %) | 7 (8 %) | 15 (11 %) |
| Physical activity | 1 (17 %) | 11 (26 %) | 18 (20 %) | 30 (22 %) |
| Substance use | 1 (17 %) | 2 (5 %) | 2 (2 %) | 5 (4 %) |
| Non-communicable disease | 0 | 1 (2 %) | 12 (14 %) | 13 (10 %) |
| Child/youth health and wellbeing | 0 | 8 (19 %) | 20 (23 %) | 28 (21 %) |
| Overweight and obesity | 0 | 0 | 7 (8 %) | 7 (5 %) |
| Women's health | 0 | 0 | 9 (10 %) | 9 (7 %) |
| Communicable diseases | 0 | 4 (10 %) | 11 (13 %) | 15 (11 %) |
| Other | 3 (50 %) | 10 (24 %) | 33 (38 %) | 46 (34 %) |
Cell sizes do not match total sample as some studies, especially reviews, were not conducted or focused on a specific country.
Other includes countries where less than a total of 5 studies were conducted in an individual country across all time periods. In most instances only one study was conducted in each individual country.
Only one intervention study assessed the effect of specific strategies designed to sustain the delivery of an intervention.
Differences in the type of research conducted between the three time periods were difficult to examine due to the small number of articles published in 2010 (n = 5). However, descriptively the type of study designs appeared similar for the later time-periods of 2015 and 2020/2021, with the largest proportion of studies being descriptive/epidemiological (64 % vs. 47 %) and the fewest focusing on measurement (0 % vs. 2 %). Similarly, the research designs employed by studies in 2015 and 2020/2021 were similar, with mixed methods (21 % vs. 24 %) and qualitative (21 % vs. 23 %) the most frequently used at both time periods. Although a larger number of cross-sectional studies were published in the most recent time period (0 % vs. 8 %) (see Table 2).
Across all time periods the country where research had most commonly been conducted was the United States (30 %), followed by the United Kingdom (8 %). Collectively, most studies were conducted in community-based settings (e.g., sporting clubs, churches, and community pharmacies) (46 %), followed by educational (25 %) and clinical settings (22 %). Few studies were conducted in a workplace setting (4 %). The most common public health focus addressed were: physical activity (22 %), child/youth health and wellbeing (21 %) and nutrition (18 %). Less than 10 % of studies focused on each public health topic area of: smoking (9 %), women's health (7 %), overweight or obesity (outside of physical activity and nutrition) (5 %), other substance use (4 %), and alcohol use (3 %). Again, it was difficult to compare differences in the setting and public health focus across the three time periods, due to the small number of sustainability articles published in 2010. However, the proportion of articles published in the more recent time periods (e.g., 2015 and 2020/2021) were similar for most topics, with the exception of a larger percentage of studies published in 2020/2021 vs 2015 for the areas of nutrition (14 % vs. 22 %), non-communicable disease (14 % vs 2 %), women's health (10 % vs 0 %) and overweight and obesity (8 % vs. 0 %) (see Table 2).
4. Discussion
Despite repeated calls for research to be more focused on the sustainability of health-related interventions [10,23,24,34], we found that the proportion of public health research in this area is low, making up just over 1 % of database research articles across the three time periods assessed. Despite an observed increase in the proportion of public health articles focusing on sustainability from 2010 (0.3 %) to more recent years of 2015 (1.4 %) and 2020/2021 (1.3 %), the absolute change in the proportion was small and remained critically low even at its peak. Such output is not of a volume sufficient to rapidly advance the field.
We found that the majority of research studies were descriptive, a finding which is consistently reported by other bibliographic reviews [27,28,30,32]. Specifically, over half of the studies identified were descriptive (51 %), which is just outside previous bibliographic reviews reporting estimates of 52 %–87 % [27,30]. As an emerging area of research, it is not surprising that the majority of work has focused on describing the phenomenon of sustainability. It also aligns with previous calls for action to identify and understand the factors that affect sustainability [10]. Unexpectedly, we found that just over one third (34 %) of all studies examined an intervention. This is higher than several other bibliographic reviews in fields of research with a more mature evidence-base that have reported less than 20 % of research output as intervention-focused [27,32]. However, only one of the intervention studies identified specifically assessed the effect of strategies designed to support EBI sustainability; with all others focused on assessing the longer term benefits and/or delivery of an EBI. There is a need for studies which adequately describe and test the effectiveness of specific strategies focused on improving the sustainability of EBIs [18,25].
Of concern was the lack of cost-effectiveness studies identified, although it is possible that cost may have been examined separately from the sustainability work or reported as a secondary outcome within another type of research design (e.g., as part of an intervention study). Nonetheless, one of the most commonly identified determinants of sustainment is funding availability or continued financial support [10]. Given this understanding, the cost implications and long-term cost-effectiveness of an EBI is essential for understanding and supporting sustainability. Such information is also crucial for decision makers who are vested in the ongoing value of investment of public health programs and initiatives. We also identified very few measurement studies (1 %). This is unsurprising given the recognised challenges in defining and measuring sustainability [11,16,35]. Encouragingly, recent reviews of implementation and sustainability measures [[35], [36], [37], [38]] suggest an increase in work being conducted in this area; although the psychometric properties of available measures may still require refinement [35,37,38].
Most of the sustainability research that we identified was conducted in community-based settings. However, only 4 % of sustainability studies were conducted in the workplace. For over 30 years the World Health Organization (WHO) has identified workplaces as a key setting for addressing chronic disease prevention [39]. Furthermore, the number of interventions and reviews that have been conducted on workplace interventions is extensive [40]. Another surprising finding is the relatively few studies (<10 %) we identified that were focused on addressing tobacco and alcohol use. In light of the sizable burden of illness attributable to these health behaviours [[41], [42], [43]], and the number of interventions and polices that have been implemented to prevent uptake and promote cessation [42,43], this is an area where future research may be warranted to ensure the funding used to deliver related policy and practices is not wasted.
5. Limitations
This study provides an overview of the proportion of research that has been dedicated to assessing sustainability in 10 leading public health journals, and how this has changed across three time periods. There are several limitations of this work that should be acknowledged. First, despite the inclusion of high-quality journals, the 10 that we included may not be representative of the breadth of public health intervention sustainability research across differing contexts; for example, low- and middle-income contexts. It is also possible that other public health-related journals have published a greater proportion of sustainability-focused research. We also assumed all articles published in the 10 selected journals had a public health focus due to their publication in a public health related journal. It is possible that some articles may not have been strictly public health focused due to the breadth of some of the journals’ scope. Second, we only assessed studies published across three time periods. It is possible that the trends in research output may have differed if a greater number of time points of different time periods were examined. Third, the small number of sustainability articles made it difficult to conduct statistical comparisons of the differences in the characteristics of research output between the three time periods. Third, it is possible that we failed to recognise some sustainability-related studies, as the definitions and terminology used to describe these terms is often vague and poorly defined [11,18], with some studies only mentioning their focus on sustainability in passing using or with unconventional terminology. However, we employed an inclusive definition of sustainability to avoid missing relevant articles. On the other hand, our broad inclusion criteria may have resulted in the classification of some studies that may not otherwise be viewed as focusing on sustainability. For example a number of the intervention studies that assessed the longer term health effects of an EBI used relatively short (i.e., <6 months) follow-up periods, yet we included these due to the authors intent of assessing sustainability of their intervention.
6. Conclusions
This bibliographic review provides an extensive and in-depth assessment of the research activity made in the area of sustainability in 10 leading public health journals over the last 10 years. The study results suggest that despite multiple calls for research into this area over the last 15+ years, only around 1 % have examined this important topic, with minimal increase over time. Despite examples of effective and cost effective interventions across a variety of settings and targeting a range of population groups [5,[44], [45], [46], [47]], and emerging research regarding methods to achieve initial implementation [[48], [49], [50]], the findings of this study suggests the evidence-base is not sufficiently geared towards informing how best to achieve sustained EBI benefits. The findings from this study should be used to direct future research efforts so that we can address one of the greatest challenges faced by the area of public health.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Availability of data and materials
Data and materials relating to this review are available from the corresponding author on reasonable request.
Funding
This project is funded through the National Health and Medical Research Council (NHMRC) as part of NN's Medical Research Future Fund (MRFF) Investigator Grant (APP1194785) and was supported by work undertaken as part of an NHMRC Centre for Research Excellence grant (APP1153479). NN is supported by a MRFF Investigator Grant (APP1194785); LW is supported by an NHMRC Investigator Grant (APP1197022); RCS by an American Cancer Society Research Scholar Grant (RSG-17-156-01-CPPB); and AS by a University of Newcastle PhD scholarship (ref. 3145402). The funders had no role in the study design, conduct of the study, analysis, or dissemination of findings.
Author contributions
LW conceived the concept for this review. AH developed the protocol with assistance from LW, NN and TM. AH undertook article screening, data extraction, analysis and led drafting of the manuscript. BM, KA and DCWL sourced all articles, screened for eligibility and completed data extraction. CG, CL and AS also assisted with data extraction. RS and NN provided extensive input and guidance on the content area. All authors contributed to the interpretation of the results, drafting and reviewing of the manuscript.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
The author team would like to thank Hunter New England Population Health, the University of Newcastle for providing infrastructure support to conduct this review.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.puhip.2023.100444.
List of Abbreviations
- EBI
Evidence Based Intervention
- CI
confidence Interval
- Q1
Quartile 1
- OR
Odds Ratio
Appendix A. Supplementary data
The following is the Supplementary data to this article:
References
- 1.(IOM). IoM . Press. TNA; Washington DC: 2012. For the Public's Health: Investing in a Healthier Future. [Google Scholar]
- 2.Care. CoADoHaA Record investment in the future of Australia's health system. 2022. https://www.health.gov.au/ministers/the-hon-greg-hunt-mp/media/record-investment-in-the-future-of-australias-health-system [Available from:
- 3.Care. CoADoHaA Preventive and public health research initiative. 2022. https://www.health.gov.au/our-work/preventive-and-public-health-research-initiative [Available from:
- 4.Leider J., Resnick B., McCullough J.M., Alfonso Y.N., Bishai D. Inaccuracy of official estimates of public health spending in the United States, 2000–2018. Am. J. Publ. Health. 2020;110:S194–S196. doi: 10.2105/AJPH.2020.305709. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Organisation W.H. 2014. The Case for Investing in Public Health: A Public Health Summary Report for SPHO 8. Geneva Switzerland. [Google Scholar]
- 6.Wolfenden L., Chai L.K., Jones J., McFadyen T., Hodder R., Kingsland M., et al. What happens once a program has been implemented? A call for research investigating strategies to enhance public health program sustainability. Aust NZJ Public Health. 2019;43(1):3–4. doi: 10.1111/1753-6405.12867. [DOI] [PubMed] [Google Scholar]
- 7.Wolfenden L., Milat A., Lecathelinais C., Skelton E., Clinton-McHarg T., Williams C., et al. A bibliographic review of public health dissemination and implementation research output and citation rates. Prevent. Med.Rep. 2016;4:441–443. doi: 10.1016/j.pmedr.2016.08.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Spoth R., Rohrbach L.A., Greenberg M., Leaf P., Brown C.H., Fagan A., et al. Addressing core challenges for the next generation of type 2 translation research and systems: the translation science to population impact (TSci Impact) framework. Prev. Sci. 2013;14(4):319–351. doi: 10.1007/s11121-012-0362-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Neta G., Sanchez M., Chambers D., Phillips S., Leyva B., Cynkin L., et al. Implementation science in cancer prevention and control: a decade of grant funding by the National Cancer Institute and future directions. Implement. Sci. 2015;10(4) doi: 10.1186/s13012-014-0200-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Proctor E., Luke D., Calhoun A., McMillen C., Brownson R., McCrary S., et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement. Sci. 2015;10(1):1–13. doi: 10.1186/s13012-015-0274-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Shelton R.C., Cooper B.R., Stirman S.W. The sustainability of evidence-based interventions and practices in public health and health care. Annu. Rev. Publ. Health. 2018 doi: 10.1146/annurev-publhealth-040617-014731. [DOI] [PubMed] [Google Scholar]
- 12.Moullin J.C., Dickson K.S., Stadnick N.A., Rabin B., Aarons G.A. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement. Sci. 2019;14(1):1–16. doi: 10.1186/s13012-018-0842-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Aarons G.A., Glisson C., Hoagwood K., Kelleher K., Landsverk J., Cafri G. Psychometric properties and US national norms of the evidence-based practice attitude scale (EBPAS) Psychol. Assess. 2010;22(2):356. doi: 10.1037/a0019188. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Moores C.J., Miller J., Perry R.A., Chan L.L.H., Daniels L.A., Vidgen H.A., et al. CONSORT to community: translation of an RCT to a large-scale community intervention and learnings from evaluation of the upscaled program. BMC Publ. Health. 2017;17(1):1–12. doi: 10.1186/s12889-017-4907-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Moore J., Mascarenhas A., Bain J., Straus S. Developing a comprehensive definition of sustainability. Implement. Sci. 2017;12(110) doi: 10.1186/s13012-017-0637-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Wiltsey Stirman S., Kimberly J., Cook N., Calloway A., Castro F., Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement. Sci. 2012;7(1):1–19. doi: 10.1186/1748-5908-7-17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Herlitz L., MacIntyre H., Osborn T., Bonell C. The sustainability of public health interventions in schools: a systematic review. Implement. Sci. 2020;15(1):1–28. doi: 10.1186/s13012-019-0961-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Hailemariam M., Bustos T., Montgomery B., Barajas R., Evans L.B., Drahota A. Evidence-based intervention sustainability strategies: a systematic review. Implement. Sci. 2019;14(1):1–12. doi: 10.1186/s13012-019-0910-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Cassar S., Salmon J., Timperio A., Naylor P.J., van Nassau F., Contardo Ayala A.M., et al. Adoption, implementation and sustainability of school-based physical activity and sedentary behaviour interventions in real-world settings: a systematic review. Int. J. Behav. Nutr. Phys. Activ. 2019;16(120) doi: 10.1186/s12966-019-0876-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Shoesmith A., Hall A., Wolfenden L., Shelton R.C., Powell B.J., Brown H., et al. Barriers and facilitators influencing the sustainment of health behaviour interventions in schools and childcare services: a systematic review. Implement. Sci. 2021;16(62) doi: 10.1186/s13012-021-01134-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Shoesmith A., Hall A., Wolfenden L., Shelton R.C., Yoong S., Crane M., et al. School-level factors associated with the sustainment of weekly physical activity scheduled in Australian elementary schools: an observational study. BMC Publ. Health. 2022;22(2022) doi: 10.1186/s12889-022-13732-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Whelan J., Low P., Millar L., Allender S., Bell C. Sustaining obesity prevention in communities: a systematic narrative synthesis review. Obes. Rev. 2018;19(6):839–851. doi: 10.1111/obr.12675. [DOI] [PubMed] [Google Scholar]
- 23.Scheirer M.A., Dearing J.W. An agenda for research on the sustainability of public health programs. Am. J. Publ. Health. 2011;101(11):2059–2067. doi: 10.2105/AJPH.2011.300193. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Shelton R., Lee M. Sustaining evidence-based interventions and policies: recent innovations and future directions in implementation science. Am. J. Publ. Health. 2019;109(52):S132–S134. doi: 10.2105/AJPH.2018.304913. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Nathan N., Powell B.J., Shelton R.C., Laur C.V., Wolfenden L., Hailemariam M., et al. Do the expert recommendations for implementing change (ERIC) strategies adequately address sustainment? Frontiers in Health Services. 2022;2(905909) doi: 10.3389/frhs.2022.905909. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Ellegaard O., Wallin J. The bibliometric analysis of scholarly production: how great is the impact? Scientometrics. 2015;105:1809–1831. doi: 10.1007/s11192-015-1645-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Lee K., Ding D., Grunseit A., Wolfenden L., Milat A., Bauman A. Many papers but limited policy impact? A bibliometric review of physical activity research. Translat. J. Am. College Sport. Med. 2021;6(4) [Google Scholar]
- 28.Yoong S.L., Jackson J., Barnes C., Pearson N., Swindle T., O'Reilly S., et al. Changing landscape of nutrition and dietetics research? A bibliographic analysis of top-tier published research in 1998 and 2018. Publ. Health Nutr. 2021;24(6):1318–1327. doi: 10.1017/S1368980021000136. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Waller A., Forshaw K., Bryant J., Carey M., Boyes A., Sanson-Fisher R. Preparatory education for cancer patients undergoing surgery: a systematic review of volume and quality of research output over time. Patient Educ. Counsel. 2015;98(12):1540–1549. doi: 10.1016/j.pec.2015.05.008. [DOI] [PubMed] [Google Scholar]
- 30.Sanson-Fisher R.W., Campbell E.M., Htun A.T., Bailey L.J., Millar C.J. We are what we do: research outputs of public health. Am. J. Prev. Med. 2008;35(4):380–385. doi: 10.1016/j.amepre.2008.06.039. [DOI] [PubMed] [Google Scholar]
- 31.Herrmann A., Mansfield E., Hall A.E., Sanson-Fisher R., Zdenkowski N. Wilfully out of sight? A literature review on the effectiveness of cancer-related decision aids and implementation strategies. BMC Med. Inf. Decis. Making. 2016;16(1):1–9. doi: 10.1186/s12911-016-0273-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Wolfenden L., Milat A.J., Lecathelinais C., Sanson-Fisher R.W., Carey M.L., Bryant J., et al. What is generated and what is used: a description of public health research output and citation. Eur. J. Publ. Health. 2016;26(3):523–525. doi: 10.1093/eurpub/ckw047. [DOI] [PubMed] [Google Scholar]
- 33.SCImago SCImago. J. Country Rank. 2022 https://www.scimagojr.com/journalrank.php?category=2739 Available from: [Google Scholar]
- 34.Scheirer M.A. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am. J. Eval. 2005;26(3):320–347. [Google Scholar]
- 35.Moullin J.C., Sklar M., Green A., Dickson K.S., Stadnick N.A., Reeder K., et al. Advancing the pragmatic measurement of sustainment: a narrative review of measures. Implement. Sci. Commun. 2020;1(1):1–18. doi: 10.1186/s43058-020-00068-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Lewis C.C., Fischer S., Weiner B.J., Stanick C., Kim M., Martinez R.G. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement. Sci. 2015;10(1):1–17. doi: 10.1186/s13012-015-0342-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Mettert K., Lewis C., Dorsey C., Halko H., Weiner B. Measuring implementation outcomes: an updated systematic review of measures' psychometric properties. Implement. Res. Pract. 2020;1 doi: 10.1177/2633489520936644. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Hall A., Shoesmith A., Doherty E., McEvoy B., Mettert K., Lewis C.C., et al. Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review. Implement. Sci. 2022 doi: 10.1186/s13012-022-01252-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.World Health Organisation . 1981. Global Strategy for Health for All by the Year 2000. Geneva. [Google Scholar]
- 40.Wolfenden L., Goldman S., Stacey F.G., Grady A., Kingsland M., Williams C.M., et al. Strategies to improve the implementation of workplace‐based policies or practices targeting tobacco, alcohol, diet, physical activity and obesity. Cochrane Database Syst. Rev. 2018;(11) doi: 10.1002/14651858.CD012439.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.GDB 2015 Tobacco Collaborators Smoking prevalence and attributable disease burden in 195 countries and territories, 1990–2015: a systematic analysis from the Global Burden of Disease Study 2015. Lancet. 2017;389:1885–1906. doi: 10.1016/S0140-6736(17)30819-X. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.World Health Organization . 2018. Global Status Report on Alcohol and Health 2018. Geneva. [Google Scholar]
- 43.World Health Organisation . 2021. WHO Report on the Global Tobacco Epidemic 2021: Addressing New and Emerginig Products. Geneva. [Google Scholar]
- 44.Hodder R., O'Brien K., Lorien S., Wolfenden L., Moore T.H.M., Hall A., et al. Interventions to prevent obesity in school-aged children 6-18 years: an update of a Cochrane systematic review and meta-analysis including studies from 2015−2021. eClinicalMedicine. 2022;54 doi: 10.1016/j.eclinm.2022.101635. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Sutherland R., Brown A., Nathan N., Yoong S., Janssen L., Chooi A., et al. A multicomponent mHealth-based intervention (SWAP IT) to decrease the consumption of discretionary foods packed in school lunchboxes: type I effectiveness-implementation hybrid cluster randomized controlled trial. J. Med. Internet Res. 2021;23(6) doi: 10.2196/25256. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Besiner E., Thomson K., Stonkute D., Mohammad T., Akhter N., Todd A., et al. Which public health interventions are effective in reducing morbidity, mortality and health inequalities from infectious diseases amongst children in low- and middle-income countries (LMICs): an umbrella review. PLoS One. 2021;16(6) doi: 10.1371/journal.pone.0251905. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Wolfenden L., Barnes C., Lane C., McCrabb S., Brown H.M., Gerritsen S., et al. Consolidating evidence on the effectiveness of interventions promoting fruit and vegetable consumption: an umbrella review. Int. J. Behav. Nutr. Phys. Activ. 2021;18(11) doi: 10.1186/s12966-020-01046-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Nathan N., Hall A., McCarthy N., Sutherland R., Wiggers J., Bauman A.E., et al. Multi-strategy intervention increases school implementation and maintenance of a mandatory physical activity policy: outcomes of a cluster randomised controlled trial. Br. J. Sports Med. 2022;56(7):385–393. doi: 10.1136/bjsports-2020-103764. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Wolfenden L., Barnes C., Jones J., Finch M., Wyse R.J., Kingsland M., et al. Strategies to improve the implementation of healthy eating, physical activity and obesity prevention policies, practices or programmes within childcare services. Cochrane Database Syst. Rev. 2020;2020(2) doi: 10.1002/14651858.CD011779.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Wolfenden L., McCrabb S., Barnes C., O'Brien K., Ng K.W., Nathan N., et al. Strategies for enhancing the implementation of school‐based policies or practices targeting diet, physical activity, obesity, tobacco or alcohol use. Cochrane Database Syst. Rev. 2022;2022(8) doi: 10.1002/14651858.CD011677.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.American Journal of Preventive Medicine. Elsevier Inc; 2022. https://www.ajpmonline.org/ Available from: [Google Scholar]
- 52.American Journal of Public Health. American Public Health Association; Washington, DC: 2022. https://ajph.aphapublications.org/page/ajph/about.html [Available from: [Google Scholar]
- 53.Annual Review of Public Health Annual Reviews. 2022. https://www.annualreviews.org/journal/publhealth Available from: [Google Scholar]
- 54.BMC public health springer nature. 2022. https://bmcpublichealth.biomedcentral.com/ Available from:
- 55.European Journal of Public Health Oxford University Press; 2022. https://academic.oup.com/eurpub Available from: [Google Scholar]
- 56.Implementation science springer nature. 2022. https://implementationscience.biomedcentral.com/ Available from:
- 57.Int. J. Pub. Health Front. 2022 https://www.ssph-journal.org/journals/international-journal-of-public-health/about Available from: [Google Scholar]
- 58.The Lancet Public Health. Elsevier Inc; 2022. https://www.thelancet.com/lanpub/about [Available from: [Google Scholar]
- 59.Prevention science springer nature. 2022. https://www.springer.com/journal/11121 Available from:
- 60.Preventive medicine Elsevier inc. 2022. https://www.sciencedirect.com/journal/preventive-medicine Available from:
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data and materials relating to this review are available from the corresponding author on reasonable request.