Skip to main content
PMC home page
HRB Open Research logoLink to HRB Open Research
. 2022 Jan 17;3:62. Originally published 2020 Sep 4. [Version 4] doi: 10.12688/hrbopenres.13089.4

Policy Evaluation Network (PEN): Protocol for systematic literature reviews examining the evidence for impact of policies on physical activity across seven different policy domains

Kevin Volf 1,2, Liam Kelly 1,2, Enrique García Bengoechea 1,2, Bláthín Casey 1,2, Anna Gobis 3, Jeroen Lakerveld 4, Joanna Zukowska 3, Peter Gelius 5, Sven Messing 5, Sarah Forberger 6, Catherine Woods 1,2,a; Policy Evaluation Network (PEN) Consortium
PMCID: PMC8567685  PMID: 34805740

Version Changes

Revised. Amendments from Version 3

A sentence has been added to the "Study selection and data extraction" specifying that the tasks of title and abstract screening and full text screening will be carried out by the same reviewers. Description of the details of the data extraction and synthesis tables has been added to the subsection entitled ‘Strategy for data synthesis’ The conclusion has been edited to make reference to study aims, methods and envisioned results. Some additions have been made to the "Urban Design", "Community Programmes" and "Primary Health Care Systems" search strings.

Abstract

Introduction: Over 40 million deaths annually are due to noncommunicable diseases, 15 million of these are premature deaths and physical inactivity contributes an estimated 9% to this figure. Global responses have included the Sustainable Development Goals (SDGs) and the Global Action Plan on Physical Activity (GAPPA). Both point to policy action on physical activity (PA) to address change, yet the impact of policy on PA outcomes is unknown.  The protocol described outlines the methodology for systematic literature reviews that will be undertaken by the Policy Evaluation Network (PEN) to address this knowledge gap.

Methods: The seven best investments for promotion of population PA identified in the Toronto Charter highlighted seven policy domains (schools, transport, urban design, primary health care systems, public education, community-wide programmes and sport) which will form the basis of these PEN reviews. Seven individual scientific literature searches across six electronic databases will be conducted. Each will use the key concepts of policy, PA, evaluation and a distinct concept for each of the seven policy domains. This will be supplemented with a search of the reference list of included articles. Methodological quality will be assessed and overall effectiveness for each included study will be described according to pre-determined criteria.

Conclusions: Each review will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined impact on PA directly or indirectly. By collating the evidence, and demonstrating the depth of the science base which informs these policy recommendations, each review will provide guidance to policymakers to use evidence-based or evidence-informed policies to achieve the 15% relative reduction in physical inactivity as defined by GAPPA.

Registration:  PROSPERO CRD42020156630 (10/07/2020).

Keywords: physical activity, policy, protocol, systematic review, evaluation

Introduction

Physical activity (PA) is defined as “any bodily movement produced by skeletal muscles that requires energy expenditure” ( Caspersen et al., 1985). The relationship between PA levels and health outcomes is well established ( Rutten et al., 2016). Insufficient PA has been identified by the World Health Organisation (WHO) as the fourth leading risk factor for mortality worldwide ( WHO, 2009) and in 2012 it was estimated that 9% (range 5.1 – 12.5%) of global premature mortality can be attributed to physical inactivity ( Lee et al., 2012). The European region has been strongly affected by the costs of inactivity, absorbing 16.9% of the disability that inactivity causes, through its contribution to morbidity from coronary heart disease (CHD), cancer, stroke and diabetes, and 21.8% of the healthcare cost ( Ding et al., 2016).

This epidemiological evidence reveals inactivity to be a substantial public health issue and advocacy by public health specialists and the academic community has demanded policy responses to this issue. For the purposes of this document, policy should be understood as “decisions, plans, and actions that are implemented by national or regional governments to achieve specific health promotion goals within a society” ( Lakerveld et al., 2020). As indicated by the WHO ( WHO Regional Office for Europe, 2010), policy can give support, coherence and visibility at the political level, while making it possible for the organisations involved at national, regional, and local levels – e.g., national government sectors, regional or local authorities, stakeholders, and the private sector – to be logical and consistent in their actions to achieve a shared goal. This applies to food and PA environments, systems and behaviours ( WHO Regional Office for Europe, 2010). In order to reflect the complexity of the policies that may affect the PA policy environment, Lakerveld & colleagues (2020) distinguish between “direct” policies, which refers to policies where improving the PA environment and increasing participation is the primary aim, and “indirect” policies, where the primary aim of the policy is not to increase PA but this may occur as a co-benefit of successful implementation.

The International Society for Physical Activity and Health (ISPAH) was established in 2009 ( Kohl et al., 2012) At its third biennial congress ISPAH promulgated the Toronto Charter calling for political commitment to achieving greater opportunities for PA ( Bull et al., 2010). To guide action on this issue the Charter was subsequently accompanied by a document titled Non-Communicable Disease Prevention: Investments that Work for Physical Activity ( ISPAH, 2012). This document declared seven domains which evidence suggested could be effectively targeted to increase PA opportunities. These were whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes and sport systems and programmes that promote ‘sport for all’. These seven domains provide a policy setting structure for systematic literature review search.

Over recent years there has been an acceleration in the production of policy responses to the epidemics of inactivity and sedentary behaviour ( Klepac Pogrmilovic et al., 2018). The Global Observatory for Physical Activity (GoPA) reports that by 2013, 139 countries were members of its PA advocacy alliance and 26.6% of these countries had already published a stand-alone PA plan ( Ramirez Varela et al., 2016). Furthermore, in 2013, the WHO published a document which recognised PA as a part of the global effort to combat non-communicable diseases (NCDs) ( WHO, 2013). Another development promoted by the WHO in the field of PA policy is the proliferation of audits of policy responses to inactivity using a tool entitled the Health-Enhancing Physical Activity Policy Audit Tool (HEPA PAT).

A significant development occurred in 2017 when, in response to demands for direction on the problem of physical inactivity, the WHO committed to publishing a stand-alone action plan on this issue. This commitment was realised in 2018 when the WHO published the Global Action Plan on Physical Activity (GAPPA), which targeted an even more ambitious PA target than the previous NCD plan ( WHO, 2018).

The recent rise in the number of national PA policies allows research into the question of which of these policies are effective in improving PA outcomes. A scoping review published in 2016 provided evidence that research into policy effectiveness lagged behind research that links PA to health and research that links PA interventions to behaviour ( Rutten et al., 2016). However, with the increase in the number of PA policies there may have been a concomitant rise in research examining the effectiveness of these policies. Furthermore, to the best available knowledge, no project has linked existing policy statements with research that corroborates or discredits the effectiveness of these statements.

As part of the Joint Programming Initiative “A Healthy Diet for a Healthy Life” (JPI HDHL), researchers from 28 institutes in seven European countries (France, Germany, Ireland, Italy, Norway, Poland, and the Netherlands) and New Zealand combine their expertise to form a Policy Evaluation Network (PEN) ( Lakerveld et al., 2020; see https://www.jpi-pen.eu/). PEN’s vision is to provide Europe with tools to identify, evaluate and benchmark policies designed to directly or indirectly address unhealthy lifestyle behaviours which contribute to overweight and obesity, while accounting for existing health inequities. Using structured evaluation principles and methods, PEN will examine the content, implementation and impact of lifestyle policies across Europe and will build on existing knowledge. PEN will provide an overview of the ‘best’ public policies most likely to sustainably support more favourable health behaviours.

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2012). These policy domains are whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes involving multiple settings and sport systems and programmes that promote ‘sport for all’. These reviews will provide evidence supporting the development of a tool named the Physical Activity Environment Policy Index (PA EPI), based on similar principles to an existing tool called the Food Environment Policy Index (Food EPI) ( Swinburn et al., 2013). The PA EPI will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined improve PA outcomes across domains. The aim of each PEN review is to evaluate the status of the evidence base for the impact of policy on PA outcomes across the different policy domains identified in the “seven best investments”.

Method

Original material examining the evidence of what works in terms of direct and indirect policies to increase PA will be identified in the following ways:

  • (1)

    A search, with no date restrictions, of the following electronic databases: four specialized sport science or biomedical databases, MEDLINE (Ebsco), SportDiscus, Cinahl, and Cochrane library, and two general social science databases, Web of Science and Scopus. Search results will be limited to articles that are identified through searching the titles and abstracts.

  • (2)

    Manual reference checks of identified original studies.

  • (3)

    Publicly available English-language resources and documents of major national and international stakeholders will be searched to identify existing reviews and position papers discussing the evidence of what works in terms of direct and indirect policies for increasing PA, e.g., the WHOs European database on Nutrition, Obesity and Physical Activity (NOPA), Global Action Plan on Physical Activity (GAPPA), the European Physical Activity Strategy (EPAS) ( WHO Regional Office for Europe, 2015) and the European Physical Activity Guidelines (EPAG) ( European Commission, 2008).

A content analysis was performed on the Toronto Charter complementary document ( ISPAH, 2012). These ‘investments’ identified the policy domains or sectors in which policies are made that could directly or indirectly impact on physical activity, i.e., schools, transport, urban design, healthcare, public education, the community and sport. This document was searched for key words to be included in the search syntax. Researchers consulted with librarians and other research staff for suggestions on search terms.

The search of electronic databases will comprise seven individual searches (corresponding to the seven best investments), each one to be run on each of the databases. The seven searches will be formed by combining the same basic search strategy (i.e. general eligibility criteria) with seven distinct search concepts (i.e. specific eligibility criteria for each domain). The basic search strategy will consist of three search concepts ( Table 1): search concept one (C1), which will combine synonyms for the keyword “policy” with the Boolean Operator “OR”; search concept two (C2), which will do the same with the keyword “physical activity”; and search concept three (C3), which will do the same for the keyword “impact”. The three search terms will be combined with the Boolean operator “AND” ( Table 1).

Table 1. General Search terms.

Keyword Synonyms
“Policy” (MH "Policy") OR (MH "Public Policy") OR (MH "Policy Making") OR (“policy”) OR (“policies”) OR (“national policy”)
OR (“national framework”) OR (“policy framework”) OR (“policy action”) OR (“legislation”) OR (“strategy”) OR (“policy
making”)
“Physical
Activity” 		(MH "Exercise") OR (MH “Sedentary Behavior”) OR (“physical activit*”) OR (“physical inactivity”) OR (“play”)OR
(“physical education”) OR (“sedentar*”) OR (“sitting”) OR (“healthy lifestyle”) OR (“health initiative”)
“Impact” (“evaluat*”) OR (“impact”) OR (“appraisal”) OR (“effect*”) OR (“assessment”)
Open in a new tab

Abbreviations: 'MH' = MeSH Heading.

Each of the seven searches will further be combined with a specific search term constructed to reflect only one of the seven best investments declared in the document Non-Communicable Disease Prevention: Investments that Work for Physical Activity ( ISPAH, 2012) ( Table 2). It is proposed that individual systematic literature reviews will be performed for each of the seven best investment domains, with an initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains in the first instance.

Table 2. Specific Search terms based on each of the seven best investments document ( ISPAH, 2012).

1. “Whole of School
Approach” 		“Whole-of-school” OR “Whole School” OR “Whole of School” OR WSCC OR “school intervention” OR “school based
intervention” OR “school initiative” OR “school based initiative” OR “school program*” OR “School health” OR
“Wellness” OR “well-being”
2. “Transport
Policy” 		“active transport*” OR “walk*” OR “cyclist*” OR “bik*” OR “bicycl*” OR “cyclist” OR “cycling” OR "active travel*"
OR “commute*” OR "transport mode" OR "transportation mode" OR "travel mode" OR “pedestrian*” OR "traffic
volume" OR "traffic count" OR “transport plan*” OR “road safety” OR “public transport” OR “transport systems”
3. “Urban Design” MH “Environment Design” OR MH “Environment” OR MH “Environment and Public Health” OR “urban design”
OR “urban environment” OR “built environment” OR “mixed-use development” OR “footpaths” OR “bikeways”
OR “street network*” OR “green spac*” OR “green areas” OR “green network” OR “blue spac*” OR “recreational
spac*” OR “urban plan*” OR “public amenit*” OR “network infrastructure”
4. “Primary Health
care systems” 		“primary health” OR “Primary care” OR “health care” OR “health system”
5. “Public
Education” 		“public education” OR “mass media” OR “mass communication” OR “social marketing” OR broadcast* OR MH
“Communications Media” OR MH “Social Media” OR “media” OR “health campaigns” OR “public education”
6. “Community
Programmes” 		“Whole-of-community” OR “Community-wide programs” “community building” OR “community strengthening”
OR "community development" OR "community empowerment" OR "community network*" OR "coalition building"
OR "community capacit*" OR "community”
7. “Sport
Programmes” 		“health promoting clubs” OR “sport*” OR “athletics”
Open in a new tab

Abbreviations: 'MH' = MeSH Heading.

The following criteria will be applied for searches in databases: language will be limited to English language only.

Eligibility criteria

In order to answer our research question some eligibility criteria were developed to screen out irrelevant documents. Studies will be included based on the following criteria for 1) type of study, 2) participants/population, 3) exposure/intervention, and 4) outcomes.

General eligibility criteria were formulated as well as “specific” eligibility criteria for each of the seven searches. Publications that do not meet the “general” eligibility criteria will be excluded from review. Publications that do not meet the “specific” eligibility criteria will be set aside and possibly reassigned to a different search category if they are not duplicates of any publication already included in that search category.

Types of study to be included/excluded. No limitations regarding study type will be placed as long as the study design allows the research questions to be addressed. In addition, reviews using a comprehensive search strategy (including systematic, scoping and realist reviews) and analysing original research on the evidence of what works, in terms of direct and indirect policies for increasing PA; and reviews and policy analysis documents issued by major national and international organisations addressing recommendations referring to the same evidence will be eligible for inclusion. Studies will be excluded based on the following criteria: a direct or indirect form of policy intervention is not identifiable; no information is provided regarding the effects of the policy under consideration on the desired outcomes.

Condition or domain being studied. Reviews examining the evidence of what works in terms of direct and indirect policies on PA.

Participants/population. Eligibility criteria relating to population characteristics are described in Table 3.

Table 3. Population related inclusion criteria.

General
criteria		 School
specific
criteria		 Transport
specific
criteria		 Urban
design
specific
criteria		 Primary
health care
specific
criteria		 Public
education
specific
criteria		 Community
programmes
specific
criteria		 Sport
programmes
specific
criteria
The study
intervention
targets the
general
human
population
or parts of
it that are
relevant for
the
respective
review		 The study
intervention
targets
students
and staff
in the
school
setting. 		The study
intervention
targets the
commuters
and
their preferred
mode of
transport. 		The study
intervention
targets the
residents
of urban
areas 		The study
intervention
targets patients
or primary care
professionals 		The study
intervention
targets the general
population
through public
outreach and mass
communication.		 The study
intervention
targets the
general
population
in the
community
setting.		 The study
intervention
targets
the general
population
in
sport
settings.
Open in a new tab

Exposure(s), intervention(s). Policies that aim to have a direct or indirect effect on PA behaviour of target groups and populations and on the PA environment that support the behaviour under consideration.

Grey literature/Other: Similar to the empirical studies, included grey literature will need to make reference to the impact of PA policy in the relevant domain.

Context. These systematic reviews are performed as a task of PEN. PEN’s vision is to provide Europe with tools to identify, evaluate and benchmark policies designed to directly or indirectly address physical inactivity. Further information on PEN is available at www.jpi-pen.eu or Lakerveld & colleagues (2020)

Main outcome(s). All study designs (e.g., reviews, empirical evidence) and grey literature/other must include the following outcome(s); a changes in PA (or proxy, e.g. fitness), assessed by means of self-report or wearable devices (e.g., accelerometer); a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention.

Study selection and data extraction

Download of title and abstract records. Titles and abstracts identified by the search will be downloaded as “Endnote import” (extension.enw) files or other file formats compatible with our software. They will be uploaded to Endnote X9, a citation management software, and Rayyan ( Ouzzani et al., 2016), an online software dedicated to managing reviews. Other freely available alternative software includes Mendeley reference manager or Zotero. Once records have been uploaded to Rayyan, the software will identify duplicate articles and one of the two identical articles will be removed. The remaining articles will undergo the first round of screening by two researchers in a shared Rayyan account.

Title and abstract review. Title and abstract reviews will be performed by at least one reviewer and checked by another reviewer. Checking will involve reviewing title and abstracts decisions to establish whether the second reviewer concurs with the screening decision. Discrepancies will be resolved by discussion to reach consensus, in consultation with a third researcher when necessary. The screening process will involve comparing the information presented in the title and abstract to the eligibility criteria. Titles and abstracts that appear to conform to the eligibility criteria will be deemed eligible for full text review while those that do not will be discarded from the next stage of the data extraction process.

Download of full articles. Full text articles will be downloaded using the resources provided by their Institution. If a full text record cannot be acquired using these resources, researchers will investigate whether they can be located through use of other libraries to which the research team has access. If a full text article cannot be located through any of these library resources, the authors will be contacted through whichever channels can be identified from the information in the title and abstract.

Full text review. Full text reviews will be performed by at least one reviewer and checked by a second reviewer. The authors involved in screening and risk of bias will differ between the different reviews; however, the authors involved in full text screening will be the same authors with responsibility for title and abstract screening. Discrepancies will be resolved by discussion to reach consensus, in consultation with a third researcher when necessary. The following information will be extracted: first author, year of publication, country, study design, data collection method, sample, recruitment/setting, sample size, and response rate.

Risk of bias (quality) assessment

Risk of bias will be assessed by at least one reviewer and checked by another reviewer. Discrepancies will be resolved by discussion to reach consensus, in consultation with a third researcher when necessary. The results of the quality assessment will be narratively incorporated into the synthesis process. A descriptive summary using the criteria described below will be presented at study level and discussed in the review. Furthermore, the methodological quality will be narratively summarized at review level.

The quality of the included quantitative studies, inclusive of randomised, non-randomised and observational studies (encompassing both longitudinal and cross-sectional studies) will be assessed by means of an adapted ‘Downs and Black’ checklist tool ( Downs & Black, 1998). This tool is apt to assess common biases in a range of study types as noted. The checklist will be modified to meet the aims of this review with some items deemed non applicable and subsequently removed.

The AMSTAR tool will be used for the assessment of systematic reviews and comprehensive reviews with a rigorous search strategy including reviews of reviews. This tool consists of 11 items and has good face and content validity for appraising the methodological quality of systematic reviews ( Shea et al., 2007). Not all items are applicable to every type of review being assessed and quality ratings will take account of this circumstance. Similar to Messing et al. (2019), to assess the quality of included studies, we will calculate percentage values for each study. Each study will be assessed by a tool appropriate for its study design and these percentage values will be calculated based on the percentages of criteria met by a study, which will be particular to the tool used to assess it.

Strategy for data synthesis

A narrative synthesis will be used to interpret and analyse the data. The results of the data synthesis will be presented in a table. Within this table, a list of short descriptive statements will be compiled based on the policy actions identified in the scientific literature. Evidence on the effectiveness of these policy actions will then be described using a method used by Panter & colleagues (2019). Specifically, when a study presents quantitative evidence about the effectiveness of one policy action a symbol will be assigned next to that particular policy action in the table. There are four categories of symbols reflecting the four possible outcomes: “significantly positive evidence” (+), “significantly negative evidence” (-), “no significance test” (?) or “inconclusive” (0). In summary, the data synthesis table will display three types of data: brief policy statements, codes which help visualise the number and direction of effects found in the literature supporting those statements, and the supporting references. In addition, data extraction tables will be designed to distinguish any demographic, environmental or other variables pertinent to synthesising the data. For example, in the schools’ review, data extraction columns will be included to reflect evidence of effectiveness stratified by gender, school level (primary, secondary, combined) or socio-economic status where appropriate.

Finally, for the included reviews and policy analysis documents, the main findings stated in the discussion and conclusions section will be extracted. Main findings of the articles will be copied into a single table along with a reference to the article itself, and details of the overall risk of bias of the study from which the information is extracted. The synthesised data will be presented in a six-column table with the different columns presenting information on the reference, study description, study type, main findings or outcomes, risk of bias and category of evidence, respectively.

Dissemination

Study findings will be presented at professional networking events such as the World Congress on Public Health. Manuscripts will be prepared for publication in scientific peer-reviewed journals and presented at academic conferences.

Study status

The submission of the first of seven intended reviews is being finalised, this focuses on the school setting A further three reviews are underway, these will focus on transport, public education and sport policy domains.

Conclusion

An aim of this project is to assist policymakers to achieve the GAPPA target of a 15% relative reduction in the prevalence of insufficient PA ( WHO, 2018). The aim of the planned work is to determine the level and type of evidence reported in the international scientific literature for policies that contribute directly or indirectly to increasing PA within 7 priority domains identified as best investments for PA ( ISPAH, 2012). This will be achieved through searches electronic databases and extensive snowballing techniques. By providing this evidence, these reviews will support the development of the PA-EPI. The PA-EPI in turn will support policy makers by facilitating the benchmarking of policies which work towards achieving this target. Achieving this target will provide health benefits such as reduced premature mortality as well as substantial co-benefits such as contributing to a sustainable environment and quality education ( WHO, 2018). We anticipate that the recommendations will mirror and expand upon some of the prescriptions made in GAPPA, the European Physical Activity Strategy or by experts in the PA community

Data availability

Underlying data

No underlying data are associated with this article.

Reporting guidelines

Open Science Framework: PRISMA-P checklist for “Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity”. https://doi.org/10.17605/OSF.IO/26QYF ( Volf, 2020).

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Acknowledgements

The authors thank Dr Sarah Taylor and Michael Lawlor for their assistance in developing the search terms used in the study. The PEN project is funded by the Joint Programming Initiative (JPI) “A Healthy Diet for a Healthy Life”, a research and innovation initiative of EU member states and associated countries. The funding agencies supporting this work are (in alphabetical order of participating countries): Germany: Federal Ministry of Education and Research (BMBF); Ireland: Health Research Board (HRB); Italy: Ministry of Education, University and Research (MIUR); The Netherlands: The Netherlands Organisation for Health Research and Development (ZonMw); New Zealand: The University of Auckland, School of Population Health; Norway: The Research Council of Norway (RCN); Poland: The National Centre for Research and Development (NCBR). Additionally, the French partners acknowledge the support through the Institute National de la Recherche Agronomique (INRA). A full list of PEN consortium members is available at www.jpi-pen.eu.

Funding Statement

Health Research Board [JPI-HDHL-PEN-2017-1]. The PEN project is funded by the Joint Programming Initiative (JPI) “A Healthy Diet for a Healthy Life”, a research and innovation initiative of EU member states and associated countries. The funding agencies supporting this work are (in alphabetical order of participating countries): Germany: Federal Ministry of Education and Research (BMBF); Ireland: Health Research Board (HRB); Italy: Ministry of Education, University and Research (MIUR); The Netherlands: The Netherlands Organisation for Health Research and Development (ZonMw); New Zealand: The University of Auckland, School of Population Health; Norway: The Research Council of Norway (RCN); Poland: The National Centre for Research and Development (NCBR). Additionally, the French partners acknowledge the support through the Institute National de la Recherche Agronomique (INRA).

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 4; peer review: 3 approved]

References

  1. Bull FC, Gauvin L, Bauman A, et al. : The Toronto Charter for Physical Activity: A Global Call for Action. J Phys Act Health. Toronto, Canada,2010;7(4):421–422. 10.1123/jpah.7.4.421 [DOI] [PubMed] [Google Scholar]
  2. Caspersen CJ, Powell KE, Christenson GM: Physical activity, exercise, and physical fitness: definitions and distinctions for health-related research. Public Health Rep. 1985;100(2):126–31. [PMC free article] [PubMed] [Google Scholar]
  3. Ding D, Lawson KD, Kolbe-Alexander TL, et al. : The economic burden of physical inactivity: a global analysis of major non-communicable diseases. Lancet. 2016;388(10051):1311–1324. 10.1016/S0140-6736(16)30383-X [DOI] [PubMed] [Google Scholar]
  4. Downs SH, Black N: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of healthcare interventions. J Epidemiol Community Health. 1998;52(6):377–384. 10.1136/jech.52.6.377 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. European Commission: EU Physical Activity Guidelines: Recommended Policy Actions in Support of Health-Enhancing Physical Activity.Brussels: European Commisssion,2008. Reference Source [Google Scholar]
  6. International Society for Physical Activity and Health: Non communicable disease prevention: Investments that Work for Physical Activity. Br J Sports Med. 2012;46(10):709–712. 10.1136/bjsm.2012.091485 [DOI] [PubMed] [Google Scholar]
  7. Klepac Pogrmilovic B, O'Sullivan G, Milton K, et al. : A global systematic scoping review of studies analysing indicators, development, and content of national-level physical activity and sedentary behaviour policies. Int J Behav Nutr Phys Act. 2018;15(1):123. 10.1186/s12966-018-0742-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Kohl HW, 3rd, Craig CL, Lambert EV, et al. : The pandemic of physical inactivity: global action for public health. Lancet. 2012;380(9838):294–305. 10.1016/S0140-6736(12)60898-8 [DOI] [PubMed] [Google Scholar]
  9. Lakerveld J, Woods C, Hebestreit A, et al. : Advancing the evidence base for public policies impacting on dietary behaviour, physical activity and sedentary behaviour in Europe: The Policy Evaluation Network promoting a multidisciplinary approach. Food Policy. 2020;96:101873. 10.1016/j.foodpol.2020.101873 [DOI] [Google Scholar]
  10. Lee IM, Shiroma EJ, Lobelo F, et al. : Effect of physical inactivity on major non-communicable diseases worldwide: an analysis of burden of disease and life expectancy. Lancet. 2012;380(9838):219–229. 10.1016/S0140-6736(12)61031-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Messing S, Rütten A, Abu-Omar K, et al. : How Can Physical Activity Be Promoted Among Children and Adolescents? A Systematic Review of Reviews Across Settings. Front Public Health. 2019;7:55. 10.3389/fpubh.2019.00055 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Ouzzani M, Hammady H, Fedorowicz Z, et al. : Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210. 10.1186/s13643-016-0384-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Panter J, Guell C, Humphreys D, et al. : Can changing the physical environment promote walking and cycling? A systematic review of what works and how. Health Place. 2019;58:102161. 10.1016/j.healthplace.2019.102161 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Ramirez Varela A, Pratt M, Borges C, et al. : 1st Physical Activity Almanac: the Global Observatory for Physical Activity-GoPA. The Global Observatory for Physical Activity. 2016. Reference Source [Google Scholar]
  15. Rütten A, Schow D, Breda J, et al. : Three types of scientific evidence to inform physical activity policy: results from a comparative scoping review. Int J Public Health. 2016;61(5):553–63. 10.1007/s00038-016-0807-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Shea BJ, Grimshaw JM, Wells GA, et al. : Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. 10.1186/1471-2288-7-10 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Swinburn B, Sacks G, Vandevijvere S, et al. : INFORMAS (International Network for Food and Obesity/non-communicable diseases Research, Monitoring and Action Support): overview and key principles. Obes Rev. 2013;14 Suppl 1:1–12. 10.1111/obr.12087 [DOI] [PubMed] [Google Scholar]
  18. Volf K: Checklist for protocol for systematic literature review examining the evidence for impact of school policies on physical activity.2020. 10.17605/OSF.IO/26QYF [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. World Health Organisation: Global Health Risks: Mortality and burden of disease attributable to selected major risks.Geneva: World Health Organisation.2009. Reference Source [Google Scholar]
  20. World Health Organization: Global action plan for the prevention and control of noncommunicable diseases 2013-2020.2013. Reference Source [Google Scholar]
  21. World Health Organization: Global action plan on physical activity 2018-2030: more active people for a healthier world.World Health Organization.2018. Reference Source [Google Scholar]
  22. World Health Organization Regional Office for Europe: Review of physical activity promotion policy development and legislation in European Union Member States.Copenhagen: WHO Regional Office for Europe.2010. Reference Source [Google Scholar]
  23. World Health Organisation Regional Office for Europe: Physical Activity Strategy for the WHO European Region 2016 - 2025.Vilnius: World Health Organisation,2015. Reference Source [Google Scholar]
HRB Open Res. 2022 Jan 26. doi: 10.21956/hrbopenres.14693.r31259

Reviewer response for version 4

Eric Breton 1,2

I am satisfied with the answers given to the points I raised in my first report.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Community health, prevention strategies, health promotion, capacity building strategies for the prevention of chronic diseases.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

HRB Open Res. 2021 Nov 17. doi: 10.21956/hrbopenres.14427.r30448

Reviewer response for version 3

Eric Breton 1,2

Dear authors,

Informing policy-makers on the best options available to improve the level of physical activity across whole populations is definitely the next step after years of research looking at the determinants of physical activity. This project is therefore likely to generate valuable evidence that will not only pinpoint the best policies but also facilitate advocacy for policy change.

I was surprised not seeing any reference to the Physical Activity Policy Analysis Tool (HEPA PAT) that allows for a diagnosis of national policy responses to physical inactivity. I think it should be mentioned as part of the current efforts to improve policies.

The general methods is appropriate but is likely going to be adapted as work on the 7 domains will be carried out. Other search and inclusion/exclusion criteria are likely to be stated when facing the large body of papers published in some of the domains.

In the following lines, I address some of the specific issues I identified.

  • Second paragraph of the introduction: the last sentence seems at odd with the topic covered by the paragraph (“The International Society for Physical Activity and Health (ISPAH) was established in 2009 (Kohl et al., 2012) and numerous articles and editorials in leading academic and medical journals have pointed out the need to address physical inactivity (Bull & Bauman, 2011; Das & Horton, 2012; Kohl et al., 2012; Woods & Mutrie, 2012).”) - It is not fitting well there.

  • In the fourth paragraph, you wrote: “Over recent years there has been an acceleration in the production of policy responses to the epidemics of physical activity and sedentary behaviour” - Seems odd talking about the epidemics of physical activity.

  • In table 1 on the search terms for policy: there is a fair bit of overlap. Searching “policy” as a single term will automatically yield hits that include “national policy”. Therefore no need to enter the latter. I assume the terms that will not be searched in the MH will be searched in the title/abstract as mentioned in the text. In the “impact” search string, shouldn’t “outcome” be included?

  • In Table 2: in the “transport policy” search string, “cyclist” appears twice and “Commute*” should be entered as “commut*”.

  • In the “urban design” one, “urban environment” appears twice as well. If I am not wrong in the literature, there are mentions along the green spaces to the blue spaces. Lakes and waterways can be settings for physical activity.

  • For the “Primary Health care systems” search string, I would add “primary care”.

  • For the “Community Programmes” string, I do think the term “capacity building” will generate many hits that are not relevant. I would remove it and add “community building” and “community strengthening”.

  • In table 3, under the “Urban design specific criteria” we read that: “The study intervention targets the residents of urban areas”. I did not understand that this idea of using the leverage of actions on urban design was only for urban areas. If that is the case, it would definitely leave behind a significant share of populations that are known to present a poorer health status.

This is all I can see reading this paper.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Community health, prevention strategies, health promotion, capacity building strategies for the prevention of chronic diseases.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

HRB Open Res. 2021 Dec 7.
Kevin Volf 1

Dear Dr Breton,

Thank you for your feedback on our paper. By our reading, you have highlighted 9 different points for improvement in the paper. We quote each point and provide our response below.

  • “I was surprised not seeing any reference to the Physical Activity Policy Analysis Tool (HEPA PAT) that allows for a diagnosis of national policy responses to physical inactivity. I think it should be mentioned as part of the current efforts to improve policies.”

The following sentence has been added to the introduction (lines 45 to 47) to add reference to the HEPA PAT: “Another development promoted by the WHO in the field of PA policy is the proliferation of audits of policy responses to inactivity using a tool entitled the Health-Enhancing Physical Activity Policy Audit Tool (HEPA PAT).”

  • “Second paragraph of the introduction: the last sentence seems at odd with the topic covered by the paragraph (“The International Society for Physical Activity and Health (ISPAH) was established in 2009 (Kohl et al., 2012) and numerous articles and editorials in leading academic and medical journals have pointed out the need to address physical inactivity (Bull & Bauman, 2011; Das & Horton, 2012; Kohl et al., 2012; Woods & Mutrie, 2012).”) - It is not fitting well there.”

We agree that this seems out of place. We have moved the first clause and deleted the second, this influences the reference list which we have also updated accordingly. Paragraph two (lines 12 to 27) now reads: “This epidemiological evidence reveals inactivity to be a substantial public health issue and advocacy by public health specialists and the academic community has demanded policy responses to this issue. For the purposes of this document, policy should be understood as “decisions, plans, and actions that are implemented by national or regional governments to achieve specific health promotion goals within a society” ( Lakerveld et al., 2020). As indicated by the WHO ( WHO Regional Office for Europe, 2010), policy can give support, coherence and visibility at the political level, while making it possible for the organisations involved at national, regional, and local levels – e.g., national government sectors, regional or local authorities, stakeholders, and the private sector – to be logical and consistent in their actions to achieve a shared goal. This applies to food and PA environments, systems and behaviours ( WHO Regional Office for Europe, 2010). In order to reflect the complexity of the policies that may affect the PA policy environment, Lakerveld & colleagues (2020) distinguish between “direct” policies, which refers to policies where improving the PA environment and increasing participation is the primary aim, and “indirect” policies, where the primary aim of the policy is not to increase PA but this may occur as a co-benefit of successful implementation.”

  • “In the fourth paragraph, you wrote: “Over recent years there has been an acceleration in the production of policy responses to the epidemics of physical activity and sedentary behaviour” - Seems odd talking about the epidemics of physical activity.”

We have amended this sentence to read "...epidemics of inactivity and sedentary behaviour” [line 40].

  • “In table 1 on the search terms for policy: there is a fair bit of overlap. Searching “policy” as a single term will automatically yield hits that include “national policy”. Therefore, no need to enter the latter. I assume the terms that will not be searched in the MH will be searched in the title/abstract as mentioned in the text. In the “impact” search string, shouldn’t “outcome” be included?”

The assumption that terms that are not searched in the MH will be searched by title and abstract is correct. As some of the reviews have been commenced and submitted, we cannot amend the general search string. However, other suggestions made below have been incorporated.

  • “In Table 2: in the “transport policy” search string, “cyclist” appears twice and “Commute*” should be entered as “commut*”.”

Since the transport review has been completed and is currently under review by an academic journal, the transport policy search string cannot be revised.

  • “In the “urban design” one, “urban environment” appears twice as well. If I am not wrong in the literature, there are mentions along the green spaces to the blue spaces. Lakes and waterways can be settings for physical activity.”

We have amended the search terms so that "Blue spac*" is added, and the second "urban environment" removed in the urban design search string.

  • “For the “Primary Health care systems” search string, I would add “primary care”.”

We have added the search term "primary care *" to the primary health care search string.

  • “For the “Community Programmes” string, I do think the term “capacity building” will generate many hits that are not relevant. I would remove it and add “community building” and “community strengthening”.”

We have amended the search terms so that “community building” and “community strengthening” are added, and “community strengthening” is removed from the community programmes search string.

  • “In table 3, under the “Urban design specific criteria” we read that: “The study intervention targets the residents of urban areas”. I did not understand that this idea of using the leverage of actions on urban design was only for urban areas. If that is the case, it would definitely leave behind a significant share of populations that are known to present a poorer health status.”

It is correct that the urban design review will not reach rural populations. It is expected that policy actions with evidence for effectiveness will include actions such as mixed urban land use. However, rural communities will be reached through the other reviews.

Kind regards,

Kevin Volf

HRB Open Res. 2021 Nov 3. doi: 10.21956/hrbopenres.14427.r30447

Reviewer response for version 3

Petru Sandu 1

This article presents very clearly the process undertaken for conducting 7 SLRs looking at the impact of PA policies in 7 priority settings towards established PA related outcomes. 

The introduction is very clear and comprehensive, gives a good overview of the research in this area and the need for the current project.

The methods are also clear, concise but comprehensive. I only have one question: the two researchers conducting title and abstract, full-text review and risk of bias assessment are the same to researchers of there are 3 sets of 2 researchers? It is not obvious from the text. Also, who is the 3r researcher in each case? I guess details about this will be described in the following papers, but I think something should be also mentioned in this paper. 

The strategy for data synthesis is also clear but I think the specific data included in the tables for each of the 7 reviews should be more clearly defined. Again, probably these will be detailed in future papers. 

The conclusion seems more like a impact paragraph to me. It is necessary and useful, but not as conclusions. Conclusions should refer to the study aim, methods put in place and envisioned results.

Also, some phrasing related feedback:

  1. Over recent years there has been an acceleration in the production of policy responses to the epidemics of physical activity and sedentary behaviour - do you mean physical inactivity and sedentary behaviour?

  2. Instead of:  Original material examining the evidence of what works in terms of direct and indirect policies on PA will be identified in the following ways - Original material examining the evidence of what works in terms of direct and indirect policies to increase PA will be identified in the following ways.

In conclusion, I think this paper is a good description of the methods put in place by PEN network in order to conduct a series of SLRs aimed at exploring the impact of PA policy on selected PA related outcome, with the final aim to assist decision makers in making evidence informed investments.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

Physical activity policy

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

HRB Open Res. 2021 Dec 7.
Kevin Volf 1

Dear Dr Sandu,

Thank you for your feedback on our review protocol. This will contribute significantly to the improvement of this paper. By our reading, you have highlighted 5 different points for improvement in the paper. We quote each point and provide our response below.

  • “The methods are also clear, concise but comprehensive. I only have one question: the two researchers conducting title and abstract, full-text review and risk of bias assessment are the same to researchers of there are 3 sets of 2 researchers? It is not obvious from the text. Also, who is the 3r researcher in each case? I guess details about this will be described in the following papers, but I think something should be also mentioned in this paper.”

As there are several reviews planned, and responsibility for these tasks will differ between reviews, it is not possible to assign authors initials to each of the roles specified in this publication. However, we have added the following to the paragraph entitled ‘full text reviews’ to provide greater clarity: “The authors involved in screening and risk of bias will differ between the different reviews; however, the authors involved in full text screening will be the same authors with responsibility for title and abstract screening” [lines 193 to 195].

  • The strategy for data synthesis is also clear but I think the specific data included in the tables for each of the 7 reviews should be more clearly defined. Again, probably these will be detailed in future papers. “To provide further clarity on how the data synthesis table can be visualised the following was added to the second paragraph of the subsection entitled ‘Strategy for data synthesis’: “In summary, the data synthesis table will display three types of data: brief policy statements, codes which help visualise the number and direction of effects found in the literature supporting those statements, and the supporting references” [lines 230 to 233].

The data extraction tables will display descriptive data on the papers including general data in of the papers (first author, date, country) descriptions of the population (sample size, demographics), description of the policy itself and reported effects.

  • “The conclusion seems more like a impact paragraph to me. It is necessary and useful, but not as conclusions. Conclusions should refer to the study aim, methods put in place and envisioned results.”

To provide reference to the study aim the following sentence has been added: “The aim of the planned work is to determine the level and type of evidence reported in the international scientific literature for policies that contribute directly or indirectly to increasing PA within 7 priority domains identified as best investments for PA” [lines 255 to 258].

To refer to the methods the following line has been added “this will be achieved through searches electronic databases and extensive snowballing techniques” [lines 258 to 259].

To address the lack of envisioned results the following sentence has been added to the conclusion: “We anticipate that the recommendations will mirror and expand upon some of the prescriptions made in GAPPA, the European Physical Activity Strategy or by experts in the PA community.” [lines 264 to 266].

  • “Over recent years there has been an acceleration in the production of policy responses to the epidemics of physical activity and sedentary behaviour - do you mean physical inactivity and sedentary behaviour?”

We have amended this sentence to read "...epidemics of inactivity and sedentary behaviour” [line 40].

  • “Instead of: Original material examining the evidence of what works in terms of direct and indirect policies on PA will be identified in the following ways - Original material examining the evidence of what works in terms of direct and indirect policies to increase PA will be identified in the following ways.”

We have amended this sentence so that "On PA" is replaced with "to increase PA" [line 88].

Kind regards

Kevin Volf

HRB Open Res. 2021 Mar 31. doi: 10.21956/hrbopenres.14427.r29226

Reviewer response for version 3

Melinda Craike 1,2

No new comments.

Is the study design appropriate for the research question?

Partly

Is the rationale for, and objectives of, the study clearly described?

Partly

Are sufficient details of the methods provided to allow replication by others?

Partly

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

Physical activity behaviour change; systematic reviews.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

HRB Open Res. 2020 Dec 17. doi: 10.21956/hrbopenres.14323.r28482

Reviewer response for version 2

Melinda Craike 1,2

I thank the Authors for their thorough responses to my comments and proposed revisions. I am happy with the revisions and have no further comments. I wish the Authors all the best with this series of reviews.

Is the study design appropriate for the research question?

Partly

Is the rationale for, and objectives of, the study clearly described?

Partly

Are sufficient details of the methods provided to allow replication by others?

Partly

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

Physical activity behaviour change; systematic reviews.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

HRB Open Res. 2020 Nov 5. doi: 10.21956/hrbopenres.14190.r28217

Reviewer response for version 1

Melinda Craike 1,2

Thank you for the invitation to review this paper. This protocol paper outlines the details of the first of several planned systematic literature reviews to assess the impact of policy on physical activity outcomes based on the “seven best investments” document (ISPAH, 2012). This is a very ambitious review series and I commend the authors on undertaking this important work. While the methods to be employed is generally clear, there are several areas that require more detailed explanation/rationale. I believe that doing so will strengthen this paper and ultimately, the review.

Introduction

It would be helpful to include specific aims of the systematic review.

P 4- Eligibility criteria

The authors need to clearly define what they mean by a ‘whole of school approach’ what are the critical components or does a policy just need to be labelled as such? For examples, does a ‘whole of school’ approach need to address the elements identified in ISPAH 2012: i.e. “prioritizing: regular, highly-active, physical education classes; providing suitable physical environments and resources to support structured and unstructured physical activity throughout the day (e.g., play and recreation before, during and after school); supporting walk/cycle-to-school programs and enabling all of these actions through supportive school policy and engaging staff, students, parents and the wider community”.

I was confused by the distinction between the “mandatory” eligibility criteria and specific “preferred” eligibility criteria and the rationale for this. I have not seen this before in a review. Can authors more clearly explain the application of these and the rationale? Perhaps by way of examples?

p. 5 Main Outcomes

The Introduction section implies that the focus of the review will be on the effectiveness of policies in increasing PA. This seems to be inconsistent with the Main Outcome(s), identified on p. 8, where studies will be eligible if they include “a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention”. I suggest that the Introduction include a section that alludes to this, and why such outcomes could be considered important (e.g., that focusing on mediators can be used as a proxy for actional PA participation outcomes).

p. 6 Full text review

Given the importance of implementation to the success of policies, I wonder if it would be worthwhile for the authors to attempt to gather data to indicate whether or not the policy was implemented? Or whether, for example, action plans were developed, and resources allocated to implementation. Given the importance of implementation to policy success, I would see this type of information as critical when evaluating the success of such policies.

Also, the inclusion of the validity and reliability of the PA measures should be included in the data extraction.

Each of the policy types is very broad, which will make it difficult to identify policy action that is needed to increase PA. For example, a ‘whole-of-school approach’ can includes several different actions. Have the authors considered providing more detail or categorising the type of whole of school approach to facilitate a clearer and more meaningful assessment about the characteristics of effective whole of school policies?

p. 6. Strategy for data synthesis

I am not clear on what is meant by: “Outcome data extracted will be tabulated to determine the impact on ( my emphasis) policy areas and policy actions (policy indicators)”. What is meant here by ‘policy indicators’ for example?

I suggest that the method described by Panter and colleagues (2019) be described in the paper (rather than just referring the reader to the paper). It is very important that the reader understands the criteria that is being used to assess the evidence of effectiveness.

I am unclear about how the findings from systematic reviews could be combined with primary research studies. This needs to be further explained.

Will there be any attempt to distinguish effectiveness based on the following: school level (primary or secondary school level), gender, level of socio-economic disadvantage?

Is the study design appropriate for the research question?

Partly

Is the rationale for, and objectives of, the study clearly described?

Partly

Are sufficient details of the methods provided to allow replication by others?

Partly

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

Physical activity behaviour change; systematic reviews.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

HRB Open Res. 2020 Nov 18.
Kevin Volf 1

Dear Dr Craike,

On behalf of the review team, I would like to thank you for taking the time for your thorough review of our paper. The team would especially like to thank you for the timely manner in which your review was conducted. We have received your feedback and have made changes to the manuscript accordingly, and believe this has strengthened the quality of the paper.

Please find below a point-by-point response to the issues raised in your review, with changes from the previous version highlighted in red, for your consideration.

Yours sincerely,

Kevin Volf, B.Sc., MM.Sc. PhD Candidate

Comment 1:

Introduction

It would be helpful to include specific aims of the systematic review.

--

We agree with the comment and the aims have added to the final paragraph of the Introduction section. To clarify the focus of this paper is to provide information on the protocol being used for seven reviews, this was unclear in the original manuscript.  The text has been edited to make this point clearer throughout. Please see amendments below, text in bold is ‘new’ and has been added for clarity:

[Introduction page 6, paragraph 2]

At its third biennial congress ISPAH promulgated the Toronto Charter calling for political commitment to achieving greater opportunities for PA (Bull et al., 2010). To guide action on this issue the Charter was subsequently accompanied by a document titled Non-Communicable Disease Prevention: Investments that Work for Physical Activity (2012). This document declared seven domains which evidence suggested could be effectively targeted to increase PA opportunities. These were whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes and sport systems and programmes that promote ‘sport for all’. These seven domains provide a policy setting structure for systematic literature review search.

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy , either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are schools, transport, urban design, healthcare, public education, the community and sport.  These reviews will provide evidence supporting the development of a tool named the Physical Activity Environment Policy Index (PA EPI), based on similar principles to the Food Environment Policy Index (Food EPI) ( Swinburn et al., 2013). The PA EPI will provide policy makers with a list of policy statements and corresponding actions which the evidence has determined improve PA outcomes across domains. The aim of each PEN review is to evaluate the status of the evidence base for the impact of policy on PA outcomes across the different policy domains identified in the “seven best investments”.

Comment 2:

P 4- Eligibility criteria

The authors need to clearly define what they mean by a ‘whole of school approach’ what are the critical components or does a policy just need to be labelled as such? For examples, does a ‘whole of school’ approach need to address the elements identified in ISPAH 2012: i.e. “prioritizing: regular, highly-active, physical education classes; providing suitable physical environments and resources to support structured and unstructured physical activity throughout the day (e.g., play and recreation before, during and after school); supporting walk/cycle-to-school programs and enabling all of these actions through supportive school policy and engaging staff, students, parents and the wider community”.

--

The reference to ‘whole of school approach’ is directly related to the ‘Seven Investments’ paper.  The use of ‘Whole of school’ within the search strategy as part of the schools domain, is not for the purpose of evaluating the impact of ‘whole of school programmes’, rather to review policies pertinent to the school setting, to determine if any evidence of policy impact exists.  The text has been modified as follows, text in bold is ‘new’ and has been added for clarity;

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy, either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011). These policy domains are whole-of-school programmes, transport policies and systems, urban design regulations and infrastructure, primary health care, public education, community-wide programmes involving multiple settings and sport systems and programmes that promote ‘sport for all’

It is proposed that individual systematic literature reviews will be performed on each of the seven best investment domains, with this initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.”

Comment 3:

I was confused by the distinction between the “mandatory” eligibility criteria and “preferred” eligibility criteria and the rationale for this. I have not seen this before in a review. Can authors more clearly explain the application of these and the rationale? Perhaps by way of examples?

--

 In essence, the “mandatory” and “preferred” nomenclature refer to “elements that are common to all reviews” and “elements that only relate to a particular review” respectively. While this was explained in the search details, to make this clearer we have edited the mandatory and preferred labels to “general” (applies to all mooted reviews) and “specific” (only applies to a particular review) as follows, text in bold is ‘new’ and has been added for clarity:

[Methods, p7 paragraph 5]:

“The search of electronic databases will comprise seven individual searches (corresponding to the seven best investments), each one to be run on each of the databases. The seven searches will be formed by combining the same basic search strategy (i.e., general eligibility criteria) with seven distinct search concepts (i.e., specific eligibility criteria for each domain).  The basic search strategy will consist of three search concepts ( Table 1): search concept one (C1), which will combine synonyms for the keyword “policy” with the Boolean Operator “OR”; search concept two (C2), which will do the same with the keyword “physical activity”; and search concept three (C3), which will do the same for the keyword “impact”. The three search terms will be combined with the Boolean operator “AND” ( Table 1).

Each of the seven searches will further be combined with a specific search concept constructed to reflect only one of the seven best investments declared in the document Non-Communicable Disease Prevention: Investments that Work for Physical Activity ( ISPAH, 2011 ) ( Table 2). It is proposed that individual systematic literature reviews will be performed for each of the seven best investment domains, with an initial review focusing on schools and subsequent reviews focusing on ‘transport’, ‘public education’ and ‘sport’ domains.

Comment 4:

p. 5 Main Outcomes

The Introduction section implies that the focus of the review will be on the effectiveness of policies in increasing PA. This seems to be inconsistent with the Main Outcome(s), identified on p. 8, where studies will be eligible if they include “a change in features of the physical and social environment (e.g., facilities, equipment, action plans, programmes) hypothesised to lead to changes in PA outcomes as a result of a policy intervention”. I suggest that the Introduction include a section that alludes to this, and why such outcomes could be considered important (e.g., that focusing on mediators can be used as a proxy for actional PA participation outcomes).

--

We agree with the inconsistency and the text has been reworded in the following location, text in bold is ‘new’ and has been added for clarity:

[Introduction page 7 paragraph 1]

This protocol paper outlines the methodology for seven complementary systematic literature reviews as part of PEN. Each review is designed to determine the impact of policy , either directly or indirectly, on physical activity outcomes across the different policy domains identified in the “Seven Best Investments” ( ISPAH, 2011).

Comment 5a:

p. 6 Full text review

Given the importance of implementation to the success of policies, I wonder if it would be worthwhile for the authors to attempt to gather data to indicate whether or not the policy was implemented? Or whether, for example, action plans were developed, and resources allocated to implementation. Given the importance of implementation to policy success, I would see this type of information as critical when evaluating the success of such policies. […]

--

While we agree that level of implementation of policies is important, and believe that ‘impact on PA’ will be reflected in the degree of implementation presented, it is beyond the scope of this study to carry out a detailed examination of type of implementation beyond what is reported as evidence of impact. This very important topic may indeed by the subject of a follow up review.

Comment 5b:

[…] Also, the inclusion of the validity and reliability of the PA measures should be included in the data extraction.

--

Detailed information on any outcome measures used to assess PA will be provided in the data extraction tables and presented in the corresponding reviews. 

Comment 6:

Each of the policy types is very broad, which will make it difficult to identify policy action that is needed to increase PA. For example, a ‘whole-of-school approach’ can includes several different actions. Have the authors considered providing more detail or categorising the type of whole of school approach to facilitate a clearer and more meaningful assessment about the characteristics of effective whole of school policies?

--

This links back to Comment 2, the focus of the systematic review is not to evaluate ‘whole of school approach’ but rather the available evidence on impact of direct or indirect policies within specific settings, e.g., the school setting, on PA outcomes.  Therefore, the reviewer is quite correct, we envisage that policy action areas will be broad, e.g., physical education or extra-curricular sport, but the specific policy actions, e.g., require a minimum PE duration or require a PE curriculum may be more defined.  Until the reviews are carried out, the level of evidence is unknown.

Comment 7:

p. 6. Strategy for data synthesis

I am not clear on what is meant by: “Outcome data extracted will be tabulated to determine the impact on (my emphasis) policy areas and policy actions (policy indicators)”. What is meant here by ‘policy indicators’ for example?

--

We agree with the reviewer that this is not clear. In order to improve clarity this sentence has been removed and the third paragraph on page 12 has been restructured as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

“A narrative synthesis will be used to interpret and analyse the data. The results of the data synthesis will be presented in a table. Within this table, a list of short descriptive statements will be compiled based on the policy actions identified in the scientific literature. Evidence on the effectiveness of these policy actions will then be described using a method used by Panter and colleagues (2019) […]”

Comment 8:

I suggest that the method described by Panter and colleagues (2019) be described in the paper (rather than just referring the reader to the paper). It is very important that the reader understands the criteria that is being used to assess the evidence of effectiveness.

--

Detail has been added to the second half of the first paragraph on page 12 to describe the coding system inspired by Panter et al. It now reads as follows, text in bold is ‘new’ and has been added for clarity:

[Strategy for data synthesis, page 12 paragraph 4]

[…] Specifically, when a study presents quantitative evidence about the effectiveness of one policy action a symbol will be assigned next to that particular policy action in the table. There are four categories of symbols reflecting the four possible outcomes: “significantly positive evidence” (+), “significantly negative evidence” (-), “no significance test” (?) or “inconclusive” (0).

Comment 9:

I am unclear about how the findings from systematic reviews could be combined with primary research studies. This needs to be further explained.

--

This may indeed be challenging, but until we examine the main results and conclusions of any systematic reviews that have reference to policy impact in the targeted domains, we cannot be sure of whether their evidence is strong enough and clearly defined in order to assess impact.  While we agree it may be difficult, in the search strategy we will include these documents in the initial search and depending on what we find include or exclude in the final data synthesis.

Comment 10:

Will there be any attempt to distinguish effectiveness based on the following: school level (primary or secondary school level), gender, level of socio-economic disadvantage?

--

Until data extraction is complete, it is difficult to identify any specific stratification variables that would be pertinent to the presentation of the results.  However, the data extraction tables will be devised to take into account any relevant stratification variables for each review as pertinent to the topic and domain.

As the reviewer’s comment applies to the systematic review corresponding to the school setting only evidence of effectiveness will be presented in a manner that makes it clear which policy actions were shown to be effective at different school levels, across gender and SES where appropriate. The following sentence has been added to the strategy for data synthesis section to clarify this, text in bold is ‘new’ and has been added for clarity.

[Strategy for data synthesis, page 12 paragraph 3]

“In addition,  data extraction tables will be designed to distinguish any demographic, environmental or other variables pertinent to synthesising the data. For example, in the schools review , data extraction columns will be included  to reflect evidence of effectiveness stratified by gender, school level (primary, secondary, combined) or socio economic status where appropriate. 

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Availability Statement

    Underlying data

    No underlying data are associated with this article.

    Reporting guidelines

    Open Science Framework: PRISMA-P checklist for “Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of school policies on physical activity”. https://doi.org/10.17605/OSF.IO/26QYF ( Volf, 2020).

    Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

    Articles from HRB Open Research are provided here courtesy of Health Research Board Ireland

    RESOURCES