Protocol for developing a core outcome set for evaluating school-based physical activity interventions in primary schools

Abstract

Introduction

Primary school-based physical activity interventions, such as The Daily Mile initiative, have the potential to increase children’s physical activity levels over time, which is associated with a variety of health benefits. Comparing interventions or combining results of several studies of a single intervention is challenging because previous studies have examined different outcomes or used different measures that are not feasible or relevant for researchers in school settings. The development and implementation of a core outcome set (COS) for primary school-based physical activity interventions would ensure outcomes important to those involved in implementing and evaluating interventions are standardised.

Methods and analysis

Our aim is to develop a COS for studies of school-based physical activity interventions. We will achieve this by undertaking a four-stage process:(1) identify a list of outcomes assessed in studies through a systematic review of international literature; (2) establish domains from these outcomes to produce questionnaire items; (3) prioritise outcomes through a two-stage Delphi survey with four key stakeholder groups (researchers, public health professionals, educators and parents), where stakeholders rate the importance of each outcome on a 9-point Likert scale (consensus that the outcomes should be included in the COS will be determined as 70% or more of all stakeholders scoring the outcome 7%–9% and 15% or less scoring 1 to 3); (4) achieve consensus on a final COS in face-to-face meetings with a sample of stakeholders and primary school children.

Ethics and dissemination

We have received ethical approval from Imperial College London (ref: 19IC5428). The results of this study will be disseminated via conference presentations/public health meetings, peer-reviewed publications and through appropriate media channels.

Trial registration number

Core Outcome Measures in Effectiveness Trials Initiative (COMET) number: 1322.

Strengths and limitations of this study.

• To our knowledge, this will be the first core outcome set developed to evaluate school-based physical activity interventions in primary schools, which will improve evidence synthesis in this field.

• The study will use a robust four-stage process including a modified Delphi technique, to incorporate multidisciplinary stakeholder perspectives, including researchers, public health professionals, educators (ie, head teachers, teachers and school governors), parents and primary school children.

• The stakeholders are drawn from an international pool and a systematic literature review of international literature.

• A limitation of this study is that primary school children are considered too young to participate in the Delphi survey rounds. To ensure we capture children’s perspectives, we will conduct a separate face-to-face meeting and their views will be considered at the final stage.

Introduction

Regular physical activity in children and young people is associated with physical and mental health benefits including musculoskeletal fitness and lower risk of depression, obesity and diabetes.^{1 2} A growing evidence base also suggests physical activity improves sleep duration, cognition³ and academic performance.^{4 5} Hence, current guidelines from the WHO recommend 60 minutes of moderate-to-vigorous physical activity every day for children.⁶ However, in high-income countries, only one in five children and young people are meeting these physical activity targets.⁷ Several school-based physical activity (SBPA) interventions have been developed and implemented to increase children’s activity levels. A Cochrane review of 44 randomised controlled trials of SBPA interventions for children aged 6–18 years found nine different outcome domains and concluded that additional research on the long-term impact of these interventions is needed.⁸

Active mile initiatives, such as The Daily Mile, which involves 15 minutes of self-paced physical activity,⁹ are encouraged by governments of several European countries. Policy makers in the United Kingdom (UK) are now promoting and incentivising their implementation in primary schools (children aged 4–11 years).¹⁰ However, the evidence base of their effectiveness is limited. Previous studies, although promising, have been small scale, and examine different outcomes using different measuring tools that are not practical for follow-up over long periods (eg, physical activity measured by accelerometers which only capture a specific period of physical activity pattern).^{11 12} It is also unclear which outcomes are most relevant for those involved in implementing and evaluating interventions.

A core outcome set (COSs) is an agreed standardised set of outcomes indicating what should be reported.¹³ The outcomes must be measurable and relevant for researchers and other key stakeholders. Core outcome sets were originally developed for clinical trials, but increasingly been developed and used in other areas.¹⁴ A COS specifies a minimum set of outcomes assessed in all studies, but is flexible to allow the inclusion of additional outcomes into any particular study.¹³ To our knowledge, there is not a COS that exists for the evaluation of primary school-based physical activity interventions. Therefore, there is a need to develop a COS to ensure that the same outcomes are being measured to allow for the direct comparison of school-based physical activity interventions across studies.

Aims and objectives

The aim of this study is to identify a COS for primary school-based physical activity interventions over time. This study will focus on what should be measured, and we will assess ‘how’ to measure each core outcome.

Study objectives include:

1. To develop a list of potential outcomes relevant to evaluating primary school-based physical activity interventions over time.

2. To prioritise outcomes of whole-school physical activity important to relevant stakeholders including professionals and researchers.

3. To achieve consensus on a minimum set of relevant outcomes for primary school-based physical activity interventions (ie, COS).

Methods

Steering group

We have formed a steering group for this project, including healthcare professionals and researchers to guide the development of this COS. We have recruited members representing different disciples and expertise including health professionals and researchers with methodological expertise in epidemiology, statistics and consensus methods. We have also identified a study management group within the steering committee to conduct day-to-day management of the study. We consulted with this committee to identify core principles that we should apply when identifying our set of core outcomes. This group determined that outcomes should be feasible for use in large-scale studies and should be both valid and reliable.

Modified Delphi

The study design uses a modified Delphi technique (the RAND/UCLA appropriateness method) to identify a set of core outcomes.¹⁵ This technique has previously been used in the development of a COS across a variety of clinical and research contexts.^{16 17} The modified Delphi process involves four stages:

1. Identifying a list of outcomes from systematic literature reviews.

2. Reduction of the list into domains for questionnaire items.

3. Prioritisation through a Delphi survey involving two rounds of questionnaires and incorporation of additional outcomes nominated by stakeholders

4. Face-to-face consensus meetings to agree a final core set with stakeholders.

Stage 1: systematic literature review

We will conduct a comprehensive umbrella review of systematic reviews and meta-analyses to identify a list of outcomes relevant to school-based physical activity interventions. The process of this systematic review has been registered with PROSPERO (CRD42019146621).¹⁸ To identify reviews, we will search MEDLINE, EMBASE, CINAHL, CENTRAL, PsycINFO and the Cochrane Database of Systematic Reviews, restricting our search to include English language only and articles published since 1990. A detailed search strategy for each database is included in online supplementary appendix A. We will also aim to include relevant papers from the grey literature and in particular, we will review the Standard Evaluation Framework for Physical Activity Interventions¹⁹ and the DAPA (diet, anthropometry, and physical activity) measurement toolkit.²⁰

We will compile studies in EndNote software and remove duplicates. Two authors will independently conduct title/abstract screening to identify eligible systematic reviews or meta-analyses. Disagreements will be resolved by discussion, or as needed, by discussion with a third author. Title and abstract screening will be followed by full-text screening. For inclusion, eligible reviews will describe physical activity interventions or processes targeted at primary school children (aged 4–11 years). All types of study designs will be included. We will exclude any studies that are not in English, focus primarily on adolescents or young adults or those that are aimed at a particular subpopulation of children as these studies would not be generalisable to the whole school population. We will use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to document the number of articles included and excluded during the searches.²¹

Once the systematic reviews are identified, we will conduct a quality assessment of the reviews using the Critical Appraisal Skills Programme (CASP)²² tool; low-quality reviews will be excluded. We will search the included studies from each review. As we are interested in studying physical activity interventions delivered in a ‘real-world’ setting, we will apply additional eligibility criteria to the studies selected from within each review. Eligible studies must include a longitudinal study design (as they may include more relevant outcomes of interest) but we will not limit the duration of the intervention, and outcomes must be applicable to primary school children (approximately 4–11 years). In addition, we will limit studies to those conducted in the last three decades. To ensure we capture all relevant papers, we will identify additional relevant studies by screening the reference list for each eligible study included. Again, this search will be performed by two study authors with disagreements resolved by discussion or through consultation with a third author.

Outcomes will be identified from the methods and results section of each paper. For each outcome, the following data will be extracted: study characteristics (eg, author(s), year, country and sample size), study population (eg, number of participants, target age, ethnic groups), how the outcomes were defined, the time points for measurement and intervention duration, the measurement tool used and whether it was validated, any reliability information (eg, test–retest reliability), and any methods used to enhance quality of outcome measurement (eg, measured twice). If the tool was validated, we will record details of the population used for validation (eg, age and country of children). All data extraction will be completed by one study author but 10% of the papers will be done by a second author to check consistency. Disagreements will be resolved by discussion or by consultation with a third author, as required.

Stage 2: establishing domains for questionnaire items

The domains for questionnaire items will be established by grouping similar outcomes that capture a broader concept.^{23 24} Domains will be identified independently by two researchers and a small number of stakeholders in discussion with a third senior researcher if there are discrepancies. The shortlisted domains will form candidate outcomes as questionnaire items in plain English for all stakeholder groups. The questionnaire will be designed and piloted with input from lay representatives to ensure its understanding and acceptability.

Stage 3: prioritisation of outcomes through a Delphi survey

Delphi Survey: round 1

The first round of the modified Delphi process will involve surveying stakeholders to prioritise each of the outcomes identified from the literature search through an anonymous Delphi survey. The advantages of this method include the low costs and avoidance of influence from strong voices in group-based decision-making. Following guidance in the literature,²⁵ we aim to recruit approximately 60 participants; around 15 members each representing four key stakeholder groups: (1) researchers, (2) health professionals, (3) educators, that is, school teachers, head teachers, school governors, and (4) parents. By ensuring heterogeneity in overall group composition it may help to identify outcomes that would be otherwise overlooked.^{13 26 27} Through our research networks, colleagues and through public health social media platforms, we will create a sampling frame of potential stakeholders to invite. In addition, we will ensure that teachers, head teachers, and school governors represent schools that are and are not taking part in SBPA interventions. We will use snowballing methods to identify further panel members and we aim to include adult panel members with a range of expertise and from different countries who are able to write and understand English. Due to the complexity of the survey rounds, we felt it would be inappropriate to include primary school children at this stage of the COS development. Instead we will include children aged 7–11 years in a face-to-face meeting (stage 4) to learn about what is important to them, and ensure their views are represented in this study. This age range reflects the age of children in primary school where children have an understanding of the improtance of physical activity.

We will invite each potential panel member by email to participate in this study. We will obtain informed consent from all participants who agree to take part, and provide them with information about the entire Delphi process and the importance of participating in all rounds of the study.²⁷ Recruitment of panel members will continue until we have a minimum of 12 and a maximum of 20 from each stakeholder group.²⁵

We will send each participant a survey by email which they will be asked to complete within 3 weeks of receipt. Participants will be required to rate the importance of each outcome using a 9-point Likert scale ranging from 0 ‘not that important’ to 9 ‘critical’. They will also be asked to suggest any additional outcomes not included in survey. All surveys will be completed online. We will send two reminder emails to encourage responses (one at the end of week 2 and one at the end of week 3 allowing for one more week to complete the survey).

All survey results will be reviewed to identify missing data, possible outliers and the range of response options used. For each outcome, the distribution of scores will be generated and the median score calculated. We will calculate these separately for each stakeholder group.

Additional outcomes suggested by at least two participants will be reviewed by the study team. If there is disagreement about whether a new suggested outcome is unique that cannot be resolved by discussion, they will consult with a third team member. New outcomes will be added to the survey for round 2 of the Delphi. All outcomes included in round 1 of the survey will be retained for the second round of the Delphi survey.

Delphi survey: round 2

We will contact all participants who complete round 1 of the survey to complete round 2. The round 2 survey will include feedback from round 1 showing their scores compared with other participants in their own stakeholder group and other groups. ²⁸ In the round 2 survey, we will ask participants to re-rate the importance of each outcome and any new outcomes. After this round, we will conduct analyses to determine consensus. Consensus that the outcome should be included in the COS will be determined as 70% or more of all panel members scoring the outcome 7%–9% and 15% or less scoring 1%–3%. Consensus that the outcome should NOT be included in the COS will be 70% or more of all panel members scoring the outcome 1%–3% and 15% or less scoring 7%–9%.¹⁴ We will divide the outcomes list into three groups: consensus that it should be included in the COS, consensus that it should be excluded and no consensus reached. Outcomes that reach consensus for inclusion and those where no consensus was reached will be retained for discussion during the face-to-face meeting.

Stage 4: consensus meeting to agree a final core outcome set

The fourth stage of this Delphi process will consist of two face-to-face meetings to obtain consensus on the final core set. We will conduct one meeting with adult stakeholders, and a separate meeting with children. The meeting with children will be first and informed by the results of the Delphi survey. Through a day of activities and discussions led by a trained facilitator, we will learn about which outcomes are important to the children. Recruitment of children for the face-to-face meeting will involve an invitation letter sent to parents identified through the educators and parents (in the UK) participating in the questionnaire rounds. A child information leaflet will be also be included. We aim to include approximately 10–15 children aged from 7 to 11 years per school, inviting a minimum of two and a maximum of four schools. In total, we aim to include 20–60 children. Written parental consent and child assent will be obtained. As the meeting with children will involve a number of activities, it will not be possible to include children from other countries. However, the children will be recruited from UK schools representing those from urban and rural, and from deprived and non-deprived areas.

For the adult stakeholder meeting, a representative sample from each stakeholder group who have completed both rounds of the survey will be invited to attend. We aim to recruit at least one international member for each stakeholder group to join the face-to-face meeting. The meeting will be run by an independent facilitator who has experience of participatory research and one of the study researchers. We will present the results of the Delphi survey to the adult stakeholders invited to attend the face-to-face meeting (including at least one international participant representing each stakeholder group). We will present the ratings for each outcome from the Delphi surveys for each stakeholder group and overall alongside the outcomes deemed important to the children. Each stakeholder group will be asked to discuss the outcomes retained after survey round 2 and present their views back to the whole group. After the discussions, each participant will be issued with a unique keypad and asked to vote each outcome as ‘include’, ‘exclude’ or ‘unsure’. All voting will be done simultaneously and individually without conferring. All participants will view the results of voting. Outcomes that are equivocal will be discussed as a group and each panel member will have a second chance to vote on these outcomes. The results will be compiled, and consensus ratings determined using the 70/15 criteria described earlier. The final list will be presented to the group for final discussion and comments. All items prioritised by the stakeholders from stage 4 will be included in the final COS for use in research in high-income countries.

Patient and public involvement

We obtained public involvement input from The Daily Mile Foundation and from participants of The Daily Mile Stakeholder Group. We obtained feedback and input on recruitment methods for research participants, incentives for survey participation and written and verbal feedback on recruitment materials. We will obtain further PPI input on the development and piloting of the Delphi survey.

Participant consent and dissemination

We will obtain written consent from all adult stakeholders, and written parental consent and child assent for children to take part in the face-to-face meeting. All survey rounds will be conducted anonymously; participants will not be told who the other respondents are or what their specific responses were. Participants’ contact information (names and emails) will be retained in accordance with Imperial College London’s data collection, retention and storage policies. During the face-to-face meeting, participants will be aware of who the other panel members are, but where possible, individual responses will remain anonymous. To limit any adverse impact on school children during the face-to-face meeting, we will aim to make the materials and activities during the meeting interactive and enjoyable. The results of this study will be shared in conference presentations, public health meetings, and via appropriate media channels. We will publish the process of developing the COS in a peer-reviewed journal, and also publish the COS as a technical operating manual for relevant audiences. This study has also been registered with COMET and an update of the study results will be published on their website.