Investigating the rigour of research findings in experimental studies assessing the effects of breaking up prolonged sitting – extended scoping review

Highlights

• •Only 50% of the published trials were registered.
• •Unregistered trials may have collected other measures that remain unreported.
• •Many additional outcomes not appearing in trial registries were published.
• •Some reported positive findings are likely spurious due to unadjusted multiple comparisons.

Abstract

Objectives

Sedentary behaviour research is a relatively new field, much of which has emerged since the widespread acceptance of clinical trial registration. The aim of this study was to investigate the trial registration and related issues in studies investigating the effect of frequent activity interruptions to prolonged sitting-time.

Methods

Secondary analysis of a scoping review including systematic searches of databases and trial registries. We included experimental studies investigating the effects of frequent activity interruptions to prolonged sitting-time.

Results

We identified 32 trials published in 45 papers. Only 16 (50%) trials were registered, with all 16 trials being completed and published. Of the unregistered trials, we identified three (19%) for which similarities in the sample size and participant demographics across papers was suggestive of duplicate publication. Identification of potential duplicate publications was difficult for the remaining 13 (81%). Results from 53 (76%) of the 70 registered outcomes were published, but 11 (69%) registered trials reported results from additional outcomes not prospectively registered. A total of 46 different outcomes (out of 53 reported outcome measures, similar measures were collated) were reported across all trials, 31 (67%) of which were collected in ≤2 trials.

Conclusions

We found direct evidence of trial registration issues in experimental trials of breaking up sitting-time. The lack of prospective registration of all trials, and the large number of outcomes measured per trial are key considerations for future research in this field. These issues are unlikely to be confined to the field of sedentary behaviour research.

Introduction

Sedentary behaviour is an emerging field of research, and as with all emerging fields there has been much excitement about early findings. As the field grows, it is important that it is closely scrutinised for potential issues of bias. Systematic reviews and meta-analyses are ways of synthesising published research literature. However, issues such as the non-publication of negative findings (publication bias), the publication of some outcomes and not others (selective reporting), the reporting of same outcomes multiple times, and the clarity of outcomes reported in separate papers arising from the same study (duplicate publication bias), affect the trustworthiness of the original publications making reviews more challenging and therefore review findings¹ and can lead to the overestimation of effects.²

In an emerging research field, it is also common to see a large number of outcomes being assessed in each trial as new mechanisms of action are hypothesised and exploratory outcomes considered. However, as the number of outcomes assessed per individual participant increases, so too does the risk of Type I error – that is finding statistically significant results that are spurious.³ Clinical trial registries are designed to guard against some of these issues, and most ethics committees, research governance bodies, and medical journals now require all clinical trials to be prospectively registered prior to participant recruitment. While the Cochrane Collaboration requires all reviews to consider issues of publication bias and selective reporting by comparing published findings with trial registry records, this is not widespread across all published systematic reviews, including those in sedentary behaviour research.⁴

In the world of exercise physiology research, there are two broad fields; physical activity research, focused on the moderate to vigorous end of the exercise intensity spectrum and sedentary behaviour research, focused on the effects of prolonged sitting. While the former is a well-established field with a long history of publication, the latter is in its relative infancy. Much of the physical activity research pre-dates the requirement to register clinical trials. However, the newer field of sedentary behaviour research, especially the literature investigating sedentary behaviour interruption with short bouts of physical activity, provides a unique opportunity to carefully examine how information found in publications are consistent with trial registration information, so as to determine the rigour of reporting of published work.

In this paper, to maintain a reasonably narrow scope, we chose to assess biases by focusing on experimental trials investigating the effect of interrupting prolonged periods of sitting time with frequent, short bouts, of physical activity or standing. In the past five years, this field of research has rapidly expanded with an increasing number of primary trials and systematic reviews being published each year. This is important, given the increasing use of these findings to inform intervention and guideline development.

Therefore, the aim of this study was to examine issues in trial registration, which may suggest publication bias, selective reporting, and risk of Type I error in studies investigating the effect of frequent activity interruptions to prolonged sitting. The specific research questions were in experimental trials investigating the effect of interrupting prolonged periods of sitting time with frequent, short bouts, of physical activity or standing;

• 1.How many published trials have been registered, and of those registered, have all completed trials been published?
• 2.Are duplicate publications from the same trial easily identifiable?
• 3.Are all outcomes recorded in trial registries published?
• 4.How many different outcomes have been measured in each trial?

Methods

Identification and selection of studies

This paper is an extension of a scoping review⁵ with systematic searches that aimed to review the evidence for the effect of interrupting prolonged sitting with frequent bouts of physical activity or standing on first or recurrent stroke risk factors. During the initial scoping review, standard guidelines were followed such as independent title and abstract, and full text screening, with discrepancies resolved by a third member of the research team.⁵ Five databases including Medline, Embase, Allied and Complementary Medicine, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and the Cochrane Library were searched from the inception to February 2018 (full search strategy and methodology previously published⁵). Experimental trials with adult participants (age ≥18 years), investigating the effect of interrupting prolonged periods of sitting time compared with supervised interventions of frequent, short bouts, of physical activity or standing published in English were included in the review. Only studies that included a control condition of uninterrupted prolonged sitting were included. While the scoping review included only studies that reported at least one outcome measure related to stroke risk factors, in this study we did not exclude studies based on outcome measures.

Identification of trial registration

We used a two-staged approach to search for registered trials. We first scrutinised all included articles for reference to a trial registry identification number. Secondly, we searched trial registries for trials meeting the same inclusion criteria. We searched the five most commonly used registers for trials published in English, including the Australian and New Zealand Clinical Trial Registry (ANZCTR), ClinicalTrials.gov, International Standard Randomised Controlled Trial Number (ISRCTN) registry, World Health Organisation International Clinical Trials Registry Platform, and the European Union Clinical Trials Register. Key terms used were a combination of “sedentary” OR “sitting” AND “break” or “interrupt” or “activity”. Where possible, results were filtered to include only trials that were “completed”, “interventional”, and/or conducted with “adult/elderly” participants.

Data extraction

Data extraction was performed independently by two reviewers and cross-checked for accuracy. Trial registry number, participant characteristics, and measured outcomes for each trial, recorded separately for registered and non-registered trials, were extracted from the included studies. Information on the number of outcome measures collected, number of outcome measures registered, number of outcome measures ‘registered and reported’, number of outcome measures ‘registered but not reported’, number of outcome measures ‘reported but not registered’, and reported outcomes that were not registered but for which significant results were found in each trial were recorded. Narrative analysis using descriptive statistics was undertaken to describe the nature of the studies and to answer the research questions. Bias definitions are provided alongside the research questions and detailed here.

Potential publication bias was examined by reporting how many published trials had been registered, and of those registered, were all completed trials been published. Any potential duplicate publication bias was observed by carefully examining papers by the same or similar author lists for substantial similarities between participant inclusion criteria, sample sizes, and methods. Possible selective outcome reporting was examined by matching outcomes reported in trial registries to those published. Lastly, any possible risk of Type I error was identified by observing how many different outcomes had been measured in each trial.

Results

Out of 7645 title and abstracts, 69 were eligible for full text screening. Out of the full text papers reviewed, 32 trials reported in 45 papers were included (Fig. 1). The sample sizes per trial ranged from 9 to 70 (Table 1), with a combined total sample of n = 606 (289 females, 48%). The mean age of included participants ranged from 21 to 69 years, and trials included a variety of population groups including people who were healthy (n = 16), overweight (n = 13), had type 2 diabetes (n = 2), or post-stroke (n = 1). All papers were published between 2012 and 2018, with 36 (80%) published in the last 5 years.

Figure 1.

PRISMA flowchart of the review.

Table 1.

Summary of trials and outcomes.

Trial registry no.	Papers	n (trial)	Participant characteristics	Outcomes registered (published outcomes are bolded)	Outcomes reported not registered (outcomes with significant positive between condition differences are italicised)
ACTRN12613000576729	Dempsey et al. (2015)7 Dempsey et al. (2016)8 Dempsey et al. (2017)6 Grace et al. (2017)11 Dempsey et al. (2018)23	24	Type 2 diabetic overweight/obese and inactive adults 10 females (41.7%) 62 ± 6 years	Postprandial glucose Continuous glucose measurement Postprandial insulin C-peptide analysis Blood pressure Fatigue	Triglycerides Other cholesterols Norepinephrine/Noradrenaline Respiratory exchange ratio Total energy expenditure Heart rate
ACTRN12615001189516	English et al. (2018)22 English et al. (2018)10	19	Post-stroke people (between 3 months and 10 years) ambulant with minimal assistance and not taking diabetic medication other than metformin 9 females (47.4%) 68.2 ± 10.2 years	Postprandial glucose Insulin control Blood pressure Fibrinogen Feasibility (adherence to protocol including fatigue)	Nil
ACTRN12609000656235	Larsen et al. (2014)15 Latouche et al. (2013)16 Dunstan et al. (2012)9 Howard et al. (2013)13	19	Overweight/obese adults 8 females (42.1%) 53.8 ± 4.9 years	Postprandial glucose levels Postprandial insulin levels Postprandial free fatty acid levels	Hemostatic parameters (fibrinogen, platelet count, activated partial thromboplastin time, Von Willebrand factor antigen) Haematological parameters (volume, Hct, Hb, RBC, WBC, PC, MPV) Blood pressure Heart rate Gene expression
ACTRN12614000624684	Homer et al. (2017)12 Mete et al. (2018)17	36	Healthy, normal weight adults 25 females (69.4%) 25.4 ± 3.9 years	Postprandial glucose Postprandial insulin Postprandial triglyceride Postprandial NEFA Substrate utilisation Energy expenditure Energy intake Appetite	None
ACTRN12611000632998	Thorp et al. (2014)19 Thorp et al. (2014)20	23	Overweight/obese sedentary office workers 6 females (26.1%) 48.2 ± 8 years	Postprandial glucose Postprandial insulin Postprandial triglyceride Postprandial free fatty acid levels	Fatigue Musculoskeletal discomfort Work productivity Workstation acceptability
ACTRN12610000657022	Larsen et al. (2015)14	19	Overweight/obese, sedentary adults 8 females (42.1%) 56.7 ± 6.5 years	Postprandial glucose Postprandial insulin Postprandial triglycerides Postprandial free fatty acid	None
ACTRN12610000953033	Peddie et al. (2013)18	70	Healthy, normal weight adults working in a predominately sedentary occupation 42 females (60.0%) 25.9 ± 5.3 years	Postprandial glucose Postprandial insulin Postprandial triglyceride Blood pressure Substrate utilisation	Heart rate
ACTRN12613000137796	Wennberg et al. (2016)21	19	Overweight/obese adults 9 females (47.7%) 59.7 ± 8.1 years	Continuous glucose measurement Cortisol Catecholamines Executive function Memory Brain-derived neurotrophic factor, BDNF Interleukin-6, IL-6	Insulin Blood pressure Fatigue Heart rate
ISRCTN48132950	Brocklebank et al. (2017)31	17	Healthy office workers in a predominately sedentary occupation 9 females (52.9%) 52.4 ± 5.1 years	Postprandial glucose Pre-prandial interstitial glucose	None
NCT02215603	Benatti et al. (2017)24	14	Healthy, physically inactive males 14 males (100%) 30.1 ± 8.8 years	Postprandial glucose Continuous glucose measurement Postprandial insulin Postprandial lipids Postprandial cytokine	C-peptide Total energy expenditure
NCT02717377	Bergouignan et al. (2016)25	30	Healthy sedentary adults 21 females (70.0%) 30 ± 5.6 years	Energy and mood level Cognitive function	Appetite ratings Epinephrine/Adrenaline Norepinephrine/Noradrenaline Dopamine Cortisol Heart rate
NCT00945165	Dijk et al. (2013)26	20	Type 2 diabetic males 20 males (100%) 64 ± 1 years	Hyperglycemia Mean blood glucose	Heart rate
NCT02135172	Henson et al. (2016)27	22	Postmenopausal, overweight/obese dysglycemic women 22 females (100%) 66.6 ± 4.7 years	Postprandial glucose Postprandial insulin Postprandial triglycerides Lipoprotein lipase activity	NEFA levels
NCT02743286	Kerr et al. (2017)28	10	Postmenopausal, overweight/obese and sedentary women 10 females (100%) 66 ± 9 years	Change in glycemic regulation Change in endothelial function Change in mitochondrial metabolites Study feasibility	Blood pressure
NCT02493309	McCarthy et al. (2017)29	34	Healthy, non-obese adults working in a predominately sedentary occupation 18 females (52.9%) 40 ± 9 years	Postprandial glucose Postprandial insulin Postprandial triglycerides Blood pressure	None
NCT02909894	McCarthy et al. (2017)30	13	Obese and inactive adults at risk of type 2 diabetes 7 females (53.8%) 66 ± 6 years	Postprandial glucose Postprandial insulin Blood pressure Cognitive function	Mood
	Holmstrup et al. (2013)41	11	Obese, young adults with impaired fasting glucose 3 females (27.2%) 25.2 ± 1.3 years	N/A	Postprandial glucose Postprandial insulin C-peptide Peptide YY tAUC Hunger and satiety
	Holmstrup et al. (2014)42
	Masaki et al. (2015)44	15	Healthy, normo-lipidaemic men 15 males (100%) 26.80 ± 2 years	N/A	Postprandial glucose Postprandial insulin NEFA levels Other cholesterols LPL protein/Apolipprotein Oxidative stress markers 3-OHB
	Miyashita et al. (2013)46
	Mullane et al. (2017)47	9	Overweight/obese and physically inactive (pre-hypertensive or with impaired fasting glucose) 7 females (77.8%) 30 ± 15 years	N/A	Sleep efficiency Heart rate Cognitive performance
	Zeigler et al. (2016)50
	Altenburg et al. (2013)32	11	Healthy young adults 6 females (54.5%) 21.4 (19.5–23.1) years	N/A	Postprandial glucose C-peptide Triglycerides Total cholesterol HDL cholesterol LDL cholesterol Muscle activity
	Bailey and Locke (2015)34	10	Healthy, non-obese adults 3 females (30.0%) 24.0 ± 3 years	N/A	Postprandial glucose Triglycerides Total cholesterol HDL cholesterol Blood pressure
	Bailey et al. (2016)33	13	Healthy, inactive, sedentary adults 7 females (53.8%) 26.6 ± 8.5 years	N/A	Postprandial glucose Postprandial insulin Peptide YY tAUC Acylated ghrelin tAUC Appetite ratings Energy intake
	Barone Gibbs et al. (2017)35	25	Overweight/obese adults with pre to stage 1 hypertension 9 females (36.0%) 42 ± 12 years	N/A	Blood pressure Pulsed wave velocity Heart rate
	Bhammar et al. (2017)36	10	Overweight/obese and physically inactive adults 5 females (50.0%) 32 ± 5 years	N/A	Postprandial glucose Blood pressure
	Carter and Gladwell (2017)37	10	Healthy adults 4 females (40.0%) 27.3 ± 8.1 years	N/A	Blood pressure Endothelial function Shear rate
	Engeroff et al. (2017)38	18	Healthy young premenopausal women 18 females (100%) 25.6 ± 2.6 years	N/A	Total cholesterol HDL cholesterol LDL cholesterol
	Hansen et al. (2016)39	14	Healthy young, normal weight and recreationally active adults 8 females (57.1%) 22 (20 – 23) years	N/A	Postprandial glucose
	Hawari et al. (2016)40	10	Overweight/obese, normoglycaemic males 10 males (100%) 33 ± 13 years	N/A	Postprandial glucose Postprandial insulin Triglycerides Substrate oxidation Total energy expenditure
	Kim et al. (2014)43	9	Healthy young recreationally active males 9 males (100%) 24 ± 4 years	N/A	Postprandial glucose Postprandial insulin Triglycerides Other cholesterols Substrate oxidation Respiratory exchange ratio
	Miyashita et al. (2016)45	15	Postmenopausal women 15 females (100%) 68.8 ± 3.2 years	N/A	Postprandial glucose Postprandial insulin Triglycerides NEFA levels Rating of perceived exertion Heart rate 3-OHB
	Pulsford et al. (2017)48	25	Healthy, inactive, weight stable males 25 males (100%) 40.2 ± 12.2 years	N/A	Postprandial glucose Matsuda index Postprandial insulin
	Thosar et al. (2015)49	12	Healthy inactive young non-obese males 12 males (100%) 24.2 ± 4.2 years	N/A	Flow mediated dilation Shear rate

Abbreviations: ACTRN, Australian New Zealand clinical trials registry; Hb, haemoglobin; Hct, haematocrit; HDL, high-density lipoproteins; ISRCTN, international standard randomised controlled trials number; LDL, low-density lipoproteins; LPL, lipoprotein lipase; MPV, mean platelet volume; N/A, not applicable; NCT, national clinical trial; NEFA, non-esterified fatty acids; PC, platelet count; RBC, red blood cell count; tAUC, total area under the curve WBC, white blood cell count; 3-OHB, 3-hydroxybutyra.

How many published trials are registered, and of those registered, have all completed trials been published?

Only 16 (50%) of published trials were registered (Table 1) in registries including the ANZTR (n = 86, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23), ClinicalTrials.gov (n = 724, 25, 26, 27, 28, 29, 30), and ISRCTN (n = 1³¹). The remaining 16 (50%) trials (published in 19 papers) were not registered.32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50 All registered trials were completed and published (Fig. 2).

Figure 2.

Flowchart of the included studies.

Are duplicate publications from the same trial easily identifiable?

Outcomes from the 16 registered trials were published in 26 papers. In all of these papers either a trial registration number was quoted or the authors clearly identified that the paper was reporting additional outcomes from a primary study. Identifying duplicate publications from the unregistered trials was more difficult. For three trials,41, 42, 44, 46, 47, 50 similarities in the sample size and participant demographics across papers was suggestive of duplicate publication. We were unable to definitively determine how many of the remaining 13 unregistered trials were duplicate versus singular publications (Fig. 2).

Are all outcomes recorded in trial registries published?

Of the registered trials, a total of 70 outcomes were registered, and of these, results from 53 (76%) were published. Eleven (69%) registered trials had at least one outcome measure that was registered but the results not published, and 11 (69%) of the trials included results from additional outcomes that were not prospectively registered (Fig. 2). Published studies rarely specified whether the outcomes reported were primary or secondary measures. As 50% of trials were not registered, the degree to which outcome reporting bias is present (e.g. due to upgrading secondary outcomes to primary outcomes and vice versa) could not be established.

How many different outcomes have been measured in each trial?

A total of 46 different outcomes (out of 53 reported outcome measures, some similar outcome measures were collated. e.g.: high density lipids, low density lipids and total cholesterol were combined as postprandial lipids) were reported across all trials (including registered and unregistered trials), the majority (31 [67%]) being collected in ≤ 2 trials (Fig. 3). The number of outcomes collected per trial ranged from 1 to 12 (mean 5 [SD 3]). The most commonly measured outcomes were postprandial glucose (26 [81%]), postprandial insulin (21 [66%]), and triglycerides (13 [41%]) (Table 1). Across all the registered trials, results from 32 additional outcomes not mentioned in trial registry records were published. Of these additional outcomes 22 (69%) reported positive, significant between-condition differences (Table 1).

Figure 3.

Frequency of different outcomes measured in included trials.

Discussion

We conducted systematic searches of databases and trial registries to investigate the issues in trial registration in experimental trials investigating the effect of interrupting prolonged periods of sitting time with frequent, short bouts, of physical activity or standing. We found direct evidence of issues in trial registration which may lead to publication bias. All completed registered trials were published. However, our finding that only half of all published trials were registered creates uncertainty concerning the number of other unregistered trials which may also remain unpublished. Similarly, while there was no direct evidence of selective reporting (results from all registered outcomes have been published), 70% of registered trials published results from additional outcomes that were not registered. The search revealed that more than half of the published, non-registered outcomes reported statistically significant positive findings. However, this may be because some reported outcomes reflect the use of new technologies or techniques that may not have been present at the time of registration, particularly for some earlier studies that used metabolomics data. It is important to note that we chose to focus on the field of experimental sedentary behaviour research, given most papers were published after trial registration became common practice. It is possible that similar issues exist in other fields of research, including those with a longer publication history such as physical activity research.

More than half of the trials were not registered, making it difficult to thoroughly assess the degree of publication bias in the literature. This is an issue not limited to the field of sedentary behaviour research. Several previous reviews of randomised controlled trials published in medical journals in the past decade have found between 25 and 50% of trials were not registered, even those published in high impact factor journals.51, 52 When trials are not registered, it is impossible to know what additional outcomes were measured but not reported. Selective reporting of the collected outcomes could be influenced by the significance of the results,53, 54 leading to potentially misleading evidence.¹ We found that the majority of non-registered outcomes for which findings were published had significant, positive findings, lending weight to this concern. Many authors favour submitting their work with positive results for publication and many editors prefer to publish significant results.55, 56 To mitigate against this risk, it is good practice that effect sizes and confidence intervals for all outcomes are reported, regardless of the significance of the findings. Journal editors can assist in addressing this issue by giving equal consideration to papers reporting non-significant findings, particularly where trials are adequately powered to show an effect. Moreover, conflict of interests such as connections with related industries may be another potential contributing factor to publication bias.

Finally, the large number of different outcomes measured across trials warrants greater attention in future trials. While this is perhaps not surprising, given the field of the physiology of sedentary behaviour is still relatively young, and much of the research remains exploratory in nature, care must be taken to guard against potentially spurious findings. For future trials we recommend careful selection of outcomes, with a clear biological rationale for each and a single, pre-registered primary outcome with sufficient statistical power, as recommended by the CONSORT statement. Consideration should be given to adjusting analyses for multiple comparisons where this is appropriate.³ The reporting of any additional outcomes, particularly those with insufficient statistical power or that were not prospectively registered, should clearly identify that the analyses were exploratory in nature. Additionally, reaching consensus on a core outcome dataset for sedentary behaviour research, as has been done in other fields⁵⁷ would be a useful step forward. We also suggest researchers conduct two separate analyses for registered and non-registered trials in future systematic reviews.⁵² Future reviews may also consider investigating which factors (such as study design, methodological quality, and impact factor of the journals) were associated with registering or not registering trials. Furthermore, in future reviews it would be interesting to investigate if outcomes have been upgraded (i.e., from secondary outcome in the register to primary outcome in the published trial), or downgraded, and if these modifications were made in light of the results of the trial. This was not possible to do in our review as many registered trials did not clearly state a single primary outcome variable. It is important that future trials pre-register a clearly defined single primary outcome measure.6, 7, 11

The strengths of this study included a comprehensive search strategy, including all relevant databases and trial registries. Moreover, the independent data extraction between two authors adds further rigour to our work. Limitations include excluding trials not published in English, excluding conference abstracts and grey literature. While we included a consultation phase, it is possible that some recent papers may still have been missed. By only searching the five most commonly used trial registries, it is possible that some trial registrations may have been missed. However, no published trials reported registration numbers from registries other than those we searched, making this unlikely. We also acknowledge that the search strategy was developed to address the research objectives of the previous published scoping review, therefore we did not include searches of unpublished studies. While we are confident that we have included all relevant literature at the time period of search, further studies may have been published since. In addition, this review was designed to provide a broad overview of evidence on issues in trial registration which may lead to publication bias, selective reporting, and risk of Type I error in experimental studies investigating the effect of frequent activity interruptions to prolonged sitting time and therefore did not report on the magnitude or direction of the findings.

Conclusion

We found a lack of widespread trial registration, which may lead to publication bias and selective reporting among experimental trials investigating the effect of interrupting prolonged periods of sitting time with frequent, short bouts, of physical activity. To further strengthen confidence in this field, we recommend that all trials are prospectively registered, trial registration numbers are clearly stated in all publications along with explicit reference to other papers reporting results from the same dataset, trials are adequately powered for a primary outcome and exploratory analyses of secondary outcomes are clearly identified as such, and finally that the development of a core outcome dataset for the field is prioritised.

Conflicts of interest

The authors declare no conflicts of interest.