Skip to main content
PMC home page
PLOS ONE logoLink to PLOS ONE
. 2022 Oct 27;17(10):e0275242. doi: 10.1371/journal.pone.0275242

Accounting for the clustering and nesting effects verifies most conclusions. Corrected analysis of: “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk”

Yasaman Jamshidi-Naeini 1, Lilian Golzarri-Arroyo 1, Colby J Vorland 2, Andrew W Brown 2, Stephanie Dickinson 1, David B Allison 1,*
Editor: Eugene Demidenko3
PMCID: PMC9612448  PMID: 36301862

Abstract

In a published randomized controlled trial, household units were randomized to a nutrient bar supplementation group or a control condition, but the non-independence of observations within the same household (i.e., the clustering effect) was not accounted for in the statistical analyses. Therefore, we reanalyzed the data appropriately by adjusting degrees of freedom using the between-within method, and accounting for household units using linear mixed effect models with random intercepts for family units and subjects nested within family units for each reported outcome. Results from this reanalysis showed that ignoring the clustering and nesting effects in the original analyses had resulted in anticonservative (i.e., too small) time x group interaction p-values. Still, majority of the conclusions remained unchanged.

Introduction

An article by Mietus-Snyder et al. [1] (hereafter “The Article”) reported the results of a randomized controlled trial that examined the effects of nutrient bar supplementation on metabolic biomarkers among adolescents with obesity and their adult caregivers. In The Article, randomization occurred at the level of household units so that adolescents and their caregiver(s) were group randomized to either nutrient bar supplementation or a control condition. However, inferences were drawn from statistical models that ignored the non-independence of observations within the same family unit. Herein, we present a valid statistical model that accounts for clustering and nesting effects within the context of other methodologic choices made by the original authors. We therefore do not intend to address any question or test any hypothesis beyond the scope of The Article.

In The Article, anthropometric and clinical measures were analyzed separately for adolescents and adults. That is, the statistical models with anthropometric and clinical measurements of adolescents as the outcome variable did not include observations from adult participants and vice versa. In the case of plasma ceramides, sphingolipid bases, and amino acids, however, adolescents and adults were analyzed together but the clustering effect of households and the nesting effect that arises from the hierarchical structure of the data were not considered in the statistical analyses. The inconsistency in the analysis of different outcomes aside, we outline below why analyzing households without accounting for the clustering and nesting renders the results and conclusions unverifiable.

When groups (clusters/households), rather than individual subjects, are randomized to experimental conditions, outcome measures of the subjects from the same group are expected to be more similar compared to those from other groups. This within-cluster correlation violates the independent observations assumption, and needs to be accounted for in the statistical analyses [2]. Ignoring the clustering effect, as done in the analyses in The Article, potentially leads to incorrect estimation of the variance and an often-inflated type I error (i.e., anticonservative [that is, too small] p-values) [3, 4]. In The Article, use of incorrect procedures occurs when caregivers and adolescents in the same household are analyzed together as independent observations. Moreover, the control group included two triad family units, one with two adults and one adolescent and the other with one adult and two adolescents. Therefore, even if adolescents and adults are analyzed separately, as reported by The Article for anthropometric and clinical outcomes, there would still be family connections between some subjects in the adult and adolescent subgroups. Therefore, all analyses require accounting for the clustering effect in The Article. A rigorous approach to analyze group randomized trials is including all observations in the statistical model while including a random effect for the cluster. This approach allows for retaining all the information, and accounts for the variability within and among clusters [5].

We attempted to conduct a proper analysis on the data made publicly available as a supplement to The Article. We first attempted to reproduce the original analyses as reported. In doing so, we failed to obtain the same results as those reported in The Article for sphingolipid bases and amino acids. This discrepancy was communicated with the authors of The Article and the journal editors, who acknowledged the errors we had detected. P-values we obtained based on their original analysis approach are reported in our Table 1 below. Additionally, the authors collegially and expeditiously shared additional information on household unit identifiers with us, which is commendable. We also report an updated participant flow diagram in our S1 Fig that corrects the sizes of household units from what is reported in The Article based on the additional information the authors shared with us.

Table 1. Time x group interaction p-values from generalized estimating (GEE) and linear mixed effect models (LMM).

Outcome measures Mietus-Snyder et al. GEE. Corrected GEEa LMM v1 (without clustering/ nesting)b LMM v2 (with clustering/ nesting)c Comparing LMM v1 to LMM v2; Significance in both, neither, or changedd
Using p<0.05 Using p≤0.002
Fasting plasma ceramides
Cer C14:0 0.007 0.007 0.013 0.018 Both Neither
Cer C16:0 0.024 0.024 0.029 0.035 Both Neither
Cer C18:0 0.192 0.192 0.229 0.236 Neither Neither
Cer C18:1 0.058 0.058 0.080 0.088 Neither Neither
Cer C20:0 0.045 0.045 0.060 0.068 Neither Neither
Cer C20:1 0.325 0.259 0.274 0.281 Neither Neither
Cer C22:0 0.040 0.040 0.055 0.063 Neither Neither
Cer C22:1 0.117 0.111 0.125 0.134 Neither Neither
Cer C24:0 0.356 0.352 0.341 0.348 Neither Neither
Cer C24:1 0.014 0.014 0.023 0.029 Both Neither
T. Cer 0.144 0.144 0.147 0.155 Neither Neither
Plasma sphingolipid bases
Sphinganine e p≤0.002 (reported in the text) 0.007 0.013 0.017 Both Neither
Sphingosine 0.02 0.503 0.573 0.577 Neither Neither
Dihydro-sphingosine-1-phosphate 0.05 0.488 0.492 0.490 Neither Neither
Sphingosine -1-phosphate 0.007 0.522 0.549 0.553 Neither Neither
Plasma amino acid metabolites
Arginine e - 0.041 0.065 0.074 Neither Neither
Serine 0.0001 0.002 0.006 0.009 Both Neither
Proline 0.0001 <0.001 <0.001 <0.001 Both Both
Aspartate 0.0001 <0.001 0.001 0.002 Both Both
Cystathionine 0.0001 0.046 0.059 0.068 Neither Neither
Sarcosine 0.0001 0.011 0.010 0.014 Both Neither
Ornithine 0.001 0.178 0.222 0.222 Neither Neither
Arginine Bioavailability Ratio 0.001 <0.001 0.002 0.003 Both Changed
Lysine 0.002 0.051 0.061 0.070 Neither Neither
Alanine 0.003 <0.001 <0.001 0.001 Both Both
Glutamine 0.007 0.087 0.108 0.115 Neither Neither
Threonine 0.008 0.032 0.048 0.050 Changed Neither
Methionine 0.01 0.102 0.144 0.155 Neither Neither
Fischer ratio 0.01 0.576 0.548 0.537 Neither Neither
Citrulline 0.02 0.005 0.014 0.019 Both Neither
Histidine 0.02 0.379 0.354 0.361 Neither Neither
Tryptophan 0.02 0.405 0.480 0.483 Neither Neither
Leucine 0.02 0.032 0.064 0.075 Neither Neither
Phenylalanine 0.03 0.082 0.115 0.125 Neither Neither
Anthropometric and clinical measures d
Activity score - 0.483 0.474 0.479 Neither Neither
Vitamin D p<0.05 in both adolescent and adult subgroups <0.001 0.003 0.005 Both Neither
Weight - 0.898 0.898 0.898 Neither Neither
Waist to height ratio - 0.137 0.162 0.172 Neither Neither
BMI - 0.672 0.674 0.677 Neither Neither
Adiponectin - 0.551 0.597 0.601 Neither Neither
HsCRP - 0.223 0.264 0.271 Neither Neither
Glucose - 0.421 0.430 0.435 Neither Neither
Insulin - 0.202 0.234 0.242 Neither Neither
Homeostatic model assessment of insulin resistance - 0.287 0.310 0.317 Neither Neither
Plasma total cholesterol - 0.110 0.134 0.143 Neither Neither
Plasma triglyceride - 0.416 0.450 0.455 Neither Neither
LDL - 0.222 0.241 0.248 Neither Neither
HDL - 0.259 0.302 0.309 Neither Neither
Systolic blood pressure p<0.05 in adolescent subgroup 0.058 0.083 0.092 Neither Neither
Diastolic blood pressure - 0.235 0.272 0.281 Neither Neither
Resting heart rate - 0.106 0.136 0.146 Neither Neither
Open in a new tab

a These p-values represent revised values of the incorrect report in The Article after discussions with the authors. These p-values are thus calculated with the model used by Mietus-Snyder et al. that ignores household clustering.

b This model still ignores household clustering and nesting.

c Including random intercepts for household units and adjusted degrees of freedom using between-within method.

d Comparing LMM models with and without clustering and nesting effects being accounted for. We note both <0.05 and ≤0.002 thresholds per the authors’ intention to use them for different outcomes. Because they were used inconsistently in the text, we report here the consequences for both.

e Exact p-values for Arginine, Sphinganine, and anthropometric and clinical measures were not reported in The Article.

To reanalyze the data using procedures that take clustering and nesting effects of households into account, we performed linear mixed effect models (LMM) with random intercepts for family units (to account for the clustering effect) and subjects nested within family units (to account for the repeated measures) on each reported outcome in Tables 2, 4, 5, and 6 of The Article. We used the between-within method to adjust the degrees of freedom (SAS 9.4). The fixed effects in our models were the study group (levels: intervention or control), time (levels: baseline or follow-up), and the interaction between time and group (as the intervention effect). Included covariates were age (in years) and sex (levels: male or female). Our code is available at https://doi.org/10.5281/zenodo.5366705. The raw data we used to generate the results and reach the conclusions in this paper are third party data. Those data (except for household unit identifiers) are publicly accessible from The Article. Household unit identifiers were shared with us by the authors of The Article through personal communications. We did not have any special access privileges that others would not have. To access household unit identifiers, others can contact the corresponding author of The Article and ask them to share household unit identifiers as they did with us. The publicly available dataset did not include participants who had dropped out or those for whom reserved plasma was not available. Therefore, we analyzed available cases only.

The Article described the statistical methods as “a generalized estimating equation [GEE] procedure determined the significance of longitudinal changes […] using age and gender as co-variates”. We outline why we switched from GEE to LMM to reanalyze the data taking clustering and nesting into account. First, The Article reported a study with 11 clusters per intervention and 7 clusters per control (see S1 Fig). GEE is a population-averaged approach [6] using an asymptotic z test that assumes large sample sizes. Thus, GEE should be avoided in the analyses of cluster randomized trials with few clusters [7]. Specifically, GEE based methods use empirical-sandwich estimation for standard errors. When the degrees of freedom are limited, empirical-sandwich estimation leads to unreliable type I error rates in hypothesis testing. That is, when the number of groups per condition is small, the increased variability of the sandwich variance estimator substantially inflates the type I error [8–11]. As stated by Murray et al. “…GEEs may have only limited application in the context of group-randomized trials. The available evidence suggests that they be limited to trials having 20 or more groups allocated to each study condition” [8]. Additionally, there does not appear to be any off-the-shelf software to account for more than two levels of clustering in GEE based models while The Article reported a three-level design (visit, individual subject, household). Thus, it would not be possible to account for the clustering effect of households and the repeated measures using GEE as we did by LMM.

In our Table 1, we present the time x group interaction p-values as reported in The Article using GEE model (which were not reproducible because of discrepancies with the data), our results from corrected GEE model (still ignoring clusters as The Article did), our LMM model that still ignores clustering and nesting (LMM v1: still ignoring clusters, used to compare to LMM v2), and our corrected reanalysis using LMM with adjusted degrees of freedom, family clusters, and repeated measures all being taken into account (LMM v2: a valid approach to clustered data). In The Article, two statistical significance thresholds (<0.05 and ≤0.002) are set for various outcomes. Therefore, in our Table 1, we indicate important differences between the two latter models based on both 0.05 and 0.002 thresholds.

As a result of conducting analyses that account for clustering and nesting effects, we found that ignoring the clustering and nesting effects in the original analysis had resulted in anticonservative time x group interaction p-values, as is well-established in statistical methodological literature. Most time x group interaction p-values increased in our LMM analyses that accounted for nesting and clustering effects, and in the cases just below the statistical significance threshold (i.e., p = 0.05), p-values reported in The Article as statistically significant changed to not statistically significant. Of 13 statistically significant effects at the 0.05 level in LMM model where clustering and nesting effects were ignored (LMM v1), one became non-significant in LMM v2 (Threonine), and of the four statistically significant effects at the 0.002 level in LMM v1, one became non-significant in LMM v2 (Arginine Bioavailability Ratio).

In theory, ignoring clustering yields unbiased estimates of regression coefficients [12], given certain assumptions including that the cluster size is not correlated with cluster-specific treatment effects. Regression coefficients of the LMM and GEE models are presented in our Table 2. Regression coefficients of LMM v1 and LMM v2 are similar for fasting plasma ceramides and plasma sphingolipid bases, and slightly differ for amino acid metabolites. Because our purpose is to provide corrected statistical procedures within the context of methodologic choices of The Article, which involved null hypothesis significance testing based on p-values, we do not elaborate extensively on interpretation of model coefficient estimates.

Table 2. Time x group interaction regression coefficient from generalized estimating (GEE) and linear mixed effect models (LMM).

Outcome measures Corrected GEE a LMM v1 (without clustering/nesting) b LMM v2 (with clustering/nesting) c
Fasting plasma ceramides
Cer C14:0 -0.017 -0.017 -0.017
Cer C16:0 -0.023 -0.023 -0.023
Cer C18:0 -0.028 -0.028 -0.028
Cer C18:1 -0.005 -0.005 -0.005
Cer C20:0 -0.256 -0.256 -0.256
Cer C20:1 -0.001 -0.001 -0.001
Cer C22:0 -0.381 -0.381 -0.381
Cer C22:1 -0.004 -0.004 -0.004
Cer C24:0 -0.864 -0.864 -0.864
Cer C24:1 -0.325 -0.325 -0.325
T. Cer -1.955 -1.955 -1.955
Plasma sphingolipid bases
Sphinganine -0.057 -0.057 -0.057
Sphingosine -0.030 -0.030 -0.030
Dihydro-sphingosine-1-phosphate -0.001 -0.001 -0.001
Sphingosine -1-phosphate 0.042 0.042 0.042
Plasma amino acid metabolites
Arginine 17.623 17.802 17.831
Serine -42.790 -42.044 -42.102
Proline -75.779 -73.463 -73.439
Aspartate -13.730 -13.730 -13.671
Cystathionine -0.216 -0.158 -0.159
Sarcosine -8.849 -8.849 -8.831
Ornithine -37.477 -37.364 -37.293
Arginine Bioavailability Ratio 0.449 0.449 0.450
Lysine -44.646 -43.479 -43.510
Alanine -84.365 -82.806 -82.774
Glutamine -73.613 -72.368 -73.116
Threonine -43.160 -42.244 -42.785
Methionine -2.551 -2.331 -2.320
Fischer ratio 0.057 0.065 0.067
Citrulline -45.836 -45.637 -45.660
Histidine 1.297 1.470 1.471
Tryptophan -3.279 -2.808 -2.823
Leucine -9.533 -9.013 -8.944
Phenylalanine -8.408 -8.357 -8.364
Open in a new tab

a This model represents revised values after discussions with the authors of The Article. These coefficients are thus calculated with GEE model that was used by Mietus-Snyder et al., ignoring clustering and nesting.

b This LMM model does not account for clustering and nesting.

c This LMM model includes random intercepts for household units with adjusted degrees of freedom using between-within method.

Rigor and reproducibility (definitions provided in S1 File) are foundations of scientific advancement, both of which are critical for verification of the results generated using the reported methods. The results reported in The Article were generated using invalid statistical methods for the study design. Our analysis with valid methods produced relatively few differences in dichotomous statistically significant findings. Yet, regardless of the magnitude of difference that valid statistical tests make, the original results are not verifiable (definition provided in S1 File) and cannot be relied upon. Indeed, in similar studies with different numbers of clusters, different numbers of individuals within clusters, or effects closer to the null, the differences in statistical significance may be more or less pronounced. Post-hoc appraisal of how an invalid analysis compares to valid analyses in a particular sample does not justify the original use of the invalid approach.

Although we highlighted changes in statistical significance herein, we recognize that some argue against null hypothesis significance testing using p-values [13]. It is beyond the scope of the current reanalysis activity to discuss the relative value of using p-values or frequentist testing. Rather, we argue that if one chooses to conduct and publish a study that is predicated on frequentist testing and p-values (as the authors of The Article did), one should calculate, use, and interpret p-values correctly. Consequently, changes in statistical significance are simply a dichotomous marker of changes in variance estimates, and thus the same concerns regarding reproducibility and rigor would apply to interpreting confidence intervals because they are based on the same mathematical information.

Our reanalysis has some limitations. First, due to the reasons described in the methods section for switching from GEE to LMM approach, it was not possible to directly compare the findings of our reanalysis that did account for clustering and nesting with the approach used in The Article that ignored those effects. Rather, we needed to conduct the reanalysis using an approach that would allow for clustering and nesting to be accounted for (LMM), but ignore them first (LMM v1). We then compared LMM v1 with the proper analysis that accounted for clustering and nesting (LMM v2). The second limitation is related to the extent we can draw general conclusions about degrees of robustness. We did not conduct simulations or mathematical derivations to show the degrees of robustness on average. Thus, our results only show the degrees of relative robustness of the results and conclusions of this paper (i.e., The Article) when switching from an analysis with incorrect procedures to one with correct procedures. That is, we cannot make any judgement about how often such changes may or may not occur or how large the errors would be. Finally, we did not explore other techniques such as parametric bootstrap to explore how the results would be different compared to LMM. These limitations can be addressed in future research.

We commend Mietus-Snyder et al. for making their raw data publicly available, and their collegiality in providing additional data on family units. In order to be probative, studies need to be rigorously analyzed and transparently reported [14]. Although most conclusions in The Article remain unchanged, ignoring the clustering and nesting effects is an important and common methodological issue in obesity research that leads to unverifiable conclusions [4, 15]. Through our ability to verify and subsequent correcting of the results, we allow them to be used by the readers who might correctly dismiss results and conclusions of The Article because the analyses are conducted using improper statistical procedures for the design. The importance of statistical analysis per the unit of randomization to avoid similar errors in future studies is vital.

Supporting information

S1 Fig. Randomization of family units by Mietus-Snyder et al.

(TIF)

Click here for additional data file. (298.4KB, tif)
S1 File. Glossary.

Definitions of Reproducibility, Rigor, and Verifiability.

(DOCX)

Click here for additional data file. (14.3KB, docx)

Data Availability

The raw data used to generate the results and reach the conclusions in this paper are third party data. Those data (except for household unit identifiers) are publicly accessible from “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk,” doi: 10.1371/journal.pone.0240437 (reference 1 of this article). Household unit identifiers were shared with the authors of this article by the authors of “Randomized nutrient bar…” through personal communications. The authors of this article confirm they did not have any special access privileges that others would not have. To access household unit identifiers, others can contact the corresponding author of “Randomized nutrient bar…” and ask them to share household unit identifiers.

Funding Statement

Authors supported by the National Heart, Lung, and Blood Institute (R25DK099080, R25HL124208) awarded to D.A., and the Gordon and Betty Moore Foundation. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.

References

  • 1.Mietus-Snyder M, Narayanan N, Krauss RM, Laine-Graves K, McCann JC, Shigenaga MK, et al. Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk. PLoS One. 2020;15(10):e0240437. doi: 10.1371/journal.pone.0240437 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Murray DM, Varnell SP, Blitstein JL. Design and Analysis of Group-Randomized Trials: A Review of Recent Methodological Developments. American journal of public health. 2004;94(3):423–32. doi: 10.2105/ajph.94.3.423 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Chuang J-H, Hripcsak G, Heitjan DF. Design and Analysis of Controlled Trials in Naturally Clustered Environments: Implications for Medical Informatics. Journal of the American Medical Informatics Association. 2002;9(3):230–8. doi: 10.1197/jamia.m0997 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Brown AW, Li P, Bohan Brown MM, Kaiser KA, Keith SW, Oakes JM, et al. Best (but oft-forgotten) practices: designing, analyzing, and reporting cluster randomized controlled trials. The American Journal of Clinical Nutrition. 2015;102(2):241–8. doi: 10.3945/ajcn.114.105072 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Golzarri-Arroyo L, Chen X, Dickinson SL, Short KR, Thompson DM, Allison DB. Corrected analysis of ‘Using financial incentives to promote physical activity in American Indian adolescents: A randomized controlled trial’ confirms conclusions. PLoS One. 2020;15(6):e0233273. doi: 10.1371/journal.pone.0233273 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Zeger SL, Liang K-Y, Albert PS. Models for Longitudinal Data: A Generalized Estimating Equation Approach. Biometrics. 1988;44(4):1049–60. doi: 10.2307/2531734 [DOI] [PubMed] [Google Scholar]
  • 7.Li P, Redden DT. Small sample performance of bias-corrected sandwich estimators for cluster-randomized trials with binary outcomes. Statistics in medicine. 2015;34(2):281–96. Epub 2014/10/24. doi: 10.1002/sim.6344 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Murray DM. Design and analysis of group-randomized trials: Oxford University Press, USA; 1998. [Google Scholar]
  • 9.Murray DM, Hannan PJ, Wolfinger RD, Baker WL, Dwyer JH. Analysis of data from group-randomized trials with repeat observations on the same groups. Statistics in medicine. 1998;17(14):1581–600. Epub 1998/08/12. doi: . [DOI] [PubMed] [Google Scholar]
  • 10.Hayes RJ, Moulton LH. Cluster Randomised Trials: CRC Press; 2017. [Google Scholar]
  • 11.Murray DM. Design and analysis of group-randomized trials: A review of recent developments. Annals of Epidemiology. 1997;7(7, Supplement):S69–S77. 10.1016/S1047-2797(97)80009-9. [DOI] [Google Scholar]
  • 12.Demidenko E. Mixed models: theory and applications with R: John Wiley & Sons; 2013. [Google Scholar]
  • 13.Cohen J. The earth is round (p<. 05). American psychologist. 1994;49(12):997. [Google Scholar]
  • 14.Wood AC, Wren JD, Allison DB. The Need for Greater Rigor in Childhood Nutrition and Obesity Research. JAMA Pediatrics. 2019;173(4):311–2. doi: 10.1001/jamapediatrics.2019.0015 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.George BJ, Beasley TM, Brown AW, Dawson J, Dimova R, Divers J, et al. Common scientific and statistical errors in obesity research. Obesity. 2016;24(4):781–90. doi: 10.1002/oby.21449 [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision Letter 0

Eugene Demidenko

25 Jun 2021

PONE-D-21-11856

Ignoring the clustering effect changes some conclusions. Corrected analysis of: “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk”

PLOS ONE

Dear Dr. Jamshidi-Naeini,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Aug 07 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Eugene Demidenko, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments:

Before finding the reviewers I read your manuscript myself and came to conclusion that (a) results of the reanalysis are similar to the original ones, (b) more technical information is required to evaluate your contribution.

Specifically,

1. The most important results of statistical analysis are the coefficients of the model; the p-values have secondary importance. Therefore I suggest reporting the coefficients in Table 1, along with the p-values.

2. Although an overseen clustering may produce a lower level of statistical significance (larger standard errors and p-values) it yields unbiased coefficients (Demidenko, E. Mixed Models. Theory and Applications with R, Hoboken, NJ: Wiley, 2013). Moreover, while the original paper uses GEE you employ LMM. Consequently, the results on p-values must be different. Indeed, as follows from Table 1 the proportion of the changed p-values (rows) is only about 5:30. In spite the correct claim of the effect of ignored clustering the results look pale.

3. Some technical description of the statistical model (dependent, independent variables, equations) and how exactly the clustering has been implemented is missing. Therefore it is impossible to evaluate the improvements and your contribution.

Despite strong criticism and weak results I still invite you to resubmit but with no guarantee that your paper will be published.

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Competing Interests section:

"In past 12 months, Dr. Brown received grants through his institution from Alliance for Potato Research & Education, Egg Nutrition Center, National Cattlemen’s Beef Association, NIH/NHLBI, and NIH/NIDDK. He has been involved in research for which his institution or colleagues have received grants or contracts from Center for Open Science, Gordon and Betty Moore Foundation, Hass Avocado Board, Indiana CTSI, National Cattlemen’s Beef Association, NIH/NHLBI, NIH/NIA, and Sloan Foundation. His wife is employed by Reckitt Benckiser. In the last thirty-six months, Dr. Allison has received personal payments or promises for same from: American Society for Nutrition; Alkermes, Inc.; American Statistical Association; Big Sky Health, Inc.; Biofortis; California Walnut Commission; Clark Hill PLC; Columbia University; Fish & Richardson, P.C.; Frontiers Publishing; Gelesis; Henry Stewart Talks; IKEA; Indiana University; Arnold Ventures (formerly the Laura and John Arnold Foundation); Johns Hopkins University; Kaleido Biosciences; Law Offices of Ronald Marron; MD Anderson Cancer Center; Medical College of Wisconsin; National Institutes of Health (NIH); Medpace; National Academies of Science; Sage Publishing; The Obesity Society; Sports Research Corp.; The Elements Agency, LLC; Tomasik, Kotin & Kasserman LLC; University of Alabama at Birmingham; University of Miami; Nestle; WW (formerly Weight Watchers International, LLC). Donations to a foundation have been made on his behalf by the Northarvest Bean Growers Association. Dr. Allison was previously an unpaid member of the International Life Sciences Institute North America Board of Trustees.

Dr. Allison’s institution, Indiana University, and the Indiana University Foundation have received funds or donations to support his research or educational activities from: NIH; USDA; Soleno Therapeutics; Dynamic AQS; American Egg Board; California Walnut Commission, Almond Board; Peanut Institute; Mondelez; National Cattlemen’s Beef Association; Eli Lilly and Co.; Reckitt Benckiser Group PLC; Alliance for Potato Research and Education; Whistle Labs, Inc;  American Federation for Aging Research; Dairy Management Inc; Arnold Ventures; the Gordon and Betty Moore Foundation; the Alfred P. Sloan Foundation; and numerous other for-profit and non-profit organizations to support the work of the School of Public Health and the university more broadly. Other authors report no disclosures. "

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4. Please amend either the title on the online submission form (via Edit Submission) or the title in the manuscript so that they are identical.

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2022 Oct 27;17(10):e0275242. doi: 10.1371/journal.pone.0275242.r002

Author response to Decision Letter 0

16 Nov 2021

Dear Dr. Demidenko,

We appreciate your insightful comments on our manuscript. Our replies are in red below.

We noted a theme of incertitude about our scientific contribution and the scope of the current re-analysis activity. Thus, we provide some clarification herein and in the revised manuscript (the version with track changes) (lines 36-40).

Current manuscript is a re-analysis of an existing published study (i.e., referred to in the manuscript as ‘The Article’). Therefore, details regarding the significance, procedures, conduct, and conclusions of the study are provided in The Article. Our concern in the current manuscript is invalidity of the statistical analyses used for generating the published results. We present a valid statistical model and the corresponding results. We do not intend to address any research question or test any hypothesis beyond the results presented in The Article. Journals often handle this type of re-analysis as ‘correspondence’ so that it is linked to the previously published work. Because PLoS ONE editorial policy does not provide such an option, we were advised to submit our re-analysis as an original research article. Thus, our scientific contribution should be evaluated within the context of post-publication re-analysis and error correction.

• The most important results of statistical analysis are the coefficients of the model; the p-values have secondary importance. Therefore, I suggest reporting the coefficients in Table 1, along with the p-values.

• Although an overseen clustering may produce a lower level of statistical significance (larger standard errors and p-values) it yields unbiased coefficients (Demidenko, E. Mixed Models. Theory and Applications with R, Hoboken, NJ: Wiley, 2013).

The well-known consequences of ignoring clustering and nesting effects in the regression models for multilevel data are underestimated standard errors (and p-values) and unbiased regression coefficients [1, 2]. However, debate about the relative value of using p-values or frequentist testing, and whether regression coefficients or the p-values are superior in importance to one another is beyond the scope of this current reanalysis activity. Rather, the authors and the journal have chosen to publish a paper that was predicated on frequentist testing and p-values. Therefore, the testing, calculation, use, and interpretation of p-values should be conducted correctly, but in The Article they were not.

We present the model coefficients in supplementary Table 1.

• Moreover, while the original paper uses GEE you employ LMM. Consequently, the results on p-values must be different.

Thanks for this insightful comment. In lines 97-111 of the manuscript, we outline the reasons why LMM is the legitimate approach to correctly account for clustering and nesting effects given the study design by Dr. Mietus-Snyder et al. Because our purpose is to focus on accounting for vs. ignoring clustering and nesting, we updated our analyses and Table 1 to compare LMMs with and without clustering and nesting effects. Please see Table 1 and lines 97-111.

• Indeed, as follows from Table 1 the proportion of the changed p-values (rows) is only about 5:30. In spite the correct claim of the effect of ignored clustering the results look pale.

Rigor1 and reproducibility2 are foundations of science advancement, both being critical for verification of the results generated using the reported methods. The results reported by Mietus-Snyder et al. are not generated using valid statistical methods for the study design (our reasoning outlined in the manuscript and elsewhere by our group and others [3, 4]). Thus, regardless of the magnitude of difference that valid statistical tests would make, those results are not verifiable3 (please see lines 130-135 too). In this regard, we highlight the following matters:

i. Editors, journals, authors, and the scientific community overall should have a commitment to accuracy and integrity of the science [5], even if the results only change modestly as a result of doing correct analyses.

ii. Hearkening back to our argument about the choice of frequentist testing, correcting the analyses changes qualitative conclusions by the standard operating procedures and social norms of the use of frequentist testing and readers have a right to know this.

iii. Finally, for an astute reader, especially someone writing a blue-ribbon panel report that might utilize such a paper, or a meta-analyst, without seeing the critique and re-analysis results, they will correctly discern that the original results cannot be relied upon because the analyses are invalid. They might therefore be unable to use the study and its findings, even though the study and its findings (once correctly analyzed) do have value. That is, an unverified result is a result that cannot be utilized and relied upon fairly. By our verifying (and correcting) the results, we allow them to be used by people who correctly understand that the current analyses are not correct.

Additionally, and as a side note, the proportion of the changed p-values is essentially a function of what one takes as the denominator and the significance level. As mentioned in the manuscript, we describe this proportion as “of 13 statistically significant effects at the 0.05 level in LMM models where clustering and nesting effects were ignored, one became non-significant, and of the four statistically significant effects at the 0.002 level, one became non-significant”.

1Scientific rigor: “the strict application of the scientific method to ensure unbiased and well-controlled experimental design, methodology, analysis, interpretation and reporting of results.” (https://grants.nih.gov/policy/reproducibility/index.htm).

2Reproducibility: “An article is designated as reproducible if the [independent investigator] succeeds in executing the code on the data provided and produces results matching those that the authors claim are reproducible” [6].

3Verifiability: a study is said to be verifiable, and to have been verified, when: (a) the study is reproducible, and the results have been reproduced (by the definition of reproducibility above); and (b) a determination is made that the methods used to generate the results reproduced are valid methods.

• Some technical description of the statistical model (dependent, independent variables, equations) and how exactly the clustering has been implemented is missing. Therefore, it is impossible to evaluate the improvements and your contribution.

The technical description of our statistical model is presented in the manuscript lines 75-89. We have deposited our statistical code at https://doi.org/10.5281/zenodo.5366705.

References

1. Maas CJM, Hox JJ. The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics & Data Analysis. 2004;46(3):427-40. doi: https://doi.org/10.1016/j.csda.2003.08.006.

2. Ntani G, Inskip H, Osmond C, Coggon D. Consequences of ignoring clustering in linear regression. BMC Medical Research Methodology. 2021;21(1):139. doi: 10.1186/s12874-021-01333-7.

3. Brown AW, Li P, Bohan Brown MM, Kaiser KA, Keith SW, Oakes JM, et al. Best (but oft-forgotten) practices: designing, analyzing, and reporting cluster randomized controlled trials. The American Journal of Clinical Nutrition. 2015;102(2):241-8. doi: 10.3945/ajcn.114.105072.

4. Murray DM, Varnell SP, Blitstein JL. Design and Analysis of Group-Randomized Trials: A Review of Recent Methodological Developments. American Journal of Public Health. 2004;94(3):423-32. doi: 10.2105/AJPH.94.3.423.

5. ICMJE. Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals 2019. Available from: http://www.icmje.org/icmje-recommendations.pdf.

6. Peng RD. Reproducible research and Biostatistics. Biostatistics. 2009;10(3):405-8. doi: 10.1093/biostatistics/kxp014.

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file. (38.8KB, docx)

Decision Letter 1

Eugene Demidenko

13 Dec 2021

PONE-D-21-11856R1Ignoring the clustering effect changes some conclusions. Corrected analysis of: “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk”PLOS ONE

Dear Dr. Yasaman Jamshidi-Naeini,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

In general, I’m not satisfied with how the authors addressed my critique. Specifically,

  1. It should be mentioned in the paper that according to the theory (Demidenko, 2013) accounting for family clustering should not affect the coefficients but increase the standard errors and reduce the p-values. The current study confirms this expectation.

  2. I suggested reporting the coefficients themselves as well, but they are missing in the current version, again.

  3. The p-values from GEE and LME are not compatible because the underlying statistical models are different. Thus, we must compare LMM v1 versus LMM v2 to understand the effect of family clustering. As follows, the difference in p-values is negligible.

  4. Graphic presentation where x-axis is time would be very beneficial to see the differences between GEE, LMM v1, and LMM v2 models in addition to Table 1.

  5. Due to my previous point paper’s contribution is marginal. Yes, there is a slight improvement, but it is wrong to say that the analysis of “The Article” is incorrect. I suggest the language “adjustment” not saying that the previous analysis is wrong. This statement must be a part of the study limitations. Is it true that n=14 as follows from S1 Fig 1?

Please submit your revised manuscript by Jan 27 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Eugene Demidenko, Ph.D.

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2022 Oct 27;17(10):e0275242. doi: 10.1371/journal.pone.0275242.r004

Author response to Decision Letter 1

14 Jan 2022

Dear Dr. Demidenko,

Thank you for your insightful comments, and giving us the opportunity to improve the manuscript. Our replies are in blue below.

1. It should be mentioned in the paper that according to the theory (Demidenko, 2013) accounting for family clustering should not affect the coefficients but increase the standard errors and reduce the p-values. The current study confirms this expectation.

2. I suggested reporting the coefficients themselves as well, but they are missing in the current version, again.

Thank you for your cogent comment. We agree that as stated in (Demidenko, 2013), [under certain assumptions, including that the cluster size is not correlated with cluster-specific treatment effects], ignoring within-cluster correlation yields unbiased estimates of regression coefficients ((1), pages 42 and 157), but artificially reduced (more significant) standard errors and p-values (2). Though, of course, they could change in any one sample. Similarly, there appears to be little bias (less than ±10%) in the estimates of regression coefficients across the methods that accommodate clustered data for 10 or fewer clusters with few (e.g., 7-14) observations per cluster (3, 4).

We have now included coefficients in our manuscript per your suggestion (see Table 2 and lines 139-146). As we report in the manuscript, “regression coefficients of LMM v1 and LMM v2 are similar for fasting plasma ceramides and plasma sphingolipid bases, but slightly differ (3% or less) for amino acid metabolites.” However, because our purpose is to provide corrected statistical procedures within the context of methodologic choices of The Article (i.e., null hypothesis significance testing using p-values), we do not dwell extensively on the interpretation of regression coefficient estimates.

3. The p-values from GEE and LME are not compatible because the underlying statistical models are different. Thus, we must compare LMM v1 versus LMM v2 to understand the effect of family clustering. As follows, the difference in p-values is negligible.

Thank you for these remarks. We agree that the comparison must be between the results derived from similar approach. Owing to your insightful comments on the previous version of our manuscript, we included LMM v1 and we are comparing LMM v2 vs. LMM v1 both in Table 1 and to summarize our findings. We also agree that the model that ignores clustering (LMM v1) yields roughly the same answers as the model that accounts for clustering and nesting (LMM v2). As we outline in the manuscript, we believe regardless of the magnitude of difference that valid statistical tests would make, results that are derived from inappropriate procedures for the design (which is the case for either GEE or LMM v1) are unsubstantiated and not verifiable. That is, incorrect analysis sometimes yields the correct answers and correct analysis sometimes yields incorrect answers (due to type I and/or type II errors). The fact that LMM v1 yields roughly the same answer as LMM v2 does not mean that ignoring clustering (and nesting) is a negligible issue. Our reanalysis of this article now allows the scientific community to interpret the results in the context of an analysis strategy appropriate to the study design, increasing the confidence in its findings.

4. Graphic presentation where x-axis is time would be very beneficial to see the differences between GEE, LMM v1, and LMM v2 models in addition to Table 1.

Thank you for this comment. We emphasize that our manuscript is not intended to be a methods paper. We are making a critique of an existing published paper because the results of that paper are produced using inappropriate procedures for its design. While we argue that the results are unsubstantiated, we cannot show that they are wrong or right because doing so would require knowing the ‘true’ answer which would be possible by conducting simulations or mathematical derivations to show the degrees of robustness. We propose that keeping the values in our table format is optimal to present our reanalysis, as the original authors did in their paper.

5. Due to my previous point paper’s contribution is marginal. Yes, there is a slight improvement, but it is wrong to say that the analysis of “The Article” is incorrect. I suggest the language “adjustment” not saying that the previous analysis is wrong. This statement must be a part of the study limitations. Is it true that n=14 as follows from S1 Fig 1?

• Thank you for raising this point. We agree that our writing has not been as clear as it could have been. Our statements about the ‘invalid’ analysis in The Article could be interpreted in two ways: 1) that the procedures used to produce the findings were incorrect for the purported properties of the study design, 2) that the results produced by the procedures used are not the results which would be produced by proper procedures. We clarify that we mean the former, not the latter: when we say the analysis is either wrong or invalid, we mean the procedures used to produce the results were incorrect, and when we say results, we mean the findings produced by the procedures used. In this case, the findings from the incorrect procedures and proper procedures are similar. We now have clarified this in our writing.

• We also agree that it is wise to include a study limitations section. Thank you for that suggestion. We now have included the following study limitations section to the manuscript:

“Our reanalysis has some limitations. First, due to the reasons described in the methods section for switching from GEE to LMM approach, it was not possible to directly compare the findings of our reanalysis that did account for clustering and nesting with the approach used in The Article that ignored those effects. Rather, we needed to conduct the reanalysis using an approach that would allow for clustering and nesting to be accounted for (LMM), but ignore them first (LMM v1). We then compared LMM v1 with the proper analysis that accounted for clustering and nesting (LMM v2). The second limitation is related to the extent we can draw general conclusions about degrees of robustness. We did not conduct simulations or mathematical derivations to show the degrees of robustness on average. Thus, our results only show the degrees of relative robustness of the results and conclusions of this paper (i.e., The Article) when switching from an analysis with incorrect procedures to one with correct procedures. That is, we cannot make any judgement about how often such changes may or may not occur or how large the errors would be. Finally, we did not explore other techniques such as parametric bootstrap to explore how the results would be different compared to LMM. These limitations can be addressed in future research.”

• According to the information shared with us by the authors, 20 clusters (i.e., family units) were randomized, among which two families dropped out, and 18 clusters (n=36) were included in the analysis.

1. Demidenko E. Mixed models: theory and applications with R: John Wiley & Sons; 2013.

2. Johnson JL, Kreidler SM, Catellier DJ, Murray DM, Muller KE, Glueck DH. Recommendations for choosing an analysis method that controls Type I error for unbalanced cluster sample designs with Gaussian outcomes. Statistics in medicine. 2015;34(27):3531-45.

3. McNeish D, Stapleton LM. Modeling Clustered Data with Very Few Clusters. Multivariate Behavioral Research. 2016;51(4):495-518.

4. Maas CJM, Hox JJ. The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics & Data Analysis. 2004;46(3):427-40.

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file. (39KB, docx)

Decision Letter 2

Eugene Demidenko

5 Aug 2022

PONE-D-21-11856R2

Ignoring the clustering effect changes some conclusions. Corrected analysis of: “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk”

PLOS ONE

Dear Dr. Yasaman Jamshidi-Naeini,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I agree with the reviewer that incorporation of the family cluster effect does not change major conclusions. Consequently, I suggest the authors retitle the paper as follows: "Ignoring the clustering effect does not change major conclusions..." I understand that the authors may disagree with my decision and submit the paper to another journal. As you know, PLOS ONE is among a few scientific journals that publish "negative results" to avoid publication bias. Although adjusting for random effect is the right way to analyze the data it just happened that for this particular study mixed model did not make considerable difference. I encourage the authors to address the comments of the reviewer and resubmit the paper.

Please submit your revised manuscript by Sep 19 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Eugene Demidenko, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

********** 

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

********** 

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

********** 

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

********** 

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

********** 

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This proposed corrected analysis of the previous publication “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk” by Jamshidi-Naeini et al suggest that application of a clustered analysis changes some conclusions of the original article.

Three main questions emerge for the authors to consider:

I could not find any mention of the conclusions that were changed, only small changes in a minority of p-values. It is stated in the final paragraph that “most conclusions in The Article remain unchanged”.

1) Can the authors explain exactly which study conclusions are changed by clustered analyses and if any, what the significance of these changes to the article’s interpretation?

If none, the title might more appropriately read “Incorporation of the clustering effect in group randomized trial does not change the main study conclusions”. It is interesting that the reference 5 provided as an example of use of this clustered approach from the Allison group (Golzarri-Arroyo et al, 2020) similarly did not change study conclusions, but stated that accurately in the title.

Both of these PLOS ONE studies the Allison group has reanalyzed incorporating the clustering effect (Mietus-Snyder et al and Short et al) involved minority adolescent populations, who would be at heightened risk for socioeconomic disadvantage. Unfortunately, an important issue relevant to health disparity that a statistical manipulation alone cannot address, is fragmentation of traditional family units and potential breakdown in the predictability of household factors.

2) Can the authors delineate which household factors are felt to be most critical in justification of the proposed clustered analysis? If dietary factors pertain, does clustered analysis reveal any difference in the diets that were reported to be uniformly poor for both adolescents and adults in the original study? If genetic or epigenetic factors are implicated, what effects do nonbiological family units or shared custody have on clustered analyses? These issues raise an important point of discussion that might be expanded upon by the authors to help explain why the application of clustered analysis – at least in this study and the study reanalyzed in reference (5) does not appear to change the main study conclusions reached without clustering.

There are two sets of changes reported upon in this Correction – the first set due to an error in the original time-by-group p-values reported that has apparently been acknowledged by the study authors. It is not apparent why those p-value corrections have not already been published as a Correction as this should be the responsibility of the original study team, once alerted to their error. The title of this Correction by Jamshidi-Naeini implies that it is the application of clustered analysis, not these time x group interaction p-value corrections that changes some conclusions. It does seem apparent that the corrected time by group p-values do not change study conclusions (with the exception of aromatic amino acids tryptophan and phenylalanine, the latter still showing a trend with p = 0.07). This is not however a central observation and does not impact the article’s interpretation.

It is the second set of p-value changes that are contingent on the newly applied clustered analysis that would still be germane to the Correction submitted by Jamshidi-Naeini et al. The authors support their claim that this approach overcomes an important methodological issue in obesity research with several recent reviews on this topic. Their point is well taken, but the frequent use the terms “correctly” and “appropriately” in describing the application of a clustered analysis seems unnecessarily pejorative, since as already noted, the reanalysis has not actually been demonstrated to change the study’s conclusions. It of course changed p-values, as would be expected with a decrease in the number of independent observations, but only a small number of p-values that were nominally significant become slightly insignificant. Trends remained trends and the strongest observations, on which the original study conclusions are based, remain very significant.

A 2019 paper cited (ref 6) for which DB Allison is also senior author on “The Need for Greater Rigor in Childhood Nutrition and Obesity Research” states what Jamshidi-Naeini et al were sure to observe in that (quoting from ref 6) “Clustering reduces power”… going on to say “there are instances in which misanalysis of clustered childhood obesity interventions has led to unsubstantiated conclusions.” This certainly is true, but the authors have not demonstrated that it is true in this instance – which circles back to the first two questions above. Which study conclusions have actually been altered by the clustered analysis? And if none, why not?

A full reading of the original paper reveals a detailed baseline principal components analysis to reduce the dimensionality of the complex lipidomic data set incorporated in baseline correlograms with traditional cardiometabolic risk factors and lipoprotein subspecies. There are also change correlation analyses that reveal numerous hypothesis-generating associations independent of pre-post metabolite p-values. Much of the discussion in the original study is rooted in the correlograms that summarize this body of data.

3) Would clustered analysis lend any further insight to the principal component and correlation analyses? Can the authors explain why their reanalysis does not address these two prominent elements of the original paper?

One small point: It is stated in the Abstract and Introduction that this was a “recent” study but the original report states that the trial was completed in the summer of 2011. Given the 9-year lag to publication, the metabolomic and lipidomic analyses performed, as stated in the original paper, on samples preserved at -70o F, were however surely more recent when mass spectroscopy technologies to explore these novel biomarkers became more widely available.

Reviewer #2: The current investigators pointed out a possible design flaw in the manuscript. That is to say “In a published randomized controlled trial, household units were randomized to a nutrient bar supplementation group or a control condition, but the non-independence of observations within the same household (i.e., the clustering effect) was not accounted for in the statistical analyses." Assuming this to be the case, the investigators provided a side by side comparison of results (clustering vs. non clustering) and proceeded accordingly. The results are very similar. However, that would have to be debated with the original investigators and not this reviewer. The work was thorough and certainly appeared to be presented reasonably in a descriptive, and not methodologic, fashion as was pointed out by the investigators.

********** 

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2022 Oct 27;17(10):e0275242. doi: 10.1371/journal.pone.0275242.r006

Author response to Decision Letter 2

19 Aug 2022

Dear Dr. Demidenko,

Thank you for your insightful comments, and giving us the opportunity to improve the manuscript. Our replies are in blue below.

Response to the Editor:

I agree with the reviewer that incorporation of the family cluster effect does not change major conclusions. Consequently, I suggest the authors retitle the paper as follows: "Ignoring the clustering effect does not change major conclusions..." I understand that the authors may disagree with my decision and submit the paper to another journal. As you know, PLOS ONE is among a few scientific journals that publish "negative results" to avoid publication bias. Although adjusting for random effect is the right way to analyze the data it just happened that for this particular study mixed model did not make considerable difference. I encourage the authors to address the comments of the reviewer and resubmit the paper.

Response: Thank you for these comments. We note that the fact that “[…] it just happened that for this particular study mixed model did not make considerable difference” is only known after we have reanalyzed the data using correct statistical procedures. That is, the results and conclusions published by Mietus-Snyder et al. are unsubstantiated because they are drawn from incorrect statistical methods. As a result of our reanalysis, results and conclusions derived from the original GEE models are verified or corrected. We have addressed the point about the title in our responses to reviewer 1.

Response to Reviewer 1:

1. I could not find any mention of the conclusions that were changed, only small changes in a minority of p-values. It is stated in the final paragraph that “most conclusions in The Article remain unchanged”. Can the authors explain exactly which study conclusions are changed by clustered analyses and if any, what the significance of these changes to the article’s interpretation? If none, the title might more appropriately read “Incorporation of the clustering effect in group randomized trial does not change the main study conclusions”. It is interesting that the reference 5 provided as an example of use of this clustered approach from the Allison group (Golzarri-Arroyo et al, 2020) similarly did not change study conclusions, but stated that accurately in the title.

Response: Thank you for raising the point about the title. We agree that a change in the title would present the consequences of accounting for clustering and nesting effects in this particular sample more clearly. We changed the title to “Accounting for the clustering and nesting effects verifies majority of conclusions. Corrected analysis of: “Randomized nutrient bar …””. The suggested title by the reviewer “Ignoring the clustering effect does not change major conclusions..." implies that we are encouraging researchers to ignore clustering (and nesting) in cluster randomized trials, which is not our intended message.

We state in our manuscript that most conclusions remain unchanged when the comparison is between linear mixed effects models (LMM) that account for clustering and nesting and LMMs that ignore clustering and nesting. We precisely state variables for which p-values are changed from statistically significant in the incorrect analysis to insignificant when clustering and nesting are accounted for (Threonine and Arginine Bioavailability Ratio). The changes we made to the title will further address this matter. On the other hand, addressing the consequence of accounting for clustering and nesting on the article’s conclusions and interpretations should occur under the assumption that the article’s conclusions and interpretations are based on reproducible results, but they are not. Such comparison would require re-writing the article’s interpretations based on a) “corrected GEE” models and b) consistent specification of the significance level (i.e., 0.002 or 0.05). This is, as correctly mentioned by the reviewer, a “responsibility of the original study team”.

• Both of these PLOS ONE studies the Allison group has reanalyzed incorporating the clustering effect (Mietus-Snyder et al and Short et al) involved minority adolescent populations, who would be at heightened risk for socioeconomic disadvantage. Unfortunately, an important issue relevant to health disparity that a statistical manipulation alone cannot address, is fragmentation of traditional family units and potential breakdown in the predictability of household factors.

Response: Thank you for raising this important matter. We agree that socioeconomic disadvantage and health disparities among adolescences have critical importance. We agree that our reanalysis efforts do not address health disparity issues among adolescents. We do not intend to address any research question with that regard in our manuscript. In this manuscript we are making a critique of an existing published paper because the results of that paper are produced using inappropriate procedures for its design.

2. Can the authors delineate which household factors are felt to be most critical in justification of the proposed clustered analysis? If dietary factors pertain, does clustered analysis reveal any difference in the diets that were reported to be uniformly poor for both adolescents and adults in the original study? If genetic or epigenetic factors are implicated, what effects do nonbiological family units or shared custody have on clustered analyses? These issues raise an important point of discussion that might be expanded upon by the authors to help explain why the application of clustered analysis – at least in this study and the study reanalyzed in reference (5) does not appear to change the main study conclusions reached without clustering.

Response: Thank you for raising these interesting matters. These questions present the potential to expand current knowledge and understanding of the factors that may explain correlation of observations within household units in cluster randomized trials. From a methods aspect, the effect of ignoring clustering would be more moderate in designs with more clusters and fewer individual participants per cluster compared to designs with fewer clusters and larger number of individuals per cluster. Situations in which families are randomly assigned to treatments are examples of smaller cluster sizes, such as the article under question here and the study we referred to in Golzarri-Arroyo et al. Any subjective explanation about dietary or genetic factors and their potential role in similarities or dissimilarities within or among households in this particular sample would be speculation, and not addressable within the context of our reanalysis.

• There are two sets of changes reported upon in this Correction – the first set due to an error in the original time-by-group p-values reported that has apparently been acknowledged by the study authors. It is not apparent why those p-value corrections have not already been published as a Correction as this should be the responsibility of the original study team, once alerted to their error. The title of this Correction by Jamshidi-Naeini implies that it is the application of clustered analysis, not these time x group interaction p-value corrections that changes some conclusions. It does seem apparent that the corrected time by group p-values do not change study conclusions (with the exception of aromatic amino acids tryptophan and phenylalanine, the latter still showing a trend with p = 0.07). This is not however a central observation and does not impact the article’s interpretation.

It is the second set of p-value changes that are contingent on the newly applied clustered analysis that would still be germane to the Correction submitted by Jamshidi-Naeini et al. The authors support their claim that this approach overcomes an important methodological issue in obesity research with several recent reviews on this topic. Their point is well taken, but the frequent use the terms “correctly” and “appropriately” in describing the application of a clustered analysis seems unnecessarily pejorative, since as already noted, the reanalysis has not actually been demonstrated to change the study’s conclusions. It of course changed p-values, as would be expected with a decrease in the number of independent observations, but only a small number of p-values that were nominally significant become slightly insignificant. Trends remained trends and the strongest observations, on which the original study conclusions are based, remain very significant.

Response: Thank you for your cogent comment. 1) We agree that it is the responsibility of the original authors and the journal to correct the irreproducibility issue of the article. Both have been notified that a portion of the analyses in the article are not reproducible. We cannot comment on the reason why these errors have not been publicly acknowledged yet. 2) Our altered title should address the issue raised about the title. 3) The purpose of our manuscript is making a critique of an existing published paper because the results of that paper are produced using inappropriate procedures for its design. We have been factual throughout: majority of conclusions that are derived from correct procedures are the same as those derived from incorrect procedures. 4) Until appropriate corrections are made to the article, any comparison of our results with the article’s interpretations would require specification of which model (LMM or GEE) one is comparing our results with and which significance level (0.05 or 0.002) they are using. 5) We understand that referring to our analysis as a ‘correct’ or ‘appropriate’ analysis may be disapproved by some. We thus modified the language where our message was still clear without those words.

• A 2019 paper cited (ref 6) for which DB Allison is also senior author on “The Need for Greater Rigor in Childhood Nutrition and Obesity Research” states what Jamshidi-Naeini et al were sure to observe in that (quoting from ref 6) “Clustering reduces power”… going on to say “there are instances in which misanalysis of clustered childhood obesity interventions has led to unsubstantiated conclusions.” This certainly is true, but the authors have not demonstrated that it is true in this instance – which circles back to the first two questions above. Which study conclusions have actually been altered by the clustered analysis? And if none, why not?

Response: Thank you for these remarks. There should be a distinction between “unsubstantiated” results vs. “incorrect” results. There are two approaches to consider this matter: 1) that the procedures used to produce the findings were incorrect for the purported properties of the study design, 2) that the results produced by the procedures used are not the results which would be produced by proper procedures. We clarify that we mean the former, not the latter: when we say the analysis is either wrong or invalid, we mean the procedures used to produce the results were incorrect, and when we say results, we mean the findings produced by the procedures used. In this case, the findings from the incorrect procedures and proper procedures are similar.

Results and conclusions of the article are unsubstantiated (i.e., unsupported) without our reanalysis. That is, regardless of the magnitude of difference that valid statistical tests would make, results that are derived from inappropriate procedures for the design are unsubstantiated and not verifiable. Incorrect analysis sometimes yields the correct answers and correct analysis sometimes yields incorrect answers (type I and/or type II errors). While we argue that the results are unsubstantiated, we cannot show that they are wrong or right because doing so would require knowing the ‘true’ answer which would be possible by conducting simulations or mathematical derivations to show the degrees of robustness.

3. A full reading of the original paper reveals a detailed baseline principal components analysis to reduce the dimensionality of the complex lipidomic data set incorporated in baseline correlograms with traditional cardiometabolic risk factors and lipoprotein subspecies. There are also change correlation analyses that reveal numerous hypothesis-generating associations independent of pre-post metabolite p-values. Much of the discussion in the original study is rooted in the correlograms that summarize this body of data. Would clustered analysis lend any further insight to the principal component and correlation analyses? Can the authors explain why their reanalysis does not address these two prominent elements of the original paper?

Response: We did not intend to reanalyze every analysis in the article, and do not claim as such in our manuscript. Principal components analysis is out of the scope of our discussion in this manuscript.

Response to Reviewer 2:

The current investigators pointed out a possible design flaw in the manuscript. That is to say “In a published randomized controlled trial, household units were randomized to a nutrient bar supplementation group or a control condition, but the non-independence of observations within the same household (i.e., the clustering effect) was not accounted for in the statistical analyses." Assuming this to be the case, the investigators provided a side by side comparison of results (clustering vs. non clustering) and proceeded accordingly. The results are very similar. However, that would have to be debated with the original investigators and not this reviewer. The work was thorough and certainly appeared to be presented reasonably in a descriptive, and not methodologic, fashion as was pointed out by the investigators.

Response: Thank you for reviewing our manuscript. We notified the original research team and the journal of the issues we detected.

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file. (28.8KB, docx)

Decision Letter 3

Eugene Demidenko

13 Sep 2022

Accounting for clustering and nesting effects verifies most conclusions. Corrected analysis of: “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members [...]”

PONE-D-21-11856R3

Dear Dr. Yasaman Jamshidi-Naeini,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Eugene Demidenko, Ph.D.

Academic Editor

PLOS ONE

Acceptance letter

Eugene Demidenko

19 Oct 2022

PONE-D-21-11856R3

Accounting for clustering and nesting effects verifies most conclusions. Corrected analysis of: “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members [...]”

Dear Dr. Jamshidi-Naeini:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Eugene Demidenko

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Fig. Randomization of family units by Mietus-Snyder et al.

    (TIF)

    Click here for additional data file. (298.4KB, tif)
    S1 File. Glossary.

    Definitions of Reproducibility, Rigor, and Verifiability.

    (DOCX)

    Click here for additional data file. (14.3KB, docx)
    Attachment

    Submitted filename: Response to Reviewers.docx

    Click here for additional data file. (38.8KB, docx)
    Attachment

    Submitted filename: Response to Reviewers.docx

    Click here for additional data file. (39KB, docx)
    Attachment

    Submitted filename: Response to Reviewers.docx

    Click here for additional data file. (28.8KB, docx)

    Data Availability Statement

    The raw data used to generate the results and reach the conclusions in this paper are third party data. Those data (except for household unit identifiers) are publicly accessible from “Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk,” doi: 10.1371/journal.pone.0240437 (reference 1 of this article). Household unit identifiers were shared with the authors of this article by the authors of “Randomized nutrient bar…” through personal communications. The authors of this article confirm they did not have any special access privileges that others would not have. To access household unit identifiers, others can contact the corresponding author of “Randomized nutrient bar…” and ask them to share household unit identifiers.

    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES