Statistical considerations in reporting cardiovascular research

Abstract

The problem of inadequate statistical reporting is long standing and widespread in the biomedical literature, including in cardiovascular physiology. Although guidelines for reporting statistics have been available in clinical medicine for some time, there are currently no guidelines specific to cardiovascular physiology. To assess the need for guidelines, we determined the type and frequency of statistical tests and procedures currently used in the American Journal of Physiology-Heart and Circulatory Physiology. A PubMed search for articles published in the American Journal of Physiology-Heart and Circulatory Physiology between January 1, 2017, and October 6, 2017, provided a final sample of 146 articles evaluated for methods used and 38 articles for indepth analysis. The t-test and ANOVA accounted for 71% (212 of 300 articles) of the statistical tests performed. Of six categories of post hoc tests, Bonferroni and Tukey tests were used in 63% (62 of 98 articles). There was an overall lack in details provided by authors publishing in the American Journal of Physiology-Heart and Circulatory Physiology, and we compiled a list of recommended minimum reporting guidelines to aid authors in preparing manuscripts. Following these guidelines could substantially improve the quality of statistical reports and enhance data rigor and reproducibility.

INTRODUCTION

Measuring variables of physiology is a foundation of cardiovascular research, and analyzing physiological measurements involves statistics. With increasing discussion over rigor and reproducibility (66, 135), the goals of these guidelines are to provide best practice information regarding statistical analysis and to recommend how to report statistics for cardiovascular physiology research. Up to 50% of studies are not reproducible, perhaps in part because the statistical analyses cannot be evaluated from the information given (8). Potential issues with statistics include studies that lack adequate statistical power, use inappropriate statistical tests, fail to confirm test assumptions, fail to account for and explain outlying values or missing data, and do not consider units of analysis. Adequate reporting of statistics will help to determine if any of these issues are applicable.

This article focuses on the statistics used in cardiovascular physiology research. Here, we review the most commonly used tests in American Journal of Physiology-Heart and Circulatory Physiology publications and summarize current best practices. We provide a checklist for authors to use in designing experiments and writing manuscripts and for reviewers to use in assessing the statistical tests and procedures reported in manuscripts. In addition, the reference section is a resource for those who want to learn more about the technical aspects of statistical approaches, which are not discussed in detail here.

We focused on statistical use in animal research, which is the majority of research reported in this journal. For statistical guidelines for clinical research, please see the recent Guidelines for the Content of Statistical Analysis Plans in Clinical Trials published by the Journal of the American Medical Association and other resources (60, 91, 137). Our guidelines add to previous guidelines on statistical use (41, 43, 44) and dovetail with recent efforts by the American Journal of Physiology-Heart and Circulatory Physiology to provide guidelines for articles on antibody use, recording sympathetic nerve activity, animal models of myocardial ischemia and infarction, and cardiac physiology measurements (19, 76, 115, 116).

MOST COMMONLY USED STATISTICAL TESTS IN AMERICAN JOURNAL OF PHYSIOLOGY: HEART AND CIRCULATORY PHYSIOLOGY PUBLICATIONS

We assessed articles published by the American Journal of Physiology-Heart and Circulatory Physiology to identify the most commonly used statistical tests and to evaluate current practices in reporting statistics. The search included all 2017 journal articles published in the American Journal of Physiology-Heart and Circulatory Physiology, from January 1 to the date of the search (October 6, 2017). Articles were identified from PubMed using the search term “[journal] Am J Physiol Heart Circ Physiol.” Of these 254 articles, those concerning corrections, errata, reviews, editorials, and articles in press were excluded, leaving 160 original research articles, of which all were downloaded for evaluation of methods used. Of these downloaded articles, 40 articles were chosen by formal random selection for an additional, indepth evaluation of the statistics used. Of the 160 articles, 14 articles were not evaluated because they were false positive selections (8 editorials, 1 historical perspective, and 5 computational or modeling articles that used no statistics), leaving 146 articles evaluated for statistical methods used and 38 articles for the indepth analysis (1, 4, 5, 7, 9–18, 21–23, 25, 26, 45–59, 61, 63, 64, 67–69, 71–75, 77–80, 82–90, 92–97, 99–101, 103, 104, 108–114, 117–121, 123–126, 128–131, 133, 136, 138–145, 147–159, 162–183, 186–189, 192–199). Three evaluators abstracted the data and performed the analysis (G. A. Gray, M. L. Lindsey, and S. K. Wood). To assess consistency across evaluators, 20 of 146 articles (5 of 38 articles) were randomly selected and analyzed twice (by G. A. Gray and M L. Lindsey); all had good degree of concordance. Both analyzers identified the same statistical tests and were identical with the indepth evaluations of the details provided by authors.

We identified six categories of statistical tests: ANOVA, chi-square tests, regression, t-tests, other two-sample tests, and other. Of the 300 tests, the t-test and ANOVA accounted for 212 articles (71%; Table 1). Because the statistics details were grouped, it was difficult to ascertain how many cases there were where multiple t-tests were used when ANOVA was appropriate. There were only a few cases (<5) where we had suspicions that a t-test had been used instead of ANOVA. Overall, authors appeared to understand what tests are appropriate to use or reviewers are requesting corrections during peer review. For the other test category, the most frequent tests were the Shapiro-Wilk normality test, the Kolmogorov-Smirnov normality test, and Bland-Altman analysis, accounting for 13 of 36 other tests (36%). Of the seven post hoc tests used, including Bonferroni, Dunnett, Holm Sidak, least significant difference, Student-Newman-Keuls, Tukey, and other, Bonferroni and Tukey post hoc tests accounted for 62 of 98 articles (63%; Table 2).

Table 1.

Statistical procedures used in American Journal of Physiology-Heart and Circulatory Physiology articles published between January 1, 2017, and October 6, 2017

Procedure	%
Analysis of variance	40
t-tests	31
Another two-sample test	7
Regression analyses	9
Chi-square tests	2
Other	11
Total	100

Table 2.

Frequency of post hoc tests used after ANOVA in American Journal of Physiology-Heart and Circulatory Physiology articles published from January 1, 2017, to October 6, 2017

Procedure	%
Bonferroni	33
Tukey	31
Dunnett	12
Student-Newman-Keuls	8
Least significant difference	6
Holm-Šídák	5
Other	5

Standard error of the mean (SE) was used to report error 82% of the time (n = 31 of 38), as opposed to four uses of the standard deviation (SD) and three cases where the type of error reported was not identified or other was used (Table 3). All 38 articles named the tests used, and of the 146 articles evaluated, only 1 article did not report what test(s) had been used. The statistical software program was reported in 66% of the 38 articles, with GraphPad Prism (https://www.graphpad.com/) and SPSS Software (IBM) accounting for 80%. Actual sample sizes for each group were reported 79% of the time; the remaining articles reported sample sizes as a range (e.g., n = 6−8 per group). The P value was reported as <0.05 for 89%, and different P value thresholds (i.e., assigning differences among P < 0.05, P < 0.01, and P < 0.001) were reported in 47% of the articles. The assumption of normality was tested for 21% of the articles, but a power analysis was reported in only 3%. In most cases, information on whether normality testing or power analysis had been completed was not provided. Practices that are not good habits in clinical research, including optional stopping or not following sequential analysis rules, could not be evaluated based on the information provided (62). Whether there is a proclivity toward collecting data until significance is reached may be an issue for animal and in vitro research. Overall, this analysis highlights that while most groups appear to be using statistics appropriately, more detailed instructions are needed on what should be reported.

Table 3.

Frequency of reporting details in the 38 American Journal of Physiology-Heart and Circulatory Physiology articles evaluated

Procedure	%
Standard error of the mean reported	82
Statistical software identified	66
Sample size listed for each individual group	79
P value reported as threshold (P < 0.05 vs. exact P value)	89
Spurious precision	47
Tests for normality reported	21
Power analysis reported	3

GUIDELINES FOR REPORTING STATISTICS: MINIMUM DETAILS NEEDED

The minimum information we recommend for reporting statistical analyses comes from several sources (Table 4) (42, 43, 105–107, 127, 132, 134). This advice is in line with published guidelines, including Animals in Research: Reporting In Vivo Experiments (ARRIVE) guidelines (98). Having a stand-alone statistical section in the methods may not be the best way to allow rigorous assessment and reproducibility of findings. Instead, incorporating statistical information in individual sections in the methods and figure and table legends may be more appropriate. Other options include hosting analysis scripts, data, and more detailed information (e.g., degrees of freedom and F-ratio) on repository sites such as FigShare and Open Science Framework (https://cos.io/our-products/osf/) (160).

Table 4.

Minimum requirements checklist for reporting statistical analyses; we recommend that the details shown be provided in manuscripts to allow the data and the study’s reproducibility to be assessed

Guideline
Experimental design	Define:
	•Hypothesis tested and purpose of the statistical analysis
	•Variables, groups, sample sizes (preferably determined by power analysis), sample randomization, and significance (α) level
methods	Provide details on:
	•Name and version of the statistical software used
	•Any procedures taken to modify raw data before analysis (e.g., transformation, ratios, combining categories)
	•Which tests were used for which comparisons, including post hoc tests for ANOVA and whether corrections were made for multiple comparisons
	•Ancillary analyses (assumptions testing as well as identification and treatment of outliers and missing values)
	•Data and details of statistical analysis should be available for requests to assess reproducibility on open repository sites (3)
results	Report:
	•Precise P values to two (for 1.0 to 0.01) or three (for 0.009 to 0.001) decimal places; precision below P < 0.001 not needed except for genetic associations
	•Variability reported using standard deviation
	•Confidence intervals
	•Data with appropriate scientific precision (e.g., report body weight with no significant digits after the decimal point)
	•Upload source data into a public repository (e.g., Figshare, https://figshare.com/) at submission
Table and figure legends	•Name tests used and sample sizes for each group in figure legends and tables
	•Provide information on sex of animals used, unless only one sex is stated in the methods
	•Data visualization: use box and whisker plots or similar instead of column graphs, to show individual responses; consider clarity of information presented

Guidelines were compiled from Refs. 42, 43, 106, and 107.

The use of P value thresholds (e.g., P < 0.05) reflects both historical, formal statistical theory and practice and the fact that P values were obtained using tables because of computational limitations. Reporting exact P values rather than threshold values is important for assessing reproducibility; this is particularly true when the P value is in the range of 0.01−0.10. For example, a P value of 0.04 in one study is statistically significant, whereas a P value of 0.06 in a replicate study is not. Reproducibility issues would arise if the only information provided were whether the threshold for significance was met. At the same time, reporting exact P values in an attempt to say that one comparison is more significant (has a lower P value) than another comparison is not appropriate.

SD should be reported when one replicate measurement is made for each data point. For example, if blood pressure is acquired once for each subject, SD should be reported. SE should be used when multiple measurements are made for each data point. For example, if blood pressure is acquired multiple times for each subject and averaged, the SE should be reported. Interquartile range is another way to show variability within a group. Confidence intervals provide details on the uncertainty about the true value of the population and keep the interpretation focused on the physiology and not merely on statistical probabilities (or chance) as an explanation for differences.

Using box and whisker plots or similar graphs to show individual responses instead of bar graphs is recommended for data visualization (190, 191). This will allow readers to assess the variation in individual responses. Showing individual responses may not be practical and may reduce clarity; for example, when using multiple line graphs, such as in articles by Brooks et al. and Zhang et al. (20, 200) We recommend that the authors select graphs that best represent the data reported.

COMMON STATISTICAL TESTS

Analysis plans should be chosen a priori, and contingency plans should be set in case there are violations of assumptions of the original tests (see http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf for more details). Flow charts can be used to determine which descriptive statistics and tests may be most appropriate for analyzing a data set, for example, figures in Rosner’s Fundamentals of Biostatistics (146). Table 5 shows a list of the common statistical tests with descriptions and assumptions. More details on these concepts can be found in the Exploration in Statistics series published by Advances in Physiology Education (29–39, 41). Additional resources also provide more details on specifics of individual tests (185). In addition, we highly recommend that a statistician be consulted as needed. All statistical tests have assumptions, so it is important to determine whether your data meet the assumptions of the analysis and whether the results of your statistical analysis are meaningful. There are a number of tests that can be performed to assess analysis quality; for example, test statistics, testing for residuals, and testing for colinearity. While not commonly used in the analysis of cardiovascular physiology, there are additional details that can be reported, including the coefficient of multiple determination, degrees of freedom, and measures of goodness of fit.

Table 5.

Common statistical tests

Test	Description	Assumptions
Descriptive statistics	Measures of center [mean (arithmetic average) and median (value in the middle)] and variability (standard deviation, mean, or median absolute deviation and IQR)	May need to be normalized; standard deviation for single measurements, IQR for data not normally distributed
One-sample comparisons	Used to evaluate a single-group one-sample t-test (parametric) and one-sample χ2-test for variances	Variables continuous, data independent, randomly selected; and normally distributed; no outliers
Two-group comparisons t-test	Used to evaluate two groups:	All; no outliers
	•Paired t-test (Wilcoxon signed-rank test is the nonparametric version)•Unpaired t-test (Mann-Whitney U-test is the non-parametric version)	•Parametric; dependent variable is continuous; subjects paired or dependent; data normally distributed or sample size large enough that central limit theorem is satisfied; homogeneity of variance; if unequal variation, log transform or use Wilcoxon signed-rank test
		•Parametric; dependent variable is continuous; independent variable is categorical; dependent variable normally distributed (or sample size large enough that central limit theorem is satisfied) and randomly selected; observations are independent
Chi-square test	•Association: determines whether the observed distribution differs from chance	Nonparametric; variables are independent; relatively large sample size (minimum expected n >5 for each group; if n < 50 for 2 × 2 table, use Fisher’s exact test)
	•Goodness of fit: determines whether an observed distribution differs from known distribution.
Kaplan-Meier	Time to an event (e.g., survival) analysis; can accommodate censored data; nonparametric log-rank test used to compare distributions	Data independent; time intervals uniform and clearly defined; censoring similar between groups
Regression	Predicts the value of one variable from a predictor (univariate) or ≥2 predictors (multivariate)	Variables are multivariate; little or no multicollinearity; limited autocorrelation; homogeneity of variance
	•Linear regression: correlation coefficients
	•Deming regression: line of best fit for a two-dimensional data set
	•Logistic regression: odds ratio (with 95% confidence intervals)
Bland-Altman plot	Analyzes agreement between two different assays	Data independent, randomly selected; and normally distributed
≥3-group comparison analysis of variance	Test for differences of means among groups	Continuous dependent variable; categorical independent variable; independent observations; data randomly sampled; dependent variables are normally distributed or sample size large enough that the Central Limit Theorem is satisfied (use log or arcsin transformation for data not normally distributed); homogeneity of variance; no outliers
	•One-way: 1 variable examined
	Multiway; ≥2 variables examined
	•Repeated measures: over time, dose range
	•Nonparametric: Kruskal-Wallis and Friedman
	Post tests evaluate which groups are different. The following are examples.
	•Parametric: Bonferroni, Duncan, Dunnett, False discovery rate, Student-Newman-Keuls, Fisher least significant difference, Sidak, Holm-Sidak, and Tukey
	•Nonparametric: Dunns

IQR, interquartile range.

Determination of Statistical Power

Power analyses should be done during the experimental design process to estimate the sample size needed to detect a difference that is scientifically important (27). Sample sizes that are too large wastes resources, whereas sample sizes that are too low are subject to false negative results (type II error). There is also a balance between theoretically ideal and practically feasible that needs to be considered when designing experiments. There are a number of online calculators for power analysis that are easy to use, including http://powerandsamplesize.com/, http://clincalc.com/stats/samplesize.aspx, and https://www.stat.ubc.ca/~rollin/stats/ssize/n2.html. The main assumption of the power analysis is that the data involve random sampling. Two other considerations are 1) the power analysis is performed a priori to set a preplanned sample size and 2) the effect size is the smallest of interest rather than a preobserved value. A more indepth discussion of power, including bias that occurs when small sample pilot studies are used to estimate the expected effect size in prospective power analysis, is beyond the scope of this article (2).

Outlier Assessment

An outlier is defined as a data point that deviates markedly from the other observations in the sample, located on the remote tail end of the true population. Physiologists filter outliers in several ways. Statistical analyses assume the data are free of outliers, and thus every data set should be evaluated for the presence of relevant statistical outliers before analysis to avoid faulty conclusions. If the outlying value was demonstrably incorrectly measured or an error occurred while documenting the data and correction is not possible, the value may be dropped. Determining whether the outlier is physiologically possible is one criteria that can be used to make this assessment. Several tests can be used to statistically detect outliers (86a).

The Dixon test determines whether a value is too small or large compared with its nearest neighbor (184). Grubb’s test determines whether a single outlier is present, whereas Generalized Extreme Studentized Deviate can detect more than one outlier (70). Of course, the physiology should be considered in this assessment, and physiological plausibility can be a criteria for inclusion. The truncated outlier filtering method first replaces the maximum and minimum or the sample population before computing the exclusion criterion. This results in a more compact criterion for the determination of the outlier (28).

Whether the outlier should be removed can be decided using the following guidelines. If the outlier does not change the results, it is acceptable to include the outlier. If the outlier affects overall results, the final statistical analysis with and without the outlier should be presented. In the end, whichever statistical method you chose and rationale you use to filter an outlier, it is critical to report this information in the methods and results.

Missing Data

Even with the most rigorous study designs, missing data or subject dropouts are possible. Although missing data imposes a serious challenge to statistical analysis, there are acceptable strategies to handle such events. Several comprehensive reviews have been written on this topic; Slinker and Glantz have reviewed how to handle missing data under conditions of two-way ANOVA (161), and He reviewed multiple imputation, a common statistical technique for analyzing incomplete data sets (81).

Big Data Analysis

Analysis of big data sets such as omics data sets are distinct from the traditional statistical approaches discussed in these guidelines and are thus beyond the scope of the present recommendations. Big data analysis requires bioinformatics coupled with statistics for data visualization. Several tools and tests can help provide new perspectives on data, including heat maps, volcano plots, principal component analysis, pathway analysis, and clustering. Statistically, controlling for false discovery rates in evaluating multiple comparisons is particularly important for large transcriptomics or proteomics data sets (40). Although big data analysis of omics data sets is currently not prevalent in American Journal of Physiology-Heart and Circulatory Physiology articles, they have appeared (57, 122, 164, 171) and more are anticipated.

RESOURCES AND SOFTWARE PACKAGES

Several resources contain more detail on the use and reporting of statistics, for example, Common Statistical Errors and How to Avoid Them (65). A number of useful decision trees on how to choose an appropriate test are available online: www.microsiris.com/Statistical%20Decision%20Tree/ and http://statpages.info/#WhichAnalysis. Commonly used software include GraphPad Prism and SPSS as well as STATA Software (https://www.stata.com/) and SAS Software (https://www.sas.com/en_us/home.html). Of these, GraphPad Prism is user friendly and great for graph development but limited in performing ANOVA because it can only do up to a 2 × 2 analysis and not a larger multivariate ANOVA. There are free, valid, point-and-click alternatives such as jamovi (jamovi.org) and JASP (jasp-stats.org), and both programs include effect size estimates and other analysis options. Additionally, R (http://cran.us.r-project.org/) has a virtually endless number of packages or extensions useful for data analysis, including a markdown useful for reducing transcription errors and several advanced data visualization options. Several other online research tools that include statistical analysis and bioinformatics platforms are available. For example, Metaboanalyst (http://www.metaboanalyst.ca/) is an online program originally developed as a comprehensive tool for metabolomics analysis and interpretation that can be used for any data set; it is not limited to only analyzing metabolomics. Metaboanalyst is a good resource of bioinformatics tools, including heat maps, volcano plots, principal component analysis, and clustering. Enrichr (http://amp.pharm.mssm.edu/Enrichr/) is an online enrichment analysis tool that contains >180,000 annotated gene sets from >100 gene set libraries (24, 102).

CONCLUSIONS

This article summarizes current practices in statistical analysis reported in American Journal of Physiology-Heart and Circulatory Physiology articles and identifies the minimum that should be included in manuscripts to allow reviewers and readers to assess data quality. The take-home messages are that statistics should be considered during the experimental design and throughout data analysis, the methods and results of the manuscript should describe sufficiently which tests were done for each evaluation, and there are a number of readily available resources to assist you with statistics and data visualization. Improving clarity in statistics will improve rigor and reproducibility of cardiovascular physiology studies.

GRANTS

The authors acknowledge funding from National Institutes of Health Grants GM-104357, GM-114833, GM-115428, HL-051971, HL-075360, HL-129823, and MH-113892; Biomedical Laboratory Research and Development Service of the Veterans Affairs Office of Research and Development Grants 5I01BX000505 and 1I21 BX002085; American Heart Association Award 15SDG22430017; Brain and Behavior Research Foundation Award 26809; British Heart Foundation Award TG/18/1/33408; and a Research Excellence Award (Cardiovascular Science, University of Edinburgh).

DISCLAIMERS

The content is solely the responsibility of the author and does not necessarily represent the official views of the National Institutes of Health, the Veterans Administration, or the American Heart Association.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

AUTHOR CONTRIBUTIONS

M.L.L., G.A.G., S.K.W., and D.C.-E. conceived and designed research; M.L.L., G.A.G., and S.K.W. performed experiments; M.L.L., G.A.G., S.K.W., and D.C.-E. analyzed data; M.L.L., G.A.G., S.K.W., and D.C.-E. interpreted results of experiments; M.L.L., G.A.G., S.K.W., and D.C.-E. drafted manuscript; M.L.L., G.A.G., S.K.W., and D.C.-E. edited and revised manuscript; M.L.L., G.A.G., S.K.W., and D.C.-E. approved final version of manuscript.