Improving rigor and reproducibility in cardiovascular research

INTRODUCTION

Reports of failures by laboratories to repeat experiments performed by other laboratories have launched a whole discussion in the biomedical research community centered on the importance of establishing rigor and reproducibility in research and how to implement plans to achieve this goal. Journals serve a critical role in the process by ensuring the best science is published, both through standardized peer review and providing implementation guides centered on rigor and reproducibility topics. The American Journal of Physiology-Heart and Circulatory Physiology recently launched a call for papers on Innovation in Improving Rigor and Reproducibility in Cardiovascular Research. This call resulted in 13 articles published in this collection (https://journals.physiology.org/topic/ajpheart-collections/innovation-in-improving-rigor). In this editorial, we highlight the three themes that arose in the articles published for this collection: 1) improving methods and models, 2) incorporating sex as a biological variable, and 3) testing assumptions.

IMPROVING METHODS AND MODELS

The first theme included articles on how to improve methods and models, with most of these centered around standardizing or automating data collection and analysis. For example, when assessing sedentary behavior in prolonged sitting studies, it is important to assess posture when cardiovascular assessments are acquired (1). The perspective by Paterson et al. (1) recommends standardizing approaches and provides practical recommendations on how to achieve consistency in studies on sedentary behavior. The Chen laboratory provides methods on automated quantification and statistical assessment of proliferating cardiomyocyte rates in embryonic hearts, which is critically important for studies on cardiac regeneration (2). Favere and team (3) detail a comprehensive practical guide on cardiac electrophysiology studies in mice by going across the jugular to improve our ability to monitor arrhythmias. They provide recommendations on catheter placement, stimulation protocols, intracardiac tracing interpretation, artifact reduction, and surface electrocardiogram (ECG) recording, as well as provide reference values, all of which save time and resources for research teams trying to establish this protocol in their own laboratories.

Any investigator who routinely uses echocardiography in their research knows that image analysis takes up the vast amount of time and effort spent to capture imaging information. In this collection, Duan et al. (4) describe a tool to fully automate mouse echocardiography analysis using deep convolutional neural networks. In addition to speeding up the process, this tool removes the issue of interreader variability that occurs with manual analysis.

Although the electrocardiogram (ECG) is a useful tool for assessing success of coronary artery ligation surgery to induce myocardial infarction in rodent models, its use to date has been limited to qualitative assessment. The DeLeon-Pennell laboratory provides a highly detailed quantitative method demonstrating how to use ECG to positively identify mice with myocardial infarction (5). Using quantitative ECG benefits investigators by providing real-time feedback during the procedure, which decreases project time and reduces animal use. In addition to the detailed protocol provided in the article, take a listen to the corresponding podcast at https://ajpheart.podbean.com/e/save-hearts-improve-efficiency/.

Heart failure with preserved ejection fraction (HFpEF) is a major worldwide health problem. A major difficulty with experimental research on HFpEF is the lack of approaches to consistently and accurately quantify diastolic dysfunction in mouse models. Numata et al. (6) developed a pacing-controlled pressure-volume loop protocol for the assessment of diastolic function at different heart rates in mice and validated that the protocol could detect diastolic dysfunction in a mouse model of HFpEF. Standardizing a technique that can be adopted by the whole community allows comparison across findings from different research teams and elevates the importance of the collective research.

The reproducibility of isoproterenol to induce a stress cardiomyopathy phenotype is inconsistent, with variation in response dependent on dose used, mode of administration (e.g., subcutaneous vs. intraperitoneal injection), and animal species studied. The Mann laboratory reports important sexually dimorphic differences in response in the C57BL/6J mouse model (7). Of note, they identified that the method used to restrain the mice for the injection was the single greatest source of variability in this model. This study highlights the need to consider factors that might be overlooked such as the handling of animals during experimental procedures.

The Langendorff whole heart technique is standard for most cardiometabolic laboratories as an invaluable research tool. Over the years, modifications to this approach by individual research teams has led to a blurring of methodology. King et al. (8) performed an extensive literature review to quantify the different method variations and provide examples on how altering individual parameters (including animal model, anesthesia, cannulation time, perfusate composition, pH, and temperature) can impact study outcomes and interpretation. This report provides a framework for interpreting seemingly contradictory results before concluding experiments are not reproducible.

Although heart rate variability (HRV) is a common index monitored in sleep and cardiovascular research, HRV reliability across various sleep stages remains ambiguous. The Carter laboratory found that time- and frequency-domain HRV measurements were reliable across stable sleep periods and remained reliable during disrupted sleep (9). Their findings support the use of HRV during sleep as a tool for indirectly estimating cardiac autonomic activity. Having reliable methods and models is a key component of rigor and reproducibility, and the articles in this section provide details to reliably replicate cardiovascular research.

INCORPORATING SEX AS A BIOLOGICAL VARIABLE

The second theme centers on how to best incorporate sex as a biological variable. Sofia Ahmed and colleagues (10) provide a framework to capture factors associated with the female sex at the preclinical, recruitment, data collection, and data analysis stages. Rytz et al. (11) proposed a road map to improve the inclusion of transgender and nonbinary individuals in the planning, completion, and mobilization of cardiovascular research. Increasing representation of understudied groups in clinical research and incorporating inclusive sex-specific risk factors in cardiovascular research not only enhances rigor and reproducibility but also improves cardiovascular health for all. As we continue to see the benefits of more inclusive research in improving cardiovascular health for all, these articles contribute to our growing knowledge of sex differences and similarities in both humans and animals, as well as the importance of considering gender in designing the most rigorous studies.

TESTING ASSUMPTIONS

The third theme covered the testing of assumptions. Within the cardiovascular community, we are at times faced with the challenge of combatting preconceived ideas that have not been tested but are assumed to be correct. In one such example, the Sm22α-Cre expression system is frequently used in studies evaluating smooth muscle cell function. The Kassiri laboratory discovered that the Sm22α-derived Adam17 deletion yielded unexpected severe skin lesions in response to high fat-diet feeding in a mouse model of atherosclerosis (12). Of interest, Adam17 deletion by another smooth muscle cell driver (Myh11-Cre) did not replicate the skin lesions in the same model. Further examination revealed that Sm22α is actually also highly expressed in keratinocytes and their model induced ectopic loss of ADAM17 in keratinocytes that led to epidermal lesions when combined with high-fat diet. This is a good example of considering both specificity and selectivity when generating mouse models and highlights the need for controls to ensure the model replicates the feature of human pathology without additional phenotypes. In addition to this article, take a listen to the accompanying podcast at https://ajpheart.podbean.com/e/sm22alpha-in-keratinocytes/.

After years of evaluating the development of skeletal muscle vasculopathy and its functional implications, the Frisbee laboratory assembled an integrated conceptual model that allowed the assessment of peripheral vasculopathy with chronic metabolic disease in the obese Zucker rat, which was previously challenged by divergent contributions from spatial and temporal origins (13). They present a conceptual model for the retrograde development of peripheral vasculopathy with chronic metabolic disease and provide insight into the timing and targeting of interventional strategies that will be important to advance the field. These two articles highlight the need to test assumptions and pay attention to experimental observations. Only in this way can you follow the data in making your interpretations, rather than trying to fit the data to match with preconceived ideas.

In summary, this call brings valuable literature to the field from a number of different perspectives. The collection emphasizes not only the importance of considering rigor and reproducibility when designing experiments but also provides means on how to achieve this goal. We are reminded of the quote by Benjamin Franklin, “It takes many good deeds to build a good reputation, and only one bad one to lose it.” The cardiovascular physiology community as a collective relies on all of us to make this effort to produce precise and accurate research, and we thank you for the efforts you contribute.

GRANTS

We acknowledge funding from National Institutes of Health Grants AG053585 (to A.J.L.) and GM151274 (to M.L.L.), Veterans Affairs Office of Research and Development’s Biomedical Laboratory Research and Development Service Grants 5I01BX000505 (to M.L.L.) and 1I01BX005943 (to A.D.B.), United States Department of Defense Grants W81XWH-19-RTRP-IDA and W81XWH-13-2-0057 (to A.J.L.), and the Gheen’s Foundation (to A.J.L.).

DISCLAIMERS

The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the funding agencies or the American Physiological Society.

DISCLOSURES

Merry Lindsey, Amanda LeBlanc, and Petra Kleinbongard are editors of American Journal of Physiology-Heart and Circulatory Physiology and were not involved and did not have access to information regarding the peer-review process or final disposition of this article. An alternate editor oversaw the peer-review and decision-making process for this article.

AUTHOR CONTRIBUTIONS

M.L.L., A.J.L., L.B., A.D.B., and P.K. conceived and designed research; M.L.L. drafted manuscript; M.L.L., A.J.L., L.B., A.D.B., and P.K. edited and revised manuscript; M.L.L., A.J.L., L.B., A.D.B., and P.K. approved final version of manuscript.