The Challenge of Reproducibility and Accuracy in Nutrition Research: Resources and Pitfalls^,,,

Abstract

Inconsistent and contradictory results from nutrition studies conducted by different investigators continue to emerge, in part because of the inherent variability of natural products, as well as the unknown and therefore uncontrolled variables in study populations and experimental designs. Given these challenges inherent in nutrition research, it is critical for the progress of the field that researchers strive to minimize variability within studies and enhance comparability between studies by optimizing the characterization, control, and reporting of products, reagents, and model systems used, as well as the rigor and reporting of experimental designs, protocols, and data analysis. Here we describe some recent developments relevant to research on plant-derived products used in nutrition research, highlight some resources for optimizing the characterization and reporting of research using these products, and describe some of the pitfalls that may be avoided by adherence to these recommendations.

Introduction

Efforts to optimize the reproducibility of biomedical research through enhanced rigor in research designs and increased comprehensiveness and transparency of research reporting span the research community (1, 2). Recent examples supporting the relevance of these efforts to nutrition research include an article titled “Five Foods We Thought Were Bad for Us Now Turn Out to Be Good” (3). Variability between published outcomes from apparently similar studies of dietary natural products is seen, for example, in a meta-analysis of randomized controlled trials that reported improvements in glycemic control associated with green tea catechin exposure (4). Variability among the 22 published trials included in the analysis is highlighted by the fact that most did not find any statistically significant effect for most of the relevant outcomes. Inconsistencies in reported outcomes may result from differences in study design, in levels (or baseline levels) of exposure to the phytochemical(s) studied (5, 6), in products or product delivery, in subtle differences in outcomes or outcome assessment methods, or in the participant populations. For plant-based natural products in particular, product complexity creates the potential for substantial variation in composition, stability, metabolism, and biological effects in individual consumers. Other aspects of nutrition research, including the pressure to test multiple hypotheses in a single study (7) and the generally small effect sizes observed for nutritional interventions, likely also contribute to variation in reported outcomes (8). The many extremely challenging sources of variation require that nutrition and other natural product researchers control, as carefully as possible, those sources of potential error that are most readily controlled if they are to build a solid foundation for eventual translation to public health or clinical application.

We focus here on an aspect of nutrition research in which the resources and methods for rigorous analysis and reporting are advancing rapidly: the identification and comprehensive characterization of complex, dietary natural products, especially those derived from plants. The use of inadequately characterized or substantially different reagents and reference materials has been reported to be a major reason for failure to reproduce research results, whether within or between research groups (2, 9). For most complex natural products, there is a lack of definitive identification of the active constituents. In such cases, it is difficult to anticipate which individual components are critical for product characterization and reproducibility of outcomes, making it important to analyze the products as comprehensively as is feasible before and during the study, as well as retain voucher specimens of the source material (10) and finished materials (e.g., extract combined with rodent diet or capsules used in a clinical trial), so that they are available later to address questions about their identity and composition (11). Comprehensive product analysis in the absence of standardization cannot ensure batch-to-batch product reproducibility, but combined with transparent reporting, it may facilitate the identification of chemical constituents that contribute to specific bioactivities. Combining the practices and cutting-edge approaches to elucidation of chemistry-bioactivity relations described here may substantially accelerate the elucidation of mechanisms of action of complex products.

This review covers the following: cutting-edge approaches to the comprehensive characterization of chemically complex, plant-derived natural products; approaches to establishing specifications for such products; some resources available from the NIH and other federal agencies to support appropriate and rigorous analyses; published recommendations for the conduct and reporting of natural products research; and pitfalls in botanical identification and characterization.

Comprehensive nutritional natural product characterization: need and challenges

Where the mechanisms of action of complex natural products are not fully established and multiple components may be involved in biological activities, as is often the case for foods and botanical dietary supplements, it is possible that small differences in chemical composition may substantially modify metabolism or biological effects in humans or preclinical models. It is then critical that composition (including primary and secondary metabolite content) be comprehensively investigated. Comprehensive chemical analyses of chemically similar but distinct preparations (e.g., different batches or fractions of a complex product) combined with standard biological or biochemical outcomes and sophisticated computational methods may enable the generation of specific, testable hypotheses regarding the active component(s) and their mechanisms of action and interaction without requiring the isolation of individual chemical constituents or bioassay-guided fractionation.

Appropriate storage of raw and test materials allows for additional analyses to be performed if previously unstudied modifications or compounds are subsequently implicated as the source of experimental variability. Nevertheless, it is always preferable to publish the requisite product analysis methods and analytical results along with the rest of the study data. Many journals now provide online access to supplementary information if analytical details are too lengthy to include in the main body of a research publication (12).

But how feasible is it to comprehensively characterize a product, given practical limitations on product availability, time, and funding? Most quantitative analytical methods require researchers to choose which classes of compounds will be recovered and detected and which will be lost or even destroyed [see, e.g., (13) and (14)]. Constituent detection is biased and not only by the analytical method(s) selected. Many choices made during sample preparation, ranging from extraction temperature and solvent through choice of direct analysis compared with hydrolysis and selection of hydrolysis conditions, may also bias the results toward the selective recovery of some compounds and the loss of others (5). These critical choices must be well justified and appropriate for the hypothesis to be tested. In some cases, multiple analytical approaches may be required to provide sufficiently comprehensive information about the chemical composition of the material. If unexpected results are obtained, it is advisable to perform the analysis with the use of more than one analytical approach, with the second approach preferably orthogonal to the first [i.e., based on different biological or physical principles; e.g., (15)].

Methods for comprehensive nutritional natural product identification, characterization, and standardization

Approaches that allow broader, nontargeted detection generally incur loss of sensitivity to some components. Efforts to develop accurate and comprehensive analytical methods with the use of orthogonal approaches to natural product characterization have a long history [as reviewed, for example in (16)]. We describe briefly the notable features of a number of newer approaches to the characterization, identification, and standardization of complex nutritional natural products (Table 1).

TABLE 1.

Methods for comprehensive nutritional natural product detection, identification, and characterization¹

Method/brief description	Major strengths	Major weaknesses	Reference
Direct analysis in real-time mass spectrometry	Real-time analysis	Limited sensitivity	(17)
	Minimal sample preparation
	Localization
Laser ablation electrospray ionization	Real-time analysis	Ionization and stability	(18)
	Minimal sample preparation	No direct quantification
	Localization	Limited spatial resolution
Quantitative NMR	Universal, quantitative	Limited (μmol/L) sensitivity	(15, 19)
UHPLC-MS for oligomeric proanthocyanidins	Polymer characterization	Limited applicability	(20)
Multiple2: differentiation of complex natural products	Orthogonal approaches	Limited applicability	(21)
Quantitative NMR: multicomponent-based standardization of NPs	Multicomponent based		(22)
Multiple3: identification of complex product metabolites	Multicomponent based		(23)

¹NMR, nuclear magnetic resonance; NP, natural product; UPLC, ultra performance liquid chromatography; UHPLC-MS, ultra-HPLC–mass spectrometry.

²Methods used included HPLC with diode array detection, UPLC with electrospray ionization–mass spectrometric detection, Trolox redox assay, ultra-fast liquid chromatography with UV detection, and multivariate analysis (partial least squares discriminant analysis).

³Methods used included liquid chromatography–tandem mass spectrometry, enzymatic degradation, and n-octanol–water partition.

General guidance

Across the NIH, a suite of initiatives aim to enhance support for rigorous research design and conduct, as well as full and transparent research reporting, with the larger goal of enhancing research reproducibility. One important result of these transdisciplinary efforts is a core set of reporting guidelines, endorsed by many peer-reviewed research journals and available from the NIH website (12). These guidelines encompass issues ranging from compliance with community-based nomenclature standards to rigorous statistical design and avoidance of unintentional bias (e.g., by randomization and blinding in preclinical as well as clinical research) and include a recommendation for the “description of biological material with enough information to uniquely identify the reagents” (12). Some details are provided for information to be obtained and provided for animals used in preclinical research, as well as for cell lines used in in vitro research. For example, approaches have been described to ensure that cell lines are correctly identified and free of contamination that could adversely affect the reproducibility of results (2).

As part of these overall efforts to enhance research reproducibility, the NIH recently announced that beginning in January 2016, it will require grant applications to include additional information on the rigor of the research supporting the scientific premise for the proposal, as well as the rigor of the proposed experimental design and methods, including (but not limited to) the inclusion of sex and age as biological variables and the authentication of key biological and/or chemical resources (24). The NIH Rigor and Reproducibility webpages (25) also offer resources on best practices and avoidance of pitfalls in cutting-edge techniques used in cell and structural biology and in genomic analyses.

Guidance and resources for research on natural products

A variety of subdiscipline-specific reporting guidelines are aggregated by the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network (26), including guidelines for reporting randomized clinical trials of herbal interventions (27). Other reporting guidelines for research on plant-derived products have focused on clinical trials using soy products (28) and flavonoid research (5).

The NIH National Center for Complementary and Integrative Health (NCCIH) has long required that applicants proposing to use natural products in research rigorously document identification and characterization of the product(s) to be used, as well as provide critical information regarding the context of use (29). The NIH Office of Dietary Supplements also applies this policy to Office of Dietary Supplements–supported research. The information required under the NCCIH Natural Product Integrity policy depends on the type of product (e.g., chemically complex botanical, probiotic), as well as on the type of research in which the product will be used (exploratory, mechanistic, preclinical, clinical).

Botanical product identification

For complex plant-derived products, the NCCIH policy requires documentation of the identification of the product (or component products) and of materials sourcing. Investigators may be required to describe the method of authentication of raw materials and describe plans for retention of an authenticated voucher specimen. A number of publications have addressed the importance of proper collection, authentication, and storage of voucher specimens (10, 30, 31). Hildreth et al. (10) describe relevant standard operating procedures. Applequist and Miller (31) note some additional parameters that may affect product chemistry [e.g., the plant part(s) used, the site (including altitude), season and time of day of harvest, and any evidence of herbivore or pathogen exposure) and should be recorded for each voucher specimen. The storage site of the voucher specimen must be noted in any publications. The Office of Dietary Supplements has supported the development of resources for the microscopic identification of botanicals (32) and for validation of methods used for botanical identification (33).

Characterization of complex natural products

In addition to requiring documentation of product identification, the NCCIH policy requires characterization of the chemistry of the final product “as thoroughly as the state of the science allows” (29); data on batch-to-batch reproducibility, homogeneity, and stability of the final product under anticipated study conditions; and specifications and tolerances for the finished product (e.g., extract mixed with rodent diet). Analyses of vehicle or control diet for compounds that may show bioactivity in the outcomes to be assessed are also required.

Natural product research resources

The NIH, often in collaboration with other federal agencies, supports a variety of resources to assist researchers in the characterization and analysis of complex botanical and other natural products (Table 2). Some of these resources have been developed by NIH-supported researchers at extramural research organizations.

TABLE 2.

Federally supported resources for natural product research¹

Resource	Description	Reference
ODS AMRM Program	AMRM program of the NIH ODS	(34)
	AMRM method validation guidelines	(33, 35, 36)
	AMRM-supported validated methods and validation studies	(37)
SRMs	SRMs for NP research via AMRM and NIST	(38)
	NIST-ODS recommendations for the optimal use of SRMs in food analysis	(39)
	AMRM and NIST quality assurance programs	(40)
NIH CARBON Program	Botanical dietary supplement and NP innovation research centers	(41)
ODS fact sheets	Dietary supplement fact sheets; typical uses, toxicities, recent research	(42)
NIH NP-Drug Interactions	Centers of Excellence for Natural Product–Drug Interaction Research	(43)
Herbs at a Glance	NIH/NCCIH publication highlighting typical uses, adverse events, recent research	(44)
USDA Nutrient Database	USDA Nutrient Database for Standard Reference	(45)
USDA special databases	USDA special interest database for flavonoids in food	(46)
	USDA databases for phytochemicals	(47)
USDA FNDDS	USDA Food and Nutrient Database for Dietary Studies	(48)
NIH RePORT	Searchable database of NIH-supported research	(49)
DS Ingredient Database	Database of analytically verified content for vitamin and mineral DS sold in the United States	(50)
DS Label Database	ODS dietary supplements label database; searchable database of DS label information	(51)
NCI NEXT Program	NIH NCI’s Experimental Therapeutics Program; research assistance	(52)

¹AMRM, Analytical Methods and Reference Materials; CARBON, Centers for Advancing Research on Botanicals and Other Natural Products; DS, dietary supplement; FNDDS, Food and Nutrient Database for Dietary Studies; NCCIH, National Center for Complementary and Integrative Health; NCI, National Cancer Institute; NEXT, NCI Experimental Therapeutics Program; NIST, National Institute of Standards and Technology; NP, natural product; ODS, Office of Dietary Supplements; RePORT, Research Portfolio Online Reporting Tools; SRM, standard reference material.

Pitfalls in the identification, characterization, and use of plant-derived natural products

In the following, we describe some of the problems that may be avoided by following the guidelines described above.

The initial identification by a plant taxonomist of source plants used for research can prevent substantial loss of effort. For example, Bauer et al. (53) reported novel constituents found in Echinacea purpurea but later discovered that the material was Parthenium integrifolium that had been substituted for E. purpurea (30). Boyd et al. (54) have also described the critical importance of appropriately retaining specimens of materials used in research; when additional material was needed to test anti-HIV activity associated with novel compounds (michellamines) detected in a plant tentatively identified as Ancistrocladus abbreviatus, the newly collected material lacked both michellamines and the anti-HIV activity. Further taxonomic study of the (fortunately archived) initial source revealed that it was subtly different from A. abbreviatus, leading to its identification as a species that had not been previously described but did contain the michellamines.

Although the chemistry of a natural product may be important for correct identification, as discussed above, most current analytical approaches provide limited information about the product. One pitfall to remember is that many natural products contain compounds that can exist as different stereoisomers. Plant-derived material will most commonly contain only one isomer of a compound, but synthetic versions of natural product constituents may contain a mix of isomers. Importantly, the different stereoisomers, although not separated by most nontargeted analyses, may have very different biological effects. Amino acids and sugars are a common example of this. Another is the report by Brown et al. (55) that the polyphenolic diastereoisomer S-(−) equol, a soy-derived polyphenol that is biosynthesized by rodents and some humans, had no effect in a rat model of breast cancer development, whereas the synthetic diastereoisomer R-(+) equol decreased the number of tumors by 43% and slowed tumor development. Smith (56) reviewed the importance of absolute chemical configuration to biological activity.

To highlight the importance of assessing the chemistry, stability, bioavailability, and potential for drug interactions of natural products used in research, we turn to a study on St. John’s wort (Hypericum perforatum, SJW). In a pilot randomized controlled trial (n = 27/group) attempting to assess the potential of SJW in children with attention-deficit hyperactivity disorder, Weber and colleagues (11) gave each child 3 capsules/d of either a placebo or a preparation of SJW standardized to contain 0.3% hypericin. Hypericin was thought to be important in the activity of SJW because it had been shown to be a monoamine oxidase inhibitor in vitro (57). After 8 wk, participants’ performance was assessed with the use of standard attention-deficit hyperactivity disorder measures, revealing no statistically significant difference between treatment and placebo groups. By the time the resulting article (11) was in print, attention had begun to focus on hyperforin, another potential active compound in SJW. Hyperforin is thought to act as a reuptake inhibitor for neurotransmitters (serotonin, dopamine, and norepinephrine), and many products were standardized to contain both 0.3% hypericin and 3% hyperforin. Although independent testing before study initiation had confirmed the presence of 0.3% hypericin, when the product was assessed at the end of the trial, it contained only 0.13% hypericin and 0.14% hyperforin, suggesting that for at least part of the study, neither putative active had been present in sufficient amount to exert the hypothesized effect. The negative outcome was therefore uninterpretable. It is possible that more frequent monitoring of the SJW and replacement of degraded product might have produced a different outcome. This SJW study underlines the importance of verifying product stability and chemistry comprehensively before starting experiments [the best candidate bioactive(s) at the end of the study may differ from those hypothesized at the outset; e.g., (58)].

The Weber et al. (11) study also exemplifies appropriate caution to avoid potential product-drug interactions. Hyperforin is reported to induce hepatic production of the cytochrome P450 3A4 isoenzyme, which metabolizes many pharmaceuticals (59). To avoid potential adverse interactions, the investigators excluded those taking relevant prescription or over-the-counter products from participation in the study.

Ensuring that exposure in controls is low enough for an intervention to show an effect compared with the control is another critical issue in natural product and nutrition intervention studies. Ensuring (or at least monitoring) protocol adherence can shed light on otherwise confusing results (60). Additional critical issues in natural products research include nonlinear dose-response curves for many products [e.g., Lappe and Heaney (6) and Nahrstedt and Butterweck (58)] and inter- and intraindividual differences in bioavailability and other responses to natural products and their constituents. Although genetic and microbiome influences on bioavailability and metabolism are not generally well understood, there are indications that bioavailability may vary even within a single individual. Ferruzzi et al. (61) reported that the bioavailability of polyphenols in rats increases following repeated exposure.

Placebo and nocebo effects may be critical in human subjects research. A 340-participant trial of SJW for major depression (62) exemplifies the potential strength of placebo effects. In this study, participants receiving all 3 interventions (SJW, placebo, and sertraline as a positive control antidepressant) showed some improvement, but there was no statistically significant difference between the interventions. Indeed, 31.9% of those receiving placebo but only 24.8% of those receiving sertraline showed the study-defined “full response.”

Many factors influence the strength of placebo or nocebo effects (63). A unifying theme among these factors is the participants’ expectations with respect to the effects of the intervention. Barrett et al. (64) reported that a 719-participant trial of an Echinacea extract for treatment of the common cold showed no overall statistically significant differences between the Echinacea and placebo groups. However, the group randomly allocated to receive no pills (unblinded placebo) tended to have longer-lasting, more severe symptoms, whereas symptoms resolved more rapidly and were less severe for those who believed Echinacea to be effective and knew they were receiving Echinacea.

Summary and Conclusions

To support sustained progress, nutrition researchers need to ensure that they are using the best, feasible approaches to the rigorous characterization and reporting of the natural products and relevant methods used in their research. Data must be reported comprehensively and transparently. Data that should be obtained and reported include the sourcing, comprehensive chemistry (including, where applicable, isomer composition, glycosylation and other modifications, and characterization of polymer sizes and linkages, as well as stability under experimental conditions), pharmacokinetics, pharmacodynamics, and bioavailability at target. Whenever possible, information on exposure to and interactions with other dietary components should be reported as well. Methods for comprehensive analysis of natural products continue to evolve. Many resources, including those described in this review, are available to provide guidance in the selection of analytical methods and to assist in the optimal application of those methods. We have described some resources that can be used to assess method performance and pointed to a variety of resources that provide guidance on the comprehensive reporting of data on products used, experimental design, conduct, and results. Many journals (12) now require detailed reporting on experimental design elements and procedures used to reduce the potential for and effects of inadvertent bias (e.g., randomization and blinding). Increased focus on the practices described here should enhance the ability to reproduce studies, to resolve methodologic issues that lead to conflicting study outcomes, and to speed the growth in our understanding of the effects of botanicals and other natural products in humans and other animals. Any additional efforts required to enhance experimental rigor, such as increased quality control of key biological resources or formal validation of analytical methods, are expected to be more than justified by accelerated research and public health progress in the longer term.