Approach for systematically assessing study reliability and relevance in evaluations of monosodium glutamate safety

Graphical abstract

Highlights

• •Food safety evaluations are increasingly considering peer-reviewed publications.
• •A framework is needed that considers reliability, relevance, and biological validity.
• •A case study of 39 MSG publications found studies of generally poor study quality.
• •The proposed systematic framework would ensure consistent review of the literature.

Abstract

Monosodium glutamate (MSG) is a well-studied food additive. In addition to numerous authoritative assessments of safety, a substantial volume of research is ongoing with MSG; this includes a growing volume of research assessing the ameliorating potential of various substances against purported MSG-induced toxicity. This work set out to develop an approach for evaluating the combined reliability and relevance of these types of investigations as part of ingredient safety assessments, and subsequently, to apply such to a case study of MSG publications involving co-exposures with other substances. The approach assesses the reliability of the studies utilizing SciRAP, and the relevance in context of study design, dose relevance, and biological validity, resulting in an overall categorization of the informativeness of an individual study, or study quality. In a case study application to 39 studies assessing MSG toxicity and ameliorative properties of a variety of substances, no publications were deemed of ‘high’ study quality for the purpose of assessing safety, due primarily to low relevance to human safety (e.g., use of an acute high dose during neonatal lifestages) and limited reliability in study conduct and reporting. The approach herein can facilitate objective assessments of the quality of MSG safety or toxicity studies and could also be tailored to other food additives or ingredients.

1. Introduction

Monosodium Glutamate (MSG) is a widely used food additive based on its ability to enhance flavor and impart an umami taste profile on foods and has been evaluated for safe use by several regulatory authorities (e.g., FSANZ, 2003; Joint and Expert Committee, 1987, The Joint and Expert Committee, 2022). The U.S. Food and Drug Administration (FDA) considers the addition of MSG to be generally recognized as safe. Despite the long history of established safety, the volume of literature related to potential MSG toxicity is ever-increasing (e.g., Chakraborty, 2019; Kazmi et al., 2017; Moldovan et al., 2021; and Zhang et al., 2020). The topics of research are vast and include reviews as well as primary investigations, using a diverse array of study designs, assessing a wide variety of nutrition and health outcomes (including benefits from use as a therapeutic agent). Within this body of research, MSG is increasingly being assessed in research studies whereby MSG is used as a causative agent to induce a specific toxicological outcome and then a wide variety of substances (typically supplements and nutritional agents) are used to assess the ameliorative capacity of the agent to resolve the outcome. Such studies are generally considered non-traditional study designs in the context of assessing ingredient safety, presenting challenges to assessors interested in reviewing or conducting updated assessments of MSG safety.

As an example, in a review by Moldovan et al., (2021), a summary of ∼ 40 ameliorative studies was included as one of the lines of evidence in an evaluation of MSG; these studies, despite their non-traditional design, were used to support the author’s overall conclusions regarding safety and suggestion to consider MSG alternatives. As another example, much of the newer research being published on MSG (as well as other food ingredients) often investigate molecular and cellular activities (e.g., oxidative stress) as well as traditional apical endpoints using a wide variety of study constructs (vs. guideline-based studies in rodents). These studies, were not, however, relied upon by authoritative agencies to establish conclusions on safety due to their indirectness (lesser relevance and generalizability) to the overall objective of assessing the safety of MSG as a food ingredient.

The principles of systematic review are increasingly being considered in the evaluation of ingredient safety which requires a thorough and transparent evaluation of the literature. Along with the uptake of these methods, practitioners/assessors have applied critical appraisal methods that assess the internal validity of individual studies using techniques such as risk of bias (EFSA 2020; USEPA, 2021a; WHO, 2021). However, internal validity alone, does not fully meet the needs of a risk assessor, as there are often multiple aspects of individual studies that impact its applicability in a safety evaluation which are not assessed using default systematic review methods (Wikoff et al., 2020). For example, study quality tools such as Klimisch categorization of reliability, are often used in risk assessment; in doing so the reliability is determined based on use or comparison to an OECD (or similar) guideline. Risk assessors need to be able to appraise both the reliability and relevance of individual studies, recognizing that not all studies will be equally informative to an assessment for reasons other than the reliability of the conduct and reporting on the study. That is, studies can be conducted well, but can still have limited or less relevance than other studies in an assessment of human safety – the level of indirectness could be due to a variety of factors including species differences, outcome differences, or dose and exposure route differences. For example, a well-conducted study assessing inhalation exposure to a substance would be less informative in an assessment of safety via dietary exposures.

Some authoritative entities using systematic review methods, as well as other investigators, have refined risk of bias approaches to more broadly encompass aspects of study reliability and relevance that are important to risk assessment. For example, the Science in Risk Assessment and Policy (SciRAP) tool – a highly comprehensive tool explicitly developed to address the non-guideline studies within the context of risk assessment – includes criteria for reporting reliability, the reliability of the methodology, and relevancy (Molander et al., 2015, Roth et al., 2021, Waspe et al., 2021). Recent assessments by European Food Safety Authority (EFSA) have differed in their approach to study quality based on available evidence (e.g., EFSA Panel on Food Additives Flavourings, 2020; EFSA Panel on Food Additives Nutrient Sources added to Food, 2017; Hernández‐Jerez et al., 2021). These assessments have evaluated internal validity using established tools such as OHAT RoB (NTP, 2015) or Klimisch scoring (Klimisch et al., 1997). External or construct validity has been evaluated using SciRAP (Molander et al., 2015). Other regulatory bodies such as US EPA’s TSCA and IRIS or the European Commissions’ SCHEER have developed in house protocols or methodologies for evaluation of internal, external, and construct validity to suit the purposes of their assessments (Commission, 2018, Commission, 2023, USEPA, 2021, USEPA., 2021b).

The indirect nature of the ameliorative studies of MSG requires an objective approach to assess the reliability and utility in context of ingredient safety. The evolving application of critical appraisal tools offer an opportunity to help assessors objectively navigate the ever-increasing and highly heterogenous research relating to MSG safety. Herein, the objective was to develop and apply a streamlined approach for appraising individual study reliability and relevance, and in doing so, integrate the concepts of internal, construct, external, and biological validity to objectively assess the study quality of ameliorative MSG studies for the purposes of ingredient safety. The proposed approach was applied to a small subset of MSG studies using co-exposure and ameloriative designs to demonstrate utility with the specific objective of assesssing the overall informativeness of studies for the purpose of safety as a food additive. It is anticipated that this proposed approach can both be applied to large bodies of MSG studies, and can be refined based on the desired research question and applied to other food safety evaluations.

2. Development of approach for assessing study quality for ameliorative MSG studies (Methods)

The first step of any risk assessment or systematic review is a planning stage in which an assessor compiles existing information and determines the specific research question or objective. As such, the first step in this approach was to consider the existing evidence base of safety assessments for MSG, and to define that the purpose of the approach is to assess study quality for MSG in the context of safety as a food additive. Because numerous authorities have conducted reviews, the recent review conducted by the European Food Safety Authority (EFSA), and the information it contains, was used as the baseline reference for topic-specific considerations and offered context for reliability and relevance. With respect to the approach for assessing study quality, numerous guidance documents and scientific manuscripts were considered (cited throughout); in doing so, the goal was to both determine an approach grounded in evidence-based techniques while also being fit-for-purpose and readily applicable as a tool for assessing MSG-related research. The resulting approach (Fig. 1) involves assessment of both reliability and relevance through evaluating multiple types of validity: internal, external, construct, and biological validity, as described below.

Fig. 1.

Summary of evidence-based approach to evaluate study quality based on reliability and relevance via internal, external, construct, and biological validity.

2.1. Assessing reliability and relevance

There are no existing appraisal methods to explicitly evaluate the internal, external, and construct validities of individual studies as part of determining study quality in the assessment of food ingredient safety, or general chemical risk assessment, though several tools and guidance documents have discussed the importance of multiple types of study validity (e.g., EFSA 2017b; Wikoff et al., 2020, Waspe et al., 2021). SciRAP (www.scirap.org) was the only tool which seperated reliability and relevance and was specifically developed to evaluate the reliability of non-guideline studies for the purposes of risk assessment; it contains many overlapping aspects with risk of bias tools and concepts (Molander et al., 2015, Waspe et al., 2021). In this tool, granular questions are posed that cover how the study was conducted and reported as poorly designed/performed studies or those with missing study details will likely have limited study quality. The specificity of questions include, ‘test compound and controls’, ‘animal model and housing conditions’, ‘dosing and administration of the test compound’, ‘data collection and analysis’, and ‘other’. By answering these questions for a study, the reliability of the underlying data is gained. Given the expanding MSG evidence base of interest contains primarily non-guideline studies in experimental models, SciRAP was used in the evaluation of reliability herein, relevancy was addressed separately detailed below. SciRAP scores for both reporting and methodology followed the weighting scheme and reported classification threshold of ‘Not Reliable’, ‘Partially Reliable’, and ‘Reliable’ as noted in (Molander et al., 2015). While SciRAP offers the most complete and thorough evaluation system for non-guideline studies, its utility for novel or unique endpoints/study designs is limited by the general lack of a priori knowledge to adequately determine the use of “reliable and sensitive test methods” for such.

In traditional risk and safety assessment practice, relevance is often based on subject-matter expertise, recognizing that there are varying levels of relevance that can inform weight-of-evidence. This contrasts with systematic review methods, where relevance is typically determined in early stages of inclusion/exclusion; there is no individual-study level assessment for degrees of relevance, rather a study is simply included or excluded; degrees of relevance may be assessed as part of indirectness in systematic review, but typically this is done on a group of studies vs. individual studies (an approach which is not sufficient for risk assessment purposes). Herein, a simplified approach that focuses on attributes that are often most important to risk assessment are proposed (Fig. 1; Table 1). The approach is based on objectively assessing the relevance to safety assessments based on concepts of construct (appropriateness and reliability of study design to assess the outcome investigated in context of food safety) and external validity (generalizable to safety assessment in humans). Recognizing the overlap in these concepts (Table 1), these validity aspects are collectively measured by: (1) route of exposure relative to human dietary intake, (2) study design relative to assessment of safety from human dietary intake, including evaluation of multiple dose or exposure levels (3) complexity and severity of the outcome (apical vs. non-apical) and context of advancing the understanding of potential toxicity in humans, and (4) dose relevance. Using the examples of guideline design and GLP conduct, a study can be conducted under GLP but not be a good construct to evaluate a specific outcome (e.g., a short-term or single exposure study investigating cancer) or have uncertain relevance (higher level of indirectness) to directly informing human health (e.g., a study in zebrafish vs. a study in primates).

Table 1.

Matrix of study attributes and ranking criteria informing individual study relevancy categorizations. ADI dose comparisons based on EFSA (2017a).

Study Attribute	Validity Aspect	High Relevance	Limited Relevance	Low Relevance	Overall Contribution to Relevance
Test System/ Duration	Construct/External	OECD-like guidance: mammalian & chronic	Other in vivo studies	Non-relevant human model: non-mammalian & short-term	High
Route of Exposure	External	Dietary	Gavage	All other	High
Outcome/Endpoint*	External/Biological	Apical endpoints/Complex responses		Mechanistic/Molecular/Cellular endpoints	Low/Med
Dosing regimen	Construct/External/Biological	≤ NOAEL for ADI (3,200)	Between NOAEL & 2x NOAEL	>2x NOAEL for ADI	Low/Med
		Multiple dose studies	Studies with a single dose
*Outcome/Endpoints defined as those that are most informative for establishing health-based benchmarks and use within risk assessments.

In recognizing that each study may (or may not) be important in a weight-of-evidence assessment, pending both the overall objectives of an assessment as well as specific considerations for the food ingredient under evaluation, each of these aspects were refined specifically to assessing the safety of MSG as a food additive as described in Table 1. Specifically, these attributes include, test system/duration, route of exposure, evaluated endpoint, and dosing regimen. Two aspects (test system/duration and route of exposure) were weighted more heavily for understanding relevance. For test system/duration, conduct of a study following guideline standards removes a large amount of uncertainty on the practical conduct of the experiment elevating its potential utility in a food safety assessment; therefore, such studies receive a high relevancy ranking as is noted in the FDA Redbook (FDA, 2007). For route of exposure, several studies have noted the toxicokinetic differences that are observed across different routes of exposure specifically for MSG (Loi and Cynober, 2022, Roberts et al., 2018). Clear differences in the timing of metabolism driven by the ‘first pass effect’ can greatly limit or exacerbate target tissue concentrations (Maluly et al., 2017, Tome, 2018). Likewise, bolus dosing of large amounts of MSG, via gavage or intravenous injection, present a very different toxicokinetic profile as would be understood from more modest dosing seen from exposure from substances within the diet or in drinking water (Loi & Cynober, 2022). As such, dietary studies offer the most relevant route of exposure for food safety assessments with bolus dosing offering only limited relevance as has been noted by some regulatory authorities EFSA Panel on Food Additives Nutrient Sources added to Food, 2017, EFSA Scientific Committee, 2017).

The later study attributes, i.e. evaluated endpoint and dosing regimen, provide insight into how the data could directly inform or update a current food safety assessment. The evaluated endpoint explores the degree of severity of the effect, adverse being endpoints traditionally used to set NOAELs/LOAELs, e.g. organ histopathology, given higher relevance vs. suggestive or mechanistic endpoints, e.g. urinalysis, oxidative stress, or gene transcriptional changes given lower relevance as has traditionally been the case in risk assessment guidelines (EFSA, 2017b). Similarly, the dosing regimen offers insight into how the level and number of doses used may inform existing evaluations. Notably, high dose studies, well above established NOAELs, may offer only little relevance to an existing safety assessment whereas doses occurring within the proximity of established NOAELs were considered more informative due to their potential in updating/refining existing safety thresholds. Likewise, the dosing regimen offers confidence into the dose–response nature of the finding with multiple dose studies given higher relevance over studies evaluating a single dose level. Given that the dosing regimen attribute is considering both the dose level and the number of doses evaluated, more priority was given to the dose level as even studies exploring a single dose can be substantiated with other studies to inform the dose–response understanding and may more directly impact the nature of existing safety assessments. Expert judgement was considered for each of the criteria taking into account the considerations noted above and, where possible, ‘formulaic’ based on Table 1.

2.2. Aggregating reliability and relevance to overall study quality

Once reliability and relevance have been assessed, overall study quality (a measure of overall informativeness) was assigned using the matrix in Table 2. Because purely quantitative methods are generally not considered good practice due to the shortcomings related to scaling and validation, categorization methods are often used to provide transparency and reproducibility while also allowing flexibility for subject matter expertise. In the proposed framework, the underlying aspects of study quality, i.e. reliability and relevance, are each categorized into one of three categories – high, moderate, and low quality (informativeness). Depending on the volume and quality of information, expert judgement is required to determine how to use categorizations in an overall assessment, though it is generally recommended to place more weight on high or moderate studies, particularly for developing conclusions on safety.

Table 2.

Matrix for Aggregation of Study Quality Assignment.

Aggregated Study Quality (Informativeness) Categorization	Relevance	Reliability
High	High	Reliable
Moderate	High	Partially Reliable
	Limited	Reliable
	Limited	Partially Reliable
Low	High	Not Reliable
	Limited	Not Reliable
	Low	Reliable
	Low	Partially Reliable
	Low	Not Reliable

As a last step in assessing quality, the biological validity of the endpoint is assessed in context of the larger body of evidence. This is particularly important in helping to determine information that may influence an assessment conclusion, if new data suggest sensitive findings or reported adversities not observed in other studies. For MSG, each study model or endpoint can be assessed relative to the findings from the comprehensive review conducted by EFSA, as this authoritative assessment included an extensive review of the available toxicological data from guideline animal studies and human studies (EFSA, 2017a). In brief, glutamate is actively absorbed and quickly metabolized in the small intestine resulting in limited systemic or neurological distribution. Generally, MSG did not cause effects in short-term and subchronic studies up to the experimental limitation of 5 % of the diet, roughly 5,000 mg/kg bw/day. Additionally, MSG was not found to be genotoxic, reproductive or developmentally toxic nor carcinogenic in three 2-year rodent studies. Within unpublished studies cited in an EFSA review, the primary finding noted in the animal studies were elevated kidney weight which was not accompanied by histological changes; therefore, was considered not an adverse effect. The NOAEL of 3,200 mg/kg bw/day was selected for the ADI from a developmental neurotoxicity study which noted delayed early swimming development and other behavioral changes in the highest exposed group. Importantly, the findings of this study have subsequently been reevaluated by the study author, and no evidence of developmental neurotoxicity was concluded (Vorhees, 2018). Regardless, EFSA continued to consider that value and endpoint for the NOAEL. Human data were also reviewed, and some effects were noted; however, the findings lacked dose responsiveness, had methodological issues related to characterizing exposure, findings typically occurred at very high doses, and were ultimately not suited for deriving human based guidance values (HBGV) per EFSA (EFSA, 2017a).

2.3. MSG case study selection

As noted above, MSG is a very well-studied food ingredient that has been routinely assessed for safety by numerous authoritative entities. Research on MSG continues, including use in a variety of study types and models generally considered to be non-traditional in context of food safety assessment. And, although regulatory and authoritative bodies may have specific guidance or mandates on assessing these study types, there is not currently a resource that can be used by the broader scientific community to assess this ever-expanding body of literature on MSG. Given such, a case study was conducted for MSG using a subset of 39 studies considered in a review on MSG (Moldovan et al., 2021). In this review, the authors provide a table of studies whereby MSG was used to elicit a specific toxicological response, and an ameliorating agent was administered to understand the beneficial capacity of the given agent. In doing so, authors characterized that these studies demonstrated protective effects “against MSG toxicity,” (Moldovan et al., 2021), without assessing the reliability or relevance of the studies relative to MSG toxicity. As such, this subset of publications: (1) provides an objective selection of studies demonstrative of a larger body of studies commonly found in the literature, e.g. studies with purportedly beneficial supplements like vitamin C and E, and (2) allows for a case-study evaluation of an approach to assess reliability and relevance of MSG studies, which can ultimately be employed by the greater scientific community in subsequent assessments.

To conduct the case study, evidence analysts extracted study information and followed the process for assessing reliability and relevance via internal, external, construct, and biological validity. This included SciRAP assessment of 39 studies, as well as consideration of attributes related to external validity (i.e., test system, route of exposure, evaluated endpoints, and dosing regimen). Studies were groups based on the endpoint being evaluated or the unique study design being considered for assessment of biological validity. Following the data collection, reviewers objectively assigned categories of reliability and relevance, followed by overall study quality categorizations.

3. Case study application to MSG studies (Results)

For application of the approach, each of the 39 studies in the case study subset was assessed for relevance and reliability. These studies were diverse, including studies in rats and mice of varying duration (most < 30 days), and largely involving a single dose level of MSG and often multiple doses of an ameliorating agent (e.g., a variety of vitamins such as vitamin C or E and plant products, including coconut water, gingko biloba extract, etc.). For brevity, detailed assessments of each study are provided in the Supplemental Materials and summarized in Fig. 2.

Fig. 2.

Summary of study reliability and relevance and overall assignment of study quality for individual studies considered in the MSG case study subset, demonstrating importance of multiple aspects of study validity in objectively assessing overall quality for risk assessment.

As an example, Paul et al., (2012) sought to “investigate the nephrotoxicity induced by chronic oral intake of MSG and cytoprotective role of α-tocopherol.” Female rats were exposed orally (authors did not specific if dietary or gavage) for 180 days either to individual exposure of MSG or α-tocopherol and co-exposure groups (combined MSG and α-tocopherol). Following exposure, renal functional markers and histology were evaluated along with mechanistic endpoints related to oxidative stress. Sufficient details were provided in the publication to deem the study as “partially reliable” via SciRAP with some animal handling information missing limiting the reliability. While the study did consider a chronic oral exposure, i.e. 180 days, and considered some adverse endpoints, i.e. renal histology, only a single dose level was assessed. The MSG dose level used was 4,000 mg/kg/day compared to the ADI NOAEL of 3,200 mg/kg/day, and the authors did not provide the type of oral exposure. Renal effects were noted in the EFSA evaluation of MSG but occurred at higher doses, i.e. 5,400 mg/kg/day, and were noted to be caused by the alkalizing effects of MSG due to the presence of increases in urine sodium and sodium excretion. As such, the study was determined to be of “limited” relevance due to the dose level considered and context of the larger body of evidence on renal effects. Overall, this study was categorized as “moderate” study quality (Fig. 2).

As another example, Bousava et al., (2016) was determined to be of “low” study quality due to the lack of reliability and low relevance. The study was meant, “…to evaluate the effect of green tea extract (in three different dosage schemes) on the activity and mRNA expression panel of drug-metabolizing enzymes in the liver and small intestine of mice with MSG-induced obesity and metabolic syndrome.” Newborn mice were exposed via subcutaneous injection during postnatal day 2–8 with 4 mg/g/day MSG then allowed to develop, unexposed, until 7 months of age when exposure to green tea extract occurred (no negative control was included). The primary endpoints evaluated were those related to xenobiotic metabolism; however, the unique study design led to it being considered with similar experimental studies. Additionally, the study was missing experimental details related to animal handling and test compound preparation as well as inconsistencies were reported which limited the reliability. The relevance of this study was deemed low due to the experimental model not being relevant to a human exposure or to general guideline studies. Specifically, due to differential development between mice and humans and the established toxicokinetics of MSG (EFSA 2017a), replicating the exposure found within the model would be exceedingly unlikely in humans. This study was categorized as “low” quality overall.

Application of the framework found the majority of 39 studies in the subset evaluated to be “partially reliability” (25/39); no studies were considered “reliable.” As can be seen in Fig. 2, the SciRAP methodological evaluation was generally higher than the reporting. One of the common elements that was driving lower reliability was the lack of reporting related to animal handling (e.g. how were the animals housed, identified, allocated to treatment groups, randomization procedure, etc.) and providing sufficient information on the test substance (e.g. purity, solvent used, etc.).

Across the various subsets, most studies were categorized as “limited” to “low” relevance. No studies were determined to be highly relevant. Only two of the 39 studies used a dietary route of exposure. Of those studies that considered an oral route, the majority were via gavage (14/22). Of the studies which were of “low” relevance (21/39), over half were of a test system of little relevance to human safety (12/39). In such, the test system consisted of a large acute bolus dose of MSG sometime during early development (within postnatal days 1–14) via either subcutaneous or intraperitoneal exposure. Regarding duration, only 6 studies had exposure lasting longer than 30 days indicating the bulk of the studies (33/39) were more acute or subacute studies. Lastly, a wide range of doses were considered across the various studies, ranging from 0.08 mg/kg/day to 16,000 mg/kg/day. When considering the NOAEL that was used in setting the ADI, i.e. 3,200 mg/kg/day, only 17 of the 39 studies evaluated considered doses lower the NOAEL (Fig. 3).

Fig. 3.

Distribution of doses evaluated in the case-study subset of literature; dose-relevance considered important to the external validities well as the biological validity of individual studies. Yellow (mean intake) and blue (high intake) note the estimated daily intakes for toddlers based on Loi and Cynober, 2022; comparison to the NOAEL used by EFSA (2017a) in developing the ADI for MSG.

In addition to the dose relevance, the assessment of the biological validity of the study designs and models used in the subset of studies was assessed relative to the existing body of evidence considered by EFSA (2017a) (Table 3). The largest biological group were studies that utilized a neonatal test system that is of limited relevance to assessment of acceptable daily intakes by EFSA, as ADI values do not apply to infants, and also because of biological and physiological specificities which make neonatal mice different than adult mice and other species, including differences in the capacities of glutamate metabolism in the gastrointestinal wall (leads to elevated blood concentrations even after low doses of glutamate), and the lesser-developed BBB (leads to elevated glutamate levels in the brain), see Table 3 for additional scientific details (EFSA, 2017a). Across the different organ systems that make up the remaining biological categories (Table 3), oxidative stress was frequently proposed by study authors or explored by ROS-related mechanistic endpoints. Oxidative stress, on its own, is generally difficult to interpret as it is non-specific, and is most informative when assessed along with apical effects (e.g., Bus, 2017). Herein, such mechanistic effects were considered biologically plausible; however, similar to EFSA (2017b), these types of endpoints were given less weight in the assessment of biological validity given that they were not accompanied by consistent apical effects in chronic, high-dose studies.

Table 3.

Biological Validity of publications noted Moldovan et al.,

Test System/ Endpoints	#	General Biological Mechanism Considered or Purported	EFSA (2017a) Conclusion or Opinion on Endpoint	Biological Validity Considerations for Current Assessment
Neonatal Exposure	12	Model for inducing neurological and metabolic effects; large single bolus dose of MSG at PND1-14 results in lesions in the arcuate nucleus of the hypothalamus leading to adulthood obesity and/or neurological effects.	“Therefore, the findings in neonatal mice cannot be extrapolated to the human situation, with the possible exception of the neonatal period. In addition, the Panel noted that direct exposure to neonates (young infants) is not in the scope of the re-evaluation of glutamates as food additives and, therefore, the histopathological lesions of the CNS and behavioral changes observed in neonatal animals are not relevant for the assessment of glutamic acid–glutamates (E 620–625) when used as food additives.”	This specific experimental construct is widely used to induce adult obesogenic and/or neurological effects; however, toxicokinetics and differential developmental in neonatal mice and the direct exposure of infants at such a large dose do not translate to human exposure in foods. Thus, this study model was considered of low biological validity for evaluating safety of food ingredients in humans.
Obesity	1	Potential inhibition of leptin leading to altered energy balance.	“Overall, the Panel considered that equivocal results were found in epidemiological studies and no effect was seen in an interventional study, making it unlikely that the observed effects in animals were relevant for human hazard characterization of glutamate.”	While some animal studies have noted modulation of leptin and altered food intake; the evidence from humans would suggest that obesogenic findings in animals is not biologically valid in humans.
Renal	3	Oxidative stress-mediated mechanism leading to alterations in kidney functions, gross anatomy, and histopathology.	“Overall, urothelial hyperplasia of the renal pelvis or papilla was observed in several rats fed MSG at doses of up to 5,400 mg/kg bw per day. The Panel considered that the alkalizing effects of MSG were responsible for the urinary bladder hyperplasia observed at this dose. In dogs, the only treatment related effects were increased urinary volume and sodium excretion in the 2-year study.”	Renal effects were observed in animal studies at very high doses; however, effects were not attributed to glutamate alone (e.g., sodium cation). Similar effects are considered unlikely in humans given typical consumption patterns.
Neurological	4	Multiple mechanisms proposed including oxidative stress, anti-inflammation, and overt excitotoxicity from glutamate.	“In studies with very high doses of glutamate, which were administered systemically, brain damage was detected in areas of the brain that were not protected by the blood–brain barrier (BBB) (Price et al., 1981; Olney and Sharpe, 1969). These results supported the concept that over-stimulation of excitatory amino acid receptors could cause neuronal death (Schwarcz et al., 1984; Albin and Greenamyre, 1992). Subsequently, this hypothesis became a popular pathogenic explanation for neuronal damage observed in acute conditions, e.g. stroke. However, it should be taken into consideration that in such cases glutamate is released within the brain. For example, during an ischemic episode, glutamate is released from brain cells and an excessive concentration of glutamate will be present in the extracellular fluid (ECF) causing extreme excitation of other neurons (Choi et al., 1987; Martin et al., 1994; Castillo et al., 1996; Rothstein, 1996). This in turn may result in the opening of receptor-coupled ionophores, for example calcium channels. A large influx of calcium associated with impaired intracellular calcium sequestration mechanisms, which activate a host of catabolic enzymes, may ultimately result in neuronal death (Benveniste et al., 1984).”	As noted by EFSA, endogenous, and potentially exogenous, glutamate can result in neurological adverse effects via potential overt excitotoxity. This often occurs following atypically high levels of glutamate which would not occur with an intact blood–brain barrier or during normal neurological function. As such, while a biological valid effect of glutamate, this effect is unlikely to have an impact on a food safety evaluation during normal human consumption of MSG.
Oxidative Stress	5	Oxidative stress activation leading to lipid peroxidation, induction of ROS-enzymes.	Such mechanistic endpoints are not typically evaluated in chronic guideline studies but lipid peroxidation was noted at very high doses (100,000 mg/kg/day for 45 days via the diet).	The presence of oxidative stress is considered biologically plausible with MSG exposure; although, such a mechanism should be accompanied by adverse apical effect (e.g. histopathology), which are largely not observed even in chronic studies at very high doses. Therefore, these mechanistic endpoints were given lower weight in overall safety considerations as the oxidative stress could not be associated with established adverse effects.
Hepatic	6	Oxidative stress or inflammation-mediated mechanism leading to alterations in serum chemistry markers and/or liver histology.	Across the number of subchronic and chronic studies evaluated by EFSA, adverse hepatic effects were not noted.	Given the absence of hepatic effects in the guideline, repeat dose studies evaluated by EFSA, the biological validity of hepatic findings reported in the subset of single-dose studies were given less weight due to their relative inconsistency (but were also evaluated for dose context to assess potential for sensitivity).
Reproductive/ Development	8	Oxidative stress-mediated mechanism leading to testicular/uterine toxicity measured as altered histology, sperm parameters, or hormone levels.	“No adverse effects were observed in reproductive and developmental toxicity studies.”	Given the absence of adverse effects in two-generation studies in rats, lack of change in histopathology nor sperm parameters, the findings reported in the subset of studies were given less weight due to their relative inconsistency (but were also evaluated for dose context to assess potential for sensitivity).

When all aspects of validity were considered for individual studies in the case study, no studies within this subset of literature were found to be of overall ‘High’ study quality (see Supplemental Materials). In this subset, the categorizations were largely driven by limitations in relevance due to lower rankings in construct, external, and biological validity, despite the moderate study reliability rankings of individual studies.

4. Discussion

The proposed approach for assessing study quality presented here combines the reliability and the relevance of individual studies which are important attributes for consideration within a risk assessment. Such evidence-based techniques consider various aspects of study validity, all of which, in the presented case study, were tailored to a specific topic: MSG as a food additive. The result is an objective process to categorize study quality of non-traditional research studies involving MSG for the purposes of assessing its safety as a food additive in conjunction with an existing evidence base. As is demonstrated in the case study herein, multiple aspects of validity – including the relevance of the study design, dose, and biological validity – are important to assess at the individual study level as well as the evidence base level to understand MSG safety. Assessing internal validity alone does not provide sufficient information needed in a standard safety evaluation, as was demonstrated in the case study of a subset of publication reporting ameliorating properties of a variety of substances on MSG-induced toxicity.

Guidance for evaluating relevancy is highly context dependent and requires the evaluator’s subject matter expertise on the topic of the safety assessment; an aspect that can be considered as a limitation generally in the conduct of risk assessment, as well as in the specific approach proposed for MSG herein. It is increasingly being recognized, however, that aspects other than internal validity – as is the focus in evidence-based methods such as systematic review – are critical for appraising and integrating evidence. For example, EFSA (2024) recently published guidance on appraising epidemiological evidence, and includes on-going discussions beyond risk of bias (internal validity) and also considering both external validity and biological relevance as part of synthesizing and integrating studies to inform safety evaluations. It is also recognized that assessment of relevancy requires consideration of both external and construct validity which contain overlapping attributes. SciRAP contains a small subset of general relevancy criteria containing both external and construct validity domains related to the test compound, test system, endpoint, route of administration (in vivo) and concentration tested or dose level and resulting tissue level (Molander et al., 2015, Roth et al., 2021) – elements demonstrated to be important in the case study herein. Given the general nature of the SciRAP relevancy criteria, they were amended with consideration of other approaches and modified/systematized as noted in Table 1 to specifically inform a food safety assessment. The relevance criteria captured herein were designed based on the existing understanding of MSG safety and toxicokinetics. Adaptation of the framework criteria to other chemistries or research questions is an inherent part of any critical appraisal, including this framework, to ensure the available knowledge is used to conduct an appropriate evaluation.

In addition to demonstrating the importance of assessing relevance and reliability in co-exposure studies, which are highly prevalent and increasing in the literature, the case study application also demonstrated the utility of the approach in objectively assessing if newly published information could potentially impact or change conclusions from established assessments. The findings reported in the subset of studies here would not have been anticipated to change safety conclusions for MSG as established by EFSA (2017a). And, while EFSA – as well as other authoritative bodies – may have specific guidance or mandates around how data are evaluated, evaluation methods like that proposed here can also be useful to the general scientific community in navigating the heterogenous and ever-expanding evidence base for MSG or other food additives. Since the publication of Moldovan et al., (2021), several other reviews have been published summarizing potential health effects of MSG, largely without consideration for the various aspects of study quality proposed herein (Dharita et al., 2023, Gottardo et al., 2022, Kayode et al., 2023; Kochmar et al., 2022; Oluwole et al., 2024, Yang et al., 2023).

One of the key aspects of understanding the value of a study conducted for food safety evaluations is ensuring that the study and test system presented is meaningful for the risk assessment for human exposure. Within the case study, a large number of studies considered a test system whereby a large bolus dose of MSG is administered to post-natal rodents to elicit a specific adulthood toxicity. One of the first reports of this effect was in the 1970′s (Olney, 1969, Takasaki, 1978). Since then, the mechanism driving this effect is understood and based on the targeted neurotoxicity of gluatamate within the arcuate nucleus of the hypothalamus (Hernández Bautista et al., 2019, Olney, 1969). Due to an immature blood–brain barrier (BBB), high levels of glutamate from a bolus dose build up in the extracellular space within the hypothalamus leading to glutamate-induced excitotoxicity. As the arcuate nucleus is responsible for key neuroendocrine functions related to appetite and energy balance, these animals develop obesity in adulthood while also being hypophagic. Additionally, some behavioral effects are also noted which some researchers have equated to Alzheimer disease-like symptoms; although, the obesogenic effects are typically how the model is used. Importantly, the exposure to mice typically occurs sometime between PND 1–14, this roughly corresponds to the later portions of human fetal development in the uterus (Workman et al., 2013). As such, replicating such an exposure in humans would be inherently challenging due to the quick metabolism of glutamate within the mother or direct exposure to the fetus in utero. While there may be utility of the model to investigate obesity or neurological effects, attribution of those effects to anticipated dietary exposure to humans is generally not considered to be relevant given current understandings.

It is also of note that the MSG-induced toxicities investigated in the subset of publications in the case study herein generally did not comport with the larger evidence base on MSG as has been reviewed by numerous regulatory authorities, and thus could be a potential limitation. However, the general inconsistency in the larger body of evidence relative to the subset of studies reporting renal, hepatic, reproductive and developmental, neurological, and obesogenic endpoints investigated was largely explained by the use of bolus-type high dose exposures, and likely by the route of exposure (e.g., subcutaneous). Given the bulk of metabolism for glutamate occurs within the small intestine and being subject to first-pass effects, the effects noted in these studies would be unlikely to occur due to the limited systemic availability of glutamate following dietary exposure. While large amounts of systemic glutamate appear to elicit effects, the underlying biological mechanisms remain uncertain although evidence suggests the involvement of oxidative stress and immune mediation. As such, the case study herein highlights the importance of topic-specific considerations, including toxicokinetics, in assessing the relevance and reliability of MSG studies. Similar conclusions were drawn by Zanfirescu et al., (2020) in an evaluation of alleged health hazards reported in epidemiological studies.

In conclusion, the proposed outlined approach demonstrates an objective and consistent way to assess the combined relevance and reliability of MSG studies for the purposes of assessing ingredient safety. Application of such a framework to a subset of publications assessing ameliorative properties of a variety of substances on MSG-induced toxicity found no studies that were considered to be of a “high” level of informativeness for the purposes of assessing the safety of MSG as a food ingredient. It is anticipated that the approach and case study herein can be applied in future assessments and reviews to aid in the objectivity and efficiency in navigating the large volume of literature utilizing MSG in similar capacities.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: ToxStrategies received consulting fees from International Glutamate Technical Committee. The authors were all employed by ToxStrategies, LLC, a consulting firm that provides services to private and public organizations on toxicology and risk assessment issues during analysis of the evidence base and preparation of the manuscript. The work reported in this article was conducted during the normal course of employment, and no authors received personal fees. The sponsors were provided an opportunity to review the manuscript prior to submission. The purpose of the review was for the authors to receive input on the clarity of the science. The literature selection, methodological protocol development, analyses, interpretation of research findings, and manuscript writing, formatting, and submission were conducted solely by the authors, and the conclusions and professional judgements were not subject to the funders’ control.

DW participates in a number of evidence-based toxicology-related activities outside the submitted work though aligned with the development and promotion of systematic review and other evidence-based methods in toxicology and risk assessment.

Appendix A. Supplementary data

The following are the Supplementary data to this article:

Data availability

No data was used for the research described in the article.