The consequences of using statistical tests on proxy measurements in place of gold standard measurements: an application to magnetic resonance spectroscopy

Abstract

The use of proxy measurements in biomedical science is ubiquitous, due to the infeasibility or unavailability of gold-standard (i.e., most precise, accurate, and/or validated) measurements. For example, in magnetic resonance spectroscopy (MRS), short-echo time (TE) sequences are frequently employed to estimate difficult-to-measure metabolites such as GABA, despite J-difference editing being the recommended gold-standard due to improved metabolic specificity. This work investigates the critical relationship between the correlation of proxy and gold-standard measurements and the associated false positive (FPR) and false negative (FNR) rates of statistical tests performed on proxy measurements. Through statistical simulations, we demonstrate that even moderately high correlations (0.6–0.7), reported in the literature for short-TE vs. J-edited estimated GABA, can lead to drastically inflated FPRs and FNRs. We show that these rates are highly sensitive to the magnitude of differential bias in the proxy measurement (δ) and the underlying true effect size (Δ). For instance, a small, unmeasured bias in short-TE estimated GABA, potentially arising from macromolecule contamination, can substantially inflate FPRs. Conversely, imperfect correlation can substantially reduce statistical power, leading to high FNRs, which may explain some discrepancies within the literature. Although this work focuses specifically on the relationship between short-TE and MEGA-edited GABA, the arguments presented here apply more broadly to other difficult-to-measure metabolites in MRS (e.g., glutathione, 2-hydroxyglutarate), or generally to any circumstance where statistical tests are performed on the readily available proxy measurements in place of gold-standard measurements.

Introduction

In biomedical science, proxy measurements are widely used in place of gold-standard measurements because the latter are often infeasible (due to resource or ethical constraints), impossible (due to inherent physical or technological limitations), or simply unavailable in existing datasets. In magnetic resonance spectroscopy (MRS), short-echo time (TE) sequences are appealing for several factors: wide availability (preinstalled stock PRESS¹ and STEAM² sequences on all major vendor platforms), high signal-to-noise ratio due to reduced T₂-weighting³, reduced sensitivity to variations in T₂ values, and easy quantification of typical major metabolites^3–5 which may serve as a reference or may also be of interest. Short-TE sequences suffer from strong spectral overlap (Fig. 1), and as such J-editing modules such as MEGA⁶ have been developed to improve metabolic specificity^7–9, and are the recommended standard^9,10 for metabolites such as γ-aminobutyric acid (GABA), 2-hydroxyglutarate and glutathione. These metabolites are characterized by being low in amplitude and heavily overlapped with other signals, particularly macromolecules which have poorly characterized spectral shape¹¹. Regardless, there are numerous examples in the literature of short-TE sequences being used to quantify GABA^12–22, glutathione^{13,15,17,23,24} and 2-hydroxyglutarate^25–27, to name a few.

Fig. 1.

Synthetic spectrum of a typical short-TE (30 ms) PRESS sequence with repetition time = 2 s generated via synMARSS^49,50 at 3T in the absence of noise with concentrations typical for healthy brain tissue. Note the low relative amplitude of difficult-to-quantify metabolites such as GABA and glutathione compared to numerous other components directly overlapping them, most notably the macromolecules. No individual moiety of either metabolite is free from spectral overlap with much more prominent resonances, which results in considerable quantification errors and moderate correlation with MEGA-edited quantification. Metabolites simulated include ascorbate (Asc), aspartate (Asp), creatine (Cr), γ-aminobutyric acid (GABA), macromolecules, glutathione (GSH), glucose (Glc), glutamine (Gln), glutamate (Glu), lactate (Lac), N-acetylaspartate (NAA), N-acetylaspartylglutamate (NAAG), phosphorylethanolamine (PE), total choline (tCho), taurine (Tau), myo-inositol (mI), scyllo-inositol (sI).

Provided that the correlation is sufficiently high between the gold-standard measurement (the focus of this work will assume this is MEGA-edited GABA) and the proxy measurement (i.e., short-TE measured GABA), simpler approaches may be appropriate. We investigate what exactly constitutes a “sufficiently high correlation” here. This is achieved using Monte Carlo simulations, and we demonstrate that the false positive rate (FPR) and false negative rate (FNR) are strongly dependent on the correlation between measurements from MEGA- and short-TE estimate measurements. Even for the moderately-high correlations (0.6 to 0.7), representative of those from the literature for this application, the FPR and FNR can be drastically higher than reported, suggesting that results which rest on the assumption of perfect correlation need to be interpreted with increased caution^28,29. A preliminary version of this work has previously been presented in abstract form³⁰.

Methods

We denoted and to be the random variables sampled from the gold-standard distributions (e.g., MEGA-edited measured concentrations for the metabolite of interest) of the control groups and cases, respectively. For example, could represent concentrations of GABA from subjects in a disease population³¹, or concentrations after a functional stimulus¹⁹. Similarly, we denoted and as proxy variables (i.e., short-TE measured GABA concentrations) of the control and case population, respectively. A conceptual diagram of this methodology is provided in Fig. 2 and a full derivation of all expressions and rationale for approximations are provided in the Appendix.

Fig. 2.

Conceptual diagram demonstrating the gold-standard distributions (left) and proxy distribution for a correlation of 0.6 (middle) and a near-perfect correlation of 0.99 (right). The case when but is depicted in the top row, where false negatives may occur. In the top row it can be seen that for a moderate correlation there can be a substantial reduction in means (and hence a higher chance to obtain a false negative), which is resolved as the correlation approaches 1. Similarly, in the bottom row for a moderate correlation there can be a substantial difference in the mean of the proxy distributions, which is then resolved as the correlation approaches 1. The purpose of this work is to investigate the relationship at which a correlation becomes “sufficiently high enough” to not drastically inflate false positive and false negative rates.

We model the gold-standard measurements by

1

2

where , and denotes sampling from the normal distribution with mean and standard deviation , both with units of millimole (mM) for concentrations, or unitless for relative concentrations. The true effect (i.e., differences in means between and ) is given by . and represents the true mean and variance of the unobserved biological signal S, respectively. is the non-biological measurement variance of the gold standard variables.

The proxy variables are then modeled as a linear combination of the true biological signal, S, and an independent noise/contamination component,

3

4

where and represent the mean and variance of the contamination signal due to the use of proxy variables. is the sensitivity coefficient (i.e., how sensitive the proxy measurement is to the true signal, and is the contamination coefficient. represents differential bias introduced due to the use of proxy variables that is present only in cases and not controls (or vice versa). Non-zero differential bias could arise in scenarios where one is measuring a difference between cases and controls that is assumed to be due to a change in GABA concentration but is actually due to a different reason (e.g., macromolecule changes in cases and controls). Further exploration of potential causes of non-zero differential bias are given in the Discussion. In order to re-parameterize and in terms of the correlation between the proxy and gold-standard measurements, we must introduce the variance ratio, defined to be , where (see Appendix).

We calculated the false positive and false negative rates numerically by running Monte Carlo simulations, where random variables representing the concentrations of proxy variables were generated according to the model defined by Eqs. 1–4. All simulations were performed with 500,000 iterations per set of parameters with NumPy version 2.1.3 and all statistical tests were two-sided independent Welch’s t-tests performed via SciPy version 1.15.3. We performed statistical tests on the proxy variables (i.e., and ) to test hypotheses about the underlying gold-standard variables. Specifically, the null hypothesis () was that no true difference exists between the gold-standard means (), and the alternative hypothesis () was that a true difference does exist (. The code to reproduce all statistical simulations is available at: github.com/karllandheer/GoldStandardVsProxy.

To investigate FPRs, we set (i.e., is true, no true difference exists between the means of the gold-standard variables) and , (i.e., a differential bias is introduced in the estimation of cases but not controls of the proxy variables, or vice versa). To investigate FNRs, we set (no differential bias) but ( is false). Initially, we array across a large range of possibilities of or (from 0.2 to 2.2 mM) to demonstrate that in certain scenarios both FPR and FNR can be strongly sensitive to correlation and refer to these as the “theoretical simulations”. The parameter was set to 1 mM, and simulations are independent of the choice of and (as shown in the Appendix). We choose the nominal false-positive rate and . We set VR = 1 (as the test statistic is independent of it if is fixed, as shown in the Appendix) and three values for  ∈ {0.5, 1, 2} when . For the test statistic is independent of , hence we do not array over it. We also investigate how the false positive and false negative rate changes as we sweep across a range of values.

We then performed statistical simulations with experimental values from the literature and refer to these as the “literature simulations”. First, to investigate the influence of correlation and the differential bias δ on the FPR, we used values from the literature which report statistically significant difference in the concentration of GABA in the visual cortex before and after behavioral training as assessed by short-TE MRS at 7T¹⁹. We assume the reported sample values are reflective of the population values, i.e., , , . We use their values of N_samples = 25, . If we assume the null hypothesis is correct (i.e., then As we only have empirical values for and (0.72 at 7T³²), we thus must assume certain values for and VR. If we set , then  = 1.2. However, we investigate how sensitive these results are to by also performing simulations with and while keeping fixed, and since is not known, but rather , we must also simulate a range of variance ratios  ∈ {0.5, 1, } (as explained further in the derivation of the t-statistic section in the Appendix). We provide the mathematical context of how such a may arise due to small macromolecule contamination in the Discussion.

To investigate the influence of correlation on FNR, we then used values from the literature which used short-TE to estimate GABA at 3T but failed to find a difference between ALS subjects¹². The relevant values from this study are (cases/controls) a correlation of 0.58 (previously measured at 3T¹⁶), and . Previously, it was observed that ALS patients had a 16% reduction in GABA (1.42 in ALS patients vs 1.70 in healthy controls³³), thus we assumed the true effect size is . Because we assume , the test statistic is independent of , and hence is not arrayed over, while  ∈ {0.5, 1, }.

Results

In the case when , FPR increases rapidly with correlations that deviate from unity if the proxy-measurement bias, , is large (Fig. 3). Even at moderate to strong correlations (0.4 to 0.7), provided is sufficiently large, the FPR can be ~ 100%. The false positive rate increases with increasing sample size, as the experiment is better powered to detect a false result. Importantly, the false positive rate decreases with increasing , which is expected as the greater noise from the use of the proxy variable drowns out the detected bias.

Fig. 3.

False positive rate vs correlation across a range of bias values (). , and an array across a typical number of subjects within the neuroimaging literature, N_samples = 10 (left column), N_samples = 25 (middle column) and N_samples = 100 (right column). Values of were 0.5 (top row), 1 (middle row) and 2 (bottom row). At even moderate to strong correlations (0.4 to 0.7), provided is sufficiently large, the false-positive rate can be nearly 100%. The false positive rate increases with increasing sample size, as the experiment is better powered to detect a false result. Similarly, the false positive rate decreases with increasing , as the contaminating signal is drowned out by the contaminating noise.

In the case when , but , even with moderate to strong correlation between gold-standard and proxy measurements, the use of proxy measurements results in a substantial increase in FNR (Fig. 4). For example, with = 25 and the false negative rate can be ~ 7% at , while at a it increases to ~ 33%. As the nominal false-positive rate is increased the measured false positive rate increases, while the false-negative rate decreases (Fig. 5). Thus, while reducing is a viable strategy for reducing the false positive rate, it concomitantly comes with an increase in false negative rate, which may or may not be an acceptable trade-off.

Fig. 4.

False negative rate vs correlation across a range of differences between gold standard measurements (). , and an array across a typical number of subjects within the neuroimaging literature, N_samples = 10 (left column), N_samples = 25 (middle column) and N_samples = 100 (right column). Values of were not swept as the FNR is independent of it (see Appendix). Even with moderate-to-strong correlations the false negative rate can be substantially inflated. For example, with N_samples = 25 and at a correlation of 1 the false negative rate is approximately 7%, while at a correlation of 0.7 it’s approximately 33%. The false negative decreases with , as expected, as the experiment is better powered to detect the positive. At low correlations regardless of differences between the true distributions the false negative rate converges to the value of , as expected.

Fig. 5.

False positive rate (beige) and false negative rate (blue) vs nominal false-positive rate, . The simulations used moderate values of mM, for the false negative rate, and mM for the false positive rate, with mM and N = 25. A strategy to mitigate false positives may appear to decrease , however this comes at an increased false negative rate, which may or may not be equally undesirable.

Using experimental values from the literature¹⁹, we demonstrate that the FPR increases dramatically with increasing magnitude of . At the estimated , the FPR ranges from 0.15 ( to 1.00 (, as shown in Fig. 6. When is large, the high variance of the contaminant noise drowns out the differential bias, and when VR is small, the large biological variance also drowns out the bias. In other scenarios the differential bias can massively inflate the FPR, with it being as high as ~ 100%, indicating that their purported result should be interpreted with increased caution.

Fig. 6.

False positive rate versus with experimental values claiming to find a statistically significant difference in GABA/Creatine before and after behavioral training¹⁹. The dotted black line indicates the empirically estimated differential bias of . Simulations were performed with values from their study¹⁹, specifically , , and . Three sets of and are depicted. The range of VRs is necessary because is estimated, not , and the range of is necessary because the t-statistic is dependent on this value when . Even in the most optimistic scenario (VR = 0.5, ), with the false positive rate is inflated to roughly three times its nominal value (15%) and can reach as high as 100% in other scenarios. While this range is large, it demonstrates that correlations alone cannot guarantee statistical validity of tests on proxy variables.

Using values from the literature¹², we show that the use of proxy variables can also substantially increase the false negative rate (i.e., reduce statistical power). At the empirical correlation of 0.58 at 3T¹⁶, the previously measured results in a FNR of 0.89, 0.83, and 0.79 for VR = 0.5, 1 and 1.3 (), respectively. With a correlation of 1 the FNR is 0.77 and 0.59, for VR = 0.5 and 1, respectively (a correlation of 1 cannot be obtained for ), Fig. 7). Thus, this apparent discrepancy between Blicher, et al.¹² and Foerster, et al.³³ may be explained by the study by Blicher et al. being simply underpowered, or due to the use of short-TE GABA which reduces statistical power.

Fig. 7.

False negative rate versus using experimental values in a study that disagreed with a previously reported significant difference in GABA/creatine between ALS patients and healthy controls¹² for the empirical correlation at 3T (blue) and if the correlation was 1 (orange). Simulations were performed with values from their study, specifically (cases/controls), and . The dotted black line indicates the empirical , obtained from a previous study³³. The range of VRs is necessary because is measured, not , while a range of is unnecessary because the t-statistic is independent on this value since . The experimentally measured correlation of 0.58 drastically reduces the statistical power for all 3 VRs displayed here, although it is worth mentioning that for the previously measured , the study is poorly powered even with .

Discussion

This work suggests comprehensively characterizing the relationship between proxy and gold-standard methods is critical for sound inference of experiments using the former. We demonstrated via Monte Carlo simulations that typical correlations found in the literature between short-TE and MEGA-edited GABA do not preclude the possibility of drastically increased false positive and false negative rates. It is important to mention that the assumptions in our model (Appendix) are shown to be consistent with the literature, as they successfully reproduce the moderate correlations often cited as a basis for using short-TE measurements. We have shown across a wide range of parameters (ρ, VR, , N, Δ ,δ) that the use of imperfectly correlated proxy measurements can result in the substantial inflation of false positive and false negative rates. Because these critical parameters are not routinely measured, our goal is to highlight the need to quantify them and to caution against interpreting proxy-based tests as if they were gold-standard results.

It is important to note that while this work focused on the MEGA-edited experiments as the purported gold standard for certain metabolites⁹, MEGA-edited measurements are themselves a proxy for true concentrations. Thus, this work also has implications for the interpretation of MEGA-edited measurements. Currently, high-quality independent validation measurements of MEGA-edited MRS with other MR or non-MR methods are limited. Selected ex vivo validation of post-mortem tissues have been performed preclinically³⁴ and clinically³⁵. For example, in one animal study, the correlation of short-TE measured GABA with ex vivo concentrations from enzyme-linked immunosorbent assay was 0.66 at 7T³⁴, once again in the range where FPRs and FNRs can be inflated. Statistical advancements on how to best combine proxy and gold-standard measurements are emerging³⁶. Other typical quantified errors do not address the issue investigated here for two important reasons: firstly, and most importantly, we demonstrate that one can be measuring effects (e.g., macromolecules) entirely distinct from the effects one believes they are measuring (GABA), even while maintaining medium-to-high correlations with the desired quantity. Secondly, estimation of uncertainty is invariably difficult in MRS, as typical estimates via the Cramér-Rao Lower Bound require knowledge of the underlying model to fit the experimental data to, which is not known³⁷, and distribution-free uncertainty estimation methods such as Conformal Predictors require access to underlying ground truth examples^38,39, which are also inherently unavailable.

To estimate the increase in FPR over the nominal value the differential bias, , must be known. While such values are experimentally unmeasured, the calculation below shows that a small variation in macromolecule concentration, due to their substantially higher amplitudes at short-TEs, could result in values comparable to those used in simulations in this work. GABA has a concentration of approximately 2.1 mM in the healthy brain^3,40, and the overlying macromolecules have an absolute proton concentration, , of 19.8 mM at 2.99 ppm, 120.8 mM at 2.04 ppm, and 49.0 mM at 2.26 ppm, which nearly directly overlap with the GABA moieties at 3.01 ppm, 1.89 ppm and 2.23 ppm, respectively. Thus, using the institutional units where GABA/tCr 7.6 from Jia et al.¹⁹ (referred to as ) and absolute concentration of GABA is mM (referred to as ), the institutional concentrations of these three macromolecule resonances would be , where the corrects for being a proton concentration¹¹, and being a molecular concentration (this factor is equal to ½ for all resonances as all moieties of GABA contain 2 protons⁴¹) and the last term corrects for the difference in T₂ values between GABA and macromolecules at the experimental TE of 36 ms. T₂ values of 63 ms were used for GABA (measured at 7T⁴²), and 20.0 ms, 14.3 ms, 19.8 ms, for M_2.99, M_2.04, M_2.26, (measured at 3T¹¹ with similar values obtained at 9.4T⁴³). Thus, , , are 10.5, 31,2, and 25.5 (unitless), respectively. Thus, if even a small change (i.e., a few %) in these macromolecules concentrations between conditions bled into the GABA signal for cases but not controls (or vice versa), it could result in a comparable to the observed value of 1.2 (Fig. 6). While Jia et al.¹⁹ used an inversion preparation module to measure the macromolecules, this only partially alleviates the issue as the macromolecule measurement can vary substantially from those in the short-TE spectrum. This is because the inversion preparation module heavily weights the T₁ spectrum¹¹. Furthermore, although macromolecules are most problematic due to their poorly characterized spectral shape, there are numerous other directly overlapped metabolites with GABA, such as NAA, NAAG, GSH, glutamate (Glu) and glutamine (Gln) (Fig. 1) which could also potentially bleed into the estimated GABA quantity. While macromolecules are a likely contender for the introduction of such bias, other experimental differences between cases and controls could also cause it. Potential causes include differences in shim quality due to tissue heterogeneity differences⁴⁴ (e.g., a tumor) or motion differences (which are known to be influenced by conditions such as Alzheimer’s⁴⁵).

One limitation of the work presented here is the assumption of normality of distributions of the random variables. While this is a common assumption used within the biomedical literature (hence the use of t-tests), there is no inherent physiological reason why such concentrations would be sampled from a normal distribution. We chose the normal distribution as, for a given mean and standard deviation, it maximizes the entropy of the random variables. Hence in the absence of additional information about the distributions it becomes a natural choice. It is also worth mentioning that while the literature simulations focused specifically on GABA/tCr, if GABA and tCr are both sampled from normal distributions and (which has previously been reported as 0.06 at 3T⁴⁶), then this ratio will also sample from a normal distribution. Work on characterizing the distributions of metabolite concentrations from both healthy and diseased populations would be a valuable contribution and could be used to further refine the results obtained here, among other simulations. Regardless, while the exact numerical results will invariably change based on choice of distributions, the problem investigated here will persist. Importantly, the Jia et al.¹⁹ study investigated GABA/tCr, and previous related studies have demonstrated that spurious correlations can arise due to the use of a common denominator⁴⁷, thus further complicating the interpretation of such results.

Appendix

Definition of model

1. The True Biological Signal (S): This is the true, unobserved biophysical quantity of interest (e.g., GABA concentration). It is a random variable with a mean and variance .

2. The Gold-Standard Measurement (G): This is the most precise, accurate and/or validated measurement of a quantity of interest (in the context of this work, it is MEGA-edited GABA). We denote and as the gold-standard measurements for the controls and cases in Eqs. 1 and 2. The gold-standard measurement represented by

where is measurement error, assumed to be sampled from a zero-mean normal distribution; and . Note, while gold-standard measurements will inherently possess a constant systematic bias relative to absolute biological truth (i.e., ), such a constant cancels out in the calculation of the effect size (, the difference in means) and does not affect the derived false positive or false negative rates, hence was set to 0 here for simplicity. Thus . We make the simplifying assumption that , i.e., the gold-standard measurement variance is dominated by the biological variance. Previous methods have found GABA + /water intra-class correlation 0.87⁴⁸. From this one can show that . Indicating that is approximately 7 × larger than .

3. The Proxy Measurement (P): This variable represents the measurement obtained from the proxy method (e.g., short-TE GABA), which we denote by and for controls and cases, as in Eqs. 3–4. We model as a weighted sum of the true signal S and an independent contamination/noise component

where is the contaminating signal (e.g., overlapping macromolecules or other metabolites), is the sensitivity coefficient (i.e., how sensitive the proxy measurement is to the true signal, and is the contamination coefficient. Because and are noise terms, we assume S, and are all mutually independent.

Derivation of proxy variance and correlation

The variance of the proxy variable can be obtained from its definition

The correlation between the gold-standard and proxy variable is

The covariance term can be expressed as , which rests on the mutual independence assumption of S, and and the bilinearity of covariance. Thus, the correlation becomes

where we have defined , and the approximation arises due to the assumption of . We then define the variance ratio, VR, as

Combining the above two expressions we obtain . Since we assume (as would imply the proxy variable is more sensitive to the true signal than the gold standard), we note that the VR provides an upper bound for the attainable correlation, i.e., Also, from these expressions it can be shown that and

Derivation of expectation values

The expectation value for the gold-standard control group is

And similarly, the expectation value for the gold-standard case group is

where is the true desired effect size (i.e., ). The expectation value for the proxy control group can be expressed as

And similarly, the expectation value for the proxy case group can be expressed as

Derivation of t-statistic

The t-statistic for a two-sample Welch’s t-test is the ratio of the true difference in means to the standard error of that difference:

where and are the sample sizes of the control and case groups, respectively. Given that and that and that this becomes

which is independent of parameters (introduction of bias in both cases and controls from use of proxy variable) and (the true mean of the biological signal). Furthermore, if we assume is a known constant then the t-statistic is independent of VR and independent of when  = 0, but proportional to if . In the fixed scenario the only effect VR has on these simulations is that it determines the upper bound for correlation, if . For this reason, we set VR = 1 for the theoretical simulations, as the purpose of this is to investigate the question of given a fixed biological signal, how does the use of proxy measurements distort the statistical results. Alternatively, we can express the t-statistic as

which is proportional to . Thus, if t is parameterized by it is dependent on VR. In the literature simulations we choose this form, as is measured.

Author contributions

MT, CJ, KL wrote the main manuscript, MT and KL prepared main figures, MT and KL devised and carried out all simulations, and KL and CJ conceptualized the project. All authors reviewed the manuscript.

Funding

Not applicable.

Data availability

The code to generate all data and reproduce all results is available at: github.com/karllandheer/GoldStandardVsProxy.

Declarations

Competing interests

KL is an employee and shareholder of Regeneron Pharmaceuticals, Inc, and a shareholder of Vertex Pharmaceuticals. Other authors declare no conflict of interest.