No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

Abstract

Objective evaluation of quantitative imaging (QI) methods with patient data is highly desirable, but is hindered by the lack or unreliability of an available gold standard. To address this issue, techniques that can evaluate QI methods without access to a gold standard are being actively developed. These techniques assume that the true and measured values are linearly related by a slope, bias, and Gaussian-distributed noise term, where the noise between measurements made by different methods is independent of each other. However, this noise arises in the process of measuring the same quantitative value, and thus can be correlated. To address this limitation, we propose a no-gold-standard evaluation (NGSE) technique that models this correlated noise by a multi-variate Gaussian distribution parameterized by a covariance matrix. We derive a maximum-likelihood-based approach to estimate the parameters that describe the relationship between the true and measured values, without any knowledge of the true values. We then use the estimated slopes and diagonal elements of the covariance matrix to compute the noise-to-slope ratio (NSR) to rank the QI methods on the basis of precision. The proposed NGSE technique was evaluated with multiple numerical experiments. Our results showed that the technique reliably estimated the NSR values and yielded accurate rankings of the considered methods for 83% of 160 trials. In particular, the technique correctly identified the most precise method for ∼ 97% of the trials. Overall, this study demonstrates the efficacy of the NGSE technique to accurately rank different QI methods when correlated noise is present, and without access to any knowledge of the ground truth. The results motivate further validation of this technique with realistic simulation studies and patient data.

1. INTRODUCTION

Medical imaging provides a mechanism to study in vivo physiological properties of the human body, and thus plays an important role in the diagnosis, prognosis, and assessment of treatment response of different diseases. To facilitate decision-making in clinical practice, quantitative imaging (QI), i.e., the extraction of numerical or statistical features from medical images, is being actively investigated.^{1, 2} QI has demonstrated substantial promise in multiple clinical applications. These include the quantification of metabolic tumor volume from oncological positron emission tomography (PET) for predicting clinical outcomes,³ quantification of dopamine transporter uptake from single-photon emission computed tomography (SPECT) to assess the severity of Parkinson disease,^{4, 5} and quantification of regional uptake from PET and SPECT for dosimetry in targeted radionuclide therapy.^6–9

Given the significant interest in QI, multiple methods have been and are being developed for QI. For clinical translation of QI, it is essential that the measurements made by those methods are reliable. Thus, there is an important need for objective evaluation of the reliability of measurements obtained using QI methods. Typically, such evaluation requires the presence of either the true value of the quantitative parameter or a reference standard. Such true values or reference standards can be available in realistic simulation and physical phantom studies.^10–14 While these studies are important for the initial development of QI methods, there is an important need for techniques that can perform objective evaluation of QI methods directly with patient data. Such evaluation then requires the presence of gold-standard quantitative values. These are typically time-consuming, expensive, and tedious to obtain. Further, even when an approximate gold standard is available, it could suffer from the lack of reliability. Thus, techniques that can objectively evaluate QI methods in the absence of a gold standard are much needed.

To objectively evaluate QI methods without the knowledge of a gold standard, a regression-without-truth (RWT) technique was proposed in a set of seminal papers.^{15, 16} The RWT technique assumes that the true and measured values are linearly related by a slope, bias, and Gaussian-distributed noise term. It was demonstrated that even in the absence of a gold standard, the values of the slope, bias, and the standard deviation of the noise term for all the considered QI methods can be estimated using a maximum-likelihood (ML) approach. These estimated parameters can then be used to rank different QI methods on the basis of precision. The efficacy of the RWT technique was demonstrated in evaluating segmentation methods on the task of estimating the apparent diffusion coefficient from diffusion-weighted magnetic resonance imaging (MRI) scans,^{17, 18} and on the task of estimating the left ventricular ejection fraction from cardiac cine MRI sequences.¹⁹ The RWT technique was then advanced further and the efficacy of the resultant no-gold-standard evaluation (NGSE) technique²⁰ was demonstrated in objectively evaluating reconstruction methods for SPECT on the task of quantifying regional uptake.^{20, 21} Further, the technique was applied to clinical oncological PET images to evaluate segmentation methods on the task of measuring metabolic tumor volume.²² While the findings from these studies are encouraging, an important assumption in these existing evaluation techniques is that the noise between measurements obtained using different QI methods is independent of each other. The noise with different QI methods arises in the process of measuring the same true value, and then can be correlated. Thus, this assumption is often violated. To address this issue, we propose an advanced NGSE technique that accounts for the presence of such correlated noise. We start by presenting the theory of this technique.

2. METHODS

2.1. Theory

Consider a clinical scenario where a total of P patients are being scanned by an imaging system. From the acquired data of each patient, a set of K QI methods are used to measure certain quantitative values. Such quantitative values can be the mean activity concentration within different organs of interest. Our objective is to estimate the parameters that can describe the relationship between the true and measured values, without access to a gold standard. These estimated parameters can then be used to rank the QI methods.

We assume that there exists a linear stochastic relationship between the true and measured values. This relationship is parameterized by a slope, bias, and Gaussian-distributed noise term. We note that this noise arises in the process of measuring the same true values but with different methods. Thus, the noise is expected to be correlated. We model this correlated noise by a zero-mean multi-variate Gaussian distribution denoted by $\mathcal{N}\left(0,\mathbf{C}\right)$. Specifically, the diagonal elements of C, i.e., $\left\{{\sigma }_k^2\right\}$ denote the variance of the noise of each method. The off-diagonal elements of C, i.e., $\left\{{\sigma }_{k,k^{\prime }}\right\}$ denote the covariance of the noise between methods k and k^′. For the p^th patient, denote the true value by a_p and the estimated value using the k^th method by ${\widehat{a}}_{p,k}$. Additionally, denote the slope and bias of the k^th method by u_k and u_k, respectively. For the p^th patient, we can then write the relationship between the true and measured values as

$$\left[\begin{array}{c}{\widehat{a}}_{p,1}\\ {\widehat{a}}_{p,2}\\ ⋮\\ {\widehat{a}}_{p,K}\end{array}\right]=\left[\begin{array}{cc}{u}_1& v_1\\ u_2& v_2\\ ⋮& ⋮\\ u_K& v_K\end{array}\right]\left[\begin{array}{c}{a}_p\\ 1\end{array}\right]+\mathcal{N}\left(0,\mathbf{C}\right).$$

Denote the vector ${\left[{\widehat{a}}_{p,1},{\widehat{a}}_{p,2},\dots ,{\widehat{a}}_{p,K}\right]}^T$ by ${\widehat{\mathit{A}}}_p$, the matrix containing {u_k} and {v_k} by Θ, and the vector [a_p, 1]^T by A_p. Based on Eq. (1), obtaining the probability of observing ${\widehat{\mathit{A}}}_p$ given the knowledge of $\left\{{\mathit{A}}_p,\mathbf{\Theta },\mathbf{C}\right\}$ depends on the true values, which are unknown. To address this issue, we next assume that the true values are sampled from a four-parameter beta distribution (FPBD) parameterized by a vector Ω.²⁰ This FPBD incorporates the fact that the true values lie within a certain range. Additionally, the FPBD provides the capability to model a wide variety of the ranges and shapes of the true distribution.

Let $\widehat{\mathcal{A}}=\left\{{\widehat{\mathit{A}}}_p,p=1,2,\dots ,P\right\}$ denote the collection of measurements made by all the K methods from all the P patients. The NGSE technique uses an ML approach to estimate the values of $\left\{\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right\}$ that maximize the probability of observing $\widehat{\mathcal{A}}$. The ML estimate of $\left\{\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right\}$ is given by

$${\left\{\widehat{\mathbf{\Theta }},\widehat{\mathbf{C}},\widehat{\mathit{\Omega }}\right\}}_{\text{ML}}=\underset{\mathbf{\Theta },\mathbf{C},\mathit{\Omega }}{\arg \max }\left\{\text{pr}\left(\widehat{\mathcal{A}}|\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right)\right\},$$

where $\text{pr}\left(\widehat{\mathcal{A}}|\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right)$ denotes the probability of observing $\widehat{\mathcal{A}}$ given the knowledge of $\left\{\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right\}$. We note that obtaining this probability does not require any knowledge of the true values.

The expression for $\text{pr}\left(\widehat{\mathcal{A}}|\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right)$ was determined to obtain the ML estimates using a constrained optimization technique based on the interior-point algorithm.²³ From the estimated parameters, we used the slope terms $\left\{{\widehat{u}}_k\right\}$ and noise standard deviation terms $\left\{{\widehat{\sigma }}_k\right\}$, i.e., the square root of the diagonal elements of the covariance matrix, to compute the noise-to-slope ratio (NSR) for each method. For the k^th method, the NSR is given by

$${\text{NSR}}_k=\frac{{\widehat{\sigma }}_k}{{\widehat{u}}_k}.$$

The NSR evaluates QI methods on the basis of precision,^{15, 16, 20} and a lower value indicates a more precise estimation performance.

2.2. Evaluating the NGSE technique using numerical experiments

We evaluated the performance of the NGSE technique using multiple numerical experiments. In each experiment, P = 200 true values were sampled from a known FPBD. From these true values, noisy measured values were generated for K = 3 hypothetical QI methods. Each method yielded outputs that were linearly related to the true values by a slope of u_k and bias of u_k. The variance of the noise of each method was characterized by the diagonal elements of the covariance matrix C. Additionally, the covariance of the noise between different methods were characterized by the off-diagonal elements of C. These noisy measurements were then input to the NGSE technique to estimate $\left\{\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right\}$. From these estimated parameters, we used the slope terms $\left\{{\widehat{u}}_k\right\}$ and the noise standard deviation terms $\left\{{\widehat{\sigma }}_k\right\}$ to compute the NSR for all methods to rank them based on precision, as described in Sec. 2.1.

In this evaluation, we sampled the 200 true values from FPBD for 4 combinations of Ω such that different ranges and shapes of the true distribution were modeled. To evaluate the sensitivity of the NGSE technique to correlated noise, we generated two sets of QI methods for each combination of Ω. The first set of methods had lower correlated noise with $\left\{{\sigma }_{1,2},{\sigma }_{1,3},{\sigma }_{2,3}\right\}=\left\{0.004,0.008,0.012\right\}$. In contrast, the second set of methods had higher correlated noise with $\left\{{\sigma }_{1,2},{\sigma }_{1,3},{\sigma }_{2,3}\right\}=\left\{0.015,0.02,0.03\right\}$. For both sets, the values of slope {u_k}, bias {u_k}, and variance of the noise $\left\{{\sigma }_k^2\right\}$ of the three methods were set to $\left\{1.1,0.9,1.05\right\}$, $\left\{0.1,0.2,0.3\right\}$, and $\left\{0.04,0.09,0.2025\right\}$, respectively. Finally, for each combination of $\left\{\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right\}$, we repeated the experiment for 20 different noise realizations. Thus, we evaluated the performance of the NGSE technique for a total of 4×2×20 = 160 trials.

3. RESULTS

We first present the performance of the NGSE technique for the set of hypothetical QI methods that had lower correlated noise. The means and standard deviations of the estimated slope $\left\{{\widehat{u}}_k\right\}$, noise standard deviation $\left\{{\widehat{\sigma }}_k\right\}$ and resultant NSR for all considered methods are reported in Table 1. As described in Sec. 2.2, these statistics were computed from a total of 80 trials. In each trial, we considered either a different combination of $\left\{\mathbf{\Theta },\mathbf{C},\mathit{\Omega }\right\}$ or a different noise realization of the synthetic measurements given the true values. We observe that the NGSE technique reliably estimated the slope, noise standard deviation, and consequently the NSR values. From the estimated NSR values, the NGSE technique accurately ranked the methods for 78% of the 80 trials. Further, the technique correctly identified method 1 as the most precise method for 97% of the 80 trials.

Table 1:

The means and standard deviations of slope, noise standard deviation, and resultant NSR estimated using the NGSE technique for the set of methods that had lower correlated noise.

Method index	True slope	Estimated slope	True noise standard deviation	Estimated noise standard deviation	True NSR	Estimated NSR
1	1.10	1.11 ± 0.08	0.20	0.17 ± 0.09	0.18	0.16 ± 0.08
2	0.90	0.89 ± 0.07	0.30	0.30 ± 0.06	0.33	0.35 ± 0.09
3	1.05	1.06 ± 0.09	0.45	0.44 ± 0.04	0.43	0.42 ± 0.07

We then present in Table 2 the results for the set of methods that had higher correlated noise. We again observe that the NGSE technique reliably estimated the slope, noise standard deviation, and resultant NSR. For 87% of the 80 trials, the technique yielded accurate rankings of the considered methods. Further, the technique correctly identified that method 1 was the most precise for 97% of the 1, 600 trials.

Table 2:

The means and standard deviations of slope, noise standard deviation, and resultant NSR estimated using the NGSE technique for the set of methods that had higher correlated noise.

Method index	True slope	Estimated slope	True noise standard deviation	Estimated noise standard deviation	True NSR	Estimated NSR
1	1.10	1.13 ± 0.09	0.20	0.17 ± 0.08	0.18	0.15 ± 0.08
2	0.90	0.91 ± 0.07	0.30	0.30 ± 0.06	0.33	0.34 ± 0.08
3	1.05	1.07 ± 0.09	0.45	0.44 ± 0.05	0.43	0.42 ± 0.07

4. DISCUSSION AND CONCLUSION

For clinical translation of QI, there is an important need for techniques that can objectively evaluate QI methods with patient data. In this context, existing statistical techniques assume that the noise between measurements obtained using different methods is independent of each other. However, this assumption can often be violated since the noise arises in the process of measuring the same true value. To address this issue, we developed an NGSE technique that models this correlated noise by a multi-variate Gaussian distribution.

Our results from the numerical experiments (Tables 1 and 2) showed that the NGSE technique yielded reliable estimates of slope, noise standard deviation, and consequently the NSR for all hypothetical QI methods. Additionally, the technique yielded accurate rankings of these methods for 83% of the total 160 trials. Further, the technique was able to identify the most precise method for 97% of the cases. This observation is especially important since when evaluating different QI methods, the objective is typically to find the method that yields the most reliable performance.²² All these results demonstrated that in controlled settings, where the true and measured values were linearly related by design, the NGSE technique was able to reliably rank QI methods without access to any knowledge of the ground truth.

The results motivate further evaluation of the proposed technique with QI methods that are developed for clinical applications. These include methods developed for reconstruction, post-reconstruction processing, segmentation, and quantification. Often, such methods are evaluated using strategies that rely on the availability of a ground truth. Further, this ground truth may not be relevant to the clinical task. For example, segmentation methods are evaluated using metrics such as the Dice score and Hausdorff distance, which quantify spatial overlap and shape similarity, respectively, between the estimated segmentation and a certain ground-truth segmentation. Such evaluation then requires access to the true segmentation, which is typically unavailable. Usually manual segmentations are used as a surrogate for the ground truth, but these can be erroneous and suffer from intra and inter-reader variability.²⁴ Similarly, denoising methods for low-dose images are evaluated by comparing the denoised image to a certain normal-dose image using metrics of structural similarity index and root mean square error. However, the normal-dose image is also noisy, and thus provides a limited measure of ground truth. More importantly, it is unclear whether the evaluation based on those conventional metrics correlates with the clinical task.^25–27 Thus, these methods should preferably be evaluated based on clinical-task performance.²⁸ The proposed NGSE technique provides a mechanism to perform evaluation on clinically relevant quantitative tasks and without access to the ground truth.

One limitation of the proposed technique is that the true and measured values are assumed to be linearly related. This linear relationship is desirable in QI since it ensures that the measured quantitative value is linearly related to the biological effect. However, this assumption of linearity may not always hold true in QI. To address this issue, one strategy is to check whether the measurements made by different methods are linearly related to each other. This will increase the confidence that the measured values are also linearly related to the ground truth.²² A second limitation is that the NGSE technique requires many patient images since multiple parameters need to be estimated. One way to reduce the required number of input images is to incorporate the prior information of the parameters to be estimated.²⁹ Thus, extending the proposed technique to incorporate such prior knowledge is an important research direction.

In conclusion, our study demonstrated the ability of the proposed NGSE technique to accurately rank different QI methods in the presence of correlated noise, and without the need for any knowledge of the ground truth. The results motivate further evaluation of the technique with realistic simulation studies and patient data.