No-gold-standard evaluation of image-acquisition methods using patient data

Abstract

Several new and improved modalities, scanners, and protocols, together referred to as image-acquisition methods (IAMs), are being developed to provide reliable quantitative imaging. Objective evaluation of these IAMs on the clinically relevant quantitative tasks is highly desirable. Such evaluation is most reliable and clinically decisive when performed with patient data, but that requires the availability of a gold standard, which is often rare. While no-gold-standard (NGS) techniques have been developed to clinically evaluate quantitative imaging methods, these techniques require that each of the patients be scanned using all the IAMs, which is expensive, time consuming, and could lead to increased radiation dose. A more clinically practical scenario is where different set of patients are scanned using different IAMs. We have developed an NGS technique that uses patient data where different patient sets are imaged using different IAMs to compare the different IAMs. The technique posits a linear relationship, characterized by a slope, bias, and noise standard-deviation term, between the true and measured quantitative values. Under the assumption that the true quantitative values have been sampled from a unimodal distribution, a maximum-likelihood procedure was developed that estimates these linear relationship parameters for the different IAMs. Figures of merit can be estimated using these linear relationship parameters to evaluate the IAMs on the basis of accuracy, precision, and overall reliability. The proposed technique has several potential applications such as in protocol optimization, quantifying difference in system performance, and system harmonization using patient data.

1. INTRODUCTION

Quantitative imaging, i.e. the measurement and use of numerical/statistical features from medical images to facilitate clinical decision making^1,2, is finding applications in many diagnostic and therapeutic procedures. For example, the absorbed organ doses measured using single-photon emission computed tomography (SPECT) for dosimetry³, imaging biomarkers to predict and monitor cancer response such as metabolic tumor volume and total lesion glycolysis measured from positron emission tomography (PET)^4–7, tumor size measured with magnetic resonance imaging (MRI)⁸, apparent diffusion constant measured with diffusion MRI^9–12, myocardial blood flow measured using PET to diagnose cardiac diseases^13,14, and bone-mineral density estimated using dual x-ray absorptiometry for diagnosing osteoporosis¹⁵.

Given the variety of applications, several new and improved image-acquisition methods (IAMs) are being developed. In this context, an IAM refers to any combination of modalities, instrumentation, or protocols. However, these IAMs often have design tradeoffs that affect the reliability (i.e. accuracy and precision) of the measured quantitative feature from the image. For example, consider the task of measuring the radiotracer uptake in a tumor using a SPECT scanner for a dosimetry study. For this task, a scanner that can provide both high resolution (to accurately capture the uptake at the edges leading to accurate measurements) and high signal-to-noise ratio (SNR) (leading to precise measurements) is ideal. To obtain high resolution, a SPECT scanner with a long fine-bore collimator is required, but such a collimator would allow only a very few photons to pass through, yielding images with poor SNR, as in Fig. 1. Conversely, if a collimator with large-bore diameters is used, the resolution of the images will be degraded. Due to such tradeoffs, the choice of the best IAM is not often obvious, and studies comparing the IAMs on the quantitative task are required.

Figure 1.

Two different sets of patients scanned using different SPECT scanners. The imaging procedure is performed to measure some quantitative value about the patient (e.g. the tumor uptake or tumor volume). The objective of the NGS framework is to rank such IAMs based on their reliability in measuring the quantitative value, in the absence of any gold standard.

It is highly desirable to compare the above IAMs using patient data, since eventually these IAMs must be used with patients. However, comparing IAMs using patient data is often impossible or impractical due to the lack, imprecision, or inaccuracy of available gold standards. Thus, animal, phantom and realistic simulation studies are instead used. However, animal studies are not definitive since the organ sizes and geometries in animals are different from humans. Similarly, physical phantoms often do not model anatomy, physiology, or patient variability well, and simulations may not model some aspects of the biology or instrumentation. These limitations reduce the confidence of physicians in the output of these studies, and thus the clinical translation and acceptance of new and improved IAMs is delayed. A no-gold-standard (NGS) evaluation technique (Fig. 1) to compare the performance of IAMs on quantitative tasks with patient data is thus highly desirable.

NGS techniques have been previously developed for evaluating quantitative imaging methods. Kupinski et al^16,17, in a seminal paper, proposed an NGS technique, referred to as the regression-without-truth (RWT) technique, to evaluate quantitative imaging methods in the absence of ground truth in the context of ranking different cardiac-ejection fraction methods¹⁸. The technique was used to evaluate segmentation methods for cardiac cine MR images¹⁹. We then extended the NGS technique to a larger range of quantitative imaging tasks and extensively validated the method, both using numerical experiments and evaluating reconstruction methods for quantitative SPECT^20,21 and segmentation methods for diffusion MRI^22,23. Further, to address the practical difficulties in applying the NGS technique to patient data, statistical methods were developed, yielding an NGS framework²⁴. This framework was applied to evaluate tumor-segmentation methods for PET images of patients with head-and-neck cancer²⁴. However, each of these existing NGS techniques require that, for each evaluated method, the data be available from all set of patients. Thus, to use these methods to evaluate IAMs would require that all the patients are scanned using each of the different IAMs. This is expensive, time consuming and can lead to increased radiation dose, for example in CT imaging. A more clinically practical scenario is where different sets of patients have been scanned using different IAMs. Our objective in this study was to design a NGS technique that could use patient data, where different sets of patients were scanned using different IAMs, to objectively evaluate the different IAMs on the quantitative task.

2. METHODS

2.1 Basis of proposed approach

Consider a clinical study to measure a quantitative value that describes some underlying physical process/anatomical feature about the patient. A different set of patients is imaged using each IAM. The objective of the NGS evaluation algorithm, as illustrated by an example in Fig. 1, is to use these patient images to compute figures of merit that can evaluate the different IAMs based on how reliably they estimate the unknown true quantitative value.

Assume that the true values, denoted by a_p for the p^th patient, have been sampled from an unknown distribution. We assume that this unknown distribution is unimodal, which is reasonable, especially after the patients have been stratified according to factors such as age, disease type, and obesity. We model this true distribution using the beta distribution (parameterized by two unknown shape parameters α and β), which, as illustrated in Fig. 2, allows us to model different possible shapes of the true value distribution. Also, consider that the p^th patient was scanned using the k^th IAM, yielding the measurement â_p,k.

Fig. 2.

A schematic illustrating that various values for α and β in a beta distribution could model different shapes for the distribution of true values.

The measured quantitative values are the result of a specific image-formation and quantitation process applied to the object being imaged. Thus, a statistical relationship between the true and measured values using the different IAMs is expected. Often, due to the linearity of the image-formation and quantitation process, it is justified to assume that this relationship is linear²⁵. This linearity has been observed in realistic simulation and phantom studies in quantitative SPECT²⁰, diffusion MRI²⁶, DaTScan imaging²⁷, PET imaging²⁸, CT imaging^29,30. Also, we have observed in clinical FDG-PET data of patients with head-and-neck cancer that the tumor-volume measurements using different imaging methods are consistent with the linearity assumption²⁴. Linearity between true and measured quantitative values is highly desirable^25,29,30 since it ensures that any changes in the true value are proportionately reflected in the measured value. The assumption of linearity also simplifies the computation of precision³¹. Due to all these reasons, imaging-system designers strive for linearity. Thus, we focus on quantitative applications where the assumption of linearity between true and measured values could be justified. The linear relationship is described as below:

$${\widehat{a}}_{p,k}=u_k{a}_p+v_k+\mathcal{N}(0,{\sigma }_k^2),$$ (1)

where u_k,v_k and $\mathcal{N}(0,{\sigma }_k^2)$ denote the slope, bias, and normally-distributed zero-mean noise term with variance ${\sigma }_k^2$. As Fig. 3 illustrates, under the linearity assumption, the method with the highest value of the noise standard deviation term and bias term would be considered the most imprecise and the most inaccurate, respectively. Further, the difference between the noise standard deviations or the biases of different IAMs could quantitatively compare the IAMs. Thus, under the assumption of this linear relationship, if we could estimate these terms, we could compute figures of merit that could compare the different IAMs.

Fig. 3.

A scatter plot of the true vs. measured quantitative values for three different IAMs. The corresponding linear-relationships for the three IAMs are superimposed on the scatter plot, and demonstrate that IAMs that are most noisy (Method 1) and most inaccurate (Method 3) have the highest noise standard deviation term and bias term, respectively.

Before describing the mathematical formalism for estimating these parameters, we first provide an intuition into how these parameters can be estimated. The intuition is very similar to the intuition provided for the RWT and existing NGS techniques in Jha et al²⁴. First note, as illustrated in Fig. 4a, that the distribution of measured values for the k^th IAM can be described by the parameters {α,β,u_k,v_k,σ_k}.

Fig. 4.

Fig. 4a: Schematic illustrating the parameterized form for the distribution of measured values

Fig. 4b: Schematic illustrating the intuition behind how the NGS technique can estimate the model parameters

We could therefore design a technique to estimate {α,β,u_k,v_k,σ_k} that maximizes the likelihood of the data from all the measurements made with the k^th IAM (a maximum-likelihood technique). However, we observe via numerical experiments that the problem is ill-posed, in that the terms are not estimated uniquely. These parameters become more identifiable, or uniquely defined, when we consider images from different IAMs if the true values for each IAM was drawn from the same underlying distribution of true values. This is because now we have different sets of independent measurements, each of which are characterized by the same parameters {α,β} but a different set of {u_k,v_k,σ_k}, as illustrated in Fig. 4b.

This premise has been previously used to develop RWT and similar NGS evaluation techniques^16,17,20 for cases where all patients have been imaged using all methods. A general version of this same idea can be used to perform NGS evaluation in the absence of the true values even when different sets of patients are scanned via different IAMs, as we describe in the next sub-section.

2.2 Theory

Using Eq. 1, the measured value using the k^th IAM, â_p,k is normally distributed with a mean of u_ka_p+v_k, i.e.

$$pr\left({\widehat{a}}_{p,k}\mid a_p,u_k,v_k,{\sigma }_k\right)~\mathcal{N}(u_k{a}_p+v_k,{\sigma }_k^2),$$ (2)

where pr(x) denotes the probability of a random variable x. However, the above distribution depends on a_p, which is not known. To circumvent this issue, we assume that a_p has been sampled from a parametric distribution. We choose this distribution to be a beta distribution characterized by parameters Ω={α,β}.

Next note that the joint distribution of â_p,k and a_p can be written as

$$pr\left({\widehat{a}}_{p,k},a_p\mid {\mathrm{\Theta }}_k,\mathrm{\Omega }\right)=pr\left({\widehat{a}}_{p,k}\mid a_p,{\mathrm{\Theta }}_k\right)pr(a_p\mid \mathrm{\Omega }).$$

Marginalizing on both sides over the random variable a_p yields

$$pr\left({\widehat{a}}_{p,k}\mid {\mathrm{\Theta }}_k,\mathrm{\Omega }\right)=\int d{a}_{p}pr\left({\widehat{a}}_{p,k}\mid a_p,{\mathrm{\Theta }}_k\right)pr\left(a_p\mid \mathrm{\Omega }\right),$$

After the marginalization, the distribution of â_p,k is no more dependent on a_p. Finally, under the assumption that the true values are independent of each other, the joint distribution of all the measurements i.e. pr({â_p,k},p=1,2,…P) can be written simply as the product of the individual distributions of â_p,k’s, i.e.

$$pr\left(\{{\widehat{a}}_{p,k}\}\mid \{{\mathrm{\Theta }}_k\},\mathrm{\Omega }\right)=\prod _{p=1}^p\int d{a}_{p}pr\left({\widehat{a}}_{p,k}\mid a_p,{\mathrm{\Theta }}_k\right)pr\left(a_p\mid \mathrm{\Omega }\right),$$ (3)

where Π is a notation for the product of the distributions, and where pr(â_p,k|a_p,Θ_k) is given by Eq. 2.

From Eq. 3, we have the likelihood of all the measurements, parameterized in terms of Θ_k and Ω, and with no dependency on a_p. Given the likelihood, we can use the maximum-likelihood (ML) procedure to estimate {Θ_k} and Ω, i.e.

$${\left\{\left\{{\widehat{\mathrm{\Theta }}}_k\right\},{\widehat{\mathrm{\Omega }}}_k\right\}}_{ML}={\mathit{argmax}}_{\{{\mathrm{\Theta }}_k\},\mathrm{\Omega }}pr\left(\{{\widehat{a}}_{p,k}\}\mid \{{\mathrm{\Theta }}_k\},\mathrm{\Omega }\right),$$ (4)

where argmax_y f(y) is the value of y at which the function f(y) is maximized. Note that the ML estimator has several important properties that make it an optimal technique for estimating these parameters. For example, if an efficient estimator exists, the ML estimator is efficient, i.e. unbiased and attains the lowest bound on the variance of any estimator, also referred to as the Cramer Rao lower bound.

From the ML estimates of the linear-relationship parameters, we can compute different figures of merit (FoMs) as required by the clinical application. For example, accuracy, precision, and reliability (accuracy and precision) of the different IAMs are quantified using the bias term v_k, the std. dev. σ_k and the mean square error (MSE) ( $v_k^2+{\sigma }_k^2$). These FoMs could be used to rank the IAMs, which is required in the context of optimizing IAM performance. They could also be used to quantify the difference in IAM performance on the basis of bias, standard deviation and mean square error of the quantitative values.

2.3 Implementation

From the above derivation, the NGS evaluation problem reduces to an optimization task. The optimization routine is simplified by taking the logarithm on both sides of Eq. 3 (which reduces products to sums). Next, using a quasi-Newton optimization technique (we use the Interior-point algorithm in the software Matlab®), we can find this ML solution.

The above treatment, although based on the same idea as existing NGS techniques^16,17,20, is general in that at no point does it require image data where the same patient has been scanned using all the available IAMs. Further, no constraints are placed on the number of true values from a given IAM. Thus, even if different number of patient studies are available from each IAM, a very realistic scenario, the NGS technique can still be used to estimate the model parameters. All these factors make the proposed NGS technique very general and practical.

3. EXPERIMENTS AND RESULTS

The proposed NGS technique was validated in the context of evaluating different IAMs on the task of estimating activity concentration in a known region of interest. Here we present preliminary results from this validation study.

Note that to validate the proposed technique on the task of evaluating IAMs, a study where the true quantitative values are known is needed for reference. For this purpose, we conducted highly realistic simulation studies using anthropomorphic phantoms. SPECT imaging of an I-131 labeled anti-CD20 antibody used for radio-immunotherapy of non-Hodgkin’s lymphoma was simulated. The object database consisted of 42-patient digital phantom population. The organ time activity curves and organ volumes for these phantoms were based on measurements from patient data, and were based on the NURBS-based Cardiac-Torso (NCAT) phantom. A Philips Precedence SPECT system with a 9.525 mm thick Nal crystal and a high-energy general-purpose collimator was simulated. Low-noise projections of eight organs were generated from photons with emission energies of 364, 637, and 722 keV and the appropriate abundances. The SimSET software, in conjunction with angular response functions tables, which accurately modeled the collimator and detector effects³², was used to simulate the projections. The software to simulate the SPECT system was validated by comparison with experimental data³². Using this software, low-noise projections were scaled and summed according to the emission abundance and activity distributions in the 42-patient phantom population, yielding 42 low-noise projection datasets. Subsequently, 50 independent noisy projection datasets were generated from each low-noise dataset using a Poisson pseudo-random-number generator.

The patient population was split into two halves. For each half, the projection data was reconstructed using two different methods. This simulated the case where different sets of patients are scanned using different IAMs. Both these reconstruction methods were OSEM-based, but provided different levels of compensation. The projection data corresponding to the first set were reconstructed using the AGS method, which compensated for attenuation (A), scatter (S), and geometric response (G) of the SPECT system. The second set were reconstructed using ADS.DWN method, which additionally compensated for collimator-detector response (D) and down-scatter from high-energy photons (DWN)³². The activity concentration in eight different VOIs corresponding to the eight different organs were estimated assuming knowledge of the true organ VOIs.

The proposed NGS technique was used to rank these two IAMs based on the task of quantifying activity concentration in the different organs. 50 trials of this process were conducted. In each trial, data from different patient subsets were reconstructed using the two reconstruction methods, thus simulating population variability. The entire process was repeated for the 50 different noise realizations of the data. Thus, in total, 2500 (50X50) trials of the NGS technique were conducted.

Since this was a simulation study, the true values of the various FoMs for the two IAMs were determined from the available ground truth. It was observed that the metrics estimated using the NGS technique yielded similar rankings as the true rankings on the basis of accuracy for more than 90% of the IAMs. This is as illustrated in Fig. 5.

Fig. 5.

The mean bias for each of the 50 noise realizations when (a) ground truth was known and (b) the NGS technique was used assuming no knowledge of true values. The error bars represent 95% confidence intervals. The ranking obtained with the NGS technique using the bias FoM were the same as the true rankings for more than 90% of the 2500 trials.

4. CONCLUSIONS

A no-gold-standard technique developed to evaluate image-acquisition methods (IAMs) when different sets of patients are scanned using different IAMs has been developed. Preliminary results demonstrate the potential of this technique in ranking the imaging methods on the basis of accuracy. In future, we propose to conduct several additional studies to comprehensively validate the performance of this technique. The proposed technique has several potential applications including protocol optimization for multi-center trials and scanner harmonization. A limitation of the technique is that it requires data from a large number of patients, which might not always be available. We are currently developing strategies that could help us overcome this requirement³³.