Ideal-observer computation in medical imaging with use of Markov-chain Monte Carlo techniques

Abstract

The ideal observer sets an upper limit on the performance of an observer on a detection or classification task. The performance of the ideal observer can be used to optimize hardware components of imaging systems and also to determine another observer’s relative performance in comparison with the best possible observer. The ideal observer employs complete knowledge of the statistics of the imaging system, including the noise and object variability. Thus computing the ideal observer for images (large-dimensional vectors) is burdensome without severely restricting the randomness in the imaging system, e.g., assuming a flat object. We present a method for computing the ideal-observer test statistic and performance by using Markov-chain Monte Carlo techniques when we have a well-characterized imaging system, knowledge of the noise statistics, and a stochastic object model. We demonstrate the method by comparing three different parallel-hole collimator imaging systems in simulation.

1. INTRODUCTION

We advocate the position that to properly quantify image quality one must clearly define the task that is to be performed and the observer who will be performing this task.¹ Tasks in medical imaging are typically detection tasks, e.g., determining whether a tumor is present, or estimation tasks, e.g., determining the value of a clinically relevant parameter. Observers can be human, human-model, or ideal observers. An ideal observer is a mathematical construct that is optimal in a well-defined sense.

The class of ideal observers includes the Bayesian ideal observer,² which sets an absolute upper limit on task performance as measured by most common figures of merit. (For simplicity, any reference to ideal observers from now on implies Bayesian ideal observers.) Because no other observer can outperform the ideal observer by these standards, it is reasonable to use ideal-observer performance as a figure of merit in comparing imaging systems. That is, the ideal observer is a useful tool for comparing and optimizing imaging systems. Furthermore, it is often useful to compare the performance of other observers relative to the performance of the best observer.

The ideal observer requires complete knowledge of the statistics of the images or image data. Thus computing the ideal observer is often considered either too difficult or impossible for image data, which tend to be large; e.g., images of size 512 × 512 are characterized by a probability density function of a 512² random vector. Researchers often resort to unrealistic assumptions that dramatically simplify the math, such as assuming fixed backgrounds.³ Clarkson and Barrett⁴ worked out the ideal-observer test statistic for a number of situations in which the statistics of the backgrounds followed certain distributions, including Gaussian and Laplacian distributions. A great deal of research has been done on the performance of nonoptimal observers such as human observers and model observers in different types of backgrounds. See studies by Abbey and Barrett,⁵ Burgess et al.,⁶ and Bochud et al.⁷ for examples. However, the performance of the ideal observer has not been computed for complicated backgrounds that are not characterized by simple density functions.

In this paper we present a method for computing the ideal observer in simulation for a class of stochastic object models. The computation of the ideal observer is accomplished by using Markov-chain Monte Carlo techniques^8,9 (to be referred to hereafter as Markov-Chain Monte Carlo). This method can be used for hardware optimization or comparison. In this paper we present the results of a simulation study in which we compare the ideal-observer performance of three different parallel-hole collimator imaging systems.

2. BACKGROUND

The process of data acquisition in imaging may be represented mathematically by

$$\mathbf{g}=\mathcal{H}\mathbf{f}+\mathbf{n},$$ (1)

where f is the continuous object being imaged, H is a continuous-to-discrete operator representing the imaging system, n is the measurement noise, and g is an M × 1 vector of image data. In radiological imaging, the object f may represent the three-dimensional activity distribution of radio tracer within a body or the two-dimensional fluence of x-rays impinging on a detector. The imaging system H describes the mapping of continuous objects to discrete image data. The statistics of the random vector n may depend on the object f, e.g., Poisson noise. The image data g may be an image as in the case of direct digital detectors or may require processing before being shown to a human observer as in the case of sinogram data. In either case the ideal observer uses the raw image data g and does not require any reconstruction or processing.

For a linear system, the continuous-to-discrete imaging operator H can be more clearly represented by

$$g_m={\int }_s\text{d}\mathbf{r}{h}_m(\mathbf{r})f(\mathbf{r})+n_m,$$ (2)

where g_m and n_m are the mth elements of the vectors g and n, respectively, r is a spatial coordinate (two or three dimensional), f(r) is the object f, h_m(·) is the mth detector sensitivity function, and S is the field of view. The entire operator H is composed of M functions h_m(·).

As discussed above, classification tasks in medicine consist of determining the presence of a signal, e.g., tumor. In this paper we consider signal-known-exactly tasks; we consider the tumor to be some known signal f_s in a random background f_b. As a result there exist two possible hypotheses for any given image: signal absent (H₀) or signal present (H₁). We mathematically represent imaging under these two hypotheses as

$$H_0:\mathbf{g}=\mathcal{H}{\mathbf{f}}_b+\mathbf{n},$$ (3)

$$H_1:\mathbf{g}=\mathcal{H}({\mathbf{f}}_b+{\mathbf{f}}_s)+\mathbf{n}.$$ (4)

For convenience we define the background image and signal image to be

$$\mathbf{b}=\mathcal{H}{\mathbf{f}}_b,$$ (5)

$$\mathbf{s}=\mathcal{H}{\mathbf{f}}_s.$$ (6)

A. Ideal Observer

The ideal observer employs the likelihood ratio, or any monotonic transformation thereof, to make decisions.¹⁰ The likelihood ratio is defined as

$$\mathrm{\Lambda }(\mathbf{g})=\frac{\text{pr}(\mathbf{g}\mid H_1)}{\text{pr}(\mathbf{g}\mid H_0)},$$ (7)

where pr(g|H_i) is the probability density of the image data under hypothesis H_i. An observer compares the test statistic, e.g., Λ(g), to a predetermined critical value (or threshold) and makes a decision accordingly. A fixed critical value corresponds to a single point on a receiver operating characteristic (ROC) curve.^11,12 An ROC curve is generated by varying this critical value over the real line. The area under the ROC curve (AUC)¹² is a commonly used performance metric that is maximized by the ideal observer.

Traditionally, to compute the ideal observer, researchers were forced to make strict and unrealistic assumptions regarding the imaging system. For example, if we were to assume that the background b and the signal s were nonrandom quantities, then the randomness in g would be due only to noise. Hence the densities pr(g|H_i) would be known, and the likelihood ratio could be easily calculated for various noise models. However, the stochastic nature of normal patient anatomy has a dramatic effect on the performance of an observer.^5,13 For example, a lesion in a mammogram may be very easy or very difficult to detect depending upon whether there is dense tissue surrounding the tumor. Ultimately, if the task is to detect a lesion in a mammogram, then the background structure of a mammogram must be accounted for.⁶

B. Object Models

Characterizing the full probability density of b is a very difficult task, which many researchers have addressed.^14,15 However, the probability density pr(b) is particular to the imaging system operator H since $\mathbf{b}=\mathcal{H}{\mathbf{f}}_b,$ and is not especially useful if we wish to perform hardware optimizations. We require a characterization of the randomness in the background objects f_b and not the background images b. However, for the sake of notational simplicity we will often refer to $\text{pr}({\mathbf{f}}_b\mid \mathcal{H})$ as pr(b), which we can know in simulation given knowledge of our object model and a complete description of the imaging system H.

We will employ a stochastic object model known as the lumpy object model, developed by Rolland and Barrett,¹³ although this method can easily be extended to other parameterized object models. The lumpy object model may be represented by the equation

$${\mathbf{f}}_b=f_b(\mathbf{r})=\sum _{n=1}^{N}L(\mathbf{r}-{\mathbf{c}}_n\mid a,s),$$ (8)

where r is a two- or three-dimensional spatial coordinate, N is the random number of lumps in the object, L(·) is the lump function, c_n is the center of the nth lump, a is the magnitude of the lump function, and s is the width. Typically L(·) is chosen to be a Gaussian function, but other functions can be chosen as well. The variable N is a Poisson-distributed random variable with mean N̄, and the centers c_n are uniformly distributed within the field of view. Note that the randomness in this object model is completely characterized by the random variables N and {c_n}. Examples of two-dimensional lumpy objects with different parameters are given in Fig. 1.

Fig. 1.

Examples of two-dimensional lumpy objects. (a) Parameters N = 39, a = 1, and s = 10; (b) parameters N = 348, a = 1, and s = 4. The field of view (FOV) for both objects is 128 × 128.

It follows from Eqs. (2) and (5) that the mth background element is given by

$$b_m=\sum _{n=1}^N{\int }_S\text{d}\mathbf{r}{h}_m(\mathbf{r})L(\mathbf{r}-{\mathbf{c}}_n\mid a,s).$$ (9)

Thus, to generate a single pixel of a noise-free and signal-free background image, we need to compute the N inner products of a sensitivity function with the lumps. To generate an entire image, we compute these inner products over all M pixels, i.e., all sensitivity functions h_m(·).

The lumpy object model is not appropriate for all imaging problems. The methodology we are about to describe, however, can be applied to many parameterized object models. We show in this paper that we can compute the ideal observer for this lumpy object model. This is a clear step forward for ideal-observer computations, which have traditionally been limited to flat or Gaussian background models. In future papers we will discuss the extension of this work to different object models.

3. METHOD

Using our knowledge of the imaging system, its associated noise properties, and the lumpy object model, we would like to calculate likelihood ratios for a set of simulated image data {g}. We begin by rewriting Eq. (7) as

$$\mathrm{\Lambda }(\mathbf{g})=\frac{\int d\mathbf{b}\text{pr}(\mathbf{b})\text{pr}(\mathbf{g}\mid \mathbf{b},H_1)}{\int d\mathbf{b}\text{pr}(\mathbf{b})\text{pr}(\mathbf{g}\mid \mathbf{b},H_0)}.$$ (10)

For a given background b, the randomness in g is due only to the noise, and hence pr(g|b,H_i) characterizes the noise distribution of the imaging system, which we know a priori. The numerator and denominator of Eq. (10) turn out to be exceptionally small numbers, because they are integrals over a product of high-dimensional probability density functions. Thus estimating the ratio accurately by computing the numerator and denominator separately is infeasible.

To make computation of the likelihood ratio possible,¹⁶ we rewrite Eq. (10) as

$$\mathrm{\Lambda }(\mathbf{g})=\int d\mathbf{b}\frac{\text{pr}(\mathbf{b})\text{pr}(\mathbf{g}\mid \mathbf{b},H_1)}{\int \text{d}{\mathbf{b}}^{\prime }\text{pr}({\mathbf{b}}^{\prime })\text{pr}(\mathbf{g}\mid {\mathbf{b}}^{\prime },H_0)},$$ (11)

where we have taken the denominator into the integral. Next, we multiply and divide by pr(g|b, H₀), which leads to

$$\mathrm{\Lambda }(\mathbf{g})=\int d\mathbf{b}\frac{\text{pr}(\mathbf{g}\mid \mathbf{b},H_1)}{\text{pr}(\mathbf{g}\mid \mathbf{b},H_0)}\frac{\text{pr}(\mathbf{b})\text{pr}(\mathbf{g}\mid \mathbf{b},H_0)}{\int \text{d}{\mathbf{b}}^{\prime }\text{pr}({\mathbf{b}}^{\prime })\text{pr}(\mathbf{g}\mid {\mathbf{b}}^{\prime },H_0)}.$$ (12)

Using Bayes’s law we are able to rewrite Eq. (12) as

$$\mathrm{\Lambda }(\mathbf{g})=\int \text{d}\mathbf{b}{\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b})\text{pr}(\mathbf{b}\mid \mathbf{g},H_0),$$ (13)

where Λ_BKE(g|b) is the background-known-exactly (BKE) likelihood ratio defined as

$${\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b})=\frac{\text{pr}(\mathbf{g}\mid \mathbf{b},H_1)}{\text{pr}(\mathbf{g}\mid \mathbf{b},H_0)}$$ (14)

and

$$\text{pr}(\mathbf{b}\mid \mathbf{g},H_0)=\frac{\text{pr}(\mathbf{g}\mid \mathbf{b},H_0)\text{pr}(\mathbf{b})}{\int \text{d}{\mathbf{b}}^{\prime }\text{pr}(\mathbf{g}\mid {\mathbf{b}}^{\prime },H_0)\text{pr}({\mathbf{b}}^{\prime })}.$$ (15)

The BKE likelihood ratio is the ideal-observer test statistic in the situation where the background is fixed and not random. Thus we have derived the random-background likelihood ratio in terms of an integral of the BKE likelihood ratio (a posterior mean).

For an imaging system with Poisson noise

$$\text{pr}(\mathbf{g}\mid \mathbf{b},H_1)=\prod _{m=1}^{M}exp[-(b_m+s_m)]\frac{{(b_m+s_m)}^{g_m}}{g_m!},$$ (16)

and

$$\text{pr}(\mathbf{g}\mid \mathbf{b},H_0)=\prod _{m=1}^{M}exp(-b_m)\frac{{(b_m)}^{g_m}}{g_m!}.$$ (17)

We are thus able to rewrite Eq. (14) as

$${\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b})=\prod _{m=1}^M{\left(1+\frac{s_m}{b_m}\right)}^{g_m}exp(-s_m).$$ (18)

Similarly, for an imaging system with Gaussian noise with covariance K_n, the BKE likelihood ratio becomes

$${\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b})=exp[{(\mathbf{g}-\mathbf{b}-\mathbf{s}/2)}^{†}{K}_n^{-1}\mathbf{s}],$$ (19)

where x^† represents the adjoint of x.

Ideally we would use Monte Carlo techniques to calculate the integral in Eq. (13) by sampling backgrounds from pr(b|g, H₀), evaluating Λ_BKE(g|b), and averaging. Sampling from pr(b|g, H₀) is difficult because the density function is generally not known. As a result, we consider alternate methods. The advantage that we have, and must use, is our knowledge of our model for f_b. We are representing our background objects f_b as lumpy backgrounds that are completely characterized by the number of lumps N (following a Poisson distribution) and the centers {c_n} (uniformly distributed throughout the field of view) for fixed a, s, lump function L(·), and field of view S. We represent the stochastic parameters as an unordered set θ = {c₁, c₂,…, c_N}.

We know that for any given θ there exists a unique b. This relationship allows us to rewrite Eq. (13) as

$$\mathrm{\Lambda }(\mathbf{g})=\int \text{d}\mathit{\theta }{\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b}(\mathit{\theta }))\text{pr}(\mathit{\theta }\mid \mathbf{g},H_0),$$ (20)

where

$$\text{pr}(\mathit{\theta }\mid \mathbf{g},H_0)=\frac{\text{pr}(\mathbf{g}\mid \mathbf{b}(\mathit{\theta }),H_0)\text{pr}(\mathit{\theta })}{\int \text{d}{\mathit{\theta }}^{\prime }\text{pr}(\mathbf{g}\mid \mathbf{b}({\mathit{\theta }}^{\prime }),H_0)\text{pr}({\mathit{\theta }}^{\prime })}.$$ (21)

Unlike pr(b), the function pr(θ) has a simple analytic form that we know from the definition of our object model. For a more detailed explanation of the replacement of b with θ in the above expressions we refer the reader to Appendix A.

A. Markov-Chain Monte Carlo

We want to estimate the likelihood ratio in Eq. (20) by using Monte Carlo integration, i.e.,

$$\widehat{\mathrm{\Lambda }}(\mathbf{g})=\frac{1}{J}\sum _{j=1}^J{\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b}({\mathit{\theta }}^{(j)}))$$ (22)

where each θ ^(j) is a sample from the density pr(θ|g, H₀). To generate the θ ^(j), we construct a Markov chain with pr(θ|g, H₀) as the stationary density for the chain. We choose an initial parameter vector θ ⁽⁰⁾ and a proposal density q(θ|θ ⁽ⁱ⁾) for the Markov chain. Given θ ⁽ⁱ⁾ we draw a sample parameter vector θ̃ from the proposal density and add it to our Markov chain with probability

$$\stackrel{\sim }{\theta }\alpha (\stackrel{\sim }{\mathit{\theta }},{\mathit{\theta }}^{(i)})=min\left[1,\frac{\text{pr}(\stackrel{\sim }{\mathit{\theta }}\mid \mathbf{g},H_0)q(\stackrel{\sim }{\mathit{\theta }}\mid {\mathit{\theta }}^{(i)})}{\text{pr}({\mathit{\theta }}^{(i)}\mid \mathbf{g},H_0)q({\mathit{\theta }}^{(i)}\mid \stackrel{\sim }{\mathit{\theta }})}\right].$$ (23)

If θ̃ is accepted, then θ ⁽ⁱ⁺¹⁾ is θ̃; if θ̃ is not accepted, then θ ⁽ⁱ⁺¹⁾ is θ ⁽ⁱ⁾. The Markov chain is the sequence consisting of

$${\mathit{\theta }}^{(0)},{\mathit{\theta }}^{(1)},\dots ,{\mathit{\theta }}^{(K)},$$ (24)

We adopt a Metropolis–Hastings approach⁸ and design our proposal density to be symmetric, i.e., q(θ̃ |θ ⁽ⁱ⁾) = q(θ ⁽ⁱ⁾|θ̃), resulting in a cancellation of the proposal densities in Eq. (23). Furthermore, we can rewrite pr(θ|g, H₀) using Bayes’s rule [Eq. (21)] and cancel the denominators, leaving an expression that we can compute exactly for a given noise model and object model. In the case of Poisson noise (which we will use for our simulation studies below), the expression in Eq. (23) becomes a ratio of

$$\begin{array}{l}\text{pr}(\mathbf{g}\mid \mathbf{b}(\mathit{\theta }),H_0)\text{pr}(\mathit{\theta })=\text{pr}(\mathbf{g}\mid \mathbf{b}(\mathit{\theta }),H_0)\text{pr}(N)\text{pr}(\{{\mathbf{c}}_n\})\\ =\left\{\prod _{m=1}^{M}exp[-b_m(\mathit{\theta })]\frac{b_m{(\mathit{\theta })}^{g_m}}{g_m!}\right\}\\ \times \left[exp(-\overline{N})\frac{{\overline{N}}^N}{N!}\right]\left(\frac{N!}{{\text{FOV}}^N}\right)\end{array}$$ (25)

evaluated at θ̃ and θ⁽ⁱ⁾, respectively. The bracketed terms in Eq. (25) are pr(g|b(θ), H₀), pr(N), and pr({c_n}), respectively. The final N! in Eq. (25) arises because pr({c_n}) is invariant under permutations of {c_n} (see Appendix A). We are exploiting our lumpy object model in which we know that the probability of a set of N centers {c_n} is given by the probability that we choose N lumps (Poisson with mean N̄) times the probability that we observed the centers {c_n} (uniform within the FOV and invariant under permutations).

In practice it is difficult to design a proposal density q(θ̃|θ⁽ⁱ⁾) that is symmetric because the number of elements in θ̃ and θ⁽ⁱ⁾ can be different. Furthermore, we cannot forgo symmetry, and we choose to compute the ratio of the proposal densities because it is difficult to design a practical, yet easily computable, density of an unordered set conditioned on another unordered set. We avoid both of these difficulties by designing a proposal density on a matrix Φ, i.e., q(Φ̃ |Φ⁽ⁱ⁾). The matrix Φ is designed such that we can write θ as a function of Φ, i.e., θ (Φ). The matrix Φ has the advantage that the number and position of elements is fixed. It has the disadvantage that not every possible θ can be represented as θ (Φ). However, the states b not achievable with Φ are those for which pr(b|g, H₀) is zero for all practical purposes (i.e., beyond computer precision).

This matrix Φ is composed of a binary column vector α of dimension N′ followed by a list of centers of dimension N′ × 2 for two-dimensional lumpy objects and N′ × 3 for three-dimensional lumpy objects, i.e.,

$$\mathrm{\Phi }=\left[\begin{array}{cc}{\alpha }_1& {\mathbf{c}}_1\\ {\alpha }_2& {\mathbf{c}}_2\\ ⋮& ⋮\\ {\alpha }_{N^{\prime }}& {\mathbf{c}}_{N^{\prime }}\end{array}\right],$$ (26)

where the centers c_n are row vectors. Here we are using an unusual notation by inserting multiple elements into the matrix Φ using the row vectors c_n. However, this keeps our definition of Φ general for two- or three-dimensional lumpy objects. The mapping function θ (Φ) is defined as

$$\mathit{\theta }(\mathrm{\Phi })=\{{\mathbf{c}}_n:{\alpha }_n=1\}.$$ (27)

The proposal density q(Φ̃|Φ⁽ⁱ⁾) is sampled by first flipping (i.e., a 1 to a 0 or a 0 to a 1) each of the binary numbers α_n with probability η. Then we randomly choose one center that is turned on (i.e., α_n = 1) in both Φ̃ and Φ⁽ⁱ⁾ and randomly shift the position of this center by sampling an isotropic Gaussian distribution with fixed width σ. In Appendix B we write down an expression of the density q(Φ̃|Φ⁽ⁱ⁾) and show that it is symmetric. The flipping of the α_n allows us to add or remove lumps, and the shifting of one of the centers allows us to make a small change to a given lump. We are limited here to N′ lumps, but we choose N′ to be much larger than the number of lumps N that went into the generation of the image g. The probability that we will ever accept N′ lumps is beyond the floating-point range of computers. Finally, we have a symmetric proposal density and are left only to compute the ratio pr(θ (Φ̃)|g, H₀)/pr(θ (Φ⁽ⁱ⁾)|g, H₀), the numerator and denominator of which are shown in Eq. (25). Note that there are typically only a couple of different lumps between b(θ (Φ⁽ⁱ⁺¹⁾)) and b(θ(Φ⁽ⁱ⁾)). Thus to generate the new background b⁽ⁱ⁺¹⁾ [Eq. 9] from the old background b⁽ⁱ⁾, we need only to replace the few lumps that have changed, which dramatically speeds up the simulation procedure.

4. SIMULATIONS

The ultimate goal of our work is to compare imaging systems based on ideal-observer performance. As an example, we simulate parallel-hole collimator imaging systems for nuclear medicine. We vary the parameters of the collimator, e.g., bore length and height, to achieve different noise levels and resolution in the generated images. More specifically, we model the imaging system transfer functions h_m(·) as Gaussian centered on the mth pixel. Each imaging system has different widths and heights for the h_m(·) functions, corresponding to different collimator parameters. By choosing Gaussian transfer functions and the lumpy object model with Gaussian lumps, we are able to compute a closed-form solution for the background, using Eq. (9).

There are numerous practical issues in Markov-chain Monte Carlo that need to be addressed. For example, if Φ⁽⁰⁾ is a completely random realization, then one must “burn in” the chain for a number of iterations to remove bias due to the initial samples. Furthermore, we must choose an N′, a flip probability η, and a shift width σ for the random shift. We are aided in this problem because we generate the image g from the lumpy model and therefore know the number of lumps N* and centers $\left\{{\mathbf{c}}_n^{\ast }\right\}$ that went into making the image. Thus the first N* entries into Φ⁽⁰⁾ are randomly shifted versions of $\left\{{\mathbf{c}}_n^{\ast }\right\}$, and they are all turned on, i.e., α_n = 1 for n = 1,…, N. We chose N′ to be 100 times bigger than N*, with the remaining 100N* − N* centers in Φ⁽⁰⁾ being uniformly distributed within the field of view. These remaining centers are all initially turned off, i.e., α_n = 0 for n = (N*+ 1),…, N′. The flip probability η was chosen to be 0.04/N′, which makes flips a rare occurrence. The Gaussian shift density was chosen to have a width of s/10, where s is the width of the lump function L(·) [Eq. (8)]. Because our initial Φ⁽⁰⁾ is chosen with knowledge of the centers that went into the creating of g, our burn-in times are relatively short (500 iterations). Finally, we iterated the chain for a total of 150,000 iterations, computing Λ_BKE(·) at each iteration.

The parameter N′ was chosen because the probability that the chain would ever accept ten times more lumps than went into the creation of the image is effectively zero. The parameters η and σ were determined empirically by studying the rejection rate of the chain. For example, if 99 out of 100 proposals are rejected on average, then the chain is not very efficient. Conversely, if the chain accepts all the proposed solutions, then it is likely that insignificant changes are occurring. This insignificance might be caused by a small σ. The burn-in of 500 iterations was determined by visual examination of Λ_BKE(·) as the chain progressed. It was clear that after a couple of hundred iterations, the chain had converged to the stationary density. If we were to choose Φ⁽⁰⁾ at random we would need to lengthen the burn-in time for our chain because it would take longer to arrive at the peak of the stationary density. Finally, the total number of iterations (150,000) was chosen because it produced estimates of Λ(g) with little variation.

We simulated three different imaging systems (A, B, and C) with different widths and heights for the sensitivity functions given in Table 1. The images acquired by all three imaging systems are 64 × 64, i.e., g is a 64² × 1 vector. For each imaging system we generated 50 signal-absent images and 50 signal-present images. We used a lumpy object model with N̄= 25, a = 1, and s = 7 [see Eq. (8)]. Example images of the same lumpy object from our three different imaging systems are shown in Fig. 2. For the signal we used a Gaussian with width 3 and height 0.1 located in the center of the object. We found that these signals were difficult to detect visually, although a psychophysical experiment was not performed. They are, however, often detected by the ideal observer.

Table 1.

Characteristics of the Three Imaging Systems

Imaging System	hm(·) Width	hm(·) Height	AUC
A	0.5	40	0.92
B	2.5	100	0.88
C	5	200	0.67

Fig. 2.

Image realizations of the same lumpy object for (a) imaging system A, (b) imaging system B, and (c) imaging system C. There is no signal in these images.

As described in Subsection 3.A, we generated a Markov chain in order to sample from pr(θ |g, H₀) in Eq. (20) to approximate the ideal observer by using Eq. (22). For each of the 300 images generated (100 for each imaging system), we generated 150,000 iterations of the Markov chain. In Fig. 3 we show Λ_BKE(g|b(θ(Φ⁽ⁱ⁾))) evaluated at the iterations of an example Markov chain. The image g used to generate this plot was without signal and was acquired from imaging system B. In all of our calculations we ignore the first 500 iterations of the chain to allow for burn-in, and we average the Λ_BKE(·) of the remaining 149,500 iterations.

Fig. 3.

Λ_BKE versus iteration number. The iteration numbers here are the states of the Markov chain described in Subsection 3.A.

5. RESULTS

We calculated the ideal observer for 50 signal-absent and 50 signal-present images for three different imaging systems. Histograms of the log-likelihood ratio are given in Fig. 4. Imaging systems A and B seem to have substantial separation between the signal-absent and the signal-present data [Figs. 4(a) and (b)]. There is a great deal of overlap between the two histograms for imaging system C [Fig. 4(c)]. Larger separation indicates that the ideal observer can more easily detect the signal. We used ROC analysis to quantify the performance of the ideal observer for these three imaging systems. Figure 5 shows ROC curves for the three imaging systems. These ROC curves were fitted by using LABROC4 software.¹⁷ The AUCs for imaging systems A, B, and C are 0.92, 0.88, and 0.67, respectively. These values provide a means of ranking imaging systems A, B, and C for the aforementioned detection task.

Fig. 4.

Log-likelihood histograms for (a) imaging system A, (b) imaging system B, and (c) imaging system C.

Fig. 5.

ROC curve comparison of the ideal-observer performance for the three imaging systems. The AUC values for imaging systems A, B, and C are 0.92, 0.88, and 0.67, respectively.

Because our method returns estimates of likelihood ratios, we reran the above experiment five times using the same 100 images from imaging system A. The five different runs used different random seeds, and thus the chains progressed differently. This experiment checks the interobserver variability that arises from the stochastic nature of Markov-chain Monte Carlo. That is, we wish to isolate the component of the variability that is due to the technique. We found that the variability in AUC as measured by the standard deviation of the five different AUC values was 0.006. This number could be reduced by running the chain for more than 150,000 iterations. This number does not reveal any information about the variability that will arise if different images (or more images) from system A are used. Similar runs were performed for imaging systems B and C with similar results.

6. CONCLUSIONS

We have presented a method for comparing imaging hardware based on the performance of the ideal observer on a signal-known-exactly detection task. The random backgrounds used in our simulations were generated by using the lumpy object model and simulated imaging systems. We regard this work as a first step toward calculations that would involve randomly varying signals, more realistic imaging-system models, and more complicated object models. Therefore the results for the particular imaging systems investigated here should be viewed as proof of concept rather than the final word on the trade-off between noise and resolution in collimator design.

APPENDIX A: REPLACING b WITH θ

We would like to show the equivalence of Eqs. (13) and (20) for the backgrounds generated with the lumpy object model. We use a standard method of transforming density functions given by

$$\text{pr}(\mathbf{b}\mid \mathbf{g},H_0)=\int \text{d}\mathit{\theta }{\delta }^M(\mathbf{b}-\mathbf{b}(\mathit{\theta }))\text{pr}(\mathit{\theta }\mid \mathbf{g},H_0),$$ (A1)

where M is the dimension of b. In general this integral is difficult because knowledge of the roots of b = b(θ) is required. However, in our problem, pr(b|g, H₀) is within an integrand, allowing us to compute the following:

$$\mathrm{\Lambda }(\mathbf{g})=\int \text{d}\mathbf{b}{\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b})\text{pr}(\mathbf{b}\mid \mathbf{g},H_0)$$ (A2)

$$\begin{array}{l}=\int \text{d}\mathbf{b}\int \text{d}\mathit{\theta }{\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b})\\ \times {\delta }^M(\mathbf{b}-\mathbf{b}(\mathit{\theta }))\text{pr}(\mathit{\theta }\mid \mathbf{g},H_0)\end{array}$$ (A3)

$$=\int \text{d}\mathit{\theta }{\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b}(\mathit{\theta }))\text{pr}(\mathit{\theta }\mid \mathbf{g},H_0),$$ (A4)

This integral expression is shorthand for the following:

$$\begin{array}{l}\sum _N\text{pr}(N\mid \mathbf{g},H_0){\int }_{S_N}\text{d}{\mathbf{c}}_1\text{d}{\mathbf{c}}_2\dots \text{d}{\mathbf{c}}_{\mathbf{N}}\\ \times {\mathrm{\Lambda }}_{\text{BKE}}(\mathbf{g}\mid \mathbf{b}(\left\{{\mathbf{c}}_n\right\}))\text{pr}(\{{\mathbf{c}}_n\}\mid \mathbf{g},H_0).\end{array}$$ (A5)

The above integral is over a region S_N contained within R^2N for two-dimensional lumpy objects. This region is defined by the requirement that all centers be within the FOV and must satisfy the inequalities c₁<c₂< … <c_N, where < symbolizes a lexicographical ordering¹⁸ of the vectors c_n. This ensures that there is only one point in S_N corresponding to the unordered set {c_n}. Also note that the volume of this set S_N is given by FOV^N/N!, which explains the expression pr({c_n}) in Eq. (25).

APPENDIX B: SYMMETRY OF THE PROPOSAL DENSITY

The proposal density is given by

$$\begin{array}{l}q(\stackrel{\sim }{\mathrm{\Phi }}\mid \mathrm{\Phi })={\eta }^f{(1-\eta )}^{N^{\prime }-f}\sum _{j:{\stackrel{\sim }{\alpha }}_j={\alpha }_j=1}\frac{1}{C}\delta ({\mathbf{c}}_{-j}-{\stackrel{\sim }{\mathbf{c}}}_{-j})\\ \times G({\stackrel{\sim }{\mathbf{c}}}_j-{\mathbf{c}}_j),\end{array}$$ (B1)

where f is the number of α’s that are flipped from Φ to Φ̃ given by

$$f=\sum _{i=1}^{N^{\prime }}\mid {\alpha }_i-{\stackrel{\sim }{\alpha }}_i\mid$$ (B2)

The function G(·) is a properly normalized, symmetric Gaussian density with variance σ². The variable C is the number of terms in the sum in Eq. (B1), given by

$$C={\mathit{\alpha }}^{†}\stackrel{\sim }{\mathit{\alpha }}.$$ (B3)

In the case in which there are no centers in common between Φ and Φ̃, the sum in Eq. (B1) is zero; i.e., its probability density is zero. For two-dimensional (respectively three-dimensional), lumpy objects, δ (c_−j − c̃_−j) is a 2C − 2 (respectively 3C − 3)-dimensional Dirac delta function. The vector c_−j is a concatenation of all the center vectors c_i satisfying α̃_i = α_i = 1, except c_j itself. The vector c̃_−j is defined similarly.

The term to the left of the summation in Eq. (B1) gives the probability of making the flips in the α_i to get from Φ to Φ̃. The summation term is a mixture of densities that represents the shift of one of the centers in common between Φ and Φ̃. To sample from such a mixture, we choose one of the terms with probability 1/C and then sample from it.

The symmetry of q(Φ̃|Φ) follows from the symmetry of the f [Eq. (B2)], C [Eq. (B3)], δ (c_−j − c̃_−j), and G(c̃_j − c_j) regarded as functions of Φ and Φ̃.