Evaluation of hardware in a small-animal SPECT system using reconstructed images

Abstract

Evaluation of imaging hardware represents a vital component of system design. In small-animal SPECT imaging, this evaluation has become increasingly difficult with the emergence of multi-pinhole apertures and adaptive, or patient-specific, imaging. This paper will describe two methods for hardware evaluation using reconstructed images. The first method is a rapid technique incorporating a system-specific non-linear, three-dimensional point response. This point response is easily computed and offers qualitative insight into an aperture’s resolution and artifact characteristics. The second method is an objective assessment of signal detection in lumpy backgrounds using the channelized Hotelling observer (CHO) with 3D Laguerre-Gauss and difference-of-Gaussian channels to calculate area under the receiver-operating characteristic curve (AUC). Previous work presented at this meeting described a unique, small-animal SPECT system (M³R) capable of operating under a myriad of hardware configurations and ideally suited for image quality studies. Measured system matrices were collected for several hardware configurations of M³R. The data used to implement these two methods was then generated by taking simulated objects through the measured system matrices. The results of these two methods comprise a combination of qualitative and quantitative analysis that is well-suited for hardware assessment.

1. INTRODUCTION

Small-animal imaging is a burgeoning field as evidenced by the range and breadth of recent imaging studies in the area^1,2 and a spate of imaging systems.^3–8 Each system has unique characteristics and will perform differently for different tasks. Therefore, the question of optimal system design is difficult. Rigorous evaluation of an imaging system can be achieved through objective, task-based analysis. However, there do exist other, more qualitative evaluation methods that allow insight into some important general features of a given imaging system.

We have designed and built a small-animal imaging system, previously introduced in this meeting,⁹ known as the multi-module, multi-resolution system (M³R) that is well-suited for hardware assessment and optimization. M³R allows for easy modification of parameters such as magnification, pinhole diameter, pinhole number, and degree of multiplexing. This system allows us to evaluate multiple hardware configurations (pinhole apertures in various combinations) simultaneously. In order to understand and improve upon these different hardware configurations, we required methods by which to evaluate the system. Because M³R is a tomographic imaging system and because of the emphasis placed upon 3D images both in clinical and pre-clinical application, we decided that these evaluation methods should be implemented on reconstructed images.

Two methods for assessing hardware configuration are described in this paper. The first method is a somewhat familiar technique¹⁰ that makes use of a system’s non-linear point response to gain insight into the resolution and artifact characteristics of a system with a particular pinhole geometry. This differential point response function (DPRF) is calculated by taking the difference between two reconstructions, one containing only a background and the other containing that same background and an additional weak point object. The second method is an objective assessment of signal detection in lumpy backgrounds using the channelized Hotelling observer (CHO) with 3D Laguerre-Gauss (LG) and difference-of-Gaussian (DOG) channels to calculate area under the receiver-operating characteristic curve (AUC). These two methods have enabled successful comparison and evaluation, both qualitative and quantitative, of different hardware configurations for M³R and would be easily applicable to other systems.

The remainder of the paper is structured as follows. The next section presents methods, including a brief description of image quality, the M³R hardware configuration used in the study, and descriptions of the procedures for the two studies. The third section presents results. The final section offers discussion and conclusions.

2. METHODS

2.1. Image Quality

In general, an imaging system maps a continuous object f(r) to a discrete data vector g via the imaging equation

$$\mathit{g}=\mathcal{H}f(\mathit{r})+\mathit{n}$$ (1)

where is a continuous-to-discrete (CD) operator representing the imaging system, n is the noise, and bold symbols denote vectors. One of our goals is to objectively determine the best (e.g., optimum magnification and pinhole configuration). Note that g is not an image for tomographic systems. Generally, a reconstruction step, represented by the operator , is required to produce an image θ, where θ = (g).

To properly optimize an imaging system, one must first identify a relevant task to be performed and then determine a figure of merit by which to measure the ability of the system to perform that task.¹¹ Tasks are often separated into two types: classification and estimation. In classification tasks, the observer is categorizing an image into one of a finite number of possible outcomes. One example is where there are two possible outcomes: signal plus background (lesion present) or background alone (lesion absent). For estimation tasks, the observer is attempting to quantify some particular image parameter, such as tumor size. M³R fulfills the imaging needs for many biomedical imaging tasks of both types. In addition, M³R allows the user to tailor to optimize system performance for a specific task.

2.2. Imaging System & Configuration

M³R’s system configuration may be modified through the interchange of pinhole plates between slots machined into the system’s shielding.⁹ The system configuration used in this study consisted of three different magnifications, two different pinhole diameters, and four different pinhole numbers. The configurations are shown in Table 1. Angle, shown in the last column of the table, is defined as the full opening angle of the pinhole, resulting from a combination of the countersink and keel. Each multiple-pinhole aperture employed a different pattern. Aperture 0 had a square arrangement of pinholes with a fifth pinhole in the center (a pattern known as a quincunx). Aperture 2 employed a 3 × 3 array of pinholes and Aperture 3 had pinholes in a square pattern.

Table 1.

Example system configuration. Aperture number, magnification, number of pinholes, pinhole diameter, sensitivity (normalized to the most sensitive aperture), and full opening angle are provided

Aperture	Magnification	Number of Pinholes	Diameter	Sensitivity	Angle
0	1.60	5	1 mm	0.59	85.4°
1	3.33	1	1 mm	0.50	85.4°
2	2.47	9	0.25 mm	0.10	60.0°
3	3.33	4	1 mm	1.00	85.4°

Data were generated through the use of a measured system matrix. Projection data are produced according to the imaging equation given by

$$\mathit{g}=\mathit{Hf}+\mathit{n},$$ (2)

where g is an M × 1 vector of projection data, f represents the N × 1 discretized object space, n is noise, and H is the M × N system matrix, responsible for mapping an object to an image.

Our group chooses to directly measure the system response, following techniques developed by Chen, et al.¹² This procedure involves moving a point source through the object space on a Cartesian grid inscribed in a cylinder and recording the system response at each location. One advantage to this method is that issues such as position-dependent detector variations, system misalignment, inaccuracies in pinhole construction, and complicated system geometry are built into the procedure. The system matrix used in this study consists of 41,888 columns, where each column corresponds to the system response to a particular voxel. Each voxel is a cube 0.9mm on a side. The object space cylinder has a diameter of 30.6mm (corresponding to 34 voxels) and a height of 39.6mm (corresponding to 44 voxels). Once in possession of the system matrix H it becomes possible to generate projection data through the mapping of simulated objects. Simulated objects are generated on the grid of points described by the system matrix (or on a grid more finely sampled through interpolation). The measured system matrix represents the system response over only one angle. To reconstruct, it is necessary to calculate system response over a range of angle. The measured system matrix is rotated (using nearest-neighbor methods) to generate the full system matrix, encompassing all necessary angles. Discretization error occurs as a result of simulating objects in this way, but in this study we have chosen to accept this penalty as a trade-off for the advantages gained through use of a measured system matrix.

Because of the Poisson nature of the physical processes governing data collection, noise can be minimized in the measured system matrix by using long exposure times. This low-noise system matrix means that simulated objects taken through the measured system matrix better represent noise-free projection data given by

$$\overline{\mathit{g}}=\mathit{Hf}.$$ (3)

Poisson noise can be added to the simulated projection data by sampling a Poisson random vector with mean ḡ

2.3. Differential Point-Response Function (DPRF)

Quantitative methods of the fashion described in the preceding subsection are the ideal tools for hardware assessment. However, these studies are typically time-consuming and difficult to implement. Properly chosen qualitative assessment methods are capable of providing some insight into the physical characteristics of an imaging system. These tools are not capable of replacing rigorous, objective studies, but can be useful complements to these studies by providing intuitive reinforcement and aiding our understanding of why certain aperture configurations perform best for certain tasks.

The differential point response function is a rapid, qualitative 3D analysis tool that lends insight into a pinhole plate’s artifact and resolution characteristics. The DPRF is formed by taking the difference of two reconstructions. This procedure has been used before and is described in Dr. Don Wilson’s thesis.¹⁰ The DPRF is generated for an aperture in M³R as follows. A simulated random textured background (e.g. lumpy background¹³) is taken through the measured system matrix for a given aperture. Two copies of the noise-free projection data are kept. One set is reconstructed without further alteration. The other set has a small number of signal counts added to each projection as shown in Figure 1. The signal counts represent activity emanating from a single voxel in object space. This second set of signal-present data is also reconstructed. The difference of the two reconstructions is taken to produce the DPRF. Because the signal is so weak, the DPRF is best viewed as the slice of the reconstruction containing the signal or as a cross-section through the signal in that slice.

Figure 1.

(a) A noise-free projection of a simulated lumpy background taken through a measured system matrix. (b) A noise-free projection of a simulated point signal. (c) The sum of the projections in (a) and (b). Note that a weakened version of the signal is shown (c) and a weaker version still is used in the actual DPRF calculation.

Eight hardware configurations of M³R were chosen for this study. Because we are generating objects in simulation and artificially rotate the measured system response, it is possible to simulate M³R hardware configurations as if any of the four apertures (for which we have measured system response data) are sitting in any of the four M³R camera slots. Four of the hardware configurations used in the study employed the same aperture for each camera while the other four used some mixture of apertures. A hardware configuration will be labeled according to the apertures used. For example, the hardware configuration employing all four apertures will be labeled 0123 while a configuration using only Aperture 1 will be labeled 1111. Aperture order should be inconsequential because the angular samples were collected over 360°.

2.4. Observer Study

A signal-detect ion study was chosen to objectively assess the same eight hardware configurations employed in the DPRF study. Five hundred lumpy background objects were generated.^13,14 A lumpy background consists of a number (chosen from a Poisson distribution) of 3D Gaussians randomly placed in the object space. The lumpy objects used in this study had mean number of 63 lumps where each lump had a 2σ width of ~ 3.3 voxels. The lump amplitude was chosen to produce an average of 1,000,000 counts per reconstruction for aperture combination 1111.

Projection data were generated for each lumpy background for each individual aperture over 36 angles. Each aperture in a given configuration contributed its full complement of 36 projection angles, resulting in a total of 144 projections used per reconstruction. Reconstructions were generated using OSEM with five iterations and six subsets.

Area under the ROC curve (AUC) was used as the figure of merit for this study. The channelized Hotelling observer was used to calculate AUC. Standard deviation was calculated for the AUC values using the One-Shot method developed for the MRMC problem.¹⁵

Channels are used to ease computational difficulties which arise when applying the Hotelling observer. The Hotelling observer requires estimation and inversion of the covariance on the data. In this study, the data consists of reconstructed images and the covariance matrix of interest is 41888 × 41888 elements. For a covariance matrix of dimension M × M, at least M + 1 images are required to obtain an invertible estimate. In practice, it has been shown that many times this minimum sample size may be required to insure an acceptable amount of error in estimating the similarity between two distributions.¹⁶ Also, once this covariance matrix has been generated, it must be inverted. While methods exist for computing this inverse, they are computationally burdensome, time-consuming, and sometimes inaccurate. The use of a channel matrix of size M × P where P is typically only 4 – 20, drastically decreases dimensionality while preserving the information in the reconstruction.

Test statistics under each hypothesis (signal-absent, signal-present) were formed according to

$$t_i={\mathit{s}}^{†}{\mathit{T}}^{†}{\mathit{K}}_{\mathit{T}}^{-1}\mathit{T}{\widehat{\mathit{f}}}_i,$$ (4)

where t_i is a test statistic for the i^th hypothesis, s is the known signal, T is the M × P channel matrix, K_T is the covariance matrix through the channels, and f̂_i is a reconstruction belonging to the i^th class.

In the case of channelized Hotelling observer, the covariance on the channel outputs (Equation 4) may be written as

$${\mathit{K}}_{\mathit{T}}=(\mathit{TW}){(\mathit{TW})}^{†},$$ (5)

where W is a N_s × M matrix composed of the N_s sample images, scaled by $\sqrt{N_s}$ and having the mean sample image ḡ subtracted.¹¹ The matrix K_T will be P × P where P – the number of channels – is typically less than twenty and often as few as five or six, making the covariance matrix easily invertible. Also, the small dimension of K_T eases requirements on sample number and noise characteristics necessary to estimate the matrix.

The signal-absent and signal-present reconstructions were each split into two sets of 250. 125 reconstructions from each of the first sets were selected for testing. 125 reconstructions from each of the second sets were selected to generate the sample signal ŝ shown in Equation 4. The second set of signal-absent reconstructions was also used to generate W.

2.4.1. Channel Types

Three well-known channel types – Laguerre-Gauss (LG), sparse difference-of-Gaussian (SDOG), dense difference-of-Gaussian (DDOG) – were employed in the channelized Hotelling observer.¹⁷ Laguerre-Gauss channels have been used in previous signal-detection tasks with radially symmetric, smooth, signals because they form a basis on the space of rotationally symmetric square integrable functions.¹⁸ Difference-of-Gaussian channels have been previously used in detection studies and shown to correlate well with human-observer performance (following noise regularization).¹⁹

The parameters for each channel type used in this signal-detect ion study are given in Table 2. The DOG parameters were chosen based on values used in work done by Abbey.¹⁹ A difference-of-Gaussian channel in frequency space is given by

Table 2.

Channel parameters used in the observer study with reconstructed images. Three channel types – Laguerre-Gauss (LG), sparse difference-of-Gaussian (SDOG), and dense difference-of-Gaussians (DDOG) – were used in the study

Channel Type	2σ width	α	Q	Number
LG	~ 5.6 voxels	−	−	6
SDOG	0.030 voxels−1	2	2	3
DDOG	0.006 voxels−1	1.4	1.67	10

$${\text{DOG}}_j(\rho )=exp\left[-\frac{1}{2}{\left(\frac{\rho }{Q{\sigma }_j}\right)}^2\right]-exp\left[-\frac{1}{2}{\left(\frac{\rho }{{\sigma }_j}\right)}^2\right]$$ (6)

where DOG_j is the jth. channel, ρ is spatial frequency, Q is a parameter determining channel bandwidth, and σ_j = σα^j is the channel width. The channel width depends on the initial channel width σ and the parameter α which must be greater than 1.

Laguerre-Gauss channel width was chosen by measuring AUC over a range of channel widths using three different aperture combinations (1111, 3333, 0303). Results of those calculations are given in Figure 4.

Figure 4.

AUC as a function of channel width. Six channels were used to generate this data for example aperture combinations 1111, 3333, and 0303. The maximum performance for all three combinations occurs at a width of 7, corresponding to 2σ width of ~ 5.6 voxels.

3. RESULTS

3.1. DPRF

The slices and cross-sections through the DPRFs shown in Figures 2 and 3 provide insight into each aperture combination’s resolution and artifact characteristics. The low-magnification and relatively high-sensitivity of Aperture 0000 are evident in its fairly broad, but artifact-free DPRF. Aperture combinations 1111 and 3333 have similar, fairly narrow peaks reflecting their high magnification but both have poorer noise characteristics than combination 0000. The noise outside the peak of combination 1111 probably results because it has only a single pinhole while the peaks at the edge of the DPRF for combination 3333 result from the redundancy in the pinhole array. The absence of those peaks in the DPRF for combination 0000 is probably because of its lower magnification but may be due to its center pinhole. Aperture combination 2222 exhibits the narrowest peak, owing to its small pinholes and relatively high magnification but also possess the poorest artifact characteristics. The two rings surrounding the central peak are due to the highly redundant pinhole pattern. Other artifacts may be due to higher photon penetration through the aperture.

Figure 2.

Center slices of the differential-point response function for aperture combinations (a) 0000, (b) 1111, (c) 2222, (d) 3333, (e) 0123, (f) 0101, (g) 0303, and (h) 1313.

Figure 3.

Cross-sections through the center slices of the differential-point response function for aperture combinations (a) 0000, (b) 1111, (c) 2222, (d) 3333, (e) 0123, (f) 0101, (g) 0303, and (h) 1313.

The DPRFs generated from combinations of different apertures tend to exhibit (qualitatively) traits corresponding to their individual aperture components. It appears, qualitatively from the shape of the DPRF for combination 1111, that Aperture 1 tends to dominate in those combinations where it is present. However, the inclusion of Aperture 0, Aperture 3, or all other apertures (as in combinations 0101, 0303, and 0123, respectively) appears to reduce the artifacts outside the primary peak. However, the penalty for the inclusion of these apertures appears to be a decrease in the height of the primary peak as evidenced by Figure 3. The DPRF for combination 0303 looks similar to that of combination 3333 but appears to have slightly weaker artifacts and a slightly broader peak, owing to the inclusion of Aperture 0. Again, to reiterate, this highly subjective analysis can not serve as the sole method for hardware evaluation but does provide quick insight into the resolution and reconstruction artifacts of a given aperture or combination of apertures.

3.2. Observer Study

AUC results from the observer study are shown in Table 3. The AUC data are shown by aperture combination in order of descending performance for each channel type (the order was the same for each channel type). The aperture combination consisting entirely of Aperture 1 (the single-pinhole, high-magnification aperture) exhibits the peak performance. Those aperture combinations containing Aperture 1 represent the next three top performers, followed by combinations containing only Apertures 0, 2, and 3. Performance was not shown to change significantly for any given aperture combination across channel type.

Table 3.

AUC for a variety of aperture combinations and three channel types – Laguerre-Gauss, sparse difference-of-Gaussians, and dense difference-of-Gaussians. The data have been listed in order of descending AUC. The order was identical for all three channel types

AUC for different apertures and channel types.
Apertures	LG	SDOG	DDOG
1111	0.913 ±0.022	0.893 ± 0.024	0.885 ± 0.024
0101	0.887 ±0.024	0.871 ±0.025	0.850 ± 0.026
1313	0.873 ± 0.025	0.853 ± 0.026	0.839 ± 0.027
0123	0.859 ± 0.026	0.838 ± 0.027	0.818 ±0.028
3333	0.821 ±0.028	0.789 ±0.030	0.799 ±0.029
0303	0.787 ±0.030	0.772 ±0.030	0.744 ±0.031
2222	0.767 ±0.030	0.705 ±0.033	0.742 ±0.032
0000	0.712 ±0.033	0.744 ±0.031	0.680 ± 0.034

The channel width used for the LG channels was chosen by computing AUC for several channel widths and three different aperture combinations. A channel width of 7, corresponding to 2σ = 5.6 voxels, was shown to return the peak AUC for the three combinations. This channel width was then used for all LG AUC computations.

The maximum DPRF value and AUC value (with LG channels) for each aperture combination were plotted. For this particular task, there appears to be some correlation between maximum DPRF value and AUC. The DPRF resolution for each camera combination was also plotted against AUC. There does not appear to be a strong correlation between aperture combination resolution and AUC for this particular task.

4. DISCUSSION & CONCLUSIONS

The DPRF has been used to qualitatively explore the resolution and reconstruction artifacts of different aperture combinations in M³R. The information provided is intuitively pleasing but may not be used as the sole evaluation for a given hardware configuration as it is highly subjective. The DPRF is attractive because it is relatively easy and fast to compute. Although difficult to discern on the images shown in this paper, the DPRF images appear to depend, especially in the peak shape and noise outside the peak, on the particular lumpy background used. Future calculations of the DPRF may wish to employ an average DPRF based on several sample lumpy backgrounds.

A possible correlation was shown to exist between the maximum DPRF value and AUC value for different aperture combinations for this particular task. Aperture 1111 and combinations including Aperture 1 (a high-magnification, single-pinhole aperture) were shown to have the largest peak DPRF value (and highest AUC values). Aperture 3333, despite having equal magnification and higher sensitivity, had a lower peak DPRF value, possibly resulting from reconstruction artifacts. These reconstruction artifacts and similar artifacts seen when using Aperture 2 probably result from the use of redundant pinhole arrays.²⁰ Further, the presence of these artifacts may be associated with the rings seen in the DPRF images for these apertures.

As mentioned above, aperture combinations containing the single-pinhole, high-magnification aperture returned the highest AUC values in the observer study. Previous studies, including work in our group with the ideal observer,²¹ support this result in that high-magnification and relatively large pinhole diameters are often associated with good signal-detection performance. Thus, the performance of the multiplexed, high-magnification Aperture 3 comes as something of a disappointment. It is not known how much the reconstruction artifacts mentioned in the previous paragraph contribute to decreased observer performance. The reduction of such artifacts in multiplexed, high-magnification apertures (through clever choice of aperture arrangement) is something which we are continuing to explore.

A second plot was generated to examine DPRF resolution and AUC as a function of aperture combination. Resolution was not shown to correlate well with observer performance for this task. The desire to improve resolution often drives aperture design. This result indicates that the drive for higher resolution should be tempered with attention to sensitivity and noise tailoring in aperture design.

The idea of using the qualitative DPRF as a predictor of aperture performance in quantitative observer studies raises several questions and must be fully explored before any conclusions may be drawn. Does the maximum DPRF always correlate with observer performance in signal-detectiont tasks? Does this correlation depend on signal size or signal strength? Do any aspects of the DPRF correlate with observer performance in other tasks such as signal activity estimation or position estimation?

Two methods have been successfully used to assess hardware configurations of the M³R small-animal SPECT system using reconstructed images. Each method produces valuable information for evaluating hardware performance and raises several questions pertaining to aperture design. The non-linear point response provides intuitively pleasing, qualitative information pertaining to resolution and artifact characteristics for a given aperture. The AUC methodology returns task-specific, objective comparison of the different hardware configurations. These techniques are generally applicable to a variety of tasks and imaging systems and should be valuable in future work.

Figure 5.

(a) Maximum DPRF value and AUC for all aperture combinations. (b) DPRF full width at half maximum value in voxels and AUC for all aperture combinations.