Foveated Model Observers to predict human performance in 3D images

Abstract

We evaluate 3D search requires model observers that take into account the peripheral human visual processing (foveated models) to predict human observer performance. We show that two different 3D tasks, free search and location-known detection, influence the relative human visual detectability of two signals of different sizes in synthetic backgrounds mimicking the noise found in 3D digital breast tomosynthesis. One of the signals resembled a microcalcification (a small and bright sphere), while the other one was designed to look like a mass (a larger Gaussian blob). We evaluated current standard models observers (Hotelling; Channelized Hotelling; non-prewhitening matched filter with eye filter, NPWE; and non-prewhitening matched filter model, NPW) and showed that they incorrectly predict the relative detectability of the two signals in 3D search. We propose a new model observer (3D Foveated Channelized Hotelling Observer) that incorporates the properties of the visual system over a large visual field (fovea and periphery). We show that the foveated model observer can accurately predict the rank order of detectability of the signals in 3D images for each task. Together, these results motivate the use of a new generation of foveated model observers for predicting image quality for search tasks in 3D imaging modalities such as digital breast tomosynthesis or computed tomography.

1. INTRODUCTION

There is a long history of utilizing mathematical models to predict human visual performance in detection tasks in image noise.^1–6 The models have progressed to include increasingly more realistic components of clinical tasks: statistical correlations in the backgrounds,^7–11 anatomic backgrounds,^12,13 components of the human visual system,^14–16 internal noise, signal variability^17–19 and 3-D components.^20–23 Researchers have also recognized the importance of the search component which, unlike the traditional simpler tasks, entails large uncertainty about the spatial location of the signal. In recent years, a number of search models have been proposed for medical imaging including the scanning observer^24,25 and a 2-stage model with an initial segmentation to identify candidate locations followed by processing with standard model observers templates.^26,27

Here, we consider visual search in 3D image modalities such as Computed Tomography (CT) or Digital Breast Tomosynthesis (DBT) which are standard for many types of lesion diagnosis and treatment assessment. A common way to display 3D volumes in a 2D screen is in a stack of 2D images. The user scrolls up and down the volume, showing one 2D slice at each time. The movement can be manually controlled by a mouse or can be played automatically at a certain speed (cine-mode).

We suggest that reading 3D images creates additional challenges for visual search.^28–30 The large volume of image data does not allow the observer to scrutinize with their high resolution fovea all regions of the 3D image. As a consequence, human search with 3D images relies heavily on visual processing with the visual periphery, which processes visual information at a coarser spatial scales. We argue that the role of peripheral visual processing during search requires the development of a new type of model observer, a foveated model observer, that takes into account the properties of the visual processing across the entire visual field.

In this paper, we compare human performance in a location-known detection task and a free search task of two different signals in 3D images and show a dissociation in their detectability across tasks. We show that classic model observers, designed only to model vision at the fovea, predict the detection task but fail to predict the detectability of signals during search. We propose a new foveated model observer, based on the Channelized Hotelling Observer, which channels vary with across the visual field. We show that this model can correctly predict the dissociation in performance between location-known detection and free search tasks in 3D images.

2. METHODS

Two different Yes/No tasks were designed for this experiment: search and detection. For the search task, observers were presented with the 3D images and were instructed to look for the signal with no time limit. In the detection task, highly visible cues indicated the possible spatial location of the signal in the 3D volume. A highly visible copy of the signal placed above the test images indicated to the observer which signal might be present for that trial (microcalcification-like or mass-like signal). The 3D volume was displayed as a stack of 2D images, which the observers were able to scroll through using the mouse in a similar manner to radiological practice. An eye-tracker was also used to track the gaze position of the observers at all times, this information was used in combination with the foveated model observer to generate the final predicted performance.

2.1 Stimuli Generation

For the generation of the stimuli, a 3D correlated Gaussian noise field was generated using a 3D power spectrum S(f) derived from X-ray breast mammograms,³¹ specifically S(f) = 1/f^2.8. The size of the 3D volumetric image was 1024×820×100 voxels. The voxel size in the synthetic images is assumed to be 50μm³. One of two different 3D signals was embedded in the image data in 50% of the trials, placed at random spatial locations in the 3D volume. The first signal was a small and bright sphere, 6 voxels of diameter (≈ 300μm), constant contrast, which was considered a simulated microcalcification (MC). The second signal was a 3D Gaussian blob, with σ = 10 voxels (lesion diameter ≈ 4mm), which was more similar to a small mass (MS). Figure 1 shows an example of the central slice of each type of signal in the noise. The stimuli was displayed in a medical grade monitor which resolution was 1280×1024 at a distance of 75cm, the number of pixels per degree of visual angle was 45. The monitor was calibrated to have linear contrast with 0 cd/m² at gray level 0 and 111 cd/m² at gray level 255.

Figure 1.

Example of a slice of the 3D simulated microcalcification (left) and mass (right) embedded in noise samples.

2.2 Search vs Detection tasks

The human observers were asked to perform two different tasks over a 3D image: search and detection. For both tasks, the observer knew beforehand whether it was a detection or search task and what type of signal (MC or MS) had to be searched. The user could scroll up and down the stack of images thus exploring the 3D volume, a scroll bar was drawn on the right of the screen showing the slice that was currently being displayed. The observer responded about the presence of the signal using an 8-scale confidence rating. In the search task, the 1st slice was shown at the beginning of the stimulus presentation while, in the detection task, the slice corresponding to the middle slice of the signal was presented initially. Additionally, for the detection task, a 2D cross was drawn in the location in which the signal might be present, the cross disappeared when the displayed slice was too far (based on the signal radius) from the center of the signal thus serving as a cue to indicate the relevant signal slices in the third dimension. On the other hand, the search task had no cues indicating the possible location of the signal. The signal contrast was set to yield performance in a reasonable range (d′ < 2) for the search task (peak contrast = 0.5437), the signal contrast for the detection task was set to be 30% lower to avoid ceiling effects (peak contrast = 0.3821).

2.3 Standard Model Observers

Four different standard model observers were developed and expanded to the third dimension as follows. 1) The 3D NPW model observer using 3D signal as the template; 2) The 3D NPWE using the temporal filter presented by Zhang et al.³² ; 3) The 3D Hotelling observer using the statistics of the 3D noise field and the 3D signal to form the optimal detection template REF; 4) The 3D channelized Hotelling observer using the hybrid approach presented by Chen et al.³³ and further discussed by Platiša et al.²⁰

2.3.1 3D NPW Observer

The 3D non-prewhitening observer is the simplest model and it is a straightforward extension to the third dimension of the well-known 2D NPW model. The response λ is calculated as Eq. 1 shows. Where w_NPW stands for the NPW template which, in this case, is equal to the 3D signal s. g stands for the test image.

$$\begin{array}{r}{\mathbf{w}}_{\mathit{NPW}}=\mathbf{s}\\ \lambda ={\mathbf{w}}_{\mathit{NPW}}^t\mathbf{g}\end{array}$$ (1)

2.3.2 3D NPWE Observer

The 3D non-prewhitening observer with eye filter adds sensitivity with both spatial⁹ and temporal components.³⁴ The temporal component is introduced to model the effects of scrolling through the stack of 2D images, we represent this by using the average slice display time from the human observer study.

The spatial eye filter was applied to the template as

$$E\left(\rho \right)={\rho }^{\alpha }\exp \left(-\beta {\rho }^{\gamma }\right),$$ (2)

using the parameters by Zhang et al.³² α = 1.4, β = 0.013 and γ = 2.6 where $\rho =\sqrt{u^2+v^2}$ is the radial spatial frequency in cycles per degree.

For the temporal component, the transient version of the temporal sensitivity function (TSF) is computed as the difference of two components

$$h\left(t\right)=h_1\left(t\right)-h_2\left(t\right)$$ (3)

where

$$\begin{array}{l}{h}_1\left(t\right)=\mathit{\text{step}}\left(t\right){[\tau (n_1-1)!]}^{-1}{\left(\frac{t}{\tau }\right)}^{n_1-1}\exp \left(-\frac{t}{\tau }\right)\\ h_2\left(t\right)=\mathit{\text{step}}\left(t\right){[\kappa \tau (n_2-1)!]}^{-1}{\left(\frac{t}{\kappa \tau }\right)}^{n_2-1}\exp \left(-\frac{t}{\kappa \tau }\right).\end{array}$$ (4)

According to the authors, the constants were fixed to the following values: τ = 4.94, κ = 1.33, n₁ = 9, n₂ = 10. The value of t was set as the average display time per slice for all the observers, specifically 0.117 s for the mass and 0.133 s for the microcalcification.

The final response of the observer can be calculated as Eq. 5 shows. Tilde symbols refer to the frequency domain.

$$\begin{array}{r}{\mathbf{w}}_{\mathit{NPWE}}={\mathsf{fft}}^{-1}\left[{\stackrel{\sim }{\mathbf{E}}}^2\stackrel{\sim }{\mathbf{s}}\right]h{\left(t\right)}^2\\ \lambda ={\mathbf{w}}_{\mathit{NPWE}}^t\mathbf{g}\end{array}$$ (5)

2.3.3 3D Hotelling Observer

The 3D Hotelling observer was built as an extension to the traditional 2D Hotelling observer in which the covariance matrix of the noise as well as the signal is 3D. Eq. 6 shows how the template w_Hot is calculated, ${\mathbf{K}}_{\mathbf{g}}^{-1}$ stands for the 3D image covariance matrix and s is the 3D signal.

$$\begin{array}{r}{\mathbf{w}}_{\mathit{\text{Hot}}}={\mathbf{K}}_{\mathbf{g}}^{-1}\mathbf{s}\\ \lambda ={\mathbf{w}}_{\mathit{\text{Hot}}}^t\mathbf{g}\end{array}$$ (6)

2.3.4 3D Channelized Hotelling Observer

The hybrid channelized Hotelling model used here³³ computes first the channelized data for each slice as a normal 2D CHO model. The channels used were a set of Gabor filters with 8 orientations and 6 bandwidths with the highest bandwidth set to 16 cycles per degree of visual angle³⁵ and are denoted by T. The template for each slice n was calculated using the following equations:

$$\begin{array}{l}\mathrm{\Delta }{\overline{\mathbf{v}}}_n={\mathbf{T}}^t{\mathbf{s}}_n\\ {\mathbf{K}}_{{\mathbf{v}}_n}={\mathbf{T}}^t{\mathbf{K}}_{{\mathbf{g}}_{\mathrm{n}}}\mathbf{T}\\ {\mathbf{w}}_n={\mathbf{K}}_{{\mathbf{v}}_n}^{-1}\mathrm{\Delta }{\overline{\mathbf{v}}}_n\end{array}$$ (7)

Following to the calculation of all the templates w_n, a test statistic is calculated for each slice n, named here as planar template t_planar.

$$\begin{array}{r}{t}_n={\mathbf{w}}_n^t{\mathbf{v}}_n\\ {\mathbf{t}}_{\mathit{\text{planar}}}=[t_1,t_2,\dots ,t_N]\end{array}$$ (8)

Finally, the planar template t_planar is used as a one-dimensional Hotelling observer to calculate the final statistic for the 3D channelized Hotelling. Eq. 9 shows the final calculation of the template and the final scalar statistic λ. In this equation, g_planar is the result of the 2D CHO templates from Eq. 8 applied to the test image g,

$$\begin{array}{r}{\mathbf{w}}_{\mathit{CHO}}={\mathbf{K}}_{\mathit{\text{planar}}}^{-1}{\mathbf{t}}_{\mathit{\text{planar}}}\\ \lambda ={\mathbf{w}}_{\mathit{CHO}}^t{\mathbf{g}}_{\mathit{\text{planar}}}.\end{array}$$ (9)

2.4 3D Foveated Channelized Hotelling Model Observer

In addition to the classic model observers, we developed a 3D Foveated Channelized Hotelling Observer (3D FCHO). This observer is an extension to the 3D hybrid CHO presented in section 2.3.4. It adds a new layer of complexity by modeling signal detectability as a function of the distance of the signal from the observer’s fixation point in degrees of visual angle (retinal eccentricity). For a given signal and noise, standard model observers result in a figure of merit (d′, area under the ROC curve, etc) quantifying the detectability of a target. This foveated model observer results in a collection of detectabilities for different points across the visual field of the human observer.

To model the decreasing spatial resolution with increasing retinal eccentricity we modified the channels of the CHO as a function of distance from the signal to the point of fixation. At 0 degrees retinal eccentricity (the fovea) the center frequencies of the channels are those of a standard Channelized Hotelling: 16, 8, 4, 2, 1 and 0.5 cycles per degree. The center frequencies for all the Gabor channels were linearly decreased by a factor of 0.3 per degree of eccentricity, always keeping the 1 octave bandwidth. The scaling factor value was selected to match human measurements of signal detectability as a function of retinal eccentricity.^35,36 This way, the model reduces the visibility of high frequency signals, such as microcalcifications, as the eccentricity increases. Figure 2 shows the center frequency of the Gabor channels as a function of the eccentricity.^18,37,38 The final performance is calculated identically to the hybrid 3D CHO in which the only difference is the used channels that change with the eccentricity.

Figure 2.

Center frequency of the 6 Gabor channels as a function of the eccentricity for the 3D FCHO model.

3. RESULTS

A total of 3 naïve human observers participated in the study. Each observer carried out 288 trials across all four conditions (signal present, signal absent, MC present, MS present), 144 for the search task and 144 for the detection task. Figure 3 (left) shows the detectability index d′ averaged across all humans for the small microcalcification-like signal (MC) and the larger mass-like signal (MS). The detectability index d′ was calculated using the area under the ROC curve averaged across all the participants for each of the four conditions: detection and search for MC and MS.

Figure 3.

Left: Average detectability index d′ for human observers for microcalcifications (MC) and masses (MS) in the detection task and search task. Right: Detectability index d′ calculated for the standard model observers for signal contrast 1.

Results show that for the detection task the MC signal is significantly more detectable than the MS signal (p < 0.05) while for the search task the MC signal is less detectable than the MS signal (p < 0.05). It is worth noting that the contrast differs between the detection and search tasks but it is the same for MC and MS within the detection or search tasks. Thus, the relative difference in d′ going from the location-known detection task to the free search task represents a significant disassociation in performance.

Figure 3 (right) shows the d′ index calculated by each one of the four standard 3D model observers for a contrast level of 1. All standard model observers result in a higher d′ for the microcalcifications than for the masses. These predictions are in agreement with human detectability for the detection task but incorrectly predict the relationship of human detectability of the signals in the search task. Note that the values of d′ are extremely high since the models did not include any internal noise, however, the trend is the same for the four models.

To obtain expected human observer detectabilities using the 3D FCHO we first calculated the model detectability as a function of retinal eccentricity for both signals. Figure 4 shows the predicted detectability for the MS and MC signals for the 3D FCHO (lines). We then utilized recorded eye position data from the human experiments to analyze the statistical distribution of retinal eccentricities for the signal across all trials of the detection and search tasks. Figure 4 shows a histogram distributions (bars) for the minimum distance between the signal location (over the slices of the 3D image stack in which the signal appears) and the fixation point for each trial for the detection and search tasks.

Figure 4.

Bars show the average histogram for all the participants in terms of eccentricity in degrees visual angle from the signal location to the closest fixation point in the 3D detection task (left) and in the 3D search task (right). Lines show the d′ obtained by the 3D FCHO in terms of eccentricity.

We then generated the human performance index d′ for both tasks by calculating a weighted average of detectabilities based on the distribution of the minimum retinal eccentricities of the signal measured in our human experiments and the d′ obtained by the 3D FCHO in the same eccentricities. Figure 5 shows the result of this weighted detectability index d′ for our tasks which can successfully predict the disassociation of signal detectabilities across the detection and search tasks.

Figure 5.

Predicted model observers detectability index calculated for the 3D FCHO as a weighted average of detectabilities across eccentricities corresponding to the distribution across trials of observers’ closest fixation points to the signal on each trial.

4. DISCUSSION AND CONCLUSION

We explored how human visual detectability for two signals varies when modifying the 3D task from a location-known detection task to a search task. Our results showed that small microcalcification type signals are easier to detect than larger mass-like signal when the location is known exactly but harder to find during the search task. This dissociation can be explained by the fact that the detectability of the small microcalcification-like signal degrades more abruptly with retinal eccentricity than the mass-like signal. Since the 3D search task involves a much larger number images to process during the stack-browsing, the observer has to rely on the visual periphery in order to scrutinize a limited number of regions with the fovea. For our signal and images, the larger mass-like signal is still visible in the periphery but the smaller microcalcification-like signal is easily missed. Only when the observer is fixating on the smaller signal, does it become easily visible. Thus, the search task introduces a higher performance cost when searching for the small microcalcification-like signal.

We showed that standard model observers (NPW, NPWE, Hotelling, Channelized Hotelling) with a 3D extension that only model foveal vision did not predict the relative performance of the signals for a free search task. These models predicted a higher detectability for the microcalcification-like signal (MC) than for the mass-like signal (MS) across tasks. This accurately describes human performance for the location-known detection task but incorrectly describes the observed performance in the search task.

The developed 3D Foveated Channelized Hotelling Observer model, combined with measurements statistics of the retinal eccentricity of the signal across trials, was used to successfully predict the dissociation in performance across the detection and search tasks. Our results argue that successful evaluation and optimization of medical image quality for search in 3D images may require a new generation of model observers that model visual detectability across the entire visual field (foveated model observers).