Zoom is Meaningful: Discerning Ultrasound Images’ Zoom Levels

Abstract

The paper explores the use of an under-utilised piece of information extractable from fetal ultrasound images, ‘zoom’. In this paper, we explore the obtainment of zoom information and conclude with a couple of potential use cases for it. We make the case that zoom information is meaningful and that convolutional neural networks can distinguish between the different zoom levels that images were acquired at, even if images were manipulated post-acquisition.

1. Introduction

As part of routine fetal ultrasound (US) scanning, a sonographer will repeatedly zoom-in and zoom-out to improve the visualization of the structure of interest [1]. As opposed to simple image magnification, once a section of the ultrasound image is, in real-time, zoomed into, the echoes outside the remaining examination area are no longer evaluated, thus improving temporal resolution as well as line density [1]. In this work, we argue that zoom information, in addition to being readily retrievable off images, can offer supplementary information during ultrasound image analyses. We show that zoom magnification information is meaningful, as in the image content acquired at a specific zoom level is distinct to a deep learning model irrespective of the kind of post-processing one might attempt to do on an image to make it appear to have been acquired at a different zoom level. This meaningfulness could then be helpful for other downstream tasks. In this work, we are assuming that one only has access to a dataset of ultrasound images such as that shown in Fig. 1, and not the original DICOM files, and therefore, requiring the application of techniques to extract information. Our contribution in this paper is to prove that ultrasound images at different zoom levels, are distinct; thereby implying the zoom level of an ultrasound image is meaningful. The way we go about this is by training a CNN model to predict the zoom level, specifically a value we introduce called the Reverse Quasi-Zoom (RQZ) value. In addition, introducing images from different structures do not significantly hamper the quality of RQZ prediction and neither do post-acqusition rescaling and fan-shaped cropping of the US images, thereby implying that zoom and scale are different despite their interchangeable use by laypeople.

Fig. 1.

An ultrasound image with the depth scale being shown as well as the magnification factor (that we do not use due to its values being difficult to interpret due to inconsistency, as in it is unclear if a larger value means an image is more magnified or less). On the depth scale, the distance between the major lines (or the dashes) is 5 cm.

Related Work

One area in medical imaging that considers images at different magnification levels is histopathology. There is work on multi-scale approaches to classify histopathological images [2], incorporating magnification into their framework. In the realm of histopathological analysis, breast cancer image datasets are often labeled with magnification levels such as 40X, 100X, etc, and Mayouf et al. [3] has relied on ordering data based on magnification to better train classification models. Dong et al. [4] attempt to predict zoom and other details pertaining to cardiac ultrasound as part of their framework. However, their definition of zoom is binary. Either the entire chest silhoutte area is visible within the fan-shaped area, covering 2/3rds of the area (1) or not (0). Patra et al. [5] and Ashtaiwi [6] explore the optimal magnification level for histopathological images on which to train Convolutional Neural Networks (CNNs) for the task of breast cancer detection.

2. Methods

Obtaining Reverse Quasi-Zoom (RQZ) Labels

RQZ labels are ‘soft labels’. They are a discrete variable ranging from 0 to 4. Images are assigned one of those five values even though magnification is actually a continuous phenomenon, ergo an image could theoretically be magnified gradually as it goes from one level of magnification to another, but for the purposes of our work, we assume a discrete approach to representing magnification values through the RQZ labels hence why we describe them as ‘soft’. ‘Quasi’ is used in its name because we are not directly working with zoom levels as the continuous values that they are but a rough approximate obtained through binning. RQZ labels are called ‘reverse’ because they have an inverse relationship with zoom or magnification. RQZ is a term that is inversely related to zoom that is based on the number of lines (‘-’) that represent the reaching of 5 cm of depth. These lines are present along the depth scale on an ultrasound image. To put it simply, a higher RQZ number indicates a more zoomed-out image.

$$\text{RQZ}\propto \frac{1}{\text{zoom}}$$ (1)

What we are suggesting is that the Graphical User Interface (GUI) provides information indicative of magnification levels, indirectly through the depth scale. On the depth scale, the delineation is marked by major lines at 5 cm intervals, minor lines representing 1 cm, and dots marking every 5 mm increment in depth. Increased zooming significantly reduces the amount of viewable depth in terms of fetal US content. As the zooming increases, the visible depth decreases significantly. This is an important consideration as it impacts the amount of information available for analysis. As the RQZ level decreases—indicating a more zoomed-in state—the visible depth correspondingly decreases. So, the consistent intervals of major and minor lines, along with the dots, serve as a standard reference to understand the extent of magnification applied to the image. In this work, we have used the major lines primarily to determine what we call the Reverse Quasi-Zoom level. The more of the lines are present, the higher the RQZ and the more zoomed out the image is. The GUI also shows a value called the Magnification Factor; however, the numbers provided are ambiguous and do not offer clear insight. A zoomed out image could have a relatively high magnification factor value as well as a zoomed in image but another zoomed out image could have a low magnification factor value. The ambiguity surrounding” Magnification Factor” hinders its utility as a measure for our analysis, necessitating our approach of obtaining some semblance of magnification through the RQZ values. Detecting the dashes (‘-’) was approached initially through basic line detection and Optical Character Recognition (OCR) techniques. However, these methods faced challenges due to the size of the lines, making detection a struggle. Consequently, a simpler method was adopted, utilizing pixel values to identify the dashes. While the line detection and OCR methods could potentially be refined with further investigation, the pixel value method was deemed sufficient for the task at hand, providing a pragmatic solution for determining the dashes and, by extension, the magnification levels (specifically the RQZ). The method of detecting dashes (‘-’) entails analyzing pixel values within a defined region, specifically between coordinates (325, 90) and (407, 945), which encompasses the scale’s location in the image. Within this region, rows (y-values) are identified where the count of non-zero values ranges between 32 to 34, as this range closely aligns with the length of a single dash, which is 33 pixels. In situations where the scale is overlaid on ultrasound (US) content, the process of detecting dashes (‘-’) necessitates a different approach due to the potential interference from the US imagery. Instead of relying solely on the count of non-zero values, the changed approach considers the approximate pixel intensity values of the dashes. This approach would allow the detection of the dashes even in the event that depth scale is overlaid onto the ultrasound image content. Utilizing OpenCV’s [7] inRange function, a specific range for the off-white color of the ‘-’ will be detected. The number of ‘-’ present within the right length range and the right color range will determine the RQZ value. The ImageJ software [8] was used as a tool to directly ascertain the pixel value (color) of the ‘-’, which is crucial for accurate detection. The RQZ value is effectively the count of dashes in a specified region R of an image I is given by the total number of sequences of pixels p of length n where 32 ≤ n ≤ 34 that satisfy the following conditions for each row r and starting column c in R:

$$\begin{array}{l}\text{RQZ }=\hfill \\ {{\displaystyle \sum _{r,c}1}}_{\left\{\begin{array}{l}\forall i\in \{0,\dots ,n-1\}:p_{\text{min}}\le (r,c+i)\le p_{\text{max}},\hfill \\ \nexists \delta \in \{1,\dots ,5\},i\in \{0,\dots ,n-1\}:\hfill \\ p_{\min }\le I(r+\delta ,c+i)\le p_{\max }\hfill \end{array}\right\}}\hfill \end{array}$$ (2)

Most Probable Timeline Plots

We introduce these plots in Figs. 3 and 4 we call the most probable timeline plots (MPTP). MPTPs show how the most probable RQZ changes with time for a structure. From these plots, we can tell that different structures have different magnification behaviours associated with them. We plotted an MPTP for nuchal translucency (NT) images. Rather than sampling every continous image frame, we sample three frames from a one-second duration when plotting MPTPs. Continuous samples might not be essential due to minimal zoom change within this time frame (1/3 of a second). MPTPs were generated for sagittal view of the heart (seg), Crown-Rump Length (CRL), and Nuchal Translucency (NT) images. Within MPTPs, we identify certain points as interesting. These points are characterized by either a transition in RQZ level or the maintenance of a RQZ level for a significant amount of time. Additionally, the start and end of the timeline, along with instances where a transition occurs, have been acknowledged as potentially interesting points.

Fig. 3.

Most Probable Timeline plot for the seg class (sagittal view of the heart). CMT and NDL stand for Changes in Magnification over Time and Number of Detected Lines, respectively. Green squares represent points of potential interest either through a change occurring in terms of RQZ or in terms of a certain threshold of the number of succeeding frames of the same RQZ value being reached. From these plots, we can begin to entertain the idea that different views and different structures would have different magnification patterns associated with them.

Fig. 4.

Most Probable Timeline plot for the NT class (Nuchal Translucency). Refer to the caption of Fig. 3 to check for the long form of the abbreviations.

Model Architecture

The architecture of the RQZ classification model used was based on Efficient-netb0 [9] with five possible classes, each associated with an RQZ level. The model was then evaluated to ascertain its ability to distinguish different RQZ levels accurately.

3. Experiments

Dataset and Data Preparation

As part of the PULSE study [10, 11, 12], information from full-length fetal ultrasound scans are acquired including ultrasound video. A commercial Voluson E8 version BT18 (General Electric Healthcare, Zipf, Austria) ultrasound machines [13] equipped with standard curvilinear (C2-9-D, C1-5-D), and 3D/4D (RAB6-D) probes were used to perform all the ultrasound scans used in our work. Pre-processing of the data samples included cropping the images to remove the US system’s GUI and resizing the US images to 224 × 224 pixels as the CNN expects. We work with three classes: CRL (crown-rump length), NT (nuchal translucency, seg (sagittal view of the heart). A total of 51,431 samples were used during training and validation, and 22,286 samples were reserved for evaluation.

RQZ Classification Task

The RQZ classification models were trained with categorical cross entropy loss. We build four different experiments with RQZ classifiers. First, classifying RQZ given images of only one anatomical structure. Second, classifying RQZ given images of multiple anatomical structures. Third, classifying RQZ given rescaled images. Fourth, classying RQZ given fan-shaped cropped images.

$$CE(y,p)=-(y\text{log}(p)+(1-y)\text{log}(1-p))$$ (3)

$$F_1=2\cdot \left(\frac{\text{precision}\cdot \text{recall}}{\text{precision}+\text{recall}}\right)$$ (4)

4. Results and Discussion

In Table 1, we can see the classification report of the RQZ classifier with images from a single anatomy, showing that frames from different zoom levels are indeed meaningful. The test accuracy of the model on test images was 99.24%. Fig. 6a. shows us the confusion matrix of the first experiment.

Table 1. Classification report of the RQZ classifier.

RQZ	Precision	Recall	F1-Score	Support
0	0.99	1.00	0.99	2146
1	1.00	0.99	0.99	5391
2	0.99	0.98	0.99	1757
3	0.97	0.99	0.98	340
4	1.00	0.92	0.96	13

Fig. 6.

a. Confusion matrix (CM) of the RQZ task, b. CM of the RQZ task with images of different anatomies, c. CM of RQZ task with fan-shaped cropped images.

While the current results are promising, it would be intriguing to evaluate the model’s performance when incorporating images from other anatomies. One would expect a significant decrease in performance; however, results in Table 2 and Fig. 6b. suggest consistent performance. We then conducted the experiment where all images were rescaled with a rate between 0.5 and 2.0. Images were centre-cropped as necessary to ensure a consistent input shape. A test accuracy of 99.06% was achieved. Fig. 7 shows the confusion matrix. Next, we discuss the experiment with fan-shaped cropping of the images. Fig. 6c. shows the confusion matrix. A test accuracy of 99% is achieved. From the results, we conclude that zoom is meaningful. In summary, the classifier performed well. The results aligned with our theoretical predictions regarding misclassifications. An analysis of the confusion matrix revealed an interesting pattern: the majority of misclassifications, when they did occur, were typically adjacent to the true class. This observation aligns with our expectations, given that RQZ levels such as RQZ=1 are inherently closer to RQZ=0 and RQZ=2 than they are to RQZ=4. This trend suggests that zoom data possesses discriminative power. The results of the RQZ classification experiment where images have been rescaled brings as to an interesting conclusion: despite the size of objects on the screen, the content looks different at different zoom levels and simply enlarging or shrinking an image post-acquisition will not make the inherent image content similar to an actual zoomed out or zoomed in image. This aligns well with the statement by [1] that make a distinction between real-time zooming during a live scan and other kinds of rescaling, but we prove it quantitatively with these experiments here. Real-time zooming leads to a greater image resolution (therefore higher image quality) that is not possible with post-acquisition rescaling. A reason for this phenomenon is that during real-time zooming, the machine stops processing the ultrasound echoes coming from outside the area to be shown on the screen, thus improving the image quality within the zoomed area. Therefore, with real-time zooming, processing resources are reallocated to the area of interest. The results of the fan cropping experiment shows us that the model is still able to learn features associated with the different RQZ levels despite the shape of the fan.

Table 2. Classification report of the RQZ classifier in the task including images from different anatomies.

RQZ	Precision	Recall	F1-Score	Support
0	1.00	1.00	1.00	4205
1	1.00	1.00	1.00	6183
2	0.99	0.99	0.99	1865
3	0.99	0.97	0.98	345
4	1.00	0.92	0.96	13

Fig. 7. CM of RQZ task with rescaled images.

5. Conclusion

Our work shows that zoom information obtained from an US image is meaningful and that an image at one RQZ level cannot be manipulated post-acquisition to appear as an image of a different RQZ level. Currently, some US images are unusable, such as those measuring fetal heart rate, because they have the depth scale out of position. There are downstream tasks that could benefit from utilizing RQZ values. Captioning could benefit from zoom level determination, especially when details regarding image scale or regions are needed. Furthermore, identifying the most zoomed-in version of an image might help to identify standard planes and ideal frames for measurement. Finally, gestational age estimation could also benefit from zoom information.

Fig. 2. The figure shows two US images at different RQZ levels with major lines circled in cyan.

Fig. 5. Model architecture for the RQZ classifier.

Fig. 8. Examples of rescaling and fan-shaped cropping.

Fig. 9. Positive qualitative results for the RQZ classifier.

Fig. 10. Negative qualitative results for the RQZ classifier.