Super-Resolution Digital Pathology Image Processing of Bone Marrow Aspirate and Cytology Smears and Tissue Sections

Abstract

Background:

Accurate digital pathology image analysis depends on high-quality images. As such, it is imperative to obtain digital images with high resolution for downstream data analysis. While hematoxylin and eosin (H&E)-stained tissue section slides from solid tumors contain three-dimensional information, these data have been ignored in digital pathology. In addition, in cytology and bone marrow aspirate smears, the three-dimensional nature of the specimen has precluded efficient analysis of such morphologic data. An individual image snapshot at a single focal distance is often not sufficient for accurate diagnoses and multiple whole-slide images at different focal distances are necessary for diagnostics.

Materials and Methods:

We describe a novel computational pipeline and processing program for obtaining a super-resolved image from multiple static images at different z-planes in overlapping but separate frames. This program, MULTI-Z, performs image alignment, Gaussian smoothing, and Laplacian filtering to construct a final super-resolution image from multiple images.

Results:

We applied this algorithm and program to images of cytology and H&E-stained sections and demonstrated significant improvements in both resolution and image quality by objective data analyses (24% increase in sharpness and focus).

Conclusions:

With the use of our program, super-resolved images of cytology and H&E-stained tissue sections can be obtained to potentially allow for more optimal downstream computational analysis. This method is applicable to whole-slide scanned images.

INTRODUCTION

While there is a global shift toward image processing and digitizing medical slides and images, there is a clear and definite obstacle: the limited depth of focus in optical microscopes and imaging systems. Due to this limitation, it is not possible to easily capture a singular image with high spatial resolution.

In general, low image pixel resolution is caused by extrinsic and intrinsic factors. The intrinsic factors are due to the interaction between wave particles of light and the substance.[1] Optical microscopes have the capability to only focus on one object plane in the field of view or a specific z-level at a certain time, disregarding crucial cellular features at different planes. The design of the optical microscope, focal length, and distance to specimen limit the depth of field to a two-dimensional plane of focus when viewing slides.

In imaging systems, changing the aperture and thus the f-number allows for focusing of different distance from the capturing device. Conversely, with a microscope, slight alterations in the distance of the specimen allows for shifts in focus of separate layers of a slide. Hence, when examining tissue section slides under a magnification, a single plane is insufficient due to slight differences in the object plane focal lengths. Subsequently, we must take multiple images of a single field of view at different z-levels to appropriately represent the entire information in that field of view on the slide.

The extrinsic factors include resolution enhancements on captured images.[2] Hence, with a set of images of different focal depths, super-resolution (SR) techniques are necessary to generate an accurate representation of the slide in one image. This allows for more efficient analysis of digital slides, expedites collaboration, and improves downstream computational analysis.

The high-resolution medical images allow for reliable digital duplication of information for collaboration in accurate diagnoses. In addition, with the rapid development of computer-assisted diagnostic models (i.e., computational algorithms that use digital image data to generate diagnoses), there is an increased focus on improving accuracy models.[3,4,5,6] An overlooked aspect in the development of these computer-assisted diagnostic models is the quality of the training data or the input images, thus providing an unrealized potential in improving accuracy of diagnostic models with super-resolved images.

SR imaging refers to a set of techniques that improve and enhance resolution of images. Harris[7] introduced the mathematical foundation of SR algorithms with a theory on solving diffraction in 1964, then Tsai et al.[8] addressed the problem in the context of enhancing the spatial resolution using the spatial aliasing effect, and subsequently, many studies have been conducted using a frequency domain approach; all these approaches use multiple low-resolution images to generate a single super-resolved image.[9,10,11] These techniques have a broad scale of applications, including security surveillance, astronomy, entomology, satellite photographs, and medical images. A commonly used approach to increase spatial resolution (i.e., the number of independent pixels values used to construct an image) is to use signal processing techniques.[2] This includes focus stacking, that is, an approach to produce SR images from a subset of low-resolution images with varying focal depths.

Currently, the most common approaches for focus stacking, or more specifically multifocus image fusion, use either Fourier analysis (i.e., to decompose an image into its sinusoidal components in the frequency domain) or edge detection (i.e., a set of techniques used to identify “sharp” regions of an image).[1,9,12] While these approaches work well in certain cases, they do not perform well for feature-oriented detection.[13] In addition, we have considered using a weighted or enhanced map method, but these approaches are aimed to reduce noise and smooth the images, which can cause a loss of valuable data from images of tissue slides. We have proposed a method utilizing a scale-normalized Laplacian operator on Gaussian smoothed images to generate the scale-space representation. This approach will detect the highest resolution regions (strongest edges) through applying the Laplacian operator to produce a gradient map, which is used to generate an image with a greater depth of field (apparent plane of focus). When applied to bone marrow aspirate and cytology smears and tissue sections, this approach utilizes a Laplacian of Gaussian (LoG) feature detector, identifying individual cells, lymphocytes, monocytes, and neutrophils that are focused in the image and maintaining the entire composition of a specific geometric cellular feature in the final image to prevent distortion. The Laplacian operator is a kernel that is convolved on an image to generate a gradient map and highlight intensity changes. Creating a single super-resolved image from a set of low-resolution images is computationally cheaper for downstream analysis as it does not require the analysis of multiple images of the same scene at different z-planes. In addition, having a single super-resolved digital medical slide allows for easy sharing of data at a low cost.

MATERIALS AND METHODS

Data

Images were taken using a SPOT Insight camera (SPOT Imaging, Sterling Heights, MI, USA), 4.0 megapixel, mounted to an Olympus BX41 (Olympus, Waltham, MA, 02453) microscope using SPOT Basic (SPOT Imaging) image capture software. All codes were written in Python on Mac OS (Apple Inc.; Cupertino, CA, USA). In addition, the OpenCV library (from https://opencv.org/) version 3.5.5 was used for basic implementations of published algorithms.

Computational methods

Image alignment

When capturing images of the slides under an optical microscope, the microadjustments in the z-plane may be accompanied with slight zoom or positioning variations. Appropriate compensation must be taken to alleviate the misalignment from these slight variations, and the images must be precisely aligned to allow for comparison.

We utilized a feature detection algorithm to locate the descriptors and key points. The Scale-Invariant Feature Transform (SIFT)[14,15] approach was used to find key points and descriptors of an image. Lowe's SIFT algorithm utilizes scale-space filtering with the difference of Gaussian, an approximation of LoG to detect blobs (local extrema) for comparison. Figure 1 shows the detected key points in the image. The SIFT algorithm was selected for this task as it performed with the highest match rate for images that were rotated of variable intensity, sheared, and noisy.[16]

Figure 1.

Illustration of identified key points using scale-invariant feature transforms. Key points are identified and noted with variably colored circles

Given these key points, we chose to use a Brute-Force matcher that accepts the descriptors of one feature and compares it with all descriptors of another image, using a set of proportional distance measurements to pinpoint a match [Figure 2]. Then, with a ratio test[14,15] set at a threshold of 80%, we determined the strong matches from the data. These strong matches were ordered from nearest to furthest.

Figure 2.

Correlation map of strong matches between two images in different z-levels. Two images of the same region of a digital slide, at different z-levels, showing “key point” identification, represented by circles in each image and their correlation, illustrated by a gray horizontal line link

Subsequently, the homography was calculated using the strong matches to describe the relation of the images in the same frame.[17] The homography matrix is a mathematical expression that outlines how to wrap and orient the images so that they can be expressed in one coordinate frame. Specifically, the Random Sample Consensus[18] algorithm was used to determine the transformational relationship of the images while filtering out the extraneous correlations.[18] The maximum reprojection error was set to 2.0 to ensure flexibility without losing precision. The images were then transformed using the homography matrix with a linear interpolation method in order to align the images with features in the exact same position in every image.

Image convolution

A differential blob detector known as the LoG is initially convolved with a kernel represented by the following Gaussian function[19] where σ is the standard deviation.

Moreover, the Laplacian filter, which is derived from discretization of the Laplace equation, is applied to the Gaussian scale-space representation.[20]

The operations of convolving an image with the LoG kernel is equivalent to calculating the Laplace of an image convolved with the Gaussian kernel. The expression below illustrates the equivalence of these two methods in two-dimensional cases.

Therefore, Gaussian smoothing was performed on the images using a kernel of size 3 × 3. Through a set of optimization experiments, we have determined that a 3 × 3 Gaussian kernel is ideal for this microscopy case.

The Gaussian smoothing is used to reduce the likelihood of false positives during feature detection. Gaussian smoothing occurs through convolving the above kernel above through each pixel of the image, resulting in a transformation of the image.

Feature detection and stacking

The Laplacian filter, as shown below, was applied on the Gaussian smoothed images. The Laplace is calculated through the addition of the x and y second derivatives.

The LoG, in this case, is often mislabeled as an edge detector. In contrast with search-based edge detection, which finds local extrema of the first order derivative, the Laplacian is a second-order derivative operator. The zero-crossings of the second derivative locate the extrema of the first-order derivative function and are the inflection points of the function. Since the Laplacian is calculated as the sum of the second derivatives, the zero-crossing will be localized along the gradient. When the second derivative in all directions is added, it will yield a negative response along a convex edge gradient transition and positive response on a concave edge gradient transition, thus resulting in zero-crossing on curved edges. The addition of these values leads to the blob detection around distinct features with a strong inflection.[20] Figure 3 illustrates the image representation after the Laplacian kernel is convolved on the Gaussian smoothed image.

Figure 3.

Plot of Laplacian of Gaussian filter on three images at different z-levels. Different z-levels of the same region of interest (×400 magnification) with the Laplacian of Gaussian filter applied. The Laplacian of Gaussian filter results are seen to the right of each image

After computing the Laplacian, the absolute magnitude of the values was calculated. The absolute value allows us to locate both the strong positive and negative responses. Then, through determining the maximum value for each pixel value from the set of images, we can create a final image with the strong in focus responses, in the shape of small blobs. Finally, using a Boolean mask, we determined which pixels and regions to use for the output.

Data analysis

Beyond professional subjective evaluation, we used six main quantification techniques to perform the analysis on whether our approach yielded high-resolution images. We selected a series of objective comprehensive analysis methods to assess the performance of our approach.

The Structural Similarity Index Measure (SSIM) is a commonly used quality metric that quantifies the similarity of images through evaluating the correlation, luminance distortion, and contrast distortion.[21,22] This metric is known to mirror the human visual system to provide an intuitive measurement. We use it as an independent control test to illustrate the presence of difference between the images with the formula below.

SSIM (f, g) = I (f, g)c(f, g) s (f,g)

The mean square error (MSE) is an error metric that can be used to assess image quality and quantify the average change in pixel value between two images.[23]First used by Carl Friedrich Gauss to quantify variance, MSE is a widely used technique for signal and wavelet strength analysis.[22] The MSE was calculated to elucidate the presence of a significant decrease in noise and shift in signal strength. The MSE between two images is calculated with the formula below, with M and N as the number of rows and columns in the images. Figure 4 illustrates the complete workflow diagram for the MULTI-Z approach.

Figure 4.

MULTI-Z workflow diagram

The peak signal-to-noise ratio (PSNR) is a logarithmic metric that is partially based off the MSE and is used as a comparative image quality measurement. PSNR provides a greater insight on images with several varying dynamic ranges. In general, a higher PSNR connotes a higher image quality and focus. The PSNR is represented with the logarithmic decibel scale and is computed on 8-bit images using the equation below.

Average gradient (AG) was calculated to illustrate the image contrast and the mean sharpness of the image. With this metric, the larger the value, the sharper the image. As a quantification of the clarity, the AG is given by the formula below.[24,25]

The Shannon entropy is a universal measure of the amount of information contained in an image.[26] The higher the entropy of an image, the greater the amount of information contained within the image. In medical applications, the difference in entropy is generally restricted as the output should not contain manufactured or predicted information and must be solely composed of the provided information. The entropy is calculated with the formula below.

The final quantitative assessment used was the variance of Laplacian method.[27] The value was calculated through convolving the Laplacian kernel and subsequently calculating the variance. Images with a larger variance, more zero-crossings of the second derivative, are more focused.

RESULTS

The aim of the described algorithm was to generate high-resolution pathology images to present the three-dimensional cellular structures that are lost with the optical microscope's two-dimensional representation of a three-dimensional surface as shown in Figure 5.

Figure 5.

Optical microscope geometric ray model. Schematic diagram of an optical microscope that details the path of light and the plane of focus, illustrating the three-dimensional aspect to a cell or tissue on a glass slide

The algorithm was applied to a series of cases and the six quantitative values were calculated as shown below. Figure 6 contains a series of images at different z-planes with respective regions of interest demarcated. The simulation and experimental results are shown in Table 1. The stacking for each set of images was repeated ten times to ensure consistent reproducibility of results. The average run time for the program for a set of three images is 8.21 s.

Figure 6.

Image result from super-resolution processing on bone marrow aspirate smears using MULTI-Z. Left: nonsuper-resolved image; Right MULTI-Z postprocessing super-resolved image. Boxed areas highlight a small region which is magnified further to clearly, but subjectively, demonstrate the increased image data and resolution provided by processing images through MULTI-Z

Table 1.

Quantitative results of image improvement following processing by MULTI-Z

The program clearly yielded images with a greater depth of field. This new application of a blob detection algorithm minimizes any visual cellular discontinuities or reduction of information with smoothing while stacking the most focused regions. The quantitative analysis from this simulation is summarized in Table 1.

The results from Table 1 indicate that there is an increase in the amount of information carried by the one image in comparison with any of the other images. In addition, the PSNR and AG show a significant increase in sharpness and focus with a 24% increase in AG. The stacking program clearly increases the sharpness of the image, providing vital three-dimensional information analysis.

DISCUSSION

Here, we describe a method for generating high-resolution digital images from several stacked images in varying z-planes. The underlying aim of this study was to be able to produce a technique to generate super-resolved pathology slide images that contain all of the relevant in-focus information in a three-dimensional field of view and not to introduce artifacts. These digital “slides” can be used for downstream analysis and easy sharing of data for collaboration. While such algorithms have been developed previously,[1,11,24,28,29] ours differs in that we are using a multifocus blob detector for detecting focused cellular geometric structures and regions to compose a final image. Applying our algorithm to bone marrow aspirate and cytology smears and tissue sections allows for a significant improvement in information entropy and image sharpness.

Several studies have evaluated the diagnostic accuracy of current z-plane stacking methods.[30,31,32,33,34] The diagnostic accuracy of stacking 20 z-plane levels was equivalent to glass slide microscopy as evaluated by 24 experienced cytologists.[32] Another impact evaluation found no statistically significant difference in diagnostic accuracy between the quality assessment of a digital slide generated from stacking three to five z-plane levels and a glass slide.[31] The study also revealed a greater average quality assessment for multiple z-plane stacked images over a single z-plane image.[31]

Our program increases the depth of field and allows for a dynamic range of information (i.e., focused material in front and behind the median focused plane of the image) to be provided, bypassing the intrinsic restrictions of optical microscopes and imaging devices. In addition, our program does not have restrictions on the number of images with varying focal planes that can be provided to the algorithm for processing. Our program is applicable to whole-slide scanned images at ×100 magnification or higher.

An obvious extension of this work is combining frequency and spatial domain approaches for image reconstruction of the three-dimensional object. Another important extension of this work could be in the use of wavelet transforms for image reconstruction.

Challenges in real-world implementation remain, such as the speed of generating these super high-resolution images after processing by MULTI-Z. Super-resolving 5 digital images at varied z-planes at a field of view of ×1000 magnification may take on the order of 1–10 s to generate a final image (depending on the size of image, CPU speed, and RAM size). Generating a whole-scanned slide at ×1000 magnification would currently be magnitude folds greater in time. Further work is needed to address this barrier to efficient utilization of the algorithm and solutions could come in the form of down-sampling of image data before alignment to increase processing capability further.

Yet questions remain such as whether the fusion of images causes introduction of microartifacts during the composition process, not distinguishable to the human visual system or metrics tested here, but significant enough to cause errors in downstream analysis. Although we did not include such data, subjective observation by three pathologists did not identify any artifacts in our analyzed and original image datasets; uniform subjective consensus was that there was a clear increase in resolution and clarity of images after SR processing by MULTI-Z.

CONCLUSION

Here, we present a novel SR processing method: MULTI-Z. This program generates SR images from multiple images at variable z-planes and uniquely performs image alignment, image convolution, and image fusion producing a final image with an objective measured increase in “relevant data” by 24%. Work to capitalize on the increased morphologic data produced by this method is an important direction of future studies, both assessment of the application of this methodology for rapid pathologic diagnoses of bone marrow aspirate specimens and cytology smears and use of images produced by this technology for downstream machine learning programs to assess morphologic features of neoplastic cells.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.