Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Mar 18.
Published in final edited form as: Proc SPIE Int Soc Opt Eng. 2023 Apr 6;12471:1247109. doi: 10.1117/12.2653923

Extended Depth of Field Imaging for Mosaic Hyperspectral Images

Armand Rathgeb a,b, Ling Ma a,b, Minh Tran a,b, Baowei Fei a,b,c,*
PMCID: PMC10947262  NIHMSID: NIHMS1973776  PMID: 38500779

Abstract

Hyperspectral imaging (HSI) is an emerging modality for digital pathology. The purpose of this study is to develop an extended depth of field (EDOF) method for mosaic hyperspectral images acquired with a snapshot camera. EDOF is a technique for ensuring that an image is in focus at all points. A stack of mosaicked hyperspectral images of hematoxylin and eosin (H&E)-stained histologic slides were acquired at different positions along the z-axis and used to output a hyperspectral histologic image that was in-focus at every point. Three different methods were compared to achieve a fully focused image. We compared conventional patch-based methods to our proposed growth-based and band-based methods. The Brenner function was used to quantitatively measure the focus quality of each image measured. The results show that both of our proposed methods performed better qualitatively and quantitatively than the patch-based method, with the band-based method performing the best, as it leveraged dividing pixels into their proper wavelengths in addition to spatially, giving the algorithm better contrast to measure. In terms of speed, the band-based method was the fastest, followed by the patch-based method, with the growth-based method being the slowest. Our proposed extended depth of field hyperspectral imaging methods can have immediate applications in digital pathology, especially whole slide imaging, and other microscopic imaging.

Keywords: Hyperspectral image, mosaic, snapshot, extended depth of field (EDOF), digital pathology, histology

1. INTRODUCTION

Computer-Aided Pathology (CAP) is an emerging research area that seeks to aid pathologists in making a diagnosis1. CAP involves digitizing whole-slide images for analysis by pathologists or machine learning methods to detect cancer and other diseases1. When the process of capturing a whole slide is not automated, it can be lengthy and error prone. After long periods in front of a microscope or computer screen, humans will tire and can capture images that are not in focus or have some other quality defect. To prevent these kinds of errors, automatic whole-slide imaging (WSI) can help obtain accurate, fully focused images.

In microscopic imaging, due to the changing thickness of slides, different planes of focus can exist in a single image. This changing thickness can lead to certain areas of a picture acquired being in-focus while others being out of focus, precluding the use of simple autofocus techniques, such as peak-search or hill-climbing algorithms, which only try to select the image with the best focus but ignore specific regions inside of each image2. One solution to this problem is to use a technique called extended depth of field imaging (EDOF). EDOF can be implemented through both hardware and software means. Hardware methods generally involve specially crafted lenses, such as the intraocular lenses which are implanted into the eyes of patients with conditions such as presbyopia3,4. These hardware options, however, can be costly as they require either a precisely manufactured lens, or the use of a specifically designed optical component in front of the lens in order to acquire a known point-spread function5, which can still result in artifacts appearing in the final image6. As a result of these limitations, it is advantageous to search for software-based approaches for EDOF. Software-based EDOF acquires a stack of images along different focal planes in a slide. A variety of methods can then be used to create a fully focused images, such as deep-learning models which depend on point-spread functions7, or by dividing the images up spatially into stacks of smaller patches and picking the most focused patch for generation.

Hyperspectral imaging (HSI) is an imaging modality that has been widely used in remote-sensing applications but is growing in popularity for use in medical applications. One important upcoming use for HSI is in the field of digital pathology, as the extra spectral information provided can be useful for a variety of analysis tasks, such as cancer detection8–12. To fully investigate a histological slide for cancer detection, whole-slide imaging is commonly employed to acquire the digital histologic image of the entire slide. It involves automatic scanning and focusing of the slide to yield an image with good quality. HSI has grown quickly in the medical field, but very few studies have been done to prove the feasibility of automatic HSI for WSI8. One drawback to HSI WSI is the length of time it takes to acquire a whole slide image, the final image’s file size, and the difficulty in implementing auto-focusing8. Many HSI WSI studies use line-scanning or spectral-scanning hyperspectral cameras which involve scanning across a wide set of spatial locations and light spectra to acquire a hyperspectral image3. As a result, the scanning time of a whole slide can be drastically prolonged and due to the many spectral bands acquired, the resulting hyperspectral image can take up to 5 GB of disk space8. Snapshot HSI methods, which involve the capture of hyperspectral images using a 2-D detector array with a single exposure13, can help to solve this problem. The image captured by the 2-D detector array can then be demosaicked to create a 3-D hypercube, a hyperspectral image where the first two dimensions are spatial dimensions, and the third refers to the wavelength at which that set of spatial pixels was acquired at13. Regarding auto-focusing, commercially available systems have already been created for the auto-scanning and focusing of color histologic images, but these do not directly work with hyperspectral images. Very few studies have achieved autofocusing and auto-scanning methods for hyperspectral images. Line-scanning and spectral-scanning microscopes especially face large challenges in fast autofocus methods due to their interleaved band format. Although snapshot cameras have fewer bands and a lower spatial resolution than other hyperspectral imaging methods, it is a good solution for this application due to its speed and simple data format.

In this study, we proposed two rapid methods for extended depth of field imaging of mosaic hyperspectral images which we compared with a conventional patch-based EDOF method. Our simple algorithms allow for the blind construction of a fully focused image from a stack of partially focused images.

2. METHODS

1.1. Automatic hyperspectral microscopic imaging system

Our automatic hyperspectral microscopic image acquisition system consisted of an inverted brightfield microscope with a customized, automated stage and a snapshot visible light (VIS) hyperspectral camera. The microscope includes a 100-Watt halogen light source, which was used for illumination. The snapshot camera has a sensor size of 2048×1088 pixels, with 4×4 mosaic filters integrated on the sensor. The acquired snapshot image can be demosaicked into a 512×272 pixel × 16 band hypercube, covering a wavelength range of 460 – 600 nm. In addition, we developed a piece of software based on the camera and microscope’s respective application programming interfaces to automatically control the microscope stage and snapshot camera to capture stacks of hyperspectral images along the whole length and width of a selected region in a matter of minutes. Figure 1 shows the imaging system we used in its entirety.

Figure 1.

Figure 1.

Open in a new tab

Our customized hyperspectral microscopic imaging system for automatic image stack acquisition.

1.2. Data acquisition

We performed hyperspectral image acquisition at 60× objective magnification, with a space of about 3 microns between each image in the z direction. We made sure to take all the images at different regions on the slide with little to no overlap. This made sure that the level of focus would vary between each image in our stack of 2-D mosaics since we changed the distance from the lens to the slide by 3 microns after every capture. We scanned the whole slide by treating it as a square divided into quadrants. Each quadrant was scanned at different heights to acquire each image stack. We captured 14 images from each region and discarded images with no tissue shown.

The acquired hyperspectral image dataset consisted of 271 stacks, each containing 14 2-D mosaicked hyperspectral images taken from a single hematoxylin and eosin (H&E)-stained histologic slide. All images were taken from different regions on tumor-normal margin slide of human head and neck cancer.

1.3. Selection of the sharpness evaluation function

The next step in the process was to select an algorithm to quantitatively measure how in-focus an image was. A variety of metrics can be used for determining how focused an image is, of which Lozano compared 9 different algorithms concluding that the Brenner function was the most accurate metric for use on hyperspectral images14. Most algorithms to determine focus use a form of edge-detection to measure how sharp an image is. They function on the principle that the sharper an image is, the more in-focus it is. We compared three different algorithms for use in this paper: the Laplacian14, Brenner14, and Tenengrad14 functions. All three algorithms have been found to work well with hyperspectral images in the past14. We implemented each algorithm in MATLAB and tested them on a stack of images acquired by graphing the measured sharpness values. We compared the curves generated by each algorithm, focusing on the slope. Better algorithms will have better defined peaks with sharp slopes, showing that the algorithm can make a clear decision as to the best image. Figure 2 shows three of the resulting graphs generated by each algorithm. All three functions show a clear distinction as to the best image and a steep slope. We also compared the images selected by each algorithm to determine which algorithm made the best decision. We used the perception-based image quality evaluator (PIQE)15 and no-reference image quality evaluation (NIQE) metrics16 from MATLAB as our standard to compare the quality of image selected by the three different algorithms, as they are based merely on human perception of the images. The time it took for each algorithm to run was also considered. All three algorithms scored essentially the same when comparing their PIQE and NIQE scores, but the Brenner function was the fastest by a small margin, so it was selected as our main method for measuring sharpness. The Brenner function calculates the focus measure of an image as follows:

F=∑x−1M−2∑y=1N(I(x+2,y)−I(x,y))2 (1)

where I is the intensity at a specific point (x,y), M is the column size, and N is the row size.

Figure 2.

Figure 2.

Open in a new tab

The curve generated from a single stack run through the (a) Brenner function, (b) Laplacian Function, (c) Tenengrad function.

1.4. Patch-based extended depth of field method

Patch-based EDOF is a method whereby each image in the stack is divided into corresponding patches of a fixed size. In this work, we used a patch size of 128×128 pixels. Each stack of corresponding patches is evaluated, and the best patch is taken from the stack for use in the final generated image. All the selected patches from different stacks will finally be used to reconstruct a full image. Figure 3 shows a graphical representation of what was done, as well as a flowchart to help visualize the way the algorithm was implemented. We implemented this method in MATLAB and tested it on each stack of images.

Figure 3.

Figure 3.

Open in a new tab

Patch-based EDOF method. (a) Flowchart describing the patch-based method. (b) A visual depiction of how the algorithm generates the final image. The red squares are different regions used in constructing the image.

1.5. Growing patch-based extended depth of field method

The growing patch-based method relies on the assumption that most images will be in-focus for all but a tiny portion of the image. Due to this fact, it is unnecessary for an EDOF algorithm to assess every single region of an image, and it can instead try to outline the in-focus area of an image being focused. Patch size for this method starts at 128×128 pixels. The index of the image containing the first patch that is determined to be the most focused in the stack of patches is stored. The patch size will then be doubled. This process repeats until the most recently selected patch no longer comes from the same image as the previous patch. At this point, the best patch from the previous image will be used in the final reconstruction. Figure 4 explains the process in greater detail. The growing patch-based method was implemented in MATLAB and tested on the same dataset as the patch-based method.

Figure 4.

Figure 4.

Open in a new tab

Growing patch-based method. (a) Flowchart describing the growth-based method. (b) A visual depiction of what the algorithm is doing when generating an image. The red squares represent regions from different images that the algorithm is using to generate the final image. Notice how the final square is cropped off at the edge.

1.6. Band-based extended depth of field method

Another method tested involved demosaicking each image prior to the EDOF reconstruction. We used a simple demosaicking method, as we reported in Pruitt et al. Each image was demosaicked from a 2048×1088 mosaic frame into a 512 pixel ×272 pixel ×16 band hypercube, and then the hypercube had the growth-based method described in the previous section applied to each band. The first band on each cube was passed through the growing patch based EDOF method with an initial patch size of 64×64 pixels, the fully focused band was then inserted into the hypercube; next, the second band was passed through the algorithm and used for the generation, and so on. This method allows for increased image sharpness in each band as pixels from the same wavelength are sorted together, separate from those of other wavelengths. Figure 5 explains the process visually.

Figure 5.

Figure 5.

Open in a new tab

Band-based EDOF method. (a) Flowchart describing the band-based method. (b) A visual depiction of what the algorithm is doing with each band. Red squares represent the different regions taken from different images. In practice, the sizes of these regions may not be so small.

1.7. Evaluation Metrics

We evaluated the performance of our algorithms with three different metrics. The first metric we used was in the quality of the reconstruction. We determined quality based on how in-focus the final reconstruction of a hypercube was. We demosaicked the final frame from the patch and growth-based methods to measure this. We used the Brenner function to score the overall sharpness of each band and then summed the individual band scores to get the total score for each hypercube. The algorithm’s final score was found by taking the average of all hypercube scores. The next metric we used for comparing the quality was speed. We measured the speed in the number of stacks processed per millisecond. Finally, we compared the computation efficiency by measuring the number of convolutions performed and the size of patch being operated on from a representative sample of images.

3. RESULTS

After running all our algorithms in MATLAB, we found that the band-based method performed better than both the patch-based and growth-based methods in both speed and image quality. To compare image quality, we demosaicked the mosaics generated by the patch-based and growth-based algorithms, and then compared the scores of the individual bands generated by all three algorithms with one another. The bands were scored using Brenner function. The patch-based method took an average of 0.428 seconds to run and scored 1.488×1010, the growth-based method took an average of 1.012 seconds to run and scored 1.490×1010, and the band-based method took an average of 0.0829 seconds to run and scored 1.500×1010. The band-based method wound up being 12.2 times faster than the growth-based method, and 5.17 times faster than the patch-based method. The slowest of the three algorithms was the patch-based method which worked at a rate of around 0.15 stacks per millisecond. The band-based method also scored slightly higher than the other two methods. Figure 6 compares the performance of each algorithm. The patch-based method scored the worst of the three methods and had areas where the reconstruction failed to select the best and the image was left very grainy as shown in Figure 7.

Figure 6.

Figure 6.

Open in a new tab

Performance of the three algorithms. (a) The average score assigned by the Brenner function to each band of the final hypercube. (b) Speed in the number of stacks processed per millisecond.

Figure 7.

Figure 7.

Open in a new tab

Comparison of errors using three methods. (a) A band taken from a hypercube generated by the patch-based method displaying errors in the selected patch. The errors are circled in red. (b) The same band but generated by the growth-based method. (c) The same band but generated by the band-based method. No errors can be seen in the growth and band-based methods.

In addition, the number of times the image was convolved when measuring quality was measured. In one case, the patch-based method made 2016 2-D convolutions with 128×128-pixel patches, while the band-based method made only 896 calls with patches that could only be up to 512×272 pixels, and the growth-based method made 70 calls to the convolution with patches that could be up to size 2048×1088 pixels. The reason the growth-based method wound up being so much slower despite how few convolutions it performed was due to the size of the patches it was convolving. While it convolved fewer patches, these patches were necessarily large and resulted in long convolution times and memory overhead when copying them back and forth.

4. DISCUSSION AND CONCLUSION

In this work, two fast extended depth of field methods were developed to achieve focus at every point in a mosaic hyperspectral image. Both methods present an improvement in the quality of images and add little overhead to existing processes. The patch-based method which our new methods were compared against made minor errors in reconstructions, likely due to noise in the image causing rapid spikes in the contrast between pixels. The fastest method was discovered to be the band-based method, where pixels were divided spectrally, as well as spatially. The reason for this increase in speed is likely due to the smaller number and size of patches being fed into the 2-D convolution used to determine the sharpness of an image. In terms of performance and speed, the number of convolutions, as well as the size of the patches used was the main factor. In the end, we found that the band-based method was the best method among the three methods for achieving an accurate reconstruction of a mosaicked hyperspectral image as it was the fastest method and produced the most accurate reconstruction.

In conclusion, our proposed EDOF methods are advantageous to improve the quality of automatically acquired hyperspectral histologic images using a snapshot camera. It promises to speed up future work by making it quick and easy to acquire large datasets with little to no human interaction. In the future, deep learning might be used to reconstruct the images.

ACKNOWLEDGMENTS

This research was supported in part by the U.S. National Institutes of Health (NIH) grants (R01CA156775, R01CA204254, R01HL140325, and R21CA231911), and by the Cancer Prevention and Research Institute of Texas (CPRIT) grant RP190588.

REFERENCES

  • [1].Madabhushi A and Lee G, “Image analysis and machine learning in digital pathology: challenges and opportunities,” Med Image Anal 33, 170–175 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [2].Zhang Q, et al. , “An autofocus algorithm considering wavelength changes for large scale microscopic hyperspectral pathological imaging system,” J. Biophotonics 15(5), e202100366 (2022). [DOI] [PubMed] [Google Scholar]
  • [3].Kohnen T, Suryakumar R, “Extended depth-of-focus technology in intraocular lenses,” J Cataract Refract Surg 46(2), 298–304 (2020). [DOI] [PubMed] [Google Scholar]
  • [4].Zalevsky Z, “Extended depth of focus imaging: a review,” SPIE Rev. 1(1), 018001 (2010). [Google Scholar]
  • [5].Moein S and Suleski TJ, “Freeform optics for variable extended depth of field imaging,” Opt. Express 29(24), 40524 (2021). [DOI] [PubMed] [Google Scholar]
  • [6].Zammit P, Harvey A, and Carles G, “Extended depth-of-field imaging and ranging in a snapshot,” Optica 1, 209–216 (2014). [Google Scholar]
  • [7].Akpinar U, et al. , “Learning Wavefront Coding for Extended Depth of Field Imaging,” IEEE Trans. Image Process 30, 3307–3320 (2021). [DOI] [PubMed] [Google Scholar]
  • [8].Ma L, et al. , “Unsupervised super-resolution reconstruction of hyperspectral histology images for whole-slide imaging,” J. Biomed. Opt 27(5), 056502 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [9].Leitch K, et al. , “Detecting aggressive papillary thyroid carcinoma using hyperspectral imaging and radiomic features,” Proc. SPIE 12033, 1203322 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [10].Ma L, et al. , “Automatic detection of head and neck squamous cell carcinoma on histologic slides using hyperspectral microscopic imaging,” J. Biomed. Opt 27(4), 046501 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [11].Tran MH, et al. , “Thyroid carcinoma detection on whole histologic slides using hyperspectral imaging and deep learning,” Proc. SPIE 12039, 120390H (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [12].Ortega S, et al. , “Hyperspectral imaging and deep learning for the detection of breast cancer cells in digitized histological images,” Proc. SPIE 11320, 113200V (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [13].Lu G and Fei B, “Medical hyperspectral imaging: A Review,” J. Biomed. Opt 19(1), 010901 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [14].Lozano KM, “Hyperspectral Microscope Auto-Focusing,” Master Dissertation, Montana State University; (2018). [Google Scholar]
  • [15].Venkatanath N, et al. , “Blind Image Quality Evaluation Using Perception Based Features”, 2015 Twenty First National Conference on Communications (NCC) 1–6, (2015). [Google Scholar]
  • [16].Mittal A, Soundararajan R, and Bovik AC, “Making a Completely Blind Image Quality Analyzer.” IEEE Signal Process. Lett 22(3), 209–212, (2013). [Google Scholar]
  • [17].Pruitt K, et al. , “A high-speed hyperspectral laparoscopic imaging system,” accepted for Proc. SPIE 12466, (2023) [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES