Real-time video mosaicking to guide handheld in vivo microscopy

Abstract

Handheld and endoscopic optical-sectioning microscopes are being developed for noninvasive screening and intraoperative consultation. Imaging a large extent of tissue is often desired, but miniature in vivo microscopes tend to suffer from limited fields of view. To extend the imaging field during clinical use, we have developed a real-time video mosaicking method, which allows users to efficiently survey larger areas of tissue. Here we modified a previous post-processing mosaicking method so that real-time mosaicking is possible at >30 frames/sec when using a device that outputs images that are 400 × 400 pixels in size. Unlike other real-time mosaicking methods, our strategy can accommodate image rotations and deformations that often occur during clinical use of a handheld microscope. We perform a feasibility study to demonstrate that the use of real-time mosaicking is necessary to enable efficient sampling of a desired imaging field when using a handheld dual-axis confocal microscope.

Graphical Abstract

Demonstration of real-time video mosaicking for guiding handheld microscopy (see supplemental video 1)

1. INTRODUCTION

Histopathology is the clinical gold-standard for disease diagnosis despite being invasive, destructive of tissues, time-consuming, and limited in terms of sampling extent.[1], [2] For certain point-of-care clinical applications, the ability to obtain noninvasive real-time pathology data could have significant advantages compared to traditional histopathology.[3]–[6] For example, noninvasive real-time pathology could be of value for disease screening in situations where invasive biopsy is not justified in terms of cost and risk.[2], [7] For surgical guidance, the time and cost of rapid frozen-section analysis prevents surgeons from examining more than a trace amount of tissue during surgery, and rapid in vivo microscopy could allow for much large amount so of tissue to be interrogated.[8]–[11] In particular, for critical organs such as the brain, noninvasive assessment of tissues is much preferred over invasive biopsies.[12], [13]

As mentioned previously, a number of clinical applications could benefit from in vivo real-time microscopy. In almost all cases, imaging a large extent of tissue is desired. For example, in the case of oral cancer detection, dentists and physicians often can visually identify suspicious lesions, but are hesitant to biopsy them since the majority of such lesions are benign. A noninvasive handheld microscope would allow for “image-guided biopsy” by providing images that approximate the gold standard of invasive histopathology.[2], [7] However, due to the small FOV of most handheld and endoscopic microscopes, it is difficult for users to know if they have adequately sampled/imaged the lesion. As shown in Figure 1a, real-time mosaicking would guide users as they seek to interrogate the lesion and would allow them to determine if they have successfully sampled the suspicious region. Another example is surgical guidance, where the goal is to minimize positive margins. Figure 1b shows that after removing the majority of a tumor, the boundary of the wound (i.e. the surgical margin) may still contain residual tumor which necessitates additional resection procedures.[12], [13] Rather than randomly sampling small FOVs with a handheld microscope, in which certain tissue regions may be redundantly imaged while other regions are missed, real-time mosaicking can help surgeons to efficiently survey as much of the surgical margin as possible in order to locate and excise residual malignancies (as described in the following paragraph).

Figure 1.

Two scenarios that could benefit from real-time video mosaicking over a large field of view (FOV). (a) For biopsy guidance, adequately sampling a suspicious lesion can be difficult with a handheld device that has a limited FOV (sub-millimeter). (b) For surgical guidance, imaging larger areas at the surgical margins improves a surgeon’s ability to identify residual disease for continued resection.

2. METHODS

The most common solution for extending the FOV of a microscopic system is the combination of an automated scanning stage with post-processed image mosaicking.[8], [14]–[22] However, the incorporation of a bulky robotic scanning mechanism is often not ideal for handheld or endoscopic microscopy applications. Post-processed mosaicking of overlapping image frames collected with a handheld device (with no robotic positioning) is possible.[14], [23]–[28] However, without real-time position tracking, it is challenging for a user to efficiently sample a large area of tissue, such that large regions of tissue will invariably be missed or redundantly sampled. Some real-time image mosaicking methods have been developed based on common features shared between adjacent image frames, but can only handle image translations without rotations or deformations.[24]–[26], [28] Since image rotations and deformations are common during handheld in vivo imaging of irregular tissue surfaces, we sought to develop a real-time mosaicking method that could handle both image translations, rotations and deformations.

Here, we describe the modification of a previous slow post-processing mosaicking method[27] so that real-time mosaicking is possible at >30 frames/sec with images that are 400 × 400 pixels in size, as for example collected with a custom-developed handheld dual-axis confocal microscope.[1], [2], [29] The primary goal of our mosaicking method is to guide clinical users as they seek to image a large tissue area both fully and efficiently, which is useful for accurately interrogating heterogenous tissues. This method also allows the users to locate and image small lesions (e.g. millimeter-scale) when using a relatively large handheld device that blocks a user’s visual access to these small lesions.

We previously developed and described a handheld dual-axis confocal microscope for high speed in vivo microscopy. This device requires direct contact with the tissue, preferably with a liquid (e.g. saline) or ultrasound gel to facilitate index matching. Our device utilizes MEMS-based line-scanning to achieve high-speed imaging at >20 frames/sec, in which the high frame rate is critical for minimizing motion artifacts during handheld use [1], [2], [29]. In order to enable accurate diagnostic interpretation, this device was designed to image with subcellular resolution, exhibiting an optical-sectioning thickness and lateral resolution of 1.7 and 1.1 μm, respectively. However, because there is generally a trade-off between resolution and field of view (FOV), especially for a miniature device, this microscope has a limited FOV of ~350-by-350 μm.

Our real-time video-mosaicking method takes raw video frames, performs real-time feature-based image registration, and then achieves image stitching (Figure 2). When imaging starts, image frames are generated by an FPGA-based frame grabber, as described previously [2], [29]. These image frames, which are generated at 20 – 30 frames/sec in this study (400 pixels by 400 pixels per frame), are streamed to a PC via a National Instruments frame grabber (NI PCIe-1473) [2], [29]. The C/C++ application that we developed, which is based on the Qt platform, performs sequential image registration on adjacent image frames (the source code can be provided upon request). For every two adjacent frames, the image registration algorithm generates a homography matrix to describe the two-dimensional spatial relationship between the two adjacent frames.

Figure 2.

Real-time video mosaicking flowchart and image-registration sub-routines. (a) Main steps of the real-time video mosaicking workflow. (b) Computer-vision-based image registration method, which accommodates image translations, rotations and deformations.

As shown in Figure 2b, the image-registration procedure consists of four sub-routines: 1) feature extraction, 2) feature matching, 3) filtered random sample consensus (RANSAC) to generate a homography matrix, and 4) image transformation, warping and stitching. The “speeded-up robust features (SURF)” algorithm is used for feature extraction.[30] Compared to the method used in a previous post-processing mosaicking method for feature detection and description, called “scale-invariant feature transform (SIFT),” the SURF algorithm is significantly faster.[30], [31] Despite this increase in speed, the registration speed of SURF, running on a CPU (Intel, i7–8700), still cannot match our desired real-time frame rate (up to 30 frames/sec). Therefore, a parallel-computing version of SURF (called from the OpenCV library) running on a standard GPU (Nvidia, GTX 1080) was implemented so that > 30 frames can be registered in one second. Further improvements to image registration speed can be achieved with the use of more-powerful CPUs and GPUs in the future.

Once key features are identified/generated for each image frame, the key features for every pair of adjacent frames are stored in the GPU memory, such that a feature-matching procedure can be used to identify common features shared between adjacent frames. For this, we use the “fast library for approximate nearest neighbors (FLANN)” method,[32] which is also a parallel computing algorithm called from OpenCV, which runs on the GPU. FLANN rapidly finds the most similar features in adjacent frames, and links them together as matched pairs.

Based on the matched pairs generated by FLANN, the “random sample consensus (RANSAC)” algorithm, with default settings from the OpenCV library, generates a homography matrix to describe the spatial relationship between two adjacent frames (which accounts for rotations, translations, and deformations). To improve the accuracy of image registration, a method was developed to reduce matching errors and converge upon an optimal homography matrix. Here we assumed that the frame rate of our handheld imaging device is sufficiently high such that rotational transformations are relatively small between adjacent frames (e.g. < 5 deg). Under this low-angle assumption, the majority of matched feature pairs should have approximately the same displacement (magnitude). As shown in Figure 2b, the matched pairs of features corresponding to the most common spatial displacement are extracted in our “filtered RANSAC” procedure so that the proportion of incorrect matches is reduced prior to computation of a final homography matrix. This filter helps to increase the accuracy of image registration, especially for low-contrast or noisy datasets.

Image-registration failures can occur when there are severe motion artifacts within single frames or when there is insufficient overlap between two adjacent images (~80% or less). A set of thresholds are used to check if there are sufficient matched features between frames (~30 correct matches or more), or if the resulting translation and rotation coordinates are within a reasonable range (< 50 microns, < 5 deg). These thresholds are empirical and can be adjusted for different imaging devices and imaging conditions. If image registration fails, the mosaicking algorithm pauses until a new frame can be correctly registered to the last stitched frame. For the tissues imaged in this study, which were stained with acridine orange (e.g. Fig. 3 & 4), the SURF algorithm would typically identify 500 – 700 features, from which FLANN would identify 150 – 300 matched feature pairs, of which 30 – 150 features would be used by the filtered RANSAC routine to generate a homography matrix.

Figure 3.

Users were tasked with imaging a 3 × 2 mm tissue region within a 2 min time frame. (a) A stitched image collected without real-time image mosaicking. The red box indicates the targeted imaging region (3 × 2 mm). Scale bar: 500 μm (b) The white areas indicate the imaged regions, and the dashed lines depict the trajectory of the device over time. (c) A stitched image collected with the aid of real-time image mosaicking. Scale bar: 500 μm (d) The white areas indicate the imaged regions, and the dashed lines depict the trajectory of the device. (e) A box plot of the coverage percentage for a 3 × 2 mm tissue region imaged both with and without real-time mosaicking guidance (over a 2-min imaging duration).

Figure 4.

(a) Post-processed image mosaic of fluorescently labeled (acridine orange) mouse kidney imaged with a handheld line-scanned dual-axis confocal microscope. The three images on the right are raw image frames corresponding to the highlighted areas. (b) A corresponding H&E histology image of mouse kidney.

With a handheld contact-based optical-sectioning microscope, some level of tissue deformation and variation in imaging depth is unavoidable as contact pressures and angles are varied over time. Our image mosaicking algorithm is not designed to stitch images in 3D based on images obtained at different tissue depths. However, at high frame rates, there is often sufficient similarity between adjacent frames such that mosaicking can be performed while the imaging depth is slowly changed. Note that this implies that our mosaicking algorithm is not globally optimized, nor does it account for non-rigid deformations. Therefore, positioning errors will inevitably increase as a function of distance traveled (errors accumulate at a rate of ~1% of the total distance traveled, based on experimental observations). Once the RANSAC procedure is complete, images are overlaid based on the homography matrices with no averaging or blending between frames (Figure 3). Since the goal of our method is to provide a real-time guide to the user for efficient sampling of large imaging fields, the image quality of the real-time mosaic is not a priority. If a high-quality mosaicked image is desired, post-processing routines can be used in which images are seamlessly blended together (Figure 4).

For the practical implementation of our methods, we designed a LabVIEW-based user interface for visualization of raw frames (20 – 30 frames/sec in this study), as well as a mosaicked image that grows in real time as the microscope is translated slowly across a tissue surface. The supplemental video demonstrates the real-time video mosaicking for guiding handheld microscopy (see supplemental video 1). As mentioned previously, the primary purpose of this real-time mosaicking method is not to generate high-quality images for diagnostic purposes, but rather to guide users to efficiently survey a large area of the tissue or to locate a desired sub-region of the tissue.

3. RESULTS

A pilot study was performed to demonstrate that the use of real-time mosaicking enables significantly more-efficient spatial coverage when performing handheld imaging with a miniature dual-axis confocal microscope (20 frames/sec, FOV = 350 × 350 μm). For this small-scale study, users were tasked with the challenge of comprehensively imaging a 3-by-2-mm area of fluorescently labeled (acridine orange) mouse kidney tissue within 2 min. Mouse kidneys were freshly excised from euthanized mice obtained through the animal donation program at the University of Washington. This task was performed both with and without the assistance of video mosaicking and was repeated ten times for each condition. To maximize scanning efficiency and to minimize redundancy, users were advised to follow a raster-scanning pattern when moving the handheld device across the tissue. However, without any real-time feedback, scanning paths were difficult to control, and users could not determine how far they had traveled in any direction, therefore leading them to image too far or not far enough. With real-time video mosaicking, users could see how far they had traveled with the device (i.e. a distance scale bar was provided in the LabVIEW-based graphical user interface). Users could therefore more-easily scan the tissue with minimal redundancy and high efficiency. The ultimate performance metric is “coverage percentage,” which refers to the percentage of the 3 × 2 mm tissue region that was sampled with the device within the allotted 2-min time frame.

Figure 3a shows one example of a stitched image collected without the guidance of real-time image mosaicking. Since users were unable to know how far they scanned, they often scanned further than needed. Figure 3b shows that large regions within the targeted 3 × 2 mm region were missed (black areas). The dashed lines in Figure 3b show the approximate trajectory of the handheld device over time. As a comparison, Figure 3c shows one example of a stitched image collected with the aid of real-time image mosaicking. Figure 3d shows that most of the targeted 3 × 2 mm region was imaged (the white areas). The dashed trajectory indicates that the user could optimize their motions to follow an efficient raster-scanning pattern with minimal redundancy or overshooting. A box plot of the percentage coverage, both with and without real-time mosaicking, is shown in Figure 3e. The median coverage rate with real-time video mosaicking was close to 95%, with all imaging attempts yielding a coverage percentage of at least 88%. As a comparison, with no real-time guidance, the median coverage percentage was ~70% with a large variation in coverage rates ranging from 55% to 75%. These results show that users can efficiently and fully image a large area of tissue with the assistance of a real-time video mosaicking method.

Finally, to show that the data collected from our study can be used to generate a high-quality stitched image (with post processing), an ImageJ plugin [33] was used. Figure 4a shows a post-processed image of a mouse kidney specimen used in our pilot study (same dataset as shown in Figures 3c & 3d).

4. CONCLUSION AND DISSCUSSION

In summary, we implemented a feature-based video mosaicking method to guide users of handheld microscopes as they seek to survey large areas of tissue efficiently and comprehensively. To further improve these methods, higher frame rates and/or larger fields of view would be helpful to ensure high levels of overlap between frames even if the device is translated more quickly across the tissue surface (the current speed limitation with our device is ~1.4 mm/s to ensure ~80% overlap between adjacent frames, as needed for reliable mosaicking). However, higher frame rates and larger images will demand faster algorithms and CPUs/GPUs for real-time computation. Currently, feature-based mosaicking methods will always fail if users move the device too rapidly, or if tissue contact is lost. In the future, adding position sensors with sub-millimeter accuracy could provide a means for guiding users back to a precise tissue location for resumed mosaicking. Such position sensors could also correct for spatial errors that accumulate over time with feature-based mosaicking. Such a strategy could be combined with global mosaicking methods,[15], [22], [25] in which image frames are registered not only with adjacent frames in time, but also with adjacent frames in space (acquired in the past) to continuously correct for accumulated spatial errors and tissue deformations. Nonetheless, we have shown through a pilot study that even the most rudimentary real-time mosaicking method can have a large impact on the usability and efficiency of a clinical handheld microscope for in vivo disease detection and surgical guidance.