Video-Mosaicing of Reflectance Confocal Images For Rapid Examination of Large Areas of Skin In Vivo

Abstract

Background

With reflectance confocal microscopy (RCM) imaging, skin cancers can be diagnosed in vivo and margins detected to guide treatment. Since the field of view of an RCM image is much smaller than the typical size of lesions, mosaicing approaches have been developed to display larger areas of skin. However, the current paradigm for RCM mosaicing in vivo is limited both in speed and to pre-selected rectangular-shaped small areas. Another approach, called “video-mosaicing,” enables higher speeds and real-time operator-selected areas of any size and shape, and will be more useful for RCM examination of skin in vivo.

Objectives

To demonstrate the feasibility and clinical potential of video-mosaicing of RCM images to rapidly display large areas of skin in vivo.

Methods

Thirteen videos of benign lesions, melanocytic cancers and residual basal cell carcinoma margins were collected on volunteer subjects with a handheld RCM scanner. The images from each video were processed and stitched into mosaics to display the entire area that was imaged.

Results

Acquisition of RCM videos covering 5.0–16.0 mm² was performed in 20–60 seconds. The video-mosaics were visually determined to be of high quality for resolution, contrast and seamless contiguity, and the appearance of cellular-level and morphologic detail.

Conclusion

Video-mosaicing confocal microscopy, with real-time operator-choice of the shape and size of the area to be imaged, will enable rapid examination of large areas of skin in vivo. This approach may further advance noninvasive detection of skin cancer and, eventually, facilitate wider adoption of RCM imaging in the clinic.

With cellular-level resolution comparable to histology, reflectance confocal microscopy (RCM) imaging is a promising approach both for diagnosis of skin cancer in vivo with high sensitivity and specificity^1,2, and for pre- and intra-operative detection of cancer margins to guide treatment.^3–5 However, RCM images are limited to a field of view (FOV) of 1 mm -by- 1 mm, much smaller than the typical size of skin lesions. Many diagnostic features cannot be reliably identified in such small FOVs. Moreover, clinicians rely heavily on visual context of the surrounding tissue to perform diagnoses. Thus, larger areas must be imaged to evaluate cellular and morphologic features with high accuracy and repeatability. To address this concern, mosaicing approaches, which increase the FOV by acquiring a matrix of adjacent images and stitching them together to display a large area, have been developed for confocal microscopy⁶.

In standard mosaicing, images are acquired while mechanically translating the microscope lens relative to the skin along pre-determined linear (straight-line) trajectories. This approach was implemented in the RCM scanner used in the cited studies^1–5, and, in fact, is now routinely used on patients. However, the mechanics of translation limit speed and coverage to pre-selected small rectangular-shaped areas, currently up to 8 mm-by-8 mm, imaged in ~4.5 minutes. Coverage and speed could be increased, of course, with larger and faster mechanical translation systems, but would add significant size and cost to RCM scanners, and would certainly not be practical for routine use on patients.

Miniaturized confocal endoscopes have been developed that allow the operator flexible control for imaging in vivo, without the constraints of mechanical translation^7,8 Similar flexibility is now possible for imaging skin with the recent advent of smaller and miniaturized handheld confocal microscopes^9,10,11. The operator manually moves the microscope along a desired curvilinear trajectory, with the lens gently pressed against the tissue, while acquiring a video sequence of images. Video microscopy enables the operator to choose the trajectory in real-time, allowing adaptive coverage of areas that can be selected in real-time during acquisition. Thus, an area with any shape and size may be rapidly imaged, without the previous constraints of straight-line trajectories and rectangular coverage. However, observing a video, by itself, merely as a time-sequence of small FOVs, does not readily provide the necessary visual context from the surrounding tissue.

In this paper, we present results from an approach for computationally transforming such videos into mosaics that display the entire imaged area. Algorithms for video-mosaicing have been developed in the fields of computational photography and computer vision¹², and their use has previously been reported for confocal endoscopic imaging^7,8.We report here application of video-mosaicing to reflectance confocal images of human skin lesions and margins in vivo.

Methods

We used a newly developed handheld RCM scanner (Vivascope 3000, Caliber Imaging & Diagnostics Inc., Rochester, NY, U.S.A.) to capture videos. The videos were acquired on subjects, under an IRB-approved protocol, by an expert clinician (co-author MC) with experience using this microscope. Imaging was performed with a 30×, 0.9 numerical aperture lens, with optical sectioning of ~3 μm and resolution of ~1 μm. Each frame (image) of the video is composed of 1000 × 1000 pixels and covers a FOV of 1 mm-by-1 mm. The imaging rate was ~8 frames/second.

Thirteen RCM videos (6 benign lesions, 2 melanocytic cancers and 5 residual basal cell carcinoma (BCC) margins) were acquired in vivo. Imaging straddled the dermo-epidermal junction, which is a key structure for cancer diagnosis and detection of margins. During capture, the operator manually moved the microscope slowly and smoothly, with the lens pressed gently against the skin, traversing a trajectory chosen in real-time on the area of interest, while attempting to avoid sudden “jumps” or discontinuities in the video.

The individual frames of the video were extracted after acquisition and identification tags were automatically cropped using an image processing algorithm written in MATLAB (Mathworks, Natick, MA, U. S. A.). The cropped frames were then stitched using freely available video-mosaicing software (Microsoft Image Composite Editor (ICE); http://research.microsoft.com/en-us/um/redmond/groups/ivm/ICE/). When the operator’s movement was sufficiently smooth, the entire sequence was processed in one batch. When there were “jumps”, we divided the video into smaller “sub-videos” separated by the discontinuities, and processed them individually, to create smaller mosaics of each corresponding area.

Upon request, we will freely provide our MATLAB executable software (with documentation) that can be used to extract the video frames. This software may be used in conjunction Vivascope 3000 videos and the ICE software (URL link above). We will also distribute our raw videos and video-mosaics, if requested.

Results

We present images of three example video-mosaics of a benign lesion, residual BCC margins and a lentigo maligna (Figures 1–3 respectively). Further details are in the figure captions. We have also included 6 video-mosaics of other skin lesions in the supplementary material.

Figure 1.

Video mosaic of a benign melanocytic nevus over a 9.4 mm² area, acquired in 50 seconds (~400 frames). Imaging was in the lower epidermis and at the dermo-epidermal junction. Features seen include normal skin (outside green boundary), nevus region (inside green boundary), hair follicle (yellow asterisks), and rings of dermal papillae (examples seen in the red ellipses).

Figure 3.

Video mosaic of a lentigo maligna over a 6.5 mm² area, acquired in 1 minute (500 frames). Atypical perifollicular infiltration (yellow ellipse), cluster of atypical cells above the dermal-epidermal junction (orange ellipse), hair follicle (yellow asterisks), and epidermal disarray (red ellipse) can be observed.

Area coverage rate of the thirteen video mosaics ranged from ~5 to ~24 mm² per minute, depending on the operator’s rate of motion. When the acquisition was smooth, conversion of a 1-minute long video (~500 frames) into a mosaic took 10–30 minutes on a standard desktop computer, without any special software optimization. The computation time heavily depended on the complexity of the image features and the motion in the video.

Discussion

Compared to the standard paradigm (linear trajectories, rectangular areas) currently employed in RCM mosaicing, video-mosaicing enables real-time operator-controlled curvilinear trajectories that allow observation of larger areas of any desired shape and size. Our results demonstrate that video-mosaicing confocal microscopy is a promising tool to further advance noninvasive detection of skin cancer.

For high quality stitching, the overlap between consecutive frames should be at least 25–50%. With a FOV of 1 mm² and imaging rate of ~8 frames/second, the rate of area coverage could, in principle, reach~240–360 mm²/min. Thus video-mosaicing can, in principle, be much faster than current RCM mosaicing, which typically images at ~14 mm²/min. However, in this study, we imaged at a rate comparable to current mosaicing, which is one to two orders of magnitude smaller than the theoretical limit, suggesting that the microscope was moved too slowly. This reflects an important consideration, due to the currently manual approach for video acquisition: optimizing the rate of microscope movement while avoiding “jumps” and keeping the focus on the desired plane. (The location and depth of the imaging plane is affected by the position and orientation of the microscope lens relative to the skin surface, and is sensitive to variations in manual pressure as well as surface topography.) Lack of sufficient control of both may lead to poor registration and blur in the mosaics. Thus, user skill is important to obtain good results. However, our experience suggests that an untrained user can develop the necessary skills after imaging ~10 lesions under the supervision of an experienced clinician.

Based on this experience, we believe that RCM video-mosaicing should be advanced to clinical testing. In the long-term, a smaller or miniaturized RCM scanner, with fully-automated (robotic) or semi-automated (for example, touch-screen) control on the motion and trajectory, may be the best approach. Our results suggest that, with better control, a considerably higher imaging rate may be achieved. Moreover, the algorithms may be further developed to account for variations in image complexity, features and quality, to automatically segment, or possibly repair, “jumps” or discontinuities in the videos, and to optimally tradeoff between resolution, blur and noise when averaging overlapping images to create the mosaics, and speed. With such advances, video-mosaicing confocal microscopy could become a critical tool for routine use in the clinic, and may facilitate wider clinical adoption of RCM imaging.

Figure 2.

Video mosaic of residual BCC margins, obtained intra-operatively in the surgical wound on a Mohs patient, over an area of 15.8 mm², acquired in 40 seconds (~400 frames). Structures seen include dark nuclei within the honeycomb pattern of keratinocytes in the surrounding intact epidermis (outside blue boundary, appears saturated here), exposed epidermis in the wound (between blue and yellow boundaries), in which the nuclei (examples in orange oval) appear bright due to the application of aluminum chloride for contrast enhancement¹³, and the underlying papillary and reticular dermis (inside the yellow boundary). Other structures, such as collagen bundles in dermis (for example, inside orange square), rings of dermal papillae (for example, red ellipses) and hair shaft (green ellipse) can be seen in the mosaic.

What’s already known about this topic?

• RCM imaging can detect skin cancers in vivo, but is limited by small fields of view;

• Mosaicing enables imaging over larger areas, but is presently slow and limited to small pre-determined rectangular-shaped areas;

• In other types of imaging, video-mosaicing can be applied to user-selected areas of any shape and size.

What does this study add?

• Video mosaicing of RCM images is demonstrated in skin lesions and margins in twelve subjects, with ability to image significant skin structures over a larger irregularly-shaped region documented.

• Considerations of speed and imaging quality are examined.