Fast, large area multiphoton exoscope (FLAME) for macroscopic imaging with microscopic resolution of human skin

Abstract

We introduce a compact, fast large area multiphoton exoscope (FLAME) system with enhanced molecular contrast for macroscopic imaging of human skin with microscopic resolution. A versatile imaging platform, FLAME combines optical and mechanical scanning mechanisms with deep learning image restoration to produce depth-resolved images that encompass sub-mm² to cm² scale areas of tissue within minutes and provide means for a comprehensive analysis of live or resected thick human skin tissue. The FLAME imaging platform, which expands on a design recently introduced by our group, also features time-resolved single photon counting detection to uniquely allow fast discrimination and 3D virtual staining of melanin. We demonstrate its performance and utility by fast ex vivo and in vivo imaging of human skin. With the ability to provide rapid access to depth resolved images of skin over cm² area and to generate 3D distribution maps of key sub-cellular skin components such as melanocytic dendrites and melanin, FLAME is ready to be translated into a clinical imaging tool for enhancing diagnosis accuracy, guiding therapy and understanding skin biology.

Introduction

Non-invasive clinical skin imaging with molecular contrast and high spatial resolution is important for maximizing diagnostic efficacy, guiding therapy and advancing the understanding of skin biology. The significance of technologies capable of 3D non-invasive imaging in living skin tissue is underscored by the continuous growth of in vivo Reflectance Confocal Microscopy (RCM) and more recently of the Optical Coherence Tomography (OCT) and multiphoton microscopy (MPM) in the dermatology field. RCM and OCT contrasts are based on variations in tissue refractive indices, which provide gray scale images. Gray scale images are generally sufficient for the purpose of diagnosis based on tissue morphology assessment. Therefore the RCM potential has been demonstrated for a broad range of applications in the dermatology field including non-invasive diagnosis of melanoma^1,2 and non-melanoma skin cancer^3,4. OCT, which generates images with lower spatial resolution than RCM and MPM, has the benefit of enhanced imaging speed, field of view and penetration depth. In the dermatology field, OCT has been proven mostly effective for non-invasive diagnosis of non-melanoma skin cancer^5,6. The parameters of interests (spatial resolution, scanning speed, field-of-view, contrast mechanism) of the existing commercial systems for label-free clinical skin imaging based on RCM (VivaScope, Caliber ID) and OCT (VivoSight, Michelson Diagnostics) are included in Table S1, Supplementary Information, along with those for the commercial MPM clinical system (MPTflex, Jenlab) and the proposed FLAME imaging platform. MPM is a nonlinear optical imaging technique that, compared to RCM and OCT, provides unique structural and molecular contrast based on endogenous signals such as two-photon excited fluorescence (TPEF) from NAD(P)H, FAD + , melanin, keratin and elastin fibers, and second harmonic generation (SHG) from collagen^7,8. The fluorescence lifetimes of these fluorophores, if detected, can further improve the molecular contrast^9,10. Therefore, based on its contrast mechanism, MPM provides color images that closely resemble the histological sections dermatopathologists use for diagnosis, a significant benefit, facilitating accurate image interpretation. Furthermore, the molecular contrast of MPM images uniquely allows non-invasive, label-free evaluation of cellular metabolism in human skin^11,12 or of skin conditions such as elastosis¹³. It also facilitates development of quantitative analyses strategies based on the TPEF and SHG signals^14–16. However, comparing to RCM and OCT, MPM involves a more complex technology to adapt to bedside. The only MPM commercial platform optimized for clinic use (MPTflex, Jenlab, Germany) has played a critical role in establishing the clinical potential of MPM for diagnosis of skin cancer^{12,14,17–20}, pigmentary skin disorders¹³ and alopecia²¹, characterization and understanding of skin pigment biology²² and keratinocytes metabolism¹¹ or monitoring the effects of cosmetic treatments²³. While these studies demonstrate the unique clinical potential of this technology, its routine implementation into clinical practice is hindered by several technological barriers. The main challenges are related to limited scanning area and speed. The capability of imaging large areas is critical for improving diagnostic accuracy, particularly for non-uniform pigmented lesions, such as melanoma arising in pre-existing nevus²⁴. Melanocytes that may develop into melanoma in these lesions may be easily missed by imaging of limited areas. False-negative findings delay diagnosis and compromise treatment efficacy. Enhancing imaging speed is required in order to improve the efficiency of the imaging procedure, minimize motion artifacts and optimize clinical work-flow. The scientific research community has been interested in MPM rapid imaging of large tissue areas with sub-micron resolution for various applications. One area of interest includes employing MPM for rapid intraoperative assessment of thick excision specimens^25–27. Commercial MPM microscopes can provide millimeter-to-centimeter scale sub-cellular resolution images acquired within minutes by detecting exogenous fluorescence signals, as recently demonstrated for breast²⁷, prostate²⁵ and skin tissue specimens^26,28. A similar performance has been reported by several research groups in the neuroscience field, who developed customized MPM platforms for imaging mouse brain vasculature²⁹, cortical tissue and neural activity^30–32 or for in vivo brain imaging of genetically modified mouse models³³. All the aforementioned applications involve detection of exogenous fluorescence signals. The challenge of optimizing the MPM for in vivo MPM imaging of human skin, our application of interest, is to reach a similar performance of the MPM system in terms of scanning area and speed while detecting weak endogenous signals such as those from skin fluorophores.

Successful clinical translation of the MPM technology for in vivo skin imaging applications requires optimization of the microscope complexity, cost and footprint in addition to significant improvement of scanning area, speed and contrast, without compromising spatial resolution. We have recently demonstrated a bench-top prototype of an MPM imaging system based on TPEF and SHG signals detection, optimized to image tissue areas of ~ 0.8 × 0.8 mm² at speeds of less than 2 s per 1 MPx frame for high SNR, with lateral and axial resolutions of 0.5 μm and 3.3 μm, respectively³⁴. In this manuscript, we present significant advances of this imaging platform, based on combining optical and mechanical scanning mechanisms with image restoration neural network computational approaches^35,36 to allow millimeter-to-centimeter scale imaging within minutes, while maintaining sub-micron resolution. The main challenge of implementing a fast-scanning approach with MPM is related to the weak endogenous fluorescence signals from molecular components of skin. We have adopted a recently validated deep learning approach, namely content-aware image restoration (CARE)^35,37,38, which represents a key solution for improving the SNR of images acquired by detecting a limited number of photons due to the high imaging speed. The proposed compact, fast large area multiphoton exoscope (FLAME) system also features time-resolved single photon counting (SPC) detection with sufficient temporal resolution to distinguish fluorophores, such as melanin, which facilitates its quantitative assessment, important in the diagnosis and the treatment evaluation of many skin conditions. A key component of this imaging platform is the light source, a femtosecond fiber laser, which has a small footprint that allows its integration into the imaging head. This allowed us to reduce the footprint of the FLAME platform 20 × compared to our previously reported bench-top prototype. We demonstrate the performance of the FLAME system by ex vivo and in vivo imaging of human skin.

Results

Main advances of the FLAME imaging platform

Key advances of the FLAME imaging platform with respect to the bench-top prototype previously reported by our group³⁴ include: (1) significant enhancement (20 ×) of the imaging system compactness; (2) increase in the scanning speed over centimeter-scale tissue areas; (3) enhancement of the molecular contrast through specific detection of melanin. The applications enabled by these advancements, along with a brief description of the imaging platform, are presented below. The hardware and software components involved in the FLAME system development are described in the “Materials and methods” section.

Enhancing compactness and portability

The FLAME imaging platform and the layout of its imaging head are presented in Fig. 1. The system is based on a customized optical and opto-mechanical design previously reported by our group^34,39 and summarized in the “Materials and methods” section. A key component of the current design is related to the excitation light source, a frequency doubled Yb-doped amplified fiber laser (Carmel X-series, Calmar Laser, USA) that generates 90 fs pulses of 780 nm light at 80 MHz and linear p-polarization with a maximum output power of 500 mW. This excitation laser source has replaced the bulky Ti: Sapphire laser (61 × 37 × 19 cm³ laser head; 39 × 27 × 39 cm³ chiller) employed for excitation in our previously reported MPM platform³⁴. The benefit of using this light source is that the laser head has a compact size (9 × 18 × 3.5 cm³), which facilitates its incorporation into the imaging scan head (Fig. 1), allowing for enhanced compactness (35 × 35 × 20 cm³) and portability of the exoscope.

Figure 1.

The FLAME system: overview (left) and layout of the scan head (right). The scan head (a) is supported by an articulated mechanical arm (b) attached to the cart (c) that houses the electronic controllers and holds the computer.

Enhancing imaging speed by utilizing image restoration neural network

We accomplish the task of extending the rapid imaging with sub-micron resolution from sub-millimeter to centimeter scale by employing computational approaches along with combined optical and mechanical scanning controlled by optimized data acquisition software. The computational approaches, described in detail in the “Materials and methods” section, involve the use of content-aware image restoration (CARE) in order to compensate for the limited number of photons detected due to the high imaging speed. Figure 2 shows representative examples of TPEF images of keratinocytes in human skin, acquired with the FLAME system and processed with CARE, a neural network model we trained on pairs of TPEF images acquired at fast and slow frame rates in human skin tissue as described in the “Materials and methods” section. The 1 MPx images of 0.27 × 0.27 mm² (Fig. 2a) and 0.9 × 0.9 mm² (Fig. 2c) were acquired by accumulating 15 consecutive frames at an effective rate of 2 s/frame. The corresponding images predicted by the trained CARE network model (Fig. 1b,d) show a significant enhancement of the cellular contrast, allowing for clear delineation of nuclei as shown by the line profiles (Fig. 2e,f) through adjacent keratinocytes in the images of Fig. 2.

Figure 2.

Performance of 2D CARE image restoration. Trained model is applied to images acquired by accumulating 15 consecutive frames of 0.27 × 0.27 mm² (a) and 0.9 × 0.9 mm² (c) image sequences (both 1 MPx, 2 s acquisition time). Restored images (b,d) display decreased noise levels and higher contrast for both fields of view. (e,f) line profiles through adjacent keratinocytes corresponding to (a,b) and (c,d) respectively (yellow lines). The line intensity profiles in the network prediction images (red) show enhanced definition of cell nuclei (black arrows) compared to the low cellular contrast to background in the raw images (blue). The insets in (g) and (h) are close-ups of the blue and red areas outlined in (c) and (d), respectively.

The content-aware image restoration has highest impact when applied to multi-millimeter scale mosaic images as it maintains/restores acceptable image contrast while using fewer photons than the current state-of-the-art commercial clinical MPM system, MPTflex by JenLab, due to increase in imaging speed (2 s/mm² vs 96 s/mm², Table S1). This approach provides two key benefits by reducing the acquisition time as well as the risk for potential photodamage by minimizing the exposure time.

Rapid, multi-scale, high-resolution imaging by utilizing optical and mechanical scanning along with image restoration

For a comprehensive interrogation of the skin tissue we have devised a multiscale scanning approach that utilizes three complementary scanning modalities (strip mosaic, tile mosaic and volumetric imaging) as described in the “Materials and methods” section. We use these combined optical and mechanical scanning mechanisms in order to mimic the current examination method for histological tissue sections, where the pathologist examines the entire tissue at low magnification and then zooms in to investigate certain regions of interest (ROI).

Thus, in our imaging procedure, the first step of the tissue examination involves a survey scan on the centimeter scale at about 3 different depths within the epidermis, dermo-epidermal junction (DEJ) and papillary dermis. We employ the single frame strip mosaic scanning mechanism at this spatial scale as it provides significantly faster acquisition rate than the tile mosaic scheme (see “Materials and methods” section). Figure 3 presents a cm-scale MPM image before (Fig. 3a) and after skin-trained CARE network application (Fig. 3b), acquired ex-vivo as a single frame at a 30 μm depth within the epidermis of a human skin tissue. The acquisition and restoration time for the image in Fig. 3 (80 MPx, 1.2 × 1.0 cm²) was 2 min 15 s. The dynamic range enhancement obtained from neural network restoration allows for trivial field curvature correction (Fig. 3b), not feasible for low SNR raw images (Fig. 3a). The insets represent the digital zoom-in to a 200 × 200 μm² area within the large-scale MPM images, showing that image restoration enhances contrast to allow visualization of keratinocytes nuclei. The slight blur appearing in this image is due to the limited digital resolution the original image was acquired at (1.2 × 1.2 μm² pixel size, see Table S2) and the single frame acquisition rate that results in a low SNR image, sub-optimal for the CARE application model. Nonetheless, the images in Fig. 3 demonstrate that, by using a strip mosaic scanning mechanism and applying a trained neural network model, large areas of skin tissue can be rapidly mapped with sufficiently high resolution and contrast to allow visualization of nuclear and cellular morphology, required to identify ROIs for certain skin conditions.

Figure 3.

Rapid, centimeter-scale MPM ex vivo imaging of freshly excised facial skin at sub-micron resolution. MPM overview map (80 MPx, 1.2 × 1.0 cm²) of a centimeter-scale skin tissue (a) prior and (b) post CARE processing, acquired as a single frame by strip mosaic scanning at 30 μm depth in the epidermis. The insets represent digital zoom-in of the 200 × 200 μm² outlined areas in (a) and (b). The intensity profiles (red) through adjacent keratinocytes show that image restoration enhances contrast to allow visualization of keratinocyte nuclei.

However, a comprehensive, detailed morphological assessment that allows visualization of microscopic features of interest, such as melanocytic dendrites, requires enhanced resolution and contrast. We accomplish this by zooming in optically and acquire images over millimeter scale tissue areas by using a tile mosaic scanning mechanism. The high resolution and contrast of these mosaic images allow for digitally zooming in to identify architectural features and ROI selection for further volumetric scanning. A movie included in the Supplementary Information (Visualization 1) illustrates the process of navigating a high resolution tile mosaic with digital zoom and lateral browsing highlighting micron-sized melanocytic dendrites that are difficult to visualize in the fully zoomed out images.

Figure 4 illustrates the high-resolution, rapid complete mapping of a centimeter to sub-millimeter scale large human skin tissue by combining optical scanning and the two mechanical scanning mechanisms as described in “Materials and methods”, followed by applying the trained CARE neural network. Close-up inspection of the features can be performed by zooming in digitally (Fig. 4d) or optically (Fig. 4e).

Figure 4.

Rapid, multiscale MPM ex vivo imaging of freshly excised facial skin at sub-micron resolution. (a) Photographic image of the excised facial skin. (b) MPM map overview of the skin tissue epidermis generated by strip-mosaic scanning; the 80 MPx image was acquired at 25 μm depth as a single frame over a 1.2 × 1.0 cm² area and restored with CARE network in 2.15 min. (c) Tile mosaic image of the orange outlined area in (a) and (b) covering 4.5 × 4.5 mm² (25 MPx, acquired and restored with CARE in 2.5 min) with hairs outlined by dashed circles. (d) Digital zoom in of the 1 × 1 mm² yellow outlined area in (c) showing a close-up of the skin folds, hairs and keratinocytes within the epidermis. (e) 1 MPx image acquired in 2 s by optically zooming into the 400 × 400 μm² blue outlined area in (d). The image shows well-resolved keratinocytes surrounding a hair follicle. (f) Optical zoom in at higher rate sampling 1 MPx, 200 × 200 μm² image allows visualization of intracellular features and protective melanin rings around the nuclei of keratinocytes (200 × 200 μm², 1 MPx image acquired in 2 s).

Volumetric imaging within selected ROIs is required for in-detail assessment of the morphological changes at different depths. Z-stacks are typically acquired as a sequence of 1 MPx en face images beneath the skin surface, encompassing volumes of about 900 × 900 × 150 μm². A z-stack of 15-accumulated frames for each image, sampling a 900 × 900 × 150 μm² volume every 5 μm, is acquired in 60 s and when needed, restored in 20 s. MPM images acquired ex vivo as a z-stack at different depths in human skin tissue are included in Supplementary Information (Visualization 2).

Enhancing molecular image contrast by time-resolved single photon counting (SPC)

Based on the assignment of the time bin integration in the fluorescence detection channel in our exoscope, as described in “Materials and methods”, melanin is predominantly detected in the corrected red channel, while fluorescence signals from other fluorophores such as keratin, NAD(P)H/FAD and elastin, are mainly detected in the green channel. Figure 5 presents a side-by-side comparison between MPM and histology images of human skin that demonstrate the co-localization of the fluorescence signal in the corrected red channel and the melanin specific stain (Fontana-Masson).

Figure 5.

Side-by-side comparison of MPM and histological images. (a) time-resolved SPC MPM image of an unstained thin section of formalin-fixed normal human skin acquired with the FLAME system; (b) Histological image obtained by Fontana-Masson staining of a human skin section from the same sample and adjacent to the one shown in (a). The images show that the fluorescence signal in the corrected-red channel of the MPM image (a) is co-localized with the Fontana-Masson melanin-specific stain (black) in the histological image (b).

This approach allows us to enhance the molecular contrast of the FLAME imaging platform without compromising speed. Thus, the time-resolved SPC images and the time-integrated intensity-based images require the same time for acquisition and restoration. Figure 6 illustrates millimeter-scale, sub-micron resolution MPM images based on time-resolved SPC, acquired in a normal pigmented human skin tissue at different depths.

Figure 6.

Rapid, millimeter-scale, time-resolved SPC, MPM ex vivo imaging of fresh scalp tissue. Tile-mosaic images (4.5 × 4.5 mm², 25 MPx) acquired and restored with CARE in 2.5 min at select depths of 30 μm (a), 50 μm (b) and 80 μm (c) corresponding to the epidermis, DEJ and papillary dermis, respectively. Mosaic images are color-coded by short-lifetime TPEF (red), long-lifetime TPEF (green) and SHG (blue). (d,e,f) Digital zoom to the 1.8 × 1.8 mm² white outlined areas in (a), (b) and (c), respectively, showing tissue heterogeneity on mm scale. (g,h,i) Digital zoom to the 450 × 450 μm² white outlined areas in (d), (e) and (f), respectively, showing non-pigmented keratinocytes and elastin fibers predominantly visualized in the green channel, pigmented (melanin-rich) keratinocytes mainly detected in the red channel and collagen fibers visualized through the SHG signal in the blue channel.

Time-resolved SPC volumetric images acquired ex-vivo from pigmented human skin illustrate the ability of the FLAME imaging system to generate 3D distribution maps of melanin (Figure S1, Supplementary Information).

Rapid multi-scale ex vivo MPM imaging of human skin while maintaining sub-micron resolution

While rapid acquisition of MPM images at millimeter-to-centimeter spatial scale in human skin is critical for enhancing the efficacy and clinical utility of the MPM technology, it is essential to accomplish this goal while maintaining the ability to resolve fine cellular structures. Melanocytic dendrites are a relevant example of fine cellular components that need to be resolved in imaging applications related to melanoma diagnosis, therapy guiding of pigmentary skin disorders or characterizing and understanding of pigment biology. The images in Fig. 7 and in the movie Visualization 1 in Supplementary Information demonstrate the ability of the FLAME imaging system to resolve and rapidly map the spatial distribution of the melanocytic dendrites in the full epidermis of a freshly excised actinically damaged facial skin tissue. A z-stack acquired at the same location as the inset of Fig. 6e shows the 3D appearance of the melanocytic dendrites (Visualization 3, Supplementary Information).

Figure 7.

Rapid multi-scale ex vivo MPM imaging with sub-micron resolution to resolve melanocytic dendrites in actinically damaged skin. (a) Photographic image of the excised facial skin tissue. (b) MPM overview map of the skin tissue epidermis generated by strip-mosaic scanning; the 80 MPx image, acquired as a single frame over a 1.2 × 1.0 cm² area and restored with CARE network in 2.15 min, is illustrated as overlapped with the photographic image of the tissue (opacity 30%). (c) High resolution tile mosaic (6.3 × 6.3 mm², 49 MPx) acquired in 3 min with 15 accumulated frames for each tile, at 15 μm below the skin surface. (d,e) Digital zoom into the blue and yellow outlined locations in (c), respectively. Images show the distribution of melanocytic dendrites (arrow heads). (f) A close-up of the melanocytic dendrites (white heads) in the outlined area in (e). A z-stack acquired at the same location as the inset in (e) that shows the 3D appearance of the melanocytic dendrites is included in the Supplementary Information (Visualization 3).

Figure S2 in the Supplementary Information shows an additional example of morphological features which can be identified in human skin lesions based on the sub-micron resolution of the FLAME imaging system. Nests of nevus cells present in a discarded tissue from the excision of a benign compound nevus could be identified in the MPM image and their appearance correlated with histology, as confirmed by a dermatopathologist (R.M.H).

Rapid millimeter-scale in vivo MPM imaging of human skin at sub-micron resolution

The ultimate goal in our effort to develop the FLAME imaging platform is to utilize it as an effective imaging tool for in vivo skin imaging in research and clinical applications that require high spatial resolution and molecular contrast. While the imaging system is a bench-top prototype at the current stage, we demonstrate as a proof-of-concept that in vivo imaging using the FLAME platform is within reach. Representative MPM images with time-resolved SPC contrast acquired in vivo at different depths in a subject’s forearm are presented in Fig. 8.

Figure 8.

Rapid, sub-micron resolution, time-resolved SPC in vivo MPM imaging of human skin forearm. MPM images acquired at different depths in human skin showing the keratinized stratum corneum (a), epidermal keratinocytes (b), pigmented cells surrounding dermal papilla (red) (c) as well as collagen (blue) and elastin fibers (green) in (d).

The corresponding full z-stack of 540 × 540 μm² en face images, acquired from the stratum corneum to the papillary dermis with 5 μm axial sampling can be found in the Visualization 4A, Supplementary Information, along with the corresponding 3D melanin distribution (Visualization 4B, Supplementary Information). The time required for acquiring and restoring 30 1 MPx frames within the z-stack was 80 s. A representative millimeter-scale MPM image (2.7 × 2.7 mm²) acquired in vivo from a subject’s forearm in 45 s, using time-resolved SPC detection, is included in Supplementary Information (Fig. S3). This is a tile-mosaic image we obtained by using a mechanical component we designed and 3D-printed to connect the miniaturized stage to the subjects’ skin.

Discussion

In this manuscript, we introduce a compact imaging platform based on label-free multiphoton microscopy optimized for rapid multi-scale imaging of skin tissue, providing up to sub-micron resolution, high contrast images.

One of the major challenges to overcome when performing label-free multiphoton microscopy at high scanning speed is the reduced image contrast as a result of the limited number of detected photons. We address this challenge by implementing a computational approach for image restoration in addition to our previously developed hardware optimization elements. We implemented the recently demonstrated CARE neural network approach³⁵, which allowed us to enhance the image contrast and extend the scanning area to centimeter scale. Although scanning area and speed are independent parameters, they are closely related in the context of clinical translation of the MPM technology. Imaging of large tissue areas is only valuable and feasible in clinical setting when performed at high scanning rates to minimize motion artifacts and optimize clinical work-flow. We used fast, miniature linear stages and custom data acquisition software design to integrate a strip-mosaic scanning scheme. This approach, in addition to our existing tile-mosaic and volumetric imaging mechanisms, extended the spatial scanning range, allowing for rapid imaging of sub-millimeter to centimeter scale areas. A similar approach has been previously described and implemented as a solution for rapid reflectance confocal imaging, although with a much larger stage suitable for ex vivo imaging^40,41.

An additional challenge to overcome in multiphoton microscopy is maintaining high spatial resolution while optimizing scanning area and speed. The high numerical aperture objective and the custom optical design of our imaging platform provide millimeter-scale images at sub-micron resolution³⁴. The centimeter-scale areas are scanned rapidly at micron resolution, sufficiently high to allow, in concert with the image contrast enhancement provided by the CARE neural network model, visualization of the cellular and fibrillar structures. This performance element is critical in the ability of the exoscope to emulate the current histological examination method, where the pathologist examines the entire tissue at low magnification and then zooms in for careful investigation of selected regions of interest. Although for the work described in this manuscript, the CARE model was trained and applied mainly on normal skin tissue samples, we expect the model to perform well on various skin conditions. While these are characterized by different architectural patterns, the main skin features used for training are the same and the variance in SNR is expected to be influenced mainly by the amount of pigmentation characteristic to the particular skin conditions (melanin is the brightest fluorophore in skin tissue). We have trained the CARE model on skin tissue samples with different degrees of pigmentation (Fitzpatrick skin types I–III).

Molecular contrast is the key benefit of MPM when compared to high-resolution imaging techniques such as reflectance confocal microscopy or optical coherence tomography, currently utilized for clinical skin imaging. The molecular contrast, commonly provided in MPM skin imaging by the intensity of the endogenous fluorescence signals, can be significantly enhanced through approaches such as, fluorescence lifetime imaging (FLIM). FLIM has been demonstrated to be valuable for many applications related to label-free skin imaging in humans⁴² and animal models⁴³. However, the conventional approach, time-correlated single photon counting (TCSPC), although it enables acquisition of fluorescence lifetime decays with high temporal resolution, for fluorophores in skin hundreds of photons per pixel⁴³ are required for accurate lifetime decay curve representation, which leads to long integration times. TCSPC imaging of skin in humans and animal models has been commonly performed at 250 ps temporal resolution and 25 s⁴² to 60 s⁴³ integration time per 256 × 256 pixel frame. These values would translate into hundreds of minutes acquisition time for megapixel size images as required for imaging millimeter-to-centimeter skin tissue areas. One solution we found for enhancing the intensity-based molecular contrast of the FLAME imaging platform without compromising speed, was to integrate time resolved SPC detection with reduced temporal resolution (780 ps) and time bins selected and grouped in such a way to allow selective discrimination of just one key endogenous fluorophore in skin, melanin. This approach is facilitated by the characteristically short fluorescence lifetime of a melanin component^42,44,45, probably eumelanin¹⁰, with respect to the other endogenous fluorophores in skin. Selective detection of melanin is essential in distinguishing dermal melanocytes and melanophages from other cellular components and in applications such as pigmented lesions diagnosis and therapy assessment of pigmentary skin disorders requiring quantitative assessment of melanin.

A key element of the imaging system design is the excitation light source, a compact femtosecond fiber laser that allows its integration into the exoscope imaging head, significantly enhancing the instrument compactness and facilitating its conversion to a portable device.

We demonstrated the exoscope performance by ex vivo and in vivo imaging of human skin. The FLAME platform can provide millimeter to centimeter scale high contrast images of skin in about 2 min with sub-micron and micron resolution, respectively. Depth-resolved images over volumes of 900 × 900 × 150 μm³ are acquired in 1 min. The spatial resolution performance was demonstrated by the ability of the exoscope to visualize melanocytic dendrites in actinically damaged human skin. Melanocytes are identified in MPM imaging by their dendritic presence. If the melanocytic dendrites are not resolved, melanocytes very often cannot be distinguished from pigmented keratinocytes. Identifying the melanocytic presence is essential in applications such as melanoma diagnosis, therapy guiding of pigmentary skin disorders or characterizing and understanding of pigment biology. An additional benefit of using FLAME as an imaging tool for these applications is related to its unique ability to provide in real-time, a volumetric mapping of melanin as demonstrated by the images we acquired ex vivo and in vivo in human skin. Rapid access to a 3D map of melanin distribution in skin is expected to expedite and enhance the accuracy of its quantitative assessment, a particularly important tool for diagnosis and therapy guiding of pigmentary skin disorders¹³ and for characterizing and understanding of pigment biology²².

While these features represent a significant enhancement of the scanning area and speed, molecular contrast and the overall compactness of the FLAME imaging platform, limited penetration depth in scattering tissues such as skin, still remains. Penetration depth can be optimized by using a longer excitation wavelength^46–48, by compensating for dispersion to maintain the laser pulse duration within the tissue⁴⁹ or by adaptive optical correction of aberrations to recover diffraction-limited performance at depth in tissue⁵⁰. Despite this limitation, the current penetration depth of 150–200 μm is sufficient to capture early signs of malignancy in skin that occur at the DEJ^14,17,19, to detect the presence of dermal melanocytes¹⁴ and to identify existing or prior inflammatory response in the papillary dermis^13,21.

In conclusion, we introduce a compact, multiphoton microscopy-based imaging platform, FLAME, highly optimized for rapid, label-free macroscopic imaging of skin with microscopic resolution and high contrast. It has the ability to provide 3D images encompassing sub-millimeter to centimeter scale areas of skin tissue within minutes. It allows fast discrimination and 3D virtual staining of melanin. This unique combination of features provides the FLAME imaging platform with highly optimized functionality and facilitates a seamless conversion to a portable device for rapid, multi-scale clinical skin imaging with high resolution.

Materials and methods

MPM imaging platform

The optical layout as presented in Fig. 1 includes a 4 kHz resonant-galvo scanning mechanism, custom-designed relay and beam expander optics and a 25 × , 1.05 NA objective lens (XLPL25XWMP, Olympus). The optical design of this system was optimized to provide sub-micron resolution images over a field-of-view of 0.8 × 0.8 mm² at speeds of less than 2 s per 1 MPx frame for high SNR ³⁴. The low image contrast caused by the limited number of detected photons due to high scanning speed was maximized by employing sensitive PMTs (Hamamatsu R9880-20 and R9880-210) in photon counting mode and by optimizing the signal collection optics to capture the maximum amount of radiation emitted in the epi-direction³⁴. We use a first dichroic mirror to separate the excitation and the emission signals (FF705-Di01, Semrock, Inc.) and a second dichroic mirror (FF506-Di03, Semrock, Inc.) to split the TPEF and SHG detection channels defined by the emissions filters: FF01-720/SP and FF01-535/150 (Semrock, Inc.) for TPEF; and FF01-375/110 for SHG. These optical elements allow the detection of the SHG signal from collagen and the TPEF signals from NAD(P)H/FAD, keratin, melanin and elastin in skin^8,51,52.

The mechanical scanning for rapid movement of the skin tissue is performed by fast, miniature linear xy stages (Q522.130, Physik Instrumente, GmbH) controlled by a customized data acquisition software (Vidrio Technologies), which synchronizes their travel with the laser beam scanning to facilitate rapid stitching of adjacent scanned areas. We selected this stage based on its performance and its reduced size (3 × 2.1 × 1 cm³), the latter being an essential requirement for its integration into the compact scanning head. The stage can be used as the mechanical scanner component while imaging either ex vivo or in vivo by taking advantage of the skin elasticity. The approach of combining the mechanical and optical scanning to rapidly image sub-millimeter to centimeter scale tissue areas is described in “Multi-scale, high-resolution imaging by optical and mechanical scanning” section.

The data acquisition software and hardware are critical components that affect the performance of the exoscope. Our current design employs the Matlab based software ScanImage (Vidrio Technologies) and data acquisition (DAQ) National Instruments (NI) hardware based on a fast digitizer (NI 5771) controlled by a FlexRIO FPGA module (NI PXIe-7975). The software enables single photon counting (SPC) detection and time-resolved SPC with a ~ 780 ps (16 bins) temporal resolution by using the 80 MHz laser sync signal, upmultiplied 16 × with a clock generator (AD9516-0, Mouser), as an external clock for the fast digitizer. Given the rise time of the PMT (R9880U-20) of 570 ps the temporal resolution is sampling-limited. Hence, detected photons may be binned into virtual output channels based on their arrival time, allowing for selective detection of some fluorophores. Thus, for time-resolved SPC imaging, the signal was detected based on the fluorescence photons arrival time relative to the excitation photons, in time bins of width: -0.4–1.2 ns (red channel) and 0.4–12 ns (green channel). The rationale for defining these time bins was for attaining selective detection of melanin. The fluorescence of melanin, is characterized by short lifetime with respect to most other endogenous fluorophores in skin^42,44,45. Due to the integration time bin overlap of the two detection channels and the modest temporal resolution, subtraction of their intensity (photon counts) signals is required to obtain a differential signature of melanin. Thus, we subtract the photon counts representing the long lifetime fluorescence signal (green channel image) from the short lifetime fluorescence signal (red channel image). The subtraction process yields both positive and negative values. The negative values are due to larger contribution to signal from photons with long fluorescence lifetime. These values are set to zero in the final image resulted from subtraction. The positive photon count values resulted from subtraction are due to larger contribution to signal from photons with short fluorescence lifetime. These values represent mainly the melanin fluorescence signal contribution and the remaining signal in the final image (corrected-red channel) resulted from subtraction. We perform the time-resolved SPC volumetric imaging using 3 channels for simultaneous detection of the SHG signal and of the TPEF signals characterized by the short and long fluorescence lifetime as described above. The 3D distribution of melanin is available almost instantaneous after correcting for signal overlap.

Laser power considerations to avoid potential photodamage

Besides minimizing exposure time through rapid scanning, we also ensure the laser fluence at the sample is below the established thermal⁵³ and DNA⁵⁴ damage threshold for two-photon microscopy of human skin. Thus, based on the focusing optics in our exoscope (NA = 1.05) and the 45 mW excitation laser power at the skin surface, the laser fluence value is lower by at least a factor of 1.7 compared to the fluence values the CE-certified clinical MPM device (MPTflex, JenLab) employs at the skin surface and the fluence values for DNA and thermal damage threshold established for two-photon microscopy of human skin^53,54.

Neural network training

We use content-aware image restoration (CARE)³⁵, recently described as an effective image denoising technique, to restore fluorescence microscopy images acquired with fewer photons for enhancing imaging speed and limit light exposure time. In this method, pairs of images are acquired at low and high SNR for training a sample-specific convolutional neural network by using the high SNR images as ground truth (Supplemental Figure S4). The trained network is then applied to restore low SNR images to predict the high SNR output. The high SNR images, serving as ground truth for training the model, were obtained by accumulating 70 consecutive frames. This number of accumulated frames corresponds to ~ 9 μs effective pixel dwell time, which results in a signal level sufficiently high to cover an 8-bit dynamic range. The model was trained with two inputs—7 and 15 accumulated frames. Using two inputs allowed us to account for two different SNR values generally characteristic to different levels of pigmentation (melanin is the brightest fluorophore in skin with MPM imaging and thus, the amount of pigmentation is generally closely correlated with the image SNR). The two inputs approach was also found to provide robustness to the network performance as described in the original publication by the developers of CARE³⁵. The reason for selecting the values of 7 and 15 as the numbers of accumulated frames for the two inputs was related to practical considerations regarding the need to minimize of imaging speed. The 7 and 15 accumulated frames values correspond to ~ 1 and 2 μs pixel dwell time that results in ~ 1 and 2 min acquisition time, respectively for 20 mm² 5120 × 5120 pixel mosaics. Based on our group’s long time experience in clinical skin imaging with MPM, these time frames would be considered acceptable in clinical setting.

Our training set consisted of 128 × 128 pixel patches (112 × 112 μm²) extracted from 1024 × 1024 pixel images (900 × 900 μm²). The patch size was selected based on the minimum amount of area required to avoid overtraining the model while covering features in skin with different spatial scales, such as cellular features (keratinocytes, melanocytes, corneocytes) on the order of 5–10 μm and tissue features (hairs, skin folds, sweat glands) on average < 100 μm. Quantification of model performance on a validation set with mean standard error revealed minor improvement increasing the training patch size from 64 × 64 pixels to 128 × 128 pixels and no observable difference for 256 × 256 pixels.

The network was trained on 550 image pairs generated by ex vivo imaging of freshly excised human skin tissue (8 excisions including pigmented and nonpigmented skin) at different depths within viable epidermis. CARE model was trained on a diverse set of features such as corneocytes, keratinocytes, melanin caps, hair follicles, sweat glands, elastin fibers and skin folds. The training set was acquired by scanning 20 mm² over 6–10 optical sections at different depths spaced out by 10 μm spanning from skin surface to the papillary dermis. This volumetric imaging procedure ensured a fair representation of skin features accessible within the depth limitation of noninvasive MPM in vivo imaging (< 300 μm from the skin surface). We used a total of 35,200 patches for the network training pooled together from all the specimens. Ten percent of these images (3,520) were randomly selected for the validation set, while the remaining 31,680 images were used for training. We applied this approach only to the TPEF images acquired by the FLAME system, as the SHG signal was sufficiently high at fast scanning rates. Training the model over 100 epochs with 30 steps per epoch resulted in a mean square error of 0.005 and a mean absolute error of 0.05 on the validation set. On an independent test set consisting of 48 900 × 900 µm² 1 MPx images from 3 subjects CARE-processed images show an increase in structural similarity (SSIM) index (0.6–0.7) relative to the high SNR ground truth (70 accumulations) compared to low SNR (15 accumulations) input images (0.3–0.5). Representative input-ground truth image pairs from the test set along with the output image generated by the model are shown in Figure S5 in the Supplementary Information.

Multi-scale, high-resolution imaging by optical and mechanical scanning

We perform imaging on millimeters to centimeter scale by employing the following scanning mechanisms for stitching adjacent fields of views and for acquiring volumetric images:

1. the strip mosaic scheme employs the resonant galvo mirror as the fast scanner in the x direction and a miniature linear stage as the slow scanner in the y direction. The width of the strip is defined by the laser line produced by the fast scanning mirror, while its length is determined by the stage travel range. The stage moves back in the –y direction and also laterally in the x direction, a distance equal with the width of the strip, before completing each of the subsequent strips. We commonly acquire 10 mm long, 0.75 mm wide strips, resulting in an aspect ratio of 1:13. Depending on the sample size, we acquire up to 16 strips resulting in a scanned area of 10 × 12 mm², typically collected as 64 MPx image and rebinned using bicubic interpolation to maintain the correct aspect ratio to 80 MPx. The acquisition and restoration time for this type of image is 2 min 15 s. The length and the number of the strips are limited by the stage speed, its travel range (13 mm) and its synchronization with the resonant scanner.

2. the tile mosaic scheme employs the galvo-resonant mirrors to scan each frame (‘tile’) and the linear stages to translate the sample in the x and y directions in order to scan adjacent tiles.

3. the volumetric imaging involves acquiring z-stacks of en face images by moving the objective in the z direction, thus scanning at different depths in the skin.

Strip mosaic provides the fastest scanning rate for single frame acquisition. We employ the single frame strip mosaic scanning mechanism for centimeter spatial scale imaging. However, multiple frames accumulation is often required at millimeter-scale FOV for enhanced contrast to allow accurate morphological assessment. In this case, our current data acquisition software provides a faster scanning rate for the tile mosaic configuration. At millimeter scale, we commonly acquire tile mosaic images at 15-accumulating frames. Figure S6 in the Supplementary Information presents a comparison of the tile and strip mosaic scanning speed for different configurations we typically employ for the multi-scale imaging. Table S2 in the Supplementary Information summarizes the parameters (digital resolution and scanning time) used for scanning different spatial scales.

Excised tissue collection and in vivo imaging

Skin specimens were obtained from the surgeries in the UCI Dermatology Clinic and consisted of remaining tissue from the wound closure procedures (“dog ears”). The tissue collection procedure was exempt from the Institutional Review Board (IRB) approval since the specimens were de-identified, but they were handled based on an Institutional Biosafety Committee (IBC) approved protocol. The specimens were imaged fresh, immediately upon collection. The in vivo measurements were conducted according to an approved IRB protocol of the University of California–Irvine with written informed consent obtained from the subjects.

Author contributions

A.F. and A.F.D designed the system components and performed the experiments. A.L. wrote the code for multidimensional data parsing, mosaic assembly and real-time image visualization. A.F. collected data, trained the CARE network and performed the image analysis. G.L. and A.F.D designed the microscope interface for the in vivo imaging. G.L. performed the tissue processing and staining. C.B.Z. provided excised de-identified tissue from surgeries. C.B.Z. and A.K.G provided feedback regarding relevant imaging parameters for the current design of the FLAME system to facilitate specific clinical applications and clinical workflow integration. R.M.H provided guidance regarding histology. M.B. supervised the study. A.F. and M.B prepared the manuscript. All authors contributed to the discussion of the results and reviewed the manuscript.

Data availability

The datasets generated and analyzed in the article are available from the corresponding author upon reasonable request.

Competing interests

M. Balu reports a patent, which is owned by the University of California and is related to the technology described in this study. The Institutional Review Board and Conflict of Interest Office of the University of California, Irvine, have reviewed patent disclosures and did not find any concerns. The other authors disclosed no conflicts of interest.

Supplementary information

is available for this paper at 10.1038/s41598-020-75172-9.