Smartphone confocal microscopy for imaging cellular structures in human skin in vivo

Abstract

We report development of a low-cost smartphone confocal microscope and its first demonstration of in vivo human skin imaging. The smartphone confocal microscope uses a slit aperture and diffraction grating to conduct two-dimensional confocal imaging without using any beam scanning devices. Lateral and axial resolutions of the smartphone confocal microscope were measured as 2 and 5 µm, respectively. In vivo confocal images of human skin revealed characteristic cellular structures, including spinous and basal keratinocytes and papillary dermis. Results suggest that the smartphone confocal microscope has a potential to examine cellular details in vivo and may help disease diagnosis in resource-poor settings, where conducting standard histopathologic analysis is challenging.

1. Introduction

Disease diagnosis in low- and middle- income countries (LMICs) is often hampered by lack of necessary resources. In developed countries, biopsy of a suspected tissue lesion and histopathologic analysis of the biopsy are the standard for diagnosis of many diseases. In LMICs, however, resources needed for biopsy and histopathologic analysis are scarce. Diagnoses are often made inaccurately solely based on clinical examination [1, 2]. When biopsy-based histopathologic analysis is employed, histopathologic diagnosis takes several weeks to complete, which leads to delay or failure in initiating adequate treatment. Recently, several smartphone-based microscopy technologies have been developed to address clinical challenges in LMICs [3–6]. Most of these technologies, however, still require tissue collection and slide preparation, which remains a challenging task in many LMICs. Dermoscopy is a low-cost imaging method that is commonly used for examining skin lesions in vivo. However, dermoscopy cannot examine cellular features under the skin surface, which are critical in histopathologic diagnosis, and therefore provides moderate diagnostic accuracy [7, 8].

Reflectance confocal microscopy (RCM) has a potential to improve disease diagnosis in LMICs since cellular changes associated with diseases can be examined in vivo without biopsy and histopathology [9]. Through many studies, RCM has been shown to provide high diagnostic accuracy for key skin diseases, including melanoma and basal cell carcinoma [10]. Conventional RCM devices, however, are expensive and not readily useable in LMICs.

Spectrally encoded confocal microscopy (SECM) is a RCM technology that uses a diffraction grating and broadband light source to spread illumination light over a line on the tissue and conduct line confocal imaging without using a beam scanning device [11]. We have previously developed SECM endoscopic devices and demonstrated comprehensive confocal imaging of esophagus in vivo [12, 13]. However, previous SECM devices needed a mechanical scanning mechanism, high-cost wavelength swept source, and high-bandwidth photo detector and data acquisition system, which made SECM devices not suitable for LMIC applications. SECM can be configured to use a slit aperture to acquire two-dimensional confocal images without using any beam scanning devices [14]. The slit SECM approach uses low-cost optical elements, an inexpensive LED as the light source, and a two-dimensional imaging sensor to acquire confocal images. CMOS imaging sensors in recent smartphones have a large number of pixels and high sensitivity and can be used as an imaging sensor in the slit SECM approach to finally construct a low-cost smartphone confocal microscope.

In this paper, we report development of a smartphone confocal microscope and present preliminary results of imaging human skin in vivo. Details of the smartphone confocal microscope are described. In vivo confocal images of human skin are presented and cellular features visualized in the confocal images are analyzed.

2. Methods

2.1 Smartphone confocal microscope

In conventional confocal microscopy devices, a focused illumination spot is serially scanned over the tissue using two beam-scanning devices (Fig. 1(a)). Our approach, instead, uses a focused illumination line from a slit aperture and a diffraction grating to spread the line over the tissue (Fig. 1(b)). Different wavelengths are illuminated on different locations on the tissue, encoding a spatial coordinate with wavelength.

Fig. 1.

Schematics of point scanning (A), spectrally-encoded line scanning (B), and smartphone confocal microscope (C).

We used this wavelength-encoded two-dimensional illumination method in our smartphone confocal microscope. In the smartphone confocal microscope (Fig. 1(c)), an inexpensive LED (XPEBPA-L1-0000-00D01-SB01, Opulent; price = $8; central wavelength = 595 nm; bandwidth = ~80 nm) was used as the light source. This LED was chosen due to its large spectral bandwidth, which allowed for wide field of view along the spectrally-encoded direction. Light from the LED was filtered by an illumination slit (width = 25 µm; length = 3mm) and collimated by a collimation lens (f = 25 mm). Collimated light was diffracted into multiple beams by a transmission grating (groove density = 1,379 lpmm) and focused by an objective lens (CFI Apo 40XW NIR, Nikon; water immersion; NA = 0.8). Each wavelength was focused into a tight line at a distinctive location inside the tissue (inset, Fig. 1(c)). Light reflected from the tissue was collected by the objective and diffracted again by the grating into a single line beam. The detection beam was focused by a focusing lens (f = 25 mm) on a detection slit (width = 10 µm; length = 3 mm), where confocal optical sectioning was achieved.

A divided pupil approach was used, where illumination beam and detection beam used opposite sides of the objective lens pupil. The illumination and detection paths were non-coaxial and separate. Compared to the conventional full-pupil approach in which the illumination and detection paths are co-axial, the divided pupil approach provided higher image contrast [15] and better rejection of the specular back-reflections from optical components. At the objective lens pupil plane, each of the illumination and detection beams had a diameter of 3 mm and was located 3 mm away from the center of the objective lens pupil.

Point spread function (PSF) of the smartphone confocal microscope was simulated by multiplying illumination and detection beams (Fig. 2) [16]. Illumination intensity was calculated by convolving the de-magnified image of the illumination slit in the object space (width = 5 µm; length = 600 µm) with the diffraction-limited PSF for the given beam diameter and objective focal length [17]. The illumination intensity was then rotated by an angle determined by the objective focal length and beam offset. Detection efficiency was calculated by convolution of the detection slit image in the object space (width = 2 µm; length = 600 µm) with the diffraction-limited PSF. Full width half maximum (FWHM) of the smartphone confocal PSF was calculated along the lateral and axial directions (x and z in Fig. 2(a)) and along the minor and major axes of the PSF (u and w in Fig. 2(a)). FWHM values were 2.1, 1.9, 4.5, and 6.0 µm for x, u, z, and w axes, respectively (Fig. 2(b)). These FHWMs were larger than the lateral and axial FWHM values for the ideal confocal microscope with infinitely thin illumination and detection slits, 0.8 and 1.6 µm, respectively. Larger slit widths were used to provide sufficient signal-to-noise ratio (SNR).

Fig. 2.

Simulated smartphone confocal microscope PSF (A) and line profiles of confocal PSF (B).

Light filtered by the detection slit was collimated by a collimation lens (f = 25 mm) and diffracted by a grating (groove density = 1,379 lpmm). Diffracted light was focused by a focusing optics of two positive lenses (f = 25 mm and 7.5 mm, respectively) to generate an intermediate two-dimensional confocal image. The intermediate confocal image was relayed to a CMOS sensor (pixel size = 1.55 µm; 12.3 Mpixels) inside a smartphone (Nexus 5X, Google and LG). The relay optics was comprised of two identical smartphone camera lenses (f/2.0; f = 4.7 mm), one as part of the smartphone camera and the other added in front of the smartphone camera. Magnification between the object plane and CMOS sensor was 3.4, which made the pixel size in the object plane as 0.46 µm.

Custom optics holders were designed and 3D-printed (Form 2, Formlabs) to assemble most optical elements. A manual translational stage with three degrees of freedom was used to achieve precise conjugate alignment between the illumination and detection slits. Outer mechanical holders were custom designed to securely hold the optics holders and block ambient light from entering the smartphone camera. The mechanical holders were made of ABS plastic using a 3D printer (Fortus 250mc, Stratasys).

2.2 Image acquisition and processing

Confocal images were acquired as either still images or videos. When still images were acquired, both JPEG and DNG files were saved. An Android camera application (Open Camera) was used to control the camera parameters such as exposure time, ISO, and image resolution and acquire still images and videos. Focus of the smartphone camera was adjusted using the manual focus function of the camera application to achieve sharpest image contrast. All three color channels of the image were summed together to generate a monochromatic confocal image.

Saved images and videos were transferred to a computer and processed with a custom Matlab code (Mathworks). Confocal images were compensated for the intensity variation due to the LED source spectrum and grating diffraction efficiency. An elliptical mask was superimposed on perimeter of the field of view, where SNR was low. For images processed from videos, running average with 2 consecutive frames was conducted to improve the SNR.

2.3 Imaging performance testing

Lateral resolution was measured by imaging a USAF resolution target. FWHM of the line spread function (LSF) was calculated along horizontal and vertical directions. An axial response curve was obtained by acquiring a confocal video of a mirror while axially translating the smartphone confocal microscope with a motorized stage. Axial FWHM was calculated from the axial response curve. Exposure time was significantly reduced during the resolution measurement to avoid sensor saturation.

Human skin imaging performance was tested in vivo. First, confocal images of the skin were acquired at multiple imaging depth levels with motorized axial scanning. A forearm was placed on the optical bench and under the smartphone confocal microscope. Videos were acquired while the smartphone confocal microscope was axially scanned with a motorized stage at 1 µm/sec speed. Still images were also obtained at several depth levels. Second, videos and images were obtained at multiple locations and depth levels while manually moving the smartphone confocal microscope on the skin. This manual scanning method is similar to the clinical imaging protocol used for the handheld commercial RCM device (Vivascope 3000, Caliber). A metallic washer was attached to the skin. The washer was then magnetically connected to the objective lens of the smartphone confocal microscope, which provided stable imaging at locations of interest while minimizing skin motion blur. Videos and still images were acquired while the smartphone confocal microscope was manually moved around by the operator. Pressure on the skin was varied by the operator to change the imaging depth. During the skin imaging, exposure time was set as 1/4.3 second.

3. Results

3.1 Resolution measurement

Photos of the smartphone confocal microscope are shown in Fig. 3. The smartphone confocal microscope had a dimension of 16cm (W) × 18 cm(D) × 19 cm (H) and weight of 1.25 kg. Light power incident on the tissue was 15 µW.

Fig. 3.

Photos of the smartphone confocal microscope. A – front view with the cover removed; and B – top view.

An image of the USAF resolution target is shown in Fig. 4(a). No significant field distortion was observed over the entire field of view. Lines in the group 8, element 6 (linewidth = 1.10 µm) were distinguishable along the spectrally-encoded direction and group 9, element 2 (linewidth = 0.87 µm) along the perpendicular direction. FWHM of LSF was measured as 2.3 ± 0.4 µm and 2.0 ± 0.1 µm along the spectrally-encoded and its perpendicular directions, respectively. Axial FWHM was measured to be 5.1 ± 0.7 µm over the entire field of view (Fig. 4(b)). The measured lateral and axial resolutions agreed well with the simulation results.

Fig. 4.

Confocal image of USAF resolution target (A) and axial response curve (B).

3.2 In vivo skin imaging

Confocal images of human forearm (Fitzpatrick’s Skin Phototype III) are shown in (Fig. 5). The field of view of each image was 430 µm × 516 µm. An image taken at depth of 18 µm (Fig. 5(a)) showed bright reflection from the stratum corneum with skin folds and corneocytes within. At 29 µm imaging depth (Fig. 5(b)), dark cell nuclei of keratinocytes were visualized against bright cell borders. At 33 µm depth (Fig. 5(c)), size decrease in keratinocytes was noticeable (inset) and melanin-containing keratinocytes and melanocytes started to appear as bright cells (arrows). In an image obtained at 50 µm depth (Fig. 5(d)), dermal papillae (asterisk) were visualized, demarcated by bright basal cells (arrows). Scattered basal cells were also visible at this depth.

Fig. 5.

Confocal images of human skin in vivo taken at imaging depth of 18 µm (A), 29 µm (B), 33 µm (C), and 50 µm (D). arrows – basal cells. Inset magnification = 2.

A confocal image generated from a MP4 video (Fig. 6(a)) was compared with the corresponding image generated from a DNG file (Fig. 6(b)). Both images were taken at approximately 50 µm depth. The two images were normalized to have similar dark levels and average tissue signal levels. While both images revealed similar cellular details such as dermal papillae and basal cells (arrows), the image generated from a DNG file showed lower noise compared to the image made from a MP4 video file (insets).

Fig. 6.

Confocal images of human skin in vivo obtained from a MP4 video file (A) and a DNG file (B). arrow – basal cells. Inset magnification = 2.

A video recorded during the manual scanning of the smartphone confocal microscope relative to the skin is presented (Visualization 1 ^{(5.7MB, avi)} ). This video showed that confocal imaging at multiple locations and imaging depth levels was feasible in near real time. Snapshot images taken from the video revealed characteristic cellular features of human skin, including honeycomb pattern of dark keratinocyte nuclei (Fig. 7(a)), cobble stone pattern of bright basal cells (Fig. 7(b)), and capillaries (arrows in Fig. 7c) in an edged dermal papilla.

Fig. 7.

Confocal images of human skin obtained at three distinctive locations (see Visualization 1 ^{(5.7MB, avi)} ). A – honeycomb pattern of dark keratinocytes; B – cobblestone pattern of basal cells; and C – capillaries (arrows). Inset magnification = 2.

4. Discussions

In this paper, we have presented first demonstration of smartphone confocal microscopy of human skin in vivo. Imaging results showed that our smartphone confocal microscope could visualize key cellular structures of human skin in vivo.

The smartphone device had similar resolutions to commercially-available confocal devices: 2 and 5 µm (lateral and axial resolutions) versus 1 µm and 5 µm. However, use of the shorter wavelength in the smartphone device than the commercial devices, 595 nm versus 830 nm, and use of the slit aperture in place of a pinhole aperture might reduce the imaging depth. Also, the imaging speed of the smartphone device is currently limited to 4.3 fps due to the low illumination efficiency, which is slower than that of the commercial devices, 6-9 fps. In future development, we will improve the imaging depth and speed by using a light source with a longer wavelength and better coupling efficiency into the illumination slit [18]. We have tested both video and still image acquisitions. Video acquisition facilitated confocal imaging at multiple fields and depths but was expected to have a degraded image quality due to the image compression. While having an increased noise compared to the still image (Fig. 6(b)), the video confocal image (Fig. 6(a)) successfully visualized cellular features. In future development, video acquisition will enable large-area imaging of the skin lesion using a video-mosaicking method [19].

Thanks to the low cost (material cost = $4,200) and portability, the smartphone confocal microscope has a potential to improve disease diagnosis and treatment in various clinical settings. The smartphone device can be used to examine skin lesions in LMICs and guide diagnosis and treatment. An endoscopic imaging probe can be used in conjunction with the smartphone device to help diagnose various conditions in internal organs such as the cervix and esophagus. In developed countries, owing to the low cost and convenience of the smartphone confocal microscope, increased clinical adoption is expected. This can lead to large-population screening and enhanced biopsy guidance for early disease detection. Furthermore, the smartphone confocal microscope may be used as a teaching tool for medical and STEM educations.

Several challenges are expected during adaptation of the smartphone confocal microscope in low-resource settings. Intermittent electric power supply in rural areas might make it difficult to charge the smartphone confocal device. The 12V rechargeable battery used in the smartphone device has a capacity of 3000 mAh, which can power the LED for 11 hours, and is easily exchangeable with another battery. During an imaging study at a rural site for an extended period, we will charge multiple 12V rechargeable batteries beforehand and bring to the study site as back up batteries. While the smartphone confocal device has a significantly lower cost than the commercial confocal devices, its cost still might pose challenges in wide adaptation in LMICs. We will develop a custom, low-cost objective lens [12, 20, 21] and utilize a grating replication method [22] to further reduce the cost. Finally, cellular features visualized in confocal images need to be reviewed to make diagnosis. Our current plan is to wirelessly transfer confocal images to trained confocal image readers, who can review the confocal images, make diagnosis, and wirelessly communicate with local health professionals in real time. In future, we will also explore feasibility of developing an automated image analysis algorithm [23], which will obviate the need for trained image readers.

Funding

National Institute of Health/Fogarty International Center (R21TW010221); National Cancer Institute (P30CA008748).

Disclosures

GJT and DK are inventors on a patent application filed by Massachusetts General Hospital on the technology presented.