A smartphone-based chip-scale microscope using ambient illumination

Abstract

Portable chip-scale microscopy devices can potentially address various imaging needs in mobile healthcare and environmental monitoring. Here, we demonstrate the adaptation of a smartphone’s camera to function as a compact lensless microscope. Unlike other chip-scale microscopy schemes, this method uses ambient illumination as its light source and does not require the incorporation of a dedicated light source. The method is based on the shadow imaging technique where the sample is placed on the surface of the image sensor, which captures direct shadow images under illumination. To improve the imaging resolution beyond the pixel size, we perform pixel super-resolution reconstruction with multiple images at different angles of illumination, which are captured while the user is manually tilting the device around any ambient light source, such as the sun or a lamp. The lensless imaging scheme allows for sub-micron resolution imaging over an ultra-wide field-of-view (FOV). Image acquisition and reconstruction is performed on the device using a custom-built android application, constructing a stand-alone imaging device for field applications. We discuss the construction of the device using a commercial smartphone and demonstrate the imaging capabilities of our system.

Introduction

Microscopes are one of the most commonly used equipment in biology and medicine, yet the size and the cost of these microscopes limit its applications in the field settings. For this reason, portable microscopic imaging systems are in high demand, especially for global healthcare and environmental monitoring. For example, diagnostics for many third-world diseases, such as water-borne parasite infections, blood-borne diseases and bacterial infections, require microscopic inspection of bodily fluids or cells/tissue samples^1–6. Also, microscopic analysis of environmental specimen is a crucial step of water quality monitoring and environmental pathogen screening⁷. A low-cost, light-weight portable imaging systems with network connectivity can greatly improve and simplify the way these tests are conducted in the field.

Recent advances in smartphone technology are having a transformative impact on global healthcare and remote sensing. Smartphone penetration is expected to surpass 60% of the global population by the end of 2019⁸. Modern smartphones deploy high computing power comparable to personal computers, high-speed mobile network connectivity and complex sensor technologies, all integrated in a palm-sized geometry. In particular, camera modules in smartphones employ state-of-the-art image sensors with small pixel sizes and high pixel counts, currently up to 40 Mega pixels. Microscopes using these built-in camera modules allow for compact and portable digital imaging platform ideal for field applications. In addition, the connectivity of these mobile devices opens up various opportunities for telemedicine and remote diagnostics in resource-limited settings^9–11.

Many efforts have been demonstrated to construct a compact microscope on mobile devices. One of the initial approaches was to add an objective lens atop of a camera module of a mobile phone¹². These imaging systems can be attached to any mobile phones without modification of the device, realizing low-cost microscopes for the mass. The performance of these microscopes is determined by the design of the optical systems, where increased resolution may result in the limited field-of-view (FOV). As an alternative approach, lensfree microscopes on mobile phones based on digital inline holography method has been demonstrated¹³. These microscopes computationally render images of the target objects by interferometric measurements under a controlled illumination and can achieve low-cost and light-weight configuration ideal for field applications. Recently, a smartphone-based contact microscope has been developed to image dense or connected samples without the holographic reconstruction, with the attachment of a tapered fibre-optic array over the camera module¹⁴.

Here, we report on a smartphone-based chip-scale microscope in a lens-free and light-source-free configuration. Our device follows a contact imaging scheme where the sample is mounted directly on top of the image sensor and uses an ambient light rather than a dedicated light source for illumination. To improve the image resolution beyond the pixel size, we rely on the user’s hand motion to manually tilt the device around the light source, such as the sun or a lamp, to capture multiple images with varying illumination angle, which are then processed with the pixel-super resolution algorithm. This scheme eliminates the illumination design as well as the lenses, thus allowing for a simple, low-cost and compact configuration that is only composed of an image sensor. Image acquisition and reconstruction is performed on a custom-built android application, constructing a stand-alone portable imaging device for field applications.

The imaging scheme based on manual angular scanning and the pixel super-resolution reconstruction is a derivation of chip-scale microscopy techniques that have been previously demonstrated in our group^15–18. The general strategy of our chip-scale microscopes requires the sample be placed on the surface of the image sensor. The shadow image casted by the sample upon illumination is collected with the resolution determined by the sensor’s pixel size. We then improve the image resolution via the pixel super-resolution image reconstruction using multiple low resolution images taken with sub-pixel shifts between each frame. In our previous work (sub-pixel sweeping microscopy, SPSM)^{16, 19}, we have used moving light sources, such as the bright pixels in a smartphone screen or a light emitting diode (LED) matrix, which creates sub-pixel shifted shadows on the detector surface. The basic concept of our smartphone microscope is similar to that of SPSM, but with further simplification of the device by using the manual scanning under ambient light as illumination.

Direct shadow imaging does not impose strict requirements on both the illumination and the sample and thus any incoherent light sources with broad spectrum can be used. We can image contiguously connected samples such as confluent cell cultures and high-density smear films, thus the biological sample can be prepared at the same concentration as in conventional microscopy slides. In addition, image reconstruction using pixel-super resolution algorithm does not require high computing power and can be performed on smartphone processors. For these reasons, our chip-scale imaging technique holds a solid ground for applications in mobile microscopy.

In the following section, we will demonstrate the working principle of our smartphone microscope and showcase the performance of our prototype device. With the compact configuration and robust performance, we envision that our smartphone microscope can be a good fit for outdoor imaging applications where the user can perform microscopy imaging with ambient illumination.

Methods

The working principle of our smartphone microscope is shown in figure 1. The technique is based on the shadow imaging method where the sample is placed directly on the surface of the image sensor. The light transmitted though the sample is collected at the photodiode of each pixel in the image sensor, providing under-sampled direct shadow images with the resolution limited by the size of the pixels. To improve the imaging resolution, a sequence of images are captured with the varying angle of illumination as the user tilts the device around the light source. Then, the captured raw sequence is processed with pixel super-resolution algorithm and reconstructed into a single high resolution image with enhanced optical resolution. We implemented a prototype system on android smartphones, by modifying the built-in camera module (Fig. 1b). Image acquisition and the processing is performed on the smartphone with a custom-built android application.

Figure 1.

(a) Working principle of the smartphone-based chip-scale microscope using sunlight as the light source for imaging. For image capturing, the user holds the smartphone with the back camera module facing the sun and slowly moves the device to capture multiple images with varying angle of illumination. (b) The prototype device uses the back camera module of an android phone. We remove the lens module of the camera and place the sample directly on the surface of the image sensor. The inset shows the image sensor module with the lens removed. (c) As the user tilts the device around, we trace the shadow of a reference target placed on a corner of the image sensor to calculate sub-pixel shifts of the sample shadows. The reference target also allows us to check the quality of illumination, such that it creates a single, sharp shadow. (d) Raw (Left) and reconstructed (Right) images of 2.5μm microspheres. 100 images were taken while we manually tilt the device and the tilt angle for each image is shown in the plot (centre).

The shadow imaging scheme does not require pre-designed illumination sources for imaging, thus allows for a compact configuration without any add-on light sources. The illumination can be any incoherent light from a single source, for example, the sun, a flash light or a lamp, such that it creates a single shadow. In this work, we have demonstrated our imaging capabilities with sunlight, a LED flashlight and a fluorescent lighting. For the field applications, the user can simply point the camera towards the sun and acquire images. In the case of an indoor use or in the presence of overcast, indirect illumination may cause multiple or diffused shadows. The user can instead obtain images with other illumination sources, such as a light bulb or a flashlight. In these cases, the distance from the device to the light source needs to be at least an order of magnitude larger than the diameter of the light source to minimize the blur of the shadows. To determine whether the illumination is suitable for imaging, we placed a known target object on one corner of the image sensor, such that the sharpness of the target’s shadow can be used as an indicator (Supplementary figure S1). Due to the high intensity and unwanted infrared bands of direct sunlight, we attached an infra-red filter and a neutral-density filter (2 Optical density units) on the back cover of the smartphone. To test various illuminations, we imaged the same microspheres under an LED flashlight (12 W/m²), a fluorescent lamp (5 W/m²) and the sun (900 W/m²) (Supplementary figure S2).

Our smartphone microscope prototype uses the built-in camera module of a smartphone. The image sensor surface was revealed by simply removing the lens module in the back camera module of the smartphone (See supplementary movie M1). Figure 1b shows the modified camera module. The camera module parts are readily available for purchase and are very easy to replace. We used two smartphone models –Samsung Galaxy S3 (GT-I9300) and S4 (GT-I9500). The pixel sizes of the image sensor is 1.4 μm (3264 × 2448, 8 megapixels) and 1.1 μm (4128 × 3096, 13 megapixels), respectively. The total FOV is same as the size of the sensing area in the image sensor, which was 4.6 mm × 3.5 mm for both devices. Due to demosaicing process in the camera, we cannot access the raw values of each pixel. Instead, we selected green pixels in the Bayer pattern, which occupy 50% of the pixels in the image sensor, and rotated the image by 45°. The effective pixel size of the raw image becomes larger by the factor of √2, which is 1.98 μm and 1.56 μm, with 50% areal fill factor.

Sample preparation step is identical to previous chip-scale microscopy techniques¹⁹. Direct shadow imaging relies on the sample being in contact with the image sensor. For the experiments in this work, we made a dry or wet film of the sample on the image sensor. For blood samples, we made thin smears by spin-coating the untreated blood sample on the sensor. The blood smear was stained for higher image contrast by dipping the entire image sensor module in methanol and Wright-Giemsa staining solutions according to the standard blood smear staining procedure. In the case of microspheres, we put a drop of microsphere solution on the image sensor and dried the solvent. For other biological samples dispersed in liquid, we made the wet film on the sensor the same way as preparing a microscope slide - a small PDMS film is placed over the sample to ensure that the particles are near the image sensor’s surface [ref]. Also, the PDMS cover film that the illumination is uniform over the entire sensor, and removes the possible lensing effect from liquid droplets. Mounting a microfluidic chamber or a cell culture well on the image sensor is also possible but has not yet been demonstrated in this work.

The algorithm of image reconstruction is similar to our previous chip-scale microscopy demonstrations. The raw sequence represents spatially under-sampled images of the original image with each frame translated by known sub-pixel shifts. Then, the algorithm re-arranges the low resolution sequence into a single high resolution matrix, according to the sub-pixel shifts of the object in each frame. In the SPSM scheme, the shadow of the sample on the image sensor plane translates with a known amount of shift for each illumination angle that is determined by the arrangement sweeping light source. In this work, we rely on user’s hand motion to tilt the device (both sample and the detector) around the light source to capture multiple low resolution shadow images with sub-pixel shifts.

Since the image reconstruction relies on the knowledge of the sub-pixel shifts between each frames, it is crucial to precisely measure the angle of incidence of the illumination in each low-resolution frame. To calculate the illumination angle of the user’s manual tilting, we traced the shadow images of a known reference target that is placed above the sensor. In shadow imaging, displacement of the shadow varies linearly with the distance between the sample and the object. The actual sample is usually located within a few micrometres range from the sensor surface (depending on the size of the sample), thus the sub-pixel shifts of the sample shadow can be scaled from the shift of the reference shadow. When the reference object is placed at a large distance away from the sensor, the reference shadow moves in large displacement and the accuracy of the measurement increases. However, if too far, reference shadow becomes blurry due to diffraction and the tracking of the shadow may fail due to the low contrast of the target shadow images. For the reference target, we placed a piece of 200μm-thick transparent film with a cross-pattern (200μm × 200μm) printed on one side. The film is placed such that the printed target pattern faces towards the illumination (Fig. 1c).

Figure 1d shows raw images and improved high resolution images of 2.5 μm microspheres imaged with a 1.4-μm-pixel sensor. Note that the bright centre of each microsphere are resolved in the improved images as oppose to the low resolution images. We used 100 images to perform 8 × 8 enhancement by rounding the sub-pixel shifts to integer multiples of 1/8^th of a low resolution pixel.

Figure 2 shows the workflow of the image acquisition. User starts the application and opens the camera to start image capture. The user is advised to point the camera towards the light source and tilt the camera in all directions. As the images are captured, the application calculates in real-time the tilt angle of each image, which are plotted on the screen and also saved in the memory for image reconstruction. After the image acquisition, the user can review and load the sequence of images and select a smaller region for reconstruction (200 × 200 pixels). The program then crops the user-defined region out of the entire sequence, normalizes the images, and performs image reconstruction with the sub-pixel shift values measured in the image acquisition step. When the data is loaded directly from the device (previously captured data), the application performs target tracing and image cropping/normalization simultaneously. The final reconstructed image is displayed on the screen and also saved automatically in the device. Image normalization and cropping takes approximately 2 minutes for 100 frames and the delay is mostly from accessing each image file in the memory. The high resolution image reconstruction takes few seconds for 200 × 200 pixels. (See supplementary movie M2)

Figure 2.

Imaging process of with the custom-built application. (a) Upon start, the user can choose to acquire new data using the camera, or load acquired images from the memory. (b) When capturing a new data set, the user points the camera towards the sun and starts capturing images. The reference target (a cross mark) shown in the upper left corner of the field of view is traced in each frame and the measured shifts are plotted over the camera view. Captured images are saved in the data storage, under a specified folder. Once the capturing if finished, user can choose to close the camera and return to the main window or to capture the data again. The sample used in this demonstration is an unstained blood smear. (c) From the main window, the user can press the photo gallery icon and load one of the acquired images from the gallery. The application prompts the user to select a smaller region to reconstruct. (d) Once a region of interest is selected, the application returns to the main window and plots a low resolution image of the selected region to confirm. The user clicks the start processing button, and the application crops the entire low resolution sequence and pre-process the data to normalize the background. (d) Once the pre-processing is done, the user can input a scale factor for desired height of in-focus plane and start reconstruction. (e) The final high-resolution image is displayed in the application and automatically saved to the data storage.

In order to remove motion-dependency in the image quality (Supplementary figure S3), the user is advised to tilt the camera around all four directions (up, down, left and right) within the boundary displayed in overlay with the camera. The application calculates the tilt in real-time as the user moves his/her hand around, plots the location in the screen to inform the user at which angle the images were taken. With the real-time processing, we can also reject images that are not suitable, for example, when an image is taken outside of the suggested boundary of illumination angle and when an image was previously taken at the same angle. The camera runs until it takes enough number of pictures. Typically, we used 100 frames for 8× enhancement at the frame rate of 3 frames/sec for data transfer and target tracing, resulting in ~40 seconds of total image acquisition time (Galaxy S3). The frame rate is variant on the data transfer rate of the device and the image size.

The real-time target tracing was performed with OpenCV-based image processing algorithm²⁰. The target object tracking process is as following; from a captured image, we first converted the region containing the target shadow into Hue-saturation-brightness (HSB) image. We then thresholded the HSB image into a binary image based on hue and saturation values to highlight the dark region (reference target’s shadow) in the image. After removing the unwanted noises in the binary image, we computed the centre of mass of the image to find the centre position of the target shadow. We dump the images where target tracing was unsuccessful or the tilt angle was out of suggested range. Otherwise, the image and the measured target location is saved in the memory for reconstruction. Upon reconstruction, the translation of the target shadow is scaled to the expected shadow shift of the sample on the image sensor surface. This scale factor, which denotes the height of the in-focus plane, is taken as user input upon reconstruction.

Results and Discussions

Figure 3 shows blood smear images taken with our Samsung Galaxy S4 prototype. We used 100 images to reconstruct a high resolution image with the enhancement factor of 8. Note that the boundaries of connected cells are not clearly visible in a single low resolution image but the boundaries of each red blood cell are resolved in the reconstructed image. For capturing of the raw images, we used the inherent white-balance settings of the camera, which were set with the presence of an IR filter, and thus the raw images show stronger red channel. We reset the white balance of the reconstructed images during the normalization step of the image processing to render grey background. To test the effect of the light sources, we imaged the same sample under sunlight, flashlight and a fluorescent lamp. (Supplementary figure S4). The quality of these images are similar between illuminations, except that the colour contrast is higher in the images with an LED flashlight and a fluorescent lamp since the illumination spectra better matches the absorption spectra of methylene blue and eosin in Wright-Giemsa stain.

Figure 3.

Wright-Giemsa stained blood smear images taken with our system. (a) A full FOV image of a blood smear made on the image sensor. Image sensor measures 4.5 mm × 3.6 mm and the scale bar indicates 1mm. (b) Raw and (c) reconstructed high resolution image of the region highlighted in (a). Images are reconstructed in the custom-built application. Scale bar indicates 20μm.

We investigated the resolution limit of our device by imaging 500-nm microspheres. We placed the microspheres on the surface of the 1.1-μm-pixel image sensors (Galaxy S4) and captured 200 images for reconstruction with the enhancement factor of 13 and 8. In both cases, the microsphere were resolved, and the diameter of the microspheres measured 580 nm and 590 nm, respectively (Fig. 4). However, the images were inverted - the rim of the microsphere appears brighter than the centre. We believe that this is caused by the microlens array on the image sensor pixels; due to the microlens pattern, small microspheres tend to locate in the grooves between the pixels. The shadow or scattering of light from these microspheres were not distinguishable from the background upon illumination angle within the acceptance cone of the microlens. When the illumination angle is large, the scattering of the microsphere appears brighter than the background. We did not observe this effect with the microspheres that are larger than the microlens size.

Figure 4.

(a) Images captured with 500-nm polystyrene microspheres on the image sensor. We used 1.12-μm pixel sensors (effective low-resolution pixel size of 1.58 μm) (b) Magnified images of the microsphere marked in (a). We used 13× and 8× enhancement for high resolution reconstruction. One pixel in each image measures 120 and 200nm, respectively. (b) Line trace of a microsphere images with 13× and 8× enhancement. In both cases, the centre of the microsphere is resolved.

Also, we suspect that the unremoved bayer filter pattern and the demosaicing algorithm of the camera causes the background checkerboard-pattern noise in the reconstructed images, which is more apparent with the low-contrast samples. However, these periodic patterns can be suppressed by 1) using a monochromatic sensor without the colour filter layer and 2) filtering out the corresponding spatial frequency components from the image. Previously, we have shown that the bright centres of 500-nm microspheres can be resolved with 2.2-μm-pixel sensors without microlens array¹⁶. Removal of the microlens array and the colour filters should remove the image distortion in small objects and yield improved resolution due to the reduced sample-to-sensor distance.

There are several other factors that affect the image quality. Image reconstruction with 8× enhancement provides sufficient high-resolution pixel sizes (200-nm per pixel) that is below the Nyquist limit for resolving 500-nm objects. Yet, higher enhancement factor yields higher precision in image registration in the shift-and-add reconstruction, at the expense of processing speed and data size. Using more low-resolution frames also allows for higher precision sub-pixel scanning, as it increases the chance of the manual scanning being evenly distributed in all directions. Image compression artefacts from JPEG compression of the camera can affect the resolution in the final images. Raw pixel data from the sensor (if available) will be much larger in size and delay the image transfer and reconstruction. The knowledge of the exact pixel function of the image sensors can be used in the image reconstruction step to further improve the resolution²¹, but the image deconvolution process may be cumbersome within a smartphone device. Imaging parameters and additional processes typically impose trade-offs between image quality and processing time, thus should be carefully chosen for each imaging application.

Next, we used our smartphone microscope to image fresh water sample taken from a koi pond on Caltech campus. We took the sample directly from the pond, dispensed 20 μL over the image sensor (Supplementary movie M2). Because of the small volume of the sample, the liquid quickly dried within few minutes, leaving the particles on the image sensor. For faster sample preparation, we used a PDMS coverfilm to press the sample down. The images show various types of green algae in the pond water (Fig. 5). We compared the images with the conventional microscope images of the same sample (20× objective lens, 0.4 Numerical Aperture). Various species of Scenedesmus (typically Scenedesmus quadricauda and Scenedesmus acuminatus), which is one of the most common freshwater genera of green algae, are found in the images²².

Figure 5.

Portable microscopy of fresh-water microorganisms for water quality monitoring. (a) We took the fresh-water sample directly from a koi pond. 20-μL of sample was dispensed on the image sensor and the particles were let settle down for few minutes before image acquisition. (b–c) Reconstructed images of green algae found in the pond. (d) Conventional microscope images of same sample taken with 20× (0.4 Numerical Aperture) objective lens. The green algae found in the sample are different species of Scenedesmus, a genus of Chlorophyceae. All scale bars indicate 20μm

In the reconstructed images, the grid-type artefacts can be seen in thick samples. This results from the shadows of the parts of the sample in different height planes moving at different sub-pixel shifts under different illumination angle, which become out-of-focus upon reconstruction. These artefacts can further be suppressed by filtering the images to remove specific spatial frequency components.

Conclusions

We have reported on a smartphone-based chip-scale microscope using an ambient light source and the user’s hand motion for angular scanning. Our imaging scheme eliminates the need for lenses and the illumination source within the device, so that a microscope can be built through a simple modification of a camera module in a smartphone. As a proof-of-concept, we have constructed prototype systems on android smartphones by removing the lens-module in a smartphone camera and placing the samples on the surface of the image sensor. The image sensor captures direct shadow images of the sample while the user tilts the device around an external light source, such as the sun, a lamp or a flashlight. The corresponding sub-pixel-shifted shadows are analysed with vision processing and reconstructed into a high-resolution image via pixel-super resolution reconstruction. We have discussed both hardware modification as well as the development of the android application for image acquisition, analysis and reconstruction using OpenCV vision library. We have shown various images of microscopic samples, blood smear, microspheres and freshwater green algae, and demonstrated the imaging capability of our system. We achieved sub-micron resolution over an ultra-wide FOV in lensless and light source-less scheme. Our smartphone microscope features one of the most compact and simple designs among portable microscope devices developed to this date.

We believe that the advantages of this technique are its simplicity and robustness - two points of consideration that are important for resource limited application scenarios. Our present demonstration of a lensless imaging method that does not require an incorporated light source and that is able to make use of ambient illumination as a light source contributes significantly to the device simplicity. This work additionally demonstrates that the computation resource available on a smartphone is sufficient for the level of computation required by the SPSM for acquiring and generating high resolution microscopy images. The extent of modifications done to the smartphone is something that a hobbyist or a skilled educator can perform. This microscope offers a large field-of-view and high resolution imaging that does not require focus adjustment.

We do note that our prototype is a proof-of-concept demonstration but it does not present a straight practical method for broad usage. One way to make this technology practical to do the following. First, we would remove the image sensor module from the smartphone and replace it with a relevant circuit and an external connectorized port. We would then commercially fabricate robust printed circuit boards that host the sensor chips (sticks). These circuit boards can be connected to the smartphone via the connectorized port. To use the microscope, we simply place the samples on the stick. The stick can be cleaned and reused by simply soaking and washing them in a cleaning solution. This scheme offers several advantages. First, the stick can be made at volume cost-effectively and broadly distributed, as it only consists of the image sensor and printed wires running from the sensor to the port. In the field, the modified smartphone would be analogous to the microscope base, while the sticks would be treated as microscope slide replacements. A second advantage is that the smartphone modification is a simple one as the image sensor can be easily unplugged and swapped out. Third, this approach will avoid contamination of the smartphone by the samples. Finally, this approach nicely leverages the very finite (average of 21 months²³) life span of an average smartphone; an obsolete smartphone can be cost-effectively purchased and given a new lease of life as a microscope by this modification approach. We anticipate that this technology may represent a viable portable diagnostic method for perform imaging-based tests such as whole-blood cell counting and diagnosis and monitoring of blood-borne parasite infections, such as malaria and trypanosomiasis.

The ability to reuse the sticks between imaging experiments may be a critical cost factor for commercial and/or diagnostic applications. With our current prototype, we have been able to wash off wet samples with water and ethanol without damaging the sensor. Protective coatings and proper washing steps (both chemical and mechanical) can be developed to effectively clean the sensor surface repeatedly without damaging the sensor. In previous versions of our chip-scale microscopes, the image sensors were robust enough to be routinely reused after plasma cleaning and autoclave. We believe that the external sticks with more robust packaging and connections would allow for easier cleaning and reconnection of the sensors.

We envision that fluorescence imaging capabilities can be incorporated with the addition of a filter layer on the image sensor¹⁸. The ability to detect fluorescent stains and immunofluorescence can enhance the specificity of image-based diagnostic tests. In addition to imaging, detection of various immunoassays and genomic assays can be performed on-chip, providing an easier route for micro total-analysis-systems on a smartphone platform.

Finally, we note that direct shadow imaging scheme allows for integration of complex microfluidic systems on the smartphone without having to construct add-on devices. Small microfluidic channels can be attached on top of the image sensor, or more complex microfluidic systems can be designed to incorporate the image sensor in the part of the system where optical detection is required. The unprocessed direct shadow images still provide decent resolution (~2 μm) to image biological samples and/or microstructures. This type of modifications opens up the possible use of the technology in sophisticated bioscience experiments. The system model of a modified used smartphones that can interface with cost-effective mass-manufactured sticks applies well in this scenario as well.