Optoelectronic Tweezers Integrated with Lens-free Holographic Microscopy for Wide-field Interactive Cell and Particle Manipulation on a Chip

Abstract

We demonstrate an optoelectronic tweezer (OET) coupled to a lensfree holographic microscope for real-time interactive manipulation of cells and micro-particles over a large field-of-view (FOV). This integrated platform can record the holographic images of cells and particles over the entire active area of a CCD sensor array, perform digital image reconstruction to identify target cells, dynamically track the positions of cells and particles, and project light beams to trigger light-induced dielectrophoretic forces to pattern and sort cells on a chip. OET technology has been previously shown capable of performing parallel single cell manipulation over a large area. However, its throughput has been bottlenecked by the number of cells that can be imaged within the limited FOV of a conventional microscope objective lens. Integrating lensfree holographic imaging with OET solves this fundamental FOV barrier, while also creating a compact on-chip cell/particle manipulation platform. Using this unique platform, we have successfully demonstrated real-time interactive manipulation of thousands of single cells and micro-particles over an ultra-large area of e.g., 240 mm² (i.e. 17.96 mm × 13.52 mm).

Introduction

Approaches for noncontact manipulation of cells and microparticles play important roles in miniaturized systems for chemical and biological analysis.^{1, 2} Versatile manipulation mechanisms utilizing optical, electrical, magnetic, and acoustic forces have already been demonstrated in the literature.^3–5 Optoelectronic tweezers (OET) forms an approach that utilizes light-induced dielectrophoretic forces (DEP) for parallel cell manipulation using light images projected onto a low cost amorphous silicon coated glass.^6–8 OET allows users to reconfigure projected light patterns in real-time for dynamic and parallel manipulation of various biological, micro- and nano-particles including DNA molecules^{9, 10}, proteins¹¹, sperm cells¹², bacteria¹³, mammalian cells^14–17, polystyrene beads⁷, semiconductor microdisks¹⁸, nanowires¹⁹, carbon nanotubes^{20, 21} and droplets^{22, 23}. Phototransistor-based OET also enables cell manipulation in physiological buffers²⁴. Floating-electrode OET enables the manipulation of aqueous droplets in electrically insulation media²². Researchers have also explored different photoconductive materials in OET such as P3HT:PCBM, one of the promising organic solar-cell materials, and organic photoconductive materials such as titanium oxide phthalocyanine ^{25, 26}.

The operation principle of OET is based on light-induced dielectrophoretic forces on a photoconductive substrate. A light beam projected on an OET device optically generates extra electrons and holes to increase local photoconductivity, which creates light-patterned virtual electrodes and non-uniform electric fields to generate dielectrophoretic forces on cells and particles. Types of light sources that have been used for triggering and patterning virtual electrodes are versatile, from laser beams²⁷, DMD projectors⁷, LCD displays²⁸, to LED pixels²⁹. OET platform can provide ultra-large area manipulation (i.e., > 100 cm²) since the technologies required to fabricate large panel OET and large display panel for driving OET already exist in e.g., display industry.

Although large-scale manipulation has been achieved on OET platforms, most manipulations, however, are passive. It means projected light patterns are pre-programmed before knowing where the objects are. Examples of passive type OET manipulations include optical conveyers, concentric-ring cell concentrators, comb-shape particle sorters, scanning line sorters, and many others^{7, 15}. A major drawback of passive type manipulation is that it cannot selectively pick up target cells and micro-particles within a randomly distributed population since cell locations are unknown and no pre-programmed optical patterns can be generated in advance. On the other hand, there are also various active-type manipulation methods, where cells and beads are observed under a lens-based microscope, and light beams are projected to trap, transport, or pattern target cells identified based on their optical signatures such as size, texture, and fluorescence spectrum³⁰. The current major bottleneck of active manipulation in OET is that the small field-of-view (FOV) of objective lenses limits the imaging and manipulation throughput. In order to resolve the image of each individual cell, an objective lens with a magnification higher than e.g., 10× isrequired. The FOV of a 10× objective lens is typically less than 1 mm², and the FOV decreases with the increase of magnification. As a result, the number of cells that can be simultaneously observed and manipulated under a microscope is limited. This is one of the major technical barriers for parallel active cell manipulation on OET.

To overcome such FOV bottleneck, here we demonstrate a novel platform that integrates OET with lensfree holographic on-chip microscopy, an emerging technique that eliminates the need for an objective lens for ultra-large-area imaging. In this approach we utilize the lensfree holographic images captured on a CCD to reconstruct the microscopic images of cells and particles.³¹ This wide-field holographic microscopy technique has been demonstrated with numerous biological applications including blood analysis,³² waterborne parasite screening,³³ and male fertility testing.³⁴ Recently, deeply sub-micron resolution (e.g., ~0.3 μm),^{35, 36} and three-dimensional imaging and tracking capability^37–40 have also been demonstrated with this holographic imaging approach. Such a lensfree on-chip microscopy technique furnishes OET with a complementary detection and imaging function to enable large-area, programmable, and parallel light-induced DEP manipulation of microscopic particles or cells with real-time position feedback.

Methods

Device and experimental setup

The experimental setup and the schematic configuration of an OET platform integrated with lensless holographic microscopy are shown in Fig. 1. An OET device is directly placed on top of the protective glass of an image sensor array, either a CCD or CMOS imager. The OET device consists of two transparent indium tin oxide (ITO) plane electrodes with the bottom electrode coated with a 50 nm n+ a-Si:H and a 1 μm intrinsic a-Si:H as the photoconductive layer. Cells and particles in aqueous solution are sandwiched between these two electrodes separated by a 100 μm thick spacer. A 10 V_pp ac voltage at a frequency of 100 kHz is applied to the top and the bottom ITO electrodes to power the OET chip.

Fig.1.

Schematic diagram of the experimental setup of OET integrated with lensfree holographic microscopy. An OET device is directly placed on a CCD or CMOS sensor array. The sensor array coupled with a partially coherent light source creates an on-chip holographic microscope with a field of view that is equal to the sensor array active area. With the microscopic information of particles, interactive OET manipulation of target objects is achieved by projecting light beams next to the targets. A computer with a customized LabView program is used to control and synchronize different modules.

Cell and particle manipulation on OET is based on light induced dielectrophresis. In the dark area without light illumination, the electrical impedance in the a-Si:H layer is large and little voltage drops across the aqueous layer. No DEP forces are produced. When a light beam illuminates the a-Si layer, it decreases the local impedance in the a-Si:H layer by generating extra electrons and holes for electrical conduction. This creates a light-patterned electrode that shifts the major voltage drop to the liquid layer. This induces non-uniform electric fields and DEP forces to attract or repel cells and particles near a light spot. By moving a light beam, the position of the virtual electrode and thus the induced electric field pattern changes in real-time to achieve dynamic cell and particle manipulation.

A commercial DMD-based projector (MP730, BenQ Inc.) is used in our experiments to provide dynamic optical images required to trigger light-induced dielectrophoretic forces for parallel cell and particle manipulation. The original projection lens set and the color filter wheel on the projector are removed and replaced with custom-built optics to generate controllable optical patterns matching the active OET manipulation region on chip.

To integrate lensfree holographic microscopy on OET, an 850-nm LED coupled to an optical fiber is used to create a partial coherent light source. This infrared LED is chosen since its wavelength can penetrate through the 1 μm thick a-Si:H layer that is not transparent to visible light. The light source illuminates the OET chip from above with a vertical distance of 10 cm. The senor array used in the experiment is a charge-coupled-device (CCD) image sensor chip (KAF-8300, Kodak) with a pixel size of 5.4 μm and an active area of 17.96 mm × 13.52 mm. When the scattered light waves from objects interfere with the unperturbed waves from the LED light source, the holograms of objects are formed and recorded by the CCD. Because the detection area of this lensfree imaging system is only limited by the active area of a CCD or CMOS (Quickcam Pro 9000, Logitech) sensor array, this imaging technique eliminates the FOV limitation in conventional microscopes imposed by objective lenses, which can be as large as e.g., 18 cm².³⁶ Therefore, parallel monitoring of hundreds of thousands of cells using this detection approach can be achieved.

Detection and manipulation algorithm

To achieve automated, real-time, interactive manipulation of randomly distributed cells and particles across a large area, the communication and synchronization between OET manipulation and lensfree holographic imaging is critical. The lensfree imaging system is responsible for imaging, identifying, and tracking every individual cell in a population within the active detection area. The locations of target cells identified are sent to the OET manipulation system to generate optical beams to manipulate the target cells. The manipulation and imaging systems form a closed feedback loop system with real-time cell position identification and cell actuation functions. The manipulation functions can be versatile, depending on users’ need. It can be collecting target cells to one or multiple desired locations, trapping and transporting target cells to form desired patterns, and various others.

LabView (National Instrument, USA) is employed to develop this close-loop, automatic, and interactive manipulation in our experiments. It is used to control the image recording, the image analysis, and the generation of projection optical patterns for OET manipulation (Fig. 2). The loop process begins by capturing a holographic image from the CCD image sensor. The raw holographic images can be analyzed by two different methods to determine the positions of cells in every holographic frame: (1) A pattern-matching algorithm which compares the recorded holograms to a prepared texture library ⁴¹; or (2) a holographic reconstruction process that generates microscopic images of detected objects.⁴²

Fig. 2.

Process chart of the integrated close-loop OET manipulation and lensfree imaging system. One loop contains two periods, i.e., image analysis and OET manipulation. During the image analysis period, the system captures a raw image using a CCD or a CMOS sensor array to identify the target objects and their locations. Once the locations of the targets are determined, the OET system projects light beams to ₅ manipulate the selected targets to their final destinations.

Method (1) is based on the concept of correlation index which simply evaluates the similarity of a recorded lensfree hologram to a mean library image of a given object type. With the defined object library image L(x′,y′), a 2D deviation map (Dev(x,y)) can be calculated for the recorded image f(x,y) by:

$$\mathit{Dev}(x,y)=\sum _{(x^{\prime },y^{\prime }\in D_L)}|f(x^{\prime }+x,y^{\prime }+y)-L(x^{\prime },y^{\prime })|$$

where D_L represents the domain of the library image. After this step, the 2D deviation map is converted into a 2D correlation map for each object type by using:

$$\mathit{Corr}(x,y)=1-\frac{\mathit{Dev}(x,y)-{\mathit{Dev}}_{\mathit{MIN}}}{{\mathit{Dev}}_{\mathit{MAX}}-{\mathit{Dev}}_{\mathit{MIN}}}$$

where Dev_MAX and Dev_MIN refer to the maximum and minimum of the deviation values for f(x,y). After the creation of this 2D correlation map, potential candidates with large correlation peak values can be distinguished by setting a certain threshold value. The coordinates of the correlation peaks determine the positions of detected particles. The type of these particles can be identified by finding the maximum correlation after matching their holographic patterns with library images of various particle types. Further discussion on this detection and tracking related techniques can be found in the reference 41.

Different from the method discussed above, method (2) uses a digital algorithm to convert the lensfree holograms of micro-particles and cells back to their microscopic images before comparing them with library images. Two different levels, coarse and full image reconstruction, were conducted in our experiments for different purposes. To speed up the image analysis process during real-time manipulation, coarse reconstruction images (without twin-image elimination procedures³⁶) were used for rapid particle tracking during real-time manipulation. This level of reconstruction is achieved by, first, decomposing the complex field by fast Fourier transform (FFT) into components with various spatial frequencies on the CCD plane. Then each component is multiplied with its corresponding transfer function component to simulate the effect of wave propagation in a uniform medium back to the object plane. Lastly, the modified frequency components are converted into a complex field through an inverse-FFT operation representing the microscopic images of micro-objects in real space.

Fully reconstructed images without twin-image artifacts can be used for particle identification when the images obtained by the coarse procedures above are not sufficient. Full image reconstruction can be done by further refining coarse images with an iterative recovery process. More details of this holographic imaging technique can be found in reference 42.

Out of this reconstruction process, method (2) provides microscope-like images of objects (example Fig. 3C and Fig. 3D), whose coordinates are determined by calculating their centroids using the intensity profile of the reconstructed images. The type of detected objects can be identified by analyzing the morphological properties of their reconstructed images (e.g., size, intensity, and eccentricity). Since the images reconstructed by this method do not overlap as much as the holograms in method (1), this approach is especially suitable for analyzing samples containing higher density of micro-objects.

Fig. 3.

(A) A full field-of-view of the holographic image showing a mixture containing 10, 20, and 30 μm polystyrene beads. (B) The holographic image of the selected area in (A). (C) The coarse and (D) the full reconstruction images of the lensfree hologram shown in (B). (B), (C) and (D) show two different methods used to determine the locations of the objects. In (B), the locations of objects can be determined by matching the holograms with a library of lensfree images. In (C) and (D), the particle types are distinguished by their sizes, determined through their reconstructed lensfree images.

According to the object coordinates determined by either of the two methods discussed above, our custom-designed LabView program can compare new object coordinates with previous ones to update the object location database. In our system, the program only searches over the whole FOV for potential candidates in the first frame. Starting from the second frame, searching is limited to the neighbourhood of each target object identified in the previous frame. This searching strategy reduces the time for image analysis and allows rapid particle tracking during real-time interactive manipulation. Based on the location database and the desired final locations of objects, the program creates a projection light pattern to induce DEP forces on target micro-objects (e.g., particles or cells) over a period of time. Target micro-objects will be moved toward the preset locations during the time when projection light illuminates. The same procedures repeat themselves after the acquisition of each holographic frame until all target particles are transported to their final desired locations.

Results and Discussions

Recognition and tracking of micro-objects

To test the performance of our system, polystyrene particles with diameters of 10, 20, and 30 μm were mixed and suspended in an aqueous medium with electrical conductivity adjusted to 0.01S/m using potassium chloride (KCl). This mixture was introduced into the OET device placed onto a CCD array (see Fig. 1). An example holographic image of this particle mixture over a FOV of 17.96 mm × 13.52 mm is showed in Fig. 3A. Fig. 3B shows the raw hologram of a small region in the full FOV. Target particles can be identified by matching each particle’s hologram with a mean library image established by averaging 20 images of target type of particles (recorded with the same wavelength at the same depth position). Fig. 3C and Fig. 3D compare the image qualities obtained by the coarse and full reconstruction methods. The 10, 20, and 30 μm particles can be clearly recognized by directly measuring their sizes in their corresponding reconstructed images. The centroids of particles were determined by our custom-designed LabView program. These results illustrate that our holographic detection process can successfully separate nearby particles even when their lensfree hologram textures overlap with each other. For example, two holograms of 20 um particles shown in the circled area in Fig. 3B overlap and could not be identified by the direct hologram pattern-matching method, but could be separated and identified in Fig. 3C and 3D using digital reconstruction procedures.

Once objects were recognized as targets, the system followed a pre-defined strategy (Fig. 2) to create light spots to manipulate target particles in parallel. In OET manipulation, cells or micro-particles can be actuated either by positive or negative DEP forces, depending on particles’ relative polarizabilities compared to the surrounding media. Particles experiencing positive DEP forces are attracted to strong electric field regions, meaning moving towards light spots. On the contrary, particles with negative DEP forces are repelled by light spots. Micron-sized polystyrene particles typically experience negative DEP forces in aqueous media.

Interactive manipulation with lensfree hologram pattern matching method

An example of parallel interactive manipulation of 901 20-μm particles randomly distributed across a 17.96 mm × 13.52 mm area using the direct hologram pattern matching method is illustrated in Fig. 4A. Part of the image is enlarged in Fig. 4B to show that every recognized particle is pushed to move in one direction (pointed by arrows) by dynamic light patterns projected on the OET chip.

Fig. 4.

(A) A full field-of-view holographic image showing parallel interactive manipulation of 901 target particles. (B) The zoom-in image of (A). (C)–(F) Snapshots of sequential holographic images demonstrating that the selected particles in a random population can be manipulated to form an elliptical pattern. These particles were recognized through pattern matching within the first frame. The arrows show the directions and final destinations that the target particles are programmed to move toward. The light spots in (C) and (D) show the OET system tracing and manipulating the target micro-objects. After 122 seconds, most of the targets were moved to their desired destinations. The color images were recorded by the CMOS sensor array.

Fig. 4C to Fig. 4F are the snapshots of holographic images showing that a group of selected 20-μm particles were first recognized, then programmed to move to target locations, and eventually formed an elliptical pattern (Fig. 4F). These particles were recognized by pattern matching method in the first cycle. The time for analysis of the entire FOV is ~6 sec on the current setup. Starting from the second cycle, particle tracking is achieved by searching the neighbourhoods of objects identified in the first cycle, which reduced the feedback control cycle time to ~2.0 sec. The whole process from Fig. 4C to 4F takes ~122 seconds. The average particle speed during this manipulation process is ~25 μm/sec. The square light spots shown in Fig. 4D and 4E are for visualization purposes. In real operation, the projection light patterns will be automatically turned off during the image acquisition step to avoid perturbing the holographic patterns recorded on the sensor array.

In our platform, the particle-moving speed can be affected by several factors. First, it is affected by the light intensity of optical patterns projected on the OET chip. Light intensity controls the on-off status of virtual electrodes on OET. Second, the relative distance between the projected light spots and the centroid of holograms is also critical. The edge of a virtual electrode usually has the highest electric field gradient to generate larger DEP forces on particles. The fastest particle moving speed is achieved by always positioning target particle at the edge of a virtual electrode during the dynamic moving process (Fig. 4D and 4E). Third, the ratio of the image analysis period to the OET manipulation period in each cycle is another factor affecting particle speed. Image analysis period includes image recording time and the computational time for particle tracing (Fig. 2). The total computational time required for tracking targets is proportional to the area of the search and the number of targets. From Fig. 4C to Fig. 4F, the system monitored the neighbourhood of 20 target objects, which reduced the image analysis time from ~6 sec to ~0.5 sec, giving a relatively long manipulation period in each feedback cycle. If the manipulation period is long, a particle moves a longer distance in each loop, which increases the range of searching neighborhood for tracking an individual particle. This could increase the time required for image analysis and the chance of losing the particle. In general, to ensure faster particle movement and lower chance of losing particles, hardware performance and particle recognition algorithm are both critical to minimize the image analysis time in each cycle.

Interactive manipulation with lensfree holographic image reconstruction method

An example of real-time interactive manipulation using the image reconstruction approach for particle recognition is shown in Fig. 5. Two different sizes of polystyrene beads, 10 μm and 30 μm, were mixed in a KCl solution with conductivity adjusted to 0.01 S/m. In this manipulation example, the system is programmed to select some 30 μm beads, marked with squares in Fig. 5B and Fig. 5C, from this mixture and transport them to desired locations. Image reconstruction was performed in every raw holographic image during the manipulation. Two different levels of image reconstruction were conducted in the experiment. One is full image reconstruction that takes longer computation time (implementing twin-image elimination³⁶) but yields more spatial details for each particle and can be performed in the first cycle of manipulation. Once target particles are identified, the image reconstruction function switches to the coarse mode, which requires shorter computation time, for rapid particle tracing and gives a longer period for particle movement in a cycle during the dynamic manipulation process. Although the coarsely reconstructed image exhibit twin-image artefacts, it is sufficient for particle tracing application. To ensure that the system does not lose track of target particles with the coarse approach, full reconstruction can always be performed during or at the end of the manipulation cycle. The average speed in Fig. 5 is ~1.3μm/sec. In this experiment, the average loop time is ~2.0 sec and the image analysis period is ~1.5 sec. The reconstruction time for a coarse image is ~1.2 sec, which increases the image analysis period in each cycle and reduces the average velocities. The velocity can be improved by adjusting the ratio of the image analysis period to the particle manipulation period.

Fig. 5.

(A) A full field of view of reconstructed image with a mixture of 10 and 30 μm particles. (B) A digitally zoomed reconstructed image. The squares show the initial positions of the three selected 30-μm particles. The circles point to 10-μm particles. 30-μm particles were selected to move to the final destinations marked by crosses. The inset 1 and 2 are the enlarged images of 30 and 10-μm particles respectively. The scale bars are 90 μm. (C) The final positions of the particles after OET manipulation.

Cell manipulation

Human ovarian cancer cells (HeLa cells) were also used in this work to verify the platform’s ability to manipulate live biological cells. HeLa cells were cultured in Dulbecco’s modified Eagle’s medium (DMEM), supplemented with 10% fetal bovine serum (Hyclone), 50 mg/ml penicillin/streptomycin, and 50 mg/ml L-Glutamine. During the experiment, the cells were first washed with phosphate buffered saline (PBS, pH7.4) and then dissociated using 1X trypsin. After washed and neutralized with PBS, cells were re-suspended into isotonic buffer (8.5% sucrose [wt/vol], 0.3% dextrose [wt/vol]). The conductivity of the cell suspension was adjusted to 100 μS cm⁻¹ in this isotonic buffer. The cell suspension was injected into the OET chip and placed on the system for characterization. To determine the locations of HeLa cells and distinguish them from other micro-objects in the chamber, the holographic textures of potential candidates were matched to a mean library cell image (Method (1) – see the Methods section).

Fig. 6 shows the sequential holographic images of a single HeLa cell been dragged towards a desired location during real-time feedback manipulation. Unlike polystyrene beads, HeLa cells experienced positive DEP force, so the cell hologram followed the projection light spot. The LabView program kept on updating the location of the cell to determine the position of the new projection spot. Therefore, the cell closely followed the designed path (defined by the light spot) toward the target destination. The average velocity of the HeLa cell is ~ 4.2μm/sec.

Fig. 6.

Sequential holographic images of a single HeLa cell attracted by a projected light spot and dragged toward the desired location.

Conclusions

We have demonstrated an optoelectronic tweezers (OET) platform integrated with a lensfree holographic microscope for large area, real-time, interactive, parallel manipulation of single cells and micro-particles. Lensfree imaging solves the fundamental small field of view limitation in conventional interactive manipulation in OET system. It can provide real-time position feedback of hundreds of thousands of cells randomly dispersed across a large area. In this integrated system, we have accomplished real-time interactive manipulation over an area of 2.40 cm², which is two orders of magnitude larger than conventional microscope based OET platforms. Methods of automated pattern matching and image reconstruction have been demonstrated for monitoring thousands of objects inside our imaging and manipulation FOV. Real-time closed-loop manipulation of polystyrene particles and human ovarian (HeLa) cells has been successfully demonstrated using this high-throughput holographic on-chip microscope-integrated OET system.