Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Dec 1.
Published in final edited form as: Microsc Res Tech. 2015 Oct 24;78(12):1075–1081. doi: 10.1002/jemt.22587

Using of a modulated CMOS camera for fluorescence lifetime microscopy

Hongtao Chen 1, Gerhard Holst 2, Enrico Gratton 1,✉
PMCID: PMC4770881  NIHMSID: NIHMS725433  PMID: 26500051

Abstract

Widefield frequency-domain fluorescence lifetime imaging microscopy (FD-FLIM) is a fast and accurate method to measure the fluorescence lifetime of entire images. However, the complexity and high costs involved in construction of such a system limit the extensive use of this technique. PCO AG recently released the first luminescence lifetime imaging camera based on a high frequency modulated CMOS image sensor, QMFLIM2. Here we tested and provide operational procedures to calibrate the camera and to improve the accuracy using corrections necessary for image analysis. With its flexible input/output options, we are able to use a modulated laser diode or a 20MHz pulsed white supercontinuum laser as the light source. The output of the camera consists of a stack of modulated images that can be analyzed by the SimFCS software using the phasor approach. The non-uniform system response across the image sensor must be calibrated at the pixel level. This pixel calibration is crucial and needed for every camera settings, e.g. modulation frequency and exposure time. A significant dependency of the modulation signal on the intensity was also observed and hence an additional calibration is needed for each pixel depending on the pixel intensity level. These corrections are important not only for the fundamental frequency, but also for the higher harmonics when using the pulsed supercontinuum laser. With these post data acquisition corrections, the PCO CMOS-FLIM camera can be used for various biomedical applications requiring a large frame and high speed acquisition.

Keywords: FLIM, Fluorescence lifetime, frequency-domain, wide-field, PCO FLIM camera, CMOS, phasor plot, harmonic frequency

Introduction

Fluorescence lifetime imaging microscopy (FLIM) has become increasingly important in the biological and biomedical research, because it offers a contrast based on fluorescence lifetime due to differences in molecular microenvironments and molecular association, in addition to traditional fluorescence intensity and spectrum measurements [1–3]. The most common form of FLIM is done in laser scanning microscope systems, i.e. confocal [4, 5] and multiphoton [6–8] using a single channel detector. However typical laser scanning systems are slower in data acquisition and the overall setup needed for FLIM is expensive. Instead, widefield FLIM systems could provide higher imaging speed because of massive parallel frame acquisition [9–11], which is particularly useful for kinetic studies of lifetime changes occurring in a large area. Widefield FLIM can be implemented in two functionally equivalent ways: the frequency domain or the time domain [1]. In frequency domain FLIM (FD-FLIM) systems, the fluorescence response of repetitively excited samples varies with the same frequency of the excitation. The fluorescence emission at each pixel can be analyzed to determine the phase delay and the demodulation of the fluorescence emission with respect to the excitation. With a proper reference sample which has a known lifetime, the fluorescence lifetime of an unknown sample at each pixel can be determined. Currently, a fast time resolved camera system typically involves an image intensifier and a scientific CCD camera with complex controllers for radiofrequency and for setting the working point of the image intensifier. More importantly the resolution (temporal and spatial) of the camera-based system is limited by the image intensifier performance and spatial resolution which limits the applications of this technique.

Recently, new developments of high frequency modulated CMOS image sensors, which are originally designed for distance measurements, become potential candidates for FD-FLIM. The principle of these sensors shows a clear analogy to frequency-domain FLIM measurements, which has been successfully demonstrated [12, 13]. Based on this principle, the CSEM (Centre Suisse d’Electronique et de Microtechnique SA) and PCO AG developed a new CMOS image sensor, QMFLIM2. Each pixel of the image sensor has two charge collection sites, called tap A and tap B, and a switch between the taps, which can be controlled by an external high frequency signal. PCO then released the first widefield CMOS FD-FLIM camera, pco.flim, which is detailed by Franke et al [14].

Briefly, this camera has a format of 1008 × 1008 pixels with 5.6 μm pixel size. The quantum efficiency is up to 39% and the maximum frame rate is 90 frames/s in dual-tap mode. Most importantly, this sensor can be modulated up to 50 MHz. The onboard circuitry can generate the modulation frequency from 5 kHz to 50 MHz, but also accepts external modulation frequencies in the frequency range from 500 kHz – 50 MHz. Moreover, possible waveforms other than sine, such as square and pulse, allow the use of high harmonics for parallel multi-frequency lifetime measurements [15–20]. In the master mode, a laser diode can be modulated by the camera, whereas in the slave mode, the camera accepts the frequency of a pulsed excitation laser. Here a 20 MHz pulsed supercontinuum (A2, SC390, Fianium Inc, USA) white laser is used. It provides picosecond pulse trains with excitation wavelengths from UV to NIR (e.g. 390 nm to 2,000 nm). The pulse train is inherently composed of a number of harmonics at integer multiples of the pulse repetition rate. In addition, the pulse excitation waveform has a higher photon efficiency than the typical sinusoidal waveform in FD-FLIM as reported earlier in [17, 21, 22]. All these properties make it ideal for a fluorescence lifetime imaging of biological systems in the range from ns to ms.

We tested this system under various imaging conditions with external modulation and derived necessary corrections to be applied during the FLIM image analysis. The output of the stack of modulated images can be analyzed by the SimFCS software (http://www.lfd.uci.edu/globals/) using the phasor approach, which is becoming routine for the analysis of FD-FLIM due to its unique fit-free advantages as discussed previously [23]. We found that the system response across the image sensor is not uniform in the phase and modulation and thus a pixel level calibration is needed for all the fluorescence lifetime measurements. A dependency of the modulation value on the intensity at a pixel was also observed and hence a calibration is required since the modulation and the phase depend on the intensity level of the pixel. In the analysis of high harmonic frequencies, we noticed that these corrections are feasible not only for the fundamental frequency, but also for higher harmonics. With these post corrections, this CMOS FLIM camera can be used for various biomedical applications with the ease of setup and daily operation and without the need of recalibration.

EXPERIMENTAL SECTION

CMOS camera based Widefield Frequency-Domain Fluorescence Lifetime Microscope

The system setup is shown in figure 1. The CMOS camera based widefield FD-FLIM was constructed based on a commercial Olympus TIRF microscope (Olympus, USA). The excitation source (Figure 1A) can be either a laser diode (A1) which can be modulated at desired frequency by the FLIM camera, or a 20 MHz white-light supercontinuum laser (A2, SC390, Fianium Inc, USA), which offers a broad spectrum range from 390 nm to 2000 nm with an average spectral power density > 1 mW/nm. A heat blocker (KG-5, Edmund Optics, USA) was added into the excitation path to reduce the heat caused by the IR beam. A computer controlled filter-wheel (figure 1A) with eight bandpass filters was placed at the laser output to select a proper excitation wavelength range. The filtered beam was then coupled into the Olympus TIRF arm via an optical fiber (Figure 1B). After entering the microscope (Figure 1C) through the TIRF pathway, the excitation beam was reflected to the sample by a dichroic mirror (Chroma, USA). The fluorescence signal was collected by a 60× (NA=1.45) oil-immersion objective and passed the dichroic mirror. The bottom prism reflected this signal to the FLIM camera (Figure 1D).

Figure 1.

Figure 1

Open in a new tab

Schematic of the CMOS based widefield FD-FLIM system. A1) A 440 nm laser diode (ISS Inc, USA) which can be modulated by the FLIM camera. A2) A 10 ps, 20 MHz supercontinuum laser (SC390, Fianium Inc, USA). Bandpass filters were used to select proper excitation wavelengths. A heat blocker (KG-5, Edmund Optics, USA) was added into the excitation path to reduce the heat. B) The excitation beam was coupled into a commercial Olympus TIRF microscope (Olympus, USA) via the TIRF arm. C) Experiments were done with a 60× (NA=1.45) oil-immersion TIRF objective (Olympus, USA). The fluorescence signal was collected by the same objective. D) The bottom prism of the Olympus microscope reflected this signal to the pco.flim CMOS camera (PCO AG, Germany). The FLIM camera was controlled by a computer for modulation and data acquisition. Acquired phase images are analyzed by SimFCS software.

The camera was connected to a computer via a USB 3.0 interface for imaging acquisition and hardware settings. The camera is capable of transferring 90 fps at full frame size. In practice, the frame rate was limited to about 20 fps since we used 50 ms exposure per frame [14, 24]. The embedded FPGA performs all the necessary operations and greatly reduces the complexity for the system setup. The acquired phase images are then processed by the SimFCS software. During our testing, the exposure time was set to 50 ms and both taps were acquired, although only Tap A was analyzed in the data shown in this paper. The phase steps were 16 and we collected two full phase sequences for each cycle, i.e. 32 frames for one full frame lifetime measurement (1.6 s). For the data presented in this paper we repeated the full frame acquisition 8 times to increase the S/N. All data were graphically analyzed by the phasor method, which allows easy fit-free lifetime analysis of FLIM images.

RESULTS AND DISCUSSION

Background Characteristics and Calibrations of System Response at the Pixel Level

First, the background was acquired at the same camera conditions later used for the measurements, i.e., the camera is modulated but the laser is off. The averaged image after removing of S&P (salt and pepper) noise is shown in Figure 2A. We noticed a non-uniform distribution of the background pattern, which implies that background correction needs to be done at each pixel. The intensity at the edge is ~30% larger than at the center in the horizontal direction whereas there is about ~10% variation in the vertical direction (Figure 2A). Note that the sensor is not illuminated so that this difference in image intensity cannot be attributed to the microscope field of view. This difference could be due to effective variable exposure of the sensor at the borders of the image or to temperature differences at the borders of the sensor with respect to the center. The background pattern is reproducible and the background image will be subtracted from each images in all measurements reported in this paper.

Figure 2.

Figure 2

Open in a new tab

Calibration images of background and phase and modulation at the fundamental frequency and the 3rd harmonic. A) Average background image (the laser is off). B) Uniform intensity image used to calibrate the phase and modulation at each pixel. C) The phase image of uniform scattered laser at the fundamental frequency of 20 MHz. D) The phase image of the same sample at the 3rd harmonic frequency at 60 MHz. E) The modulation image of uniform scattered laser at the fundamental frequency of 20 MHz. F) The modulation image of the sample at the 3rd harmonic frequency of 60 MHz.

Second, we took a FLIM measurement of the scattered laser light via a diffuser to obtain a uniform illumination across the sensor (Figure 2B), which ideally should give a constant value of the phase and modulation across the image. After background subtraction, the phase and modulation images were calculated and are shown in Figure 2C and 2E, respectively. The phase and modulation for each pixel i of the image are calculated by the following expressions:

gi(ω)=∫0∞Ii(t)cos(ωt)dt/∫0∞Ii(t)dt=Mi×cos(ϕi)si(ω)=∫0∞Ii(t)sin(ωt)dt/∫0∞Ii(t)dt=Mi×sin(ϕi)

Note that the above formulas correspond to the phasor transformation, they require no fit and these formulas are easily extended to the higher harmonics as shown in Figure 2D and 2F.

Interestingly, although the sample was relatively uniform in intensity and in lifetime, the phase and modulation are not uniform across the image plane. The phase image is shown in figure 2C. In the horizontal direction, the difference is ~ 2 degrees from the edge to the center with a symmetric profile. In the vertical direction, the variation is ~ 14 degrees from the edge to the center with a non-symmetrical profile. The calculated modulation image in Figure 2E shows similar behaviors. In the horizontal direction, the difference is ~0.002 from the edge to the center whereas in the vertical direction, the difference is ~0.09 from top to bottom. The same result was also obtained at the 3rd harmonic frequency, i.e. 60 MHz. Figure 2D shows the phase image of the same sample at the 3rd harmonic (60 MHz). In the horizontal direction, it has the same behavior as the fundamental frequency but with a larger difference, which is ~6 degree from the edge to the center, while in the vertical direction, the difference from the edge to the center is ~45 degree. However the modulation image shows slightly different properties at the 3rd harmonic frequency (Figure 2F). In the horizontal direction, the profile is flat. In the vertical direction, the difference is ~ 0.08 from the top to the bottom.

These non-uniform distributions of phase and modulation of a “uniform” sample indicate that at each pixel, the system response is different. Hence calibration of system response needs to be done at every pixel using the uniform illumination method. We use these uniform intensity images to produce a “phase” and “modulation” correction image.

However, we noticed that if we change the intensity of the uniform image and we take the ratio of one image to the other for the modulation images, we obtain a pattern that is changing (quasi) linearly along the vertical direction (Figure 3) at both fundamental frequency and 3rd harmonic frequency. Using one image at a given intensity as a reference (“middle” intensity, 2350 counts), we measured this linear dependence for a range of intensities. We calculate the slope and offset of these linear dependencies as a function of the average image intensity and we built intensity dependent correction factors to be applied for each pixel of an image to the phase and modulation values. These factors are computed once forever and then stored to be used as correction factors for all FLIM measurements

Figure 3.

Figure 3

Open in a new tab

Ratio of two modulation images measured at two different intensities 2350 counts and 3500 counts, respectively. A) Modulation ratio at 20 MHz and B) Modulation ratio at 60 MHz. Red line is vertical direction and blue line is horizontal direction of the image

As an example, Figure 4 shows the phasor diagram of a series of images taken at different intensities corrected for the reference image and also with the additional correction that account for the intensity dependent phase and modulation correction. When these corrections are applied, the phasor plots are almost identical for all the intensities used in our calibrations.

Figure 4.

Figure 4

Open in a new tab

Phasor plots at 20 and 60 MHz corrected for intensity dependence of modulation. Middle intensity is the reference. Before correction: A) and B) Phasor plots at 20 and 60 MHz for high intensity. C) and D) Phasor plots at 20 and 60 MHz for middle intensity. E) and F) Phasor plots at 20 and 60 MHz for low intensity. After correction: for the modulation dependence on intensity and for the pixel position. G) and H) Phasor plots at 20 and 60 MHz for high intensity. J) and K) Phasor plots at 20 MHz and 60 MHz for low intensity.

In summary, in order to fully calibrate the phase and modulation response at each pixel, a uniform sample with a known fluorescence lifetime (scattering is OK) needs to be measured and calibrated at each pixel for all the frequencies that will be utilized and under the same camera settings. This is not particularly difficult, since one measurement of the image stack using a pulsed source already contains all the harmonic frequencies in parallel.

Sample Measurement

With the corrections discussed above (phase, modulation and intensity dependence of these parameters), the FLIM camera is ready for fluorescence lifetime measurement. All the measurements should be done under the same settings since the calibration and corrections are only valid under the same conditions, including the modulation frequency and the exposure time. As a demonstration, we measured cells transfected with Cerulean (donor only) and with a construct of Cerulean-Venus that shows FRET. A 438/24 excitation filter and a 475/35 emission filter for donor fluorescence were used together with the white laser. First, we calibrated phase and modulation of the system with coumarin 6 in ethanol, which has the known lifetime of 2.5 ns. Note that only one image is measured for the coumarin 6 sample since the intensity dependent modulation and phase was already obtained using the uniform image method previously described. Then 3 images of Cerulean-Venus transfected cells were obtained as well as 2 images of Cerulean only cells as shown in Figure 5A and 5B. The corresponding phasor images and phasor plot in Figure 5C, 5D and 5E clearly show the shortening of donor fluorescence lifetime under Cerulean-Venus construct.

Figure 5.

Figure 5

Open in a new tab

FLIM-FRET measurement: cells transfected with Cerulean (donor only) and cells transfected the Cerulean-Venus construct (FRET pair). A) Three different images of cells transfected with the Cerulean-Venus construct. B) Cells transfected with the Cerulean donor only. C) Selection of the phasors that correspond to the shorter lifetime of the FRET construct at donor fluorescence. D) Selection of the phasors corresponding to the donor only cells. E) The phasor plot of the 5 images (3 of FRET pair and 2 of donor only) shows a lifetime difference of donor fluorescence due to the FRET effect.

In Figure 6, the fluorescent convallaria sample was also measured and analyzed with corrections discussed above. Figure 6A shows the average intensity image of the sample. Figure 6B and 6D show the phasor image and the corresponding phasor plot at the fundamental frequency, i.e. 20 MHz, whereas Figure 6C and 6E show the phasor image and the phasor plot at the 3rd harmonic, 60 MHz. The fluorescence lifetime is obtained at 20 MHz and different lifetime components are well distinguished in Figure 6B. With 3rd harmonic frequency at 60 MHz, due to the high resolution properties, more detailed separation is displayed in Figure 6E.

Figure 6.

Figure 6

Open in a new tab

FLIM measurement of convallaria at the fundamental frequency (20 MHz) and the 3rd harmonic (60 MHz). A) the intensity image of the convallaria, scale bar = 20 μm. B) FLIM image and the phasor plot at 20 MHz in D. C) FLIM image and the phasor plot at 60 MHz in E.

CONCLUSIONS

We described here practical corrections and necessary processing details for FLIM images acquired by the first FLIM camera and external modulation based on a modulated CMOS sensor. The flexible coupling and easy setup makes this camera an ideal choice for the widefield FD-FLIM instruments. The use of a pulsed supercontinuum laser and the phasor analysis method further simplified the system construction and data analysis with capability of parallel multi-frequency detection. Although the FLIM camera only supports up to 50 MHz modulation directly, the flexibility of internal waveform control by FPGA or synchronization with external pulsed laser allows the use of high harmonic contents, for example 60 MHz as shown in this paper. Moreover the cross-correlation technique, which can minimize the effect of photobleaching and DC background [1, 25, 26], is also possible to be adapted in this camera.

Our results show that the calibration of the system response needs to be performed at the pixel level at each camera settings. This is a typical routine measurement in FD-FLIM experiments. The more complex correction is the intensity dependence of the measured phase and modulation. We have shown the basic procedure to obtained the intensity dependent correction factors. The correction can be stored by the operating program in the PC so that only the calibration with a compound of known lifetime is performed prior to each measurement, which is the routine procedure used in FLIM.

Acknowledgments

This work was supported by the National Institutes of Health (NIH)-P41 GM103540 and the NIH-P50 GM076516 grants and it was supported by German Federal Ministry for Education and Research (BMBF) by 13N9242 and 13N11177. Also, we would like to thank Milka Stakic for cell culturing and Robert Franke for a technical review of the camera part.

Footnotes

Conflict of interest statement

Doctor Gerhard Holst is an employee of the PCO Company that has provided the PCO-FLIM camera for evaluation for this work. HC and EG have no conflicts to disclose.

References

  1. Berezin MY, Achilefu S. Fluorescence Lifetime Measurements and Biological Imaging. Chemical Reviews. 2012;110(5):2641–2684. doi: 10.1021/cr900343z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Buurman EP, et al. Fluorescence lifetime imaging using a confocal laser scanning microscope. Scanning. 1992;14(3):155–159. [Google Scholar]
  3. Chen YC, Clegg R. Fluorescence lifetime-resolved imaging. Photosynthesis Research. 2009;102(2):143–155. doi: 10.1007/s11120-009-9458-7. [DOI] [PubMed] [Google Scholar]
  4. Clegg RM, et al. Methods in Enzymology. Academic Press; 2003. Fluorescence lifetime-resolved imaging: Measuring lifetimes in an image; pp. 509–542. [DOI] [PubMed] [Google Scholar]
  5. Dong CY, et al. Methods in Cell Biology. Academic Press; 2003. Fluorescence-Lifetime Imaging Techniques for Microscopy; pp. 431–464. [DOI] [PubMed] [Google Scholar]
  6. Elder A, Schlachter S, Kaminski CF. Theoretical investigation of the photon efficiency in frequency-domain fluorescence lifetime imaging microscopy. J Opt Soc Am A. 2008;25(2):452–462. doi: 10.1364/josaa.25.000452. [DOI] [PubMed] [Google Scholar]
  7. Elder AD, Kaminski CF, Frank JH. f2FLIM: a technique for alias-free frequency domain fluorescence lifetime imaging. Opt Express. 2009;17(25):23181–23203. doi: 10.1364/OE.17.023181. [DOI] [PubMed] [Google Scholar]
  8. Elder AD, et al. The application of frequency-domain Fluorescence Lifetime Imaging Microscopy as a quantitative analytical tool for microfluidic devices. Opt Express. 2006;14(12):5456–5467. doi: 10.1364/oe.14.005456. [DOI] [PubMed] [Google Scholar]
  9. Esposito A, et al. All-solid-state lock-in imaging for wide-field fluorescence lifetime sensing. Optics Express. 2005;13(24):9812–9821. doi: 10.1364/opex.13.009812. [DOI] [PubMed] [Google Scholar]
  10. Franke R, Holst GA. Frequency-domain fluorescence lifetime imaging system (pco.flim) based on a in-pixel dual tap control CMOS image sensor. Proc. SPIE 9328; 2015. [Google Scholar]
  11. Franke R. PhD Thesis. 2012. Hochintegriertes Kamerasystem für die Multifrequenz-Lumineszenzabklingzeit-bildgewinnung. [Google Scholar]
  12. Gratton E, BB Multifrequency phase fluorometry using pulsed sources: Theory and applications. Spectroscopy. 1986;1(6):28–36. [Google Scholar]
  13. Gratton E, et al. Fluorescence lifetime imaging for the two-photon microscope: time-domain and frequency-domain methods. Journal of Biomedical Optics. 2003;8(3):381–390. doi: 10.1117/1.1586704. [DOI] [PubMed] [Google Scholar]
  14. Grimm JB, et al. A general method to improve fluorophores for live-cell and single-molecule microscopy. Nat Methods. 2015;12(3):244–50. 3–250. doi: 10.1038/nmeth.3256. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Heβ H, Albrecht M, Schwarte R. PMD - New Detector for Fluorescence Lifetime Measurement. Proceedings Opto 2002; 2002; pp. 181–186. [Google Scholar]
  16. Holub O, et al. Fluorescence Lifetime Imaging (FLI) in Real-Time - a New Technique in Photosynthesis Research. Photosynthetica. 2001;38(4):581–599. [Google Scholar]
  17. Leray A, et al. Optimized protocol of a frequency domain fluorescence lifetime imaging microscope for FRET measurements. Microscopy Research and Technique. 2009;72(5):371–379. doi: 10.1002/jemt.20665. [DOI] [PubMed] [Google Scholar]
  18. Morgan CG, Mitchell AC, Murray JG. Prospects for confocal imaging based on nanosecond fluorescence decay time. Journal of Microscopy. 1992;165(1):49–60. [Google Scholar]
  19. Philip J, Carlsson K. Theoretical investigation of the signal-to-noise ratio in fluorescence lifetime imaging. J Opt Soc Am A. 2003;20(2):368–379. doi: 10.1364/josaa.20.000368. [DOI] [PubMed] [Google Scholar]
  20. Piston DW, Sandison DR, Webb WW. Time-resolved fluorescence imaging and background rejection by two-photon excitation in laser-scanning microscopy. Proc. SPIE 1640; 1992; pp. 379–389. [Google Scholar]
  21. Sanabria H, et al. Spatial diffusivity and availability of intracellular calmodulin. Biophys J. 2008;95(12):6002–15. doi: 10.1529/biophysj.108.138974. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Schlachter S, et al. mhFLIM: Resolution of heterogeneous fluorescence decays in widefield lifetime microscopy. Opt Express. 2009;17(3):1557–1570. doi: 10.1364/oe.17.001557. [DOI] [PubMed] [Google Scholar]
  23. So PTC, et al. Time-resolved fluorescence microscopy using two-photon excitation. Bioimaging. 1995;3(2):49–63. [Google Scholar]
  24. Spencer RD, Weber G. Measurements of Subnanosecond Fluorescence Lifetimes with a Cross-Correlation Phase Fluorometer. Ann NY Acad Sci. 1969;158:361–376. [Google Scholar]
  25. Squire Verveer, Bastiaens Multiple frequency fluorescence lifetime imaging microscopy. Journal of Microscopy. 2000;197(2):136–149. doi: 10.1046/j.1365-2818.2000.00651.x. [DOI] [PubMed] [Google Scholar]
  26. Wu Q, et al. Optical refocusing three-dimensional wide-field fluorescence lifetime imaging microscopy. Opt Express. 2012;20(2):960–965. doi: 10.1364/OE.20.000960. [DOI] [PubMed] [Google Scholar]

RESOURCES