Phasor-based widefield FLIM using a gated 512×512 single-photon SPAD imager

Abstract

Single-photon avalanche diode (SPAD) imagers can perform fast time-resolved imaging in a compact form factor, by exploiting the processing capability and speed of integrated CMOS electronics. Developments in SPAD imagers have recently made them compatible with widefield microscopy, thanks to array formats approaching one megapixel and sensitivity and noise levels approaching those of established technologies. In this paper, phasor-based FLIM is demonstrated with a gated binary 512×512 SPAD imager, which can operate with a gate length as short as 5.75 ns, a minimum gate step of 17.9 ps, and up to 98 kfps readout rate (1-bit frames). Lifetimes of ATTO 550 and Rhodamine 6G (R6G) solutions were measured across a 472×256 sub-array using phasor analysis, acquiring data by shifting a 13.1 ns gate window across the 50 ns laser period. The measurement accuracy obtained when employing such a scheme based on long, overlapping gates was validated by comparison with TCSPC measurements and fitting analysis results based on a standard Levenberg-Marquardt algorithm (>90% accuracy for the lifetime of R6G and ATTO 550). This demonstrates the ability of the proposed method to measure short lifetimes without minimum gate length requirements. The FLIM frame rate of the overall system can be increased up to a few fps for phasor-based widefield FLIM (moving closer to real-time operation) by FPGA-based parallel computation with continuous acquisition at the full speed of 98 kfps.

1. INTRODUCTION

Fluorescence lifetime imaging microscopy (FLIM) is an important tool in life sciences, including biophysics and biochemistry¹. Its advantages compared to fluorescence intensity imaging include the possibility to reject short-lived excitation light background by time-gating, the relative insensitivity of lifetime to fluorophore concentration or medium thickness, and the sensitivity of some fluorophores to oxygen and pH levels making it a tool for molecular environment sensing^2–5. To capture fast phenomena in live cell imaging, real-time fluorescence lifetime (FL) image generation is essential. To achieve this, both data acquisition and analysis need to be performed at high speed. High-speed widefield data acquisition with a single-photon imager requires low dead time and fast readout.

Another imager feature essential for FLIM is time-resolved operation. The conventional time-to-digital converter (TDC) based architectures offer high photon efficiency thanks to the high timing resolution of state-of-the-art TDC’s. However, these architectures limit the miniaturization of the pixel due to the relatively large silicon footprint of TDCs. Currently, few scalable options to implement TDCs exist for array formats approaching megapixel scale. An alternative approach to in-pixel TDCs is the sharing of TDCs between groups of pixels⁶. While this approach mitigates the area constraints, it imposes a dead time in the group of pixels sharing the TDC for large arrays. Time-gating is another method that addresses the scalability issue of TDCs: a gate can be added to a pixel with a single transistor, which has minimal effect on its size. However, the transmission of a square-shaped gate signal across a large array is non-trivial in 2D CMOS technologies. Therefore, the typical gate sizes for large-format imagers are longer than the timing resolution of a TDC.

Last but not least, fast data processing requires algorithms using functions that are not computationally intensive and/or can be efficiently parallelized.

This work aims to achieve real-time widefield FLIM by combining a fast, large format single-photon imager with a lifetime extraction method that is implemented by simple arithmetic operations. SwissSPAD2, an array of gated single-photon avalanche diodes (SPADs) with 97.7 kfps binary readout is the latest in this line of research⁷. The phasor method was chosen for lifetime extraction due to its use of only elementary linear operations suitable for fast data processing^8,9 Measurements were performed several fluorescent dyes characterized by single-exponential decays, using a series of 13.1 ns-long overlapping gates, translated with steps as short as 17.9 ps. To assess the feasibility of fast FLIM, we report the dependence of various FLIM performance metrics on gate number. We conclude that it is possible to achieve photon-efficient FLIM with less than 15 gate positions, which, when using FPGA-based parallel data processing which allow real-time operation.

2. PHASOR-BASED FLIM METHOD

The FLIM method that we used in this work is summarized in Figure 1. To capture timing information, a gate that is longer than the typical lifetimes to be measured (Figure 1(a)), is used for a specific integration time. The gate position, which can be adjusted with fine steps of 17.9 ps, is repeatedly shifted across the laser period T in order to obtain a “gated” fluorescence decay profile at each pixel. Gate positions t_i are associated to phasors with 2D coordinates, g_i = cos(2π t_i/T) and s_i = sin(2πf t_i/T), which are positioned on a unit circle with equal steps, as shown in Figure 1(b), and weighted proportionally to the recorded intensity in each gate, I_i. In other words, the phasor of the fluorescent decay is calculated as the average of the g and s coordinates of all individual gate phasors. The phasor of a single exponential decay is located on a semicircle in the first quadrant of the coordinate plane. The position of the phasor on the semicircle determines its lifetime: it gets closer to the origin as its lifetime increases (Figure 1(c)). The lifetime of the single-exponential decay is extracted from the phasor using the phase lifetime equation⁹:

$$\tau =\frac{1}{2\pi f}\left(\frac{s_{avg}}{g_{avg}}\right)$$ (1)

Figure 1.

Conceptual description of the phasor-based FLIM with long, overlapping gates. (a) A long gate is scanned across a fluorescent decay, in successive exposures, with fine steps that are significantly shorter than the gate length. (b) The phasors corresponding to time delays assigned to each gate positions are distributed along a unit circle in the (g, s) coordinate plane. The phasor Pavg, calculated from the average of all plotted phasors, weighted by each gate’s intensity is located close to the universal semicircle for single-exponential decays⁹. (c) The lifetime is extracted using Equation (1) from (ϕ), the angle of the line segment connecting the origin to the phasor⁹.

3. HARDWARE SPECIFICATIONS

3.1. SPAD and Imager Architecture

SwissSPAD2 is a time-gated SPAD imager with 512×512 pixels⁷. It consists of two identical, vertically mirrored, independently operated 512×256 sub-arrays, which form a square array with uniform spacing. Among the two sub-arrays, only one was connected for these experiments. Its produces a binary image every 10.2 μs, at a 97.7 kfps frame rate. The pixel pitch is 16.38 μm and the native fill factor is 10.5%. The SPAD used in the imager has one of the best photon detection probability (PDP) and dark count rate (DCR) combinations among the SPADs fabricated in standard CMOS technologies^10,11. The comparison of SwissSPAD2 with other large-format CMOS SPAD imagers is presented in Table 1⁷.

Table 1.

State-of-the-art comparison of large-format SPAD image sensors⁷

	This Work	12,13	14	15	16	17	18
Process Technology	180 nm CMOS	130 nm CIS	350 nm HV CMOS	350 nm HV CMOS	130 nm CIS	130 nm CIS	350 nm
Array Format	512×512 (472×256 accessible array)	320×240	160×120	32×32	256×256	256×256	512×128
Pixel Pitch	16.38 μm	8 μm	15 μm	25 μm	8 μm	16 μm	24 μm
Fill Factor (Nominal)	10.5%	26.8%	21%	20.8%	19.6%	61%	5%
Fill Factor (Microlens)	26.5–46.9% (Preliminary Results19)	50%	-	-	-	-	60%
Chip Size	9.5×9.6 mm2	3.4×3.1 mm2	3.42×3.55 mm2	-	3.5×3.1 mm2	5×5 mm2	13.5×3.5 mm2
Max. PDP	~50% @520 nm (Vex = 6.5 V)	39.5% @480 nm (Vex = 1.5 V)	-	-	-	39.5% @480 nm (Vex = 3 V)	46% @490 nm (Vex = 4.5 V)
Median DCR	7.5 cps/px 0.26 cps/μm2 (Vex = 6.5 V)	47 cps/px 2.7 cps/μm2 (Vex = 1.5 V)	580 cps/px 12.3 cps/μm2 (Vex = 2.5 V)	500 cps/px 3.8 cps/μm2 (Vex = 5 V)	50 cps/px 4.0 cps/μm2 (Vex = 2 V)	6.2 kcps/px 40 cps/μm2 (Vex = 1.5 V)	366 cps/px 12.7 cps/μm2 (Vex = 4.5 V)
Readout Noise	0	0.168e−	-	-	-	Negligible	0
Uniform SPAD Pitch	Yes	No	No	-	-	No	Yes
Max. Frame Rate	97.7 kfps (1 bit)	16 kfps (1 bit)	486 fps (5.4 bit)	50 fps (1.5 V analog)	4 kfps (3-bin histogram)	100 kfps (1 bit)	156 kfps (1 bit)

3.2. Imager Operation

A pixel in the camera consists of a 1-bit digital memory and a gate. An electrical pulse is generated by an incoming photon and stored in the memory only if the gate is open. The global gate signal generated by an FPGA can be synchronized to the laser pulse for timing information, or the gate can be kept permanently open for maximum sensitivity in intensity imaging. After a fixed number of laser pulses, the 1-bit information in the entire array is transferred row-by-row. An 8-bit photon count for a given gate position requires repeating this process 255 times. During the transfer of an 8-bit frame to the PC via USB 3.0 link, the next 8-bit frame is accumulated in FPGA-based counters in a pipelined scheme. Because a pixel can store only one photon between two subsequent readout events of the same row (10.2 μs at the fastest speed), 1/10.2 μs = 97.7 kHz is the maximum photon count rate per pixel that the sensor can sustain. However, in practice, such high count rates (or even lower count rates) result in a number of photons being undetected because of the finite (1-bit) capacity of the pixel counter, requiring pile-up correction to be applied¹⁰.

4. PHASOR-BASED FLIM PERFORMANCE

In this section, we describe the phasor-based FLIM experiment that was performed with SwissSPAD2 on multiple fluorescent dyes as well as quantum dots. The purpose of the experiments was to assess the ability to measure lifetimes as a function of signal level, and to investigate the minimum resolvable lifetime difference using the phasor plot.

4.1. Samples

Four fluorophores were used in this experiment. Cy3B (literature τ = 2.8 ns), ATTO 550 (literature τ = 3.6 ns), and Rhodamine 6G (literature τ = 4.08 ns) solutions were used for spatial non-uniformity measurement, while the quantum dot (QD585) solution was used for lifetime imaging. Due to concentration, extinction coefficient and quantum yield differences, the recorded signal intensity for the four samples were quite different, which allowed an analysis of lifetime precision dependence on photon counts.

4.2. Gate Characterization and Photon Efficiency Analysis

To characterize the performance of the phasor method for long and overlapping gates, FLIM was performed on two fluorescence dye pairs with different total photon count and lifetimes. For each sample, a 13.1 ns long gate was stepped in 17.86 ps increment over the entire 50 ns laser period, corresponding to 2,800 gate positions. The purpose of this test was to investigate the dependence of lifetime accuracy and precision on the number of gates. To achieve this goal, smaller data sets with fewer gates (than the original 2,800) were obtained by decimating the original data sets. The evolution of the accuracy and precision of the extracted lifetime as a function of gate number is shown in Figure 5(c–d). In this test, the 472×256 accessible pixel array was binned in groups of 4×4, in order to increase signal-to-noise ratio (SNR), making the effective spatial resolution 118×64. The average lifetime and standard deviation (σ) correspond to the extracted lifetimes of 7,552 binned pixels. With gate numbers as low as 14, no significant change was observed for the accuracy (Figure 5(c)), whereas the dispersion increased with decreasing number of gates (Figure 5(d)).

Figure 5.

(a) Characteristics of the gate used in the FLIM experiment. (b-e) Total photon count, average lifetime, standard deviation over the array (assuming spatially uniform lifetime), and F-value of ATTO 550 and R6G FLIM measurements. A 472×256 accessible sub-array of SwissSPAD2 was binned in groups of 4×4 pixels. IRF compensation was performed on a pixel-by-pixel basis using Cy3B solution as a reference dye. Its measured lifetime with a TCSPC system, 2.5 ns, was used instead of the literature lifetime of 2.8 ns.

While the standard deviation is useful for finding the resolving power of a FLIM experiment for a particular lifetime pair, another figure of merit of interest is the photon efficiency, which is defined as the precision of the extracted lifetime for a given number of detected photons. To determine the evolution of photon efficiency with varying number of gates, a figure of merit that is insensitive to the number of detected photons, the F-value, introduced by Gerritsen and collaborators²⁰, was used. The F-value quantifies the discrepancy of the measured lifetime dispersion with respect to the ideal situation of pure Poisson distribution. While it is ignoring the details of the data reduction process, it remains a good indicator of photon efficiency²¹. An F value of 1 means that the precision of the measurement is limited purely by the number of detected photons (shot noise), the theoretical minimum. As more sources of uncertainty contribute to the dispersion, the F-value of the measurement may increase. Figure 5(e) shows that the photon efficiency of ATTO 550 remains constant for gate numbers down to 14, whereas a degradation in the photon efficiency is observed in R6G with increasing gate number. The F-value calculated here includes the spatial non-uniformities of the SPAD imager, since the standard deviation used to obtain the F-value is computed over the whole array (and not over consecutive measurements in time). The behavior of ATTO 550 indicates that the change in precision between multiple data sets is essentially due to changes in the number of photons, not in the number of gates. If the number of photons increases continuously, eventually other sources of uncertainty start dominating the dispersion. In that case, the standard deviation remains constant and thus the F-value increases with higher number of photons. This behavior is visible in Figure 5(e) in the case of R6G. By contrast, the lower photon count in the ATTO 550 sample results in shot noise limited results throughout the range of gate numbers.

The accurate extraction of lifetimes shorter than the gate length requires characterization of FLIM performance dependency on the gate length. This investigation is beyond the scope of our work, and was performed in Ref.²².

4.3. Fluorescence Lifetime Dependency on Intensity

Figure 6 shows the comparison of intensity and lifetime images of a dried QD585 solution on glass. Quantum dots are known to be characterized by multiple lifetimes, and also by some residual size distribution resulting in additional dispersion of their photophysical properties²⁴, therefore the relative inhomogeneity in average phase lifetimes noticeable in Figure 6(b) is not surprising. Dependence of photophysical properties of quantum dots on excitation intensity has been reported before²⁵, therefore the relatively weak cross-correlation displayed in Figure 6(d) could be as signature of a weak dependency of the lifetime on fluorescence intensity. Said otherwise, a visual comparison of Figure 6(a) and (b) shows that while the intensity difference due to the excitation power is not as pronounced in the lifetime image, the contrast due to varying quantum dot concentrations exists in both images. The exact nature of this correlation will be further investigated in the future.

Figure 6.

(a) Fluorescence intensity and (b) fluorescence lifetime images of quantum dot (QD585) deposited on a glass surface. The whole field of view is 65.7 × 48.1 μm. (c) Phasor distribution of the pixels in (b). (d) Cross-correlation map between the intensity and lifetime images. Normalized 2-D cross-correlation function of MATLAB was used for the construction of this map. This function is based on the fast normalized cross-correlation formula in Ref.²³.

5. CONCLUSIONS

This paper demonstrates phasor-based widefield FLIM using a time-gated, large format SPAD imager. The imager, SwissSPAD2, consists of a 512 × 512 array of pixels with 16.38 μm pitch. The core of the pixel is a SPAD with one of the best DCR and PDP combinations among standard CMOS-based SPADs. Using microlenses, its 10.5% native fill factor has been recently improved by a factor of 2.5–4.5, depending on the incident angle of the photons¹⁹.

Phasor-based FLIM was used as the lifetime extraction method, due to its use of linear operations only, potentially enabling real-time processing, and because of the simple representation of lifetimes it offers, which allows qualitative visual assessment⁹. Using an integrated time-gate width of 13.1 ns and minimum step of 17.9 ps, single-exponential FLIM analysis was performed on ATTO 550, Cy3B, Rhodamine 6G and quantum dot (QD585) samples. The accuracy and photon efficiency (F-value) were preserved for number of gates ranging from 2,800 to 14, which potentially paves the way for real-time FLIM.

Future work will extend this analysis to FPGA-based real-time data processing and performance analysis of multi-exponential fluorescent dye mixtures.

Figure 2.

(a) Die micrograph and block diagram of the SwissSPAD2 image sensor⁷. (b) PDP characterization of the p-i-n SPAD¹⁰. (c) Timing resolution characterization of the p-i-n SPAD¹¹. (d) PDP and DCR state-of-the-art comparison of SPADs fabricated in standard CMOS process^7,11.

Figure 3.

(a) Timing diagram of the gating function of SwissSPAD2. (b) Conceptual illustration of the gating operation. The gate is shifted by Δt after the integration of the previous gate position. The minimum value of Δt is 17.86 ps.

Figure 4.

Gated fluorescence decay (gate duration: 13.1 ns) of (a) ATTO 550, (b) Cy3B, (c) Rhodamine 6G (R6G), and (d) quantum dot (QD585) solutions. The fluorescent samples differ in lifetime and total photon count.