A review of SPAD array chip design for direct time-of-flight LiDAR

Abstract

As the requirements of LiDAR for data processing volume, ranging accuracy, and anti-interference capability continue to rise, significant transformations have occurred in recent years in the system architecture, data flow design, and algorithm modules of LiDAR receiver chips. This paper reviews the current design status of direct Time-of-Flight (dToF) ranging sensor chips, focusing on three key aspects for systematic summarization: first, innovations in ranging methods, covering two novel technical schemes, namely histogram-free and variable histogram resolution; second, anti-interference and accuracy optimization, including the chip’s suppression of ambient light, mitigation of inter-LiDAR interference, and calibration methods for ranging accuracy; third, comprehensive design optimization of chip Power, Performance, and Area (PPA). Finally, several targeted improvement suggestions are put forward to address existing design bottlenecks, providing references for the subsequent research and development of ToF ranging sensor chips.

Introduction

Light detection and ranging (LiDAR) sensors have the advantage of better measurement resolution and more compact size over common ranging technologies like radar [1]. LiDAR in conjunction with the direct time-of-flight (dTOF) measurement principle has found increasing use in commercial applications requiring fast and accurate depth sensing over a large range of distances [2]. Examples are service and industrial robotics, automobile self-driving, security surveillance and virtual/augmented reality. Although in the past dTOF LiDARs have often employed linear photo-response avalanche photo diode or PIN diode [3], the trend is to favor single photon avalanche diodes (SPADs) owing to their high sensitivity, fast reaction time, low timing jitter, and importantly, superior CMOS integration compatibility [4]. A number of SPADs can be connected in parallel to form an analog silicon photomultiplier (SiPM) which output analog current pulse with intensity proportional to the number of concurrent photon hits [5]. Alternatively, the SPAD pulse signal can also be readout and processed digitally, using fast on chip timing electronics and arithmetic logic units to measure the round-trip time of the laser pulse and derive the depth information.

In applications where 3D (direction and depth) scene acquisition is required, using SPAD arrays or SiPM arrays allows the extraction of 3D information with fast acquisition time, wide FOV and flexible laser illumination scheme.

In comparison to a single SPAD, analog SiPM can directly output the overall received signal intensity, and its large detection area effectively reduces deadtime. However, analog SiPM can also suffer from a prominent long pulse tail caused by inter-SPAD crosstalk. It also requires dedicated analog processing and timing circuits to extract the TOF information [6]. In terms of CMOS compatibility, there have been reports on CMOS based single analog SiPM [6, 7], however reported cases using multiple analog SiPMs as the sensor array is still scarce [8]. On the contrary, a group of CMOS based SPAD array or digital SiPM array designs have emerged [1, 4, 9–14], tailored for dTOF measurement.

The most commonly adopted statistical measurement technique for SPAD based dTOF is the time correlated single photon counting (TCSPC). In a typical implementation, after a pulsed laser emission, the SPAD current pulse is first digitized in its analog front end (AFE) circuit. In the conceptually simplest implementation, each SPAD’s digit pulse is timestamped by a time to digital converter (TDC), and the time difference with respect to the timing reference signal is recorded and buffered. This process can be repeated multiple times, forming a histogram representing the time distribution of received light with respect to the laser emission time. The histogram is then analyzed in the digital signal processor (DSP) to extract the range information, and finally the point cloud data will be analyzed. A workflow chart of the SPAD signal processing is shown in Fig. 1.

Fig. 1.

Schematic of a typical CMOS SPAD dTOF signal chain [4]

The aforementioned workflow as depicted in Fig. 1 should be viewed as the baseline version of designing a SPAD array dTOF LiDAR chip. The goal of this review is to explore the existing and possible variants to this workflow, providing more implementation details with a focus on novel ideas and system level optimizations not confined to circuitry design. There are two main goals for chip design, which are better functionality and lower cost; and there are the three main performance parameters for LiDAR, the depth and angular resolution, the frame rate and the ranging distance. Improvement of those performance parameters arrives at a cost of increasing complexity and power(area) consumption of the system. In addition, mitigation methods against adversary effects, nominally high ambient light, pulse shape distortion, LiDAR inference, crosstalk between SPADs and adversary environment conditions, usually inflicts a reduction in the best possible main performance parameters. In spite of these downsides, the trend of the SPAD chip design is to integrate an increasing level of processing into the SPAD sensors to ultimately achieve a single-chip receiver solution, driven by the desire for a solid-state dTOF system with fast acquisition time, reduced system total power consumption, and more resilient against ambient light [4].

In order to address the multiple facets of the SPAD chip design optimization, the rest of this review is organized as follows. We start from exploring different levels of processing integrated in the SPAD in Sect. 2, as a walk through of the chip design architecture. Then a discussion of different ranging principles is presented in Sect. 3. In Sect. 4 countermeasures against ambient light are presented. Mitigation methods against another important potential risk factor, the inter-LiDAR interference, are discussed in Sect. 5. We then shift gears in Sect. 6, in which histogram correction algorithms for improving the measurement accuracy are reviewed. Finally, we dedicate Sect. 8 to chip area and power consumption optimization. We then conclude the review in Sect. 9.

Before ending this section, we shall comment on recent SPAD/LiDAR design reviews, and what new insight this review might offer. A review of common chip modules of SPAD imaging sensors is given in Ref. [15]. Ref. [16] reviews the biological applications of SPAD sensor, such as in multi-wavelength and Raman spectroscopy research. Ref. [17] provides a general review of data processing methods on Flash SPAD LiDARs rather than concentrating on chip design and implementation considerations.

In Ref. [18], the SPAD sensor design of existing commercial products and research prototypes is discussed, with a focus on product and function specification rather than technical solutions. Analog circuits design for LiDAR chips is reviewed in [19], covering SPAD sensor, TDC and signal processing circuits, while short of in-depth discussion on chip architecture optimizations. Ref. [20] reviews the SPAD and OPA based LiDAR integration from a system design perspective, however chip inner structural design is not thoroughly discussed. In [21] a review of silicon photonics LiDAR is provided while the focus is not on TOF LiDAR.

To summarize, this review is distinct from the aforementioned works, with a focus on 3d-stacked logic wafer SOC design, state-of-the-art innovations in dTOF ranging principles such as the histogram-free paradigms, mitigation of anti-interference and accuracy optimization algorithms, as well as the PPA-driven design optimizations.

SPAD sensor processing integration

Following Fig. 1, we shall categorize the SPAD sensor architecture according to the timing and histogramming integrated on the SPAD sensors. In the first case, the individual SPADs signal are readout directly, as shown in Fig. 2(Left) [9]. A complex combinational logic performs data suppression prior to sending the digitized waveform to flash TDCs which are then stored in ripple counters to form a histogram, which are all done outside of the SPAD sensor area.

Fig. 2.

(Left) A 32 by 32 SPAD array readout using combinational XOR trees into flash TDCs, and to histogram generating counters. (Right) A 256 x 128 SPAD array with 8 x 16 sub-modules each featuring its own shared TDC. A progress time gating and coincidence gating scheme is also applied to filter out background photons and minimize TDC pile-up effects

In the second case, photon timing circuit is integrated using TDCs serving one or multiple SPADs. One such example is a 256 x 128 SPAD array with 8 x 16 sub-modules each featuring its own shared TDC [22]. To avoid potentially high rate of TDC event collisions, a progress time gating as well as a coincidence gating scheme is also applied to filter out background photons.

A golden rule in SoC is to minimize wire-lengths and put interconnected components closer when possible, to minimize energy cost per operation and also improve operating frequency [4]. Therefore, to further shorten the path from SPAD to TDC and to histogram circuit, it is reasonable to have an architecture which generate histograms in-pixel(SPAD) or outside the SPAD array as column parallel logic. For the latter case, the transferring of large array data out of the SPAD pixel array can be challenging [23]. In-pixel histogramming avoids any bottleneck in transferring data outside of the SPAD array. One drawback of in-pixel histogramming is the limited pixel size restricted the achievable length of the histogram [24], therefore limits longer range applications.

To overcome the drawback of limited histogram length, partial histogramming approaches have been applied. Three trending partial histogramming techniques are illustrated in Fig. 3. The zooming method performs a peak bin identification and zooms into the corresponding time range with a fixed number of total bins. The sliding method, on the other hand, has fixed number of bins and scans over the total time region. Interestingly, a study [26] reveals that the sliding approach performs better in high background applications as the peak detection of the zooming method became unreliable. Finally, a tracking approach is recently proposed [25] building on the sliding approach, but rather having a ‘lock on’ peak detection logic as once a peak is identified from several consecutive partial histograms, as shown in red in Fig. 3, and the remaining time range is ignored. It also features an optional center-of-mass processor for sub-bin resolution. The entire partial tracking sensor and the close-up of the layout of each 4x4 SPAD pixels are shown in Fig. 4.

Fig. 3.

Three trending partial histogramming techniques [25]. The zooming method performs a peak bin identification and zooms into the corresponding time range with a fixed number of total bins. The sliding method, on the other hand, has fixed number of bins and scans over the total time region. Finally, a tracking approach is recently proposed building on the sliding approach, but rather having a ‘lock on’ peak detection logic as once a peak is identified from several consecutive partial histograms, as shown in red, and the remaining time range is ignored

Fig. 4.

The entire partial tracking sensor and the close-up of the layout of each 4x4 SPAD pixel as presented in Ref. [25]

Ranging principles

Here we shall briefly recall the measurement principle of TCSPC for dTOF LiDAR ranging and imaging; As shown in Fig. 1, the laser source emits a narrow width laser pulse while the RX unit records the laser echo signal from the object. The distance d from the object to the receiver can be inferred from the time delay between the emitted and received light, expressed as

1

where c is the speed of light. The rising (and falling) edge of the photo-detector pulse signal can be time-stamped as a TDC count, representing the detected photon arrival time. After multiple laser pulse emissions, a time histogram illustrating the photon arrival time distribution is accumulated in reference to the laser emission time and can be analyzed to yield distance and intensity information.

For 1D and 2D scanning LiDAR applications, multiple pixel arrays are required in order to perform live imaging of the observed scene, and the number of TDCs scales with the number of pixels, reaching the bottleneck of improving performance as need to overcome the multi-Gbits/sec level data throughput rate and the large power consumption [27]. In addition, the ranging resolution is dependent on the timing resolution. In order to achieve a balance between cost, ranging resolution and power consumption, a widely adopted strategy is to use a combination of coarse and fine time binning.

A plethora of smart design features are proposed, notably are (a) use a coarse global clock cycle until a trigger signal is emitted, then starts the fine TDC in a short data acquisition window [13]; (b) use finer resolution to enhance the short range resolution [28]; (c) use fine resolution binning shared between sub-pixels while use coarse binning among SPADs in each sub-pixel for correlation detection and event weighting [29]; (d) accumulate coarse histograms first, perform on-chip histogram analysis to determine if switch to fine bin mode [1]. A variant of this approach is to use partial histogram as we already discussed [30]; (e) a combination of coarse-fine binning and time-gating histogramming [22, 31].

In some sense, time-gating, especially the progressive time-gating method [22] is just like the sliding partial histogram approach, however with a subtle difference in that the SPAD is ONLY activated during the gated time. Therefore, a reasonably short time gate can get rid of pile-up caused by the SPAD deadtime. Time-gating is also an effective way to suppress ambient light as will be discussed in Sect. 4.

Apart from the aforementioned techniques, a group of novel dTOF ranging design architectures have been recently proposed, with a focus on low-level data compression and high-efficiency distance information extraction, as will be discussed in the rest of this section.

Histogram-free methods

Machine learning processor on FPGA

The elimination of histogram in the data-processing stage could significantly loosening the demand on sensor chip’s memory and processing power. In Ref. [32] the SPAD signal is time-stamped by the TDC and then sent to an event buffer, which is then processed on-the-fly using FPGA. A Long-Short-Term-Memory (LSTM) neural network processor reads in the histogram and locates the position of the signal pulse peak through supervised regression.

Differential intensity extractor

In order to obtain relative depth information of the scene from the pixel array readout with small data footprint compared to the conventional approach, a unique SPAD relative intensity recording scheme is demonstrated [33]. A group of SPADs can be configured into either a local pixel or a macro pixel, depending on their spatial distributions. For a pair of pixels A and B, in any given laser cycle, a counter increments / decrements if A / B first record an event. This detection method is sensitive to the difference in photon flux at the two pixels even when both are saturated because of high photon flux. To better understand this point, assuming both pixels contain a single SPAD, facing photon counting rate , with laser cycle period T, and the expected number of counts in the counter is

2

Under large photon rate conditions, even if the exponential term vanishes, the difference term on the numerator is preserved, as long as the counter dynamic range is not saturated.

In application scenarios where the shape of the illuminated objects are of interest; the relative depth information of the object surface is related to the TOF difference between the two pixels. The numerical relation between the counters counts and is

3

in which is the laser time width and an ‘AND’ logic is implemented to eliminate single photon events.

In terms of data processing requirements, this method is economic compared to conventional methods, as the elimination of histograms results in data throughput two orders of magnitude smaller. In addition, as shown in Fig. 5, the counter circuity is compact and easy to scale up to function in larger pixel arrays. The obvious limitation of this method is that it is rather an imaging than a ranging method. To acquire absolute depth information requires capturing a few absolute ToF measurements by sparsely distributing TDCs across the pixel array. It is possible that a hybrid of this method and conventional DTOF can be integrated together on chip to achieve high quality 3D ranging and imaging simultaneously.

Fig. 5.

Operation principle of the proposed method in Ref. [33] and its circuity cost in comparison with the conventional SPAD based imaging methods. Of note is the remarkable reduction in temporal and spatial footprint because of the elimination of histogram accumulation and TDCs, respectively

Dynamic width histogram

In analyzing the DTOF measurement histogram, the ROI of the signal peak is confined to the width of the laser pulse, usually of only a few ns. On the other hand, in order to achieve a reasonable full-scale range, the histogram total length is usually much longer than the ROI. A natural instinct is to make the histogram bin width in the ROI narrower, which leads to the partial histogramming technique [30]. A drastically different solution is to use unequal width histogram [34], in which the histogram width is inversely proportional to the measured photon rate at that bin. As shown in Fig. 6, a 10-bin unconventional equal-depth histogram has much finer bins at the peak region compared to the conventional histogram with the same total bin number. The triggered event time are recorded using a novel ‘Race Logic’ design [35] in which values are represented as time delays, rather than as analog or digital quantities, avoids the energy required to convert return events to digital timestamps. The core logic of generating the equal-depth histogram is built on N stages of binary split ‘binner’ circuit. During a laser cycle, in each binner, a preset control value splits the recorded events into an ‘early’ stream and an ‘late’ stream. If there are more events in the ‘early’ stream, the control value will be moved earlier before the next laser cycle, and vice versa. The control value eventually settles close to the overall median. A binner at one stage feeds streams of early and late photon events to two binners at the next stage in the tree. Eventually, the controls values from the binners will form a histogram with bins a majority of the bins cluster around the true peak location.

Fig. 6.

Adapted from Ref. [34].(a) The traditional equal-width histogram of a certain group of student’s age distribution. (b) The peaky structure of the student’s age distribution is much better captured using the equal-depth histogram, due to its narrower bin width around the peak location

A drawback of this architecture is the binning positions may not converge because of the random nature of the photo arrival time. In addition, the resolution should decrease dramatically poor in high ambient light scenarios.

To summarize, the traditional TCSPC technique remains the mainstream solution owing to its robustness and compatibility in a variety of industrial application scenarios. It’s natural support of multi-peak processing is suitable for complex ranging requirements. Partial histogramming, dynamic histogram or histogram-free method, on the other hand, saves SRAM space at the cost of limited peak information output capability; therefore, it might be advantageous for in-door applications with less multi-path interference and crosstalk.

Ambient light suppression

A challenge that most modern dTOF LiDAR sensing chips have to face is the high dynamic range of ambient photon flux, which can change by orders of magnitude as the working environment changes, with a maximum intensity up to 100 klux. A common mitigation method is to use optical band-pass filters [36] for wavelength selection. Nevertheless, high level ambient light flux could still saturate the detector circuit’s time-resolving capability [37] and put a challenging load on the output bandwidth. In SPAD based pixel designs, high background light severely degrades the SNR as well as distorting the pulse shape owing to increased pulse pile-up. Both effects negatively impact the ranging resolution. Time-gating is a viable solution to reduce pile-up, by combining the SPAD’s quenching circuits and gating schemes [38], at the expense of reducing the SPAD’s duty-factor [3], and the proper choice of the time window is in general difficult [31]. Active quenching and recovery circuit [39] could substantially shorten the SPAD dead time, however the after-pulsing probability also increases, and a trade-off between detection rate capability and pulse quality has to be made.

The suppression of ambient light background can be done after the histogram has been collected, by employing dedicated background subtraction and pulse finding subroutines when post-processing the histogram. The drawback of this approach is it requires large overhead on the output bandwidth especially when high timing precision is needed, which in turn limits the available frame rate. The chip’s power consumption is also increased. Consequently, it is desirable if ambient light suppression or rejection can be integrated during the formation of histogram. In generation, most of such endeavors can be categorized as utilizing the temporal [40] or spatial difference between the laser echo signal and the ambient light backgrounds, respectively.

Coincidence detection methods

The basic idea is to detect photons in close proximity in time and create coincidence detection gating, as the laser echo signal arrives within the laser pulse width, as opposed to the ambient light background which arrives randomly [22, 41]. BenMoussa et al. [13] explore an architecture using digital SiPM pixels with smart trigger logic for concurrent event detection, in which the detected photon counts after a triggered event is compared with a preset threshold number during a short time window, and only output those data segments which passed the threshold. In addition, a trade-off between power consumption and timing resolution is made by using a combination of coarse-fine resolution TDC. The time window is started by the arrival of a SiPM pulse. The fine resolution TDC immediately starts to work until the end of the time window if the threshold is not met, or the next global clock starts if the threshold is met, and a validation signal trigger is produced. For longer range applications two short counters are chained and measure the clock cycles until the event detection. Schematic of the readout circuitry timing logic is shown in Fig. 7.

Fig. 7.

A high-level pixel array schematic and its associated timing circuit logic showing the smart trigger logic proposed in [13]

In Ref. [42] a similar approach for Flash LiDAR application is presented with different pixel architecture, a large pixel is subdivided into smaller sub-pixels each contains several SPADs. A fine TDC records the arrival of the first incoming photon and creates a coincidence window based on laser pulse width; each sub-pixel can have its own coarse timestamped counter serve as a threshold filter. The TDC data is written to bus only if the threshold is met at the end of the coincidence window. In such readout scheme that require TDC sharing, loss of events due to pulse collision can be recovered by recording both the leading and trailing edges timestamps of a pulse and identify pile-up events as having pulse width wider than the laser width [1].

In order to effectively reject background events while retaining signal events with high efficiency, the threshold value must be optimized. This is especially challenging as both the signal and the background will have large dynamic range due to variation in target reflectivity, laser beam incidence angle, weather conditions, etc. An adaptive threshold setup can improve the signal to background ratio [43] by adjusting the coincidence window setup according to the detected photon rate and measured coincidence level.

Another design approach [44] eliminates the generation of histograms which frees the otherwise required large storage space and instead using peak holder to update the number of concurrent photons, and an analog multi-hit TDC to record the highest bin position after each laser emission. Although this method has the potential to achieve high frame rates, it requires large number of SPADs per FOV when used in scanning applications, and its ranging precision is limited by the bin width as no pulse shape information can be extracted.

The coincidence detection approaches introduced in this section can effectively reduce the circuit layout density which is ideal for high resolution SPAD array chip design. A drawback is the signal miss-counting issue, which might impede the system performance in low light environments. In practice, the choice of whether to implement the coincidence detection depends on the pixel resolution and light environment requirements.

Spatial laser pattern recognition methods

The typical 1D or 2D scanning LiDAR emits laser pulses having an illumination region matches with the FOV of a small number of pixels on the sensor array. An optimized data readout scheme should therefore be able to readout only the pixels related to the laser illumination signal, thus avoiding unnecessary bandwidth saturation. To further optimize the performance, a smart ROI setup can be employed to keep tracking and monitoring the laser spot in real time. This is especially useful for short range applications, as the systematic effect due to laser illumination pattern distortion or disparity between TX and RX can be attenuated. One such design idea is to add a calibration laser TX phase before the ranging TX sequence [1, 45], as presented in Fig. 8. During the calibration phase, for each pixel, a logic circuit compares the number of photon counts with/without laser beam on and selects those pixels with enough echo signal counts. One disadvantage of this method is that when the target moves across the scene with fast angular velocity, the object corresponding to the ROI shifts, blurring the acquired image.

Fig. 8.

A high-level schematic showing the stages of data processing per scanned frame as proposed in [1]. An in-pixel laser spot detection phase is performed and decides which pixels will be used for dTOF measurement. After a certain number of coarse bin histograms are measured, the ASIC logic analyzes those histograms and determines if it should switch to the fine bin mode

The spatial laser pattern recognition technique cover in this section is now a standard LiDAR configuration module. Its implementation scenario is however limited to the optical focusing process of the TX/RX lense.

Reducing LiDAR interference

When multiple DTOF LiDARs are operating in the same scene, one LiDAR can generate a light echo signal which is received by other LiDARs. As a result, the LiDAR system would have difficulty determining whether the detected laser pulse signal was generated by its own light emitter. Such inter-LiDAR interference must be reduced to a minimum level to guarantee the LiDAR’s proper functionality.

There are several common techniques to resolve inter-LiDAR interference. The Pulse position modulation (PPM) technique [46, 47] spreads out the emitted LiDAR pulse temporal profile utilizing a digitally controlled delay line (DCDL). As shown in Fig. 9, when a TOF histogram is generated, a demodulation process is performed, after which the modulated pulse is reshaped back into its original narrow temporal profile while the alien interference pulse height is drastically reduced by the demodulation. The drawback of this technique is 1) the method could fail if the alien interference pulse signal is much stronger compared to the true signal and 2) it cannot distinguish the interference if both pulses are modulated by the same pattern.

Fig. 9.

A schematic showing the principle of PPM as proposed in [46]. The optimized parameter of the modulation is shown in the middle right plot

A second design architecture is based on the Optical Code Division Multiple Access (OCDMA) technique, in which multiple laser pulses were generated with a time pattern encoded according to the Optical Orthogonal Codes (OOC) [48, 49]. Each OOC C is a family of fixed sequences of 0,1 numbers. When applied in time-coding the LiDAR laser emission, each sequence can be regarded as a consecutive TDC values, where the ‘1’s are pulses and the ‘0’s are null bins. The unique feature of OOC is that when performing convolution in the time domain, those sequences have thumbtack auto-correlation as well as small cross-correlation, expressed as:

4

for any and any integer T in the range ; and

5

for any and any integer T(mod n). The number of ‘1’s in the sequence is the weight W of the code. A proper minimization of the correlation results in . By encoding the LiDAR laser emission using OOC, inter-LiDAR interference can be suppressed during data post processing, in which the histogram is convoluted with a filter sample sequence of the OOC. The price to pay is the diminished ranging capability. A useful figure-of-merit is the weight W. The maximum ranging distance shrinks by .

The above methods require relative sophisticated pulse shaping and pattern design, as well as dedicated on-chip circuit design or heavy duty post-processing. Instead, a conceptually simple and easy to implement method is emitting two laser pulses per TX with a controllable time interval [14]. The interference pulse can be completely eliminated except for the rare occasions where an interference pulse happens to have the same time interval and similar pulse height with respect to the emitted pulse pairs. In order to further suppress the interference signal in rare “corner" cases, a random laser emission technique can be employed [50], in which a random is added to the otherwise fixed laser TX interval. Similar to the PPM method, after the accumulation of multiple TX data, a persistent interference pulse will be spread out in the time range determined by the random time shift width, in the worst-case scenario. The random nature of the SPAD avalanche facilities the design of low-cost, compact random number generator, for instance using a pair of noisy SPADs and store which one first avalanche as a [0,1] bit. By repeating this process the pair of SPADs function as a binary random number generator [29] Fig. 10.

Fig. 10.

From Ref. [49]. (Upper) OOC code with parameters , as defined in Eq. 4 and Eq. 5. The maximum number of sequences belonging to this family is 5, labeled as . (Lower) The auto-correlation and cross-correlation pattern of

The LiDAR interference reducing techniques cover in this section has become a key standard module in harsh electro-magnetic interference enviornments.

Histogram correction algorithm

The ranging precision of the DTOF LiDAR relies on the hardware time-stamping resolution, the timing jitter of the photo-detector’s response function, the laser’s intrinsic width and timing uncertainty, and importantly, the statistical fluctuations of the acquired signal events on top of background events. In a typical pulsed laser DTOF setup using SPAD based sensor chip with TDC histogramming, the single event timing precision can be expressed as [51]

6

where is a combination of the signal binning width error, the TDC non-linearity error and the TDC jittering. is usually the dominate error source, determined by the laser pulse width. Finally, is the SPAD time-jitter. A laser echo signal with signal counts and background counts in its peak region can have its peak position located with a precision [52]

7

The ranging precision is simply . From Equation. 7, the measurement precision is affected by both the number of signal counts and the signal-to-background ratio. Consequently, to improve the ranging precision requires 1) suppress the background level, as discussed in Sect. 4 and 2) increase the signal counting rate.

Pile-up distortion recovery

Unfortunately, the signal counting rate of the SPAD device cannot be increased indefinitely because of the SPAD deadtime, during which it cannot detect another incoming photon. Furthermore, the measured signal pulse shape experiences a pile-up distortion effect as those late arriving photons within the same laser pulse will not be recognized, reducing the measured pulse intensity and introduces a systematic error which shifts the peak position earlier, thus can degrade the distance measurement accuracy.

The pile-up issue has haunted single photon detection measurements starting from the 60 s [53, 54], at which time a simple formula is proposed to compensate for the pile up effect. The basic idea is to estimate the number of pile-up photons based on the poisson nature of the incoming photon flux. An example of such correction formula reads [5, 55]

8

in which the true count rate at the k-th bin is corrected from the pile-up distorted measured histogram counts . The formula is well-suited to correct the distortion at low to medium counting rate but lost its prediction capability at high counting rate applications where the denominator in Eq. 8 is approaching zero.

Alternatively, pile-up correction could be performed by detecting the rising edge of the laser echo signal as a stable timestamp regardless of the subsequent pile-up. The precision of this method is limited by the TDC resolution and the large statistical uncertainties of the rising edge signal counts. In practice, this problem can be alleviated by incorporating both the rising and falling edge position and slope to perform regression against the pulse peak position [56].

A more sophisticated and yet indirect solution requires the extraction of pulse shape features, such as the pulse width, pulse height, rising and falling edge, center-of-mass, etc., and to perform multi-variable regression using those parameters to correct for pulse pile-up. A commonly used procedure is to apply different convolution kernels to the measured histogram, then analyze the corresponding change in the measured shifted peak position. Deep learning techniques, such as CNN [57] or LSTM [32] neural network can realize automated feature extraction without the need to manually specify which features are to be used, at the cost of dedicated computational resources.

After-pulsing effect

A related phenomenon that affects the SPAD response is the after-pulsing effect, caused by defect-trapped carriers releasing the charge and triggers secondary avalanche. This after-pulsing effect can be reduced by setting a longer SPAD recharge time [58]. After-pulsing can also be optically induced by secondary photons which are re-absorbed in the same SPAD’s neutral region, and the charge carriers diffuse to the high field region. This after-pulsing probability can be reduced by using a low-lifetime substrate [59]. Nevertheless, such events distort the accumulated pulse histogram profile and can negatively impact the ranging precision. Recently, active correction algorithms have been developed in order to reduce the distortion effect. The de-trapping lifetime can be obtained by firing laser onto a hard target and extracted through an exponential fit of the after-pulsing miscounting probability [60]. Bin-to-bin corrections can then be made by subtracting the spurious after-pulsing counts according to the observed counts time distribution [60, 61].

Crosstalk 

Another source of non-ideality is the crosstalk between adjacent SPADs. This effect is originated from hot carrier light emission during the avalanche of an ‘aggressor’ SPAD and triggers the emission of light in a neighboring victim SPAD either promptly or after a time delay [5, 12, 59]. Apart from device level mitigation methods such as the deep trench pixel isolation technique [12, 63], software correction algorithms have also been invented, by making use of the pixel-wise correlation matrix specifying the trigger rate between the ‘aggressor’ SPAD pixel and its neighbors, and then perform hot pixel correction using a combination of clock-recharging SPAD nonlinear response function, the crosstalk correlation matrix and a 1D median filter method [62], as shown in Fig. 11.

Fig. 11.

(Upper) Schematic of the carrier recombination and inter-pixel light crosstalk. (Lower) Image processing flows a multi-step optical crosstalk correction algorithm proposed in Ref. [62]. The raw image is first corrected through a nonlinear correction function ‘F’ to associate the observed counts with predicted true counting rate. Meanwhile, the crosstalk matrix is applied to the raw image and subtracted from the ‘F’ corrected image, and applied a 1D median filter

Adversary environments

Bad weather conditions such as fog or snow can reduce the LiDAR’ sensing range and disturb its imaging capability. A deconvolution method [64] has been proposed to distinguish objects from fog, where the received light intensity is

9

where is the impulse response of the object or fog at distance , and is the received power having lost energy by passing through fog [65]. Assuming homogeneous fog density, is just a function of distance with a constant scale factor. Using as the deconvolution kernel, the fake ’peak’ on the histogram caused by fog will be flattened out as its is broad while the true laser return signal will be peaky.

Feasibility and cost considerations

The histogram correction algorithms discussed in this section typically requires a high volume of convolution, multiplication or special function computations. It is therefore not suitable for on-chip integration, as parallel processing between multiple pixels will consume a lot of registers, meanwhile going through a large number of range bins will increase the area and power consumption. As an example, the ’benchmark’ Sony IMX459 SPAD Sensor Chip opts to off-chip corrections using the LiDAR point cloud information such as peak height, position and profile in external SOC. An alternative approach adopted by LiDAR manufactures such as HESAI technology is to integrate standard SOC on-chip for better flexibility.

Mitigating highlight saturation

One major challenge the LiDAR sensor needs to overcome is to provide high fidelity ranging and imaging when the received laser signal intensity has large spatial variations. In application scenarios such as automobile auto-driving, this can be especially tricky because of the frequent use of retro-reflective road signs [66, 67] in modern road traffic environments. The surface of retro-reflective material strongly reflects light in the same direction as the light arrives, due to its total internal reflection design. This is drastically contrary to the reflection behavior of normal material surface, which have both diffusive and specular reflection components [68]. Unless the emitted laser happens to be perpendicular to a glass or a high reflective metal surface, the measured reflectivity should be orders of magnitude weaker compared to a retro-reflective surface. Unfortunately, the SPAD or SiPM light response is non-linear by nature, suffers from event pile-up caused by the SPAD deadtime. As a result, the pixels corresponding to high reflective objects can be saturated and the detail within the object is lost. An associated parasitic effect is the expansion of the apparent FOV of the retro-reflective object. The cause of this phenomenon is manifold; the aforementioned optical crosstalk between pixels is usually the major contributor, while stray light reflection inside the RX lens causes veiling glare as a halo like disk artifact [69], and the lens intrinsic astigmatism also contributes to this effect.

In order to minimize the impact of high reflectivity induced effects, a number of mitigation measures can be taken from both the hardware and software perspectives. Anti-reflective coating [70] and optical baffle [71] are common practices in optical lens design which can effectively suppress stray light reflection. Optical crosstalk reduction and correction methods need to be implemented, as already discussed in Sect. 6.3. Compared to Flash LiDAR, 1D and 2D scanning LiDAR design can benefit from having a smaller sensor ROI, nonetheless the high light flux saturation effect within the ROI needs to be dealt with. One might adopt a similar approach as presented in Fig. 8 [1], where an initial scanning stage performs fast counting and distance estimation using on-chip ASIC logic to analysis the histogram for each ROI. For those ROIs having high photon counts that are suspected to be high reflective objects, the corresponding laser emission intensity will be lowered accordingly in the subsequent formal measurement stage.

Area and power consumption optimization

Reducing the power consumption of the LiDAR sensor chip plays a critical part in the LiDAR system design. In section 2 we have discussions about data compression and area and power consumption reduction from the point of view of the signal workflow optimization. In this section we shall give a more comprehensive discussion on other relevant energy and area saving methods. The major contributors to total power consumption can be summarized in this formula [72]

10

in which the subscript LS, SPAD, TDC, MP and IF stands for the laser source, the SPAD circuit, the TDC circuit, the digital processor and the interface power consumption, respectively. The design principle is to minimize the total power consumption in a cost-effective way, while maintaining LiDAR’s ranging capability. Following this logic, we categorize the existing efforts of reducing power consumption into three categories; 1) improving CMOS technology to reduce the SPAD and the electrical circuit’s area and power consumption; 2) Streamlined photon-sensing and timing design focusing on reducing the data processing power and 3) Optimization of the RX data processing circuit to satisfy the laser scanning needs with minimum cost. For item 1), one could start from the famous Denard Scaling, in which the dynamic power consumption P is related to the clock frequency f, the transistor capacitance C and the switch voltage V as

11

where k is a scale constant and is the ‘leakage’ or static power consumption, proportional to the leakage current times the threshold voltage. The capacitance C roughly scales with area, and V rough with the linear dimension of the transistor, therefore advanced CMOS node technology should in principle help reducing both power and area consumption. In modern 3D CMOS multi-layer stacking process technology, a popular CMOS node choice is 40 nm which achieves a good balance between analog requirements and digital design. In addition, the 3D multi-layer stacking process technology has leading to minimum X and Y size for top die limited devices such as the SPAD pixels. Its rectangular digital floorplan shape improves routing efficiency and performance, as typical monolithic U or L shaped sensor floorplan suffers from limited routing efficiency and increased power consumption [73].

As this review is not focused on semiconductor technology and devices, in the remainder of this section, we shall focus on items 2) and 3) and go through a number of practical implementation ideas.

Data rate compression

Memory occupancy increases with the need of detection range, range resolution, and field of view (FOV). In high precision and high frame rate LiDAR measurements, one bottom-neck is the data throughput and storage capacity. To address this issue, data compression in the 3D point cloud processing stage has been presented, including a 3D mapping framework based on octree and uses probability occupancy estimation to model unmapped area [74], and a real-time compression approach which supports incrementally data decompression and analysis [75], both being efficient with respect to runtime memory usage. For low-level data compression, i.e., prior to the point cloud processing stage, including the aforementioned coarse-fine binning methods and time gating methods, as well as a variety of lossless data compression approaches, namely 1) the binary arithmetic coding [76], a dictionary-based compression method which uses a combination of sliding window and dynamic dictionary to compress unknown data [77], and the Golomb-Rice encoding method [78], having good performance when small number are assumed more likely to be presented in the data stream than big ones, such as in high ambient light environment. These encoding methods, however, may be slow as it requires extra on-chip data manipulations, and the data reduction factor can barely reach 50%. Therefore, compression sensing strategies such as adaptive ROI sampling is proposed, based on a fast estimation of the depth map using iterative likehood fits of a coarse grid of pixels until it converges to build a probability map of the regions of interest [79]. Similarly, a data sketch method based on a generalized MLE fitting scheme can achieve similar resolution as compared to the coarse-fine binning scheme [80].

The importance of data rate compression technology discussed in this section becomes increasingly prominent as the resolution of LiDAR chips continues to improve, as higher chip resolution leads to a significant increase in data transmission volume, placing higher demands on the chip’s data transmission and storage capabilities, and in turn is driving data compression technology to become a core design focus for high-resolution LiDAR chips. For example, the EM4 LiDAR equipped with Robosense’s self-developed SPAD-SOC can achieve up to 2160 scan lines, owing to the chip’s point cloud data compression design to enable efficient data processing and output, despite its large data transmission volumes.

Reducing TDC area and power consumption

The SPAD event timing generation and acquisition is usually achieved using high-bandwidth TDC circuits. The simplest and most straightforward strategy is to arrange one TDC per SPAD and the TDC is in a continuous-running mode, meaning it is ’always on’. However, this is not very efficient because 1) large chip area needs to be reserved for the TDC block; 2) The sheer number of TDC required will render the inter-block wiring a near impossible task; 3) a large number of TDC counts will be triggered by ambient light events, and the TDC processing power is wasted. As a result, event-driven [81, 82] and shared-TDC architectures [2] are proposed. In event-driven TDC design, the TDCs turn on upon receiving a photon event, and stop by the end of the time frame. In shared-TDC architecture, a number of SPAD sensors share a single TDC, while using combinational logic circuits to process the event counts. A column-wise [23] shared-TDC architecture is shown in Fig. 12 (Middle), where the TDCs can operate either in an event-driven mode or in a combination mode. Alternatively, the TDC can be operated in a continuous-running mode shared among several SPAD blocks, as shown in Fig. 12 (right). In both cases the total power consumption w.r.t. the timing circuit is a combination of the TDC consumed power and the combinational consumed power, plus the power consumed by PLL circuit. The compromise here is the loss of events due to the TDC deadtime. The output rate can be increased by adding a pulse-shortening monostable circuit [83], as shown in Fig. 13 (Middle) or using an XOR-based combination tree [9], as shown in Fig. 13 (Right).

Fig. 12.

From Ref. [2]. Examples of different TDC arrangements. (a) A per-SPAD, event-driven TDC architecture; (b) Column-wise shared event-driven TDC; (c) Continuously running, shared TDC concept

Fig. 13.

Adapted from Ref. [9]. (a) A simple OR logic combinational tree. The hold-off time is roughly equal to the SPAD deadtime from the anode and inverter output. (b) OR tree with monostable pulse shaper to shorten the TDC response. (c) An XOR Tree with toggle flip-flop input. (c) Continuously running, shared TDC concept

The choice between shared or per-pixel TDC depends on the system requirement and event-loss rate tolerance. In general, high per SPAD conversion rate applications favor per-pixel TDC arrangement, however the gain in accuracy and efficiency asymptotes to a limit where SPAD deadtime prevails. In Ref. [10] a quantitative comparison is performed to benchmark different application scenarios, showing that there is an optimum number of TDCs for each application scenario, above which the area and power consumption will increase without enhancing the TCSPC accuracy.

Finally, we should remark that the ultimate design principle is to achieve a balance between conversion rate, area and power consumption. The aforementioned coarse-fine timing approach is also an effective way to reduce power consumption [1, 11, 24]. For example, in a ring-oscillator(RO) based TDC the presented TDC power consumption scales with resolution by simply changing the RO frequency [11]. While in Ref. [84], the sensor is operated to recognize the two-laser-pulse pattern to reject interference, the fine TDC is triggered only when the second laser pulse is correctly detected, reducing unnecessary power consumption.

Dynamic power consumption control

The on-chip signal processing circuit hardware design has been modified to minimize its associated area and power consumption, such as implementing a low power micro-controller design [82] or re-usable functional core for the analog circuitry [85]. Meanwhile, studies on bench-marking the low-power operation algorithm and efficient data processing methods have been performed [86–88]. Nominally, a sequential task scheduling for minimizing the total power consumption within a given processing time constraint has been studied, useful when the static power consumption of the sensor chip cannot be neglected compared to the dynamic power consumption [89]. Novel environment sensing-based LiDAR sensor control has been proposed by letting the micro-processor enter sleep mode according to the vehicle’s speed and surrounding environment [72].

Conclusion

Multiple facets of the SPAD based dTOF chip design has been reviewed, with an emphasis on novel ideas and smart chip design architectures which mitigate adversary effects such as SPAD pile-up and crosstalk, as well as improving the system’s robustness against challenging working conditions such as high ambient light, high reflectivity object detection, and the system’s proper functioning amidst multiple LiDAR interference. We investigated novel ranging principles which can benefit the system’s measurement precision such as the coarse-fine binning schemes, in the meantime achieving superior performance under large background light flux by using techniques such as partial histogramming and coincidence timing. A hybrid solution of the above mentioned methods should achieve a good balance between cost, power consumption and performance, paving the way towards a single-chip receiver solution compatible with a solid-state dTOF scanning system.

To conclude this review, we shall briefly project the future development trends of SPAD-array-based dTOF LiDAR chip design by trying to answer three important open questions in this field. The first question is, what kind of technical architecture can survive commercialization. At present, the demand for key performance factors such as ranging distance, angular resolution, and frame rate of LiDAR products has increased drastically, leading to higher burden on chip PPA design, as a result promoted the continuous innovation of ranging technology, signal processing methods and algorithm integration solutions. However, the actual performance of various innovative ideas, such as the histogram-free paradigm, has not yet reached the performance level of the classic TCSPC scheme, and the relevant technologies still need to be verified by actual scenarios before they can be commercialized.

The second question is, what are the physics-level limits that cannot be broken through in this field. Judging from the existing research literature and current status of commercial chip products, the physical limitations in this field are mainly reflected in four aspects: first, the size of SPAD devices is relatively large, with a typical size of about 10m of conventional products, which is much larger than that of CIS devices. second, the conventional operating voltage of the SPAD PN is higher than 20V, causing frequent avalanche currents and large analog power consumption, which becomes a major obstacle to power consumption optimization. Third, there is an upper limit to the chip clock frequency of approximately 1 G or 500MHz, making it difficult for the time resolution of TDC modules to reach the ps level, and higher ranging accuracy needs to be achieved through indirect methods such as phase delay and centroid interpolation. Fourth, the scale of data flow continues to expand, and when the front-end adopts the TCSPC ranging method, it is necessary to count a large number of avalanche counts through histograms, which requires higher I/O frequencies of the SRAM, resulting in higher digital power consumption.

The third and last question is, what open problems remains unsolved in this field. At present, there are two critical problems in the industry that need to be broken through: first, there is no physical solution to the pile-up effect caused by the SPAD dead zone, which cannot be eliminated from the device itself, and the signal must be processed by correction algorithms to restore the true ranging distance; The second is the parallax-free fusion problem of SPAD and 2D images, that is, to realize the monolithic integration of SPAD and 2D CIS.

Author contributions

Lianghua Mo and Shihua Huang wrote the manuscript and prepared figures and tables. They contributed equally to this work and are designated as co-first authors. Yi Yang and Tian-Ling Ren critically reviewed the manuscript, and Ren served as corresponding author. All authors read and approved the final version of the manuscript.

Funding

This research received financial support from the Shenzhen Science and Technology Program [Grant No. KJZD20240903103811015].

Data availability

No datasets were generated or analysed during the current study.

Declarations

Ethical and consent to participate

Not applicable.

Consent to publish

This manuscript has not been published or presented elsewhere in part or in entirety, and is not under consideration by another journal. All the authors have approved the manuscript and agree with submission to your esteemed journal.

Competing interest

The authors declare no Conflict of interest.