Robust real-time estimation of non-uniform angular velocity and sub-pixel jitter in images captured with resonant scanners

Abstract

Images captured with resonant scanners are affected by angular velocity fluctuations that result in image distortion and by poor synchronization between scanning and light detection that creates jitter between image rows. We previously demonstrated that both problems can be mitigated in post-processing by recording the scanner orientation in synchrony with the image capture, followed by data resampling [Opt. Express ³⁰, 112 (2022)]. Here we introduce more robust algorithms for estimation of both angular velocity fluctuation and jitter in the presence of random and deterministic noise. We also show linearization of the scanner oscillation model to reduce calculation times by two orders of magnitude, reaching 65,000 jitter estimations per second when using 2,800 samples per image row, and 500,000 when using only 500 samples, easily supporting real-time generation of jitter-corrected images.

1. Introduction

Optical resonant scanners [1,2] are used in a wide range of applications, including microscopy [3–5], intraoperative imaging [6–8], ophthalmic imaging [9–11], optical coherence tomography [12–14], automotive light detection and ranging, [15–17], high resolution displays [18–21], flow cytometry [22], underwater imaging [23] and printing [24,25]. The angular velocity variation in these devices through the scan cycle introduces sinusoidal image warping when pixels are equally spaced in time. This image distortion can be corrected by interpolating over a grid of uniformly spaced pixels, after calibration using a periodic test object such as a Ronchi ruling [11,26–31], fitting a physical model to the scan cycle [10,11,32–35], or using non-uniform temporal pixel acquisition [3,36,37].

In addition to warping, images captured with resonant scanners are also affected by imperfect synchronization between the optical scanning and pixel acquisition, resulting in jitter between image rows [27,36–39]. This jitter can be mitigated using custom electronics known to be sensitive to noise and to require non-trivial tuning [36,40–44], a secondary optical system in combination with custom electronics [39], or as we recently demonstrated, by capturing scanner orientation signal samples in synchrony with the image acquisition [45]. This approach, which exploits built-in orientation sensors present in most resonant scanners, is accurate but, as proposed, is too slow for real-time applications, requiring ~0.7 ms per image line with 2,500 pixels.

Here, we first improve the scanner orientation signal model by incorporating deterministic and random noise. Then, we demonstrate methods for identifying and excluding the scanner orientation samples most affected by deterministic noise, to deliver robust image distortion and line jitter estimation. Finally, we propose the linearization of the expanded cosine model to reduce calculation times without compromising accuracy.

2. Methods

2.1. Data collection

A custom adaptive optics scanning ophthalmoscope [11] which can be thought of as a confocal microscope, was used to synchronously capture images and resonant scanner orientation samples for every image pixel. The adaptive optics in this instrument played no role in the desinusoiding and dejittering that are the focus of this work. The horizontal scanning of the imaging raster was performed with an SC-30 galvanometric scanner by Electro-Optical Products Corporation (Ridgewood, NY, USA) resonating at 13.842 kHz. Image pixels and resonant scanner orientation samples were captured using an Alazartech 16-channel 14-bit waveform digitizer (model ATS9416; Pointe-Claire, QC, Canada) with the pixel clock operating at 40 MHz, as depicted in Fig. 1. Sequences of 100 images, each with 650 rows and 2816 columns were captured using a lens with negligible distortion. This large number of columns was chosen to capture almost the entire resonant scanner cycle (~2,890 samples). Because of the minimal distortion, we consider the scanner angular orientation samples and their corresponding pixel locations on the image plane linearly related.

Fig. 1.

Depiction of point-scanning instrument electrical connections for synchronous sampling of images (top) and resonant scanner orientation to capture both image warp and sampling jitter (bottom). The gray rectangle is a voltage divider that reduces the scanner orientation signal amplitude, so that it is within the dynamic range of the digitizer. The object image (top) is mirror-symmetric along the horizontal direction because pixels are captured through both the clockwise and counterclockwise portions of the resonant scanner cycle.

2.2. Data model

The angular orientation samples ${\theta }_{ij}$ of a resonant scanner in an imaging instrument can be modeled as,

$${\theta }_{ij}={\theta }_{\mathrm{a}}\cos \left(\frac{2\pi }{T}{t}_{ij}+\phi \right)+{\theta }_o+\zeta \left(t_{ij}\right)+{\eta }_{ij},$$ (1)

with $i$ and $j$ denoting pixel row and column, respectively, ${\theta }_{\mathrm{a}},T,\phi$ and ${\theta }_o$ cosine amplitude, period, phase and offset, $t_{ij}$ the time at which pixels are captured, $\zeta \left(t_{ij}\right)$ deterministic additive noise (assumed identical for all scan cycles), and ${\eta }_{ij}$ random additive noise.

There are multiple potential sources of image row jitter, that is, lateral shift of the sampling of each image row. This work is motivated by jitter due imperfect synchronization between the resonant scanner with period $T$ and the times $t_{ij}=n_{ij}{t}_{\mathrm{p}\mathrm{i}\mathrm{x}}$ at which pixels are captured, where $n_{ij}$ is an integer and $t_{\mathrm{p}\mathrm{i}\mathrm{x}}$ is the time between pixels. In practice, $T$ is not an exact multiple of $t_{\mathrm{p}\mathrm{i}\mathrm{x}}$, that is, $T/t_{\text{pix}}=N_{\text{cycle}}+\epsilon$, with $N_{\text{cycle}}$ being the integer quotient and $\epsilon$ being the remainder. Therefore, the first pixel of each scan “drifts” in relation to the position within the scan cycle by the pixel fraction $\epsilon$, which in the cosine model corresponds to a phase shift of $2\pi \mathrm{\epsilon }{\mathrm{t}}_{\mathrm{p}\mathrm{i}\mathrm{x}}/T$ radians.

Most opto-mechanical resonant scanners output an electronic digital pulse that can be used to synchronize the scanning and image acquisition to within the nearest pixel, and thus, in a noise-free instrument, the phase drift would become image line jitter within the [−0.5, 0.5] pix interval. Therefore, jitter is caused by the accumulation of fractional ε pixel shifts minus its nearest integer, as it can be readily seen in the images in Fig. 2, where the vertical lines of a Ronchi ruling appear slightly skewed (left and middle) and/or randomly dislocated (right).

Fig. 2.

Ronchi ruling images captured with a point-scanning instrument in which the horizontal scanning was performed with a resonant scanner. The image warping is due to the angular velocity variation of the horizontal scanner, while the sub-pixel misalignment of the image rows being caused by the lack of synchronization between the pixel acquisition and the resonant scanning.

In this model, $t_{ij}$ can be replaced with $t_j+\delta t_i$, where $t_j$ denotes the times at which pixels would be captured in the absence of jitter (the same across rows), and the jitter $\delta t_i$ term, which varies across image rows, but not across columns. In this way, Eq. (1) becomes

$${\theta }_{ij}={\theta }_{\mathrm{a}}\mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{2\pi }{T}{t}_j+\frac{2\pi }{T}\delta t_i+\phi \right)+{\theta }_o+\zeta \left(t_j+\delta t_i\right)+{\eta }_{ij}.$$ (2)

2.3. Robust estimation of model parameters

In order to estimate the model parameters ${\theta }_{\mathrm{a}},T,\phi$, and ${\theta }_o$, as well as the deterministic noise function $\zeta \left(t\right)$, let us start by noting that when the jitter values $\delta t_i$ are a small fraction of the scan period $T$, Eq. (2) can be approximated using the first-order Taylor series,

$${\theta }_{ij}\approx {\theta }_{\mathrm{a}}\mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{2\pi }{T}{t}_j+\phi \right)-{\theta }_{\mathrm{a}}\frac{2\pi }{T}\delta t_i\mathrm{s}\mathrm{i}\mathrm{n}\left(\frac{2\pi }{T}{t}_j+\phi \right)+{\theta }_o+\zeta \left(t_j\right)+\delta t_i{\zeta }^{\prime }\left(t_j\right)+{\eta }_{ij},$$ (3)

with prime denoting first derivative. If we now average the scanner orientation signal across $N$ image rows as ${\bar{\theta }}_j=\sum _{i=0}^N{\theta }_{ij}/N$, we get

$${\bar{\theta }}_j\approx {\theta }_{\mathrm{a}}\mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{2\pi }{T}{t}_j+\phi \right)-{\theta }_{\mathrm{a}}\frac{2\pi }{T}\overline{\delta t_i}\mathrm{s}\mathrm{i}\mathrm{n}\left(\frac{2\pi }{T}{t}_j+\phi \right)+{\theta }_o+\zeta \left(t_j\right)+\overline{\delta t_i}{\zeta }^{\prime }\left(t_j\right)+\overline{{\eta }_{ij}}.$$ (4)

Assuming that both jitter and random noise have zero-mean, and that the averaging is performed across an adequately large number of scan cycles, we have that Eq. (4) simplifies to

$${\bar{\theta }}_j\approx {\theta }_{\mathrm{a}}\mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{2\pi }{T}{t}_j+\phi \right)+{\theta }_o+\zeta \left(t_j\right).$$ (5)

The absence of random noise and jitter in this formula allows the estimation model parameters by fitting an average of experimental data, with the deterministic noise emerging as the fitting error or residual. This fitting was performed in the interpreted computing language Python (version 3.10), using the damped least-squares minimization algorithm by Levenberg and Marquardt included in the Python module Scipy (version 1.11.1). This algorithm was chosen because it was found to be ~20 times faster than the downhill simplex method by Nelder and Meade used in Ref. [45]. Fitting robustness, aimed at mitigating parameter estimation error due to the deterministic noise, was implemented by repeating the fitting, discarding all data points with fitting error higher than one standard deviation after the previous iteration.

2.4. Cosine model linearization for jitter estimation

Once the cosine parameters and the deterministic noise are estimated from the fitting the scanner orientation signal averaged over many cycles, we seek to estimate the image row jitter with lower calculation times than those from our previous iterative non-linear least square fitting [45], and with equal or superior accuracy. To do so, let us rearrange the terms in the linearized cosine model in Eq. (3) to get,

$${\theta }_{ij}=\left[{\theta }_{\mathrm{a}}\mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{2\pi }{T}{t}_j+\phi \right)+{\theta }_o+\zeta \left(t_j\right)\right]+\left[{\zeta }^{\prime }\left(t_j\right)-{\theta }_{\mathrm{a}}\frac{2\pi }{T}\mathrm{s}\mathrm{i}\mathrm{n}\left(\frac{2\pi }{T}{t}_j+\phi \right)\right]\delta t_i+{\eta }_{ij}.$$ (6)

In this way, the scanner orientation data from each image row can be seen as a set of linear equations that can be solved for $\delta t_i$ using any mathematical computing package with least-squares and/or robust linear fitting algorithms.

3. Results

3.1. Robust 4-parameter cosine fitting

Before trying to estimate the parameters of the cosine model, we evaluate how additive random noise in the resonant scanner orientation signal is mitigated by averaging. This was investigated by first averaging 65,000 scan cycles to create a signal that could be considered “free” of random noise. Then, we subtracted this average signal from averages of less cycles and plotted their standard deviation on logarithmic axes, as shown in Fig. 3. If the zero-mean random noise was described by a normal distribution, then the calculated standard deviation would decrease with the square root of the (increasing) number of averaged cycles, that is, the slope of plotted data would be −0.5. Our data seems consistent with that hypothesis, as a least-squares fitting of this data to a linear model shows a slope value of −0.503 and a correlation coefficient of −0.9997.

Fig. 3.

Standard deviation of resonant scanner orientation signal after subtraction of the average across 65,000 cycles (assumed to be noise-free) as a function of the number of averaged cycles. The green line is the linear fit of data logarithm with a slope value of −0.503 and a correlation coefficient of −0.9997. The dashed lines highlight the average of 650 cycles (lines in one image).

Based on this plot, in what follows, we estimate the cosine model parameters from an average of 650 scan cycles of a single image, as it is keeps the noise standard deviation to approximately one tenth of a digital unit, that is, less than 0.02% of the scanner amplitude. We then use this average signal, together with the proposed robust iterative fitting of the cosine model to obtain the deterministic noise of the model as the fitting error after the final iteration. Visual examination of the fitting error with iteration number, shown in Fig. 4, reveals a residual cosine with the same period as the raw signal, but about 1,000 times smaller in amplitude, and that reduces amplitude with increasing number of fitting iterations. Comparison of the resonant scanner orientation analog signal and its digital synchronization pulses in an oscilloscope (data not shown here) suggests that the deterministic noise is due to cross-coupling of these two electrical signals. The two large excursions from zero seemingly show the derivative of the TTL pulse rising and falling edges, indicating that, perhaps, the coupling between these two electrical signals is due to an inductance.

Fig. 4.

Average scanner orientation signal and fitting error for various iterations.

3.2. Cosine parameter and latency variability

The cosine model fitting can be used to monitor and correct resonant scanning changes over time, for example, due to thermal effects. To illustrate this, we captured images of a vertical Ronchi ruling and the corresponding resonant scanner orientation data every ten minutes during a 2-hour period, with the fitted cosine parameters plotted in Fig. 5. The variation of scanning parameters during this period was measured as follows: 1.9 d.u. for scan amplitude (0.03%), 0.35 pix for scan period (0.01%), 0.5 pix for phase (0.02% of period), 0.35 d.u. for offset (0.01% of amplitude), and a more substantial 4.3 pix for latency (0.15% of period), which should be thought of as an additional phase term, and that was estimated as previously described [45].

Fig. 5.

Variation of resonant scanner orientation cosine model parameters: amplitude, period, phase, offset, and latency between optical scanning and electrical scanner orientation signal.

We can now study these small changes by examining three composite images created by vertically stacking one image row from each of the 13 image sequences captured over the 2-hour period. In the first composite image (top panel of Fig. 6) the horizontal warping was removed by using the cosine parameters and latency of the first image sequence across all 13 rows. If the scanning had not changed across the 13 datasets, then, the image would consist of vertical straight lines, but they are not. The fact that latency experienced the largest relative change of all parameters and that the curved vertical fringes resemble the latency plot rotated 90° clockwise, suggests that latency changes are the source of the fringe distortion.

Fig. 6.

Composite images formed by vertically stacking 13 lines of Ronchi ruling images captured with a resonant scanner every 10 minutes over a 2-hour period, after desinusoiding. The top panel was created by using the cosine model parameters and latency estimated for the first image line across all lines, the middle panel using the same parameters, and individualized latency values, and the bottom panel using individualized model parameters and latency.

This is further supported by the fringes in the second composite image, undistorted using the latency measured at each line and the model parameters estimated from the first image only, are straighter. Finally, we can see that the third composite image showing image rows corrected with individualized cosine model parameters and latency does not look noticeably straighter, confirming that latency is the dominant source of scanning changes over time.

3.3. Jitter estimation

Estimation of jitter using the linearized model introduced in Section 2.4 requires knowledge of the analytical derivative of the deterministic noise, which is rarely available. Instead, we propose to use the difference between consecutive samples as a coarse approximation. The absolute value of these differences can then be compared to the absolute value of the scanner orientation model derivative, that is, $-{\theta }_{\mathrm{a}}2\pi \mathrm{s}\mathrm{i}\mathrm{n}\left(2\pi t_j/T+\phi \right)/T$, to discard the samples for which the former is not negligible in comparison to the latter (see Fig. 7). In this dataset, a ratio threshold of 1/20 results in the discarding of 23% of the samples. The remaining samples are then plugged into Eq. (6) without the deterministic noise derivative term (considered negligible for these samples), to generate one set of equations for each image row, as shown in Fig. 8, with the plot slope being proportional to the desired image row jitter.

Fig. 7.

Plot of terms in the linearized scanner orientation cosine model (top), and their absolute value ratio (bottom), which is used to discard samples from the jitter estimation.

Fig. 8.

Single raw scanner orientation signal cycle (top) and its linearized version using a cosine model with jitter, with the samples no longer uniformly distributed along the x-axis (bottom).

The jitter values obtained using the linearized cosine model were compared with those from the non-linear model for all the rows in a single image as validation, first using all the samples, and then, only those with negligible deterministic noise derivative. The results, plotted in Fig. 9 below, show that the largest difference between methods was 0.003 pix (standard deviation 0.001 pix), and thus, we consider these fitting methods equivalent. In the same figure, comparison of the jitter estimated using all scanner orientation samples against only the samples with negligible deterministic noise derivative shows a maximum difference of 0.07 pix (standard deviation 0.02 pix). We consider this difference non-negligible, and therefore, conclude that the latter method is more accurate. How the improved jitter estimation benefits the imaging is application-specific, and therefore, beyond the scope of this work.

Fig. 9.

Estimated image row jitter using fitting of linear and non-linear resonant scanner orientation signal models (top panel). The standard deviation of the difference between these methods (bottom panel) including all the samples is ~0.001 pix, and when ignoring the samples most affected by deterministic noise is 0.002 pix.

Finally, we sought to compare the calculation times of linear and non-linear fitting methods implemented in Python with timing results summarized in Table 1 for various sample numbers in a general-purpose computer with an Intel Core i7–6850 K CPU (Intel Corp., Santa Clara, CA, USA). The least-squares fitting of the linearized model using either of the two evaluated Numpy algorithms reduces calculation times ~10–20 times when compared to the non-linear fitting. A custom implementation of the least-squares estimation of the slope (i.e., ignoring the intercept), shows 85–220 times faster calculation times, which translates in jitter estimation faster than 65,000 times per second when using 2,800 samples per image row, and about 500,000 times per second when using only 500 samples. This is adequate for real-time jitter correction, given that most current resonant scanners operate at resonant frequencies ≤ 30 kHz.

Table 1.

Image row estimation times (in μs) using least-squares fitting algorithms

Fitting algorithm	Full cycle (2,800 samples)	Half cycle (1,400 samples)	Quarter cycle (700 samples)	Robust quarter cycle (500 samples)
Levenberg and Marquardt	824	596	497	443
Numpy linalg.lstsq	80	48	35	31
Numpy polyfit	46	21	11	8
Numpy custom slope fitting	15	7	3	2

3.4. Desinusoiding and dejittering

The process of removing image distortion due to resonant scanner non-uniform angular velocity while ignoring jitter is often referred to as “desinusoiding.” Desinusoiding can be achieved by calculating the coordinates of new image pixels by inverting the cosine model,

$$t=\frac{T}{2\pi }\left[{\mathrm{c}\mathrm{o}\mathrm{s}}^{-1}\left(\frac{\theta -{\theta }_o}{{\theta }_{\mathrm{a}}}\right)-\phi \right],$$ (7)

with uniformly distributed (i.e., equally spaced) angular values. The pixel values at these new coordinates should then be interpolated from pixel values along the same image row, and not using adjacent row pixel values, as they are affected by jitter. The choice of interpolating algorithm, beyond the scope of this work, depends on factors including sampling in relation to the resolution limit, image blur and image noise. As we saw earlier, in Section 3.2, the phase term in the cosine model must incorporate changes in latency over time.

The calculation of dejittered images is identical to that of desinusoided images, other than for the addition of the jitter $\delta t_i$, which is unique to each image row,

$$t=\frac{T}{2\pi }\left[{\mathrm{c}\mathrm{o}\mathrm{s}}^{-1}\left(\frac{\theta -{\theta }_o}{{\theta }_{\mathrm{a}}}\right)-\phi \right]-\delta t_i.$$ (8)

These two processes, namely desinusoiding or desinusoiding and dejittering combined, yield images such as those shown in Fig. 10, where jitter correction from the linearized cosine model are shown, first with all the samples, and then with only those with negligible deterministic noise derivative.

Fig. 10.

Ronchi ruling images captured with a point-scanning instrument with a horizontal resonant scanner, and its orientation signal (top two panels), with pixels along the horizontal axis being equally separated in time. The magenta curves represent the scanner orientation and the orientation signal, respectively, averaged across all image rows, with their horizontal shift representing orientation sensor latency. The grid image is split in two intervals (cyan rectangles for left to right scan and yellow for the opposite), desinusoided (left), dejittered using the linearized cosine model using all the samples (middle) and using only those with negligible deterministic noise derivative (right).

The benefit of jitter correction could be coarsely quantified in our vertical Ronchi ruling images by calculating the standard deviation of the pixel values along each image column, as shown in Fig. 11. The numerical values will, of course, vary with the change in values between adjacent pixels, which are determined by the structure being imaged, the blur induced by the imaging system and the pixel spacing (sampling). In the particular image tested, the standard deviation of the desinusoided image columns can be as large as 20 gray levels, with dejittering bringing these values down to just above the ~2.5 noise floor set by the darker regions of the image. This noise floor is informative, because it indicates that for this particular experimental setup, little additional benefit that could be gained from a more refined dejittering method, should it become available.

Fig. 11.

Standard deviation of pixel values along the columns of the desinusoided and dejittered images in Fig. 10.

4. Summary

We presented a model of the analog electrical orientation signal provided by optical resonant scanners, improved by the consideration of deterministic and random additive noise. The model was used to demonstrate robust fitting to generate desinusoided and dejittered images, measuring and correcting minute changes in amplitude, period, phase, offset and latency over a 2-hour period. The robustness in both methods relies on the exclusion of outliers affected by deterministic noise. A previously introduced jitter estimation algorithm was linearized to reduce calculation times without sacrificing accuracy, reaching 2–15 μs per image row, easily allowing real-time generation of jitter-corrected images without specialized computing hardware.

Data availability.

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.