Correction of non-uniform angular velocity and sub-pixel jitter in optical scanning

Abstract

Optical scanners are widely used in high-resolution scientific, medical, and industrial devices. The accuracy and precision of these instruments are often limited by angular speed fluctuations due to rotational inertia and by poor synchronization between scanning and light detection, respectively. Here we demonstrate that both problems can be mitigated by recording scanner orientation in synchrony with light detection, followed by data resampling. This approach is illustrated with synthetic and experimental data from a point-scanning microscope with a resonant scanner and a non-resonant scanner. Fitting of the resonant scanner orientation data to a cosine model was used to correct image warping and sampling jitter, as well as to precisely interleave image lines collected during the clockwise and counterclockwise resonant scanner portions of the rotation cycle. Vertical scanner orientation data interpolation was used to correct image distortion due to angular speed fluctuations following abrupt control signal changes.

1. Introduction

Optical scanners [1–3] are used in applications as diverse as intraoperative imaging [4–6], ophthalmic imaging [7–9], microscopy [10–12], optical coherence tomography [13–15], DNA sequencing [16], automotive LIDAR [17–19], high resolution displays [20–23], flow cytometry [24], quality control [25], underwater imaging [26], data storage [27] and printing [28,29].

Reproducible but undesired scanning speed variations introduce static distortion in images sampled at uniform time intervals, as illustrated by the images shown in Fig. 1(a) and 1(b). On the left, an image of a grid of vertical lines shows the horizontal warping characteristic of the resonant scanner cosinusoidal oscillations, which causes features near the left and right of the field of view appear horizontally stretched. Image distortion due to reproducible angular speed oscillation in a vertical scanner can be seen in the image of a grid of squares in panel (b). Both types of distortion can be corrected by interpolating the values of uniformly spaced pixels, after calibration using a known, and typically periodic, test object such as the Ronchi ruling seen in Fig. 1(a) [9,30–35]. This approach, however, is only accurate if the calibration object allows adequate (i.e., Nyquist or higher) sampling of the scanner orientation within the field of view, or adequate fitting of a physical model of the scanning cycle. An example of the latter is the cosinusoidal modeling of resonant galvanometric scanners [8,9,36–39]. Alternative distortion correction methods, using non-uniform temporal pixel sampling to achieve uniform angular or spatial sampling, require complex electronics known to be affected by temperature changes, suffer from random pixel clock phase errors, and to date, have limited precision [10,40,41].

Fig. 1.

Images of grids of lines and squares captured with a point-scanning microscope to illustrate: (a) horizontal image stretching away from the image center due to the cosinusoidal angular speed variation of the horizontal resonant scanner; (b) vertical image warping (after horizontal cosinusoidal warping correction) seen as wavy square edges, due to angular speed oscillations of the vertical scanner around its nominal constant speed; (c) and (d) images after horizontal warping correction showing line jitter due to scanners with 15.1 and 13.9 kHz resonant frequencies, respectively, unsynchronized from the 40 MHz pixel sampling clock.

In addition to static image distortion, changes in scanner angular speed over short and long periods of time due to, for example, vibrations or thermal changes, respectively, introduce dynamic image distortion [40,42–45]. Lack or poor correction of static and/or dynamic distortion degrades the accuracy in the estimation of image feature dimensions, spacing and localization.

A second issue is that an imperfect or lack of synchronization between optical scanner(s) and image sampling results in jitter between image lines, as it can be seen in Fig. 1 panels (c) and (d), degrading resolution and localization of image features. In resonant scanners, which are of particular importance for high-speed and high-throughput applications, it is almost always the case that the scanner cycle period is not an exact number of ticks of the sampling clock. This results in line jitter with amplitude in the [−0.5, 0.5] pixel range, irrespective of the optical setup [31,40,41,46,47]. This jitter can be more or less evident, as the panels in Fig. 1 illustrate with images captured using two scanners of different resonant frequencies. This jitter can be mitigated using complex electronics that are known to be sensitive to noise and require non-trivial tuning [40,48–52]. An alternative approach using a secondary optical system in combination with custom electronics recently showed substantial jitter suppression [47]. However, due to its complexity, cost, and opto-mechanical limitations, this method is not generally applicable.

A third issue is the doubling of imaging frame rates by capturing data during both the clockwise and counterclockwise rotation of the faster (often horizontal) scanner. Poor interleaving of these clockwise and counterclockwise image data manifests as line jitter between alternating image lines [41,44,45,53,54]. Yang et al. [44] recently used the convolution of the clockwise and counterclockwise images to address this problem with ∼0.5 pixel accuracy. This approach, however, does not correct the jitter due to poor or lack of synchronization between the scanning and imaging sampling, and it relies on assumptions of image content that are not always met.

In what follows, we demonstrate that the sampling of a scanner orientation signal in synchrony with the image sampling can be used to correct image distortions due to non-uniform angular speed, sub-pixel jitter, and precise image interleaving in resonant scanners, as well as non-uniform angular speed in non-resonant scanners. This approach exploits built-in analog orientation sensors present in most optical scanners [55–60]. In the methods section that follows, we describe: the scanning imaging instrument used for this work; a cosinusoidal model used to fit the resonant scanner oscillation; and a local interpolation alternative for when models of the scanning angular speed are not available or desired. The results section presents the validation of the distortion and jitter correction algorithm using synthetic and experimental resonant scanner data, followed by image distortion correction using experimental data from a piezo-ceramic scanner driven by a sawtooth signal. Finally, in the discussion section, we summarize the results and their implications for imaging and sensing applications.

2. Methods

The proposed correction of distortion and jitter in images or sensing data from scanning instruments, consists of three steps. First, the sampling of the scanner orientation in synchrony with the sampling of the light detectors, followed by the calculation of the coordinates of the desired uniformly spaced samples, and finally, the calculation of the sample values at these new coordinates through interpolation.

2.1. Image and scanner orientation sampling

A custom scanning ophthalmoscope [9], which for the purpose of this work can be thought of as an epi-illumination reflectance confocal microscope, was used to collect synchronous scanner orientation and image samples. The adaptive optics components of the instrument play no role in this work. In this device, a rectangular imaging raster was created with a horizontal galvanometric scanner with either 15.1 or 13.9 kHz resonant frequency (model SC-30; Electro-Optical Products Corporation, Ridgewood, NY, USA) and a piezo-ceramic vertical scanner (model S334; Physik Instrumente, Karlsruhe, Germany), driven by a sawtooth control signal. For the images captured to demonstrate the horizontal cosinusoidal warp and jitter correction, this sawtooth rise and fall times were 10 and 50 ms, respectively, with the images being captured 10 ms after the start of the fall period, to mitigate the angular velocity oscillations during the imaging portion of the scan cycle. For the images captured to demonstrate the correction of image distortion due to angular velocity fluctuation in the vertical scanner, the rise and fall times were 0 and 55 ms, respectively, with the imaging starting with the fall portion of the scan cycle. Starting imaging right after the scanner input discontinuity was intentionally done to capture a worst-case scenario that best illustrated the benefit of the proposed distortion correction method.

The scanner orientation and image samples were captured in synchrony using 3 of the 16 channels in a 14-bit waveform digitizer operating at 40 MHz (model ATS9416; Alazartech, Pointe-Claire, QC, Canada), as depicted in Fig. 2 below. In this way, each traditional image now had two corresponding images, in which their pixel values encode the orientation of the vertical and horizontal optical scanners, respectively (see Fig. 2). The electronic drivers of both scanners provide analog orientation signals that were scaled down with voltage dividers formed by an external resistor (270 Ω for the horizontal scanner and 510 Ω for the vertical scanner), and the 50 Ω internal resistance of the Alazartech digitizer channel inputs [61].

Fig. 2.

Depiction of the point-scanning instrument electrical connections used for synchronous sampling of object images (top), horizontal scanner orientation (middle) and vertical scanner orientation (bottom), achieving 1:1 pixel correspondence. The lines in the horizontal scanner image in this case correspond to a cosine that captures both the image warp and sampling jitter, while the vertical scanner orientation image captures the vertical ramp driving the scanner and the orientation fluctuations due to the scanner angular inertia. The two gray rectangles indicate the voltage division performed by passive resistors which reduce the amplitude of the scanner orientation signal so that it is with the dynamic range of the digitizer. Note that the object image (top) is symmetric because pixels are capture through both the clockwise and counterclockwise portions of the scanner cycle.

2.2. Global fitting of scanner motion

Often, the functional variation of a scanner orientation or speed with time is known, as it is the case with resonant scanners. In this situation, the scanner orientation data within each image line can be used to perform a single “global” fit to a cosinusoidal model of the digitized scanner orientation values ${\theta }_i$ captured at times $t_i$ as,

$${\theta }_i={\theta }_a\cos \left(\frac{2\pi }{T}{t}_i+\phi \right)+{\theta }_o,$$ (1)

where ${\theta }_a$ is the cosine amplitude, T is the cosine period, $\phi$ is the cosine phase at time $t=0$ , and ${\theta }_o$ is an angular offset.

This 4-parameter model could be used to fit each individual image line, although this might require unacceptably long computing times or resources. This is the case in our instrument, as we aspire to implement this jitter correction in real-time, that is, at ∼20 frames per second, with each frame having > 500 lines. So, after some tests in our instrument, we determine that during the few seconds that data capture takes in our application (retinal imaging), the cosine amplitude, period and offset do not change appreciably. Thus, we pursued a two-step fitting approach. First, we average the recorded scanner orientation values across the 650 image lines and performed a least-squares fit using the Nelder-Meade minimization algorithm. This minimization, using ∼2500 values, took ∼15–20 ms using Matlab’s fminsearch function (Mathworks, Natick, MA, USA) driven by the Frobenius norm of the difference between the fitted and the averaged experimental data. We then used these cosine amplitude, period and offset values fitted using the average data to fit the phase of each line individually. That is, a second fitting in which only the phase of the model was allowed to vary was implemented, while the amplitude, period and offset values remain fixed. This second one-parameter fitting was about 20 times faster (0.7–0.8 ms), required about 0.5 s/frame using Matlab’s fminbnd function, a 1-dimensional minimizer, driven by the same metric as the first fit and using a single computing thread. This fitting is parallelizable, as the phase (jitter) of the cosine model for each image line is independent from the rest, and thus, amenable to real-time implementation.

The parameters from these fitting steps, namely amplitude, period, offset and phases (one per image line), were used to unwarp each line from the raw image as follows. First, we calculated the coordinates of the pixels of the unwarped (raw) image, in which angular separation in pixels (rather than time) is uniform, by inverting the cosine model, that is by solving Eq. (1) for $t_i$ with equally spaced $x_i$ values, over each monotonic cosine interval. Then, the corresponding pixel values were calculated using Matlab’s interp1 function (Mathworks, Natick, MA, USA) as a 1-dimensional spline interpolator. Unwarping the image in this manner using the same phase, estimated from the average sensor signal, in all lines, yields “desinusoided” images. If instead, we unwarped each line using its own phase estimate, then the resulting image is both unwarped and dejittered. Finally, a third image can be generated by interleaving the unwarped and dejittered images corresponding to the clockwise and counterclockwise rotation portions of the scan. These images are often used to double the frame rate of instruments with resonant scanners. Key to this interleaving is the estimation of the “pseudo-latency” between the optical sampling and the sensor signal. Here by pseudo-latency, we mean the half a cosine cycle modulus of the actual latency. For this work, we first coarsely (∼1 pixel) estimated this latency by capturing a single bi-directional image of a vertical Ronchi ruling with a stationary vertical scanner, and then finding the location of the auto-convolution maximum. This process allows finding the pixel in which the scanner reverses the sign of its angular speed, relative to one of the cosine extremes of the fitted model (a similar approach to that used by Yang et al. [44]). This coarse latency estimation was then refined by using the 1-dimensional minimizer fminbnd Matlab function driven by the norm of the difference of the images to be interleaved (i.e., the images captured when the scanner was rotating clockwise and counterclockwise). We have observed that this latency changes with temperature (data not shown), shows repeatable 2-pixel latency change during the first 20 minutes of operation, an additional ∼0.15 pixels in the following 40 minutes, and less than 0.1 pix thereafter.

2.3. Local fitting of scanner motion

When the variation of a scanner orientation over time cannot be easily modeled, the local fitting performed by common interpolating algorithms can be used instead. This approach will be demonstrated in the next section with the vertical scanner of our instrument driven by a sawtooth control signal. In this case, for simplicity we will only consider the portion of the scan cycle in which the scanner orientation signal changes monotonically. The calculation of both the uniformly spaced new pixel coordinates, and the actual pixel values, were performed using the previously mentioned 1-dimensional spline interpolator. For this or any other interpolation to be adequate, the scanner orientation must be sampled at or higher than its Nyquist frequency, which is not related to the Nyquist frequency of the imaging. This Nyquist limit could be obtained either from the manufacturer’s data sheet (e.g., known resonant frequencies), or experimentally. In our case, the 40 MHz sampling is orders of magnitude higher than the highest frequency oscillation that we see in the response of our vertical scanner to a step function (∼1.2 kHz). Thus, we consider our scanner orientation sampling adequate.

3. Results

3.1. Resonant scanner synthetic data

The unwarping and dejittering algorithm for a scanner with cosinusoidal orientation variation was validated using simulated data with jitter and images of a vertical grid of fringes with cosinusoidal intensity cross-section, as shown on the top two panels of Fig. 3. The data span both the clockwise and counterclockwise portions of the resonant scanner cycle, indicated by the cyan and yellow rectangles, respectively, with the arrows indicating the scanning direction. The simulated scanner amplitude (1000 digital units), the scanner period (2500 pixels) and the offset (1000 digital units), were the same for all image rows, with only the cosine model phase randomly varying between lines (average value π/20), assuming zero latency between the imaging and the sensor signal.

Fig. 3.

Ronchi ruling images from a point-scanning instrument with a horizontal resonant scanner, and its orientation signal (top two panels), with pixels along the horizontal axis being equally separated in time. The magenta curves represent the scanner orientation and the orientation signal, respectively, averaged across all image rows, with their horizontal shift representing orientation sensor simulated latency (zero). The grid image is split in two intervals (cyan rectangles for left to right scan and yellow for the opposite), unwarped (left), dejittered (middle) and interleaved (right). In these panels, the horizontal axis corresponds to points equally separated in space (as opposed to time).

The 4-parameter cosinusoidal fitting of the scanner orientation signal averaged across all 650 simulated image rows retrieved jitter with 0.005 pixel accuracy (standard deviation of absolute error) and the other cosine parameters with ≤ 0.3% accuracy (also standard deviation of absolute error). The fitting parameters were then used to calculate unwarped, unwarped and dejittered, and interleaved (also unwarped and dejittered) images shown in the central and bottom panels of Fig. 3, for both the clockwise and counterclockwise portions of the scan cycle.

3.2. Resonant scanner experimental data

A second test of the proposed unwarping and dejittering algorithm was performed using captured (as opposed to simulated) images of a Ronchi ruling with its fringes oriented along the vertical direction while the vertical scanner was stationary. In this way, each image line shows the same portion of the grid (other than for jitter), eliminating image distortion due to the instrument optics as a confounding factor. The raw and processed images in Fig. 4, show that the jitter after correction of the warping due to the cosinusoidal scanning alone (left column), is ∼0.3 pixels (left bottom plot), which is consistent with the standard deviation of a uniformly distributed set of random numbers in the [−0.5, 0.5] interval. This jitter was estimated by upsampling each image line using Matlab’s interp1 spline interpolator and then finding the shift of the peak of the normalized cross-correlation with the first image line. The same jitter estimation method applied to the dejittered and interleaved images (center and right columns, respectively) showed jitter mitigation by a factor of ∼6.7. This jitter correction method was applied to a sequence of 20 raw images (Visualization 1 ^{(2.2MB, avi)} ) in uncompressed monochrome AVI files which produced two unwarped image sequences (Visualization 2 ^{(5.7MB, avi)} and Visualization 3 ^{(5.7MB, avi)} ), two unwarped and dejittered image sequences (Visualization 4 ^{(5.7MB, avi)} and Visualization 5 ^{(5.7MB, avi)} ) which were interleaved to create Visualization 6 ^{(11.3MB, avi)} . We recommend viewing these at 300% magnification or higher to appreciate the jitter correction.

Fig. 4.

Ronchi ruling images from a point-scanning instrument with a horizontal resonant scanner, and its orientation signal (top two panels), with pixels along the horizontal axis being equally separated in time. The magenta curves represent the scanner orientation and the orientation signal, respectively, averaged across all image rows, with their horizontal shift representing orientation sensor latency. The grid image is split in two intervals (cyan rectangles for left to right scan and yellow for the opposite), unwarped (left), dejittered (middle) and interleaved (right). The bottom plots show the line jitter on the corresponding images.

3.3. Resonant scanner and linear vertical scanning experimental data

In this final test, we provide an example of a scanner movement that might not be trivial to model, and that is relevant to a large fraction of point- or line-scanning instruments. Here the vertical scanner was controlled by a sawtooth with a 65 ms period, with a discontinuity at the end of each cycle. The plots in Fig. 5 show that the scanner actual orientation (top panel), takes ∼50 image lines to catch up with the abrupt change in control signal, due to its angular inertia and the latency of the driver electronics. More subtle, however, is the oscillatory angular speed changes around its desired value (bottom panel), which are attenuated towards the bottom of the image and appear to be dominated by a slow and a high frequency oscillation. The latter corresponds to the mechanical resonant frequency of the scanner itself, as reported by the manufacturer (∼1 kHz). Each point in these plots corresponds to the average of the scanner orientation signals across each image row and across 100 images. Using less averaging in this scanner results in poor image distortion correction. Importantly, the angular speed plot informs of the image distortion amplitude, which is almost a full row at the start of the image and about an order of magnitude smaller towards the bottom of the image.

Fig. 5.

Vertical scanner orientation and its control signal over time in units of image rows (top) and speed (bottom) over a monotonic portion of the rotation.

In the interest of keeping the interpolation process simple, we only used the portion of the image over which the scanner orientation signal is monotonic (green curves to the right of the vertical dashed lines in Fig. 5). In this case, the test object is a rotated grid of squares after the horizontal unwarping and dejittering discussed above (central panel in Fig. 6). The rotation of the grid reveals the vertical oscillation through the wavy edges of the squares which are more clearly visible towards the top of the image (top-central panel of Fig. 6), and that are drastically reduced after unwarping (see top-right panel of Fig. 6) based on the digitized analog scanner orientation signal.

Fig. 6.

Point-scanning images of a rotated square grid captured with a horizontal resonant scanner and a vertical non-resonant scanner driven by a sawtooth control signal. The effect of the sawtooth discontinuity can be seen at the top of the image on the left center panel, in which the grid squares appear massively compressed. Because of this, we only try to unwarp the portion of the image in which the vertical scanner angular speed has positive sign (i.e., the image lines are monotonically sorted), as highlighted by the green rectangle. The magnified insets show how the image unwarping noticeably mitigates the vertical oscillations, as revealed by the correction of the wavy edges of the squares.

4. Discussion

Optical scanners are widely used in high-precision scientific, medical, industrial and commercial imaging and sensing applications. The precision and accuracy of instruments using mechanical optical scanners, irrespective of the method of actuation (e.g., piezo-ceramic, galvanometric) can be improved through the sampling of analog orientation signals provided by scanner electronic drivers. Here we demonstrated how the scanner orientation can be performed in synchrony with the image (or sensor) data acquisition without modifying the optical setup or custom electronics, except (potentially) a passive voltage divider formed by two resistors. This sampling can then be used to resample the imaging or sensing data to correct for non-uniform scanner angular speed and poor or no synchronization between scanning and sampling. The coordinate transformations (resampling) to unwarp and dejitter the images should precede or be combined with correction of image distortion due to the optics of the imaging system, to avoid multi-step interpolations.

In addition to showing the validation of the proposed approach using synthetic data, we demonstrate its successful application to data from an imaging instrument with both a resonant and a non-resonant optical scanner. It should be noted that the benefit of dejittering depends on the image sampling relative to the resolution element due to the optics. This is because in most instruments the jitter is in the [−0.5, +0.5] pixel interval, irrespective of the optics and/or wavelength(s) of light used. Therefore, the lower the spatial sampling in the image relative to the resolution element, the greater the benefit of the jitter correction.

Finally, let us emphasize that the approach presented here, irrespective of fitting algorithm and model or interpolation method, is applicable to scanning patterns beyond the rectangular scanning raster demonstrated here. Scanning patterns in which the angular acceleration is greater will show the greater performance benefit, when the proposed correction is applied. Examples of such non-conventional patterns can be seen in optical coherence tomography for speckle reduction [62], angiography [63], improved spatial registration [64,65], as well as patterned or random access microscopy [66–68].

Funding

Research to Prevent Blindness10.13039/100001818 (Departmental award); National Eye Institute10.13039/100000053 (P30EY026877, R01EY025231, R01EY031360, R01EY032147, R01EY032669).

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.