Software-Based Phase Control, Video-Rate Imaging, and Real-Time Mosaicing with a Lissajous-Scanned Confocal Microscope

Abstract

We present software-based methods for automatic phase control and for mosaicing high-speed, Lissajous-scanned images. To achieve imaging speeds fast enough for mosaicing, we first increase the image update rate tenfold from 3 to 30 Hz, then vertically interpolate each sparse image in real-time to eliminate fixed pattern noise. We validate our methods by imaging fluorescent beads and automatically maintaining phase control over the course of one hour. We then image fixed mouse brain tissues at varying update rates and compare the resulting mosaics. Using reconstructed image data as feedback for phase control eliminates the need for phase sensors and feedback controllers, enabling long-term imaging experiments without additional hardware. Mosaicing subsampled images results in video-rate imaging speeds, nearly fully recovered spatial resolution, and millimeter-scale fields of view.

I. Introduction

MINIATURIZED, high resolution, laser scanning microscopes are usually constrained by small fields of view (FOVs) or slow frame rates due to mechanically scanning across an area, compiling images point-by-point. These devices commonly use microelectromechanical systems (MEMS) scanning mirrors [1], [2], [3] and piezoelectric fiber scanners [4], [5] for beam deflection. Raster scanning, in which one axis is driven at resonance while the other is scanned much more slowly, is simple to implement, but is susceptible to small FOVs, low electromechanical efficiency, and image warping caused by low frequency mirror actuation (Figure 1a). Lissajous scanning, in which both axes are scanned at resonance [6], maximizes FOV while using lower driving voltages, but suffers from inefficient sampling (Figure 1b) and phase drift (changes in phase lag over time). In this work, we aim to address the traditional disadvantages of Lissajous imaging with a miniaturized microscope: inefficient spatial sampling, slow frame rates, small FOVs, and phase drift.

Fig. 1.

Image comparison between (a) raster scanning and (b) Lissajous scanning using the same microscope and MEMS mirror. This mirror exhibits signs of inter-axis coupling, and thus distortion, during raster scanning. The Lissajous scan achieves a larger FOV and is less susceptible to image warping, but suffers from a slight phase misalignment and a few unsampled pixels near the center of the image. Group six of a reflective 1951 USAF resolution target shown. The FOVs in (a) and (b) are 90 μm × 215 μm (approximate due to warping) and 140 μm × 260 μm, respectively. Scale bars indicate 50 μm.

Lissajous patterns’ inefficient sampling has previously been addressed either by choosing driving frequency combinations resulting in nonrepeating patterns [7], or by updating the image prior to complete image sampling [7], [8]. On its own, increasing update rate results in undersampled, sparse images, reducing spatial resolution and requiring interpolation.

Reconstruction techniques developed to improve on linear interpolation can be categorized into three groups: (1) basic interpolation using local averaging; (2) inpainting [9], [10], which extends image features into unsampled regions; and (3) compressive sensing reconstruction [11], which attempts to find the sparsest description that fits the sampled data. Some of these techniques require minor modification to beam-scanning hardware and additional control and data acquisition electronics, and all of these techniques are computationally complex, requiring on the order of seconds to reconstruct each image on a standard computer.

Lissajous patterns slow frame rates have precluded the implementation of image mosaicing due to motion-based warping and blurring. Mosaicing is a fundamental technique in image processing in which consecutive images (frames) are automatically registered, de-warped, overlapped, and blended together to build a single, large image. This process increases both FOV and signal-to-noise ratio (SNR), and enables software-based image stabilization, but depends on fast frame rates. Previous works in this field has focused on applying mosaicing using fixed [12], fibered [13], and handheld [14], [15] microscopes, some in real-time [16], [17], [18].

The lack of phase control in Lissajous scanning has mostly led the field toward hardware-based solutions to measure the phase between driving waveforms [19], [20]. Phase-locked loops (PLLs) are most commonly used for tracking and controlling periodic signals [19], and have been applied in MEMS devices [21], AFM [22], and resonant fiber scanners [23]. Phase control hardware increases both the size and cost of the optical system, inhibiting miniaturization. For cases in which the Q-factor of the scanner is sufficiently low, and the imaging duration relatively short, manual adjustment of phase through a graphical user interface (GUI) is possible [7].

Our solution for tackling inefficient sampling, slow frame rates, small FOVs, and phase drift is comprised of four steps: First, in Section II, we boost the update rate of Lissajous-scanned images, thereby subsampling the FOV to form a sparse image. Second, in Section III, we interpolate the image in real-time, obtaining a coherent image with which to perform phase measurements and template matching. Third, in Section IV, we analyze every n^th image in real-time to maintain phase control for each axis. Lastly, in Section V, we mosaic images together to recover the resolution and SNR that was sacrificed by undersampling. We demonstrate these techniques using a handheld, dual-axis, confocal (DAC) fluorescent microscope utilizing a 2D MEMS scanning mirror [24]. This microscope features a 10 mm outer diameter, with all optical components and beam paths contained within the central 5 mm. Excitation occurs at 660 nm and 785 nm. This solution enables high-speed Lissajous imaging with high resolution, large fields of view, and automatic phase control.

II. Spatial Scanning and Sampling

Pattern repeat rate, $f_{pr}$, describes the periodicity of a 2D scanning pattern. Update rate, $f_u$, is the rate at which consecutive frames are displayed and saved, regardless of how completely we sample the image’s FOV. We distinguish $f_u$ from frame rate, a term that usually implies full frame sampling prior to update (i.e., when $f_{pr}=f_u$). All of these rates are expressed in units of frames per second (fps), or Hz.

A. Scanning Patterns

The complex harmonic motion of a Lissajous curve is described by the parametric equations:

$$x(t)=\frac{1}{2}{A}_x\mathit{sin}\left(2\pi f_{x}t+{\phi }_x\right)+\frac{1}{2}{A}_x,$$ (1)

$$y(t)=\frac{1}{2}{A}_{y}sin\left(2\pi f_{y}t+{\phi }_y\right)+\frac{1}{2}{A}_y.$$ (2)

In these equations, f is driving frequency, $\varphi$ is instantaneous phase, and A is the extent of FOV (amplitude) for each axis. Phase is a function of both frequency and elapsed time (Δt), and must be calculated in real-time in order to smoothly match driving signals and to properly construct images:

$$\phi =2\pi f\text{Δ}t.$$ (3)

In the case of raster scanning, $f_u=f_{pr}$, and the scanning patterns shape is insensitive to frequency (Figure 2a). The scanning paths of Lissajous curves, on the other hand, are highly sensitive to the ratio $f_x/f_y$ [25] (Figures 2b-d). For repeating patterns (Figure 2c), which occur if and only if $f_x/f_y$ is rational, the pattern repeat rate, $f_{pr}$, is given by

$$f_{pr}=\frac{f_x}{n_x}=\frac{f_y}{n_y},$$ (4)

where n_x and n_y are the smallest integer divisors of f_x and f_y, respectively, and determine the number of cycles scanned on each axis, each period [8]. This equation introduces an inherent tradeoff for repeating Lissajous patterns: increasing line density decreases pattern repeat rate. In other words, lateral resolution and frame rate are inversely related.

Fig. 2.

Common laser scanning driving modes. (a) Raster scan, in which $fy\ll fx$. (b) Lissajous radial scan, in which $f_x=f_y$, ${\phi }_x=0°$, ${\phi }_y=-90°$, and both driving amplitudes ramp at the same linear rate. (c) Repeating Lissajous scan, in which $f_x/f_y$ is rational. (d) Nonrepeating, or sliding, Lissajous scan, in which $f_x/f_y$ is irrational. Less commonly used Lissajous patterns for imaging include a line scan, in which $f_x=f_y$ and ${\phi }_x={\phi }_y=-90°$, and a parabola scan, in which $f_x/f_y$ = 2 and ${\phi }_y=90°$ (neither shown). Color represents elapsed time during a single frame period. Simulations generated in Matlab.

While this work focuses on repeating Lissajous scanning patterns (Figure 2c), the techniques described later are applicable to all of the scanning patterns shown in Figure 2. We elected to use a repeating pattern so that we could directly compare images with identical scanning patterns (and thus interpolation patterns, discussed later) to determine how image quality is affected by sampling, interpolation, and phase control, especially for undersampled images. If we were to use nonrepeating patterns, each image would be sampled and interpolated at slightly different spatial locations, making it difficult to directly compare images and quantify results, particularly during phase control testing.

B. Spatial Sampling and Update Rate

Mechanical scanning properties (scanning amplitude, frequency, phase, etc.), optical resolution, sample rate, and update rate all affect sampling sparsity. Of these parameters, update rate is the simplest to manipulate. The effect of varying update rate on spatial sampling is shown in Figure 3. White pixels in Figure 3a-d correspond to unsampled “gaps” in the FOV. Gap height is the vertical distance between sampled pixels, which varies as a function of position, and maximum gap height occurs near the center of the FOV. Note that because scanning frequency determines the number of lobes and periods for each axis, gap height is almost always less than gap width because $f_x\gg f_y$ for our scanner. Although these varying gaps all affect image quality, image resolution is ultimately limited by the maximumx gap height.

Fig. 3.

Sampling efficiency and spatial resolution versus update rate. (a) 3 Hz ($f_u=f_{pr}$), (b) 10 Hz, (c) 20 Hz, and (d) 30 Hz Lissajous scans. Color represents the number of times each pixel is sampled, with yellow, blue, and white representing greatly oversampled, Nyquist-rate sampled, and unsampled pixels, respectively. (e) Plot showing how sampling is affected by update rate. (f) Plot showing how spatial resolution is affected by update rate. All scanning parameters other than sample rate match those used for our microscope: FOV = 250 pixels × 450 pixels, f_x = 2034 Hz, f_y = 315, and starting phases ${\phi }_x={\phi }_y=-90°$. Sample rate SR = 5.77 MHz, satisfying Nyquist. Simulations generated in Matlab.

The pattern in Figure 3a naturally closes because $f_x/f_y$ is rational for our scanner (f_x = 2,034Hz and f_y = 315Hz). From Equation 4, the pattern has a natural repeat rate of 3 Hz. Reducing the update rate to below 3 Hz does not reduce the maximum gap height, regardless of starting phase. Increasing the update rate past the repeat rate increases the maximum gap height for each frame and causes sequential frames to have different scanning trajectories until the pattern reaches the end of its period.

Figure 3e shows that spatial sampling efficiency (how often we sample each pixel just once) improves logarithmically with update rate. For example, as we increase update rate from 3 Hz (Figure 3a) to 30 Hz (Figure 3d), the number of greatly oversampled pixels (those sampled ten times or more per frame) decreases by ~95%, while the number of unsampled pixels increases by just ~ 64%, depending on initial phase. Note that there are plateaus in the red plot of Figure 3e, and in both plots of Figure 3f extending from 1 Hz to 17 Hz, rather than from 1 Hz to 3 Hz as expected. This is due to the chosen starting phase combination, and is not representative of all starting phase combinations.

Figure 3 is calculated using the same initial phase value of 90°. In Figure 3a, $f_{pr}=f_u$, meaning each axis’ phase value ends precisely where it starts, at 90°. Thus the scanning patterns for two consecutive frames are identical. In Figures 3b-d, these traits no longer occur since $f_{pr}>f_u$. In these cases, ending phase does not match starting phase, and consecutive frames have different scanning patterns, sampling different regions in the FOV. This phenomenon will prove to be vital in Sections III and V.

C. Temporal Sampling

For one-dimensional harmonic oscillation, the scanning points maximum velocity $v_{max}$ is

$$v_{\mathit{max}}=\pi Af.$$ (5)

Pythagorean theorem gives us maximum velocity for a two-dimensional system:

$$v_{max}=\sqrt{v_x^2+v_y^2}=\pi \sqrt{A_x^2{f}_x^2+A_y^2{f}_y^2}.$$ (6)

Maximum velocity occurs at the equilibrium position for our mass-spring system, in the center of our FOV, when acceleration is minimized and the MEMS springs are neither pushing nor pulling on the mirror. This explains why Lissajous images can exhibit poor sampling and resolution near the center of the FOV. Plugging in values for our microscope’s MEMS mirror (370μm, 650μm, 315Hz, 2.034kHz for A_x, A_y, f_x, and f_y, respectively) results in $v_{max}=4.17m/s$.

The experimentally measured full width at half maximum (FWHM) spatial resolution of the microscope is 3.0 μm for both horizontal and vertical axes, determined by scanning a mirrored knife-edge target across the beam. This represents the diffraction-limited, transverse confocal response of the system [26]. A digital acquisition (DAQ) card samples voltages from the photomultiplier tube (PMT) at a sample rate (SR) of 10 MHz. Using our value for $v_{max}$, our spatial sampling rate is ≤ 0.417μm/pixel, greatly exceeding Nyquists sampling criterion.

To form the final image, each sample is integrated with the nearest pixel of a 250 × 450 pixel image using sample locations calculated ahead of time. With a 3 μm resolution and a 370 × 650 μm FOV, this image size gives us a ratio above 2 pixels per resolution element, also satisfying Nyquist. An integration map is incremented by one at its corresponding pixel location for each sample. The final image is then the quotient of the integrated signal divided by the integration map, and unsampled pixels are assigned to zero.

III. Image Interpolation

With the goal of broad applicability and real-time, video-rate use in mind, we linearly interpolate along the vertical axis:

$$I=I_1+\left(y-y_1\right)\frac{I_2-I_1}{y_2-y_1},$$ (7)

where y and I are pixel location and intensity, respectively. This process takes less than 1.5 ms for each 250 × 450 pixel image, more than fast enough for real-time use at 30 Hz.

Vertical interpolation has three benefits: First, interpolation occurs along the shortest route between sampled pixels. Second, horizontal resolution of static mosaics is unaffected because interpolated pixels are not averaged with sampled pixels during the blending process. Thus, for undersampled, interpolated images at fast update rates ($f_u>f_{pr}$), maximum gap height is the limiting factor in spatial resolution of static mosaics, and is a function of update rate alone. For moving samples, this effect is overwhelmed by motion artifacts and registration errors. Third, although interpolation has a low-pass filtering effect, this effect is constrained to unsampled pixels; sampled pixels are unaffected. This is necessary for resolution recovery through mosaicing.

Figure 4 shows the vertical blurring effect for individual, undersampled images (Figures 4b,e), and for static mosaics (Figures 4c,f). In calculating Figures 4c,f, thirty consecutive frames are averaged together, each with image quality comparable to that of Figures 4b,e. However, because each frame adds new spatial information through sampling slightly different scanning patterns, as discussed in Section III, the end result is comparable to simple integration (Figures 4a,d). Thus, through static mosaicing, we recover lateral resolution and SNR lost through subsampling to increase temporal sensitivity. Note that if we were to keep track of sampled locations, avoid interpolating, and then integrate according to number of times a given pixel is sampled (as apposed to simple averaging), we would obtain results identical to those shown in Figures 4a,d.

Fig. 4.

Image quality versus update rate, and its potential effect on mosaicing. From top to bottom: (a₁, d₁) a single 1 Hz frame; (b₁, e₁) a single 30 Hz frame after interpolation; and (c₁, f₁) 1 second’s worth of 30 Hz images averaged together, representing a static mosaic. (a₂-f₂) Corresponding 2D Fourier transforms, contrast enhanced to show detail and cropped to conserve space. In comparison to 1 Hz images, 30 Hz images show significant degradation due to their sparse sampling pattern. 30 Hz images integrated for the same amount of time show only a slight decrease in vertical resolution, confirmed by the darker top and bottom regions of the corresponding spectra. (a₁-c₁) 15 μm diameter fluorescent beads. (d₁-f₁) Fixed and cleared mouse brain tissue, labeled with LIQOR785 (red) and DRAQ5 (green) with dimethyl sulfoxide (DMSO) to increase label penetration. The vertical dark feature is a micro-tear in the tissue sample. Scale bars indicate 100 μm.

Interpolation is crucial for both mosaicing and phase feedback. For mosaicing, image sparsity introduces high contrast corners and patterning that would prohibit feature detection and image registration. For phase feedback, this patterning introduces high-frequency artifacts in the Fourier domain, skewing the signal used for phase alignment.

IV. Phase Control

A. The Phase Control Problem

The MEMS scanner at the heart of our microscope is an underdamped oscillator driven in open-loop with a sinusoidal force. The phase lag $\varphi$ (or mapping phase) between the oscillation and driving force is

$$\varphi =atan\left(\frac{2\omega {\omega }_0\zeta }{{\omega }^2-{\omega }_0^2}\right)+n\pi ,$$ (8)

where ω is the driving frequency, ω₀ is the undamped angular frequency, and $\zeta$ is the damping ratio. If instantaneous phase $\phi$ (from Eqs. 1 and 2) defines an input signal’s position relative to one full period, phase lag $\varphi$ measures the physical system’s delay relative to that input signal. Both values are needed for accurate image reconstruction. As driving frequency $\omega$ approaches and surpasses resonant frequency ${\omega }_r$, phase lag transitions from 0° to −180°. As damping ratio decreases, quality factor (Q-factor) increases, resulting in higher resonance amplitude (enabling larger FOVs) and phase sensitivity to frequency. This destabilizes the scanner, making it more difficult to control.

For raster scanning, phase drifts are presented as small shifts of the entire image along the resonant axis. For Lissajous scanning, phase drift occurs for both axes, causing interlaced copies of the image to shift along the vertical and horizontal axes, despite the beam path scanning along changing, non-orthogonal directions. These interlaced copies are sometimes referred to as fields, referring to when the mirror is, for example, scanning left to right (even) or right to left (odd). When these fields aren’t properly aligned, high frequency artifacts are introduced into the image, artificially increasing contrast.

B. Phase Algorithm and Sweep

The algorithm for determining phase lag for Lissajous scanned images is shown in Algorithm 1 and Figure 5. The intent is to generate a signal over a range of test phase lag values for which the peak corresponds to how “in phase,” or smooth, the image is, indicating properly aligned image fields. Peak signal corresponds maximizing the center, low-frequency bands of the image’s Fourier spectrum, corresponding to an image free from the high-frequency artifacts introduced from misaligned image fields. This is inversely related to classic autofocus contrast detection schemes, in which maximum contrast occurs at the focal plane [27], [28].

Fig. 5.

Two-dimensional phase sweep using images from a random location in fixed and cleared mouse brain tissue with DRAQ5 and LICOR785 staining. (a-c) Horizontal and (d-f) vertical phase sweeps corresponding to minimum, median, and maximum signal values, respectively. (g) Full plot of horizontal (blue) and vertical (red) phase sweeps, with image locations labeled. The initial x-axis phase value is held constant during the y-axis sweep, and vice versa. Images were taken at 10 Hz and tested every 0.1°, totaling 22 images in all. Scale bars indicate 100 μm.

After sampling the FOV to obtain an image vector, we (1) map the vector into an image according to Equations 1 and 2; (2) vertically interpolate the image according to Equation 7; (3) normalize the image to within [0,1]; (4) Fourier transform the image, shifting the DC term to center; and (5) sum the center portion (12.0% of each dimension by length, or 1.44% by area) of the absolute value of the image’s spectrum to obtain an output signal. The dimensions of this center portion correlate with sampling most of the center lobe of the Fourier transform, including the DC term, but not extending into higher frequencies. Fortunately, this dimension does not need to be precisely calibrated, as the artifacts introduced from misaligned fields are higher in frequency than our optical resolution. This will always be the case for Nyquist sampling or better. The example dimension is a result of roughly minimizing computation while sampling enough pixels to provide strong signal.

Figure 5 shows the resulting signals generated from Algorithm 1, along with representative images for each axis’ phase lag search. The phase sweep is performed separately for each axis. Algorithm 1 shows inputs corresponding to an x-axis sweep, holding y-axis phase lag ${\varphi }_y$ constant (Figure 5, a-c and g, blue). A y-axis sweep would then naturally follow, where ${\varphi }_x$ would be held constant (Figure 5, d-f and g, red).

Our method for computationally reconstructing images is purely software-based in that we reconstruct images using test phase lag values and display results in real-time (with the first few seconds of images being shown out of phase), however driving phase is kept constant to keep the mirror operating smoothly. This is one of the key differences between a hardware- and software-based phase control solutions: the quick feedback from circuitry enables the active control of driving phase, forcing the scanning beam along a predefined path. Because a method like this one calculates phase lag through processing image information, it takes orders of magnitude longer, and thus cannot actively control driving phase. Regardless, both methods are capable of providing images “in phase” in real-time.

This process is completed in ~360 ms per image, the majority of which is spent mapping our sampled data vectors into image form. Unfortunately this step cannot be replaced with less computationally expensive means (by, for example, simply shifting image pixels), since sinusoidal mapping dictates that integration is a function of position in the image. Processing the 22 test images in the example shown in Figure 5 takes ~7.5 s.

Before beginning the 2D phase lag search, initial values are first roughly set with the GUI using the real-time image display as feedback. Once within an acceptable range (±1° – 10°, depending on sample), the software begins a coarse phase sweep, in which image quality is tested over a wide range of phase values using coarse increments. After the initial sweep, the software updates the phase lag with an initial solution, then decreases the scanning range by a factor of ten and adjusts the increments such that the total sweep time remains constant.

Algorithm 1:

Phase Control Algorithm

This process is repeated until the phase lag is determined to within 0.001° (0.1% of 1°). After testing a set of phase values, the software normalizes the responses (from step five, above) to within [0,1] and picks phase lag value that corresponds to maximum signal.

To test the algorithm’s sensitivity to parameters such as SNR and optical resolution, we varied noise, sample rate, and Gaussian filtering on tissue images undergoing simulated phase testing like that shown in Figure 5. In these tests, we define sensitivity as the range between the maximum and minimum observed signal values, corresponding with the best and worst aligned images of the test, respectively. Error is defined as the root-mean-square deviation (RMSD) between each plot’s normalized shape (Equation 9). As sample rate drops from 10⁷ to 10⁶, below Nyquist, sensitivity remains relatively constant, but error rises exponentially. To test sensitivity to SNR, we added zero-mean, Gaussian white noise. As noise variance rises to 0.01, sensitivity and accuracy both drop linearly. Together, these results confirm that skipping small numbers of samples during the 2D mapping step can increase algorithm speed with limited effect on performance. To test the algorithm’s sensitivity to optical resolution, we applied Gaussian filters with sigma values ranging from 1,000 (close to original) to 1 (no discernible features). For σ > 20, decreasing sigma has virtually no affect on sensitivity and accuracy. For σ < 20, sensitivity and accuracy begin to drop exponentially, with the algorithm losing effectiveness below σ = 10. These results suggest that low-pass filtering images may be an effective step toward adapting the algorithm for use with images with low SNR.

We have found the Fourier-based approach used here to be more robust than typical edge detection schemes (such as with Sobel or Canny operators) mainly due to noise considerations. Weak fluorescent staining and deep tissue imaging are common issues with fluorescence that cause significant drops in SNR. However, we have a few additional sources of phase lag for each axis (Figures 6d,e). Before the test started, we manually adjusted phase to within ±1° for each axis, set the final precision level to 0.1% of a degree, and set frame rate to 2 Hz. Each time the phase sweep completes, in this noise unique to our setup that cause variation in SNR within each image. First, due to the microscope’s optical design, our focal plane is parabolic, meaning imaging depth decreases with MEMS mirror deflection, and maximum imaging depth occurs in the center of the image. Second, the excitation and collection beams in our microscope are slightly misaligned; a problem that is exacerbated when the mirror nears maximum deflection.

Fig. 6.

Real-time phase control during one continuous hour of imaging. (a) Sample image of 15 μm diameter fluorescent beads. (b) Percent absolute difference between frames at times t = 0 min and t = 60 min, color-coded and capped at 50% to enhance contrast. Insets in (a) and (b) show the same magnified region. (c) RMSD values (in units of pixel intensity normalized to [0, 1]) for each frame compared to the first image (blue) and for consecutive image pairs (red) over time. (d) X-axis phase lag over time. (e) Y-axis phase lag over time. Scale bars indicate 100 μm.

C. Maintaining Phase Control

To test phase control over a long period of time, we continuously imaged a phantom of 15 μm diameter fluorescent microbeads suspended in polydimethylsiloxane (PDMS) for 60 minutes while saving each image (Figure 6a) and recording phase lag for each axis (Figures 6d,e). Before the test started, we manually adjusted phase to within ±1° for each axis, set the final precision level to 0.1% of a degree, and set frame rate to 2 Hz. Each time the phase sweep completes, in this case every 16 frames, or 8 s, phase lag gets updated, and a new sweep is initiated with the previous test’s phase solutions as starting points.

To evaluate phase control performance, each image is compared to the first image of the test (Figure 6c, blue) as well as to the previous frame (Figure 6c, red) by calculating the RMSD between said pairs of reconstructed images, defined as

$$RMSD=\sqrt{mean{\left(I{m}_1-I{m}_2\right)}^2},$$ (9)

where each image, Im, is converted to double precision and normalized to within [0,1]. When compared to the previous frame, RMSD consistently hovers just below 0.01, with an average value of 8.9 × 10⁻³, indicating that each pairwise image changes by less than 1% on average. When compared to the first frame, RMSD grows linearly with time, suggesting either consistent accumulation of errors in phase control or gradual phantom shift. Note that this linear RMSD ramp does not correlate with the fluctuations observed in tracked phase (Figures 6d,e). Visually assessing the differences between the first and last images in the test (Figure 6b) confirms that this ramp is due to phantom shift. Recall from Section IV-A that inaccurate phase control results in horizontal and/or vertical translation of even and odd fields of the image. This kind of error would present itself as doubling (or quadrupling) of the beads along either axis in the grayscale image data, resulting in pairs of mirrored crescent-shaped patterns in Figure 6b. Instead, we observe two major features in the absolute difference image: roughly symmetric rings (Figure 6b, inset), suggesting axial translation of the sample, and crescent-like shapes, without mirroring, suggesting lateral translation. Either of these features could be from contraction or relaxation of the PDMS, or from a shift in imaging depth from the piezoelectric linear actuator. Other possible sources of FOV instability include evaporation of the index matching gel between the microscope’s objective and the sample, fluctuations in room temperature, and inadequate vibration control. Although the microscope’s scanner exhibited phase drift of varying severity for the two axes (maximum observed severity was 0.06 deg/min, or 10⁻³ deg/s; Figures 6d,e), image alignment was consistent over time (Figure 6c, red).

V. Image Mosaicing

With update rates increased and phase lag controlled, mosaicing with Lissajous scanned images is now possible. Our registration method uses normalized cross-correlation [29], a form of template matching in which one finds areas of an image that matches a template image kernel. In this approach, a response map R is calculated from scanning a template T over an image I, with the brightest locations indicating the best matches:

$$R(x,y)=\frac{{\sum }_{x^{\prime },y^{\prime }}\left(T\left(x^{\prime },y^{\prime }\right)\cdot I\left(x+x^{\prime },y+y^{\prime }\right)\right)}{\sqrt{{\sum }_{x^{\prime },y^{\prime }}T{\left(x^{\prime },y^{\prime }\right)}^2\cdot {\sum }_{x^{\prime },y^{\prime }}I{\left(x+x^{\prime },y+y^{\prime }\right)}^2}}.$$ (10)

To form our template, we crop the second image of each pair down to size w × w pixels, defining the search window, and amount of overlap, between sequential frames. This assumption constrains panning speed (how quickly the user can slide the microscope across the tissue surface) according to the linear relationship

$$v_{max}=w{f}_u,$$ (11)

but keeps calculations to a minimum. For example, if we set our search window to ±30 pixels (±15 μm) per axis and image at 30 Hz, we can pan the microscope at speeds of up to 450 μm/s. Operating the microscope by hand under this constraint is challenging, but possible, whereas handheld operation at single digit update rates is impractical. Thus, at slower update rates, we depend on linear translation stages to control panning. Stages also facilitate maintaining optical contact between the microscope’s objective and the sample, enabling larger mosaics.

Cross-correlation is simple, computationally efficient when done in Fourier space, and performs well enough to avoid needing secondary fine-tuning. Methods like pyramidal optical flow, SIFT, and SURF were attempted early in development, but their dependence on high SNR, resolution, and number of observable features limited them to update rates below ∼15 Hz. Although frequency domain-based approaches can also be sensitive to noise, we found this phase correlation method to be robust to both noise and occlusion from, for example, small pieces of dust or tissue. Through experimentation, we found increased repeatability through normalizing each pair of images prior to registration.

Whether the microscope is stage-mounted or handheld, we make no assumptions about direction of travel. All registration is applied to a single channel of image data (the channel imaging DRAQ5-stained cell nuclei). We experimented with using both channels of data, which under different circumstances (e.g., a different modality with better SNR) is likely be a viable improvement. In our tests, the LIQOR785 channel didn’t show enough contrast for consistent template matching for all frames.

Image blending is handled with simple averaging, such that SNR is improved in proportion to the square root of the number of samples (in overlapping regions). Once imaging has ended, the mosaic goes through two post-processing steps to enhance aesthetics: trimming and contrast enhancing. The trimming step removes empty borders around the mosaic that are automatically padded on when the FOV approaches any of the mosaic’s borders. To enhance contrast, we apply contrast-limited adaptive histogram equalization (CLAHE) [30], assuming a linear distribution of image intensity values. This integration. The difference between the static mosaics of and other closely related approaches have been applied to mosaics in micro-angiography [31] and astronomy [32], and works well here to control lighting uniformity without greatly oversaturating cell nuclei.

Regardless of update rate, the algorithms used for image interpolation and integration remain constant. However, template matching high update rate, low-resolution images requires one critical tweak: low-pass filtering. In preparation for template matching, each pair of images is temporarily smoothed using a median filter with a kernel size that increases with update rate (e.g., 30 Hz images require a kernel size of 11 pixels, or 5.5 μm). Despite varying initial image resolution, we observed no difference in registration precision or mosaic resolution with update rates of up to 30 Hz.

Mosaicing is commonly employed to accumulate large imaging areas over time from smaller individual FOVs, to provide image stabilization, and to boost SNR in the overlapped regions of consecutive frames. However, using mosaicing to recover the resolution lost through subsampling images is novel. Recall from Sections II-B and III that although each individual frame can be greatly degraded, integrating additional high-resolution information along with interpolated information can result in image quality comparable to simple Figures 4 and those shown in Figures 7, 8, and 9 is lateral translation between the microscope and the sample (and longer total imaging times). To prevent motion blurring, we have increased the update rate. To determine distance and direction traveled, we perform template matching and overlay data accordingly. Thus, combining fast updating and mosaicing enables us to remove the assumption that the microscope is static while integrating, without sacrificing resolution. This approach is applicable for all manners of imaging with miniaturized microscopes, whether static, intentionally panned across tissue, or for image stabilization.

Fig. 7.

Multispectral, real-time 5 Hz mosaic of fixed and cleared mouse brain tissue stained with IR783 (red) and DRAQ5 (green). Top: Single 5 Hz frames before (a) and after (b) interpolation. Bottom (c): Mosaic comprises 400 consecutive images taken over 80 s. White rectangle shows same FOV as in (a) and (b). Sample translated via manual stage controller. Single frame FOV is 370 × 650 μm. Total Mosaic FOV is 1.37 × 1.43 mm. Scale bars indicate 100 μm (top) and 500 μm (bottom).

Fig. 8.

Multispectral, real-time 10 Hz mosaic of fixed and cleared mouse brain tissue stained with IR783 (red) and DRAQ5 (green). Top: Single 10 Hz frames before (a) and after (b) interpolation. Bottom (c): Mosaic comprises 6,398 consecutive images taken over 10.7 min. White rectangle shows same FOV as in (a) and (b). Sample translated via manual stage controller. Single frame FOV is 370 × 650 μm. Total Mosaic FOV is 4.43 × 4.27 mm. Scale bars indicate 100 μm (top) and 1 mm (bottom).

Fig. 9.

Multispectral, post-processed 30 Hz mosaic of fixed and cleared mouse brain tissue stained with IR783 (red) and DRAQ5 (green). Left (a): Mosaic comprises 385 consecutive images taken over 12.8 s. White rectangle shows same FOV as in (b) and (c). Right: Single 30 Hz frames before (b) and after (c) interpolation, showing severe image degradation due to subsampling. Microscope translated in hand. Single frame FOV is 370 × 650 μm. Total mosaic FOV is 1.27 mm × 679 μm. Scale bars indicate 500 μm (left) and 100 μm (right).

Figures 7, 8, and 9 show multispectral mosaics of fixed, cleared mouse brain tissues imaged at 5 Hz, 10 Hz, and 30 Hz, respectively. Each tissue sample was stained with IR783 to show general tissue morphology (red) and DRAQ5 to counterstain cell nuclei (green). These figures represent the first mosaics comprised entirely of Lissajous scanned microscopy images. In Figures 7 and 8, the microscope was inverted and mounted to a static post. The tissue was mounted to a glass slide and taped down to a manual, linear translation stage to control panning. These figures were mosaiced in real-time. In Figure 9, the microscope was detached from its mount and held by hand, keeping the tissue static. Due to limited computational power, this figure was mosaiced in post processing. In contrast to imaging at slower update rates, manual stages are not necessary when imaging at video-rate. Although each individual image shows low resolution and SNR, integration during the mosaicing process mitigates both these issues, with no appreciable difference in resolution between mosaics with different update rates.

VI. Discussion and Conclusions

With the inclusion of this work, there are now two categories for real-time phase control during Lissajous imaging: hardware- and software-based phase control. The work demonstrated here is not intended to replace physical sensors, but to offer an alterative for certain applications. For tabletop setups or in situations where bandwidth and correction response time is critical, or when frequency control is also needed, physical sensors are ideal. Such methods can also control beam path deflection in real-time, ensuring scanning occurs over a predefined path. However, hardware-based methods add size, complexity, and cost, making them particularly difficult to implement in miniaturized microscopes designed for in vivo imaging. In comparison, software-based methods can enable phase control for any imaging system, but require real-time imaging as part of their feedback loop and time to calculate reconstructions from sampled data. Scanners with high Q-factors, or those exhibiting gyroscopic sensitivity, may be too unstable for phase control methods that take on the order of seconds to respond. Additionally, these methods require constant optical contact with the sample to ensure a steady stream of image information with which to work. Thus, our method is not applicable to non-imaging Lissajous scanning systems, such as video projectors.

Image processing techniques like the one presented here are sensitive to SNR, optical resolution, and sample rate. With that in mind, it is applicable to any modality with comparable image quality. For example, operating a confocal microscope in reflectance mode offers identical resolution and sample rate, but increases SNR greatly. Optical coherence tomography (OCT), which typically displays images in which SNR degrades with imaging depth, is comparable to our fluorescent image data, where portions of the image appear out of focus due to optical misalignment, and image quality degrades with penetration depth. Even relatively low-resolution images (e.g., from magnetic particle imaging) are applicable.

Alternatives to linear image interpolation were considered early in the project. One idea is to update the image prior to complete image sampling, replacing only those pixels sampled [7]. However, this method may not work well with the template matching or feature tracking processes needed for mosaicing, as there exists a mixture of new and old information in each image. In our approach, although we have a mixture of high-and low-resolution information in each image, each frame consists entirely of new information in a temporal sense, enabling accurate motion tracking. A second idea is to shift each image half a FOV at a time, sampling the same region once in the low-resolution center of the FOV and then again in the highly sampled edges or corners. This places a major constraint on how the microscope is panned across the FOV, requiring motorized movement between frames, and preventing software-based image stabilization. A third option is to amplitude modulate the driving signals, alternating between full and reduced FOVs in an attempt to fully sample the center region every n^th image. This method could help with the sampling issues inherent in Lissajous scanning, but at the costs of artificially limiting sampling bandwidth and complicating phase control. Finally, more advanced interpolation methods were considered, and likely would have been implemented if real-time mosaicing had not been the end goal. Choosing a simple interpolation method allowed us to direct more computational load toward mosaicing.

The spatial and temporal sampling advantages of Lissajous scanning, discussed in Section II, can also be thought of in terms of overall information sampling rates. Space bandwidth product (SPB) is a unitless metric defined as FOV × maximum spatial frequency. To compare the SBP of a raster- or Lissajous-scanned microscope (i.e., characterizing bandwidth independent of optics), we compare three major factors: (1) FOV, (2) rate of travel of the focal point, and (3) sampling efficiency. For fixed amplitude driving signals, driving both axes in resonance instead of just one results in a factor of up to 2–3 larger imaging area, depending on the Q-factor of the scanning mirror and assuming a lack of other limiting factors such as off-axis aberrations or vignetting.

To compare rates of travel, we use the average scanning speed for a raster-scan (approximated as a 1D equation),

$$v_{ave,raster}\approx 2A_{res}f,$$ (12)

and that of a Lissajous-scan,

$$v_{ave\text{,}Lissajous}\approx 2\sqrt{A_x^2{f}_x^2+A_y^2{f}_y^2}.$$ (13)

Switching from raster scanning (f_x = 2034 Hz, f_y = 5 Hz) to Lissajous scanning (f_x = 2034 Hz, f_y = 315 Hz) increases the average scanning speed by just 0.4%. However, as shown in Figure 2, raster scanning is unidirectional, meaning only one direction of the fast axis’ sinusoidal motion is utilized. Bi-directional raster scanning is possible, but less commonly implemented due to requiring phase control but not increasing FOV. Thus, Lissajous scanning gains a factor of approximately two in effective scanning speed over raster scanning as a direct tradeoff for needing phase control.

As discussed in Section II, Lissajous sampling efficiency is a function of update rate. In fact, sampling efficiency relies so heavily upon update rate that these accumulations in SBP can be nearly negated for slow update rates, or substantially increased for fast updating. For example, let us compare a microscope in either raster (FOV = 150μm × 650μm, 100% sampling efficiency) or Lissajous (FOV = 370μm × 650μm, 64% sampling efficiency) scanning modes and update rates of 5 Hz for both. Since optical resolution, one axis’ deflection angle, and update rate cancel out, the increase in SPB is then 0.64 × 370 = 236.8 for Lissajous vs. 150 for raster, a difference of just 1.58X. When we increase the Lissajous update rate to 30 Hz, sampling efficiency drops to 45%, and the Lissajous vs. raster increase in SPB is then (0.45 × 370 × 30)/(150 × 5) = 6.66X.

Our solutions for phase control, boosting update rate, and mosaicing are applicable to most Lissajous scanning imaging modalities and setups. Modern optical microscopy techniques such as confocal [7], [33], [34], [35], [36], multiphoton [8], [37], [38], [39], [40], OCT [41], [42], [43], [44], [45], [46], [47], and resonant fiber scanning [37], [38], [39], [40], [43], [44], [45], [46], [47], [48] have all had recent success with Lissajous scanning, and are immediate choices in which to effectively implement the work shown here due to their similarity with our own microscopy setup. Other applicable modalities include high-speed atomic force microscopy (AFM) [20], [49], [50], [51], [52], magnetic particle imaging (MPI) [53], [54], [55], [56], and light detection and ranging (LIDAR) [57], [58], [59], [60]. Many of these systems are being employed on moving objects such as cars, drones, and other robots, where mosaicing is commonly used in autonomous applications, and and replacing phase sensing hardware with software would lighten the load.

This work could also be applied to raster scanning in two ways. First, our phase control algorithm (reduced to 1D) could be used for aligning even and odd fields of a raster scan, converting traditional raster scanning to bi-directional scanning, increasing frame rate by a factor of two with virtually no alterations to the experimental setup. Second, frame rate could be further improved by increasing the slow axis scanning speed past that dictated by Nyquist, resulting in images subsampled in the vertical direction, much like in this work. Through subsampling, vertical interpolation, and mosaicing to recover lost resolution, frame rate could be increased by at least another factor of two. Combined, these two techniques could improve the effective frame rate of a raster scanned imaging device by a factor of at least four, potentially boosting single digit frame rates to video-rate.