Calibration-free sinusoidal rectification and uniform retinal irradiance in scanning light ophthalmoscopy

Abstract

Sinusoidal rectification (i.e. desinusoiding) is necessary for scanning imaging systems and is typically achieved by calculating a rectification transform from a calibration image such as a regular grid. This approach is susceptible to error due to electronic or mechanical instability that can alter the phase of the imaging window with respect to the calibration transform. Here we show a calibration-free rectification method implemented from live video of a scanning light ophthalmoscope (SLO) with or without adaptive optics (AO). This approach, which capitalizes on positional differences in the images obtained in the forward and backward scan directions, dynamically keeps the imaging window in phase with the motion of the sinusoidal resonant scanner, preventing errors from signal drift over time. A benefit of this approach is that it allows the light power across the field-of-view (FOV) to be modulated inversely to achieve uniform irradiance on the retina, a feature desirable for functional imaging methods and light safety in scanning light ophthalmoscopes.

Images from SLOs that use resonant scanners contain sinusoidal distortion [1, 2], usually requiring offline or real-time rectification before further processing such as registration [3, 4, 5] is applied.

In offline rectification, a calibration image is recorded from a grid, a grating, or other target. This calibration image typically serves two major functions. First, it defines two optimal imaging windows in the forward and backward portions of the scanning interval, avoiding the turn-around intervals at the margins of the scan. Once the imaging window is determined, a conversion matrix or a lookup table is generated based on numerical interpolation that is then used to rectify the sinusoidally distorted images in space linearly. The second function of the calibration image is to achieve uniform pixel density across the rectified image and square pixels.

The limitation of the offline approach is that it only provides a static correction from one point in time. Calibration images are recorded either before or after an imaging session. The interval between calibration and retinal imaging can span from tens of minutes to several hours or even longer, so the conversion matrix from the last calibration image will be used to rectify all retinal images until the conversion matrix is renewed from a new calibration image. This is a reliable approach as long as the imaging window does not change over time. In reality however, drift of the imaging window over time is common in SLO systems. This can occur for several reasons. For example, the frequency of the resonant scanner may decrease over time when it is not able to dissipate heat quickly enough or if the mass of the attached mirror is too great. This issue causes a small change in frequency, but an appreciable change in imaging window position over timescales of a few to tens of minutes. Fig 1 illustrates image displacement from acquisition window drift over time in an AOSLO system. In this instrument, the drift amount is ~3.8 arcmin over two hours (~4.2% of the size of a typical 1.5°×1.5° AOSLO image). This drift causes two problems: 1) the imaging FOV of the forward and backward scans are no longer in register and 2) the calibration image no longer captures the true image distortion.

Fig. 1.

Drift of image over time due to shift of imaging window in an AOSLO. Data was calculated from pixel-level cross correlation; drift magnitude in arcmin was converted from the number of whole pixels.

Fig 2 compares sinusoidal rectification from a correct inversion matrix to an incorrect one acquired at a different time point. The correctly rectified image has uniform spacing (middle image) accurately representing the image of the Ronchi ruling. The incorrect conversion matrix distorts this uniformity and spacing varies across the image (bottom).

Fig. 2.

An AOSLO image of a uniformly spaced Ronchi ruling before rectification (top) and after rectification with either the correct (middle) or incorrect (bottom) conversion matrix. The incorrect conversion matrix (obtained after a 4.2% shift of the imaging window) introduces a non-linear distortion.

To avoid imaging window drift and to eliminate the need for offline rectification, a hardware solution has been reported in confocal microscopy where variable pixel clocks are generated in real time from the scanning system [6]. The purpose of variable pixel clocks is to achieve linear spatial sampling by applying nonlinear temporal pixel clocks on the digitizer. However, this approach has several disadvantages: 1) it requires additional optics making the optical system more complex, 2) it is not flexible when the temporal and/or spatial sampling intervals require frequent changes, which is common in scientific research, and 3) it does not solve the problem of non-uniform irradiance at the edges of the scan field as the light source dwells longer on these regions due to the slowing of the scan before it stops and changes direction. This non-uniform light irradiance on the retina is undesirable in many physiological studies and for functional imaging and is particularly important for minimizing light exposure for safety reasons when imaging human eyes.

We developed a new approach to dynamically detect and adjust temporal/spatial position of the imaging window during real-time data acquisition to eliminate image acquisition window drift without using additional hardware. At the same time, we use this information to modulate light intensity to achieve uniform irradiance across the FOV.

In a conventional SLO system, the timing of a full cycle of the resonant scanner (T_H) is illustrated in the five intervals in Fig 3: the front porch (T₁), the forward scan imaging window (T_f), the turnaround interval between the forward and backward scans (T₂), the backward scan imaging window (T_b), and the back porch (T₃). In some digitizers, T_f, T₂, and T_b are merged into one imaging window and the data in T₂ is then cropped out. Resonant scanner electronics output a TTL clock signal. Ideally, the rising (or falling) edge of the TTL pulse delineates the start of the forward (or backward) scan. The TTL clock is commonly used as the H-sync of the digitizer where T₁, T_f, T₂, T_b, and T₃ can be conveniently represented in the number of pixels. The summation of them is

$${\mathrm{T}}_1+({\mathrm{T}}_{\mathrm{f}}+{\mathrm{T}}_2+{\mathrm{T}}_{\mathrm{b}})+{\mathrm{T}}_3={\mathrm{T}}_{\mathrm{H}}$$ (1)

where the duration of T_H, T_f, T_b, and T₂ remains constant during imaging. It should be noted that T_H is not strictly constant. In one of our AOSLO systems, we measured a T_H increase of 0.059% over the course of two hours. However, for our purposes, assuming a constant T_H is reasonable because it only causes a tiny change in the AOSLO field of view. The phase shift in the imaging window results from the continuous accumulation of small changes in frequency.

Fig. 3.

Fixed timing intervals cause imaging window drift. Top panel illustrates the timing intervals of a full cycle of the resonant scanner. T₁: front porch, T_f: forward scan imaging window, T₂: turnaround interval between forward and backward scans, T_b: backward scan imaging window, T₃: back porch. The TTL pulse phase with respect to the scanner motion drifts over time (e.g. from green to red trace positions) causing the imaging windows to drift (e.g. from green to red dashed outlines).

In many scanning systems, the TTL clock does not always synchronize and stay in phase with the physical position of the resonant scanner due to electronic and/or mechanical instability. It should be noted that some resonant scanner electronics do not output a TTL clock but rather a sinusoidal motion signal and this problem persists in these systems as well. As a consequence, when T₁, T_f, T₂, T_b, and T₃ are fixed, a gradual shift of the TTL clock against the physical position of the resonant scanner occurs, causing a gradual drift of the sampled images. Ideally, images acquired during the forward (T_f) and backward (T_b) scan intervals should be aligned symmetrically (i.e. when the TTL clock synchronizes with scanner motion); this is illustrated by the green dashed outlines in Fig. 3. They become asymmetric when the TTL clock shifts its phase against the physical position of the scanner; this is illustrated by the red dashed outlines in Fig. 3.

Image drift in the direction of slow scanning which is also caused by the drifting imaging window, can be reasonably treated as translational eye motion because the slow scanning is linear in space (when ignoring its retrace period). However, the gradually drifting images in the fast direction cause rectification to introduce additional distortion by causing the conversion matrix to become invalid, as illustrated in Fig 2. It should be noted that this distortion is usually not visible to the eye in the retinal images but is actually present and can have detrimental implications for image registration and image stitching (i.e. montaging).

To solve this problem, durations of T₁, and position of T_f, T₂ and T_b must be adjusted dynamically to compensate for the unknown phase shift between the physical position of the resonant scanner and the rising/falling edge of the TTL clock. In our approach, we measure the imaging window drift by calculating the positional difference between the images obtained in the forward and backward scan directions (e.g. red or green dashed outlines in Fig. 3). This measurement, obtained through cross correlation of the forward and backward scan images, is used to dynamically adjust T₁ to keep the imaging windows at the same position, and in phase with the motion of the sinusoidal resonant scanner. Durations of T_f, T₂, and T_b are constants, and T₃ is updated automatically with the constraint from Equation (1) (i.e. the sum of all intervals does not change). If positional differences in the images from T_f and T_b are defined as x(t) and y(t) which change over time, T₁ will be updated in the form of

$${\mathrm{T}}_1\left(t\right)={\mathrm{T}}_1(t-1)-x\left(t\right)∕2$$ (2)

The initial value of T₁ is setup manually only once, during system calibration, unless imaging parameters such as the frequency of the pixel clock are updated.

Since our approach is used for living eyes that blink periodically, we use the mean pixel value and standard deviation from the images to determine whether a blink (or no data at all) occurred. Cross-correlation is turned off during blinks or periods where no data is acquired, but otherwise is activated every 3-5 seconds. This temporal update is sufficient due to the slow imaging window drift (Fig 1). We crop two strips, e.g., 512×128 pixels from the two full images, e.g., in 808×576 pixels (before rectification) because the relative motion between the two strips in the slow scan direction is no more than 3 pixels from a 1.5°×1.5° AOSLO image and much smaller with larger FOV, even if the worst case of a microsaccade with a speed of 120°/sec [7].

After the dynamic adjustment of T₁, the centers of T_f and T_b will be located at the peak velocity of the resonant scanner, with error +/− 0.5 pixels expected from the pixel-level cross correlation. It should be noted that it is possible to improve this and adjust the sampling window with sub-pixel clock accuracy, by using sub-pixel cross correlation combined with oversampling from the digitizer. We use conventional interpolation to generate a conversion matrix for sinusoidal rectification.

To achieve uniform irradiance on the retina, we start with motion of the resonant scanner which can be defined as,

$$x\left(t\right)=A-A\cdot \cos \left(\omega t\right),$$ (3)

where x(t) is position of the scanner at time t, ω is its angular velocity, 2A is its scanning window (or FOV). From Equation (3) it is not difficult to derive light irradiance on unit area of the retina,

$$I\left(x\right)\propto P_0\cdot \frac{dt}{dx}=P_0\cdot \frac{1}{A\omega }\cdot \frac{1}{\sqrt{1-{\left(1-\frac{x}{A}\right)}^2}}$$ (4)

where P₀ is power of the light source. It can be seen that I(x) is the lowest at the center of scanning FOV where x = A, but is significantly higher at the edges of the FOV. To achieve constant I(x) on the retina regardless scanner position x, the light source needs to be modulated in the following form,

$$I\left(x\right)\propto M\left(x\right)\cdot P_0\cdot \frac{dt}{dx}=M\left(x\right)\cdot P_0\cdot \frac{1}{A\omega }\cdot \frac{1}{\sqrt{1-{\left(1-\frac{x}{A}\right)}^2}}\equiv I_0$$ (5)

where M(x) is the modulation formula, giving

$$M\left(x\right)=\sqrt{1-{\left(1-\frac{x}{A}\right)}^2}=\sin \left(\omega t\right)=M\left(t\right)$$ (6)

In Equation (6), M(x) is shown to be equal to M(t), a function of time t instead of location x, because time t is associated with the pixel clock of the digitizer, which is directly accessible from the device. Therefore, for example, if 80% of the scanning window is used for imaging, from Equation (6) it is not difficult to get the modulation curve in Fig 4. Due to the fact that the resonant scanner does not travel strictly sinusoidally close to the two edges of the scanning FOV, light is usually turned off in the two shaded areas in Fig 4 to avoid unnecessary retinal exposure. Modulation of the red curve in Fig 4 can be achieved conveniently by implementing these two steps simultaneously: 1) sending a TTL signal to turn off the light source outside the imaging window, and 2) employing a fast opto-electronic device (e.g., an acousto-opto modulator (AOM)) to modulate the intensity of the light source inside the imaging window.

Fig. 4.

Modulation of light power across scanning window (dashed curve) and imaging window (solid curve) to achieve uniform retinal irradiance. Light source is turned off at two edges of scanning FOV as shown in the shaded regions.

To validate our approach, we measured light power across the imaging window of an AOSLO by dividing a ~2.45°×3.4° FOV (fast scan × slow scan) into 23 equal-space strips, without overlap. Each strip was ~0.11° in the fast direction and the full 3.4° in the slow direction. This was accomplished by using an AOM to rapidly turn off the light at the desired positions. The power meter was placed at the pupil plane of the AOSLO, and its reading was integrated over a 10-second interval.

System latency and nonlinearity curves are used to modulate the AOM. The system latency comes largely from two sources: 1) digital-to-analog latency, and 2) AOM latency. These latencies can be calibrated and they are constant as long as the hardware does not change. The imaging window can have any size equal to or smaller than the scanning window. This is particularly important for real-time stabilized stimulation where the stimulus size is usually much smaller than the image size, but the location of the stimulus will change dynamically to compensate for eye motion, hence the nonlinear curve for power modulation also has to be updated dynamically. Stabilized stimulation and its applications have been reported earlier [8, 9, 10, 11], and the details are beyond the scope of this article. Any nonlinearity of the AOM as well as transverse chromatic aberration (TCA) [12] must be considered in the engineering implementation. An important advantage of this light modulation is that it improves estimation of light exposure. Without modulation, calculation of light safety must consider the power at the areas of highest light exposure (i.e. the two edges of the scanned area).

The improved imaging this method provides is illustrated in the images shown in Fig 5: a line-interleaved image from a wide-FOV SLO, where each odd line comes from the forward scan and each even line comes from backward scan, after dynamic adjustment of the imaging window.

Fig. 5.

A line interleaved image from a wide-FOV SLO after dynamic adjustment of imaging window.

Fig 6 illustrates an example of imaging window modulation with correction of system latency and AOM nonlinearity. The blue curve is the theoretical retinal irradiance from Equation (4), the red squares are measured light power without any modulation which reflects light distribution on most of the existing AOSLO systems, and the black triangles are measured light power with modulation from Fig 4 plus corrections of system latencies and the AOM nonlinearity. RMS error of the black triangles in Fig 6 (fluctuation of measured power) with power modulation is ~1.43%. The averaged energy level across the imaging window after irradiance correction is decreased ~19.5%.

Fig 6.

Retinal irradiance in an AOSLO before and after light modulation. Measured light intensity before modulation (red squares) is close to the theoretical expectation (blue curve) and is uniform after modulation (black triangles). Light is turned off in the shaded gray areas.

In summary, we demonstrated a calibration-free approach to accurately rectify SLO images that dynamically compensates for image drift due to mechanical and/or electronic instability. It facilitates image interlacing and efficient use of data acquired from both scan directions (e.g. locking the same FOV in each), and it eliminates the need for any manual calibration steps. This approach is beneficial for all types of scanning imaging systems and is particularly beneficial for imaging systems that incorporate real-time image based optical stabilization and/or registration [13, 14]. An added benefit of this approach is that it allows for precise light source modulation for uniform retinal irradiance, allowing for precise retinal stimulation for functional imaging [15] and improved safety by reducing light exposure to the retina.