Increasing the field-of-view of dynamic cardiac OCT via post-acquisition mosaicing without affecting frame-rate or spatial resolution

Abstract

Optical coherence tomography (OCT) allows imaging dynamic structures and fluid flow within scattering tissue, such as the beating heart and blood flow in murine embryos. For any given system, the frame rate, spatial resolution, field-of-view (FOV), and signal-to-noise ratio (SNR) are interconnected: favoring one aspect limits at least one of the others due to optical, instrumentation, and software constraints. Here we describe a spatio-temporal mosaicing technique to reconstruct high-speed, high spatial-resolution, and large-field-of-view OCT sequences. The technique is applicable to imaging any cyclically moving structure and operates on multiple, spatially overlapping tiled image sequences (each sequence acquired sequentially at a given spatial location) and effectively decouples the (rigid) spatial alignment and (non-rigid) temporal registration problems. Using this approach we reconstructed full-frame OCT sequences of the beating embryonic rat heart (11.5 days post coitus) and compared it to direct imaging on the same system, demonstrating a six-fold improvement of the frame rate without compromising spatial resolution, FOV, or SNR.

1. Introduction

Optical coherence tomography (OCT) is increasingly being adopted as an accurate imaging method to study embryonic cardiovascular anatomy and dynamics. It allows for non-destructive imaging through several millimeters of biological tissue with single cell resolution. OCT has been used for a wide array of biomedical applications, including structural imaging of whole mammalian and avian embryos [1,2], flow measurements [3,4], and 4D imaging of mammalian and avian embryonic hearts [5–7]. High frame rate, signal-to-noise ratio (SNR), and spatial resolution as well as the ability to image over a large field of view are all equally important factors for accurate visualization and analysis.

Improvements in light sources, scanner efficiency, data transmission protocols, and storage capacity have collectively produced improvements in the acquisition frame rate for OCT [8, 9]. The time over which the signal is gathered to produce one image limits the spatial resolution when imaging dynamic samples since sample motion introduces blurring if the image capture time is too long [10]. Increasing the frame rate can be achieved through faster scanning but decreasing the dwell time also reduces the SNR, as fewer photons contribute to the signal. For cardiovascular systems, which exhibit velocities of the order of millimeters per second, framerates of 100 frames per second or more are required to achieve a spatial resolution of 10 μm and below. Thus, there is a need for advanced methods that improve the frame rate while preserving the spatial resolution.

For optical and electron microscopy of static or nearly static samples, tiled imaging, followed by mosaic reconstruction, has become a central tool to extend the limited field of view of high-numerical aperture microscope objectives. Several methods have been proposed to automatically or semi-automatically spatially register (align) individual microscopy tiles to form an image mosaic [11–14] extending methods initially developed for panoramic photography [15, 16]. These technique, however, produce static images and are therefore not directly applicable in our case.

Recently, we demonstrated that dynamic 3D images of the embryonic zebrafish heart can be reconstructed by post-acquisition synchronization of nongated slice sequences acquired on a fast confocal microscope [17, 18]. Building upon our ability to accurately register image sequences in time, we here present a combined (rigid) spatial and (non-rigid) temporal registration method that allows us to extend the scope of static mosaicing methods to improve the field of view and resolution of dynamic 2D OCT by combining tiled image sequences of cyclically moving structures in the mammalian embryonic heart.

2. Method

Our method operates as follows. We assume the object to be imaged has a local scattering intensity that is periodic, that is

$$I(x,y,t)\hspace{0.17em}\approx \hspace{0.17em}I(x,y,t\hspace{0.17em}+\hspace{0.17em}T)\hspace{0.17em}\forall t\hspace{0.17em}\in \hspace{0.17em}ℝ,$$ (1)

where T is the period of the heart beat (assuming heart imaging). We acquire N image sequences (see Fig. 1(a) for an example with N = 2) that are spatially tiled such that

$$I_n(x,y,t)\hspace{0.17em}=\hspace{0.17em}I(x_n\hspace{0.17em}+x,y,t_n+t),\hspace{0.17em}0\hspace{0.17em}\le \hspace{0.17em}n\hspace{0.17em}<\hspace{0.17em}N,$$ (2)

for 0 ≤ x < L_x, 0 ≤ y < L_y, and 0 ≤ t < L_t, where L_x and L_y are the width and height of the tiles, respectively, L_t is the duration of each sequence and where x_n is the spatial offset in the x-direction of the nth tile, and t_n is the time at which recording of the nth sequence started. The duration of the sequence, L_t, must be at least two periods long, so that one full heart beat period, starting at any arbitrary time in the cardiac cycle, is included in the sequence.

Fig. 1.

Tiled acquisition and automatic mosaicing procedure for dynamic images. (a) Overlapping cardiac image series, are acquired sequentially. (b) Each sequence is collapsed to a static image (temporal maximum). (c) Static images are spatially aligned. (d) Image sequences are spatially transformed and padded with zeros. (e) Spatially aligned sequences are temporally synchronized. (f) Synchronized image sequences are blended. (Color online).

We assume that the spatial and temporal offsets, x_n and t_n, respectively, are unknown. Our task is to estimate them, which is an image registration (and synchronization) problem (we assume, here, that tiles are aligned in the y direction, but this is not a fundamental requirement of our approach, which could be generalized to include registration in both the x and y directions). The periodicity assumption allows us to decouple the spatial and temporal registration problems as follows. We first collapse each sequence along the temporal direction (Fig. 1(b)) to obtain a spatial signature of the tile that is independent of time

$${\overline{I}}_n(x,y)\hspace{0.17em}=\hspace{0.17em}\underset{0\le t<L_t}{\text{max}}\hspace{0.17em}{I}_n(x,y,t).$$ (3)

We next pair-wise align the N static tiles Ī_n, that is, we estimate the tiling-offsets x_n as follows

$${\tilde{x}}_n\hspace{0.17em}=\hspace{0.17em}{\tilde{x}}_{n-1}\hspace{0.17em}+\hspace{0.17em}\text{arg}\hspace{0.17em}\underset{0\le x<L_x}{\text{min}}{\int }_0^{L_y}\frac{1}{L_x-x}{\int }_x^{L_x}{\left|{\overline{I}}_n(x\prime -x,y)\hspace{0.17em}-\hspace{0.17em}{\overline{I}}_{n-1}(x\prime ,y)\right|}^2\hspace{0.17em}\text{d}x\prime \text{d}y,$$ (4)

with x₀ = 0 (Fig. 1(c)). In words, we find the horizontal position for which the mean-squared difference between the previous and current tiles’ grayscale values (calculated over the region of overlap of the two tiles) is minimal. In practice, we use a rigid-body, pyramid-based, registration implementation proposed by Thévenaz et al. [19], which we constrain to register tiles along the horizontal direction.

We next define zero-padded image sequences

$$I_n^{\prime }(x,y,t)\hspace{0.17em}=\hspace{0.17em}\{\begin{array}{ll}0\hfill & 0\hspace{0.17em}\le \hspace{0.17em}x\hspace{0.17em}<{\tilde{x}}_n\hfill \\ I_n(x-{\tilde{x}}_n,y,t)\hfill & {\tilde{x}}_n\hspace{0.17em}\le \hspace{0.17em}x\hspace{0.17em}<\hspace{0.17em}{\tilde{x}}_n\hspace{0.17em}+\hspace{0.17em}{L}_x\hfill \\ 0\hfill & {\tilde{x}}_n\hspace{0.17em}+\hspace{0.17em}{L}_x\hspace{0.17em}\le \hspace{0.17em}x\hspace{0.17em}<\hspace{0.17em}{\tilde{x}}_N\hspace{0.17em}+\hspace{0.17em}{L}_x,\hfill \end{array}$$ (5)

in which the original sequences have been placed at their estimated spatial offset (see Fig. 1(d)). These sequences are spatially, but not temporally aligned. Proceeding recursively for n = 1,...,N – 1, and choosing the sequence I′₀ as a reference, we align pairs of sequences I′_n and I′_n–1 via a non-uniform time-warping procedure in which we minimize the following cost functions (1 ≤ n < N):

$$\begin{array}{l}{Q}_n\left\{w\right\}\hspace{0.17em}=\hspace{0.17em}\lambda \hspace{0.17em}{\int }_0^T{\int }_0^{L_y}{\int }_{{\tilde{x}}_n}^{{\tilde{x}}_{n-1}+L_x}\left|I_{n-1}^{\prime }\hspace{0.17em}(x,y,w_{n-1}(t))\hspace{0.17em}-\hspace{0.17em}{I}_n^{\prime }(x,y,w(t))\right|\hspace{0.17em}\text{d}x\text{d}y\text{d}t\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\hspace{0.17em}(1-\hspace{0.17em}\lambda )\hspace{0.17em}{\int }_0^T\left|1-\frac{\text{d}}{\text{d}t}w(t)\right|\hspace{0.17em}\text{d}t,\end{array}$$ (6)

where w_n−1 (with w₀(t) = t, the identity function) is the warping function applied to the sequence I′_n−1, and w is a candidate cost function to warp In so that it matches one period of the warped version of I′_n−1. The first integral compares the two neighboring, temporally warped sequences (sequences whose time-axis has been deformed), while the second integral keeps the extent of this deformation in check by penalizing candidate warping functions w that stretch or compress the time axis. These two contributions to the cost function are balanced by the parameter 0 < λ ≤ 1 to favor either good matching of the warped and reference sequences (λ = 1) or the temporal integrity of the warped sequence (λ → 0). We constrain the search for warping functions w_n to the set ℳ = {w ∈ C¹([0, T))|0 ≤ w(t) < L_t and w(t₁) < w(t₂), t₁ < t₂} of continuous, non-negative, and strictly increasing functions bounded by L_t and defined over the interval [0, T). To determine the warping function w_n that minimizes the cost function, we use a previously described dynamic programming algorithm [20]. The optimal warping function w_n produced by this algorithm is such that w_n ∈ {w̄_n ∈ ℳ |Q_n{w̄_n} = min_w∈ℳ Q_n{w}}. Applying this warping function to the sequence I′_n(x, y,t) results in sequence I′_n(x, y, w_n(t)), which is synchronized to all previously warped tiles, I′_i(x, y, wi(t)), i = 0,..., n – 1 (see Fig. 1(e)).

Once all warping functions are determined, we blend the time-warped sequences according to

$$I_{\text{out}}^{(n)}\hspace{0.17em}(x,y,t)\hspace{0.17em}=\hspace{0.17em}{B}_{{\tilde{x}}_n,{\tilde{x}}_{n-1}+L_x}\hspace{0.17em}(x)\hspace{0.17em}{I}_{\text{out}}^{(n-1)}\hspace{0.17em}(x,y,t)\hspace{0.17em}+\hspace{0.17em}(1-B_{{\tilde{x}}_n,{\tilde{x}}_{n-1}+L_x}\hspace{0.17em}(x))\hspace{0.17em}{I}_n^{\prime }\hspace{0.17em}(x,y,w_n(t))$$ (7)

where, B_a,b(x) is a function defined for a < b such that

$$B_{a,b}(x)\hspace{0.17em}=\hspace{0.17em}\{\begin{array}{ll}1,\hfill & x<\hspace{0.17em}a\hfill \\ 1\hspace{0.17em}-\hspace{0.17em}\frac{x-a}{b-a},\hfill & a\hspace{0.17em}\le \hspace{0.17em}x\hspace{0.17em}\le \hspace{0.17em}b\hfill \\ 0,\hfill & x\hspace{0.17em}>\hspace{0.17em}b.\hfill \end{array}$$ (8)

and

$$I_{\text{out}}^{(0)}\hspace{0.17em}=\hspace{0.17em}{I}_0^{\prime }\hspace{0.17em}(x,y,t).$$ (9)

The final output of our algorithm (see Fig. 1(f)) is a blended, synchronized image series that covers one full period

$$I_{\text{out}}(x,y,t)\hspace{0.17em}=\hspace{0.17em}{I}_{\text{out}}^{(N-1)}\hspace{0.17em}(x,y,t),\hspace{0.17em}0\hspace{0.17em}\le \hspace{0.17em}t\hspace{0.17em}<\hspace{0.17em}T.$$ (10)

3. Experiments and Results

To verify our method in practice, we acquired image sequences of a live rat embryo (11.5 days post coitus) using a swept-source OCT (SS-OCT) system as described previously in [3]. Figure 2 shows a representative frame of an SS-OCT image series of the embryonic heart and yolk sac acquired at different frame rates and spatial resolutions. Increasing the frame rate limits the spatial resolution over the region of interest (Fig. 2(a); 512 x 64 pixels, FOV=2.8 mm × 1 mm, 150 frames per second, SNR=15.76dB ). Improving the spatial resolution decreases the frame rate over the same region (Fig. 2(b); 512 × 500 pixels, FOV=2.8 mm × 0.97 mm, 25 frames per second, SNR=15.93dB ). Figure 2(c) shows the series of consecutive image sequences with smaller FOV (each 512 × 64 pixels, FOV=2.8 mm × 0.125 mm, 150 frames per second, 1000 frames (6.7 s, about 13 periods), SNR=15.76dB ) stitched together. The overlaps of the series are marked at the bottom. Figure 2(d) and Media 1 ^{(5.7MB, MOV)} show the result of the synchronization of the same series of sequences. Importantly, the latter reconstructed sequence has the same spatial resolution as the series acquired at low frame rate, Fig. 2(b) (512 × 500 pixels, FOV=2.8 mm × 0.97 mm), but with the same temporal resolution as the low spatial resolution sequence in Fig. 2(a) (150 frames per second) and a better signal to noise ratio (SNR=16.30dB ), thereby demonstrating that we can jointly achieve high spatial and temporal resolutions while preserving the field of view. The increase in SNR is due to the fact that in the areas of overlap the resulting signal is a weighted average of two series.

Fig. 2.

Dynamic mosaicing overcomes frame rate, resolution, and FOV interdependence for fast SS-OCT cardiac imaging. (a) High frame rate, low spatial resolution, large FOV: 150 fps, 512×64 pixels (b) Low frame rate, high spatial resolution, large FOV, 25fps, 512×500 pixels. (c) High frame rate, high spatial resolution, small FOV (12 tiles, before dynamic mosaicing, each 150 fps, 512×64 pixels). Marks at bottom of panel indicate lateral position of individual tiles. (d) High frame rate, high spatial resolution, high FOV (after dynamic mosaicing), 150 fps, 512×500 pixels. Inserts represent time-course (500 ms total duration) of pixel intensity (arbitrary units) at location marked by arrows. SNRs were computed over rectangular boxes. Scale bars are 0.2 mm. ( Media 1 ^{(5.7MB, MOV)} ).

To evaluate the accuracy of our method, we generated test datasets by extracting subsequences from each of N = 11 measured tile image sequence (512 × 64 pixels, FOV=2.8 mm × 0.125 mm, 150 frames per second, 1000 frames; see above and Fig. 2(c)). Specifically, we first split each measured sequence into two overlapping, left and right tile sequences, both of size 512 × 42 pixels and same duration as the original sequence (Figs. 3(b) and 3(c)). We further extracted frames covering two nonoverlapping time intervals (one interval at the beginning and one toward the end of the sequence, both spanning at least two heartbeat periods) from both the original sequence and the left and right sequences. Each measured image sequence thereby resulted in six subsequences labeled A through F (Figs. 3(a)–3(c)).

Fig. 3.

Accuracy evaluation of tiled acquisition and automatic mosaicing procedure. (a) Original sequence (one of the tiles in Fig. 2(c)) is split into (b) a left tile sequence and (c) a right tile sequence. (d)–(g) Sequence subsets are registered in space and/or time. (d) Temporal alignment of first period in subsequence A to subsequence B. (e) Spatio-temporal alignment of first period in subsequence C to subsequence F. (f) Spatio-temporal alignmnent of first period in subsequence E to subsequence D. (g) Spatio-temporal alignment of first period in subsequence C to subsequence E.

Given the six subsequences generated from any one of the 11 measured sequences, we registered the first cardiac cycle in subsequence A to subsequence B (an operation we denote by A→B, see also Fig. 3(d)), and similarly, the first cardiac cycle in subsequences C, E, and C to subsequences F, D, and E, respectively (i.e. C→F, E→D, and C→E, see Figs. 3(e), 3(f), and 3(g), respectively).

We first verified that the registration algorithm produced similar reconstructions when merging left or right tiles from an early cardiac cycle (subsequence C or F, respectively) to right or left tiles, respectively, extracted from later cardiac cycles (subsequence F or D, respectively) or when synchronizing non-tiled sequences directly (subsequences A and B). The temporal registration results for C→F and E→D were therefore compared against each other and against those obtained when directly synchronizing non-tiled frames of the same early cardiac cycle (subsequence A) to the same later cardiac cycles (subsequence B), i.e. A→B.

Assuming the notation W, X, Y, or Z for arbitrary subsequence labels A through F, we compared temporal warping function $w_n^{\text{W}\to \text{X}}(t)$, obtained when synchronizing subsequences W and X, with $w_n^{\text{Y}\to \text{Z}}(t)$, obtained when synchronizing subsequences Y and Z (with all four subsequences, W, X, Y, and Z, extracted from the same tile n), by computing a discrete equivalent of

$$\mathrm{\Delta }{t}_n^{\text{W}\to \text{X},\text{Y}\to \text{Z}}\hspace{0.17em}=\hspace{0.17em}\frac{1}{T}\hspace{0.17em}{\int }_0^T\left|w_n^{\text{W}\to \text{X}}\hspace{0.17em}(t)\hspace{0.17em}-\hspace{0.17em}{w}_n^{\text{Y}\to \text{Z}}(t)\right|\hspace{0.17em}\text{d}t,$$ (11)

which captures the average temporal discrepancy between the two synchronization operations.

Also, we verified that the spatial match of left and right tiles was found correctly (assuming that the sample’s spatial motion was identical from one heartbeat to the next). We calculated the spatial registration error $\mathrm{\Delta }{\tilde{x}}_n^{\text{W}\to \text{X}}$ incurred when aligning subsequence W to subsequence X (both extracted from tile n), given the overlap x_n (known by construction) and the estimated offset ${\tilde{x}}_n^{\text{W}\to \text{X}}$, as

$$\mathrm{\Delta }{\tilde{x}}_n^{\text{W}\to \text{X}}\hspace{0.17em}=\hspace{0.17em}\left|{\tilde{x}}_n^{\text{W}\to \text{X}}\hspace{0.17em}-\hspace{0.17em}{x}_n\right|.$$ (12)

The average spatial error and temporal discrepancies were computed in each case for N = 11 tiles, i.e.,

$$\mathrm{\Delta }{\tilde{x}}^{\text{W}\to \text{X}}\hspace{0.17em}=\hspace{0.17em}\frac{1}{N}\hspace{0.17em}\sum _{n=1}^N\mathrm{\Delta }{\tilde{x}}_n^{\text{W}\to \text{X}}$$ (13)

and

$$\mathrm{\Delta }{t}^{\text{W}\to \text{X},\text{Y}\to \text{Z}}\hspace{0.17em}=\hspace{0.17em}\frac{1}{N}\sum _{n=1}^N\mathrm{\Delta }{t}_n^{\text{W}\to \text{X},\text{Y}\to \text{Z}}.$$ (14)

Numerical results are reported in the first three columns of Table 1 showing discrepancies of less than a frame in time and spatial registration accuracy of the order of one pixel.

Table 1.

Registration Evaluation^*

Registration Experiment
(W→X)	C→F	E→D	C→F	C→E
Spatial Registration Evaluation
Average spatial discrepancy
Δx̃W→X (pixels)	0.4 ± 0.1	1.7 ± 0.5	0.4 ± 0.1	0.0 ± 0.00
Temporal Registration Evaluation
Reference transform (Y→Z)	A→B	A→B	E→D	Identity
Average temporal discrepancy
ΔtW→X,Y→Z or ΔtC→E (frames)	0.2 ± 0.1	0.6 ± 0.4	0.6 ± 0.4	0.0 ± 0.00

^*Each experiment was carried out on N = 11 tiles and the average result is provided. The sequence labels A–F correspond to those in Fig. 3. The notation W→X, stands for “First cardiac cycle of subsequence W is registered to subsequence X”, where symbols W, X, Y, and Z stand for labels in the range A–F.

Finally, as a control to ensure that the spatial registration procedure was accurate and did not affect the accuracy of the temporal registration procedure, we also merged tile-pairs generated from the same cardiac period. To this end, we registered subsequence C to subsequence E (within the same tile, see Fig. 3(g)). The spatial error was obtained as above, while the temporal error was obtained as

$$\mathrm{\Delta }{t}_n^{\text{C}\to \text{E}}\hspace{0.17em}=\hspace{0.17em}\frac{1}{T}{\int }_0^T\left|w_n^{\text{C}\to \text{E}}(t)\hspace{0.17em}-\hspace{0.17em}t\right|\hspace{0.17em}\text{d}t.$$ (15)

Both spatial and temporal registration errors, reported in the last column of Table 1, show that registration was perfect in this case for all 11 tiles tested.

4. Conclusion

In summary, we have demonstrated the applicability of an image acquisition protocol and reconstruction algorithm for OCT images of cardio-vascular structures whose motion is cyclical, as dictated by the cyclic contraction of the heart. Dynamic mosaicing overcomes the limitations that arise from the frame rate, resolution, and FOV being interdependent for fast OCT cardiac imaging. Although we believe our space-time registration method is well adapted to the task at hand as it provides a simple solution to separately address the spatial and temporal registration problems (which considerably reduces computational complexity) other methods for space-time registration developed in the context of human cardiac imaging in adults using magnetic resonance imaging, single-photon emission computed tomography, or X-ray computer tomography could be used for similar purposes. In particular, use of registration techniques that are not only nonrigid in time but also in space [21, 22] could likely improve our mosaic reconstructions in cases where the spatial motion of the embryonic heart is not conserved from one heartbeat to the next.

Future work will include investigating possible improvements that could result from using such methods as well as combining our procedure with previously developed noise removal strategies [23] or techniques to reconstruct high frame rate 3D+time volumes from 2D+time series [17].