High definition live 3D-OCT in vivo: design and evaluation of a 4D OCT engine with 1 GVoxel/s

Abstract

We present a 1300 nm OCT system for volumetric real-time live OCT acquisition and visualization at 1 billion volume elements per second. All technological challenges and problems associated with such high scanning speed are discussed in detail as well as the solutions. In one configuration, the system acquires, processes and visualizes 26 volumes per second where each volume consists of 320 x 320 depth scans and each depth scan has 400 usable pixels. This is the fastest real-time OCT to date in terms of voxel rate. A 51 Hz volume rate is realized with half the frame number. In both configurations the speed can be sustained indefinitely. The OCT system uses a 1310 nm Fourier domain mode locked (FDML) laser operated at 3.2 MHz sweep rate. Data acquisition is performed with two dedicated digitizer cards, each running at 2.5 GS/s, hosted in a single desktop computer. Live real-time data processing and visualization are realized with custom developed software on an NVidia GTX 690 dual graphics processing unit (GPU) card. To evaluate potential future applications of such a system, we present volumetric videos captured at 26 and 51 Hz of planktonic crustaceans and skin.

1. Introduction

Optical coherence tomography (OCT) is a depth-resolved imaging modality which provides micrometer-scale cross-sectional and three-dimensional (3D) information of the scattering properties of biological samples [1]. Ophthalmic imaging and endoscopic imaging are the two main application fields of OCT. Recently, the possibility to use OCT for surgical guidance, e.g. in an operating microscope, is drawing increasing attention as a further application [2–8]. Even though already a 2D OCT functionality is highly beneficial for this application, a video rate real time 3D volumetric OCT (4D-OCT) could generate a new class of optical tools in clinical practice. As will be explained below, this requires the combination of a sufficiently fast OCT imaging setup including scanners and data acquisition as well as adequate real-time data processing that can keep up with the data rate generated by the OCT system. It is the combination of all these parts that makes the realization of real-time 4D OCT particularly challenging.

Volumetric OCT imaging to capture sample dynamics has been demonstrated quite early. Already in 2002, Laubscher et al. [9] implemented a time domain (TD) OCT with a detector array to capture volumes with 58 x 58 x 58 voxels at 25 Hz video rate. However, the sensitivity of 76 dB limited the image quality and the sample choice. In 2007, Yelin et al. [10] demonstrated OCT at up to 20 volumes/second to visualize embryonic heart microstructure. The depth scan rate of 54 kHz limited the acquired volume sizes to 60 x 54 depth scans and no real-time processing was performed. A different approach to overcome the OCT speed limitation was gating the image acquisition [11, 12], but this is applicable for repetitive motion only and is in most cases incompatible with real-time visualization.

Pushing the speed of OCT has been a major driving force in the field from the earliest days on. The slow data acquisition speed of early TD-OCT systems in the range of ~1 kHz initially limited OCT imaging to capturing single B-frames. Later, the introduction of frequency domain (or Fourier domain; FD) detection techniques for OCT allowed a substantial increase in imaging speed. This was enabled by both, an inherent sensitivity advantage of FD-OCT over TD-OCT in typical OCT samples as well as by eliminating the need for an inherently slow scanning mechanism required in TD-OCT [13–18]. Due to advances in technology, depth scan rates of ~50-200 kHz are now common for both spectrometer-based (SD-OCT) and swept-source OCT (SS-OCT, also called optical frequency domain imaging, OFDI) [19–26]. Even faster systems with at least 0.5 MHz line rate and acceptable image quality include a dual-camera system with 0.5 MHz [27], line-field SD OCT at ~0.8 MHz [28], streak-mode FD-OCT [29] at 1 MHz, vertical cavity surface emitting laser (VCSEL) based OCT pioneered by Jayaraman et al. [30–34] at up to 1.2 MHz line rate, full-field OCT with 1.5 MHz equivalent line rate [35], a 1060 nm FDML-based setup with up to 2 x 3.35 MHz [36], a discrete spectrometer SD-OCT setup [37, 38] with useful image quality up to ~10 MHz, as well as FDML-based setups [39–43] with up to 4 x 5.2 MHz [39]. Despite their high speed, most of these demonstrations have acquired only one or a few 3D data sets at a time at the specified speed. The main technological reason impeding sustained volumetric imaging was in many cases a bottleneck in data transmission rate. Once removed, the major remaining obstacle is sufficiently fast data processing.

Real-time data processing and visualization of large amounts of OCT data has recently become possible at moderate cost due to advances in graphics processing unit (GPU) computing. However, in many cases fast data processing with impressive performance has been applied to OCT data acquired previously at much slower speed, as in [44–48]. Apart from conventional OCT, GPUs have also been applied to functional imaging like Doppler processing [49–51] and speckle variance OCT [52]. Also, more complicated OCT reconstruction algorithms like dispersion encoded full-range OCT [53, 54] and non-uniform FFT [47] have been implemented on GPUs, showing the great potential of GPU computing in OCT.

By combining both, fast OCT and fast computing, only a few groups have published 4D volumetric OCT imaging with real-time visualization [6, 7, 38, 47, 49, 55–57]. The fastest complete 4D OCT system with real-time visualization previously reported is described by Choi et al. [38] featuring a real-time volume rate of 12 Hz with volume sizes of 256 x 256 x 160 voxels. They also report higher volume rates by reducing the volume sizes accordingly, yet always with voxel rates below 200 MVoxels/s. As a noteworthy difference to most other real-time OCT systems, Choi et al. use a massive amount of dedicated hardware including more than 20 FPGAs (field programmable gate arrays) for several steps in the OCT data processing. Here, we present a more than 5-times faster real-time 4D OCT system featuring more than 1 GVoxel/s. The system can acquire, process and display volume sizes of 320 x 320 x 400 voxels at a sustained rate of 26 volumes/second. All data processing is performed on a single desktop computer using a consumer grade dual-GPU graphics card. This is to our best knowledge the fastest real-time OCT imaging system today. It combines a video speed volume rate with a volume size comparable to commonly used single-shot 3D volumes in recent commercial OCT devices.

2. System design considerations

2.1 Sustained voxel rate as measure for OCT speed

The purpose of OCT as a 3D imaging modality is to extract volumetric information from a sample. Hence, the most natural way of measuring OCT speed is the number of obtained voxels per second (voxel rate) [39, 58]. However, care has to be taken that these voxels (1) can actually carry information and (2) that this information is not redundant with neighboring voxels along the depth axis or fast scanning axis. Also, (3) dead times during acquisition introduced by galvanometric scanners, the ADC trigger re-arming, data transmission, etc. have to be subtracted to accurately characterize the sustained throughput in voxels/second.

To ensure criterion (1), the signal-over-depth roll-off performance of the whole imaging system has to be considered. Depending on the specific system, the roll-off is usually limited by one of the following: In SS-OCT (a) by the bandwidth limit of the analog detection path including photodetector, amplifier, digitizer, (b) by the instantaneous linewidth (coherence length) of the swept source, (c) by insufficient sampling rate in the digitizer. In SD-OCT one limit is given by (a) the finite number of pixels used in the spectrometer. The other limit (b) is the roll-off imposed by the limited spectral resolution of the spectrometer. Besides these two effects limiting SD-OCT performance there are additional mechanisms degrading the roll-off, but they are in most cases not dominant [59, 60]. Since OCT images usually give meaningful contrast way beyond the −6dB roll-off point and have a dynamic range of ~40dB at high speeds, we proposed the −20dB point limit in [39]. Today, this mostly applies to SD-OCT setups because recent high speed SS-OCT systems (especially VCSEL and dispersion compensated FDML based ones) are usually limited by the detection bandwidth or sampling rate before reaching the coherence length.

For criterion (2), the strictest definition would be to measure the number of resolvable points along the axial direction. However, this method is impractical because it can vary along the depth direction. Also the definition of resolution in the case of an imagining system with speckle contrast is not straight forward [61]. However, in many OCT setups where the sampling rate is not much larger than twice the analog bandwidth (Nyquist criterion) and no zero padding is used, the point spread function typically spreads <2 samples. A simple yet useful method which is compatible across SS-OCT and SD-OCT is to divide the number of raw ADC samples by 2 to give an upper limit for the usable voxels per depth scan. The division by 2 accounts for the negative frequencies after the Fourier transform and is not applicable for full range OCT.

Table 1 shows a compilation of previous work on real-time 4D OCT. Only publications which presented full working OCT systems were included. Figures in the table refer to actual OCT performance with all required steps from light source, sample scanning, data acquisition to 4D display. In many cases, data acquisition was the bottleneck. Therefore, publications which demonstrated the feasibility of high speed data processing and/or visualization only without acquiring data at the required speed were not included. Specified OCT speed in voxel rate refers to the true voxel rate without artificially generated voxels by zero padding or interpolation. As explained above, the true voxel rate cannot be higher than half the data rate in samples per second (or full data rate in case of full range OCT). For instance [38], makes use of 320 depth samples which results in 160 “true” net depth voxels after FFT (and removal of the complex conjugate parts) and hence a “true” voxel rate of ~122 MV/s which is close to half the data rate. In comparison, with zero padding the 256 samples after FFT result in a theoretical raw voxel rate of ~195 MV/s which is also quoted in [38] but this value is above half the data rate, hence contains redundant information. Analogous to these considerations, the net voxel rate of our system is also lower than the raw voxel rate after FFT. In this paper, when taking into account the FFT size of 1024, the raw, partially redundant, voxel rate would be ~1.40 GV/s while the “true” net voxel rate is 1.07 GV/s.

Table 1. Previous work on real-time 4D OCT presenting full OCT systems including light source, data acquisition and real-time display. Columns 2, 3 specify the raw line and frame rates. The raw line rate is the swept source sweep rate or camera line readout rate. The raw frame rate is the galvo scanning rate. Column 5 shows the calculated “true” net depth scan size in image voxels along the depth axis, i.e. without any zero-padding or interpolation which would artificially create voxels and with complex conjugate frequency samples removed. (‘F’ denotes full-range OCT without removal of negative frequency samples.) Columns 6, 7 are the usable frame and volume sizes in depth scans / frames. Raw values before removing back-scan/turning point parts are included in brackets. Column 8 is the raw data rate in 10⁶ samples per second at the acquisition device. The voxel rate cannot be larger than 0.5x (1x) this value for usual (full-range) OCT, respectively. Column 9 is the effective depth scan rate after removing all those scans not displayed (e.g. during galvo turning, back-scan,…). Finally, column 11 specifies true OCT speed in million voxels/second and column 10 the efficiency which is the effective scan rate divided by the raw scan rate or in other words the fraction of non-wasted raw A-scans. * denotes numbers where due to missing information, an estimate close to the optimum value was used resulting in potentially over-estimated efficiencies close to 100%.

1	2	3	4	5	6	7	8	9	10	11
Source, Pub. date	Raw line rate	Raw frame rate	Volume rate	True depth scan size	Frame size (raw) usable	Volume size (raw) usable	Data rate	Effect. scan rate	Efficiency	OCT speed
	kHz	Hz	Hz	Voxels	Scans	Frames	MS/s	kHz	%	MV/s
[55] 2010	215	566	7	512	(380) 300	*(81) 80	220	168	78	86
[56] 2010	128	~140	9	1024	*(130) 100	*(109) 100	250	90	70	92
[62] 2010	125	1000	10	512	*(125) 110	(100) *98	128	108	*89	111
[6] 2011	128	500	5	F: 1024	(256) 250	(100) *98	131	122	*95	125
[48] 2012	128	800	10	F: 1024	(160) *155	(80) *78	131	121	*95	124
[38] 2012	2 500	4000	12	160	(625) 256	(333) 256	251	786	31	122
[38] 2012	10 000	4000	12	160	(2500) 256	(333) 256	251	786	8	122
[38] 2012	10 000	8000	41	160	(1250) 128	(195) 128	215	672	7	107
This paper	3 216	4290	26	400	(376) 320	(328) 320	2200	2670	83	1070
This paper	3 216	4290	51	400	(376) 320	(168) 160	2200	2610	81	1040

In Table 1, column 10 is the efficiency of the OCT system and specifies which fraction of the A-scans can actually be used. Usually, camera based OCT systems can achieve good efficiency because they are not limited by the data transfer rate but only by the scanner dead times as can be seen in the first 5 rows of Table 1. The values of 89% and 95% were actually estimated by us and are probably over-estimated to err on the side of higher OCT speed. In contrast [38], which applies a discrete spectrometer, is able to theoretically produce data at very high rates – much faster than what we present in this paper – but the data transfer rate is limited to only ~250 MS/s which results in rather low efficiency. In effect, this system could achieve a higher voxel rate if the bottleneck in the data transfer was removed. In our setup, the efficiency of 83% (or 81% in double speed mode) is given by dead times at fast axis galvo scanner turning points due to sinusoidal scanning and slow axis back-scan.

2.2 Design considerations

To design an OCT system with real-time visualization that delivers more than 1 GVoxel/s requires careful layout of all system components. It is easiest to walk the flow of data in reverse order. Since dense and with respect to resolution isotropic sampling offers many advantages [40], a well-balanced combination of parameters for 1 GVoxel/s at video rate is to choose a volume rate of 25 Hz and a volume size of 320 x 320 x 400 voxels (width, height, depth). At proper scan area, this results in isotropic sampling using 320 x 320 depth scans. Each depth scan has 400 voxels which results in a cube with balanced edge lengths (in voxels) and provides sufficient depth samples for typical OCT imaging.

3D visualization: 1 GVoxel/s at an update rate of ~25 Hz translates into a volume size of ~40 MVoxels. We find that recent GPUs are fast enough to perform good quality ray casting on volumes of that size even for typical full-screen resolutions. It is however easier to implement smooth updates and good response to user input (such as rotating the displayed volume) by reserving a dedicated GPU for the process of visualization only.

OCT data processing usually involves the steps of resampling, apodizing, Fourier transformation, compression (logarithm) and clipping. For 1 GVoxel/s and an A-scan size of 400 usable voxels along the depth axis, this corresponds to 2.5 million A-scans per second. Each A-scan will have ~800 samples of raw data, a corresponding FFT size of 1024 and finally 400 usable voxels because half the samples are lost after Fourier transform (negative frequencies). As has been shown previously, recent high end consumer GPUs are fast enough to compute millions of A-scans per second of similar and even larger size [7, 44, 46]. Due to processing speed available on GPUs, we find that one can even use computed 3rd order interpolation rather than usual texture-based linear interpolation and benefit from better image quality.

Data transfer: Interestingly, today the data transfer rate off the AD-converter board is typically the main bottleneck for real-time OCT system performance. Some previous publications in the past have mentioned that a significant fraction of time was spent for copying the raw data from host memory onto graphics memory for GPU processing (e.g [50].). In contrast, we find that with today’s hardware and proper software design, CPU to GPU transfers do not at all affect processing speed because these transfers can be done asynchronously while the GPU is processing the previous data chunk. For the 1 GVoxel/s system with 800 raw samples per A-scan, ~2 GS/s (10⁹ samples per second) are required. This corresponds to 2 or 4 GBytes/s for 8 or 16 bit wide samples, respectively. Recent graphics cards with PCIe generation 3 offer beyond 8 GBytes/s actual transfer rate which is more than sufficient. However, data acquisition cards typically lack behind with respect to data streaming capabilities. Today, typical ADC cards (e.g. Alazar Technologies Inc., Gage Applied Technologies Inc., Innovative Integration, Signatec / DynamicSignals LLC, SP Devices Sweden AB, etc.) with PCIe interface can stream beyond ~1 GByte/s (PCIe generation 1, 8 lanes) although very recently, PCIe generation 2 cards have become available that offer beyond 3 GBytes/s (e.g. Alazar Technologies, Gage Applied). Therefore, we chose to use two 8 bit ADC cards in parallel where each card can stream slightly more than sustained 1 GByte/s (and 1.4 GByte/s peak) resulting in a combined sustained transfer rate beyond 2 GS/s. The cards are operated in a volume interleaved acquisition mode (see below).

Scanning: With 320 frames per volume and a volume rate of 25 Hz, the required frame rate is ~8 kHz. Scanning across an area of ~1cm² at that speed can no longer be done with standard galvanometric scanners. So, for the fast axis, a resonant scanner was used. Because of the sinusoidal motion of this type of scanner, a part of the movement near the turning points cannot be used. We found that a good compromise is to use the most linear ~85% [39]. Therefore, a frame with 320 usable A-scans requires about 376 raw depth scans. For the slow axis, usual non-resonant scanners can be used to scan linearly and uni-directionally across the sample. Since these scanners need some time to move back after completing a volume, some unused frames need to be inserted after every volume. In our setup, we found 8 frames to be sufficient for the back movement.

Swept source: For a volume with 320 x 320 depth scans at 25 Hz, the A-scan rate needs to be 2.56 MHz. Factoring in an ~85% scanning duty cycle along the fast scanning axis and 8 dead frames along the slow axis, the required A-scan rate is ~3.1 MHz. Since 400 voxels along the depth axis are desired, the whole analog detection path (including coherence roll-off) must provide a usable OCT signal up to ~1.2 GHz. To fulfill the Nyquist criterion, the required sampling rate must be at least ~2.5 GHz. For the OCT system presented here, a 3.2 MHz depth scan rate was chosen. The detection path has a nominal −6dB bandwidth of 1 GHz and a sampling rate of 2.57 GHz and features a −20dB roll-off point very close to Nyquist frequency.

3. Experimental setup and data processing

3.1 Buffered FDML laser

The video rate 4D OCT setup uses an FDML laser as swept laser source. Figure 1 shows a schematic of the laser source which is an enhanced version of the one presented in [42]: It represents the first demonstration of a 3.2 MHz FDML laser with a dispersion compensated cavity, improving speed by a factor of 2 compared to our previous results. The FDML cavity features a dual fiber Bragg grating (FBG) dispersion compensation module (DCM), first presented in [63], to enhance the roll-off performance [64]. Since the DCM requires a 2.5 km long FDML cavity corresponding to a fundamental FDML round-trip frequency of merely 80 kHz, the laser is operated in the 5th harmonic at 402 kHz. Using an 8x buffering stage [21], the sweep rate is further increased to 3.2 MHz. For the 8x buffering, the SOA is switched on only for 12.5% of the time by a high speed laser diode controller in the laser cavity (WL-LDC10D, wieserlabs.com). The phase of this modulation is adjusted such that it coincides with the most linear part of the sinusoidal wavelength sweep [65]. The resulting 312 ns long wavelength sweep with 12.5% duty cycle is delayed and time-interleaved with copies of the original sweep in a series of 3 fiber delay stages. To ensure proper amplification in the final booster SOA and to prevent stripes in the image, the polarization of each delay stage has to be carefully adjusted.

Fig. 1.

Schematic of the high speed FDML laser followed by an 8x buffer stage and a booster optical amplifier. The filter is driven at 400 kHz resulting in a 3.2 MHz scan rate after the buffer stage. SOA: semiconductor optical amplifier, FBG: fiber Bragg grating, PC: polarization controller, LDC: laser diode controller, FRM: Faraday rotation mirror, FP-TF: Fabry-Pérot tunable filter, ISO: isolator, CTRL: Driver electronics and arbitrary waveform generator, OSA: optical spectrum analyzer.

3.2 OCT interferometer and data acquisition

The OCT interferometer is built in a Mach-Zehnder configuration using a circulator in both reference and sample arm as shown in Fig. 2. The reference arm power was attenuated to ~500 µW by slight misalignment of the free space reference delay. The power on the sample was ~40 mW for all measurements. We measured a sensitivity of −102 dB which is close to shot noise limit of ~105 dB when taking into account a 3 dB back coupling loss.

Fig. 2.

Schematic of the interferometer and the data acquisition. After the photo diode, the radio frequency (RF) signal is split and subsequently fed into two digitizer cards. These two digitizers operate at 2.57 GS/s in volume interleaving mode. Sampled data is streamed into computer RAM. For real-time visualization, RAM only serves as a temporary buffer for a few volumes. Bidirectional scanning allows an 85% scan duty cycle along the fast axis resulting in a sustained average total data transfer rate of ~2.1 GBytes/s.

The interference fringes are detected by a 1 GHz differential photoreciever (WL-BPD1GA, wieserlabs.com). The electrical fringe signal is then split by a passive radio frequency (RF) splitter to feed 2 identical high speed digitizer cards (Signatec PX1500-4). To ensure proper sampling of the analog signal which has components beyond 1 GHz, the ADCs are operated at 2.57 GHz sampling rate with a common external sampling clock.

Due to limited streaming capability of slightly more than 1 GBytes/s, two separate PCIe digitizer cards have to be used. These cards are operated in volume interleaving mode: All the trigger signals for the 1st, 3rd, and all following odd-numbered volumes are routed to the first digitizer card. The 2nd, 4th and all subsequent even-numbered volumes are captured by the second ADC card. The routing of the trigger signals is performed by a home developed FPGA board. While capturing a volume, the 8 bit ADC card runs at 2.57 GS/s and hence produces sample data at a rate of 2.57 GBytes/s. This data is buffered in the on-board FIFO RAM on each card. Since sampling is only performed during the most linear 85% of fast axis galvo scanner movement (see next section), the average net data rate per acquisition card is ~1.1 GBytes/s. The data transmission from the card into host memory on the computer is running continuously in the background. The rather odd sampling rate of 2.57 GHz was found out experimentally to be the fastest one which reliably allows continuous streaming from both cards without losing any data.

3.3 Scanning protocol for 26.1 Hz volume rate

In the sample arm, a 2 axis galvanometric scanner and a 50 mm lens in telecentric configuration were used to scan the imaging beam across the sample. To reach the required high scan speeds, the fast axis employs a 4.29 kHz resonant scanner (SC-30 from EOPC). The scanner is not running in self-resonant mode but is instead synchronized to a suitable fraction of the laser sweep rate to ensure stable operation across all volumes during an imaging session. The sinusoidal motion of the resonant scanner introduces distortions in the image. As described in [39, 42], a reasonable approach is to perform bidirectional scanning and discard ~15% of the scan time near the turning points while applying numerical correction in post-processing to the remaining ~85%. To save data transmission bandwidth, ADC sampling is performed only during the useful ~85% of the time. The phase of the scanner is adjusted with respect to data acquisition so that the most linear part of the sinusoidal motion is used and such that the acquired portion of both sweep directions covers the same area on the sample.

The scanning protocol is shown in Fig. 3. Due to bidirectional scanning, the raw frame rate during volumetric imaging is 8.58 kHz resulting in 375 theoretical depth scans per frame. Since depth scans at the turning points of the scanner are thrown away, only ~85% of them are used, resulting in an actual usable net frame size of 320 depth scans.

Fig. 3.

Scanning protocol used by the video rate 3D OCT setup. The fast axis uses a 4.29 kHz resonant scanner. Depth scans near the turning points of the sinusoidal galvo motion are not used leaving a usable frame size of 320 out of 375 depth scans. 8 frames are lost during fly back of the slow axis. The net frame rate is 8.37 kHz and the volume rate is 26.1 Hz resulting in usable volumes of size 320 x 320 x 400 voxels. The overall acquisition duty cycle is 83%.

The scan along the slow axis is performed in uni-directional manner with a modified sawtooth waveform at 26.1 Hz using a 6215H scanner from Cambridge Technologies Inc. A raw sampled volume consists of 328 frames. 8 of these frames cover the fly back and are removed during image processing. The remaining 320 usable frames form the displayed volume. The resulting net frame rate is 8.37 kHz, the overall scan duty cycle contributing to the true voxel rate is 83%. The voxel rate is therefore 320 x 320 x 400 x 26.1 = 1.07 GVoxels/s.

3.4 Scanning protocol for 51 Hz volume rate (“double speed mode”)

Since the fast axis resonant galvo is fixed in speed, the only option of tuning the volume rate is by changing the slow axis movement. By reducing the usable frames per volume by a factor of 2 to 160 ( + 8 for the back scan), the OCT system can be set to acquire and display volumes of 320 x 160 x 400 voxels at a volume rate of ~51 Hz. This is referred to as “double speed mode” in this publication and is also visualized in real time. Due to increased dead time by the slow axis scanner, this translates into a slightly lower voxel rate of 1.04 GVoxels/s.

Even though videos acquired at 51 Hz are displayed live at full 51 volumes per second on the computer screen, they can also be recorded and played back later at 25 Hz enabling volumetric “slow motion OCT”.

3.5 Data processing and visualization

Data processing and visualization is performed on a single Intel Core i7 3930 based desktop computer running Debian Linux 64 bit. The computer hosts both ADC cards as well as an NVidia GTX690 dual GPU card. The whole video rate live OCT acquisition and 3D display is performed by a self-developed multi-threaded C + + software. A schematic of the data flow is shown in Fig. 4 (left).

Fig. 4.

Left: Data processing on the host computer. After the acquisition, the data is transferred into GPU RAM. The NVidia GTX690 is a dual GPU card. The first GPU performs OCT data processing while the second GPU is dedicated to 3D visualization using a ray caster. Right: Screen shots from NVidia visual profiler showing the timing sequence of OCT data processing. The 4 steps in the most zoomed-in view are resampling and apodizing (violet), FFT (cyan), magnitude compression (red) and copying into the output texture memory (yellow).

Data from the ADC cards is continuously streamed into a dedicated host memory area capable of buffering the raw data for several volumes. Since the data processing is faster than the data acquisition, this buffer usually holds no more than 2 partially transferred volume data sets (one from each ADC card). As soon as a volume is complete, it is copied to the on-board RAM of the GPU at a rate of >9 GBytes/s. Copying is performed asynchronously while the previous volume is still processing on the GPU.

The GTX690 is a dual GPU card featuring 2 NVidia “Kepler” chips with 1536 stream processors each. One of these GPUs is used for OCT data processing: Resampling with a 3rd order Hermite kernel, apodizing to Hann shape, fast Fourier transform (FFT) with 1024 samples, finally dynamic range compression (logarithm of magnitude) and cut level clipping. The interpolation process generates 1024 interpolated samples out of 800 raw samples. Since both the negative frequency components as well as those high frequency components introduced by interpolation are discarded, the processing pipeline generates depth scans with 400 voxels each. Figure 4 (right) shows screen shots from NVidia visual profiler while running life OCT at 1 GVoxel/s. 3 different zoom levels are shown. As can be seen, there is sufficient idle time between volumes so that consistent live display is performed without losing any data. Two streams are used so that host-to-GPU copy operations can run concurrently with data processing. Resampling and apodizing are performed by one and the same kernel.

Although the GPU code used for this publication is fast enough to process the incoming data in real time, it is expected that further optimization of the algorithms on the first GPU will reduce computation time further and allow for more complex processing to be carried out in real time. For instance, Zhang and Kang [48] have demonstrated that an NVidia GTX590 dual GPU card with 2 x 512 stream processors can perform FD-OCT data processing for complex-conjugate free full-range OCT with 16 bit raw data at ~0.45 GVoxels/s when using both GPUs for the task. The GTX690 used in this publication has 3 times more stream processors per GPU and twice the PCIe bandwidth suggesting that even full-range OCT at 1 GVoxel/s is feasible if both GPUs are used for processing and a further dedicated GPU card is inserted for volume rendering.

The second GPU runs a ray casting algorithm for high quality 3D visualization [66]. The ray caster always operates on the most recent volume available and can visualize 3D volumes at a rate higher than the 26.1 Hz (or 51 Hz) supplied from the OCT imaging system and data processing. Hence, it displays a live 3D visualization of the acquired volumes at 26.1 Hz video rate (or 51 Hz real-time rate in double speed mode). The software allows to interactively rotate and zoom the 3D image in real time as well as to change the color palette and to encode and store the 3D visualizations into a H.264 video in real time.

In summary, the data processing and 3D display is capable of handling the data rate of over 2 GBytes/s generated by the ADCs. It generates volumes with a usable size of 320 x 320 x 400 voxels at a rate of 26.1 Hz which is slightly more than 1 GVoxel/s.

4. Results

4.1 Power, sweep range

The laser was operated at a sweep range of 80 nm in the 1310 nm band. This results in a total depth range of ~3.3 mm in water (8.3 µm per voxel). A higher sweep range would increase the resolution at the cost of an even smaller depth range. Ultimately, a higher detection bandwidth and even faster real-time sampling would be desirable but the parameters chosen here were already at the technical limit of our specific setup.

The lateral voxel spacing is ~30 µm. Due to an optical spot size of ~50 µm, this results in a slight lateral over-sampling.

The optical power after the final booster SOA is about 100 mW. For imaging, the power on the sample was ~40 mW which is spread over a ~1 cm² area.

4.2 Video rate volumetric 3D movies: sample applications

In this section we demonstrate different imaging examples of the fast live 4D OCT. The examples are chosen such that they resemble OCT imaging applications where MHz live OCT imaging might be highly desired in the future.

The real-time live OCT system has two major operating modes: A setup mode where the background data is acquired by manually blocking the sample arm and the resampling information is acquired by inserting a mirror into the sample arm. After that, the software is switched into live 3D imaging mode. In this mode, the image can be freely rotated smoothly and cut levels can be adjusted in real time. From this mode, movie recording of the displayed 3D reconstructions can be started and stopped any time by a button. The H.264 movie is encoded in real time and stored on hard disk and there is no limit on the length except disk space. To demonstrate the high speed live OCT, several different volumetric movies were created, each of them representing an important potential future application of MHz live video rate 3D OCT:

Application 1: MHz live video rate 3D OCT for volumetric turbulent flow analysis: The study of flow is fundamentally a “3D problem” and is of interest in various cases, e.g. for studying heart development [67] or atherosclerotic lesions due to fluid shear stress [68–70], understanding micro fluidics and turbulence, e.g. in mixing applications. Time gating or temporal sub-sampling cannot be applied to non-repetitive processes, or processes that are dominated by turbulent flow or if particular features (like swimming cells) need to be tracked in order to reconstruct the flow.

Figure 5 (top, left) (Media 1 ^{(6.8MB, MOV)} ) shows a fluid dynamic visualization: A resistive wire spiral in water was heated electrically giving rise to convection. The video first shows a view from top onto the water surface which is then rotated to a side view. To visualize the water convection, a drop of milk is then put into the water. The flow can now be seen in the OCT.

Fig. 5.

Examples from live display: Top left: Convection at wire spiral - (Media 1 ^{(6.8MB, MOV)} ). Top right: Daphniae - (Media 2 ^{(8.3MB, MOV)} ). Bottom left: Dynamics of filter legs, Triops - (Media 3 ^{(9.4MB, MOV)} ). Bottom right: Syringe over finger - (Media 4 ^{(7.8MB, MOV)} ).

Application 2: MHz live video rate 3D OCT for capturing dynamic 3D movements and morphology in real time. Sufficiently fast 3D OCT allows capturing volumetric data sets of e.g. small animals like Daphnia and other crustaceans in their natural state without the need to immobilize them either mechanically or chemically. This is of interest e.g. to study morphology changes induced by the presence of predators [71]. 4D OCT adds the capability to capture motion and assess e.g. the coordination of movements while these animals actively move or eat. Also the heart beat and the fluid flow inside their bodies (see also application 1 above) can be of interest.

Figure 5 (top, right) (Media 2 ^{(8.3MB, MOV)} ) shows an application in biology featuring Daphniae which are water planktonic crustaceans with a length of 1-2 mm. The video shows several of these semi-translucent animals moving around in a tiny water reservoir. One can see the arms used by these animals to propel themselves in the water as well as the coat-like shape of their carapace. A part of the movie was created in “double speed mode” (51 Hz) and is shown as a 2 x slow motion playback.

Figure 5 (bottom, left) The Triops in (Media 3 ^{(9.4MB, MOV)} ) is a somewhat larger crustacean. The imaged animal had a length of ~15 mm and was too large to fit in the imaging size in a single view so it is slowly moved across. It was glued with the back of its carapace to the bottom of a shallow bowl filled with water. OCT imaging from the top hence gives a view on the front side with its multiple filter legs. With the slow-motion in “double speed mode” one can easily time resolve and identify the wave-like movement of the legs particularly well.

Application 3: MHz live video rate 3D OCT for surgical guidance. A major future application of 4D OCT might be surgical guidance, especially during micro-surgeries [2, 3, 6–8, 72–74]. One difficulty is that a usually reflective and non-transparent tool is used over and within highly scattering tissue. Both the surface reflex as well as the reflex of the surgery tool can adversely affect image quality, especially since 8 bit ADCs with limited dynamic range have to be used for speed reasons. For this reason, this application is especially challenging. We show that high quality imaging in this situation is possible at the GVoxel rate.

Figure 5 (bottom, right) (Media 4 ^{(7.8MB, MOV)} ) shows a needle touching and moving around on a fingertip. One can very well see the elastic properties of the skin and the orientation of the needle tip.

5. Conclusion and outlook

This paper evaluates high quality video rate live 3D MHz OCT. A technical description of the FDML light source, the OCT imaging system, the data acquisition, the parallel processing and live visualization is given. The critical management of the involved asynchronous data transfer, the timing of the individual processing threads and the various transfer bottlenecks are analyzed. We demonstrated, for the first time, a complete OCT system capable of performing OCT with real-time live display at >1 GVoxel/s. This allows the acquisition, computation and 3D visualization of volumes with 320 x 320 x 400 usable voxels at 26 Hz update rate or 320 x 160 x 400 at 51 Hz. This is achieved with a combination of a very fast swept laser source, dual high end digitizer cards and a state of the art consumer grade GPU card with dedicated processing software. The overall OCT scanning duty cycle is 83% because ~15% “dead time” is introduced around the turning points of the fast axis resonant galvo scanner movement and another ~2% for the back movement of the slow axis.

With the described system, three different imaging applications are demonstrated and analyzed representing some of the most important groups of future OCT applications where MHz live OCT might be highly desired: (a) 3D transient vectorial flow analysis (b) volumetric live OCT of non-immobilized objects including free-moving small animals (c) live surgical guidance.

Real-time live OCT with large volumes at flicker free video rate has the potential to create a new class of optical instruments for special applications like surgical guidance, product monitoring or real-time monitoring of biological samples.

Exposure limit calculations suggest that future systems, if necessary, might achieve even higher speed at similar imaging quality. Higher volume rates, e.g. at the demonstrated 50 Hz, or even faster, might be used to visualize fast processes and generate slow-motion 3D videos. On the other hand, volumes with a higher voxel count, like 512³ might enable real time monitoring applications where large and rapid OCT survey scans are required.

We expect that in the future, high speed OCT will greatly benefit from current trends in telecom technology of optical networks. Especially the transition to coherent receivers, which require the analog detection of the signals, motivates the development of multi-GHz ADCs, and MHz live 3D OCT will ultimately also benefit from these developments.