ULA-OP 256: A 256-channel open scanner for development and real-time implementation of new ultrasound methods

Abstract

Open scanners offer an increasing support to the ultrasound researchers who are involved in the experimental test of novel methods. Each system presents specific performance in terms of number of channels, flexibility, processing power, data storage capability and overall dimensions. This paper reports the design criteria and hardware/software implementation details of a new 256-channel ULtrasound Advanced Open Platform (ULA-OP 256). This system is organized in a modular architecture including multiple front-end boards, interconnected by a high-speed (80 Gbit/s) ring, capable of finely controlling all transmit and receive signals. High flexibility and processing power (equivalent to 2500 GFLOPs) is guaranteed by the possibility of individually programming multiple DSPs and FPGAs. 80 GB of on-board memory are available for storage of pre-, post-beamforming and baseband data. The use of latest generation devices allowed to integrate all needed electronics in a small size (34 x 30 x 26 cm). The system implements a multi-line beamformer that allows obtaining images of 96 lines by 2048 depths at a frame rate of 720 Hz (expandable to 3000 Hz). The multi-line beamforming capability is also exploited to implement a real-time vector Doppler scheme in which a single transmit and two independent receive apertures are simultaneously used to maintain the analysis over a full PRF range.

I. Introduction

This paper reports the technical details and first applications of the ULA-OP 256, a novel open platform whose main features were presented at the 2015 IUS [1].

The development of ULA-OP 256 has been encouraged by the upward demand of ultrasound research, which needs increasing data storage capability, operating flexibility and computational power. All of these features characterize the Synthetic Aperture Real-time Ultrasound System (SARUS), a unique research platform located at the University of Denmark, which is connected to a 144-core Linux cluster to manage up to 1024 channels and 128 GB of acquired data [2]. Commercial research scanners, even though not as powerful and flexible as the SARUS, are also available for laboratory and clinical use. For example, BK Ultrasound (Richmond, BC, Canada) provides its top level diagnostic system, SonixTouch Research, with an ultrasound research interface (URI) that enables research-specific capabilities, such as modification of low-level parameters and beam sequencing. The system needs to be coupled to the SonixDAQ parallel channel data acquisition tool for acquiring raw ultrasound data [3]. Another example is the Aixplorer system (Supersonic Imagine, Aix en Provence, France), which stands out for its high-frame rate (HFR) capability that enables ultrafast Doppler imaging and dynamic elastography. Its research version allows the acquisition of raw radio frequency (RF) data either before or after beamforming and the acquisition of HFR images for offline processing [4]. One of the most powerful commercial research scanners is the Vantage (Verasonics, Redmond, WA) that controls up to 256 independent channels. It offers programmable transmission based on three-level transmitters and allows access to the pre-beamforming samples. The Vantage system also provides a Matlab® interface to modify the signal and image processing parameters and to incorporate custom algorithms [5]. Another interesting example of open scanner using graphics processing units (GPU) for real-time data processing is described in [6].

The ULtrasound Advanced Open Platform (ULA-OP), which was entirely developed at the University of Florence, is a research system characterized by an accurate balance among computational power, cost, dimensions and flexibility [7]. By using only five Field Programmable Gate Arrays (FPGAs) and one Digital Signal Processor (DSP), it was possible to integrate all of the needed electronics in two programmable boards, coupled to a host PC through USB 2.0 link. A powerful, compact and relatively cheap architecture, capable of finely controlling 64 elements selected out of a 192-element array probe, was thus obtained. The ULA-OP was made available to the ultrasound community and was intensely used in several laboratories worldwide. The fields of application of the system span from vector Doppler investigations [8]–[12] to motion estimation [13]–[16], from nonlinear contrast imaging [17]–[19] to tissue characterization [20]–[22]. The system was also employed to develop innovative beamforming schemes to improve the image quality [23], [24] to compensate for the refraction induced by skull bone [25]–[27] as well as to validate novel ultrasound image formation models [14], [28].

Although successfully used in so many different applications, ULA-OP is not sufficiently powerful to satisfy all requirements of emerging methods such as, for example, real-time HFR [29]–[32] or vector Doppler imaging [10]. We thus decided to start the development of ULA-OP 256, a novel platform characterized by the highest number of channels and computational power compatible with mobility, which is a fundamental feature to make the system usable in different laboratories. In addition, the new scanner should have a flexibility at least comparable with that of SARUS: in particular, all of the active probe elements should be individually controllable, in both transmit (TX) and receive (RX) modes.

The design and implementation of ULA-OP 256 are described in this paper, which is organized as follows. The major criteria that inspired the new system design are reported in Sec. II. The modular hardware architecture of ULA-OP 256, based on the use of multiple 32-channel boards, is then described in Sec. III, which also includes a description of the software and firmware associated with the new system. Sec. IV summarizes the main technical features that have been tested so far and presents the first results obtained by the on board high power arbitrary waveform generator and by the high-speed sequential beamformer for HFR and vector Doppler applications. The conclusions and the prospects for this work are discussed in Section V.

II. Rationale

Some of the emerging ultrasound research topics require to finely manage a number of channels higher than 64 and/or to have high computational power available for possible real-time implementations.

A large number of channels is needed, for example, in the control of bi-dimensional (2D) arrays for volumetric imaging applications [33]. Since it is impossible to directly control thousands of elements, this number can be reduced by means of the so-called micro-beamformers [34], [35], or by reducing the array sampling in sparse array probes [36]–[39]. In both cases, it is convenient that the number of channels, N, is maintained as high as possible. By taking into consideration the dimensions and weight that would be assumed by the corresponding cable, N was set equal to 256 in the new scanner. Such a high number of probe elements should be connected to an equal number of high power arbitrary waveform generators and flexibly grouped in different TX and/or RX sub-apertures.

High computational power is required, for example, in high frame rate imaging approaches, which transmit defocused beams, or even plane waves, and simultaneously recover multiple image lines by applying parallel beamforming in receive [29]. The required processing power thus linearly increases with the number of “parallel” lines to be beamformed. In the new system, a high processing power should be achieved by distributing a number of computational resources (FPGAs, DSPs) in different modules, each controlling a limited number (32) of elements (modular architecture). The direct access to the FPGAs and DSPs, as well as to the correlated PC software, should grant the possibility of implementing a large class of real-time processing modules.

III. Methods

A. Hardware organization

1). General architecture of ULA-OP 256

The modular architecture of the new system is illustrated in Fig. 1. The architecture is based on the use of multiple front-end (FE) modules, each capable of controlling 32 elements of the probe. 256 ultrasound channels are thus managed by 8 FE boards, although a lower number of boards can be used in association to specific probes with less elements (for example, only 6 boards are needed for managing 192-element arrays, while 4 boards are sufficient to control the probes with 128 elements).

Fig. 1.

General architecture of ULA-OP 256

All of the FE boards are interconnected by a SerialRapidIO (SRIO) link running at 5 Gbit/s on 4 lanes. Each board has 4 SRIO interfaces that are interconnected through a ring running on the system backplane with bandwidth of 80 Gbit/s - full duplex. The use of a high speed serial bus is mandatory when interconnecting a high number of devices, since the performance of parallel buses rapidly decreases with increasing load and bus length. The ring topology, although less performing than a star connection, is also more easily routable.

The acquisition and synchronization clock, with frequency of 78.125 MHz, is generated on the backplane. A low-jitter phase locked loop (PLL) buffer (AD9522-4, Analog Devices) distributes the clock to the FE boards. All the clock lines were skew-matched and routed using differential lines to ensure high noise rejection. To avoid any beat frequency, all other clocks on the system were chosen to be multiple of the acquisition clock.

The Master Control (MC) board, which is housed on the same backplane, hosts a DSP from the 320C6678 family (Texas Instruments, Austin, TX, USA) that features eight cores running at 1.2 GHz and addresses up to 8 GB of DDR3 memory. The MC board collects and, when needed, further processes the data arriving from the multiple FE boards. Furthermore, it manages the interaction between the system and a host PC by means of an USB 3.0 SuperSpeed connection. For such connection, a speed up to 200 Mbyte/s was experimentally verified.

2). 32-channel Front-End board

The FE module integrates the electronics needed to manage the TX, RX and real-time processing for 32 channels (Fig. 2). As explained in major detail in Sec. IV.A, an FPGA from the ARRIA V GX Family (Altera, San Jose, CA, USA) is programmed to generate 32 bit-streams, each at 468 Mbit/s rate [40], in Low voltage Differential Signaling (LVDS) format. These bit streams are sent to a daughterboard, where each one is converted to an analog driving waveform for a piezoelectric element by a low pass filter and a power amplifier. Integrated T/R switches are used to steer the 32 ultrasound echo-signals back to the FE board, where they are amplified and digitized at 78.125 MSPS with 12-bit resolution by four 8-channel ultrasound front-end integrated circuits (AFE5807, Texas Instruments). The same FPGA (hereinafter called Beamforming FPGA) performs the delay and sum operations required to beamform 32 channels (see Sec.IV.B).

Fig. 2.

The Front-End board. Top: Block-diagram; Bottom: view of the upper board side showing the FPGA (on the right) and the two DSPs. The high power TX and T/R switch board is hosted by the high-speed connectors indicated by white arrows.

The FE board hosts two DSPs from the 320C6678 family. These DSPs are in charge of classic real-time processing operations such as coherent demodulation and filtering, but further custom processing modules, such as compounding, cross-correlation and Fast Fourier Transform (FFT), can be added at firmware level.

3). Data storage

Ultrasound researchers are frequently interested in designing a transmit sequence, acquiring the channel data, transferring the raw unprocessed data to the PC and performing RF processing, such as beamforming and/or image reconstruction on the PC offline. By taking this motivation into account, we designed the system with a large amount of memory. Each FE DSP is equipped with a standard 4GB DDR3 Small Outline Dual In-line Memory Module (SO-DIMM). The Beamforming FPGA directly controls 2 GB of memory. When 8 FE modules are installed on the system, a total of 80 GB of DDR3 memory is thus available. The system memory can be expanded to 144 GB by means of specially designed DDR3 SO-DIMMs having a capacity of 8 GB each. Such a huge memory allows storing significant amount of RF raw data, post beamformed RF data or quadrature demodulated (baseband) signals. Depending on the pulse repetition interval (PRI) and on the depth of the region of interest, up to 30 s of raw data can be saved.

4). Probe adapter module

For research systems, the possible connection with different probes, even if using different types of connectors, might be really important. The general solution to this problem is the use of external adapters. This solution has a major drawback when looking to the signal to noise ratio, since the insertion of an additional external board can add a significant contribution of noise pick-up. For this reason, we designed an internal, fully shielded and replaceable probe adapter board. Such board contains the specific probe connector that is routed toward an array of high-speed shielded connectors. These fit a corresponding array that is housed on the system backplane. The adapter board can be replaced by simply pulling it out of the backplane connectors. This solution allows high flexibility in probe selection while maintaining good signal to noise ratio. Two adapters were so far developed. One implements an ITT Cannon DL5-260R receptacle, which is compatible with the previous ULA-OP system and allows connecting 192-elements linear, convex and phased-array probes. The second adapter hosts a DLP408R receptacle with 408 available contacts, which allows using 256-elements probes. Fig. 3 shows both sides of the 256 channels adapter.

Fig. 3.

Probe adapter module for 256-element probes. Top: the shielded DLP408R connector; bottom: the shielded connectors that mate with the system backplane.

B. Software/firmware organization

1). General software control

ULA-OP 256 is managed by a modular and reconfigurable software, written in C++, running on the host PC. This software, which is a direct porting of the previous ULA-OP real-time software, initializes the hardware upon startup, provides a user-friendly interface and displays the results of ultrasound data processing on the screen.

Proper text configuration files, easily adjustable by the user, provide the main startup settings for all of the resources in the system. Once ULA-OP 256 is configured, the acquisition begins and the real-time processing results are displayed on the screen. Depending on the configuration, the main window of the software may be split into multiple display frames, each one coupled to a different processing module. Directly from the toolbar of the main software interface, visible in Fig. 4, the user can freeze the system, change the pulse repetition frequency (PRF), save the acquired signals or start screen capturing. Moreover, some parameters of specific processing modules can be adjusted through appropriate control panels. All panels are hidden by default, but can be shown on the screen through a mouse click. For example, Fig. 4 shows the panel that allows modifying, for displayed B-Mode images, the noise rejection threshold and the dynamic range, the video filtering level, the scale range of graphs, and, finally, to turn on/off contrast enhancement methods. Every time the software needs to configure or to change any parameter in the system, a command is transmitted by the PC through the USB link, and received by one of the ULA-OP 256 DSPs. The DSP interprets the command and either modifies some of its internal parameters or forwards the adjustment to the appropriate programmable device(s).

Fig. 4.

Screenshot of the real-time interface frozen during a HFR exam of the common carotid artery of a healthy volunteer. The panel on the top-right allows adjusting B-mode image settings such as the noise rejection threshold, the dynamic range, the video filtering level, and contrast enhancement.

Real-time acquisition, processing and streaming of the ultrasound data to the host PC are managed by the software as concurrent processes. RF or baseband signals are continuously saved on the DDR memories: it is possible to stop the acquisition at any time and the data so far collected can be stored into binary files. The data streaming operation is currently possible, within the speed limits of USB 3.0 connection, for both baseband and video data, while streaming of RF beamformed data is under development.

2). DSP firmware

Similar to the software running on the host PC, the previous DSP firmware was adapted to run on ULA-OP 256. Although this preliminary version of the firmware does not fully exploit the new system’s multi-core architecture, it sped up the test and the development of the platform, making available a first library of 18 processing algorithms.

The adapted firmware is run by the DSP present on the MC board, whilst the multiple DSPs on the FE boards run a new, purposely-developed, firmware. According to a programmable parameter set that may be changed every PRI, they can simultaneously demodulate on quadrature channels, filter and downsample the digital signal, to produce blocks of up to 512 complex base-band samples. Finally, the FE DSPs send the samples to the DSP onto the MC board. The MC DSP features concurrent processing modules which the user may activate, to implement standard algorithms, such as B-Mode imaging and Doppler spectral analysis, as well as advanced methods such as Multigate Spectral Doppler [41], strain imaging [21] and pulse compression [42].

The cumulative processing power of the installed DSPs is 5000 GMAC or 2500 GFLOP, and since this computational power is distributed among several boards, as well as among multiple cores inside each device, a flexible firmware architecture is necessary, and is currently under development, to allow the implementation of most complex and processing power demanding algorithms. All resources are hierarchically balanced and no master device is assumed a priori. Each of the 128 available cores is capable of receiving and transmitting data packets from and to any other, while inside a single core the processing can involve multiple modules, each one embedding a different task. A firmware package was specifically developed to create a communication engine that provides the modules with a common interface that abstracts from the hardware peripherals and connects them by means of a data channel. The channel provides a direct wakeup of the receiving module, when the two modules reside in the same core and no effective data movement is needed, and either involve a DMA (Direct Memory Access) transfer or a SRIO stream, in case the modules reside in different cores of the same device or in case the modules reside in two separate devices, respectively.

An advanced processing chain, made up of several distinct modules spanning multiple devices, can thus be realized, possibly joining or splitting data flows, depending on the application requirements and the provided user configuration.

IV. Results

A first prototype system was assembled and extensively tested in our laboratory by using both 192- and 256-element linear array probes. TABLE I summarizes the main technical features of the system that have been tested so far. The following sub-sections describe some of the main ultrasound functionalities that are already active in the new system.

Table I. ULA-OP 256 MAIN FEATURES.

General features	Open platform 32 to 256 independent TX/RX channels Size: 34 x 30 x 26 cm
Transmitter	32 to 256 arbitrary waveform generators based on a 468 Mbit/s sigma-delta bitstream Max output voltages: 200 Vpp Frequency: 1 to 20 MHz
Receiver	Bandwidth: 1 to 30 MHz Analog Gain: 2 – 54 dB with 0 to 40dB programmable TGC 12 bit @ 78.125 MSPS ADCs
Beamformer	Programmable apodization and delays (dynamic focusing delay and sum beamformer, with resolution of 1/16 of the sampling period)
Processing modules	Coherent demodulation, band-pass filtering, downsampling, B-mode, multigate spectral Doppler, vector Doppler, etc (with possibility to create new custom modules by the end user)
Storage capabilities	Up to 144 GB for RF (pre- or postbeamformed) and baseband data

A. Transmission of high-power arbitrary waveforms

One of the key features of the system is represented by the capability of independently generating arbitrary waveforms to be transmitted to the probe elements. This generation is based on a sigma-delta modulator that produces one bit-stream for each channel [40]. The bit-streams are outputted at 468 Mbit/s rate by the FPGA in LVDS format. The LVDS signals are converted to analog by second order low-pass filters and amplified by an array of MAX14807 (Maxim Integrated, San Jose, CA, USA) up to 200 Vpp.

Fig. 5 shows an example of a 3μs-long linear frequency modulated chirp with 50% Tukey tapering, 8 MHz central frequency, and 10 MHz bandwidth, where the peak-to-peak amplitude was set to 80 Vpp. A 100 Ohm resistive load was placed at the output of the transmitter, directly on the probe connector. In particular, Fig. 5 (top) shows the actual excitation signal acquired on the probe connector of the ULA-OP 256 by a digital oscilloscope (TDS 5104, Tektronix Inc., Beaverton, OR, USA). Although the time domain signal might look distorted, Fig. 5 (bottom) shows that the actual spectrum exhibits the in band flatness expected in presence of the tapering window. A maximum difference of only 1dB between the expected and acquired spectral components was observed for the highest in-band frequencies, according to the low-pass behavior of the transmit circuit.

Fig. 5.

Designed (dotted lines) and acquired (continuous lines) linear FM excitation signal with Tukey tapering (top) and related spectra (bottom).

B. Real-time beamforming

The user can choose to downsample the acquired channel data by a factor 2, 3 or 4, before beamforming. This option can speed up processes like the multi-line beamforming described in the next paragraph. Whichever is the frequency of input data, the beamformer implements an interpolation stage with a delay resolution of 1/16 of the sampling period.

Real-time beamforming is performed in two steps. A first step is done inside the Beamforming FPGA of each FE board, and concerns a group of echo-signals from 32 elements, which are suitably weighted, delayed and summed. Full 256-channel beamforming is achieved by suitably summing the contributions from multiple FE boards. The final summation can be performed either by one of the DSPs present on a FE board or by the DSP hosted on the MC board.

C. Multi-line beamforming

A first pilot application was developed to test the overall system capabilities. The Beamforming FPGA was programmed to operate a single sequential multi-line beamforming [43]. Channel raw RF data are stored in a buffer inside the FPGA to be read-out at 234 MHz and beamformed multiple times by the same FPGA. One of the front-end DSPs is in charge of receiving the beamformed data and of performing coherent signal demodulation and low-pass filtering before sending them to the MC board. The MC DSP finally implements the B-mode image reconstruction and the streaming of images to the PC.

A first high-frame rate imaging test was done by transmitting plane waves and beamforming a full frame of 96 lines by 2048 points, between two consecutive transmissions. By including the time needed by the FPGA to acquire the received echo data and to read the up-to-date beamforming parameters for each step, the PRI was limited to 1.388 ms. Accordingly, full B-mode images were obtained at 720 Hz frame rate in real-time. Fig. 4 shows one screenshot obtained freezing one of these images.

A massively parallel multi-line beamformer was also implemented and is currently under test. The Beamforming FPGA can embed up to 8 parallel beamformers, thus potentially sustaining an output sample rate of 1.8 GSPS, which can be handled by the FPGA SRIO interface (32 Gbit/s of sustained transfer rate). Considering, for example, the same image size of 96 lines by 2048 depths, as above, this rate will enable imaging rates up to 3000 frames/s.

D. Real-time vector Doppler processing

The sequential multi-line beamforming capability was here exploited to test the possibility of improving the performance of the dual-beam angle tracking vector Doppler procedure described in [8]. In the previous implementation, the reference and the measurement beams were transmitted (and the related echoes were received) by two separate apertures in two consecutive PRIs. This yielded a halved PRF, which may generate aliasing during in vivo measurements.

ULA-OP 256 was programmed to fire a single transmission along the reference line. The backscattered signals are simultaneously gathered by two separate apertures and properly beamformed during the same reception interval. With this single-transmitter double- receiver scheme [44], Doppler data are collected at PRF rate for both the reference and the measurement beams. The multi-line beamformer is run twice in the same PRI, each time with a distinct pattern for the delays and apodization coefficients, so that the FPGA delivers a set of beamformed samples for both Doppler lines at double slow-time sampling rate compared with the previous implementation.

The use of separate apertures for TX and RX permitted by the ULA-OP 256 flexibility also lowers, according to the Doppler equation, the amount of phase shift assessed during flow investigations. Though this may reduce the accuracy of the measurement, it also further reduces the chance of aliasing in case of fast flow, which is often observed in conjunction with stenosis.

The improved angle tracking procedure was tested in-vivo on the carotid artery of a healthy volunteer. Fig. 6 shows a screenshot frozen during the real-time test. The B-Mode image (upper left) highlights the directions of the TX-RX reference beam (yellow line) and of the RX measurement beam (cyan line). The symmetric spectrogram on the top-right, related to the sample volume at the crossing of the above lines, shows that the flow is transversely oriented to the reference line. The second spectrogram (bottom-right) related to the Doppler signal detected by the measurement aperture from the same sample volume, can thus be converted to velocity by the Doppler equation in which the flow direction is known. The result is shown in the bottom panel. The ULA-OP 256 was here programmed to use an 8 kHz PRF, sufficient to assess blood velocities as high as nearly 1 m/s.

Fig. 6.

The B-Mode image shows the carotid artery of a healty volunteer and the position of the two inspecting Doppler lines. The spectrograms on the right are obtained from the reference line (top) and the measuring line (bottom), respectively. The velocity profile estimated through the measuring line is shown on the bottom.

V. Discussion and conclusion

The ULA-OP 256 is based on a modular architecture that integrates all the electronics needed to store huge amounts of RF or baseband data and to perform computationally intensive processing algorithms. Compared to the previous ULA-OP, the new platform manages a higher number of active channels (256 vs 64), which can be individually controlled to deliver different high voltage arbitrary waveforms to the corresponding probe elements. The processing power is hugely enhanced by the use of multiple last generation devices instead of the 5 FPGAs and the single DSP used in ULA-OP. In addition, the on board memory is increased from 1 GB to 80 GB. In spite of the extensive use of high-end devices, the system was maintained repeatable and mobile thanks to the efforts addressed to integrate the electronics in a limited number of boards. The final overall dimensions (34×30×26 cm) make the system suitable to be moved to other labs [45], so that different fundamental and pre-clinical research needs may be fulfilled.

Special attention was dedicated to the possibility that external users may want to develop their own imaging schemes, signal processing or image reconstruction methods. The transmission and reception strategy, as well as the size of the acquisition buffers at any point of the reception chain, can be easily programmed. Furthermore, software and firmware sources will be made available to experienced developers to facilitate the development of novel real-time algorithms, even though the used devices (FPGA and DSPs) are more demanding, in terms of programming skills, than the architectures based, for example, on GPUs [6], [46].